How do we block or slow down WISEnutbot SE Bot? sv-crawlfw4.looksmart.com Its been hammering the site for 4-5 days now
1st of all ask yourself the question do you want more traffic or do you prefer to shut down your site or have it secret - just for friends since you may prefer second option you may disallow ALL bots in your robots.txt - this is done by User-agent: * Disallow: / if you like SOME traffic fut only from certain SE and NOT wisenut and all its sister SE's looksmart, zeal directory then you may exclude in your robots.txt just that one bot a full robots.txt tutorial is here http://www.clockwatchers.com/robots_main.html that also includes the solution to use .htaccess for blocking a particular bot - sme bots may disobey robots.tyt robots.txt for one bot exclusion is User-agent: BotName Disallow: / replace BotName with the bot you want to exclude
Thanks for the information hans, might get me somewhere closer to understanding how all this bot stuff works. Sorry to intrude, but why would anyone want to stop or slow the indexing of their site?
bots are the little "dwarfs" that bring your pages to the SE who then bring you traffic and thus happiness into your life ... be happy to have bots visiting your site your server can surely handle ten thousands of bot visits a day
I figured that they would only improve traffic, hence why I wondered why somebody may wish to delay or stop bots indexing their sites altogether I'm actually getting a fair few hits now from bots, and whilst I'm only really seeing the big names, I certainly won't be complaining about it!
That depends on what bot is crawling Not all bots crawl your site for the sole purpose of indexing your content and sending you visitors. Many bots are not search engine related and only crawl for reasons of their own, so they use your bandwidth and grab your content without ever sending you a single visitor.
For some sites I placed it in my robots.txt User-agent:WISENutbot Disallow:/ If someone have a 50000 pages site all generated from php or perl and mysql, for the few visits a small engine can bring is better to stop instead of slowing the server. I considered looksmart a small engine because for all my sites send very little traffic if any before disallowing it.
Ok next bot hammering us is Search results for: 209.85.32.65 OrgName: Everyones Internet OrgID: EVRY Address: 390 Benmar Address: Suite 200 City: Houston StateProv: TX PostalCode: 77060 Country: US NetRange: 209.85.0.0 - 209.85.127.255 CIDR: 209.85.0.0/17 NetName: EVRY-BLK-15 NetHandle: NET-209-85-0-0-1 Parent: NET-209-0-0-0-0 NetType: Direct Allocation NameServer: NS1.EV1.NET NameServer: NS2.EV1.NET Comment: RegDate: 2005-12-14 Updated: 2006-07-17 Are they good? I dont see why they have done 1600+ they just seem to be using b/w. The referr is XP I.E 6.0?
The company is legit, is a internet service provider, probably someone are using a dedicated server on their datacenter or a dialup account. below is resources to ban robots that not respect the robots.txt file http://www.kloth.net/internet/bottrap.php http://danielwebb.us/software/bot-trap/ http://www.fleiner.com/bots/
My advice is to make a comprensive robots.txt for robots that follow the guidelines of robot.txt, so that only the bad robots use the trap, so you can save more cpu usage by utilizing mod_rewrite only for bad robots.