Bots that you disallow because they eat bandwidth

[websiteideas]

Are there any bots that you disallow because all they do is eat up your bandwidth and deliver you no visitors? If so, which ones do you disallow and what does your robots.txt file look like? If you would, please post the code for your robots.txt file here.

[mjamesb]

I stop a few bots...one for example is
User-agent: linksmanager_bot
I did link exchanges with some sites that use this service. Sorry but a few links isn't worth the stupid spider hitting my site a thousand times in a week.

[websiteideas]

I'm surprised that bot would hit your entire site? What's the purpose? It's trying verify that you have reciprocated the link?

[tpn87]

I only let the major engines in, the ones that are going to provide traffic to my site.


User-agent: Googlebot
Disallow:
User-agent: MSNBot
Disallow:
User-agent: Inktomi Slurp
Disallow:
User-agent: Slurp
Disallow:
User-agent: Teoma
Disallow:
User-agent: *
Disallow: /

[MatthewN]

None at the moment. I run a dedicated server and have plenty of bandwith at the moment though, so no worries there.

[minstrel]

I only let the major engines in, the ones that are going to provide traffic to my site.


User-agent: Googlebot
Disallow:
User-agent: MSNBot
Disallow:
User-agent: Inktomi Slurp
Disallow:
User-agent: Slurp
Disallow:
User-agent: Teoma
Disallow:
User-agent: *
Disallow: /

1. that's sort of a self-fulfilling prophecy, isn't it? sicne you disallow other SE bots, how are they ever going to provide you with traffic in the future? a rather short-sighted solution

2. the major problem bots aren't going to pay any attention to your robots.txt file - what you've just done is turn away some well-behaved bots to make more room for the badly behaved ones.

In general, I don't think you gain anything by banning bots in robots.txt - you really should only be using it to disallow indexing of certain folders or files.

[tradealoan]

yes allow the major search engines only as they are the one who delivers you the traffic.

[minstrel]

yes allow the major search engines only as they are the one who delivers you the traffic.
That's bad advice. Re-read my previous post.

[websiteideas]

I've found that the following three lines in my robots.txt file can save my server from getting loaded down by this bot that doesn't ever bring me any traffic.

User-agent: BecomeBot
Crawl-Delay: 30
Disallow: /cgi-bin

Anyone else notice this bot just eats their bandwidth and doesn't ever bring them traffic?

[RectangleMan]

BecomeBot is the worst...

[tpn87]

BecomeBot is the worst...


Bandwidth eater....

[websiteideas]

Does anyone find that allowing this bot actually brings them more traffic?

[websiteideas]

Here's a few more lines you might want to add:


User-agent: WebStripper
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: Offline Explorer
Disallow: /

[tbfilly]

Does anyone know anything about the Alexa bot? Has shown up to crawl my site, Knowledge Creates Power. I'll be on the lookout for some of the ones Everyone has mentioned, Thanks!
Cheers

[minstrel]

What are you asking? What the Alexa bot is? or if you should ban it?

I don't understand the rush to ban bots. First, on an average site (let's face it - most of the member's sites here are not Microsoft or even Webmasterworld), how much of a problem can it be. In North America, at least, bandwidth these days is pretty cheap. And most of the suggested bots banning robots.txt files I've seen ban bots that simply should not be banned (not long ago, someone asked me to look at his site to find out why Google wasn't indexing it - pre-Big Daddy - and it turned out he'd been banning user agents and inadvertently included Googlebot and Slurp and probably MSNbot).

[websiteideas]

The problems usually arise when you have a huge site with thousands of pages like an Amazon clone or a huge article directory. For most people, you're right - banning bots is not something they need to rush to do.