robots.txt - How to ban everyone except Google

[lim (x² - 5x³) = -∞]

Hello, I want to make robots.txt that will allow only Google to index web site. So it will be like:

User-agent: Google
Allow: /

User-agent: *
Disallow: /

But I am not sure if it will allow access to ALL google bots.

Do I need to add separate rule for each type of bot? Like:

User-agent: Mediapartners-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

etc.

And if yes, can you give me the list of all google bots, by User-agent?

PS yes, I understand that many bots ignore robots.txt

[lim (x² - 5x³) = -∞]

digital bump

[ssandecki]

You must make a different entry for each of Google's web robots, so yes you must make the entry for the other two. The real question is why would you want to block Yahoo!, Live! Search & msn. :confused:

[lim (x² - 5x³) = -∞]

This is one of experiments, I don't need other traffic sources. Anyone can help to make a list of bots?

[The Stealthy One]

But will that work with the *, too? Wouldn't that over-ride the first Allow?

[enriquerojas]

It should work. By logic the agents gets matched when it hits Google if it doesn't match it will continue over the list.

[vitalous]

Google spiders sites good in Live & Yahoo...
Aren't you shooting self in foot?

[windtalker]

That will not work, you need to address each individual agent, as example:

User-agent: Google
Allow: /

User-agent: Slurp
Disallow: /

User-agent: Msn
Disallow: /

[ssandecki]

This is one of experiments, I don't need other traffic sources. Anyone can help to make a list of bots?

http://www.user-agents.org/

[guidyy]

I dont know what kind of experiment are you doing, but some bad robots will not follow directives from your robots.txt.
If you want to block all bots except google u will need a bad bot trap also.

[ashisharora_83]

That will not work, you need to address each individual agent, as example:

User-agent: Google
Allow: /

User-agent: Slurp
Disallow: /

User-agent: Msn
Disallow: /

This looks ok to me... thanks windtalker for sharing...

--Ashish

[manish.chauhan]

Hello, I want to make robots.txt that will allow only Google to index web site. So it will be like:

User-agent: Google
Allow: /

User-agent: *
Disallow: /

But I am not sure if it will allow access to ALL google bots.

Do I need to add separate rule for each type of bot? Like:

User-agent: Mediapartners-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

etc.

And if yes, can you give me the list of all google bots, by User-agent?

PS yes, I understand that many bots ignore robots.txt

No you do not need to allow separate Google bots, just allow only Google, all its bots will be allowed automatically...
Here is the list of Google Bots (http://www.iplists.com/) and also you can find spammy bots here (http://seocrazy.blogspot.com/2008/04/spammy-robots-list.html), what you should block to reduce your bandwidth usage and cut down your cost.

[manish.chauhan]

PS yes, I understand that many bots ignore robots.txt
In that case you can block them through their IP adress by using .htaccess..
I hope..this will help you :)

[tekz999]

you have a very 31337 us3rn3m3, ph34r j00.

[lim (x² - 5x³) = -∞]

No you do not need to allow separate Google bots, just allow only Google, all its bots will be allowed automatically...

Thats nice, thanks for information, but dont you remember where you read about it?

[manish.chauhan]

You can find more info at
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40364

[lim (x² - 5x³) = -∞]

You can find more info at
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40364

Thanks, I think thats what I need.

[manish.chauhan]

you have a very 31337 us3rn3m3, ph34r j00.

????:confused:

[jasoncdu]

????:confused:

He means You have a very elite username ... fear you? Kinda iffy on that last part. The reason he types in 1337 language is because the creator of this post's username is all in numbers, characters etc.

Jason

[manish.chauhan]

He means You have a very elite username ... fear you? Kinda iffy on that last part. The reason he types in 1337 language is because the creator of this post's username is all in numbers, characters etc.

Jason
Thanks to make it clear..

[jasp]

That will not work, you need to address each individual agent

This is not true if the robot is standards compliant. I'd be surprised if the big 5 didn't understand the official robots.txt standard.

To allow only specific robots, like those from google you can use

User-agent: *
Disallow: /

User-agent: Googlebot
User-agent: Googlebot-Mobile
User-agent: Googlebot-Image
User-agent: Mediapartners-Google
User-agent: Adsbot-Google
Disallow:



Remembering that the 'allow' directive is non-standard.

[dermax]

I can confirm jasp's solution.