robots.txt - How to ban everyone except Google

Hello, I want to make robots.txt that will allow only Google to index web site. So it will be like:

User-agent: Google
Allow: /

User-agent: *
Disallow: /

Code (markup):

But I am not sure if it will allow access to ALL google bots.

Do I need to add separate rule for each type of bot? Like:

User-agent: Mediapartners-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

etc.

And if yes, can you give me the list of all google bots, by User-agent?

PS yes, I understand that many bots ignore robots.txt

 

digital bump

 

You must make a different entry for each of Google's web robots, so yes you must make the entry for the other two. The real question is why would you want to block Yahoo!, Live! Search & msn.

 

This is one of experiments, I don't need other traffic sources. Anyone can help to make a list of bots?

 

But will that work with the *, too? Wouldn't that over-ride the first Allow?

 

It should work. By logic the agents gets matched when it hits Google if it doesn't match it will continue over the list.

 

Google spiders sites good in Live & Yahoo...
Aren't you shooting self in foot?

 

That will not work, you need to address each individual agent, as example:

User-agent: Google
Allow: /

User-agent: Slurp
Disallow: /

User-agent: Msn
Disallow: /

 

http://www.user-agents.org/

 

I dont know what kind of experiment are you doing, but some bad robots will not follow directives from your robots.txt.
If you want to block all bots except google u will need a bad bot trap also.

 

This looks ok to me... thanks windtalker for sharing...

--Ashish

 

No you do not need to allow separate Google bots, just allow only Google, all its bots will be allowed automatically...
Here is the list of Google Bots and also you can find spammy bots here, what you should block to reduce your bandwidth usage and cut down your cost.

 

In that case you can block them through their IP adress by using .htaccess..
I hope..this will help you

 

you have a very 31337 us3rn3m3, ph34r j00.

 

Thats nice, thanks for information, but dont you remember where you read about it?

 

You can find more info at
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40364

 

Thanks, I think thats what I need.

 

????

 

He means You have a very elite username ... fear you? Kinda iffy on that last part. The reason he types in 1337 language is because the creator of this post's username is all in numbers, characters etc.

Jason

 

Thanks to make it clear..