how to block all other BOT except Google, MSN, Yahoo

[danieloffice]

Hi,

Need help,

in robot.txt, how to exclude all other BOT except Google, MSN, Yahoo?


Thanks

[SwapsRulez]

You have to allow Google, MSN, Yahoo to crawl the whole site & disallow other bots to look into the root also..

here is the sample code..

User-agent: google
Disallow:

User-agent: yahoo
Disallow:

User-agent: msn
Disallow:

User-agent: *
Disallow: /


The above code will match the user agent by checking the substrings of the name of the robot of the perticular system... So that they will work perfectly fine... peace!

[sajidmm]

You have to allow Google, MSN, Yahoo to crawl the whole site & disallow other bots to look into the root also..

here is the sample code..

User-agent: google
Disallow:

User-agent: yahoo
Disallow:

User-agent: msn
Disallow:

User-agent: *
Disallow: /


The above code will match the user agent by checking the substrings of the name of the robot of the perticular system... So that they will work perfectly fine... peace!

wrong info.

i know about google. it should be

User-agent: googlebot
Disallow:

i am not sure about yahoo and msn bots.

[lhughes33309]

Hi,

The only problem with using the robots.txt file, is many bots don't follow the rules and are there to simply scrape the site. You need to use a script like perl that will feed your html etc. to all users and MSN, GOOGLE and Yahoo, but will trap the others and either give them a 500 server error or redirect them some where else.
I have written many scripts to do just that.

Thanks,
lhughes33309

[manish.chauhan]

Check this...This will help you in basic understanding of robots.txt

http://seocrazy.blogspot.com/2008/04/robotstxt-stop.html

[al-zabir]

I think this will help all: :)

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Yahoo-slurp
Disallow:

User-agent: Msnbot
Disallow:

[rozane]

it should be
User-agent: Googlebot
Disallow:



I think this will help all: :)

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Yahoo-slurp
Disallow:

User-agent: Msnbot
Disallow: