How I can stop index my sites with robots.txt by Yahoo, msn, excite, all-the-web, altavista, alexa ? I make this - all is correct ? I need only Google and many local SE.
If you would like to not allow certain bots to index your pages, you can refer to bots by their name. For example, many webmasters prefer not to allow Google's Image Search or MSN's Pic Search bots to index any of the images on their site. So you would use the following. User-Agent field you list the name of the bot. Disallow field we use the forward slash / to prevent indexing. User-agent: Googlebot-Image Disallow: / User-agent: psbot Disallow: /
Zafar Ahmed, I know this. But I give my code - is it correct for search engines Yahoo, msn, excite, all-the-web, altavista, alexa ? I don't sure that I know thei bots correct name.
kevster1, please think before write. I don't want to index by Yahoo, msn, excite, all-the-web, altavista, alexa. Not - ALL search engines. * - stop index by ALL SE !
Use this code in your robots.txt User-agent: Slurp //For yahoo bot Disallow: / User-agent: msnbot // For MSN bot Disallow: / User-agent: ArchitextSpider //For Exite bot Disallow: / User-agent: FAST //For Alltheweb bot Disallow: / User-agent: Scooter //For Altavista Bot Disallow: / User-agent: ia_archiver //For Alexa Bot Disallow: / // now allow rest of search engines User-agent: * Disallow: