I want to dissallow a folder (and it's subfolders) from crawling by these 3 bots What to write in robots.txt?
User-agent: ?? Disallow: /private/ What to write as user-agent if I want to dissallow google, yahoo and msn (not all robots, just these 3) Please help!
More Detailed: GOogle: DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0) (216.239.39.x) Generic Mobile Phone (compatible; Googlebot-Mobile/2.1) Google Talk Googlebot-Image/1.0 (66.249.72.xxx) Googlebot-Image/1.0 (66.249.72.xxx) Googlebot/2.1 66.249.64.XXX Googlebot/2.1 66.249.64.XXX Googlebot/Test 66.249.64.XXX gsa-crawler (Enterprise; GID-01422; ) (216.239.xx.xx) gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com) (216.239.xx.xx) gsa-crawler (Enterprise; GIX-02057; ) (212.35.100.1xx) gsa-crawler (Enterprise; GIX-03519; ) (129.41.20.1xx) gsa-crawler (Enterprise; GIX-0xxxx; ) (216.239.xx.xx) KDDI-SN22 UP.Browser/6.0.7 (GUI) MMP/1.1 (Google WAP Proxy/1.0) (216.239.33.x) Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html) Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1) 66.249.66.xxx Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1) 66.249.66.xxx Mozilla/5.0 (compatible; Googlebot/2.1) Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com Nokia7110/1.0 (05.01) (Google WAP Proxy/1.0) MSN: msnbot-media/1.0 msnbot-Products/1.0 MSNBOT/0.xx 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx msnbot/x.xx 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx MSNPTC/1.0 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx Yahoo: DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0) (203.216.197.xxx) Mozilla/4.0 Yahoo Mindset (66.228.182.1xx) Mozilla/4.0 (compatible; crawlx, ) (68.142.211.1xx) Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid) Ya(203.141.52.) Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya) (202.93.76.xx) SpiderMan (202.165.102.xxx) Y!J-BSC/1.0 (211.14.8.2xx) Y!J-SRD/1.0 (203.216.197.xxx) Y!J/1.0 (211.14.8.2xx) Y!OASIS/TEST no-ad Mozilla/4.08 [en](X11; I; FreeBSD 2.2.8-STABLE i386) Y!TunnelPro Y!TunnelPro Yahoo! Mindset Yahoo Mindset (66.228.182.1xx) Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5 ) (209.191.83.1xx) Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com) (206.190.43.xx) Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com) (66.77.73.xx) Yahoo-Test/4.0 Yahoo-VerticalCrawler-FormerWebCrawler/ 3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler (66.77.73.3x) YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5 YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5) ( 68.142.195..x) YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5) ( 66.196.93.x) YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5) ( 66.196.93.x) YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5) ( 66.196.93.x) YahooSeeker/bsv3.9 (compatible; Mozilla 4.0; MSIE 5.5) ( 68.142.195..x) YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-webmaster@yahoo-inc.com ) (64.157.137.xxx) Nutch-0.9-dev Unknown Yahoo robot Altavista (Yahoo): AltaVista Intranet V2.0 AVS EVAL AltaVista Intranet V2.0 Compaq Altavista Eval AltaVista Intranet V2.0 evreka.com AltaVista V2.0B AV Fetch 1.0 AVSearch-3.0(AltaVista/AVC) Scooter-3.0.EU Scooter-3.0.FS Scooter-3.0.HD Scooter-3.0.VNS Scooter-3.0QI Scooter-3.2 Scooter-3.2.BT Scooter-3.2.DIL Scooter-3.2.EX Scooter-3.2.JT Scooter-3.2.NIV Scooter-3.2.SF0 Scooter-3.2.snippet Scooter-3.3dev Scooter-ARS-1.1 Scooter-ARS-1.1-ih scooter-venus-3.0.vns Scooter-W3-1.0 Scooter-W3.1.2 Scooter/1.0 Scooter/1.0 Scooter/1.1 (custom) Scooter/2.0 G.R.A.B. V1.1.0 Scooter/2.0 G.R.A.B. X2.0 Scooter/3.3 Scooter/3.3.QA.pczukor Scooter/3.3.vscooter Scooter/3.3_SF Scooter2_Mercator_x-x.0 Scooter_bh0-3.0.3 Scooter_trk3-3.0.3 Inktomi (Yahoo): Inktomi Search Mozilla/3.0 (Slurp/cat) 72.30.61.xx(x) Mozilla/3.0 (Slurp/si) 72.30.61.xx(x) Mozilla/5.0 (compatible; Yahoo! Slurp China) (202.160.180.xxx) Mozilla/5.0 (compatible; Yahoo! Slurp) (via 66.196.xx.xxx) Mozilla/5.0 (Slurp/cat; ) Mozilla/5.0 (Slurp/si; ) Slurp/2.0 (slurp@inktomi.com) Slurp/2.0-KiteWeekly (slurp@inktomi.com) Slurp/si (slurp@inktomi.com) Slurpy Verifier/1.0 72.30.61.xx(x)
Are you looking to block these three bots only? or other bots as well? Because if you want to block bots in general then just write the following User-agent: * Disallow: /whatever/