Name of yahoo, google and msn bot

Discussion in 'robots.txt' started by blabla11, Aug 8, 2007.

  1. #1
    I want to dissallow a folder (and it's subfolders) from crawling by these 3 bots

    What to write in robots.txt?
     
    blabla11, Aug 8, 2007 IP
  2. blabla11

    blabla11 Peon

    Messages:
    164
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #2
    User-agent: ??
    Disallow: /private/

    What to write as user-agent if I want to dissallow google, yahoo and msn (not all robots, just these 3)

    Please help!
     
    blabla11, Aug 9, 2007 IP
  3. kop16

    kop16 Peon

    Messages:
    28
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    User-Agent: googlebot
    User-Agent: msnbot
    User-Agent: yahoobot
     
    kop16, Aug 18, 2007 IP
  4. evera

    evera Peon

    Messages:
    283
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #4
    More Detailed:

    GOogle:
    DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0) (216.239.39.x)
    Generic Mobile Phone (compatible; Googlebot-Mobile/2.1)
    Google Talk
    Googlebot-Image/1.0 (66.249.72.xxx)
    Googlebot-Image/1.0 (66.249.72.xxx)
    Googlebot/2.1 66.249.64.XXX
    Googlebot/2.1 66.249.64.XXX
    Googlebot/Test 66.249.64.XXX
    gsa-crawler (Enterprise; GID-01422; ) (216.239.xx.xx)
    gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com) (216.239.xx.xx)
    gsa-crawler (Enterprise; GIX-02057; ) (212.35.100.1xx)
    gsa-crawler (Enterprise; GIX-03519; ) (129.41.20.1xx)
    gsa-crawler (Enterprise; GIX-0xxxx; ) (216.239.xx.xx)
    KDDI-SN22 UP.Browser/6.0.7 (GUI) MMP/1.1 (Google WAP Proxy/1.0) (216.239.33.x)
    Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html)
    Mozilla/4.0 (MobilePhone SCP-5500/US/1.0)
    NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1) 66.249.66.xxx
    Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE
    (compatible; Googlebot/2.1) 66.249.66.xxx
    Mozilla/5.0 (compatible; Googlebot/2.1)
    Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com
    Nokia7110/1.0 (05.01) (Google WAP Proxy/1.0)

    MSN:
    msnbot-media/1.0
    msnbot-Products/1.0
    MSNBOT/0.xx 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx
    msnbot/x.xx 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx
    MSNPTC/1.0 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx

    Yahoo:
    DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0) (203.216.197.xxx)
    Mozilla/4.0 Yahoo Mindset (66.228.182.1xx)
    Mozilla/4.0 (compatible; crawlx, ) (68.142.211.1xx)
    Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid) Ya(203.141.52.)
    Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya) (202.93.76.xx)
    SpiderMan (202.165.102.xxx)
    Y!J-BSC/1.0 (211.14.8.2xx)
    Y!J-SRD/1.0 (203.216.197.xxx)
    Y!J/1.0 (211.14.8.2xx)
    Y!OASIS/TEST no-ad Mozilla/4.08 [en](X11; I; FreeBSD 2.2.8-STABLE i386)
    Y!TunnelPro Y!TunnelPro
    Yahoo! Mindset Yahoo Mindset (66.228.182.1xx)
    Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5 ) (209.191.83.1xx)
    Yahoo-MMAudVid/1.0
    (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com) (206.190.43.xx)
    Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com) (66.77.73.xx)
    Yahoo-Test/4.0
    Yahoo-VerticalCrawler-FormerWebCrawler/
    3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler (66.77.73.3x)
    YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5
    YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5) ( 68.142.195..x)
    YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5) ( 66.196.93.x)
    YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5) ( 66.196.93.x)
    YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5) ( 66.196.93.x)
    YahooSeeker/bsv3.9 (compatible; Mozilla 4.0; MSIE 5.5) ( 68.142.195..x)
    YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD
    ;cafekelsa-dev-webmaster@yahoo-inc.com ) (64.157.137.xxx)
    Nutch-0.9-dev Unknown Yahoo robot

    Altavista (Yahoo):
    AltaVista Intranet V2.0 AVS EVAL
    AltaVista Intranet V2.0 Compaq Altavista Eval
    AltaVista Intranet V2.0 evreka.com
    AltaVista V2.0B AV Fetch 1.0
    AVSearch-3.0(AltaVista/AVC)
    Scooter-3.0.EU
    Scooter-3.0.FS
    Scooter-3.0.HD
    Scooter-3.0.VNS
    Scooter-3.0QI
    Scooter-3.2
    Scooter-3.2.BT
    Scooter-3.2.DIL
    Scooter-3.2.EX
    Scooter-3.2.JT
    Scooter-3.2.NIV
    Scooter-3.2.SF0
    Scooter-3.2.snippet
    Scooter-3.3dev
    Scooter-ARS-1.1
    Scooter-ARS-1.1-ih
    scooter-venus-3.0.vns
    Scooter-W3-1.0
    Scooter-W3.1.2
    Scooter/1.0
    Scooter/1.0
    Scooter/1.1 (custom)
    Scooter/2.0 G.R.A.B. V1.1.0
    Scooter/2.0 G.R.A.B. X2.0
    Scooter/3.3
    Scooter/3.3.QA.pczukor
    Scooter/3.3.vscooter
    Scooter/3.3_SF
    Scooter2_Mercator_x-x.0
    Scooter_bh0-3.0.3
    Scooter_trk3-3.0.3

    Inktomi (Yahoo):
    Inktomi Search
    Mozilla/3.0 (Slurp/cat) 72.30.61.xx(x)
    Mozilla/3.0 (Slurp/si) 72.30.61.xx(x)
    Mozilla/5.0 (compatible; Yahoo! Slurp China) (202.160.180.xxx)
    Mozilla/5.0 (compatible; Yahoo! Slurp) (via 66.196.xx.xxx)
    Mozilla/5.0 (Slurp/cat; )
    Mozilla/5.0 (Slurp/si; )
    Slurp/2.0 (slurp@inktomi.com)
    Slurp/2.0-KiteWeekly (slurp@inktomi.com)
    Slurp/si (slurp@inktomi.com)
    Slurpy Verifier/1.0 72.30.61.xx(x)
     
    evera, Aug 18, 2007 IP
  5. trichnosis

    trichnosis Prominent Member

    Messages:
    13,785
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #5
    it's an intresting request.

    why do you need to block google and yahoo from your sites?
     
    trichnosis, Aug 21, 2007 IP
  6. n3o_the_on3

    n3o_the_on3 Well-Known Member

    Messages:
    1,422
    Likes Received:
    62
    Best Answers:
    0
    Trophy Points:
    165
    #6
    Yep Trichnosis! its very interesting. why do you want to block yahoo and google?
     
    n3o_the_on3, Aug 29, 2007 IP
  7. cheapez

    cheapez Active Member

    Messages:
    1,123
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    78
    #7
    Anyone know the name of the Google bot that crawls the Adsense on your site?
     
    cheapez, Sep 2, 2007 IP
  8. evera

    evera Peon

    Messages:
    283
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Its:
    User-agent: Mediapartners-Google
     
    evera, Sep 2, 2007 IP
  9. blabla11

    blabla11 Peon

    Messages:
    164
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #9
    I'm testing BH script and don't want to get IP banned
     
    blabla11, Sep 4, 2007 IP
  10. seoworld

    seoworld Active Member

    Messages:
    471
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    58
    #10
    Are you looking to block these three bots only? or other bots as well?

    Because if you want to block bots in general then just write the following

    User-agent: *
    Disallow: /whatever/
     
    seoworld, Sep 11, 2007 IP