1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Name of yahoo, google and msn bot

Discussion in 'robots.txt' started by blabla11, Aug 8, 2007.

  1. #1
    I want to dissallow a folder (and it's subfolders) from crawling by these 3 bots

    What to write in robots.txt?
    SEMrush
     
    blabla11, Aug 8, 2007 IP
    SEMrush
  2. blabla11

    blabla11 Peon

    Messages:
    164
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #2
    User-agent: ??
    Disallow: /private/

    What to write as user-agent if I want to dissallow google, yahoo and msn (not all robots, just these 3)

    Please help!
     
    blabla11, Aug 9, 2007 IP
  3. kop16

    kop16 Peon

    Messages:
    28
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    User-Agent: googlebot
    User-Agent: msnbot
    User-Agent: yahoobot
     
    kop16, Aug 18, 2007 IP
  4. evera

    evera Peon

    Messages:
    283
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #4
    More Detailed:

    GOogle:
    DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0) (216.239.39.x)
    Generic Mobile Phone (compatible; Googlebot-Mobile/2.1)
    Google Talk
    Googlebot-Image/1.0 (66.249.72.xxx)
    Googlebot-Image/1.0 (66.249.72.xxx)
    Googlebot/2.1 66.249.64.XXX
    Googlebot/2.1 66.249.64.XXX
    Googlebot/Test 66.249.64.XXX
    gsa-crawler (Enterprise; GID-01422; ) (216.239.xx.xx)
    gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com) (216.239.xx.xx)
    gsa-crawler (Enterprise; GIX-02057; ) (212.35.100.1xx)
    gsa-crawler (Enterprise; GIX-03519; ) (129.41.20.1xx)
    gsa-crawler (Enterprise; GIX-0xxxx; ) (216.239.xx.xx)
    KDDI-SN22 UP.Browser/6.0.7 (GUI) MMP/1.1 (Google WAP Proxy/1.0) (216.239.33.x)
    Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html)
    Mozilla/4.0 (MobilePhone SCP-5500/US/1.0)
    NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1) 66.249.66.xxx
    Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE
    (compatible; Googlebot/2.1) 66.249.66.xxx
    Mozilla/5.0 (compatible; Googlebot/2.1)
    Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com
    Nokia7110/1.0 (05.01) (Google WAP Proxy/1.0)

    MSN:
    msnbot-media/1.0
    msnbot-Products/1.0
    MSNBOT/0.xx 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx
    msnbot/x.xx 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx
    MSNPTC/1.0 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx

    Yahoo:
    DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0) (203.216.197.xxx)
    Mozilla/4.0 Yahoo Mindset (66.228.182.1xx)
    Mozilla/4.0 (compatible; crawlx, ) (68.142.211.1xx)
    Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid) Ya(203.141.52.)
    Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya) (202.93.76.xx)
    SpiderMan (202.165.102.xxx)
    Y!J-BSC/1.0 (211.14.8.2xx)
    Y!J-SRD/1.0 (203.216.197.xxx)
    Y!J/1.0 (211.14.8.2xx)
    Y!OASIS/TEST no-ad Mozilla/4.08 [en](X11; I; FreeBSD 2.2.8-STABLE i386)
    Y!TunnelPro Y!TunnelPro
    Yahoo! Mindset Yahoo Mindset (66.228.182.1xx)
    Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5 ) (209.191.83.1xx)
    Yahoo-MMAudVid/1.0
    (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com) (206.190.43.xx)
    Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com) (66.77.73.xx)
    Yahoo-Test/4.0
    Yahoo-VerticalCrawler-FormerWebCrawler/
    3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler (66.77.73.3x)
    YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5
    YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5) ( 68.142.195..x)
    YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5) ( 66.196.93.x)
    YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5) ( 66.196.93.x)
    YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5) ( 66.196.93.x)
    YahooSeeker/bsv3.9 (compatible; Mozilla 4.0; MSIE 5.5) ( 68.142.195..x)
    YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD
    ;cafekelsa-dev-webmaster@yahoo-inc.com ) (64.157.137.xxx)
    Nutch-0.9-dev Unknown Yahoo robot

    Altavista (Yahoo):
    AltaVista Intranet V2.0 AVS EVAL
    AltaVista Intranet V2.0 Compaq Altavista Eval
    AltaVista Intranet V2.0 evreka.com
    AltaVista V2.0B AV Fetch 1.0
    AVSearch-3.0(AltaVista/AVC)
    Scooter-3.0.EU
    Scooter-3.0.FS
    Scooter-3.0.HD
    Scooter-3.0.VNS
    Scooter-3.0QI
    Scooter-3.2
    Scooter-3.2.BT
    Scooter-3.2.DIL
    Scooter-3.2.EX
    Scooter-3.2.JT
    Scooter-3.2.NIV
    Scooter-3.2.SF0
    Scooter-3.2.snippet
    Scooter-3.3dev
    Scooter-ARS-1.1
    Scooter-ARS-1.1-ih
    scooter-venus-3.0.vns
    Scooter-W3-1.0
    Scooter-W3.1.2
    Scooter/1.0
    Scooter/1.0
    Scooter/1.1 (custom)
    Scooter/2.0 G.R.A.B. V1.1.0
    Scooter/2.0 G.R.A.B. X2.0
    Scooter/3.3
    Scooter/3.3.QA.pczukor
    Scooter/3.3.vscooter
    Scooter/3.3_SF
    Scooter2_Mercator_x-x.0
    Scooter_bh0-3.0.3
    Scooter_trk3-3.0.3

    Inktomi (Yahoo):
    Inktomi Search
    Mozilla/3.0 (Slurp/cat) 72.30.61.xx(x)
    Mozilla/3.0 (Slurp/si) 72.30.61.xx(x)
    Mozilla/5.0 (compatible; Yahoo! Slurp China) (202.160.180.xxx)
    Mozilla/5.0 (compatible; Yahoo! Slurp) (via 66.196.xx.xxx)
    Mozilla/5.0 (Slurp/cat; )
    Mozilla/5.0 (Slurp/si; )
    Slurp/2.0 (slurp@inktomi.com)
    Slurp/2.0-KiteWeekly (slurp@inktomi.com)
    Slurp/si (slurp@inktomi.com)
    Slurpy Verifier/1.0 72.30.61.xx(x)
     
    evera, Aug 18, 2007 IP
  5. trichnosis

    trichnosis Prominent Member

    Messages:
    13,784
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #5
    it's an intresting request.

    why do you need to block google and yahoo from your sites?
     
    trichnosis, Aug 21, 2007 IP
  6. n3o_the_on3

    n3o_the_on3 Well-Known Member

    Messages:
    1,422
    Likes Received:
    62
    Best Answers:
    0
    Trophy Points:
    165
    #6
    Yep Trichnosis! its very interesting. why do you want to block yahoo and google?
     
    n3o_the_on3, Aug 29, 2007 IP
  7. cheapez

    cheapez Active Member

    Messages:
    1,123
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    78
    #7
    Anyone know the name of the Google bot that crawls the Adsense on your site?
     
    cheapez, Sep 2, 2007 IP
  8. evera

    evera Peon

    Messages:
    283
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Its:
    User-agent: Mediapartners-Google
     
    evera, Sep 2, 2007 IP
  9. blabla11

    blabla11 Peon

    Messages:
    164
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #9
    I'm testing BH script and don't want to get IP banned
     
    blabla11, Sep 4, 2007 IP
  10. seoworld

    seoworld Active Member

    Messages:
    471
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    58
    #10
    Are you looking to block these three bots only? or other bots as well?

    Because if you want to block bots in general then just write the following

    User-agent: *
    Disallow: /whatever/
     
    seoworld, Sep 11, 2007 IP