what are all the google bots?

Discussion in 'Google' started by Link.ezer.com, Dec 1, 2008.

0
  1. #1
    I am looking for all the names of google bots and the functions of each bots.

    Any one knows? Please list them or tell us where to find the information.

    Thanks
     
    Link.ezer.com, Dec 1, 2008 IP
  2. yousellstuff1

    yousellstuff1 Active Member

    Messages:
    1,195
    Likes Received:
    17
    Best Answers:
    0
    Trophy Points:
    75
    #2
    Well, beside the regular googlebots, there's

    MediaBot - used to analyze AdSense pages
    user agent "Mediapartners-google"

    ImageBot - crawling for the Image Search
    user agent "googleBot-Image"

    AdsBot - checking AdWords landing pages for quality
    user agent "AdsBot-google"

    Regarding Adwords
    This bot
    "googleBot/2.1" has been crawling one page of my site daily. The page being crawled is the target of an adwords campaign. So Adwords is definitely looking at target page quality. Note the upper case "Bot". This is ("googleBot/2.1") the entire referrer string as well. It has used multiple IP addresses.
    This bot ("googleBot/2.1") also detected a change to one of my robots.txt files (more liberal for google) and appeared to almost immediately trigger a deep complete crawl of the site from the conventional bot. I believe all the bots cooperate in collecting the robots.txt file.

    The conventional bot has a lower case "bot"
    "Mozilla/5.0 (compatible; googlebot/2.1; +http://www.google.com/bot.html)"

    google has made it a little difficult to sort out all of their bots with one search string "/2.1" works but does find some extra unrelated odds and ends in logs. (Except of course googlebot-Image/1.0)

    Referrer strings extracted directly from my logs


    Adsense: "Mediapartners-google/2.1"
    Adwords: "googleBot/2.1"
    google-: "Mozilla/5.0 (compatible; googlebot/2.1; +http://www.google.com/bot.html)"
    Image--: "googlebot-Image/1.0"

    Of course as mentioned, there's Froogle,Mobile,and Feed for RSS feeds.

    Also google now appears to be exclusively using HTTP/1.1, until recently there has been a mix of HTTP/1.1 and HTTP/1.0. One thing important to note is google is now always requesting GZIP compressed content if your server provides it. Your website might get an "attaboy" if you served GZIP compressed content to the "bots". This could cut your website's bandwidth usage and boost your site's performance.

    all of this information was taken from
    http://www.webmasterworld.com/forum30/34547.htm

    Hope this helps
    Brandon
     
    yousellstuff1, Dec 1, 2008 IP
    Link.ezer.com likes this.
  3. Link.ezer.com

    Link.ezer.com Peon

    Messages:
    647
    Likes Received:
    28
    Best Answers:
    0
    Trophy Points:
    0
    #3
    thanks Yousellstuff1,
    good rep added

    regards,
     
    Link.ezer.com, Dec 1, 2008 IP
  4. baiduspider

    baiduspider Peon

    Messages:
    45
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    there is another google bot use in mobile now:

    Google Wireless Transcoder

    IP: 209.85.138.136 209.85.170.136 72.14.252.136...
     
    baiduspider, Dec 1, 2008 IP
    Link.ezer.com likes this.
  5. zk200083

    zk200083 Peon

    Messages:
    211
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #5
    It very helpful
     
    zk200083, Dec 1, 2008 IP