Google Bot Question

Discussion in 'Google' started by Nick_Mayhem, Nov 9, 2006.

0
  1. #1
    Hello friends,

    I have read it somewhere that it can be tuned up that Google BOT don't eat up more bandwidth.

    Because google bot on one of my site is using 59 MB Bandwidth. :(

    It is not a big deal I know but can it be minimized?

    I don't want to put extra load on server if this 59 MB's can be minimized or saved.

    Some input would be highly appreciated.

    Thanks.
     
    Nick_Mayhem, Nov 9, 2006 IP
  2. Richie_Ni

    Richie_Ni Illustrious Member

    Messages:
    10,721
    Likes Received:
    1,175
    Best Answers:
    0
    Trophy Points:
    410
    #2
    So you wanted to prevent the bots from crawling your site?
    Use Robots.txt
     
    Richie_Ni, Nov 9, 2006 IP
  3. visiaa

    visiaa Peon

    Messages:
    81
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    And also there is possiblity that there are some virsus is there which is eating your bandwidth, so why dont you check that possilblity also.
     
    visiaa, Nov 9, 2006 IP
  4. pixel_boost

    pixel_boost Peon

    Messages:
    55
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    You also have the "Crawl-delay: number of seconds" directive in robots.txt this is helpfull if you see that the bots are too fast in crawling your site
     
    pixel_boost, Nov 9, 2006 IP
  5. Nick_Mayhem

    Nick_Mayhem Notable Member

    Messages:
    3,486
    Likes Received:
    338
    Best Answers:
    0
    Trophy Points:
    290
    #5
    Hummm... I think it was called something like IF - Modified.

    I am not looking to block the google bot.
     
    Nick_Mayhem, Nov 9, 2006 IP
  6. Hosting Advisor

    Hosting Advisor Banned

    Messages:
    406
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Well, i dont think the bots are not accumulating a lot of space when it comes to bandwidth, check in, may be you have something to fix.:cool:
     
    Hosting Advisor, Nov 9, 2006 IP
  7. Nick_Mayhem

    Nick_Mayhem Notable Member

    Messages:
    3,486
    Likes Received:
    338
    Best Answers:
    0
    Trophy Points:
    290
    #7
    Googlebot 4013 59.97 MB 08 Nov 2006
    Inktomi Slurp 779 11.90 MB 08 Nov 2006
    MSNBot 457 7.66 MB 08 Nov 2006
    Voyager 53 1.16 MB 07 Nov 2006

    Maybe the above thing will give someone some idea about what could be done.

    Its not very important. I am happy that google is showing me at nice positions for nice keywords. But I just thought if this is saveable then I would be good. Otherwise I will have to move this site some place else :)
     
    Nick_Mayhem, Nov 9, 2006 IP
  8. Mong

    Mong ↓↘→ horsePower

    Messages:
    4,789
    Likes Received:
    734
    Best Answers:
    0
    Trophy Points:
    235
    #8
    I think its option somewhere in your sitemap account ;)
    Dig it .. :)
     
    Mong, Nov 9, 2006 IP
  9. Richie_Ni

    Richie_Ni Illustrious Member

    Messages:
    10,721
    Likes Received:
    1,175
    Best Answers:
    0
    Trophy Points:
    410
    #9
    Ummm,probably it's the "crawl rate"?
     
    Richie_Ni, Nov 9, 2006 IP
  10. Jean-Luc

    Jean-Luc Peon

    Messages:
    601
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Exactly!

    It is supported by static html pages. When a client visits the page for the first time, the code is 200 and the page is sent. The next time the client software (browser or robot) visits the page, if it supports this "if modified", it first checks if the page has been modified by the server since the last time the client read it. If the page has not been modified, the server answers with a 304 code and it does not resend the page. If the page has been modified a 200 code is sent and the new content of the page is sent.

    Some very clever dynamic pages (some pages in WordPress) can manage this too.

    Jean-Luc
     
    Jean-Luc, Nov 9, 2006 IP
  11. Nick_Mayhem

    Nick_Mayhem Notable Member

    Messages:
    3,486
    Likes Received:
    338
    Best Answers:
    0
    Trophy Points:
    290
    #11
    Means my PHP script will not be able to handle it. :(

    And it might handle it but it will drive me insane with research before something good happens. :(

    Got it friends. Thanks a lot for your support.
     
    Nick_Mayhem, Nov 9, 2006 IP