Any way to limit the G's bot visits?

Discussion in 'Google' started by pixads, Jan 10, 2007.

  1. #1
    Is there a way I can limit Google's bot visits?

    Now it hits me every 5 seconds and causes quite good usage on my account

    Thanks
     
    pixads, Jan 10, 2007 IP
  2. Dekker

    Dekker Peon

    Messages:
    4,185
    Likes Received:
    287
    Best Answers:
    0
    Trophy Points:
    0
    #2
    This will pass, probably just doing some deeplinking
     
    Dekker, Jan 10, 2007 IP
  3. pixads

    pixads Well-Known Member

    Messages:
    2,015
    Likes Received:
    45
    Best Answers:
    0
    Trophy Points:
    145
    #3
    this has been going on for the last month so it does not pass :(

    my server crashes from time to time because of it
     
    pixads, Jan 10, 2007 IP
  4. sterlingpixel

    sterlingpixel Peon

    Messages:
    264
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Im sure you can tell gbot to visit less often if you use sitemaps
     
    sterlingpixel, Jan 10, 2007 IP
  5. robotdex

    robotdex Peon

    Messages:
    16
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    If you have a sitemaps account with google you can go into the options and request a slow crawl rate.
     
    robotdex, Jan 10, 2007 IP
  6. nevetS

    nevetS Evolving Dragon

    Messages:
    2,544
    Likes Received:
    211
    Best Answers:
    0
    Trophy Points:
    135
    #6
    Googlebot supports the crawl-delay tag if I'm not mistaken:

    
    User-agent: googlebot
    Crawl-delay: 180
    
    Code (robots.txt):
     
    nevetS, Jan 10, 2007 IP
  7. dimva

    dimva Well-Known Member

    Messages:
    681
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    130
    #7
    Could this work?
    <meta name="revisit-after" content="7 days">
    <meta name="robots" content="index, follow">
     
    dimva, Jan 10, 2007 IP
  8. pixads

    pixads Well-Known Member

    Messages:
    2,015
    Likes Received:
    45
    Best Answers:
    0
    Trophy Points:
    145
    #8
    it's not the revisit rate that bothers me just the new pages visiting
    the site now has 170 000 pages in G and it ads more and more
     
    pixads, Jan 10, 2007 IP
  9. jc22

    jc22 Peon

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Go here:

    http://www.google.com/webmasters/

    Once you take ownership of your site, you can tell Google what crawlrate to use: Slow, Normal, Fast (if they enable it for you).
     
    jc22, Jan 10, 2007 IP
  10. pixads

    pixads Well-Known Member

    Messages:
    2,015
    Likes Received:
    45
    Best Answers:
    0
    Trophy Points:
    145
    #10
    the verification is out of order :(
    i will try with the crawl delay
     
    pixads, Jan 10, 2007 IP
  11. adleym

    adleym Peon

    Messages:
    795
    Likes Received:
    22
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Personally, I like it when the Googlebot visits. It can visit as many times as it wants, even drink my beer, sleep with my wife, whatever!;)
     
    adleym, Jan 11, 2007 IP
  12. Seiya

    Seiya Peon

    Messages:
    4,666
    Likes Received:
    404
    Best Answers:
    0
    Trophy Points:
    0
    #12
    The easiest way is to access the sitemaps feature as mentioned before... :)
     
    Seiya, Jan 11, 2007 IP
  13. pixads

    pixads Well-Known Member

    Messages:
    2,015
    Likes Received:
    45
    Best Answers:
    0
    Trophy Points:
    145
    #13
    seems like this would be the only was as the bot does not even read my robots file and does not delay the visits :(

    I hope the verification will be back working soon

    Added:
    so verification is back. let's see if the setting realy works

    the interesting thing is the message I got when changed the rate to slow:

    This site is currently set at a Slower crawl rate. This rate will return to Normal on Apr 11, 2007.

    I do not want it to return to normal. Why would they do it auto?
     
    pixads, Jan 11, 2007 IP
  14. indoman

    indoman Peon

    Messages:
    17
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #14
    by anchor text google will visited our site
     
    indoman, Jan 11, 2007 IP
  15. nevetS

    nevetS Evolving Dragon

    Messages:
    2,544
    Likes Received:
    211
    Best Answers:
    0
    Trophy Points:
    135
    #15
    I stand corrected - I found this page on Matt's site http://www.mattcutts.com/blog/googlebot-keep-out/ where he specifically states that Googlebot does not support the crawl-delay command. I'm not the only person who was fooled though - http://slashdot.org/robots.txt
     
    nevetS, Jan 11, 2007 IP
  16. pixads

    pixads Well-Known Member

    Messages:
    2,015
    Likes Received:
    45
    Best Answers:
    0
    Trophy Points:
    145
    #16
    the sitemaps option did not make things better
    now Gogglebot is on the site every 2-3 seconds
    any more ideas how to limit the bot without banning it?
     
    pixads, Jan 13, 2007 IP
  17. johan-cr

    johan-cr Well-Known Member

    Messages:
    2,034
    Likes Received:
    170
    Best Answers:
    0
    Trophy Points:
    135
    #17
    I have tried to search a lot for the solution for this problem. Have the same problem with my sites and i have still not found any solution. If anyone has an solution that would be greatly appriciated.
     
    johan-cr, Jan 13, 2007 IP
  18. Ganceann

    Ganceann Peon

    Messages:
    842
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    0
    #18
    Is this possibly linked to Matt Cutts stating that the cache for supplemental index would be updated more frequently?

    I feel it could be a related issue and that the main googlebot would observe the slower crawl rate requested through webmaster tools/google sitemap - but the supplemental bot isn't paying attention to the request?

    Without knowing much, it is a possibility.
     
    Ganceann, Jan 13, 2007 IP