Google Bots crashing my DB server

Discussion in 'vBulletin' started by sansei, Jun 13, 2005.

0
  1. #1
    I have a number of forums which get crawled fairly regularly by the bots. I also have my concurrent connections settings set at 1500 visitors which I though would be enough.

    anyway, when the bots visit, they regularrly cross the 1500 connections at a single time and create a DB error.

    MY question is that how are the bots registering this error. Assmuning that the other 1499 times they don't get an error with the DB, am I gettign dinged by the bots(from a SERP perspective) for the one DB failure?

    Thanks for thoughts and suggestions.
     
    sansei, Jun 13, 2005 IP
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    What's the URL of your site? I've never heard of Googlebots coming anywhere remotely close to 1,500 concurrent connections.
     
    digitalpoint, Jun 13, 2005 IP
  3. sansei

    sansei Well-Known Member

    Messages:
    12
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    113
    #3
    I agree with you, but that was the most recent theory I received from the server side guys since almost all the DB crashes have come from Bots and not actual users.

    Anyway, you will know the forums I am talking about. They are located at Firetown.com and National-real-estate-directory.com
     
    sansei, Jun 13, 2005 IP
    Blogmaster likes this.
  4. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #4
    Well then I would say your server guys have no idea what they are talking about. vBulletin shows you the maximum number of users it ever had at once (your 2 forums have an all-time maximum user count of 24 and 429). By default, that's a 15 minute window... So even at 429, you are averaging a user more than 2 seconds apart.

    1,500 "concurrent" users (even if that was within 5 seconds) would give you a maximum user count of 450,000.
     
    digitalpoint, Jun 13, 2005 IP
  5. sansei

    sansei Well-Known Member

    Messages:
    12
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    113
    #5
    sounds like the technology terms I am using is wrong. What I was trying to communicate is that when a bot is in the forum, it is following all the links one after the other. If as a single user, it is checking these links faster than a human can check links, could this be causing the DB failures and if so, will this ding the over all ranking in the forum since not all the links it is testing are being validated(working)
     
    sansei, Jun 13, 2005 IP
    Blogmaster likes this.
  6. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #6
    Typically a bot comes in waves... fetching a group of pages, then coming back. But regardless of what the underlying cause is, yes... if the DB server is failing, the page that gets spidered will be an error message from the forum about the DB server failing and not the actual content of the URL.
     
    digitalpoint, Jun 13, 2005 IP
    Blogmaster likes this.
  7. petertdavis

    petertdavis Notable Member

    Messages:
    1,494
    Likes Received:
    159
    Best Answers:
    0
    Trophy Points:
    235
    #7
    I suspect a server problem. You're not on a shared server, are you?
     
    petertdavis, Jun 13, 2005 IP
  8. sansei

    sansei Well-Known Member

    Messages:
    12
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    113
    #8
    Only in the aspect that all our clients are on this server. No outside sites are on the server
     
    sansei, Jun 13, 2005 IP
    Blogmaster likes this.