1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to keep your site online during a DIGG!

Discussion in 'General Chat' started by TheHoff, Jun 24, 2006.

  1. #1
    Thanks to Nintendo and the Google spam story last week, my blog was dugg to the front page twice. It resulted in about 50k uniques with a huge wave of traffic coming as soon as the story hit the front. With shared hosting accounts, running PHP or mySQL and getting Dugg = death.

    [​IMG]

    I put together a list of steps to take to make sure your site stays online when Dugg:

    http://digg.com/technology/Can_Your_Website_Survive_a_DIGG_
     
    TheHoff, Jun 24, 2006 IP
    anthonycea likes this.
  2. anthonycea

    anthonycea Banned

    Messages:
    13,378
    Likes Received:
    342
    Best Answers:
    0
    Trophy Points:
    0
    #2
    You don't need a list, you need IBM to host your site! :eek: :D
     
    anthonycea, Jun 24, 2006 IP
  3. TheHoff

    TheHoff Peon

    Messages:
    1,530
    Likes Received:
    130
    Best Answers:
    0
    Trophy Points:
    0
    #3
    TheHoff, Jun 24, 2006 IP
  4. KLB

    KLB Peon

    Messages:
    1,167
    Likes Received:
    68
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I believe I have the bandwidth and server allocation to handle being slashdotted or dugg, but thus far it has never happened :( I'd love to see my site really tested. I'm certain I could handle 100,000 page views in one day. I've done over 50,000 without the server even breaking a sweat.
     
    KLB, Jun 24, 2006 IP
  5. TheHoff

    TheHoff Peon

    Messages:
    1,530
    Likes Received:
    130
    Best Answers:
    0
    Trophy Points:
    0
    #5
    It isn't the quantity over a day, it is the # that slam it over the first few minutes. If the traffic were spread out over 24 hours, most could handle it. I'm not saying you can't, but the biggest factor is if your pages are dynamic-- PHP and a database? That is the killer... can your dB keep up with 100's of queries per second?

    And are you using a design with more than a few images? Too many HTTP requests for images will bring sites down as well.
     
    TheHoff, Jun 24, 2006 IP
  6. KLB

    KLB Peon

    Messages:
    1,167
    Likes Received:
    68
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Okay its the queries per second that could pose a challenge. I think, however, I could reduce this concern as most of my content is not database driven, only supplemental stuff is. I might be able to even reduce the number of these if necessary.
     
    KLB, Jun 24, 2006 IP
  7. Nintendo

    Nintendo ♬ King of da Wackos ♬

    Messages:
    12,890
    Likes Received:
    1,064
    Best Answers:
    0
    Trophy Points:
    430
    #7
    Sweet!! :D:D Now what's da odds of that!!! There are only three that you can read what it is...and da wacko get's one of them!! :D Is that how Matt found the page!! :D

    Would you be able to handle five visitors a second!!!
     
    Nintendo, Jun 24, 2006 IP
  8. KLB

    KLB Peon

    Messages:
    1,167
    Likes Received:
    68
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Sustained for hours might cause problems, but over a short period of time it should be no problems. Even at that it would only cause minor failings on my site. There might be some SQL errors generated, and some things would go missing, but it wouldn't spell disaster.

    I'm thinking of coding back up plans to reduce the impact of my MySQL server going down. I like to be ready for anything the Internet can throw at me as much as technically possible.
     
    KLB, Jun 24, 2006 IP
  9. Nintendo

    Nintendo ♬ King of da Wackos ♬

    Messages:
    12,890
    Likes Received:
    1,064
    Best Answers:
    0
    Trophy Points:
    430
    #9
    It was for about three or four hours that I got slammed with visitors, like a DOS attack (Apache went down once.). As soon as the link left the index page of Digg, then it started going down. Here's some stats (Day one covers only about six hours!) and massive log, that only cover the Digg page.
     
    Nintendo, Jun 24, 2006 IP
  10. anthonycea

    anthonycea Banned

    Messages:
    13,378
    Likes Received:
    342
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Keep it up and your entire network will be down, you should get a different server for your Digging man!
     
    anthonycea, Jun 24, 2006 IP
  11. Nintendo

    Nintendo ♬ King of da Wackos ♬

    Messages:
    12,890
    Likes Received:
    1,064
    Best Answers:
    0
    Trophy Points:
    430
    #11
    One Apache crash isn't so bad!!! Simply try not to let vBulletion threads make the front page!! Look at Shawn to see what happens there!!! :D:D Hince I made sure the page was pure static with just one SSI code, to get the comments, and make the log. :D:D My brain just knew it would hit the front page, since every one hates spammers at Spamoogle, so I did what Boy Scouts do...be prepaired!!!! 100% success rate at making the index page is quite nice!!!!
     
    Nintendo, Jun 24, 2006 IP
  12. KLB

    KLB Peon

    Messages:
    1,167
    Likes Received:
    68
    Best Answers:
    0
    Trophy Points:
    0
    #12
    I'm on a shared hosting environment where there are only four other accounts on the same server and I'm alloted 300 GB per month ( http://pair.com/services/high_volume/hv1.html ). When I signed up for this account it was only allocated 60 GB per month, but over the years they increased the allotment. My chief concern is bringing down the MySQL server, not the website itself. With some streamlining, I could reduce dependence upon the database server as the core content does not depend upon MySQL anyways.
     
    KLB, Jun 24, 2006 IP
  13. Xitanto

    Xitanto Peon

    Messages:
    434
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #13
    I think you're referring to the fact that the forum spams the admin e-mail address every time it is offline and someone tries to connect?

    :S

    I really need a a good idea for a digg story. =|
     
    Xitanto, Jun 24, 2006 IP
  14. Nintendo

    Nintendo ♬ King of da Wackos ♬

    Messages:
    12,890
    Likes Received:
    1,064
    Best Answers:
    0
    Trophy Points:
    430
    #14
    Ouch!! I used them back in 2001 for mySQL. They were crummy when it came to just handling my mySQL database, a vBulletin board. I had time outs from using too many queries. I quickly upgraded to a dedicated server. And that's the plan I was on. For all the non mySQL stuff, they were great. Make it to Digg, and the site will probably be destroyed while it's on the front page, atleast if it's a page that uses mySQL!!!!!

    With a dedicated server you can atleast go in to root and restart something when it crashes!! Even on a dedicated server I had to do that once because of Digg.
     
    Nintendo, Jun 24, 2006 IP
  15. KLB

    KLB Peon

    Messages:
    1,167
    Likes Received:
    68
    Best Answers:
    0
    Trophy Points:
    0
    #15
    I rarely have problems with Pair. Well there were a few glitches with one of the database servers I use for one of my forums over the past few days, but it was only caused problems for a few seconds at a time. Somebody's database was behaving badly and chewing up all the CPU time. As soon as I reported the problem it went away.

    This evening, I retooled my main site such that it does not fail even if the database server does not respond. Most of the site really only uses the database server for secondary tasks that it can operate without if necessary. The sections of the site that require the database (e.g. my chemical database) now simply throw up a notice that they are temporarily unavailable if the database server fails to respond.

    Now if one of my core articles gets dugg or slashdotted, I'm pretty sure it won't bring down my entire site. If I do get enough traffic to bring down the webserver, I'd be really impressed.

    With Pair, if being on a shared database is a problem (like it was for you), one can get a dedicated database server for something like $90 per month. Since other people's databases rarely cause me problems, I'm too cheap to spring for it. Eventually I'd like to have enough traffic to justify going to a dedicated database server such that someone else's bone head database query doesn't bring down my databases, but for now that is a luxary.

    For those worrying about surviving a massive traffic load, one key point is to really look at the design of your site, don't make the server run through hoops it really doesn't need to go through. Now obviously if you are using a canned package like Word Press or vBulletin you have no choice in the matter. If, however, you have a completely custom site like mine, you can really pick and choose what you ask your database server to handle and it is SQL queries that can bring a server down the fastest.
     
    KLB, Jun 24, 2006 IP
  16. natterbu

    natterbu Peon

    Messages:
    269
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    0
    #16
    Any good UK hosting companies that can handle such spikes in traffic.This is exactly the kind of thing that has caused me problems in the past

    My old host ended up charging me £400 per month because i exceeded the limits constantly and i have never really prepared for massive spikes.

    Anybody out there that can recommend good reliable hosting company?

    Thanks
     
    natterbu, Jul 4, 2006 IP