Hello, Yesterday one of my pages got dugg. Over 3000 diggs in less than 12 hours. As a result over 50.000 visitors arrived at one page in only a few hours time. The page that got dugg was a total size of around 700kb... At first I thought I was safe with my DEDICATED server, which has the following specs: OS: CentOS 4.x Dual Xeon 2.8 HT/2048MB Ram/2x300GB SATA Drivers/100Mbps Port/ 3000GB bandwidth per month. One would think a server like this can handle these amounts of traffic right? But according to my host it cannot and the server showed so. It was constantly overloaded and half of the time it was completely offline... Bandwidth was not the problem... It consumed many Gigabytes, but 3000 is more than sufficient. According to my host I need at least a cpu with 8 CORES and a SCSI drive... Can anyone confirm that this hardware is needed to handle large amounts of traffic like this? I was convinced that a dedicated server - dual xeon with 2GB's of RAM could handle it.
It's always interesting to compare specs and traffic. Thanks for sharing. To figure out what you need to survive Digg you need to know where your bottleneck is. I guess that in your case it was CPU but guessing doesn't really help either of us. If it is CPU, you can go either way by adding more CPU power or by reducing the CPU your website uses. The CPU usage could be coming from the PHP scripts or from the database queries but you will need to do some profiling to figure that out. If it happens again (fingers crossed), try running "top" on the server while it's happening. Do you have a server stats monitoring package like Nagios, Orca or Cacti ? The graphs available from those packages could tell us a lot about the bottleneck. How much RAM was being used during this period ? What was the load average ? How many Apache processes were running ? What was the disk usage like ? In short, if your site was purely static HTML then it would almost certainly be able to handle that sort of traffic. With PHP and MySQL then all bets are off until we know more about the PHP code and the SQL queries. Unfortunately I don't have any stats from a Digging that I can share. We got on the front page of Digg last year and the 2 core (1280MHz per core) box that site was on couldn't handle the traffic either but that doesn't tell you much about what can handle Digg. We moved that site onto a load balanced cluster of two dual core dual processor boxes running at 2500MHz per core but it's sharing that cluster with five other sites, two of which get more than ten times the traffic on a normal day and we haven't been on Digg again since then anyway.
Well, every server is different. If you are using a site management tool like Plesk or Ensim or any of the other multitude of management tools out there, then you most likely will not be able to handle very much of a load. They in my experience end up eating about 1/3 - 1/2 of your resources. Even when your system is not doing anything at the time. A page with a 700kb size will only really affect your bandwidth usage. The real question when it comes to resources running a dynamic site comes from your queries and how optimized they are and the optimization of your PHP script. There are various ways to time your php script and how long it takes to run. If you can login to your box with SSH and run top look at the top right hand corner and you will see three numbers. The numbers are in decimal form. As long as the numbers are on the right hand side of the decimal you should be alright. If they make it to the left hand side of the decimal then you are starting to get in trouble. If they make it to the right hand side of the decimal that is basically how many processes have to wait in line to get access to your processor. I think that is per processor as well. So if you have a dual processor machine (single core's) then that is 2.00 before you run out of processors per process. There are many optimizations that can be done with Apache as well as PHP. Using APC or other caching components can help quite a bit as well. The upcoming PHPTek conference in Chicago has a speech being given by the Facebook CIO (or some other high up) talking about how they use PHP cacheing to reduce their server loads drastically. Good luck and congratulations on getting on the front page of Digg. That is an accomplishment. Often times peoples servers are overloaded though when they do get dugg like that, so people learn how to get around it or if they really want to read it will wait and come back. Fred
Just to get an idea of the traffic, content and the server you're running, could you let us know the page that got dugg? I'd be interested in analysing the requirements for running a site with so many hits. Thanks.
1. optimize your software and os with linux utils. You are probably using apache right? 2. If so remove apache and install lighttpd + xcache + memcached + fastcgi and your server will handle the DIGG diggs with no problem 3. Also uninstall,remove,kill any other programs or services you may use, but you don't need.
caching your pages is the solution. this way the hits to your database would be less giving lesser load to your server. there is a plugin to cache your wordpress though. just look for it in the repository, on the other hand you should install memcached as well. best of luck.
replace apache with lightppd, i have experienced digg with more then 8k diggs, and the server never screamed, fastcgi is a lifesaver as well.