apache web server HELP! extreme laggy pages

Hello all,

I work for a company with a highly trafficked site (about 100,000 unique daily visitors). We have our "MANAGED" hosting through Rackspace for quite the pretty penny per month. It includes a dedicated back end / DB server and cloud servers for the front end.

The setup has been functioning perfect for 10 months now; but this past Monday the speed of the site immediately dropped. Page load times fell from 1-2 seconds to between 10-20 seconds, and sometimes not at all. As far as we know (and as far as Rackspace says), no server setting were modified. No new code was introduced on our end. It's a mainly static site, with minimal user interaction with the backend at all.

Can any expert offer some advice? We've monitored the traffic, checked IPs, etc. We've even tunes down several site features in the interest of reducing server load. Upon a server reboot, the active threads/processes running on it IMMEDIATELY jump back up to maxed out levels. It seems like once our daily traffic reaches 10MB/s, a type of queue forms and the delays begin. Rackspace assures us that we're not limited to that.

Please advise - thanks! -Jay-

EDIT: Some more background info: The site is typically busiest from 7am until 3pm EST. For the past few days, we've noticed that between 7am and 9-10pm the server has just lagged incredibly. However, at around that 9-10pm mark, something changes and the pages go back to loading almost instantly. (There is still decent traffic though.) Then at around 7am again it slows to a crawl.

Rackspace has offered solutions such as spinning up another server and incorporating their load balancing - they are in the process of this BUT they do NOT think the traffic is the issue. At one point they actually said there was potential packet loss somewhere in the network, but no progress has been made.

Versions in Use:
OS: cent OS on cloud
OS: Redhat on Dedicated Server
Apache: 2.2
PHP: 5.3 / MySQL: 5.1.69

 

You need a good firewall to block any bad traffic, then tweak apache and mysql, etc from there.

 

UPDATE:

We've had packet loss when pinging from one server to the other; and the servers are in the same datacenter. We believe there is some type of network issue. Rackspace originally told us that it looked to be a DDOS attack; they then said it was much more than our normal traffic; and then they said they were investigating their hardware. Eventually we determined SOMETHING had to be done, so another server was thrown up. Of course it helped immensely... for a price of course.

I'm still curious as to how out of nowhere one day the web server could no longer handle the load. It wasn't like the page load times were slowly getting worse; one day they just diminished to a crawl. Here's 2 charts that show our standard traffic:

jjmenterprises.com/images/bandwidth2.png

I believe the cap looks to be at/around 10 Mbps. Our IT Director said that he believes we do have a gb card in the hardware. Thoughts on why the traffic seems to have a nice rollercoaster chart for the past 8 months but for 4 days last week (until we load balanced with another server on Friday) it would keep plateauing? (During these times pages would take 15-20 seconds to load, sometimes timing out, and this would last for a good 10 hours from 7am thru the early evening hours; then all of a sudden at 8 or 9 pm the site would start loading instantly again.)

We don't mind much the cost of the additional server if it's necessary; just feel there is something else at play here.

 

There are a lot of causes and possibilities to this: Networking issues, Kernel, TCP/IP stack limitations, DDoS, Misconfigurations, etc. This can't be determined without actual access to the involved servers. If you are getting a lot of traffic then you may consider replacing apache with a more powerful webserver such as nginx or litespeed.

 

Were you able to resolve this issue by now? If you actually get packet loss, I rather think it's a network issues than an issue with Apache. However, there are different possibilites what could cause the network issues. Faulty kernel settings, a faulty NIC, a faulty config on your hoster's node, bad routing, or DDoS attacks. If it's DDoS, you would have to figure the type and size of the attack first, to figure the best defence plan. You could first try to harden your server and optimize the kernel settings to handle more packets per second than a default config and also install something like CSF. If that's not enough, you should think about getting a remote DDoS protection. On r00t-Services.net you can find very affordable anti DDoS solutions. However, if the attacks are small enough, you might as well be able to block it via software directly on your server or by using the free CloudFlare plan, which will also protect you from a few small attacks. But as other users already mentioned, it's just impossible to guess the root of your problems without access to the server or more information. Where does the packet loss occure? At the last hop or maybe on the path already? Do you see anything suspicious (ie. erros) in /var/log/messages? What do you see in your monitoring? High memory or CPU usage or many software interrupts or packets per second? Any weird repeated requests in your access logs?

"Upon a server reboot, the active threads/processes running on it IMMEDIATELY jump back up to maxed out levels."
Although this sounds rather like a layer 7 attack (HTTP flood) than a network/server issue. Your best bet would be to have a sysadmin investigate the root of this issue and then think about appropiate counter-measures.