server hanging from too many httpd processes

Every few days my server will spawn 450 or so httpd daemons in a few minutes and effectively go offline. It requires a shutdown -r to resolve and that usually takes 45 minutes to complete.

I'm trying to track down what's causing it. I suspect that either I've got a misbehaving bot or an obnoxious email address harvester hitting it.

I've looked at the logfiles via webalyzer but don't see anything obvious and the message log does not have any entries that look suspicious.

Any advice on how to figure out what’s causing it?

Are there apache configuration parameters that will help?

Any suggestions would be appreciated.

Regards,

-jay

 

This kind of condition may also be triggered by a bug in the code that is being executed. For example, if you have an endless loop that will tie up one of the worker threads, eventually all of them will end up hanging, locking up the server. Try to see if the amount of traffic a few minutes before the server hangs is increasing compared to the time it's working all right. Check CPU usage - if you have an endless loop that does something like string comparison, your CPUs will go to 100%.

J.D.

 

Next time it happens it would be quicker to reset the httpd daemon, instead of doing a full reset, how to do this depends on your distro, redhat/fedora which are the most common, would be:

service httpd restart

Code (markup):

When it crashes check the end of your and access and error, to see what was hitting the server before it went down:
Access logs:

tail /var/log/httpd/access_log -n 30

Code (markup):

Error logs:

tail /var/log/httpd/error_log -n 30

Code (markup):

You might need to change the log paths to reflect your directory structure.

If you're logged while it happens you can do this to see what clients are hitting your box:

netstat -tpu

Code (markup):

If it's one host causing the problem you can block it with your firewall:

iptables -I INPUT 1 -s xxx.xxx.xxx.xxx -j DROP

Code (markup):

xxx.xxx.xxx.xxx is the ip to block.

This will also block a host on most systems:

echo xxx.xxx.xxx.xxx >> /etc/hosts.deny

Code (markup):

Otherwise you will probably have to make some changes to the apache config file to limit the number of allowed processes.

 

Thanks for the suggestions.

I'll have a window later tonight to make these changes and dig into exactly what's going.

Regards,

-jay

 

Thanks for the help

Here's what I discovered (and not).

1. There was a lot of "files not found errors" in the error log - *fixed*

2. My .htaccess "deny" is working well for the Nigerian 419 scammers I'm looking at the host.deny and other IP level blocks to improve efficiency - my .htaccess is at 16k (even with using CIDR for the addresses).

3. There was no indication of sudden traffic from a single IP address or range of addresses - So at this point I'm going to assume that there's some buggy code somewhere. I've set up some traps to try to isolate it.

4. The MaxClient was set at 400, I changed it to 100 and will lower it further if the problem shows up again.

5. Restarting the http daemon took 2 minutes instead of 45 minutes for a full reboot

Thanks again for the advice!

Regards,

-jay

 

Did you say 45 *minutes*?!

 

YES I DID

It was very frustrating.

-jay

 

45 minutes is an extremely long reboot time. You probably need to look at your init scripts, and check your logs, somethings causing that lapse.

 

If a restart/reboot takes longer than a few minutes, I usually kill the process that causes it.

J.D.

 

A normal reboot takes about 4-5 minutes.

The 45 minute reboot happens when 300-400 httpd daemons are spawned in several minutes and overloads the server.

I'm still trying to determine the cause and I'm hoping that the recommendations made earlier will help mitigate the impact on the server (at least until I figure out what's causing it).

-jay

 

OK, 4-5 minutes is OK for a normal reboot.

Does the large spawning happen at a particular time of day, or is it totally random?

 

I understand. What I'm saying is that I cannot imagine a machine being out of circulation for over 4-5 minutes and when this kind of thing happens, I usually kill the process that causes the problem after a short timeout (typically a minute or so). The only thing to watch out here for is that if it's not httpd, but something that else (e.g. DBMS), then killing the process may have repercussions on integrity of the data it's the killed process was working on at the time it was killed.

J.D.

 

When it gets in this vegetative state the response time on the SSH/telnet session is so slow that it takes 5 minutes to enter one command and at this point its spawned so many processes that its difficult and slow to dig through them looking for the culprit.

As you can tell from my posts I’m not a sys admin nor am I a programmer – (not since the early eighties anyway –I loved those old Sperry Univac DCPs!).

I appreciate your experience, insights and recommendations so please keep them coming!

-jay

 

The top program will print out CPU/Memory/etc usage for the most active processes

 

That means your CPU is pegged at 100%. I think it is a bad loop somewhere in the code. May be not your code, but something is spinning its wheels when this happens.

J.D.

 

I'm running Linux Fedora Core 2.

Is it "top" - no arguments?

 

Yes just top. It can take arguments to customize the output (which might be useful in your case), do "man top" for more info.

 

This and the other suggestions should give me plenty to do this weekend!

Thanks again for the help.

-jay

 

You can record top's output every 60 seconds or so, in case if you want to leave it running for some time:

top -d 60 -b > top.txt

Each output will be 5-10K, so watch your drive usage if you change the timeout or want to leave it running for a while.

J.D.