Is there a way to limit the number of concurrent connections per IP address, but have an exception list as well? Recently, I noticed that several IPs were spamming, scraping, etc. or otherwise overloading our website server with requests. We solved this quickly by installing mod_limitipconn module for apache and set a limit on how many concurrent connections would be allowed per IP address. However, we now realize that by doing so we may have inadvertently hindered search engine crawlers ability to crawl our site. I did some searching and found a patch for mod_limitipconn here: http://dominia.org/djao/limitipconn.html that would limit by IP, user, vhost or load average and would be perfect but it only works for Apache 1.3 and we are running 2.0. I’ve done a lot of searching and asking around and no one seems to have a solution for this. It seems very strange to me considering that we can’t be the first website that has had this problem. Is there anyone here with experience dealing with this problem? Any suggestions on how to solve?
It would probably be easiest to just make a small PHP script linked to a database with all the IP's you want blocked/allowed.
My first thought would be to hire a C programmer to update that 1.3 patch to work with 2.0 Shouldn't be too much and you could probably recoup at least some of the cost by selling/installing the solution to others. You'ld be wrapping a layer of rubber around the rim, but at least it's not re-inventing the wheel.