How to deal with a sudden intensive bot traffic from many ASNs/IPs to a vBulletin4 forum+Cloudflare

Hello, I wanted to share my experience how I have rather successfully eliminated intensive bot traffic from many ASNs and subnets using free Cloudflare account + I would like your feedback on what that traffic is and if i could do it better way, maybe even without Cloudflare...

My vbulletin4 forum website went down due to a CPU limit reached on a shared hosting account.
According to a logs, the reason was many visitors, sometimes several per second.
They did not come from the same IP or User-Agent, nor similar subnets /16 or /24 but many ASNs, most visits (according to Webalizer stats - i have pulled these into Calc app and sorted) I have noticed from:
HostPapa
RackNerd
Web2Objects

I had no better budget friendly idea than setting Interactive challenge (captcha) for visits which seemed resource intensive and rarely used by regular visitors. These rules were set at https://dash.cloudflare.com/idhere/mysite.com/security/security-rules
like this:
"When incoming requests match…"
Field = URI Path, Operator = wildcard, Value = /tags.php*
Field = AS Num, Operator = equals, Value = 36352

Full expression:

at that page /security/security-rules, i have also set another rule for Interactive challenge for IPs/subnets listed in https://dash.cloudflare.com/idhere/configurations/lists/idhere :
Field = IP Source Address, Operator = is in list, Value = blockedips
(blockedips is name of my list)

what really made a change in my logs (in terms of a traffic reduction) was that last rule:

http.request.uri wildcard r"/[I].php*s=[/I]"

Code (markup):

Sample traffic before applying the rule:

am I blocking legitimate traffic using that s= rule? I have used it because when I am browsing the site as a human, i do not see these s= in URLs nor in logs near my visits.

Note that also as a next measure to reduce bot flood I am using .htaccess firewalls:
https://perishablepress.com/8g-firewall/
and
https://perishablepress.com/ultimate-ai-block-list/

and having set crawl-delay in robots.txt (most bots does not respect, but such bots may be at least reported)

 

The last time I used .htaccess to block bots, my forum got very slow.
I ended up putting most of it visible only to logged in users.

Now everybody has a bot using AI... It's insane.

 

Looks like typical scanners/bots from datacenters, nothing unusual. What you did with Cloudflare is solid — you’re just filtering junk before it hits your server.
As for s= — if you’re not using it anywhere, you’re most likely blocking bots, not real users.
I’d also add some rate limiting and not bother mass-blocking ASNs — they’ll just rotate anyway. Without Cloudflare, this would be much more painful on shared hosting.

 

I also wanted to suggest that one may try challenging all visits from datacenters/cloud with captcha using CIDRAM (free and open source PHP software) and block known threats. It may reduce some load.
But before installing it, i suggest to check the load without CIDRAM being attached to the site and then while it is attached, in order to ensure that load really decreased and not increased.
I wish there is more ways a shared hosting user can fight this bots issue for free or for a couple of $, without hiding content or using Cloudflare. I have not yet researched about solutions on web server level regarding GEO blocking/whitelisting (based on countries of visitors).

 

I had today a couple of websites down due to intense hits from bots.
All of these bots are getting out of control... Each bot was hitting over 130/hits per second. They put my websites down. It's insane..

 

Ran into the same thing on a small PHP site last year. A few things that helped on shared hosting without Cloudflare:

1. Check the user agents on the heavy ranges. A lot of that RackNerd / HostPapa traffic is AI and SEO crawlers that ignore robots.txt, and many of them still send a recognizable UA. Blocking by UA in .htaccess is much cheaper than long IP lists, which is probably why mike30's forum got slow. Apache walks every Deny line on every request.

2. If you do block ranges, block whole ASNs as a short list of large CIDRs, not hundreds of /24s. Datacenter hosting ranges almost never carry real forum readers, so the false positive risk is low. Home connections that get rented out as exit nodes are a different story, those you can't block by range without hitting real people.

3. Cache guest pages. vB4 renders everything through PHP, so every bot hit costs CPU. Even a simple full page cache for logged out users cuts the load a lot, and bots rarely log in.

4. Rate limit search and member list pages specifically. Those are the expensive ones and the ones scrapers love (the s= thing you noticed is a good sign it's automated).

5. Look at request timing, not just IPs. Real readers open a thread, wait, scroll, click. Scrapers hit pages at fixed intervals with no CSS/JS/image requests in between. If your logs show HTML only requests with no assets, that's a bot regardless of which IP it came from.

The rotating IP part is why pure IP blocking keeps failing. Whatever you block, they move to the next range, so the behavior and cache side ends up doing most of the work.