How to deal with a sudden intensive bot traffic from many ASNs/IPs to a vBulletin4 forum+Cloudflare

Hello, I wanted to share my experience how I have rather successfully eliminated intensive bot traffic from many ASNs and subnets using free Cloudflare account + I would like your feedback on what that traffic is and if i could do it better way, maybe even without Cloudflare...

My vbulletin4 forum website went down due to a CPU limit reached on a shared hosting account.
According to a logs, the reason was many visitors, sometimes several per second.
They did not come from the same IP or User-Agent, nor similar subnets /16 or /24 but many ASNs, most visits (according to Webalizer stats - i have pulled these into Calc app and sorted) I have noticed from:
HostPapa
RackNerd
Web2Objects

I had no better budget friendly idea than setting Interactive challenge (captcha) for visits which seemed resource intensive and rarely used by regular visitors. These rules were set at https://dash.cloudflare.com/idhere/mysite.com/security/security-rules
like this:
"When incoming requests match…"
Field = URI Path, Operator = wildcard, Value = /tags.php*
Field = AS Num, Operator = equals, Value = 36352

Full expression:

at that page /security/security-rules, i have also set another rule for Interactive challenge for IPs/subnets listed in https://dash.cloudflare.com/idhere/configurations/lists/idhere :
Field = IP Source Address, Operator = is in list, Value = blockedips
(blockedips is name of my list)

what really made a change in my logs (in terms of a traffic reduction) was that last rule:

http.request.uri wildcard r"/[I].php*s=[/I]"

Code (markup):

Sample traffic before applying the rule:

am I blocking legitimate traffic using that s= rule? I have used it because when I am browsing the site as a human, i do not see these s= in URLs nor in logs near my visits.

Note that also as a next measure to reduce bot flood I am using .htaccess firewalls:
https://perishablepress.com/8g-firewall/
and
https://perishablepress.com/ultimate-ai-block-list/

and having set crawl-delay in robots.txt (most bots does not respect, but such bots may be at least reported)

 

The last time I used .htaccess to block bots, my forum got very slow.
I ended up putting most of it visible only to logged in users.

Now everybody has a bot using AI... It's insane.

 

Looks like typical scanners/bots from datacenters, nothing unusual. What you did with Cloudflare is solid — you’re just filtering junk before it hits your server.
As for s= — if you’re not using it anywhere, you’re most likely blocking bots, not real users.
I’d also add some rate limiting and not bother mass-blocking ASNs — they’ll just rotate anyway. Without Cloudflare, this would be much more painful on shared hosting.

 

I also wanted to suggest that one may try challenging all visits from datacenters/cloud with captcha using CIDRAM (free and open source PHP software) and block known threats. It may reduce some load.
But before installing it, i suggest to check the load without CIDRAM being attached to the site and then while it is attached, in order to ensure that load really decreased and not increased.
I wish there is more ways a shared hosting user can fight this bots issue for free or for a couple of $, without hiding content or using Cloudflare. I have not yet researched about solutions on web server level regarding GEO blocking/whitelisting (based on countries of visitors).

 

I had today a couple of websites down due to intense hits from bots.
All of these bots are getting out of control... Each bot was hitting over 130/hits per second. They put my websites down. It's insane..