Referer Spam, how do you fight it?

Hello all, recently Im getting hit with bots that spamming my site logs with refer info, I tried banning by ip, then banning by referer domains, no luck, they always change.

So there's no way to fight this stupid bot?

 

What type of spam comes from these bots?

 

they visit your site say 30-100 times per day and rotate refer info which are different sites(http:****.com, etc.) also they dynamically change ips, so banning by ip doesnt help. I guess I just have to sit and watch them

Some websites publish the refer info on their sites, like "last visitor came from bla bla bla", I think this is what these bots are after, to be published. I dont have such hacks on my website, so to me its just a waste of traffic and spammy logs.

 

This type of spam can be filtered on the basis of the collected query statistics. Most likely the IP address as well as repeated and referrers hosts. Perhaps there are particular in the headers which can uniquely identify the data bots. Can you show the log with records?

 

what about banning the whole range of IP??

 

expertofexperts, the ips totally different, some starts with 178, others with 50, its impossible, I think the bot uses proxies all over the world.

supportex, the domains rotate, up to 50 domains, with new and new domains everyday, how are you going to ban that? sample logs:

2011-09-28 15:39:55 W3SVC10642 NS1 [my server ip here] GET /somepage.asp 80 - 109.230.246.54 HTTP/1.0 Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+pl;+rv:1.9.1.3)+Gecko/20090824+Firefox/3.5.3 - http://www.slubny-fotograf.e-rzeszow.com.pl www.mysite.com 200 0 0 19099 279 515
2011-09-28 18:56:03 W3SVC10642 NS1 [my server ip here] GET /somepage.asp 80 - 109.230.246.54 HTTP/1.0 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+.NET+CLR+1.1.4322;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.04506.30) - http://www.ekonomia.artfoto.katowice.pl www.mysite.com 200 0 0 17199 323 515
2011-09-28 02:42:19 W3SVC10642 NS1 [my server ip here] GET /somepage.asp 80 - 173.208.62.13 HTTP/1.0 Mozilla/5.0+(Windows;+U;+Windows+NT+6.1;+en-GB;+rv:1.9.1.3)+Gecko/20090824+Firefox/3.5.3 - http://www.buy-fansfacebook.info/ www.mysite.com 200 0 0 28437 272 187
2011-09-28 00:15:10 W3SVC10642 NS1 [my server ip here] GET /somepage.asp 80 - 108.62.26.27 HTTP/1.0 Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.9.0.14)+Gecko/2009082707+Firefox/3.0.14+(.NET+CLR+3.5.30729) - http://www.sex-dating.co/ www.mysite.com 200 0 0 16937 419 140
2011-09-29 11:51:26 W3SVC10642 NS1 [my server ip here] GET /somepage.asp 80 - 113.254.11.191 HTTP/1.0 Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.9.0.14)+Gecko/2009082707+Firefox/3.0.14+(.NET+CLR+3.5.30729) - http://grannysmusic.com/amazon-music www.mysite.com 200 0 0 14741 302 968

Code (markup):

and so on

 

As a variant you can filter users by User Agent. In your logs bots are using older browsers that are currently hardly used (http://www.w3schools.com/browsers/browsers_mozilla.asp). But in this case is likely to lose some honest users.

 

supportex, yes I know that. I think its not worth it. Let them visit, at least they dont heavily hit the site, 30-50 hits per day is nothing. I just thought someone had a solution to this new spam method

 

Unfortunately there is no simple and unique solution to fight this type of bots.