Stopping Referrer Spam

[Smyrl]

Two Questions

1. Will banning the IP of the site spamming my log files keep spammer out?

2. If I use .htaccess to combat the spam and want to block the domain myjunk dot com and all subdomains can it be done with the one line

RewriteCond %{HTTP_REFERER} ^http://www\.)?myjunk.com.*$[NC,OR]

or will I have to specify each sub domain seperately?

Thanks for your help.

Shannon

[Bernard]

I'm not an expert on this, but in answer to Q1, I think the answer is no. Referral spam is usually handled by a program and they can usually be configured to operate through proxies, so the the spam is not necessarily coming from the same IP as the site they are "promoting".

[Smyrl]

Thanks, Bernard. What you said made sense.

I have made an attempt at altering a .htaccess file for first time today. Since I have never touched one before I do not know whether it will work or not but at least I can still reach site.

Shannon

[neterslandreau]

Originally Posted by Smyrl

1. Will banning the IP of the site spamming my log files keep spammer out?

I'm not sure what you mean by Referer Spam but I always get crackers trying to hack into my servers. When I notice someone attempting to breach my security, I add the IP to my hosts.deny file and deny them access to my machine, period.

[Smyrl]

neterslandreau, I am not the best person to describe referrer spam. You really should read something about it. If you look at your stats and the list of referrers you will see URLs for sites that have no liinnks to your pages. Porn, pharmeseuticals, gambling sites are among the worst.

I have read three stop-gap methods of altering your .htaccess file if you are running Apache. Since I know nothing about syntax for coding and am such a horrid typist I hated to touch mine but touch two I have. I will be watching to see if I continue to get spam.

It seems we all need to understand the blackest of black hat to try protect our sites.

Shannon

[Bernard]

Originally Posted by Smyrl

... If you look at your stats and the list of referrers you will see URLs for sites that have no links to your pages. Porn, pharmeseuticals, gambling sites are among the worst.

... and web hosting companies.

[neterslandreau]

Originally Posted by Smyrl

It seems we all need to understand the blackest of black hat to try protect our sites.

That's very sad, isn't it?

BTW, I googled referrer spam, saw all kinds of ways to protect against it (thank you ), but in all the time I've been admin'ing sites, I've never run across this problem, or at least never realized I was being exploited.

[J.D.]

Originally Posted by Smyrl

1. Will banning the IP of the site spamming my log files keep spammer out?

Some of them operate from behind a DSL connection and blocking these addresses is impractical - who knows, may be somebody who'd get the address next would buy something from you I do check on a case-by-case basis who's the owner of the IP address and if it's a dedicated server, I usually block it for some time.

Originally Posted by Smyrl

2. If I use .htaccess to combat the spam and want to block the domain myjunk dot com and all subdomains can it be done with the one line

You didn't show the rule, so I'm assuming that you are redirecting spammers somewhere else. I haven't checked, but I would think that in this case the log line would still show the spamming referrer, which kind of defies the purpose of this exercise. Some log analyzers process log lines with HTTP errors differently, so if you return some error (e.g. 404, not found), you may avoid this referrer in the reports.

J.D.

[john_loch]

Hi Smyrl,

The only static way to kill referrer spammers is by IP. Unfortunately that's the crux of it. The only dynamic way to do this (that I've used) is to look for discreet requests to pages or specific files that:
a. are files that should have a local referrer (ie images/css files that are typically subrequests of a page) or:
b. Should result in subrequests for images and the like (ie .html, .php etc) but don't.

This will be problematic for users who either:
a. Use text browsers (ie Lynx etc)
b. Use custom software for the visually impaired.

At the end of the day though (no matter how you garner the info), you end up having to filter them on an IP basis.

Bit of a bugger really :|

Cheers,

JL

[Smyrl]

Thank each of you for your replies.

I will continue making attempts to block spam.

Shannon

[SEbasic]

Keep us informed...

Referral spam on my sites has been increasing steadily over the past few months and It's now really starting to tick me off.

[T0PS3O]

Isn't referral spam only an issue when you have public stats that can be indexed by the spiders?

[J.D.]

Originally Posted by T0PS3O

Isn't referral spam only an issue when you have public stats that can be indexed by the spiders?

Very good point. Everybody has to make sure that access to all stats pages is properly authorized. At the very minimum all stats pages must have noindex and nofollow meta tag defined.

J.D.

[T0PS3O]

I just password protected (htaccess) my webalizer stats folder. The way I understood the phenomenon is that it's killed by doing just that.

No access = no index = no link popularity = no reason to spam

Searching Google for two distinct webalizer (or similar stats package) lines of text (including the misspelled explaination) shows who can be spammed successsfully:

http://www.google.co.uk/search?hl=en...G=Search&meta=

Just 47 results but if you get creative with the search there must be millions.

[SEbasic]

They still spam you either way.

My stats are protected on all of my sites.

[J.D.]

Originally Posted by T0PS3O

I just password protected (htaccess) my webalizer stats folder. The way I understood the phenomenon is that it's killed by doing just that.

No access = no index = no link popularity = no reason to spam

Some of these pesky spammers make a quite few hits in order to be noticed and this distorts the stats for some sites quite a bit. Blocking IP's or doing some other filtering helps to avoid this problem.

J.D.

[T0PS3O]

Originally Posted by SEbasic

They still spam you either way.

But then all you get is inflated stats. I'd worry more about customers than spammers.

[SEbasic]

I don't like inflated stats.

Makes quick stat checks difficult.

[neterslandreau]

Originally Posted by SEbasic

I don't like inflated stats.

I don't either. Because I pretty much operate on a shoestring budget and it is impossible for me to ignore my own clicks, I always create a subdomain that keeps no logs and do my developing thru that.