Stoping Referrer Spam

kusadasi-guy

Mar 24th 2005, 3:23 pm

i am hitting by wan*er referrer spammers (like you all). How can i stop them?
They are using proxy servers, so that banning thier ip addresses is not a solid idea. ANd i dont want to reduce my servers performance with putting hundreds of lines to htaccess ( deny from spammer-casino/dot/com ...etc)

i scanned my logs and if i block some keywords then it would be prevent %80 of these spammers.

For example; if referrer link contains "casino" , "pharmacy" keywords, then stop it.

How can i do that with htaccess?

digitalpoint

Mar 24th 2005, 3:35 pm

There isn't a "sure fire" way to do it unfortunately... but if you check some of the similar threads (look at the bottom of this page), there are some other threads about it.

Smyrl

Mar 24th 2005, 3:38 pm

I do not have the answer, wish I did. I have seen two methods used that have some appeal to me. Both involve using .htaccess and doing something with sites based on

Words appearing in URL
URLs havng two or more dashes in them.

I have lost the URL showing first method. If someone has it I would appreciate your posting the link.

Thanks,
Shannon

kusadasi-guy

Mar 24th 2005, 3:51 pm

Smyrl

Mar 24th 2005, 7:19 pm

I found it!

RewriteEngine On

SetEnvIfNoCase Referer ".*(casino).*" BadReferrer
SetEnvIfNoCase Referer ".*(pharmacy).*" BadReferrer
SetEnvIfNoCase Referer ".*(gambling).*" BadReferrer
order deny,allow
deny from env=BadReferrer

IF a domain name (or its subdomain name) contains "casino", "pharmacy" and/or "gambling" then deny it.

I can prevent %70 of all my spammers with just few words.

Should i also put these lines to httpd.conf file to prevent all of the domains in the server?

Here is other code I have found.

RewriteEngine On
RewriteCond %{HTTP_REFERER} ^(http://www.)[a-z]+-[a-z]+- [NC]
RewriteRule ^(.*) http://%{REMOTE_ADDR}/ [R=301,L]

Can I use both to try to a double barrel shotgun approach?

Shannon

danpadams

Mar 25th 2005, 8:39 am

Smyrl, What does your code do exactally?

iShopHQ

Mar 25th 2005, 9:30 am

If you're on a Windows box, you can set up a similar function in the global.asa file. Grab incoming referers, do an INSTR() check, and then a response.end that stops them in their tracks.

J.D.

Mar 25th 2005, 9:39 am

Smyrl, What does your code do exactally?It checks if the referrer string matches URLs like these:

http://www.abc-def-
abc-def-

And if it does, redirects the browser to the IP address of the browser itself.

J.D.

danpadams

Mar 25th 2005, 9:45 am

I hate to say it, but that is sorta being downright mean. In my opinion from the way I have my personal stuff setup, it wouldn't be as mean as possible as I have a webserver on my outside address, but that would mean the bandwidth would be used (the webserver is not in my own box but on the same IP #, why not just point to http://127.0.0.1?

Smyrl

Mar 25th 2005, 9:51 am

It checks if the referrer string matches URLs like these:

http://www.abc-def-
abc-def-

And if it does, redirects the browser to the IP address of the browser itself.

J.D.
J.D., Can I use this

RewriteEngine On

SetEnvIfNoCase Referer ".*(casino).*" BadReferrer
SetEnvIfNoCase Referer ".*(pharmacy).*" BadReferrer
SetEnvIfNoCase Referer ".*(gambling).*" BadReferrer
order deny,allow
deny from env=BadReferrer

and

RewriteEngine On
RewriteCond %{HTTP_REFERER} ^(http://www.)[a-z]+-[a-z]+- [NC]
RewriteRule ^(.*) http://%{REMOTE_ADDR}/ [R=301,L]

both in my .htaccess?

Shannon

J.D.

Mar 25th 2005, 10:04 am

J.D., Can I use thisI never used SetEnvIf and can't say anything regarding its performance, but the fact that it assigns variables makes me think that it is probably not as fast as mod_rewrite.

I would rewrite the expressions as keyword instead of .*(keyword).* for performance reasons, but otherwise, either method or both should work fine.

J.D.

kusadasi-guy

Mar 25th 2005, 10:52 am

Hello JD,

As i understand from your reply, these lines are better than SetEnvIf rules, right?

RewriteEngine On
RewriteCond %{HTTP_REFERER} (casino) [OR]
RewriteCond %{HTTP_REFERER} (pharmacy) [OR]
RewriteCond %{HTTP_REFERER} (gambling) [NC]
RewriteRule .* - [F]

J.D.

Mar 25th 2005, 1:11 pm

These rules will return 403 (forbidden) when referrer contains any of the specified words:

RewriteCond %{HTTP_REFERER} poker [OR,NC]
RewriteCond %{HTTP_REFERER} casino [NC]
RewriteRule ^.? - [F]

Edit: I think rewrite rules will work faster, but I didn't actually run a test to verify this.

J.D.

J.D.

Mar 25th 2005, 1:38 pm

Also, don't forget, the order in which rewrite rules are specified does matter. Rules denying access should go first (otherwise some of the perpetrators may slip through if they hit a preceeding rule with an [L] flag).

J.D.

kusadasi-guy

Mar 25th 2005, 1:45 pm

Seems work perfectly J.D.
Thank You so much

Here is my new htaccess;
RewriteEngine on
RewriteCond %{HTTP_REFERER} pharmacy [NC,OR]
RewriteCond %{HTTP_REFERER} viagra [NC,OR]
RewriteCond %{HTTP_REFERER} porn [NC,OR]
RewriteCond %{HTTP_REFERER} casino [NC,OR]
RewriteCond %{HTTP_REFERER} gambling [NC,OR]
RewriteCond %{HTTP_REFERER} phentermine [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^LWP* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^lwp* [NC]
RewriteRule .* - [F]

BTW, whats the difference between "RewriteRule ^.? - [F]" and RewriteRule .* - [F] ?

J.D.

Mar 25th 2005, 1:51 pm

RewriteCond %{HTTP_USER_AGENT} ^LWP* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^lwp* [NC]These are the same - [NC] stands for canse-insensitive string comparison.

BTW, whats the difference between "RewriteRule ^.? - [F]" and RewriteRule .* - [F] ?The first one will work faster because the regular expresion parser will have to match only one optional character at the beginning of the referrer string. The second one means "one or more" and the parser may need to process more characters.

J.D.

kyle422

Mar 25th 2005, 2:25 pm

Thanks for posting the solution kusudasi. I put it in my .htaccess and it works well for me. :)

kusadasi-guy

Mar 25th 2005, 2:36 pm

J.D.

Mar 25th 2005, 2:50 pm

Thanks to JD actually, i learned from him.

Just last thing, how can i do it for my server's websites? Should i add that lines (except "RewriteEngine on") to my httpd.conf file? or add to another file?Rewrite rules can be placed in the server config, virtual domain sections or in the .htaccess file. Your choice.

J.D.

TwisterMc

Mar 25th 2005, 3:09 pm

Any chance this is for a wordpress blog? I just implemented a plugin that helps block referrer spam.

kusadasi-guy

Mar 25th 2005, 3:31 pm

works perfect for all kind of websites that experiencing referrer spam.

danpadams

Apr 1st 2005, 9:59 am

Is this plugin available for others of us, or can someone sum up the .htaccess code that is thought of to be the best?

Smyrl

Apr 1st 2005, 10:06 am

I have different code running on different sites. Am looking to start over this month and monitor stats on daily basis and see which methods work best for me. I just did not feel like I have time to monitor the referrer stats daily. As someone said, no matter what approach we use think we are going to have to use some individual blocking for sites that do not fit patterns.

Shannon

kusadasi-guy

Apr 1st 2005, 10:13 am

RewriteEngine on
RewriteCond %{HTTP_REFERER} poker [NC,OR]
RewriteCond %{HTTP_REFERER} viagra [NC,OR]
RewriteCond %{HTTP_REFERER} porn [NC,OR]
RewriteCond %{HTTP_REFERER} casino [NC,OR]
RewriteCond %{HTTP_REFERER} gambling [NC,OR]
RewriteCond %{HTTP_REFERER} phentermine [NC,OR]
RewriteRule ^.? - [F]

i stopped %58 of my spammers with that htaccess file. instead of running lots of query in my MYSQL server (i have DB driven website), i am now showing basic html file (403 forbidden file).

here is a result from my log file;
09.03.2005;
total visits: 1044
blocked visits (403 forbidden): 223

i notice that there are still 130-160 or so visits that trying referrer spam.
So, my success is 223/160+223 = %58

danpadams

Apr 1st 2005, 10:23 am

Can you refresh my memory, what does the NC stand for as in the [NC,OR] part?

kusadasi-guy

Apr 1st 2005, 10:45 am

NC makes it not case sensitive

danpadams

Apr 1st 2005, 10:51 am

I tried to implement it and the exact code I used is

RewriteEngine On
Options +FollowSymlinks
RewriteBase /
#
RewriteCond %{HTTP_REFERER} poker [NC,OR]
RewriteCond %{HTTP_REFERER} viagra [NC,OR]
RewriteCond %{HTTP_REFERER} porn [NC,OR]
RewriteCond %{HTTP_REFERER} casino [NC,OR]
RewriteCond %{HTTP_REFERER} gambling [NC,OR]
RewriteCond %{HTTP_REFERER} phentermine [NC,OR]
RewriteRule ^(.*)$ http://spam.infochi.net/referrer.html [L]

When I tried it in a broswer request with an empty or blank referrer, it sent me to the referrer.html page that I put in there. Can anyone help me understand why?

TwisterMc

Apr 4th 2005, 11:53 am

The plugin i tried earlier, didn't work worth crap. I don't know what I did wrong. I installed referrer karma for wordpress and that works, just not as good as hoped. I have something screwed up with that too. I'll look into that htaccess idea next. It sounds good to me.

I just don't see how referrer spam can help the referring site? So what's the point?