Referrer logs are being spammed

[dchapman]

My awstats logs are full of this crap:

- http://www.progressiveupdate.net/poker-games.html 171 171
- http://www.progressiveupdate.net/free-poker.html 95 95
- http://www.progressiveupdate.net/empire-poker.html 85 85
- http://www.progressiveupdate.net/poker-rules.html 84 84
- http://www.progressiveupdate.net/poker.html 83 83
- http://www.progressiveupdate.net/party-poker.html 82 82
- http://www.progressiveupdate.net/texas-hold-em.html 81 81
- http://www.progressiveupdate.net/pacific-poker.html 79 79

How do I tell if my logs are open to the public and this is being counted as links out from my site? What can I do to prevent this kind of crap?

Thank you for your help.

[Smyrl]

If you are running an Apache server block with your .htaccess file.

So a search on this forum for referrer spam with some lines of code you can add to your .htaccess files or do a Google Search for referrer spam for various suggestions.

Referrer spam is an ongoing battle.

Shannon

[J.D.]

How do I tell if my logs are open to the publicView you stats pages through a proxy of some kind (e.g. http://www.anybrowser.com/siteviewer.html). If they can see it, anybody else can.

J.D.

[sarahk]

here's a list of people who have got their logs open to Google

http://www.google.com/search?q=progressiveupdate+poker%2Ehtml

are you on it?

Yes, use .htaccess

Don't have the links in your logs clickable or use <a href="" rel="nofollow">

don't fret about it but consider banning the IP if it's causing bandwidth problems.

I've got an article from 2003 on referral spam: http://sarahk.pcpropertymanager.com/blog/referral-spam/55/ it's not going to go away - we just have to make sure we don't fall into their traps :)

Sarah

[J.D.]

here's a list of people who have got their logs open to Google

http://www.google.com/search?q=progressiveupdate+poker%2Ehtml

are you on it?The fact that his website isn't on Google doesn't mean that his logs are not publicly accessible. Besides, he's running AWStats and this package has the following statement that prevents legitimate robots from indexing stats pages:

<meta name="robots" content="noindex,nofollow" />

Don't have the links in your logs clickable or use <a href="" rel="nofollow">There's no such value as nofollow for rel:

http://www.w3.org/TR/REC-html40/types.html#type-links

J.D.

[sarahk]

It's the individual link that has no follow

http://www.w3.org/TR/REC-html40/struct/links.html#h-12.2

and here's the official word from Google

http://www.google.com/intl/en/googleblog/2005/01/preventing-comment-spam.html

Sarah

[I, Brian]

Referrer spamming is actually an old issue. I'm not sure if the SE's actually ever totally ignored those links, but they are certainly aware of the method. Doesn't stop it still being employed, of course - I see plenty of crap from gambling, porn, and pharma in my logs.

[J.D.]

It's the individual link that has no follow

http://www.w3.org/TR/REC-html40/struct/links.html#h-12.2No it doesn't. Read the page you quoted. According to W3C, it is valid to define your own link types, but it has to be accompanied by a reference to the appropriate profile:

Authors may wish to define additional link types not described in this specification. If they do so, they should use a profile to cite the conventions used to define the link types. Please see the profile attribute of the HEAD element for more details.

This makes nofollow specified for an individual link a Google-specific extension. If other search engines pick up on this initiative, then it will be a good tool to fight blog comment spam.

Note that in case of the referrer spam, all stats pages should be accompanied by the noindex/nofollow meta tag to prevent engines from even indexing these pages (indexing these pages is a good source of info for hackers).

J.D.

[Bernard]

J.D., Last I knew, Google, Yahoo & MSN are on board with support for rel='nofollow'. Teoma/Ask Jeeves is not.

[J.D.]

J.D., Last I knew, Google, Yahoo & MSN are on board with support for rel='nofollow'. Teoma/Ask Jeeves is not.Good to know. Thanks!

[just-4-teens]

heres a nifty bit of code, put it in your .htaccess (requires apache)

RewriteEngine On

RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?pharmacy(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?phentermine(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?incestporn(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?rapeporn(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?diet-therapy(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?blonde-mature(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?phntrmn(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?car-insurance(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?sex(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?erotic(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?texas-hold(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?weight-loss(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?free-ringtone(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?airline-ticket(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?poker(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?ringtone-download(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?progressiveupdate(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?cialis(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?merchant-account(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?cialis5(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?cialisonline(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?phentermine2(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?phentermine45(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?scalaroi(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?cruelnitio(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?health-insurance(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?tramadol(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?vicodin(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?hydrocodone(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?permento(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?airline(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?cardinalo(-|.).*$ [OR]

RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?auto-insurance(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?free-asian(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?nylon-pics(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?jamie-lee-curtis(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?homesexsearch(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?spyware-removal(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?bigsitecity(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?djsdesigns(-|.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?astellsci.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?twentyfirstcc.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?lassaro.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?setiatpalm.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?iberbrasconsultoia.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?ybsearch.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?noamkts.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?setiromania.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?free-spyware-removal-download.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?download-free-ringtone.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?racepointfunding.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?warrenzanes.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?fresnay.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?adipex.epinoy.*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?bigsitecity.*$

RewriteRule .* - [F,L]

[exam]

The only thing is huge .htaccess files can slow down your server.

[sarahk]

That's the trade off in this case... Slow server but reduced bandwidth drain or the other way around.

I get my post notifications through gmail - it didn't like the post with the .htaccess code. Thought it was spam. You'd have thought it would think "not even a spammer would have that much crap in one email!" :)

Sarah

[jorge]

can't you put the awstats behind user/password?

[Smyrl]

Supposedly mine is but I still get spammed. Maddening.

Shannon