HTTrack 3.0x webcopier, how to block it....

I've just checked my server logs for this month, and I got an (unpleasant) surprise when I got to the the 23. nov..

40-50,000 pages extra and 1gb more than usual, but no more visitors..

So opened my raw server logs and found this one (a lot) in there:
"Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)

Looks like some b***ard have been using this http://www.httrack.com/?pat to download my entire site and forum..

How do I prevent this from happening again?? .htaccess ??

And I've also noticed that my traffic has gone down 30%-40% the last few days.
But I can't see any disturbing changes in the SERPs, at least not for the keywords I monitor..

Is there any way I can find out if is because of the copying (duplicate pages in google??)

Thanks..

 

I've tried copyscape, but it will be impossible for me to check all of my pages there, especially the forum..

I found nothing on the index page, or some of my high ranked pages.

But I can't understand the sudden drop in traffic at all, all of the keywords I try in google ranks like they have been for a while (or better)..

 

I hope it's that..

Didn't really know that it was a holiday weekend, I live in Norway myself..
Thanks for telling me..

How long did (do?) the holiday last??

The number of visitors dropped from 11-13,000 pr day on average last week (15-19 nov), to 6,500-9,000 this week (22-26 nov.)



Any suggestions to how I can block webcopiers like that in the future??

 

If you want to block by IP: In your .htaccess file put (back up your original .htaccess as a precaution):

deny from 211.157.8.44
deny from 202.

The first example blocks one ip; the second blocks a range in the 202 block.

BLock an address like so:

RewriteCond %{HTTP_REFERER} ^http://www.domainname.com/* [OR]

I have an example somewhere for blocking useragent as well; looking for it.

Here we go; found it:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCapture.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Webdupe.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Pockey.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^DiscoPump.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InternetSeer.com.* [NC,OR]
RewriteRule .* - [F,L]

If you use that list; be careful. You can lock yourself out of your site via http.

To block httrack; try using:

RewriteCond %{HTTP_USER_AGENT} ^HTTrack 3.0x.* [NC,OR]

or

RewriteCond %{HTTP_USER_AGENT} ^HTTrack.* [NC,OR]

You may have to find a different variation.

 

You can change Httrack's user-agent to whatever you want, so I think your only option is to block by IP.

 

I don't think it's any good to block the IP either, since anyone can use a webcopier, and the guy who used HTTrack will probably not visit my site again (or in a while) anyway...

Thanks for the info Shannon, I hope the traffic is back on track on monday again..

 

Unless you see some malicious activity (eg. a deliberate DoS attack), there's really not much you can do about this, except throttling bandwidth per connection (say @ 100-200 KBps). You can do a massive download like this using just about anything nowadays, even IE (using offline links crawler).

IP address blocking won't give you much unless you start blocking entire ISPs, which may affect your legit users.

Blocking user agents is a bit better - at least average users won't be able to hit you with massive downloads like this. User agent strings may be easily changed, however, so this extra work is hardly worth the effort.

J.D.

 

I think I'll block some known useragents for now..

Thanks for your help guys and gals..