Blocking random robots

Discussion in 'Site & Server Administration' started by dpak, Oct 14, 2005.

  1. #1
    Has anyone had issues with random robots scanning their pages frequently? There was a particular IP address spidering my site everyday. The User Agent just said it was IE and there was no Referer. I'm only guessing it was a spider because the pages it was visiting were not connected to each other by links and the time between page views was not normal for a user (too long between different page views). My site has a fixed amount of information and is not one that you would come back to everyday...

    For right now, my pages are just not displaying content for that IP, but I sort of worry that it might be a legitimate robot from link-vault.com or linkmarket.com (both of which I am a part of)...

    Anyways... my question is... has anyone else had problems with strange spiders? And if so, did you do anything about it?
     
    dpak, Oct 14, 2005 IP