zipdisk
Mar 13th 2006, 3:31 pm
Is there anyone bothered by this bot "FAST MetaWeb Crawler?" It has been downloading my pages every 1-3 minutes for more than one day. What is it doing? I wonder if you have any good solution to prevent it. I have tried banning its IP and robots.txt. But both are useless. It is still downloading my pages and consuming my bandwidth.