So this new spider has been visiting my site for the past few days. http://www.cuill.com/twiceler/robot.html I don't really care what they do, but they are visiting random pages, (and are served 404 pages). Why are they visiting those pages? What are they trying to achieve? No other search engine EVER go to those pages, (not google, yahoo, nobody), so that make me think that the link does not exist and they are trying their luck and my site security. Does anybody know more about those guys? should I just ban their IP? FFMG
Twiceler is all over my forums. Hopefully there will be some return in the future on all the bandwidth it uses.
Forgive my ignorance, but what the heck does Cuill do? They claim to be a search company... yet no search functionality exists on their site. So why are they crawling?
they are only expiremental as far as I can tell, so it may be a while before any search functionality exists. My sites have been hit also but I'll leave them spider all they want at the moment and see what happens later.
Like I said I don't mind new spiders, but I wonder why they are hitting so many 'not found' pages. Their algo must be broken or something, 'cause I have no idea where they would have picked up some of those links. Either that or they are trying something fishy on my site. FFMG
Samething happened with me few months back. Check it for further details here. http://forums.digitalpoint.com/showthread.php?t=377143 Now no more worries for me DON.
Thanks, I'll send them an email. And if it doesn't stop I'll ban them, (seen that they ignore robot.txt requests). FFMG