Any possible solutions to get rid of this bots permanently... Checked my last visitors and found that below given bot visiting my site continously. Agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) So i have added below given in my robots.txt User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) Disallow:/ Still it's visiting my site and consumes bandwidth unnecessarily. What to do now to get rid of this bots
Thanks for this info. I have already added below given to my .htaccess RewriteCond %{HTTP_USER_AGENT} ^Twiceler RewriteRule ^.* - [F] But still its visiting
Its times like this we need Nintendo's advice on .htaccess. You might want to search around on Google on blocking the IP address of the bot, if they are using the same IP that is. Did you contact them on to let them know they are ignoring the robot.txt and its eating your bandwidth?
Their ip address is not same everytime Already blocked their few ip addressess) Hope Nintendo going to help me to solve this problem..
Just sent PM to him and asking for help. Hope to hear from him soon as currently my website is down due to reach its bandwidth limit.
The guy actually does respond to email. He responded within a few hours of my query and stopped the robits coming to one of my sites. Apparantly, the spider does obey robots.txt, but has a 7 day cache which is why it does not take effect straight away.
Oh! That means i have to wait 7 day for that. Thanks Claudek for this useful info. I am not aware about this. BTW i have also send them email with my site url details and hope they also stopped the robots coming to my site. Edit : Surprise that i got reply within just 20 mintues from them. They said sorry for this and stopped the robots comming to my site.
I'm getting the same problem with that bot. I have sent an email to them. I hope i will get success like you
Sure why not. You have to send them email on including your url where bot visiting requesting to stop it. They are very co-operative as per my experience. DON.
This is strange, I have blocked them last night by domain name. deny from cuill.com but it still coming in. Like you guys say, robots.txt is no use. I see he's got a more detailed ID so I tried that.. deny from crawl-16.cuill.com Still no use. This robot seems more persistant than others. When I blocked the IP, he switched and kept going. I emailed them too. This is really a problem where these guys and any joker with a PC can just kill your site and run it into the dirt.
If you had even bothered to read this thread, you would have found the answer on how to stop the bots. I won't give it to you in this post - go read the thread.
Yes ive noticed that the bot Twiceler is hammering my site. I don't mind it if its of use but does anyone know exactly what it is doing? If its pointless and of absolute no benefit i'll just get shot of it but if I knew what it was actually for I might let them do it. Did they give any indication in the emails you received? Thanks guys
I did email Cuill and they got back to me within 24 hours. They didnt really explain much about it apart that it was for a new search engine. They gave me a few options for how to approach the bot going forward. Very polite people. For the time being I am going to leave it and keep monitoring to see how things develope to see if there is any advantage to allowing the Twiceler robot to scan my site. Thanks guys.
Just to chime in here, that Twiceler robot has been pounding my site for a few weeks now. I don't have any performance issues because the load balancing has been up to the task, so I've let it be under the hope that there might be some future benefit. I saw that this thread went dead a few months back and thought I'd raise it and see if anyone had any new info on exactly what this is. Above it simply said some new type of search to come out in the future... Thx