Twiceler wont stop even when banned from robots.txt

If you are using apache you could try and block then via .htaccess. See here for more info.

 

Its times like this we need Nintendo's advice on .htaccess.

You might want to search around on Google on blocking the IP address of the bot, if they are using the same IP that is.

Did you contact them on to let them know they are ignoring the robot.txt and its eating your bandwidth?

 

PM him and see if he'll reply. Its been a while since I've actually seen him around these parts.

 

Send those wankers an email and let them know what they are doing.

 

The guy actually does respond to email. He responded within a few hours of my query and stopped the robits coming to one of my sites.

Apparantly, the spider does obey robots.txt, but has a 7 day cache which is why it does not take effect straight away.

 

Yeah, they are suprisingly fast to respond and very friendly as well.

 

Brilliant stuff! Good to see he responded so quickly.

 

I'm getting the same problem with that bot. I have sent an email to them.

I hope i will get success like you

 

This is strange, I have blocked them last night by domain name.

deny from cuill.com

but it still coming in. Like you guys say, robots.txt is no use.

I see he's got a more detailed ID so I tried that..

deny from crawl-16.cuill.com

Still no use. This robot seems more persistant than others. When I blocked the IP, he switched and kept going. I emailed them too. This is really a problem where these guys and any joker with a PC can just kill your site and run it into the dirt.

 

If you had even bothered to read this thread, you would have found the answer on how to stop the bots.

I won't give it to you in this post - go read the thread.

 

Yes ive noticed that the bot Twiceler is hammering my site. I don't mind it if its of use but does anyone know exactly what it is doing? If its pointless and of absolute no benefit i'll just get shot of it but if I knew what it was actually for I might let them do it.

Did they give any indication in the emails you received?

Thanks guys

 

I did email Cuill and they got back to me within 24 hours. They didnt really explain much about it apart that it was for a new search engine. They gave me a few options for how to approach the bot going forward. Very polite people.

For the time being I am going to leave it and keep monitoring to see how things develope to see if there is any advantage to allowing the Twiceler robot to scan my site.

Thanks guys.

 

Just to chime in here, that Twiceler robot has been pounding my site for a few weeks now. I don't have any performance issues because the load balancing has been up to the task, so I've let it be under the hope that there might be some future benefit.

I saw that this thread went dead a few months back and thought I'd raise it and see if anyone had any new info on exactly what this is. Above it simply said some new type of search to come out in the future...

Thx