I have some links which has nofollow but yandex crawl is simply crawling and makes my all script to run. Please tell me how can i stop this crawling? Actually, I am redirecting to other site by link and sending mail but i am geting number of mails because of this yandex crawl. I already used below but its not working. User-agent: yandex bot Disallow: /
if your host have mod_rewrite, you can use this tutorial http://diveintomark.org/archives/20...pybots_and_tell_unwanted_robots_to_go_to_hell
Look here: http://help.yandex.ru/webmaster/?id=996567 You should write: User-agent: Yandex Disallow: /