Hi, I need to block the Ask Jeeves bot. It has taken down my site and my provider won't re-enable my site until I agree to block it. However, I'm not too sure of the correct syntax for the robots.txt. Will the following do the trick? User-agent: *ask* Disallow: / User-agent: *Jeeves* Disallow: / Thanks, fcmisc.
Check out this page http://www.robotstxt.org/wc/active/html/askjeeves.html According to it, the exclusion tags are "Teoma" or "Ask Jeeves" or "Jeeves". Try it out. Sorry - I'm not allowed to post a clickable url, you'll need to copy / paste
Thanks. I tried that. But Jeeves doesn't seem to check the robots.txt file too often so I used IP deny to block it. I wish I could redirect it to a dirty p0rn site. That robot is so naughty! Even worse, this downtime has hit me in Google and I've lost 90% of my coop weight! I'm now tempted to block all bots apart from the big 3. If they don't send me traffic why should I pay good money to let them access my site?
I am SO GLAD someone has finally posted a public complaint about ASK JEEVES! I have written them several times and I even called their offices and they just pay no attention so I just blocked them, too. Their bot is indeed a bad bot. They come in and download an ENTIRE copy of the site and all dynamic pages almost every week. They are the Number One Data Downloader in my Webalizer Program every week, and we're talking 10 times what the 2nd highest downloader stat is. If you use programs with a database it will surely bring down your site and .... I NEVER get traffic from them because they fill the first scroll with PAID ADVERTISERS ONLY! I say we all boycott them. We pay for their access, bandwidth, and then they cheat us out of leads. Yep. No traffic, no access.
It took a while before they stopped coming to my site. What I did in the end was to block the spider's IP address using the my website's control panel. That way, all the spider saw was a forbidden page. I didn't get any traffic from Ask and don't care that my site isn't in their search engine.