Ask Jeeves/Teoma userAgent: 'Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)' has crawled my site, no problem so far, but it has downloaded 20GB of data in the last 4 weeks and haven't sent me one (1) single user... I've the same experience with other sites: a lot of crawling (up to 1/sec, all day long), no referrers. Should I put this spider in my /robots.txt or does it take loooooooong before I can expect results? any ideas on this anyone?
That's a great question! ask/Teoma bot has been in my site every second for two weeks. It seems to be trying to crawl all the pages. I've not banned it because they appear to be the distant #4 to G/Y/M and I'd love to get some traffic from them (as you, none yet). Let me know if you see any changes. -jay