hello, this is my first post..hope this is the right section. Got a problem on a italian website: yahoo sends me about 5% of the traffic sent by Google. No good. So, analyzing the logs, I've found that Yahoo slurp tries to crawl directories even if they're not linked. For example, given a page /somedir/somepage the spider tries to crawl /somedir/ even if there are no links to /somedir/ Somedir doesn't exist because I use htaccess, and the spider gets a wrong page with no content and <meta name="robots" content="noindex,nofollow">. can this be the reason why i get poor rankings on Y? thanks to everyone who will share his/her knowledge
Yes, I noticed Slurp doing that on my website. I suggest you setup some RewriteRules in your .htaccess to serve 404 or 301 in those cases. I don't think that this affects your rankings on Yahoo, though it is possible that they are looking for a good directory structure. You could experiment and add some valid page for /somedir/ so that the bot does find something to parse... Cryo.
Thank you, you've helped me.. I was wondering if there was some unwanted link to /somedir/ on my website, but if you noticed Slurp doing that on your website, then it's fine. Now I'm trying to redirect 301 to some valid page for /somedir/ so that the bot does find something to parse
Yahoo sometimes will ignore nofollow attribute, you'd better use robot.txt if you don't want yahoo spider to crawl and index a page.