Yahoo's spider crawls not linked directories

hello, this is my first post..hope this is the right section.

Got a problem on a italian website: yahoo sends me about 5% of the traffic sent by Google. No good.

So, analyzing the logs, I've found that Yahoo slurp tries to crawl directories even if they're not linked. For example, given a page /somedir/somepage the spider tries to crawl /somedir/ even if there are no links to /somedir/

Somedir doesn't exist because I use htaccess, and the spider gets a wrong page with no content and <meta name="robots" content="noindex,nofollow">.

can this be the reason why i get poor rankings on Y?

thanks to everyone who will share his/her knowledge

 

Yes, I noticed Slurp doing that on my website. I suggest you setup some RewriteRules in your .htaccess to serve 404 or 301 in those cases.

I don't think that this affects your rankings on Yahoo, though it is possible that they are looking for a good directory structure. You could experiment and add some valid page for /somedir/ so that the bot does find something to parse...

Cryo.

 

Thank you, you've helped me.. I was wondering if there was some unwanted link to /somedir/ on my website, but if you noticed Slurp doing that on your website, then it's fine.

Now I'm trying to redirect 301 to some valid page for /somedir/ so that the bot does find something to parse

 

Yahoo sometimes will ignore nofollow attribute, you'd better use robot.txt if you don't want yahoo spider to crawl and index a page.

 

for the last few months slurp has been running havoc on my site...I just let it be