I notice MSN and yahoo do not crawl a number of my links. It visits links with depth 2, like /content/homepage.html but not longer. I am using mod_rewrite to make some of the links "nicer", which is making them longer. as an example: /product/home/bathroom/tile/cleaner.html these types of links are located on a page linked off the hompage, so the bot does not have to crawl far to find them. google and ask visit those pages, i think due to the sitemap.xml file. why won't msn or yahoo visit them? I am tempted to do some magic to make them shorter, but wanted to verify that is the true issue.
Someone has said that you can use the xml sitemap within the robots file to get MSN to index your site deep. Does anyone have any idea on how to go about this?