I find several errors in my site's C Panel Error log. Most of the errors are caused by MSN bot asking for non-existant files (to be precise, files are there, but the path is incorrect). For example, the file [Fri Apr 6 16:16:50 2007] [error] [client 65.55.209.10] File does not exist: /home/vy010101/public_html/networking/about-us.html is not there. But, a file is there at location: /home/vy010101/public_html/about-us.html There is a sub-domain by name "networking", that shows up as a folder under public_html. Most of the errors are of the above type. The file name exists, but the path is incorrect. The site have several sub-domains, and as a result, several file errors. I have no problem with other bots or SEs. Any feedback is highly appreciated. PS: Reverse DNS on IP is resolving to livebot-65-55-209-10.search.live.com.
Are you sure that the internal linking on the subdomains is correct? Also, the bots can also follow outdated links from other sites to your pages.
The internal linking has been good. Now, I have put htaccess file in all sub domains, with "GONE" directive for each "File not found" server error on my domain.