I have a large number of sites that are all showing the following strange .bot behavior. MSN.bot drops by, sometimes 2 to 6 times a day. First the .bot looks up my robot.txt file, then it visits my index.html file. But that's where it stops. The .bot is not crawling any further on any of my sites. I tried changing my robot.txt file, then I tried putting up a blank robot.txt file. No change. The bot is still stopping at the 2 pages mentioned above. I then tried adding a meta tag, instructing robots to "follow", hoping that this would cause a deeper crawl. No luck. Any ideas of why this is happening? Thanks for any suggestions! Brian
There have been some problems going on with MSN lately, replace the current robots.txt with the old robots.txt, wait for a week, lets see if MSN can repair itself. IT