All of my sites were indexed by MSN before Google and Yahoo, but lately, I am finding that it's indexing new pages and sites alot slower (2 weeks and I'm still waiting to have new pages indexed compared to 2-3 days before) Is anyone else finding the same? I read somewhere that Google would be knocked off the No1 search engine spot in 2006 - what a joke that is with the likes of MSN and Yahoo as it's competitors! Darren
8h18m is the average for MSN crawling my pages 4h55m for google Yet, i am getting more results from MSN at the moment. Just in case you were interested in some stats. Good luck.
It definitely says something that there's one thread for people who think they're getting crawled too much and another one for people who think they're not getting crawled enough, and that they're BOTH active. In many cases, when someone has complained that we quit crawling his or her site, it has turned out that recent changes to the robots.txt file had accidentally excluded us from most/all of the site. Remember that Microsoft interprets robots.txt in a case-independent way, so if you exclude us from "/I" (for example) you have told us not to crawl your index.htm file, even if that wasn't your intent.
Eh? What if there is no robots.txt file and even then the site's not being crawled anymore? And what's with "us"? Are you a search.msn.com representative?