Hi, I'm monitoring the spider activity of a large (newly redesigned) site. Google has indexed 17.000 pages since the first february but Yahoo is stuck on 200 pages. (MSN has 2.000 pages indexed.) Yahoo was the first spider to come and index and indexed thousands of pages in january but it stopped arround the first february. Anyone an idea why Slurps acts like that or how to push it into spidering again ?
yahoo is a lot slower at updating its database of pages from a site, if the yahoo bots is always on your site i would not worry about it.
That has been true for as long as I've been watching Slurp... the only bot I ever had to slow down because it was hitting pages at a rapid rate and it wasn't translating into anything very useful, at least at the time, in terms of Yahoo rankings. By the way, for those of you who want to do this, add the following to robots.txt: User-Agent: Slurp Crawl-Delay: 10 Code (markup): You can also use this for MSNbot: User-Agent: msnbot Crawl-Delay: 10 Code (markup): The above code instructs Slurp and MSNbot respectively to wait the specified amount of time, in seconds (10 seconds above, default is 1 second if not specified) before requesting another page from your site. MSNBot and Slurp have been known to index some sites very heavily, and this allows webmasters to slow down their indexing speed. Googlebot is normally better behaved by default.