I've had a site up for about 8 months now. Recently MSN has had anywhere from 9k-19k pages indexed, yahoo has had lows of 900 to highs of 9100. Google however, has stayed ~120 for almost the entire time. It barely even crawls me. It got stuck on a phpbb sid and crawled that about 6000 times, and then just seems to have quit. This seems like very, VERY poor performance on google's part. Anyone else see anything like this? MSN crawls me everyday, yahoo is a bit slower, and erratic in it's indexing of pages, but google is down right terrible!
Yes, I have experienced all that you have mentioned above from google Google at times only visits two pages a day on the site I had the same problem as yours.
Yea, it crawls about 5 a day. And many times, the pages it has crawled never get indexed, can't find them via searching. I've done some experimentation, randomizing parts of the page and such. Bypassing the heirarchy or links, since they seem to crawl depths at a time, and my site has several categories, then large lists. And yes, now I have a google friendly phpbb install. But not until I saw google had crawled the same page 6000+ times, literally. It just seems very poor design for their search engine to not recognize a sid, or not recognize the same content. I mean, sure, I could see it crawling it 50 times, or 100, but after 3000 times, where the sid is the only content changing, perhaps it's time to move on. Neither MSN or Yahoo had a problem with sids that I could see. I find it very strange. Every slightly tech-ish person says google is the best, but after having this site up, I'm pretty sure msn must list much more new content every day, google is just too slow to keep up, or so it seems.