What happened with the GoogleBot? Last month, or it might've been the month before that, and all the months before, the googlebot would visit 800 (average) times per day. Now its 220 so far for Feb. MSN bot is doin pretty good though, 2500 hits from them so far for feb. Is anyone seeing the same thing as I am?
I agree to that, Google visits and rankings decreased for my sites, but MSN rankings and visits have risen a lot.
I was at school today and noticed there were 320 bots on my site, wonder if they're changing things? Dunno if they were Google, MSN, yahoo, or whatever though
Same thing happens to me. And MSN's spider has a nice nest on my site and visits it every single day...
We are seeing the same thing. MSN have been crawling alot for the past month. I think they are increasing the size of their index for the new MSN search launch. Fast crawling + fast indexing. MSN is taking some major steps to catchup to Google. Google is not sitting still either. In response to MSN crawling activities, Google have increased theirs also. We are seeing this trend accross all domains. I think SE battle is on!
Slurp and MSN are showing a lot of activity in all my websites, Slurp is going crazy really... As for Googlebot, I ain't seeing "him" that much latelly...
Slurp is going ape tonight or what? We are seeing lots of activities. We have been getting 20-30 slurp hits per seconds for couple of hours now. Looks like Yahoo just put on their battle gear. Is anyone else seeing all these crawling activities by the top 3 SE? Don't know how much the servers can take all the pounding by these spiders.
I'm seeing the same decrease in Google activity, especially on my forum, though it's not affecting SERPs. In the past, Google used different bots for SE spidering and AdSense spidering. While Googlebot seems to be down, I have AdSense on my forum and MediaPartners Googlebot visits are up, along with MSNbot and Slurp. Is it possible the old AdSense bots are now doing double duty?
I just finished checking my site stats and Google did the deepest crawl yet - over 10k pages yesterday. Up until yesterday the bot would come by 30 or so times a day but only do a few pages per visit (average of around 350 pages per day). It must have liked what it saw because G search referrals are up over 150%! Rock on G! No change in Yahoo - Slurp activity has stayed the same - about 400 pages a day. The MSN bot must be taking a break - only 1,000 pages - normally it goes through 3-4k. -jay
how soon after google spidering your pages do you see them in the index ? I mean how much time does google take to process the spidered pages and add them to the index. Has it happened to anyof you guys that google spidered a lot of pages but didnt really add all of them to index.
spidering doesn't mean indexing, I think it's usual that a lot of the spidered pages aren't indexed in the top 100 results.
What do you mean by "indexed in top 100 results". I am not talking about SERPS. I am just talking about google indexing my page. I should be able to see it in the site: query.
# Hits Files KBytes Visits Hostname 1 23200 23174 1144040 36 crawl-66-249-65-207.googlebot.com 2 7388 7370 227129 1 crawl-66-249-65-101.googlebot.com 4 7967 7826 169448 108 msnbot.msn.com 6 3656 3625 102410 19 crawl-66-249-66-144.googlebot.com 8 1273 1232 53521 12 crawl25-public.alexa.com # Hits Referrer 12 3348 http://search.msn.com/results.aspx 16 2110 http://www.google.com/search The big G is doing deeper but fewer crawls but msn is indexing more pages and sending in nearly 2/3 more hits. At least the gbot visits are somewhat back to normal.
Google Has been crawling one of my sites very actively. I wonder if this is a sign that it will be de-sandboxed soon.
Googlebot (Google) (payes8170) (bw194.17 MB) 10 Feb 2005 - 11:13 ok googlebot has moved in in my forums, i think he wants bed and breakfast, he never leaves, and sometimes i have 37 of him in there at once. now in 4 days he has used 180mb of BW. not bad for a site thats only 21mb in size, 7000+ pages for a site with less than 300 pages,
It can be a couple of days before they appear in the index. They are definitely not added to the "visible" index immediately. But, if you try the site: command a day or two after a crawl, you should be able to see the pages there.