So far google has used just over 5GB of bandwidth on 1 of my sites, yet it only has 103 pages indexed! And those 103 pages are just text!! Whats that all about?? I wouldnt mind it using so much bandwidth if it actually fully indexed the site! Has anyone experienced this before? Im hoping it means i will soon have many, many more pages indexed in G, what dya think?
From Google faq 3. Googlebot is crawling my site too fast. What can I do? Please contact us with the URL of your site and a detailed description of the problem. Please also include a portion of the weblog that shows Google accesses so we can track down the problem quickly.
Actually, I think unless it starts looking like my bandwidth limit will be neared because of it I will leave it as it is... I dont want to risk not being indexed properly!
I have the same problem. Below are my Webalizer stats for the last few days. Normal bandwidth use is about 50 megs per day. 3 5338 12.24% 4310 11.93% 1854 11.56% 667 12.02% 658 16.64% 50182 0.69% 4 5150 11.81% 4297 11.89% 1917 11.95% 612 11.03% 603 15.25% 1797426 24.67% 5 4111 9.43% 3441 9.53% 1618 10.09% 569 10.25% 547 13.83% 2879848 39.53% 6 4297 9.86% 3417 9.46% 1685 10.51% 552 9.95% 511 12.92% 1349960 18.53% 7 4365 10.01% 3773 10.44% 1612 10.05% 561 10.11% 537 13.58% 54485 0.75% 8 5785 13.27% 4891 13.54% 2070 12.91% 703 12.67% 650 16.44% 47360 0.65% 9 4369 10.02% 3847 10.65% 1726 10.76% 689 12.42% 639 16.16% 1024777 14.07% Here is the problem: 796 1.83% 557 1.54% 6898907 94.69% 119 2.14% 66.249.65.44 Here is the user agent info: 11916 27.33% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET I thought it was done after three days of it, but I see they were back again last night. It's not that they are crawling to fast. The site total uses less than 200 megs so this kind of bandwidth use is insane. Maybe this is the kind of thing that got WMW to shut off the bots? In my case it doesn't mean more pages will be indexed; they were already all indexed. After reading all the posts where people are praying for the Googlebot to visit I have been reluctant to complain
I have heard of some instances where the bots were stuck in a look on the site. Keep tabs and contat them if they dont move onto other pages. Good luck.
well google earlier 'claimed' to have indexed 33k pages but just displayed url's for most of them. For most of this month it has been down at around 100. I have other sites with more than 33k pages and gbot only uses 10's of mb's on them. Will just wait and hope it means a nice full index for me soon