Why is Google crawling so much?

Google and yahoo are crawling the crap out of my new blog, and driving up bandwidth.
Googlebot 21331+98 1.04 GB

Is this normal? I have to set aside 2g worth of bandwidth just for google?

Anybody know if this is normal?
What are you guys getting crawled at?

 

you postd the same thing about yahoo..

you should look into sitemaps? they're not crawling to index.. they're crawling to find what to index...

if you're already using sitemaps just give up on the internet.. and remove disallow robots ;P


I get crawled pretty much all day by google... doesnt matter though..

 

I thought I had sitemaps activated.
It's only these 2 ,(Google and Yahoo, especially yahoo) the other bots seem to be normal.
Can you elaborate a little.
It shows that Google has over 4,000 pages indexed, what are they still looking for?

 

It is normal if you always update your contents. Mine also being crawled by Google whenever I updated my blog's new contents...Don't worry about it... Be proud that your site is always visited by Google...

 

But the bandwidth! Three different urls from yahoo are hitting me for 2g's this month and another 1.5g's from Google. I know that is not normal..is it?

 

I guess that is normal, specially if you have lot of backlink from reputable sites. If bots have lot of way to go to your site they will visit it frequently. 2GB BW is not that high, I have a entertainment site getting more than 10GB BW from google and I love it because everytime I update my site google index it immediately and sooner or later start getting traffic from that new page..

Like other says be proud if google bot loves your site....

 

2GB seems high.

mine are - Googlebot 9311+41 35.71 MB

9311 x 3 = 27933 (that would work out more hits than yours but would only work out to be 100mb bandwidth)

is google trying to download files from your site (.zips, .mp3s etc?)

 

If you have 4,000 pages in their index, maybe. I wish they crawled my site half as often!

 

it's normal.

 

you can set duration for spider throught robot tage. how many day after it will visit ur site or u can manually ping to google and yahoo when u update ur site if u are not updating ur site soo much frequently.

 

Thanks guys. I am only going on the word of one of my associates that is telling me that this abnormal activity.
I though the same way you did, that it is good, especially with the amount of content that I am adding, and having the pages translated with a global translator.

None of that spider activity counts as stats, ie: page views correct?
Just hits, right ? Or not?
It is still very new, and the page views compared to visits, are high.
This is my first blog of this size, so it all looks strange to me.
Thanks for the help.

 

I wouldnt worry about it. 2Gb isnt a lot of bandwidth and only costs a couple of dollars a month at most.

 

Yeah, I think I may be getting some less than expert opinions from my associate..
Thanks

 

I would not complain if I were you. It is a good sign to come

 

I don't have anything to download, most of the links go to an affiliates site for downloads, (Music)

 

i wouldnt worry then, probably just down to the amount of pages u have and how big each page is etc.

how does yahoo/msn etc compare to google? do they use as much (in equivalent)

 

Yahoo is sending bots from 3 different URLs at about 2.5 G's this month.

216.109.121.44 l 12663 l 12663 l 437.80 MB
216.109.121.41 l 12652 l 12653 l 443.23 MB
216.109.121.42 l 12474 l 12474 l 436.16 MB

MSN, not so much and only 20 pages indexed [545+99]

 

Google, I guess has got lots of dirt cheap bandwidth. Google bot are crawling and using more bandwidth then used by other methods through search engines...

 

Yeah, after I tracked down the Yahoo IPs, now it seems that all is well.
Those 3 Yahoo addresses that were eating so much bandwidth?
After further investigation, it seems that they are each 1 individual IP address, but they are the main addresses for Yahoo ISP's all over the world, like SBC.
So each one is responsible for hundreds of Yahoo DSL IP's.

I am no longer freaking out, and it seems that I have traffic
I never handle server admin, so this is all new to me.
I have a list if anyones interested.
Thanks for all the help guys.