Hey Guys My new site www.tubefetch.com is only 3 weeks old, and i have noticed that the site is being crawled huge amounts...especially by google bot. In the first 3 weeks we had 26 different robots crawl the site, and google bot ALONE used over 12GB of bandwidth. Today (first day of the month) we have had 6 different crawlers including google bot, adsense, MSN and Yahoo and Google Bot has used 1.1GB of bandwidth (today!). The re-visit meta tag is set to 20 days, but the google, msn and yahoo bot visited both today and yesterday. Does anybody know why this is? It just doesn't seem to make any sense to me (not that i am complaining!). Thanks Josh
You could set your .HTACCESS to keep google from caching all your images. I would set up some web exclusions as well.
Thanks for the replies.....I dont know if google is indexing the videos, but even if it did, there would not be that amount of videos. Also there is nowhere near enough images on the site to cause that amount of usage.
You can find a robot exclusion .htaccess that will prevent offline browsers and non-name / malicious robots from accessing your page. Who gives a damn if johnnysearchbucket.org's home-made search robot caches your page? Just worry about Google, Yahoo, and MSN, mainly.
I have had that happen in the past. A site with less than 25 megs of total content (all static pages) suddenly started using multiple gigs of bandwidth to googlebot. It lasted for a day or two, stopped for a couple of days, did it again and then stopped. Fortunately it stopped before my site got shut down for bandwidth overages I could never tell what caused it and saw no corresponding jumps in SERP's as a result of it.
I'll have to see how it goes...but yes i am only really worries about the main search engines, of which all are visiting daily
I looked at your site and I can't say just what is causing this. I had a similar thing happen with one of my sites last summer. Google was chewing up over 1 gig of bandwidth in less than a weeks time. It slowed down after about a month and finally dropped back down to normal. I never did really figure out why it was happening. The site is question was straight html with no scripts at all.
Maybe are malicious bots using Google's "user agent". You should check their IP range (66.249.64.0 - 66.249.95.255) I'm verifying this on my site with this regular expression: /^66\.249\.(6[4-9]|[7-8][0-9]|9[0-5])\./ Sometimes Google is using others IP and others "user agents" to ensure you're not fooling Googlebot. You can be banned from Google index if you provide different content for Googlebot and for others.