My site has 42500 hits from googlebot these days A few days ago it had only some hundreds.. Why is that?
I believe it has started indexing each city from my base.. If not I cant explain it! When you search in England for cities starting with "a" you take these links
O God Much spamm from google. Just make your Robots.txt to dont allow robots to visit your site only 1 time per day. (2.2gb bandwidth lol)
The last time Googlebot started hammering my site it turned out to be from a hack where the pages were bloated but only for the search engines UA. Is your site new in the last year?
it is a few months old.. Googlebot 153978 5.45 GB 21 Apr 2010 - 01:48 Unknown robot (identified by 'robot') 972 27.49 MB 21 Apr 2010 - 01:08 Unknown robot (identified by 'spider') 88 2.11 MB 20 Apr 2010 - 08:39 Yahoo Slurp 81 1.36 MB 20 Apr 2010 - 16:06 MSNBot-media 81 389.56 KB 19 Apr 2010 - 13:04 Alexa (IA Archiver) 79 8.05 MB 20 Apr 2010 - 07:30 Unknown robot (identified by 'bot*') 65 1.58 MB 20 Apr 2010 - 23:15 MSNBot 49 780.87 KB 20 Apr 2010 - 23:35 Unknown robot (identified by empty user agent string) 47 1.43 MB 20 Apr 2010 - 05:07 Ask 15 1.96 MB 20 Apr 2010 - 12:12 Unknown robot (identified by 'crawl') 5 127.64 KB 20 Apr 2010 - 02:54 Unknown robot (identified by '*bot') 2 50.94 KB 19 Apr 2010 - 09:53 Netcraft 2 24.76 KB 09 Apr 2010 - 03:08
You can also use this in robots.txt, which should help: User-agent: * #put spider name in here, or leave it as wildcard Crawl-delay: 10
a) you have not got a robots.txt so it's looking at every possible thing you let it. b) a look at google shows what it's indexing (many with the same page title ) site:www.theweatherland.com
High Bandwidth Mate. Go to your webmaster tools, decrease the speed of the crawling. So that your bandwidth will be saved.
Hey, this 7.62 GB is tooooo much. There must be some issue on your server or you have selected high crawl rate by setting it up within Webmasters. Check it out, otherwise you'd be overlapping your monthly bandwidth at hosting.
Have you created any kind of coding loop which will get the Google robot stuck? Else I would suspect someone is using your bandwidth/some images on your site by identifying themselves as the Google Robot. Go in to webmaster.google.com and check your site there, see the crawl rate and the average number of pages crawled per day. If that number doesn't nearly add up towards the mentioned bandwidth, ask your host and see if they can help you out by identifying different IP numbers accessing the site.