They have both been on my site for about 2 days non stop (or atleast everytime i check theyre there. Not sure if this is due to my site just being uploaded but there cant be any more than about 500 pages (its a forum with 170 members and about 600 posts 200 topics) They shouldnt spend that long on there should they? As it happens theyve not used up anymore than about 30mb bandwidth each so its not the end of the world but if this goes onto the end of the month its gonna be a fair amount of bandwidth they use up. Is this normal? If not how do i stop it? Obviously do want them to visit ive just never had a site where a bot stays for 2 days - let aloe 2 of them.
You can slow down google by adding the site to webmaster tools - https://www.google.com/webmasters/tools/siteoverview . Once its in, go to the Crawl Rate page and change the rate from Normal to Slow. I don't think there's anything you do with Yahoo. Personally I wouldn't worry about it and would be happy. If they're not crawling then you're not getting pages in the index. -jay
google has finally goe after 3-4 days yahoo is still there, any suggestions? i dont mind them coming and going but 50mb of baswidth in 1 go is just rediculous. im hoping it was a 1 off due to me changing everything
Google bot came 215 times in 14 days to my site and I'm very happy about it. I don't know why are you woried? p.l.u.r.
ive never had a site where they have been on my site pretty much continuously for 4 days. Not only is there 1 but 2 of them. Its as though theyre stuck on my site so, im not complaining at the minute but if it carries on then im gona have a lot of bandwidth usage.
If you want to rank well you need to let them spider as much as they want. They won't be getting stuck on your site unless you have session id's and they are thinking you have thousands of pages. Don't restrict the spidering as you won't be able to get your site indexed as quickly. Lots of spidering is a very good sign google trusts your site.
That is most likely the case. In order to keep their index fresh... the more a site is updated, the more the spiders will crawl it. Once it sees that things are not changing as fast, it should slow down it's rate of return.
Frequent visits = better crawling = better indexing of pages! There is nothing to worry about unless you are using free webhosting with limited bandwith. For serious bigger websites it is a good sign.
im not on free hosting but my site is quite small, 100 reigstered users, bandwidth isnt too much but its about 1/10th if not more of my monthly limit, which is fine for the time being but if it gets more then im gonna need to increase it
In my clients case I had to do something because he had about two dozen spiders eating a lot of bandwith and his bandwith was limited.