How much load can search engine bots put on a server? Past two days I've had a high and constant load on my main server, around 13-16. Couldn't think of what it could be until I saw that Yahoo has around 150 bots reindexing my forum. Would that be a likely cause for the high load or does search engine bots not produce much load at all?
Well the forum is old and has over 1.7 million posts....so if there is a lot of reindexing going on I guess there can be an awful lot of bots at the same time.
Thanks...added a delay. Unfortunatly the load just keeps going up with about 1 per hour though, sitting at 20 now.
If yahoo bot was the culprit for the load, let it reindex the robots.txt...sometimes robots take more than 24hours to honor the new robots.txt.
It actually turned out not to be all those bots, even though they probably added a bit of load, but the real cause was stuck cron jobs. Started to look into strange possibilites because the load was up around 25 but it was not affecting the server response time at all, and it turned out to be stuck/bugged cron jobs that were not given any CPU time so they just added to the load number without actually using any server resources.
Hey I think bots are nothing but visitors. So they should not put any load on the server lol But yes if you have too many sites and if many of them are accessed simultaneously then it may put a load