What a strange problem...too many spiders. One suggestion off the top. Can you send some pages to directories which you then tell the spiders not to go to? I suspect that much of your SE traction comes from home page and then one directory level down; beyond that there is little point in having the spiders crawl. Second, how many of these things do you have? Are you auto generating content...you must be. I ask out of sheer amazement. Third, try a different stats package which can give you dig down on referrers. Fourth, if you need more detailed analysis I'd be happy to help if I can. PM me.
Not sure what you mean by this. You want/need for spiders to crawl your entire site - if they don't crawl then you don't get any SERPs for those deeper pages. [quote Second, how many of these things do you have? Are you auto generating content...you must be. I ask out of sheer amazement. [/quote] 'auto generating content' has a bad ring to it - like scraping or random gibberish. Obviously there are ways of generating massive amounts of content without scraping - using wikipedia, dmoz etc etc. What I'd love to know is how you get the spiders to hit a new site so hard.
What a strange problem...too many spiders. Yes... it is an interesting challenge sometimes. One suggestion off the top. Can you send some pages to directories which you then tell the spiders not to go to? I suspect that much of your SE traction comes from home page and then one directory level down; beyond that there is little point in having the spiders crawl. I am not convinced that it is all home page and one directory down. I recently installed asRep across all of the directories to find out if they are coming in "deeper into the site" than that. I think they are, but will not know until I get some more data for certain. Second, how many of these things do you have? I have quite a few. Are you auto generating content...you must be. I ask out of sheer amazement. No, I am actually paying people to write content. I am "buying" about 50 "articles" per day. Third, try a different stats package which can give you dig down on referrers. I am doing that with asRep. Fourth, if you need more detailed analysis I'd be happy to help if I can. PM me. Thanks. I may take you up on that offer. Sincerely, James
I do not "auto generate" content (see my last post above). All I "do" to start getting indexed is to link it to other sites (like this) that I already have... some seems to index well right away. Some not so well at all. Some get hit hard a month or two or three later... it is really odd! That is why I asked about domain names and extensions above. Sincerely, James
Hey, Compared to my experiences it seems odd that you are having a problem server loads. I run a p4/3ghz with 1gig ram - I have about 50 sites and I know at least two of them do about 75k pages per month with the others around 5k-25k -- they all run databases for content - and my load averages barely get above zero (and that's with a perl spam filter running as well!) Two suggestions. 1. If you are using open source stuff such as OScommerce you should be aware they are not always well written from the DB access point of view. I mention OSC as I read it has some 150+ database access per page load! 2. Use caching, via PHP, perl or however you do it. You'll get around 100x increase in speed / decrease in load. 3. read up on robots.txt - you can actually change some spiders page reading frequency with the right config. Oops - that was three. And if you're not making much money with adsense yet, you should be once all the spidered pages show up in the SEs.
I think the load issue partly comes that the ONE SITE above (and I have multiple ones running on one server) are averaging over 50K pages per day (I did not actually average it but just did a quick estimate based on the stats posted above). Your site is doing about 75K per month (about 2K to 3K per day). If you have 50 sites at 3K per day that is 150K per day in traffic. I would get 150K per day with just two sites (and I have more than that on a single box). I think that is why I have a load issue. Sincerely, James