How often do the spiders and bots visit your website(s)?

First off, thanks for looking

The company I work for has five websites that make up our service offering. Our primary sales site (a PR6) is typically crawled at least once per week. Our other sites, a PR6, PR5, and PR3 aren't crawled quite as often, yet our PR4 blog seems to get crawled as often as our primary sales site. This gets me to thinking about the correlation between PageRank and cache frequency. I've heard people say that PageRank has nothing to do with cache frequency, but I've also heard others say that it does.

What say you? Also, how often are your sites cached?

 

Hi CJan_NH,

PageRank is believed to be one of the factor the bots consider when re-visiting your site. The most important (and the primary) factor is the Content Freshness. The most often you update your content, the more often the bots will visit your site. That is the reason why blogs are indexed frequently.

 

just constantly run rss/xml automatic updating feeds on your site, they change regularly so this boosts spiders noticing you. So updating your site is automatic and every day, increase spider site.

 

I agree with moneyspeaks. However, if your site is indexed and has no fresh content it will make no difference if it is crawled or not. Change at least one paragraph of your index page weekly if you want regular indexing.

 

I must agree. Our sites get new content a few times/week and generally get visited by the Big 3 daily and the others (Ask/Tehoma,Findlinks,metacrawler etc...) avery few days

There is also 'deep crawling'. It is when the spiders occasionally dig deeper into you site to look for new/updated content.

The whole PR thing is another issue alltogether. It has nothing to do with crawl frequency.

My 2 cents for ya

Dave

 

Thank you for the input everyone

Ever get the feeling that just when you feel like you're getting a handle on something another curve ball comes your way?

 

Oh I hear ya! I have to try and deal with SEs (search engines) every day. It never ends. google threw the biggest curve ball this year with a new algorithm over haul called Big Daddy..

Tons o fun... so yes... keep reading and learning... (and asking questions, lol)

Dave

 

In my company a lot of us turned blue because of Big Daddy, but fortunately we got through it okay-aside from some dropped cached pages and a temporary PageRank drop on one of our sites. I'm aware of the green fairy dust argument with PageRank, but when one of your sites goes from a PR6 to a PR0, and then back to a PR6 within 24 hours it's a great way to have a stroke, right at your desk

I kicked around the idea of billing google for all of the Maalox we drank during the update

 

BD was/is fun huh?

Good story. I started writing about BD on my site. For the last few months I get a bunch of traffic for a variety of DB terms. So Big Daddy did help some. I got burned on some percieved 'dupe content' tohugh. Had to fix some things all over the place. Clients istes and mine. Drag.
Should bill them with each update.

Were you around for hurricane 'Florida' update?

 

If it happened prior to last spring it was before my time. Please explain-it sounds like another "priceless Maalox moment"

 

Bots are always on my site, just depends on which one. There is the rare occasion that they aren't on my site, but as I said, rare.