I am curious as to how often my page gets crawled. The page is a PR 3 as are most of its subpages disreguarding pages that I recently addes within the last month, approximately three. I know from reading this forum that a better PR usually means ones site gets crawled more often by google. I have dont a lot of optimization techniques lately (still not much but better than it was for sure), and was curious if anyone can gve me a ball park answer as to how often google might crawl my site. About two weeks ago I was getting 40-60 uniqes a day but I recently dropped to 10-15 tops, it hurts and its hard for me to keep working on the site after a huge drop like that (huge for my site). I'm not sure if thats relevant but I figured I'd post just in case that may affect my PR. Thanks for taking the time to read my post.
This is one area where PR actually tends to have some significance. There are no set rules as to how often a site will be crawled, but a higher PR indicates that the site is linked to be either high pr sites (their high PR reflecting that they have many links to them), many lower pr sites, or a combination of the two. In all of the above situations, the frequency at which a site is crawled will increase with PR. Conclusion > higher PR results in an increased number of visits from spiders.
Is there a reason that you are concerned about frequency of crawling? Frequent crawling doesn't necessarily lead to higher ranking, but it does help the SE to have the most up to date info about your pages if they change often.
From my experience higher PR sites also tend to have new content indexed more quickly, and the site is generally included more thoroughly altogether.
Yes... absolutely.. kkibak and i would like to add that the more the search engines crawl your page the better chance you have to give them fresh content and by that way you would be loved by google and rank higher...
Also if google find your content changes between the crawls they will increase the frequency of the crawls. If they find the content hasn't changed they will decrease the frequency.
Actually, more IBLs get you crawled more often. That's the real cause, not the PR. PR just happens to also be a result of the IBLs. The more links pointing to you the more your URLs will be added to the crawler's queue. The more spots you have in the queue the more you will be crawled. I believe that you will get crawled more often if you have tons of low PR links (and hence your own PR is low-ish) compared to if you have you have just one really high PR link (and hence your own PR being higher). Though you could argue that if you get a high PR link from say the Google homepage, you indirectly benefit in terms of crawling from the thousands of links that high PR page gets. Though judging by the way most crawlers work, it doesn't really work like that.
That is completely untrue.. Spiders do not visit your site everytime an IBL is found. They simply record that there is a back link and move on. If you have a high page rank and frequent new content then you will get spidered more often than a site with a low page rank no matter how many links are pointing to it, fact!
"Is there a reason that you are concerned about frequency of crawling? Frequent crawling doesn't necessarily lead to higher ranking, but it does help the SE to have the most up to date info about your pages if they change often." I was under the impression that if I did some onsite optimization my site wold have to be crawled before I reaped any benefits from it so, I thought if I was crawled after doing a lot of optimization stuff, then crawled, I would most likely have a higher ranking, not by PR necesarilly but concerning not being on the 30th page of google but maybe the 25th.
That's what I meant. They crawl a document and add any links they find to the crawling queue. Which will be processed later, subject to sorting algorithms probably. What I'm saying is that if many documents link to you, your URL will be in the queue more than URLs that don't have many IBLs. It depends on the sorting whether or not this means you will be ahead in the queue or not but I don't think high PR beats quantity of links. Doh, you're comparing a high PR page + fresh content (2 things and remember high PR is often due to many links as oppose to few high PR links) on one side to just low PR on the other side. Fair comparison! And if you state something as fact, you might as well show some proof. I just believe that a new page, say on an earth shattering news topic, which gathers millions of low PR links (from other news sources = new pages = no PR yet) overnight will get spidered quicker and more often than an older page with high PR and just thousands of links.
Agree with Tops here. Also if you have natural backlinks I.E. deep links to content within your site rather than everything to the front page YUK! you are more likely to be spidered. The spider resource algorithm wants to make best use of its spiders. Because of this they move the time of spidering of a site where content rarely changes. Sites where content changes constantly will be spidered constantly. My riugby forum for example gets spidered more or less constatntly, as do other busy forums. My own web site gets spidered now and again though.
well we are talking about frequency of crawling here not how quick its takes to get a page spidered. obviously a page with thousands of backlinks is statiscally gonna get discovered quicker than a page with a few hundred high PR backlinks. I can go out a spam a few thousand links over night and sure enough my page will get spidered very quickly but in my experience its not gonna get re-visted as frequently as a page with a relatively low number of links and yet a high page rank which is updated at the same frequency. At the end of the day there are too many different metrics to provide a concrete example so we go with experience and in my experience the later case is true.
actually amnezia, this shouldn't be the case. i don't believe that thousands of low pr backlinks will (statistically) make you more likely to get your page discovered than a few high pr links that results in the same overall pr level. if you think about the logic, the high pr sites have high pr because they have lots of low (or high or medium) pr links to them. this results in them being crawled constantly, which means their links out (e.g. a high pr link to your site) are also followed more frequently. thousands of low pr links might take even longer (or IMO just as long) to get followed.
This is one area where PR actually tends to have some significance. There are no set rules as to how often a site will be crawled, but a higher PR indicates that the site is linked to be either high pr sites (their high PR reflecting that they have many links to them), many lower pr sites, or a combination of the two. In all of the above situations, the frequency at which a site is crawled will increase with PR. Conclusion > higher PR is likely to be representative of in an increased number of visits from spiders.
PR is not everything, there are also different ways to get scores rather than getting only nice PR as ultimately your goal is to bring traffic and high conversion.
KKibak, why have you repeated your first post? I see no point in doing this in the same thread I disagree with you when you say the frequency will increase with the pr, as this is not the case per se. Certainly the DEPTH of crawl will increase with a higher pr, but the spidering tends to be based more on the frequency pages are updated. and the number and atrget page of links, rather than pure PR value.