HI I have seen a website (zsl.com), it contains around thousands of pages , i know once crawler landed to any website , it completely crawls each and every page at a single day, right ? but i can see different crawl date for many page on the particular website . how does it happens to this site ?? can some one explain this
I don't think crawlers necessarily crawls every link on a given website at the same day. Even if they are, there could be tons of reasons for this. example for a few of them: - not all the pages are from the same date so when the site was crawled, some of the pages didn't exist yet. - maybe some of the pages has duplicated content so ignore them. - the more traffic a page has, the more frequent crawls it would get
Google doesn't crawl static pages as much as dynamic pages. For example a locked thread here on DP will get crawled, but since there won't be any new replies even after weeks Google won't crawl it daily/weekly/monthly. The famous 'proxy discussion thread' does get new replies almost daily, so Google will crawl that page much more. Hopefully that's an answer to your question.