hey guys! Are there any cases with anyone where google has crawled your website only partially and not all pages ( An instance where spide visits your site). If yes, please share your experiences and what you have learnt from it.
It actually depends on whether you have submitted your site to Google or not. If submitted your site's all the pages will be crawled by Google but if you haven't submitted the site then you will not get all the pages of your site indexed.
I have found Google indexes all your pages and links from them unless you specify no follow. I find it is best to use a good sitemap and then submit it to your Google or Bing Webmaster Account. Also, check your robots.txt file making sure it allows crawling on those pages that are not showing up and check the frequency is appropriate to the content (if the content is updated on a regular basis or not).
Do you intend to say that if we do not submit a site to google, there are chances that only few pages get indexed? and rest don't? And if we do submit all of them get indexed? Have you had any such experiences? If yes, do share it here.
Yes, Google have 2-level bot as from my exp. Fast bot to crawl what and how many page in this website and deep bot to crawl deep in each page.
Yes, there is a possibility of partial crawling... If you have poor internal linking structure or broken links, crawler may get lost and leaves your website. So, you should take care of all these aspects.
So internal linking is a factor to get all pages crawled? Can i expect something more and price answer? And if such case should occur, what should be next step to get the rest pages crawled?