I know that google uses something called a crawler which is in fact a method of getting random webpages and indexing it. I still can't get that how it works. Does it visit random new links that it comes across and then goes on to index it, or is there another factor which makes it move that way. How long will it take for the crawler to finally get to one webpage after it was created. It must be hard work, finding and indexing millions of pages that are created every day, and un-indexing the pages that no longer exist.
"Crawler" is a generic term for any program (such as a robot or spider) used to automatically discover and scan websites by following links from one webpage to another. Google's main crawler is called Googlebot. This table lists information about the common Google crawlers you may see in your referrer logs, and how they should be specified in robots.txt, the robots meta tags, and the X-Robots-Tag HTTP directives.
Crawler is a program which crawls the web non-stop. It keeps on adding new links in its link graph, which get crawled later on. Search engines don't have just one program to deal with huge web... there are million of programs running out there to deal with newly created webpages, existing pages and dead pages.
To make sure that the creepy crawler visits your new page as quickly as possible, make sure you're updating your site regularly, rather than just once in a while. This will encourage Googlebot to stop by more often, as it will realise that you often has useful new content for it to consume and deliver to searchers. You could also try resubmitting your updated XML sitemap in Webmaster tools, which could lead to Googlebot stopping by.
Crawler is a computer software which have instruction to crawl each & every webpage around the globe...
Crawler is a computer software which have instruction to crawl each & every webpage around the globe... For example: "Robots can only do that thing for which he has the instruction" A robot is a mechanical or virtual intelligent agent that can perform tasks automatically or with guidance, typically by remote control. In practice a robot is usually an electro-mechanical machine that is guided by computer and electronic programming. Crawler is same as robot that why we often called googlerobots....
Crawler is a generic term for any program (such as a robot or spider) used to automatically discover and scan websites by following links from one webpage to another.