Here you can get all the informations about crawling the website.Let us see how search engines work. In simple terms, we can say, there are three pieces of software which make the search engine. They are Spider software, Index software and the query software. Spider Software It crawls the web looking for new pages to collect and add to the search engine indices. We say so. In reality, it requests pages a from a web site in the same way as Microsoft explorer, or firefox or whatever browser you use to request pages to display on your screen . The difference is the browser collects image and formatting whereas the spider collects only the text, links and URLs from which they come. Links attracts spider more because it leads to other web pages that has the things like Text, links, and URLs. Index Software It can catch everything that the spider throws at it. The index makes the sense of the mass of text, links and URLs using what is called an algorithm – a complex mathematical formula that indexes the words, the pairs of words and so on. These algorithms analyses the pages and links for word combinations and assigns score that allows the search engine to judge how important the page might be to the person who is searching. And of course, it stores all the information and makes it available. The Query software This is the front end and what you see when you go to a search engine. The main feature of the query software is the box in which people type their search terms. The visitors can type in their words, and the search engine will help to match the words by searching the web. We say so. But in reality the query software does not search. It just checks the records that have been created by its own index software. And those records have been made possible by the raw materials the spider software collects.
my question is How Internet Search Engines Work from starting of search engines to now or history? or serach engine alralgorithm
Find out from a search engine http://www.searchenginehistory.com/ http://en.wikipedia.org/wiki/Web_search_engine