Hi All, Can any one tell me the difference between crawling and indexing??? And the code for stopping crawling for particular page?? Should i place robot.txt or "noindex,nofollow" in meta ???
Buddy,,,, Robots.txt is used for blocking sites or inner pages so that googlbot can't index the pages that u don't want to show up in search index ex: images, private, cgi-bin etc,,,,, Robots.txt is the best one example as show in below For a site : www.xyz.com/robots.txt User-agent: * Disallow: /cgi-bin Disallow: /cp Disallow: /downloads Disallow: /images1 Disallow: /widgets By applying this rules u can disallow a folder or directory which u don't want to crawl as mentioned above. Thanx Google Lover
Crawling is what bots to do to get all the info from your site. indexing is that info displayed in search results. I would use the robots text to restrict bots from searching certain files or directories.
Crawling is the process of reading your website pages by search engine bots. Indexing is the process of listing your site pages ranked for different keywords.
Crawling: The act of a spider following links and gathering the data from the pages it visits. Indexing: The process whereby a Search Engine adds web pages to its database. See the link below: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=35769 Read the last point in Technical guidelines. Also, for the robots.txt: http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html http://www.google.com/support/webmasters/bin/answer.py?answer=40362
Hiiii From here you'll get more information about crawling and indexing http://forums.digitalpoint.com/showthread.php?t=912204 Sumesh
Crawling is the process of reading the data on the page, indexing is putting this data (and other meta data) in the search engines database (index).