Under what circumstances would we look to exclude pages from .............. .......................search engines using robots.txt vs meta robots tag?
Robots.txt is a permissions file that can be used to control which webpages of a website a search engine indexes. The file must be located in the root directory of the website for a search engine website-indexing program (spider) to reference.Robots.txt is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes. Meta tag manipulation is lost it’s importance in search engines.The most important Meta tags among all is the Description Meta tags. Because Google use this Meta tag information to index a page in categories and sometimes may consider meta tag when gives rank in SERP. Even if you don’t get a ranking boost for your Meta tag description but still you try to write an interesting as well as relevant description Meta tag. An interesting description increases the CTR (click through rate), because Goggle displays the Meta tag description of a page in SERP.
"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that. If your web host prohibits you from uploading "robots.txt" to the root directory, or you simply wish to restrict crawlers from a few select pages on your site, an alternative to "robots.txt" is to use the robots meta tag.
robots.txt is appropriate when you prefer to hide entire directories or specific files from spiders, however, the META robots noindex,nofollow technique is useful when a specific page should not be indexed and you would prefer not to list it in a place like robots.txt where a human visitor could easily determine its location. Refusing to divulge a page's location is hardly a way to "hide" the content (a recursive download of your site would include the content), however, I find it useful for one-off script-generated pages which may or may not return content dependent upon the script's action.
In robots.txt file you can mentioned all the pages which have errors and which you not want to index one by one. Where as in meta robots tag you have to put manually the tag on all the pages to which you want to keep away from indexing by Google.
robots.txt is appropriate when you prefer to hide entire directories or specific files from spiders, however, the META robots noindex,nofollow technique is useful when a specific page should not be indexed and you would prefer not to list it in a place like robots.txt where a human visitor could easily determine its location. Refusing to divulge a page's location is hardly a way to "hide" the content (a recursive download of your site would include the content), however, I find it useful for one-off script-generated pages which may or may not return content dependent upon the script's action.