Under what circumstances would you look to exclude pages from search engines using robots.txt vs meta robots tag?
If you use meta tags Google will just crawl your site but will not index your site. But if you use robot.txt Google not even crawl. Google suggest robot.txt.
Robot Tag is the allows the Robots to crawl & index (if you use - content="index, follow" otherwise not). By the robot.txt file you can instruct the robots where to crawl & where not to crawl. You can also restrict the robots as per your preferred search engine. The syntax of the XML sitemap is also useful on the Robot.txt file.
well my site's personal or not so interesting pages is what I want to be not followed. I will add no follow to the pages like privacy policy, user agreement etc.
Robot tag we use for particular page like if we use on contact us page it will not get index in google... but let suppose that if you dont want to index your search page or search results from your site then you need to use Robot.txt file and disallow search there...