we should use robot.txt if we want to restrict crawlers to crawl any private pages. You can do it using notepad file and upload it in your server. Thanks
robots.txt tells google, yahoo, bing crawlers what not to crawl ( take an inventory of ) on your pages. Robots.txt works in conjunction with your sitemap.xml file. The sitemap.xml tells the crawlers what pages you have and where they are located. If you do not have a robots.txt file, the crawler crawls all your site. If your robots.txt file has page A as a restriction and you declare page A on your sitemap.xml, you get a crawl error on your Google Webmaster Tools panel. An example of how we use it is the following. We have a test directory on our server where we upload sites to be tested before we go public. Obviously, we do not want the crawlers to take note ( inventory ) of this directory since it is a directory for testing only. Below is the robots.txt file contents. # www.robotstxt.org/ User-agent: * Disallow:/testing/ Good luck AndresNeil. Braulio
robot.txt is one kind of things that will help you on crawling in search engine. To add a robot.txt for your web just do the following thing : Google Webmaster link log in and add your site .Then Click on submit then click on 3 number option and you are done . See the picture :