It provides restriction to search engine robots. If you want to keep some files or folder private then you can write command and robots will never visit the file or folder : disallow : /images/ (restrict robots not to visit this folder) disallow : /photos/mypicture.jpg (restrict robots not to visit this file in your web server)
Robot.txt is the file which informs Spider that which pages it should not crawl. By default spider crawls all pages, so you just have to mention those pages which you want to restrict to crawl by spider. Example: Disallow: /tutorials/meta/ Disallow: /tutorials/images/ Disallow: /tutorials/assets/ Disallow: /tutorials/404redirect/ After mentioning all these you have to upload this file at the root level of your site.
If you want to find easy and best solution about robot.txt than must go in www.google.com/webmaster A complete solution for your needs is present here...
Hello, This simple little file can get you listed or keep you from ever being listed. Robot.txt is a text file which contains instruction for search engine to act on.
It prevents search engines from accessing some parts of your site. A quick search on DP or Google can get you detailed explanation on robot.txt
Hi, you can read all about for robot.txt see in more given links www.robotstxt.org google.com/robots.txt Thanks!
every search engine has different indications in robot.txt files, ex: User-agent: Googlebot Disallow: /admin Disallow: /cgi-bin Disallow: /func Disallow: /inc User-agent: msnbot Disallow: /admin Disallow: /cgi-bin Disallow: /func
Hello, In DP there is a sub-forum for robots.txt Below is the url : http://forums.digitalpoint.com/forumdisplay.php?f=50