You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one).
Robots.txt file contains restrictions for Web Spiders, telling them where they have permission to search. A Robots.txt is like defining rules for search engine spiders (robots) what to follow and what not to. It is helpful to keep out unwanted search engine spiders like email retrievers, image strippers, etc.
The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map of what they can, and cannot index. The file must reside in the root directory of your web. The URL path (web address) of your robots.txt file should look like this... www. your site name .com/robots.txt
Hello Meenka , Thanks for sharing your knowledge about Robots. It's very useful for me & other users, but you wrote less about it. Can you please explain in details Thank you, Dewang Jani
Robot.txt files is a sort of tag used to place in particular site for not indexing it.In simple words its the substitue of "No follow"
Hi, Robot.txt is working as commander for Search engine spiders, we can allow to search engine spiders through Robot.txt which page we want to crawl or which page we don't want to crawl. hope you have gotten what you want.
Follow this link given below http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
My site is showing crawl errors and a20 urls are shown here. Restricted by robots.txt ‎(120)‎. Please guide me that how this can be fixed and if not fixed what is the harm...
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter†on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
robots.txt is a file, resident on web-sites. It contains a list of filenames that should not be searched by search engines such as Google. You should use robots.txt to indicate pages that are not worth indexing - such as pages with a lot of variable data.