What this txt file contain?????????Why it's necessary for SEO does anyone can help to get the actual answer.????????????
robots.txt is a file which tells robots ( search engine crawlers ) which pages they should visit and which they should not.....this file we put in root of our site & can easily be accessible by visiting www.yourdomain.com/robots.txt
A robots.txt file restricts access to your site by search engine robots that crawl the web. You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one).Google bot won't crawl or index the content of pages blocked by robots.txt.
Robot.txt is text file where we can inform to search engine crawlers that we want to crawl entire website and some time if we don't want to crawl any page then we can make it none crawl by search engines. In short we can use rebot.txt as command file for search engines.
Search engines use to crawl your site fully and when you do not want any of your page dis allow them, you can add that to it.
thanks for telling the robot.txt in such a good manner...i was always having a wrong perception about the robot.txt...which is very much clear after reading this thread...
for blocking google spider our website site pages and images which we do not want index...and want more details about of robots.txt then search in google....
robots.txt is a text file which is specially used for Google spider.Google spider crawl your site for little bit sec.so within this period is not possible to crawl all the pages of site.by using robots.txt file we disallow the some pages where there is no value.you can use http://www.yoursite.com/robots.txt.you can also get more information from this http://www.robotstxt.org/wc/robots.html
The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web ... A text file present in the root directory of a site which is used to control which pages are indexed by a robot. Only robots which comply with the Robots Exclusion Standard will follow the instructions contained in this file. A text file located in the root directory to restrict search engine spider's access to certain files or folders of a web site. a file used to exclude some or all robots from crawling some or all the files or directories on a website. ... Robots.txt is a file which spiders read to determine which parts of a website they may visit and may not visit
While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project can appear in Google search results. In order to use a robots.txt file, you'll need to have access to the root of your domain. If you don't have access to the root of a domain, you can restrict access using the robots meta tag.
Sometimes, a content from your website can be copied to any blog submission pages. you can able to know that by checking. So you can disallow the duplicate copy of your content using robots.txt Then, there is no need to visit your cached pages to be visited by search engine bots. You can also disallow those pages using robots.txt