What is robot.txt files.

I want to know more about robot.txt files............
Plz tell me about robot.txt files.........

 

you can check on www.robotstxt.org

 

It basically is a set of instructions for search engine bots, spiders

 

It is used to call the search engines to crawl the website.

 

robots.txt file is a special text file that is located in web servers root directory.It defines the search engine spiders what to follow and what to not.

 

in robot.txt file you tell the crawler which pages to crawl or which not to crawl

 

robot.txt is a text file, in this it is defined that spider will follow this and won't follow that.

 

it's important value for search engines. it determines the crawling behavior and authority which directory or folder can search engine have authority or not. Webmaster can give authority or denied search engines to access specific folders or pages. it's very important to specific optimization over internet.

 

robots.txt file for a website tells the crawler about which page is to be crawled and which not to be crawled

 

used to include some pages in serach and others not

 

Even u can check the example of my Robots.txt file

 

Sometimes, a content from your website can be copied to any blog submission pages. you can able to know that by checking. So you can disallow the duplicate copy of your content using robots.txt Then, there is no need to visit your cached pages to be visited by search engine bots. You can also disallow those pages using robots.txt

 

robots.txt file is a text file that search engine bots check if they are allowed to index the sites or not. Google, Yahoo, Bing complies with robots.txt file.

 

yes i'm fully agree with you.