What is the robots.txt

Hi there!my dear friends what is role of robots.txt file of website and where include which file and where code.

 

You put it in your root directory (usually the "www" folder) and it tells spiders what they can and cant crawl/index.

 

A file written and stored in the root directory of a website that restricts the Search Engine spiders from indexing certain pages of the website. ...

 

Read the robots.txt FAQ:
http://www.robotstxt.org/faq.html

 

Robots.txt is a regular text file present in the root directory of a site that is used to control spiders that visit your website.
This file permits the access to certain folders, file types, and specific files depending on the robot accessing the site.

For Example :
User-agent: *
Disallow: /admin/
This says that - all robots (indicated by "*") are instructed not to index any of the admin (indicated by "/admin/") directory and its descendents.

 

thanx all of for reply my qustions very nice ans plz send everything robots.txt file.i am not find txt file.

Thanx

 

It's exactly right.

your url would be for robots.txt

yoursite.com/robots.txt

 

http://www.seoresource.net/robots-metatags.htm

Read this link. You will get answer.