I wrote a post about this the other day. Hopefully it will give you the basic information you requested. Robot.txt
The robots.txt file tells crawlers which files and directories should or should not be visited. Should specific areas of the site be protected from crawling, the folder or file location should appear after the ‘Disallow:’ instruction. For example: 1. To exclude all robots from a server: User-agent: * Disallow: / 2. To allow all robots complete access: User-agent: * Disallow: 3. To exclude all robots from parts of a server: User-agent: * Disallow: /cgi-bin/ 4. To exclude a single robot: User-agent: BadBot Disallow: / The robots.txt file can be created in Notepad. It needs to be saved to the root directory of your site-that is the directory where your home page or index page is located. It can be viewed by visiting your URL http://www.yourdomain.com/robots.txt That's it. This will allow all robots to index all your pages. It's free! Cheers