Mobile Phones - Recreation websites directory - Flights - Prison Break - Direct TV

PDA

View Full Version : Optimal commands in robots.txt


dshah
Nov 28th 2005, 8:46 pm
Could anyone please suggest what should be minimum commands present in robots.txt so that site is well crawled by search engines.

1. Having NO robots.txt is good or bad?
2. If its good to have rotobts.txt, what should be minimal standard commands in it (I dont need anything specific)

Thanks In Advance!

Wipqozn
Nov 29th 2005, 4:26 am
robots.txt is useful when you disallow access on the site for some of the robots ;-) However, You can even put an empty file on a server.

For more info visit http://www.robotstxt.org/wc/robots.html

minstrel
Dec 1st 2005, 8:58 pm
For a standard robots.txt file, use this:

User-agent: *
Disallow:

Create this in a text editor like Notepad and save as ASCII/ANSI format (plain text rather than unicode or anything else).