Optimal commands in robots.txt

Discussion in 'robots.txt' started by dshah, Nov 28, 2005.

  1. #1
    Could anyone please suggest what should be minimum commands present in robots.txt so that site is well crawled by search engines.

    1. Having NO robots.txt is good or bad?
    2. If its good to have rotobts.txt, what should be minimal standard commands in it (I dont need anything specific)

    Thanks In Advance!
     
    dshah, Nov 28, 2005 IP
  2. Wipqozn

    Wipqozn Peon

    Messages:
    29
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Wipqozn, Nov 29, 2005 IP
  3. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #3
    For a standard robots.txt file, use this:

    User-agent: *
    Disallow: 
    
    Code (markup):
    Create this in a text editor like Notepad and save as ASCII/ANSI format (plain text rather than unicode or anything else).
     
    minstrel, Dec 1, 2005 IP