1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Optimal commands in robots.txt

Discussion in 'robots.txt' started by dshah, Nov 28, 2005.

0
  1. #1
    Could anyone please suggest what should be minimum commands present in robots.txt so that site is well crawled by search engines.

    1. Having NO robots.txt is good or bad?
    2. If its good to have rotobts.txt, what should be minimal standard commands in it (I dont need anything specific)

    Thanks In Advance!
     
    dshah, Nov 28, 2005 IP
  2. Wipqozn

    Wipqozn Peon

    Messages:
    29
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    robots.txt is useful when you disallow access on the site for some of the robots ;-) However, You can even put an empty file on a server.

    For more info visit http://www.robotstxt.org/wc/robots.html
     
    Wipqozn, Nov 29, 2005 IP
  3. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #3
    For a standard robots.txt file, use this:

    User-agent: *
Disallow:
    Code (markup):
    Create this in a text editor like Notepad and save as ASCII/ANSI format (plain text rather than unicode or anything else).
     
    minstrel, Dec 1, 2005 IP