1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

right and default format of robots.txt

Discussion in 'robots.txt' started by zeel_shah, Mar 13, 2013.

0
  1. #1
    Hello friends,

    i have confussion about robots.txt file. so someone tell me which format is default and right for php websites. :)

    User-agent: *
    allow: /
    is this right or not??
     
    zeel_shah, Mar 13, 2013 IP
  2. wonay

    wonay Well-Known Member

    Messages:
    468
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    140
    #2
    Here is the right format.....

    # Allows all robots
    User-agent: *

    And you can use this if you want to disallow spidering of certain folders....

    # Allows all robots
    User-agent: *
    Disallow:/cgi-bin/

    In this case cgi-bin is the disallowed folder.
     
    wonay, Mar 13, 2013 IP
  3. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #3
    The correct "default format" is not to have one unless you need it.

    User-agent: *
allow: /
    Code (markup):
    This is not valid and robots.txt is typically used to disallow rather than allow. In fact, avoid using allow as it is not a standard.

    Don't try to get clever by putting invalid entries in it. This is where you can end up making a mistake and disallowing your whole site. So unless you need to disallow a page/folder, don't use one.

    Read this.
     
    ryan_uk, Mar 14, 2013 IP