Question Re: Robots.txt

Discussion in 'robots.txt' started by NewComputer, Jun 12, 2004.

0
  1. #1
    I have a question regarding the actual tesxt involved. If you have this:

    User-agent: Ihatebots*
    Disallow:

    and

    User-agent: Ihatebots*
    Disallow: /

    The first one means allow and the second one mean disallow, correct?
     
    NewComputer, Jun 12, 2004 IP
  2. Help Desk

    Help Desk Well-Known Member

    Messages:
    1,365
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    180
    #2
    I belive second means that you are dissallowing indexing of the root or the "/" directory and its sub-directories.
     
    Help Desk, Jun 12, 2004 IP
  3. Smyrl

    Smyrl Tomato Republic Staff

    Messages:
    13,740
    Likes Received:
    1,702
    Best Answers:
    78
    Trophy Points:
    510
    #3
    The following allows all robots to visit all files because the wildcard "*" specifies all robots.
    User-agent: *
    Disallow:

    This one keeps all robots out.
    User-agent: *
    Disallow: /

    S
     
    Smyrl, Jun 12, 2004 IP
  4. Cloud Income

    Cloud Income Active Member

    Messages:
    69
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    83
    #4
    that is correct NC/

    = disallow all
     
    Cloud Income, Mar 29, 2012 IP
  5. anna30

    anna30 Well-Known Member

    Messages:
    281
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    123
    #5
    You are Correct.
     
    anna30, Apr 3, 2012 IP