A Question About Robots.txt

Discussion in 'General Chat' started by Masterful, Oct 5, 2008.

  1. #1
    If I put a Robots.txt file into my root directory with instructions to block all bots, will it affect my subdomains (in subdirectories), too?
     
    Masterful, Oct 5, 2008 IP
  2. Masterful

    Masterful Well-Known Member

    Messages:
    1,653
    Likes Received:
    28
    Best Answers:
    0
    Trophy Points:
    140
    #2
    Anyone know?
     
    Masterful, Oct 5, 2008 IP
  3. alarik

    alarik Active Member

    Messages:
    382
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    58
    #3
    depends on what exactly did you write in the robots.txt file.
    If you block root then normally your whole website should be blocked from bots.
    BUT I had several experiences back then when I did the same thing but STILL, my website was indexed in google. I heard that this happened to a lot of people so I suggest that you don't rely on robots.txt .
    Also I don't know this for a fact but I think if you restrict root and you directly post a link to a subfolder of that root, google bot will still access the subfolder using that link and bypassing the root restriction.
     
    alarik, Oct 5, 2008 IP
  4. PromaneX

    PromaneX Peon

    Messages:
    76
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    if you have you sub, for example, at mysub.mydomain.com and its folder is in mydomain.com/mysub and your robots.txt file is in mydomain.com/ then it will have no effect on the mysub.mydomain.com because google will access it via that domain (not your main domain) so will never come accross the robots.txt file. Place the file in each sub domains root dir.
     
    PromaneX, Oct 5, 2008 IP
  5. coachtraffic

    coachtraffic Peon

    Messages:
    187
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Also matters if the bots play by the rules too.
     
    coachtraffic, Oct 5, 2008 IP
  6. Masterful

    Masterful Well-Known Member

    Messages:
    1,653
    Likes Received:
    28
    Best Answers:
    0
    Trophy Points:
    140
    #6
    I only want to block the root, not the subdomains. Is that possible?
     
    Masterful, Oct 5, 2008 IP
  7. Masterful

    Masterful Well-Known Member

    Messages:
    1,653
    Likes Received:
    28
    Best Answers:
    0
    Trophy Points:
    140
    #7
    I have 2 different domains. One is in the root directory; the other is in a sub-directory. So, according to what you're saying, if I put the Robots.txt file into the root directory, the other domain in the sub-directory won't be affected. Is that correct?
     
    Masterful, Oct 5, 2008 IP
  8. I-G4L

    I-G4L Peon

    Messages:
    312
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #8
    If you put it in root, it does not affect any folders, subdomains.
     
    I-G4L, Oct 5, 2008 IP
  9. Masterful

    Masterful Well-Known Member

    Messages:
    1,653
    Likes Received:
    28
    Best Answers:
    0
    Trophy Points:
    140
    #9
    OK. Thanks, guys. I've added points to all of your reputations. :)
     
    Masterful, Oct 5, 2008 IP
  10. justinlorder

    justinlorder Peon

    Messages:
    4,160
    Likes Received:
    61
    Best Answers:
    0
    Trophy Points:
    0
    #10
    It doesn't work on subdomains.
    All subdomains are considered a new site.
    By the way, you should name the file with "robots.txt" all low case, not "Robots.txt"
     
    justinlorder, Oct 5, 2008 IP