Robots.txt has 'Disallow: /forums/-/' line, what is the meaning

Discussion in 'robots.txt' started by Xfactormaster, Mar 1, 2016.

0
  1. #1
    The forum software is xenforo, robots.txt has one of the line as
    Disallow: /forums/-/
    Is it the correct one? Does it effect google rank or crawling?

    Thank you.
     
    Xfactormaster, Mar 1, 2016 IP
  2. Localnode

    Localnode Active Member

    Messages:
    33
    Likes Received:
    7
    Best Answers:
    1
    Trophy Points:
    65
    #2
    It will disallow all robots from accessing /forums/-/ and /forums/*
    I'm not 100% great with robots.txt, but I believe it will stop all robots from accessing any URL with /forums/ (so /forums/*)
    Example:
    
User-Agent: *
Disallow:
Disallow: /forums/-/
    Code (markup):
    Another:
    
User-Agent: *
Disallow:
Disallow: /forums/
    Code (markup):
    To allow:
    
User-Agent: *
Disallow:
Allow: /forums/
    Code (markup):
    In these sorts of things an * represents a wildcard.
    And User-Agent has the types of robots such as
    User-Agent: Googlebot
    and User-Agent: bingbot
     
    Localnode, Mar 6, 2016 IP
  3. stilliard

    stilliard Greenhorn

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #3
    In case it helps, you can use: http://robots-txt-parser.stapps.io/ to test robots.txt rules with example urls :)
     
    stilliard, Mar 7, 2016 IP