1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt has 'Disallow: /forums/-/' line, what is the meaning

Discussion in 'robots.txt' started by Xfactormaster, Mar 1, 2016.

0
  1. #1
    The forum software is xenforo, robots.txt has one of the line as
    Disallow: /forums/-/
    Is it the correct one? Does it effect google rank or crawling?

    Thank you.
     
    Xfactormaster, Mar 1, 2016 IP
  2. Localnode

    Localnode Active Member

    Messages:
    33
    Likes Received:
    7
    Best Answers:
    1
    Trophy Points:
    65
    #2
    It will disallow all robots from accessing /forums/-/ and /forums/*
    I'm not 100% great with robots.txt, but I believe it will stop all robots from accessing any URL with /forums/ (so /forums/*)
    Example:
    
User-Agent: *
Disallow:
Disallow: /forums/-/
    Code (markup):
    Another:
    
User-Agent: *
Disallow:
Disallow: /forums/
    Code (markup):
    To allow:
    
User-Agent: *
Disallow:
Allow: /forums/
    Code (markup):
    In these sorts of things an * represents a wildcard.
    And User-Agent has the types of robots such as
    User-Agent: Googlebot
    and User-Agent: bingbot
     
    Localnode, Mar 6, 2016 IP
  3. stilliard

    stilliard Greenhorn

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #3
    In case it helps, you can use: http://robots-txt-parser.stapps.io/ to test robots.txt rules with example urls :)
     
    stilliard, Mar 7, 2016 IP