Pattern Matching in Robots.txt !

Discussion in 'robots.txt' started by Charlee22, Jun 22, 2010.

  1. #1
    I created the following Robots.txt file and want to block access to URLs ending in a certain pattern.

    The URLs I want to block look like this:

    http: //www.XYZ.com/-white/car.html
    http: //www.XYZ.com/-blue/car.html


    I dont want to block any URL that looks like this:

    http: //www.XYZ.com/white-car.html
    http: //www.XYZ.com/blue-car.html

    so I have my robots.txt file as follows:

    User-agent: *
    Disallow: /*/car.html$

    So this should only block URLs ending with " /car.html "
    I want to make sure it only blocks URLs that have a " / " right before the "car.html" and not ones ending like" -car.html"

    Can you guys let me know if I have this concept correct.
    I would appreciate all feed back.

    Thanks,
     
    Charlee22, Jun 22, 2010 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    This is fine. You can go ahead with this.
     
    manish.chauhan, Jun 23, 2010 IP