Can I use * in my robots file

Discussion in 'Google' started by premierfabio, May 6, 2011.

  1. #1
    I want to disallow thiese category from indexing.
    e.g
    /abc/xyz/2011/ and /abc/zyx/2011/

    the red parts are different. Can I use *to represent xyz and zyx in robots?
     
    premierfabio, May 6, 2011 IP
  2. saqib889

    saqib889 Peon

    Messages:
    1,291
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #2
    do you want to deny these red parts?
     
    saqib889, May 6, 2011 IP
  3. premierfabio

    premierfabio Peon

    Messages:
    17
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I want to deny /abc/xyz/2011/ and /abc/zyx/2011/, but want /abc/xyz/ and /abc/zyx/ to be indexed.
     
    premierfabio, May 7, 2011 IP
  4. tattoos

    tattoos Prominent Member

    Messages:
    1,903
    Likes Received:
    150
    Best Answers:
    0
    Trophy Points:
    335
    #4
    Some bots will respect it, some won't.
    Most of the bots respect blocking a particular file type. EG: to block access to all pdf files in your site use... Disallow: /*.pdf$

    To block the /2011/ folder in your example, use Disallow: /2011*/
    Make sure you test it in Google webmaster tools before uploading it to your site. Not sure how many bots respect that, but most of the major ones should.

    Further reading
    http://ghita.org/search-engines-dynamic-content-issues.html
    http://en.wikipedia.org/wiki/Robots.txt
    http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449&from=40367&rd=1

    Cheers
    James
     
    tattoos, May 7, 2011 IP