Block specific part of the site

Discussion in 'robots.txt' started by duffer77, Nov 22, 2010.

0
  1. #1
    Hi Guys,

    I was wondering if any of you guys know how to use robots.txt file to block acces to a specific portion of a site;

    subdomain.com/language_id/content_url

    We have about 40 subdomains and same ammount of language id's.
     
    duffer77, Nov 22, 2010 IP
  2. ramnet

    ramnet Peon

    Messages:
    32
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    This should do it:

    User-Agent: *
    Disallow: /language_id/content_url
    Allow: /
     
    ramnet, Nov 23, 2010 IP
  3. The_Hatta

    The_Hatta Greenhorn

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #3
    ramnet is correct:

    User-Agent: *
    Disallow: /language_id/content_url
    Allow: /

    or could also add a

    <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

    if theres a page you don't want them to follow. :)
     
    The_Hatta, Nov 24, 2010 IP