Robotx.TXT Files

Discussion in 'Search Engine Optimization' started by Steeves, Mar 18, 2009.

0
  1. #1
    I am setting up the robots.txt for the http://archive.womansday.com/ site
    Here is my Code:

    User-agent: *
    Disallow:/archive.womansday.com/

    Will it only disallow that one URL above or all the URLs under that subdomain. Thanks
     
    Steeves, Mar 18, 2009 IP
  2. StevePro

    StevePro Peon

    Messages:
    106
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #2
    You want to put the robots.txt file in the root of the subdomain and use

    User-agent: *
    Disallow:/

    this will disallow all pages within that sub domain from being indexed.
     
    StevePro, Mar 18, 2009 IP
  3. SEO_WatchDog

    SEO_WatchDog Well-Known Member

    Messages:
    1,148
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    115
    #3
    You need to use separate robots.txt for every subdomain and domain
     
    SEO_WatchDog, Mar 18, 2009 IP