1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Can someone tell me what is wrong with my robots.txt file?

Discussion in 'robots.txt' started by Boxwell, Nov 24, 2016.

0
  1. #1
    Hello,

    I have a robots.txt file that has the following content:

    User-Agent: *
Disallow:
Disallow: /developmentsite/
    Code (markup):
    I'm looking to stop the indexing of the development section of my site (anything that falls after /developmentsite/. The robots.txt is placed in the root of the site. When I check it using http://tools.seobook.com/robots-txt/analyzer/ it says that robots are allowed to index this area.
    I'm very confused and would appreciate any help you can give me.

    Best wishes,

    Graeme
     
    Boxwell, Nov 24, 2016 IP
  2. Zoti Media Group

    Zoti Media Group Notable Member

    Messages:
    1,598
    Likes Received:
    113
    Best Answers:
    2
    Trophy Points:
    265
    Digital Goods:
    2
    #2
    Remove the first Disallow:
    The empty one.
     
    Zoti Media Group, Nov 24, 2016 IP
  3. GameO.Over

    GameO.Over Active Member

    Messages:
    45
    Likes Received:
    7
    Best Answers:
    1
    Trophy Points:
    65
    #3
    Are you sure it says that?

    I just tested it:

    Url: http://www.test.com/developmentsite/
Robots disallowed: All robots
    Code (markup):
    Thats a very, very poor URL for your development site.. It is completely up to the crawler whether it honours your robots.txt rules. May not be a problem with Google but it could become indexed elsewhere. Your development site should ideally be offline, if thats not possible why not create a sub domain? dev.yoursite.com. If it must be online make sure it is password protected with .htaccess rules so that it can't be accessed at all via the outside world.
     
    GameO.Over, Nov 24, 2016 IP