Ban subdomain from robots, stop Google from indexing it

Discussion in 'robots.txt' started by Mr.Dog, Dec 1, 2018.

0
  1. #1
    Hi,

    I created a subdomain for testing a Wordpress site. It is a huge site, so it will take up a lot of space. I want to ban robots, stop Google from indexing the site etc. Basically ban everything except human visit to the page, so that I can still see it.

    If I put this into the robots.txt and upload it onto the subdomain's files, will it only affect the subdomain or the entire site?

    User-agent: *
    Disallow: /

    Because, the main domain should be left alone.

    I know this code bans robots from visiting.
    But: should I put anything else in it? Will this also block Google from indexing any pages, images found on the subdomain?
     
    Mr.Dog, Dec 1, 2018 IP
  2. mmerlinn

    mmerlinn Prominent Member

    Messages:
    3,197
    Likes Received:
    819
    Best Answers:
    7
    Trophy Points:
    320
    #2
    Using that code ONLY bans bots that RESPECT your wishes. ROGUE bots will still index your site.

    The ONLY sure ways not to get indexed are

    1) password protect your subdomain AND/OR
    2) NEVER access your subdomain publicly AND/OR
    3) make sure there are NO links from your public domain to your subdomain
     
    mmerlinn, Dec 1, 2018 IP
  3. Mr.Dog

    Mr.Dog Active Member

    Messages:
    912
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    60
    #3
    The original question was: does this code only ban bot from the subdomain or, does it also apply for the main domain?

    I want the main domain to be indexed (it is already), but I want to exclude the subdomain.

    As for password protection: I will still access the subdomain, so I will not block it entirely. I want to block unwanted traffic and robots.

    The subdomain is for a staging environment - for testing the new version of the site.
     
    Mr.Dog, Dec 2, 2018 IP
  4. mmerlinn

    mmerlinn Prominent Member

    Messages:
    3,197
    Likes Received:
    819
    Best Answers:
    7
    Trophy Points:
    320
    #4
    Read what I said again. I answered your question then added some other comments to help you, all of which allow you to access your whole site, but limits what the bots can access.

    And I know what I said works because I have a public facing website of 13,000 pages and an even larger private website with additional technical information that is useless to the public.
     
    mmerlinn, Dec 2, 2018 IP
  5. jamesandersonicb

    jamesandersonicb Peon

    Messages:
    25
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    3
    #5
    User-agent: *
    Disallow: /
     
    jamesandersonicb, Feb 14, 2019 IP
  6. surendrad dhote

    surendrad dhote Active Member

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    56
    #6
    Hi, I found this step by step guide on allowing and disallowing subdomains from a website. https://www.theproche.com/2020/05/03/robots-txt-to-disallow-subdomains/
     
    surendrad dhote, May 3, 2020 IP
  7. Borislav Arapchev

    Borislav Arapchev Greenhorn

    Messages:
    4
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    11
    #7
    Hello!
    mmerlinn is right - only password protection will save the subdomain from Google visits for sure.
    If Googlebot found a single link pointing to this subdomain - it can visit and crawl it :)
    so .. robots.txt is not solution.
     
    Borislav Arapchev, May 13, 2020 IP