1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How NOT to get indexed?

Discussion in 'Search Engine Optimization' started by syda, Apr 24, 2015.

  1. #1
    I'm creating a website that I don't want to get indexed. Well, actually I want the main site, mysite.com NOT indexed and I'll make a blog as a sub-domain (blog.mysite.com), which I DO want indexed.

    There is a feature in WP settings to Discourage search engines from indexing this site. However, can I be sure that if I tick this, the site will definitely not be indexed?
     
    syda, Apr 24, 2015 IP
  2. Wasi88

    Wasi88 Active Member

    Messages:
    56
    Likes Received:
    4
    Best Answers:
    2
    Trophy Points:
    55
    #2
    I don't know anything about WP, sorry. But I know how to do it directly within the robots.txt or the HTML output. Don't worry about your blog. You tell Google not to index www. and blog. will not be effected. Use the following methods with caution. Meaning it's up to the spider if they actually want to look at your 'noindex' properties or not. Google and all the other big ones do respect your settings.

    The best way would be setting the right properties within the robots.txt:
    Disallowing everything for everyone:
    User-agent: *
    Disallow: /

    OR disallowing everything for Googlebot only:
    User-agent: Googlebot
    Disallow: /

    Google Support: https://support.google.com/webmasters/answer/6062608?hl=en

    An other way would be using Meta-Tags:
    <meta name="robots" content="noindex">
    <meta name="googlebot" content="noindex">

    Important! For the noindex meta tag to be effective, the page must not be blocked by a robots.txt file.
    Google Support: https://support.google.com/webmasters/answer/93710?hl=en

    So, just enable the setting within WP and see what method they use. Call www.mysite.com/robots.txt. If this file does not exist, has no content or not any content that actually says 'disallow' then take a look at the HTML response within www.mysite.com and check for the proper meta tags.
     
    Last edited: Apr 25, 2015
    Wasi88, Apr 25, 2015 IP
    syda likes this.
  3. syda

    syda Well-Known Member

    Messages:
    430
    Likes Received:
    39
    Best Answers:
    0
    Trophy Points:
    115
    Articles:
    9,213
    #3
    Thanks! I will try that.
     
    syda, Apr 25, 2015 IP
  4. Wasi88

    Wasi88 Active Member

    Messages:
    56
    Likes Received:
    4
    Best Answers:
    2
    Trophy Points:
    55
    #4
    Let us know if you need further help.
    Btw: There is a decent robots.txt checker within Google Webmaster Tools:
    google.com/webmasters/tools/robots-testing-tool
     
    Wasi88, Apr 25, 2015 IP
  5. dscurlock

    dscurlock Prominent Member

    Messages:
    4,564
    Likes Received:
    260
    Best Answers:
    0
    Trophy Points:
    300
    #5
    I imagine you can use all the tricks there is not to index.
    at the end of the day, it is up to google or other engines
    to honor your request; google will do what they well please.
     
    dscurlock, Apr 25, 2015 IP
  6. DocuMaker

    DocuMaker Active Member

    Messages:
    427
    Likes Received:
    16
    Best Answers:
    0
    Trophy Points:
    60
    #6
    Using the noindex meta tag is the only thing that makes sense to me. Via robots.txt, you can't allow access to a subdirectory if the main directory is blocked. Bots need to access the main directory to get to the subdirectory!

    Nevertheless, both approaches only work with bots that honor these tools. There are a number of bots that don't.
     
    DocuMaker, Apr 25, 2015 IP
    syda likes this.
  7. affiliolabs

    affiliolabs Greenhorn

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #7
    You can do this with tons of free wordpress plugins....
    https://wordpress.org/plugins/ultimate-noindex-nofollow-tool-ii/
    https://wordpress.org/plugins/easy-noindex-and-nofollow/

    hope this helps!
     
    affiliolabs, Apr 25, 2015 IP
  8. syda

    syda Well-Known Member

    Messages:
    430
    Likes Received:
    39
    Best Answers:
    0
    Trophy Points:
    115
    Articles:
    9,213
    #8
    Thanks for explaining. So maybe it's best to do it the other way round? Have the main site indexed and put the part I don't want to get indexed under a sub-domain?
     
    syda, Apr 26, 2015 IP
  9. meethi jain

    meethi jain Active Member

    Messages:
    169
    Likes Received:
    0
    Best Answers:
    1
    Trophy Points:
    56
    #9
    use robots.txt
     
    meethi jain, Apr 27, 2015 IP
  10. Auraiya ITS

    Auraiya ITS Member

    Messages:
    39
    Likes Received:
    0
    Best Answers:
    2
    Trophy Points:
    46
    #10
    You need update your sitemap & resubmit it. Also do search engine submissions.
     
    Auraiya ITS, Apr 28, 2015 IP