1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

noindex by x-robots-tag is invalid

Discussion in 'Apache' started by Uchi T, May 13, 2018.

  1. #2
    I want to set "noindex" in my site, but there are so many pages so I used X-robots-tag.
    I already tried, but "noindex" isn't set.(checked by search console Fetch as google)

    so please tell me how to rewrite x-robots-tag.

    I tried 2 patterns as below.

    RewriteCond %{REQUEST_URI} (^/aaa/.+|^/bbb/ccc/.*|^/ddd/[a-z]+|^/bbb/ddd/[a-z]+|^/eee/[a-z]+|^/bbb/eee/.*|^/fff/eee/[0-9]{2}/[0-9]+)
    RewriteRule .* – [env=NOINDEX_HEADER:1]
    Header set X-Robots-Tag "noindex" env=NOINDEX_HEADER


    RewriteCond %{REQUEST_URI} ^/aaa/.+
    RewriteCond %{REQUEST_URI} ^/bbb/ccc/.*
    RewriteCond %{REQUEST_URI} ^/ddd/[a-z]+
    RewriteCond %{REQUEST_URI} ^/bbb/ddd/[a-z]+
    RewriteCond %{REQUEST_URI} ^/eee/[a-z]+
    RewriteCond %{REQUEST_URI} ^/bbb/eee/.*
    RewriteCond %{REQUEST_URI} ^/fff/eee/[0-9]{2}/[0-9]+
    RewriteRule .* – [env=NOINDEX_HEADER:1]
    Header set X-Robots-Tag "noindex" env=NOINDEX_HEADER

    <caution>
    ・I want to set noindex which directory biginning from "/aaa/" , except for /aaa/ (TOP).
    ・directories biginning from /ddd/ OR /eee/ is "noindex" target pages, but its target is which follows alphabet.
    ex./ddd/abcd → noindex /ddd/1123 → index
    ・PDF noindex is valid.(checked by search console Fetch as google) code is as below.
    <Files ~ "\.(swf|fla|pdf)$">
    Header set X-Robots-Tag "noindex"
    </Files>
     
    Uchi T, May 13, 2018 IP
  2. hostechsupport

    hostechsupport Well-Known Member

    Messages:
    413
    Likes Received:
    23
    Best Answers:
    7
    Trophy Points:
    138
    #3
    To inform search engines not to index your page, just add the following to the </head> section:

    <meta name=”robots” content=”noindex, follow”>

    The second part of the content tag shows all the links mentioned on this page must be followed, which will be discussed below:

    Alternatively, the nofollow tag can be used in an X-Robots-Tag in the HTTP header:

    X-Robots-Tag: noindex
     
    hostechsupport, May 14, 2018 IP