noindex by x-robots-tag is invalid

Discussion in 'Apache' started by Uchi T, May 13, 2018.

  1. #2
    I want to set "noindex" in my site, but there are so many pages so I used X-robots-tag.
    I already tried, but "noindex" isn't set.(checked by search console Fetch as google)

    so please tell me how to rewrite x-robots-tag.

    I tried 2 patterns as below.

    RewriteCond %{REQUEST_URI} (^/aaa/.+|^/bbb/ccc/.*|^/ddd/[a-z]+|^/bbb/ddd/[a-z]+|^/eee/[a-z]+|^/bbb/eee/.*|^/fff/eee/[0-9]{2}/[0-9]+)
    RewriteRule .* – [env=NOINDEX_HEADER:1]
    Header set X-Robots-Tag "noindex" env=NOINDEX_HEADER


    RewriteCond %{REQUEST_URI} ^/aaa/.+
    RewriteCond %{REQUEST_URI} ^/bbb/ccc/.*
    RewriteCond %{REQUEST_URI} ^/ddd/[a-z]+
    RewriteCond %{REQUEST_URI} ^/bbb/ddd/[a-z]+
    RewriteCond %{REQUEST_URI} ^/eee/[a-z]+
    RewriteCond %{REQUEST_URI} ^/bbb/eee/.*
    RewriteCond %{REQUEST_URI} ^/fff/eee/[0-9]{2}/[0-9]+
    RewriteRule .* – [env=NOINDEX_HEADER:1]
    Header set X-Robots-Tag "noindex" env=NOINDEX_HEADER

    <caution>
    ・I want to set noindex which directory biginning from "/aaa/" , except for /aaa/ (TOP).
    ・directories biginning from /ddd/ OR /eee/ is "noindex" target pages, but its target is which follows alphabet.
    ex./ddd/abcd → noindex /ddd/1123 → index
    ・PDF noindex is valid.(checked by search console Fetch as google) code is as below.
    <Files ~ "\.(swf|fla|pdf)$">
    Header set X-Robots-Tag "noindex"
    </Files>
     
    Uchi T, May 13, 2018 IP
  2. hostechsupport

    hostechsupport Well-Known Member

    Messages:
    413
    Likes Received:
    23
    Best Answers:
    7
    Trophy Points:
    138
    #3
    To inform search engines not to index your page, just add the following to the </head> section:

    <meta name=”robots” content=”noindex, follow”>

    The second part of the content tag shows all the links mentioned on this page must be followed, which will be discussed below:

    Alternatively, the nofollow tag can be used in an X-Robots-Tag in the HTTP header:

    X-Robots-Tag: noindex
     
    hostechsupport, May 14, 2018 IP