I dont want to index some of dynamic pages and i blocked them through robots.txt file i.e. User-agent: * Disallow: User-agent: * Disallow: /search?sq*/ i want to block site search pages but they are indexing on google search. What wrong i am doing please guide
This is a better solution for completely blocking the index of a dynamic pages is to use a robots noindex on a per page bases. <meta name="robots" content="noindex">the page is not indexed, but links may be followed <meta name="robots" content="noindex,nofollow"> the page is not indexed & the links are not followed Allow indexing of everything User-agent: * Allow: / Disallow indexing of everything User-agent: * Disallow: /