I have created a page for the moderators at my site. An html page. Now that page is hidden meaning there are no links pointing to it, if that page gets indexed even though i have the <meta name="robots" content="noindex, nofollow" /> tag, if someone posts the link to that page on a forum or social network site, will i be penalized for "hidden pages"? Or is this ok?
Thanks for the input, so this is not what it means when SEO says a big no no on hidden pages? MVG, im afraid to put in the wrong code in the robots.txt and end up restricting other pages. Any guides on that?
Sorry for double posting but will this work? Is this the right way to all bots can crawl all pages except /vipchat.html ? User-agent: * Disallow: /vipchat.html Allow: /
Thanks the google toolbar generated this one for me: User-agent: * Disallow: /vipchat.html Allow: / User-agent: Mediapartners-Google Allow: / Code (markup): It looks right?
You would be foolish to block the page with Robots.txt. 1) Anyone looking to hack your forums is going to first look at your robots.txt to see what all you might be 'hiding'. Disallowing the page in robots.txt is just advertising, "Hack this 'secret' page!" 2) Contrary to popular belief, a disallow in a robots.txt does NOT prevent a page from being shown in Google's search results. If that page has links from external sites, even though they cannot crawl it Google can still show it in the SERPs if they think it is relevant to the search query based on the link text used to link to it. You will know when this has happened when you see a SERPs listing with a title and a URL but it has no snippet. The reason the snippet of text is missing is that Google could not crawl the page. Therefore, they could not see the meta description to determine if it should be shown as the snippet. Nor could they construct a snippet from the content of the page because they could not crawl it. You had the right solution to begin with. Simply include a <meta name="robots" content="noindex"> element in the head of the page you do NOT want to show in the SERPs and allow the crawlers to crawl the page (i.e. do NOT disallow the URL in robots.txt). The engines have to be able to crawl the page to find the meta robots noindex to figure out that you do not want them to index or show the page in their SERPs. Once they find the meta robots element they will abide by it. And chances are that no one will ever link to it anyway.