I have 2 pages in my site which are for prior customers only to access. They offer the product at a discount to those who have bought before. Both these pages are appearing in search engines (allbeit on page 32!) and this is enabling anyone to come in and get the discount right off the bat. How can I stop this using robots text? I expect it is something very simple and basic but I have not got a clue! mogsta
Insert the code below into the Meta Tags (at the top of your html code) in your website pages that you don't want to be indexed in the search engines. <meta name="robots" content="noindex,nofollow"> Mike
You may need to login to google webmaster tools and request removal once you've done the above - I had problems with a page staying in the index after I had excluded it in robots.txt, this solved it. BTW, the way to do it with robots.txt is User-agent: * Disallow: /directory/pagename.htm
I tried this request as google had indexed registration form fo my site and showing it at the top results for my site name but they rejected saying blah blah.. this cant be done After some time the search results went fine. is there a better way to restrict the crawlers and crackers to know where is our important data ,instead of giving them robots.txt and telling them where is our safe!
If you want to prevent Googlebot and all other bots from visiting some parts of your web-site, you should put them in a pass-word protected directory, whose contents are only visible after login.