I have placed a robots txt file to prevent google crawling my website. but its still crawling it and my website is coming in google search result. I have done soemhing wrong? my robots.txt file is User-Agent: * Disallow: http://beta.healthcarejobssource.com/
how to do that? I dont want a single page from my webiste to b listed on any search engine till i complete it
yes and i dont want any of my web pages to get indexed by Google or any search engine.. please help anyone
Since now the pages have been indexed you will have to use removal tools from Google Webmaster Account to get the URLs out of index. In addition to robots.txt file you will have to apply the restrictions at your page level within the meta tags.
if robots.txt is not working for you, then add robot meta tag to each page with nofollow and noindex attributes to prevent google from crawling and indexing your pages.
Your robots.txt file is as follows: This is an INVALID robots.txt. You cannot Disallow a protocol or a domain. Your robots.txt goes in the root of your web. I can ONLY disallow folders and files on that web. Robots could care less what protocol is used and what domains and/or subdomains resolve to that root web folder. The correct way to disallow your entire beta site is in the root folder of beta.healthcarejobssource.com place a robots.txt that looks like: This says, "Disallow crawling of anything from the root folder down. WARNING: IF beta.healthcarejobssource.com resolves to exactly the same folder as healthcarejobssource.com then the above Disallow: / will prevent them from crawling your production web as well. But if the DNS for healthcarejobssource.com resolves to a folder called ROOT on your webserver and beta.healthcarejobssource.com resolves to some subfolder like ROOT/BETA on your webserver (or to a totally different web server) then placing the above robots.txt in the ROOT/BETA folder (or on the separate server for beta.healthcarejobssource.com) will work. PS: Once you have the robots.txt in place THEN you'll need to go to Google Webmaster Tools, go through the Google site verification process for the beta.healthcarejobssource.com site to prove you're the webmaster. And then request a URL removal for that entire site. NOTE: Google can STILL continue to show your beta pages in their index if other sites are linking to those pages even if you have the beta site blocked by robots.txt AND you get Google to remove the URLs from the index. If Google feels your page is relevant to a search they can STILL show the URL and construct a title to show in the SERPs based on the link text used to link to it without ever crawling those pages. These types of entries show in the SERPs with no snippet. Best solution IMO: An even better approach to prevent Google indexing your site would be to NOT disallow the crawl and simply render ALL beta.healthcarejobssource.com pages with a <meta name="robots" content="noindex"> element in the <head> of each page. The only way they can find the meta noindex element on the beta pages is if you ALLOW them to crawl. The meta noindex will also cause them to remove the pages from the index without having to submit a URL removal request.
Thanks for the detailed overview. the website has more than 1000 pages wo would it b possible to place noindex tags on all of them?
IMHO, you can disallow the root directory with disallow command as you have already done so and ask for a removal request from google webmaster central located at http://www.google.com/webmasters/tools that will be much Hope it helps !