Can anyone tell me how I can tell the search engines not to index a certain page? I included a robots.txt file for that but Google has indexed it which means your download link is visible. Dlguard is expensive. What shall I do?
Sign up at google webmaster tools for free and verify your ownership of the website takes around 2 minutes for the entire process once done then log in to the webmaster account and go to site configuration -> Crawler access -> Remove url . This should remove your url in just 2 to 4 hours and it will never be indexed again unless you chose to index it back. You can always cancel this at any time so just use the method above to remove it from google same applies for yahoo and bing.
There could be a possibility that the page URL is not properly specified in the robots.txt file. However, as its already been indexed now, you can post a request to remove the URL through your GWT. I'm not sure whether Google removes or it may deny removing it... I haven't tried it though.
My page was indexed at the time of this thread posting but some URLs were not indexed. Then I searched i Google and found the info that it is possible to tell Google or other search engines which ones not to index via robots.txt file. So I did that too. But after a while when I checked it, they were all indexed. Robots.txt file no longer used? I will try the Google webmaster suggestion now.
i'm a newbie, but i'm thinking why you do not certain page to be indexed? for what purpose? if it is useless, why not to delete it? thank you for advance.
only dess71 can answer your question as its related to him and its up to him to answer or not. The general reason would be, if the webmaster wants to post some huge content... and if he has no time to place it at a time then he would like search engines not to index his content. Until the design of the site is completed, its not good to allow search engines to index the pages of the site.
Very sorry for the delayed response. I have a dial up connection and digitalpoint will not load most of the time that is why I am late to respond. This is why I wanted the search engines not to index a certain page in my domain. I have a downloadable product on my page and if Google indexes it then it is visible to all. Everyone can get the product without payment. You know what I mean. You can make a certain kind of querry that reveals every indexed pages of a domain. and it is site:domain.com this query and may be another query like allinurl: will give you all the pages of a domain indexed by that search engine. So basicaly I wanted to protect my product but Google wouldn't listen. Then I can't do anything. I also can not afford to buy the download manager dlguard. That's all.
I post pages to the Internet so I can access them myself from anywhere, but they are not for the public. There is no reason to have those pages indexed. It is also easy to just password protect any pages you do not want other seeing.