The title of this thread sounds weird, since most webmasters want traffic to their site, but let me elaborate and explain myself. My site is steadily increasing in traffic, and the search engines are sending me visitors for certain keywords they type in. However, the pages these visitors are landing on are pages I don't want them to access without being a member, which requires them to have a unique username and password. I can't password protect every page on the site, because I use a basic Sitebuilding Software that only allows you to use one password per page, and when this site gets big, I will have a lot of people who have unique usernames and passwords, and simply having one password per page won't work. The simple question is this: How do I stop Google, or any other search engine, from sending visitors to specific pages to my site when they enter certain keywords into the search box?
Your post is not very clear, but robots.txt is the way to deny search engine bots to index some of your pages
I think you need a members only script, where all pages within a certain area will be password protected automatically.
Ideally you need all the restricted pages in one directory of the site. Then you can password protect the entire directory and SE's will also not be able to get in. As far as just making sure the pages never get listed in the SE's then as per rehash's post you will need to use robots.txt file to specify what you want and don't want the SE's to list.
If these pages were properly secured, your visitors would get a "accessed denied" or "register to view" meassage. Might be worth tightening up your members only pages.
I'm a bit new at this. How do I use the robots.txt? Once I implement robots.txt, will it automatically stop people from accessing certain pages on my site when typing certain keywords in? Do you know of a script that works with HTML based sites? A PHP or ASP.net script is worthless to me. Also, do you know of a script that is simple to install? Most scripts are very hard and frustrating to work with.
I really don't need to solve this problem immediately, since I won't start charging monthly memberships until about January 1, 2008. However, is there anyone here at DP who sells and can intall a members on script on my site? I use a sitebuilding software. Is this still possible?
I do not think it is the se's problem I think you need a new site design some kind of script that will easier facilitate your needs
I'm happy with my site's current design. I don't need a new one, and even if I wanted one, I only have a month before I'm reading to go live with the site and start charging member ships, which isn't enough time to totally design a new site. The only thing I need is a members only script that can be installed on a Site Building program like Homestead Sitebuilder, one that blocks people from accessing certain areas of the site. I'm willing to pay for both the script and the installation to anyone who can help.
robots.txt is simply to tell SE's which pages and directories of your site can be indexed. It won't stop people visiting them if you have links to those pages on your site or people visiting them through links that are already out there in the SE's (until the SE's refresh their indexes in line with your robots.txt file). As far as members script... a quick way (assuming you have a cPanel access which is standard with pretty much any hosting now-a-days) is you can simply use Password Protect directories option which will instantly secure any directory you specify on the site and only allow access to users you give usernames & passwords to.
One easy way to stop the SEs from indexing a page is to add the following line to the header: <meta name="robots" content="noindex"> That isn't the only way to do it, but it's one of the easiest.