Don't Want Ads on a Certain Page

Though my robots.txt Disallows access to www.researchtechs.com/fast.htm somehow Google decided to index it. Since the ad network was showing zero weight without the ads on this site, I put them on for now, but I would really like not to have to put them there. Is there any way to exclude a single address from being required? Note: I recently added the meta tags noindex and nofollow as well, but they had not previously been there. Does Google totally ignore robots.txt?

 

In the Webmaster Guidelines is a links somewhere to get unindexed I believe. Have you tried changing the robots file to say /fast.htm? Not sure if that's it but seems better to me.

 

I know Google doesn't totally ignore robots.txt, but when Google finds a direct link to a page they might be indexing it without reading the robots.txt...

 

I'm pretty sure google respects robots.txt for all their stuff. People would jump on them in a second if they didn't.

 

There is a post on Searchenginewatch about Google and robots.txt

http://forums.searchenginewatch.com/showthread.php?t=3436

Unfortunatelly my experience confirms that google shows at least some first level pages excluded by robots.txt as URL only listings.

I know of a few pages on different sites that can prove it and anyone can check. At first I thought that happened because some of them might have been excluded in the robots.txt (or the robots meta tag) after being indexed. However that is not the case as I know myself of 2 pages that have been excluded in the robots.txt from the very beginning - when the site was first uploaded- and they still apear in the google index as URL only listings.

 

You can add 'noindex' to the page itself in a Meta tag, and then somewhere on Google's site, you can request that they de-index a page. Googlebot will come take a look at the page, make sure the 'noindex' is there, and then remove the page from Google's index pretty quickly.

I did this a few weeks ago for a single page and it was gone in about 24 hours.

 

The problem is that some sites add/update/change regularly a large number of pages (100s or 1000s). How do you do it then as de-indexing by url becomes virtually impossible!
On top of this google now "indexes" even many pages excluded in the robots.txt file as "url only" listings.

As far as I know, the coop validation process can check at random any page on your site that is in google's index (including "url only" listings and pages no longer existent)

 

My guess is that the API does not return URL Only listings.

(I realize this was not a good solution for hundreds of pages. The original poster only had problems with a single page)

 

I'm afraid in my experience it does! I got spot checks even on the pages that usually appear in the: