I've just noticed a big jump in the number of pages Google has cached, but some of them are of pages that my robots say's NO to. Why is this happening? User-agent: * Disallow: /forum/admin/ Disallow: /forum/db/ Disallow: /forum/images/ Disallow: /forum/includes/ Disallow: /forum/language/ Disallow: /forum/templates/ Disallow: /forum/common.php Disallow: /forum/config.php Disallow: /forum/faq.php Disallow: /forum/groupcp.php Disallow: /forum/login.php Disallow: /forum/memberlist.php Disallow: /forum/modcp.php Disallow: /forum/posting.php Disallow: /forum/privmsg.php Disallow: /forum/profile.php Disallow: /forum/search.php Disallow: /forum/viewonline.php Disallow: /mail/ My forum is at: w w w.sanlucar-de-barrameda.com/forum/index.php AND the above robots is correct to me! Thanks Ian
When did you add the robots.txt file? It might take a while for Google to pick up the new version (they certainly don't request it every time) and longer still for them to automatically purge any pages left in their results/cache.
I used to use the php file excludes too. They seemed to be ignored then too. Mine were exactly the same as your's. I dunno, I just forgot about it till I read this.
It was just uploaded the weekend just gone, a very slight change on the previous with 3 additions being DB, groupCO and modCP. Guess we can't really do anything about it. On another site I had an OLD file that no longer existed which was indexed by Google and I wanted to add the co-op ads to but couldn't as I didn't have htaccess file, so waited about 2 months before it left Google's index! Thanks Ian
Take a look at http://www.google.com/remove.html. Under "Remove part of your website" it has a link for urgent removals.
Google started ignoring my noindex,nofollow,noarchive metas for a couple of days. Heck, they even removed them from the cached copies! But then a day or two later things went back to normal. I almost posted something on it but it fixed itself so I didn't bother.