I've been using a Google sitemap for one of my WordPress blogs. I have the sitemap and robots.txt setup so that Google indexes the posts and the categories but NOT the pages (i.e. site.com/page/2/), so that I can decrease my supplemental results. My robots.txt has this line in it: Disallow: /page/ But I can't remove all pages (site.com/page/2/, site.com/page/3/..., etc) through the URL removals in Google Webmaster Tools, it just ends up saying denied like there is no robots.txt restriction on indexing them. I also ran the URL test through the Google "Test URLs against this robots.txt file" and it said my /page/#'s could be indexed. Any help would be appreciated!
The robots.txt file only tells the SE not to enter. It does not tell the SE to remove. We have this problem all the time and you can check the correct robots.txt commands at http://en.wikipedia.org/wiki/Robots.txt. But it takes Google forever to remove the pages. The best way I have found to remove a page is to place <meta name="robots" content="noindex,nofollow,nocache" /> in the header of the page(s) that I want to remove. Then place a link on the home page to the page(s). This forces Google to index the page. Google then will follow the meta instructions. The remove URL tool of Googles still doesn't work.
Google will only remove the URL if your robots.txt says disallow to the URL or the page has that meta tag. Inserting those meta tags into WordPress is a bit tricky I was looking at how to do it and I don't think it's possible to get just the pages I want efficiently. What I am really asking is what's wrong with my robots.txt? I modified it a few days ago with "Disallow: */page/*" instead of "Disallow: /page/" and Google says the page is now disallowed (that's good), but that also disallows mysite.com/category/page/2/ (bad)
supplemental results are mostly related with the page rank. i think you have to focus getting more quality backlinks