I'm developing in a sandbox right now. I need the adsense ads on my page to take a look at the overall design. I want to allow the google adsense bot to view the page, and present me with a real ad, but I don't want the page indexed in google. Is there a way to do this with robots.txt ? My other options are screenshots, or manually coding it.
You might try using this in the head of the page. It tells the bots not to index the page, and not to follow links. <meta name="robots" content="noindex,nofollow"> Code (markup):
You could block all bots except the ad bot. User-agent: googleadsensebotwhosenameescapesmerightnow Disallow: User-agent: * Disallow: /
Like this, the * at the end of the user-agent is a wildcard. I think the current version is Mediapartners-google/2.1. So this takes care of it if Google changes the version of the bot. user-agent: Mediapartners-Google* Disallow: User-agent: * Disallow: / Code (markup):