AdSense, subdomains and robots.txt

[tripower]

Hi,

I've come up against an issue that I can't seem to resolve with Adsense. I've emailed Google, but their replies are not helpful (it seems that they either aren't thoroughly reading my emails or don't understand them). I've searched the web for hours with little luck.

The problem I'm encountering is that Google AdSense does not seem to be crawling my subdomain pages and thus, relevant AdSense ads are not being displayed.

I have a somewhat unusual situation. We run a website that provides a mirror of our sites to affiliate partners. These mirror sites contain the same content as our main site, but we change out the logos and colors to match our affiliate's taste. These mirror sites are all hosted on our server as subdomains of our main site....like this ->

www.domain.com (our main site)
affil1.domain.com (affiliate 1)
affil2.domain.com (affiliate 2)
and so on...

Because all of the subdomains have the same content as our main page, we have purposely placed a robots.txt file on each subdomain so that the search engines would not think we're spamming them with mirror sites and blacklist us.

I believe the problem is that AdSense cannot crawl our subdomain pages. In order to fix this, I have modified our robots.txt file according to the AdSense FAQ so that the AdSense crawler can crawl the pages, but none of the search engines will be allowed to crawl. According to the Google AdSense FAQ:

"If you would like to grant our crawler access your pages, you can do so without granting permission to any other bots. Simply add the following two lines to the top of your robots.txt file:

User-agent: Mediapartners-Google*
Disallow:
"

I have done this, but the AdSense ads on the subdomains are still irrelevant to the content of the page and seem to be default text link ads for our top domain.

I created two test pages to try to figure this out. Both pages have the EXACT same content on them. Both pages should be displaying 2 AdSense ads on them. As you can see, the second page only displays 1 ad and it's not relevant to the content on the page.

TEST PAGE 1 on top domain:
http://www.chooseyouritem.com/adsense-test.html

TEST PAGE 2 on subdomain:
http://greatvehicles.chooseyouritem.com/adsense-test.html

Here's the robots.txt file for the subdomain (found at: http://greatvehicles.chooseyouritem.com/robots.txt)

User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /

I can't just delete the robots.txt file on the subdomains for fear that the search engines will blacklist us for spamming them with mirrored content. How do I get AdSense to crawl the pages and return relevant ads?

Any and all suggestions appreciated!

Thanks!

[nevetS]

I ran into that problem. It took a couple of weeks after updating my robots.txt file for good adsense ads to show up again. Lucky for me it was a tester site.

Maybe use the code for framed adsense ads to provide the location of your regular page.

[tripower]

I ran into that problem. It took a couple of weeks after updating my robots.txt file for good adsense ads to show up again. Lucky for me it was a tester site.

Maybe use the code for framed adsense ads to provide the location of your regular page.

Sheesh! I hope I don't have to wait a couple of weeks for the crawler.

On your test site, did you originally have all crawlers blocked and then change your robots.txt file to allow the AdSense crawler?

I understand your suggestion about using the framed AdSense ads, but I'm not sure how I can direct Google to the main domain pages for retrieving the content of the pages.

Thanks!

[nevetS]

I originally had it set up to not allow any spiders. Then it turns out their PSAs were one huge ad instead of a series of 4 text ads, so I couldn't have a realistic way of designing the site. It seemed like the bot was coming by, but it kept on showing the PSAs for an annoyingly long period of time.

[tripower]

I originally had it set up to not allow any spiders. Then it turns out their PSAs were one huge ad instead of a series of 4 text ads, so I couldn't have a realistic way of designing the site. It seemed like the bot was coming by, but it kept on showing the PSAs for an annoyingly long period of time.


You were right on the money! The AdSense ads finally started to show relevant links in my subdomains. It just took time (longer than I thought it would). There was no delay on my top domain (which doesn't have a robots.txt).

I wouldn't have thought that a change in the robots.txt file would cause any delay in crawling the site, but apparently it did.

Thanks!

[paymentapprovaltooslow]

I found that tip works well, too. Good advice. The ads were better.