The AdSense crawler can't access your sites...

Discussion in 'Google' started by unlisted80, Dec 2, 2008.

  1. #1
    I received this email and i dont know why...I dont have any robot.txt on any of my site. Does anyone know what google adsense support email i can email them:

     
    unlisted80, Dec 2, 2008 IP
  2. yousellstuff1

    yousellstuff1 Active Member

    Messages:
    1,195
    Likes Received:
    17
    Best Answers:
    0
    Trophy Points:
    75
    #2
    what is your host?
    Are you using any script?
    Some host automatically make one for you and also some scripts. I was using vb on one of my websites and it blocked some bots. I had to make a custom rewrite for it when the new update happened.

    What is your website?

    Hope this helps
    Brandon
     
    yousellstuff1, Dec 2, 2008 IP
  3. tekteker

    tekteker Peon

    Messages:
    476
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    0
    #3
    try this,

    make a robots.txt and add these

    User-agent: Mediapartners-Google
    Allow: /
     
    tekteker, Dec 2, 2008 IP
  4. HarriL

    HarriL Member

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #4
    I just received this same message.. Weird, I have no robots.txt on my sites and always get targeted ads on my sites. How many of you have received this? It would be nice to know if they are sending this to everyone.
     
    HarriL, Dec 2, 2008 IP
  5. RTM

    RTM Guest

    Messages:
    160
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I got that too, and carefully analysed the embedded link to see if it was one of the many Adwords / Adsense phishing emails making the rounds.

    You can see your sites' issues with regards to the Mediapartners bot by clicking on the "Diagnostics" link from inside your Adsense control panel. It will show you specific URLs as well as reasons (such as robots.txt) that cannot be accessed by the bot...

    Keep in mind that if you exclude certain pages from being indexed by Googlebot (for the search engine index, not Adsense) such as "search pages" or similar on your site, it may prevent the Adsense bot from analysing them to provide the most relevant contextual ads.

    The email simply is trying to highlight the nuances between blocking the Googlebot while letting Mediapartners index your page for the purpose of serving the most relevants Adsense ads.

    Rob
     
    RTM, Dec 2, 2008 IP
  6. fitriulina

    fitriulina Peon

    Messages:
    111
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Im got that too... I just check my hosting file.. I didnt found robots.txt. So what can i do?
     
    fitriulina, Dec 2, 2008 IP
  7. JohanBru

    JohanBru Member

    Messages:
    80
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    41
    #7
    I got the same mail today too.

    We do have a robots.txt file, but it isn't disallowing any pages which ads are displayed on.


    Following RTM's advise:
    - Login to your Adsense account
    - In the Reports tab, click Site Diagnostics
    - Sort the list by Reason Blocked
    - Look for URLs blocked because of a Robots.txt File

    For me it listed only these URLs:
    hxxp://www.windowslivetranslator.com/BVFrame.aspx?s=tgt&lp=en_zh-cht&a=http://<MYSITE>.com/*
    hxxp://www.saywhatyouwant.info/browse.php?u=<ENCRYPTED-STRING-OF-MYSITE>
    hxxp://www.online-translator.com/url/translation.aspx?viewMode=result&sid=XXXXXXt&sourceURL=http://<MYSITE>.com/*
    hxxp://www.bronzecover.info/browse.php?u=<ENCRYPTED-STRING-OF-MYSITE>

    I don't own any of these domains, but as you could guess from the URLs these are translation services and web proxies loading my site in an iframe.

    Looking at the robots.txt files, these third parties do indeed block Mediapartners-Google or all bots:
    http://www.windowslivetranslator.com/robots.txt
    http://www.saywhatyouwant.info/robots.txt
    http://www.online-translator.com/robots.txt
    http://www.bronzecover.info/robots.txt

    So in my case blocking the Mediapartners bot is out of my control. All I could do is ask the owners of these external sites to stop blocking, or I could redirect/block any traffic coming from these sites. For now I'll just ignore Google's mail, if that means loosing paid ads on those pages so be it, only a very small percentage of my visitors use these services and pages loaded normally would still show paid ads. The percentage is probably lower than those of visitors using adblock, and I wouldn't want to prevent my visitors from translating my site in their language, or loading it trough proxies.

    People who have the same mail but no robots.txt on their own server (or at least nothing in there blocking Mediapartners-Google), can you confirm this?
     
    JohanBru, Dec 2, 2008 IP
  8. peepin2me

    peepin2me Well-Known Member

    Messages:
    2,097
    Likes Received:
    66
    Best Answers:
    0
    Trophy Points:
    115
    #8
    ah, Google seems to have become pretty dumb of late.
     
    peepin2me, Dec 2, 2008 IP
  9. namln

    namln Active Member

    Messages:
    280
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    58
    #9
    namln, Dec 2, 2008 IP
  10. Guillo

    Guillo Peon

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    The fact that you didn-t have a robot.txt file is a problem in itself. Even if you dont use AdSense, a robot.txt file that follows Google guidelines is a must for SEO purposes.
     
    Guillo, Apr 8, 2011 IP