1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Google sitemap says can't access site yet robots.txt unchanged

Discussion in 'Google Sitemaps' started by travelfish, Apr 29, 2006.

  1. #1
    Logged into Sitemaps this afternoon see see a bunch of URLs no longer crawlable by Googlebot. The report reads:

    "Below are URLs we tried to crawl (found either through links from your Sitemaps file or from other pages) that we didn't crawl because they are listed in your robots.txt file. You may have specifically set up a robots.txt file to prevent us from crawling this URL...<snip> My bold"
    SEMrush
    The thing is my robots.txt doesn't list any of the files that are no longer being crawled. Indeed when I test the forbidden URLs against the robots.txt file with the tool supplied by Google, they pass wthout problem!

    Simply bizarre Google behaviour? Anyone else having problems?
     
    travelfish, Apr 29, 2006 IP
    SEMrush
  2. sas410

    sas410 Active Member

    Messages:
    128
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    53
    #2
    I've just noticed the exact same problem with my account today! I can't understand why this is happening because my robots.txt file also doesn't list any files not to be crawled.

    I wonder what's going wrong! :-(
     
    sas410, May 1, 2006 IP
  3. sas410

    sas410 Active Member

    Messages:
    128
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    53
    #3
    hmmm I've found this post on the official Google sitemaps blog that says such 'errors' have now been resolved... I guess they have except for mine :-((

    http://sitemaps.blogspot.com/2006/04/updated-robotstxt-status.html
     
    sas410, May 1, 2006 IP
  4. travelfish

    travelfish Peon

    Messages:
    46
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Yeah I saw that post too -- the errors are still showing on mine, though I assume its just till googlebot tries again.
     
    travelfish, May 1, 2006 IP