1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Network Unreachable: robots.txt unreachable

Discussion in 'Google Sitemaps' started by CypherHackz, Jun 2, 2007.

  1. #1
    i have problem with my sitemap. all of my 4 websites got the same error. here is what google says:

    ==============
    Network unreachable: robots.txt unreachable
    We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.
    ==============

    fyi, all of my websites using worpress 2.2 and the sitemap was generated by a wordpress plugn. and i didnt set anything that can blocks my robots.txt in .htaccess but still, google bot can't reach my sitemaps and robots.txt.
    SEMrush
    any helps?

    -cypher.
     
    CypherHackz, Jun 2, 2007 IP
    SEMrush
  2. cormac

    cormac Peon

    Messages:
    3,662
    Likes Received:
    222
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Is there a last date for the error?

    Are the 4 blogs hosted on different servers?
     
    cormac, Jun 3, 2007 IP
  3. CypherHackz

    CypherHackz Well-Known Member

    Messages:
    447
    Likes Received:
    24
    Best Answers:
    0
    Trophy Points:
    155
    #3
    im not sure. when i login into my webmaster dashboard, i found out all of my 4 websites have error. and nope. all of them are in the same server.

    -cypher.
     
    CypherHackz, Jun 3, 2007 IP
  4. cormac

    cormac Peon

    Messages:
    3,662
    Likes Received:
    222
    Best Answers:
    0
    Trophy Points:
    0
    #4
    There should be a list of URL's that it couldnt reach and once you click on one it should give a date on the right hand side.

    What does the robots.txt analysis tool show within your account? It should tell you the status of your robot file.

    I was thinking those sites were all on the same server. You server could have been unavailable when a bot was trying to visit, this would have the bot return an unavailable error code. If the unavailable dates on the 4 sites are the same then I would bet on the server being down.
     
    cormac, Jun 3, 2007 IP
  5. CypherHackz

    CypherHackz Well-Known Member

    Messages:
    447
    Likes Received:
    24
    Best Answers:
    0
    Trophy Points:
    155
    #5
    last success visit was on 27th may 2007. and there were only 3 urls it can't reach.

    and in the robots analysis, last downloaded was on 27th may. all of my 4 websites show the same last downloaded date.

    -cypher.
     
    CypherHackz, Jun 3, 2007 IP
  6. cormac

    cormac Peon

    Messages:
    3,662
    Likes Received:
    222
    Best Answers:
    0
    Trophy Points:
    0
    #6
    That sounds right. One of my sites shows the 29th.

    Do you still have the uploaded sitemap verification file within the root of each site? If those files are there then your site should be ok.
     
    cormac, Jun 3, 2007 IP
  7. CypherHackz

    CypherHackz Well-Known Member

    Messages:
    447
    Likes Received:
    24
    Best Answers:
    0
    Trophy Points:
    155
    #7
    yes. the verification file is still there.

    btw i just checked my raw access log and found there is no last access from google bot to my robots.txt file. yahoo, msn, and other bots are still got their access to my robots.txt. but google, none.

    is there any other way to make the google bot reach to my robots and sitemap?

    -cypher.
     
    CypherHackz, Jun 3, 2007 IP
  8. cormac

    cormac Peon

    Messages:
    3,662
    Likes Received:
    222
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Yes get some quality back links to the site.

    Are you getting that error when trying to submit the site?
     
    cormac, Jun 3, 2007 IP
    Addurobi likes this.
  9. CypherHackz

    CypherHackz Well-Known Member

    Messages:
    447
    Likes Received:
    24
    Best Answers:
    0
    Trophy Points:
    155
    #9
    before this, everything was fine. no errors, no problems at all. google reach my sites everyday day. i'm not sure when it start giving me the error but as i said at the above, last success visit was 27th may. hurm...

    other bots are not having any problems. they can crawl my sites as usual (based on my raw access log).

    -cypher.
     
    CypherHackz, Jun 3, 2007 IP
  10. cormac

    cormac Peon

    Messages:
    3,662
    Likes Received:
    222
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Google will determine certain factors about your sites and of them is how often they'll visit to crawl your pages. To have them on your site more often update pages as much as possible.
     
    cormac, Jun 4, 2007 IP
  11. CypherHackz

    CypherHackz Well-Known Member

    Messages:
    447
    Likes Received:
    24
    Best Answers:
    0
    Trophy Points:
    155
    #11
    i update my website every day. about 2 to 4 posts per day.

    btw cormac, do you have any idea why google give me the error?

    -cypher.
     
    CypherHackz, Jun 4, 2007 IP
  12. cormac

    cormac Peon

    Messages:
    3,662
    Likes Received:
    222
    Best Answers:
    0
    Trophy Points:
    0
    #12
    I cant say for sure exactly as I would need to see inside your Sitemap control panel for a better idea.

    I am thinking, however, the error is back dated as the robots test passed and with 4 sites having the same error it point towards the server being unavailable.

    Whats the commands in your robots file? Can you post the full file here?

    Here is what I would try, a long shot but it might work.

    Remove one site from the Google account.
    Delete the sitmap verification file attached the site you removed.
    Delete the XML sitemap for the site
    Download & run link sleuth and gsitecrawler. If these two tools have problems crawling the site you can be sure the GoogleBot will too. Use these tools to fix broken urls etc
    Once you confirm everything is in order create a new XML sitemap and upload it to your site - Gsitecrawler can do this.
    In your robots file place the following:

    Enter the site again within the Google sitemap account and upload the new verification file and wait to see what happens.

    Feel free to post screenshots here of what your seeing inside your sitemap account.
     
    cormac, Jun 4, 2007 IP
  13. longhand

    longhand Peon

    Messages:
    255
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #13
    I've exactly the same problem in one of my hostgator shared hosting accounts. In the root is installed PHPLD script. I thought that maybe some of its .htaccess config settings disturb google in the subfolder addon domains, but i'm not sure.
    Cypherhackz! Your site is in an subfolder too? What happens if do you open a new folder and new domain google give the same result or that one reachable ok?
     
    longhand, Jun 4, 2007 IP
  14. CypherHackz

    CypherHackz Well-Known Member

    Messages:
    447
    Likes Received:
    24
    Best Answers:
    0
    Trophy Points:
    155
    #14
    here is my full robots.txt command

    btw cormac, based from my site statistic, its seem like googlebot has crawled my site.

    GoogleBot: 162 visits

    but yahoo bot crawls my site more frequent.

    Yahoo! Slurp: 1037

    is that mean google still can crawl my site?

    longhand, what do you mean by sub foldder? if it is about public_html, yeah. my site is in a subfolder name public_html.

    thank you! ;)

    -cypher.
     
    CypherHackz, Jun 4, 2007 IP
  15. CypherHackz

    CypherHackz Well-Known Member

    Messages:
    447
    Likes Received:
    24
    Best Answers:
    0
    Trophy Points:
    155
    #15
    ok this is just for update this thread. it seems google is now can crawls my websites as usual. i think maybe it is because of the server or maybe google has problems.

    anyway, i'm feeling happy now. :D and thanks to cormac and longhand.

    -cypher.
     
    CypherHackz, Jun 6, 2007 IP
  16. elvirs

    elvirs Active Member

    Messages:
    614
    Likes Received:
    24
    Best Answers:
    0
    Trophy Points:
    75
    #16
    so did this problem resolve itself or you did sth?
    I got the same error from nowhere today:(
     
    elvirs, Aug 19, 2007 IP
  17. inatmo

    inatmo Active Member

    Messages:
    338
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    58
    #17
    Hi
    Have the same problem. Google tells me that robots.txt is unreachable.
    It can't crawl the site for 3 weeks. Sad things. I'm getting sad.
    I tried to use different sitemaps generators, but no use.
    When I add to the GoogleWtools sitemap it creates a new site in panel:
    http://www.mysite.com/http://mysite.com/
    and tells that we'll crawl this site soon with updated sitemap. And in http://www.mysite.com panel this sitemap doesn't show up.
    May be it's some kind of update or else.
    What to do?
     
    inatmo, Aug 20, 2007 IP
  18. elvirs

    elvirs Active Member

    Messages:
    614
    Likes Received:
    24
    Best Answers:
    0
    Trophy Points:
    75
    #18
    What I did was rebuild the sitemap and ping google about your new sitemap at: hkkp://www.google.com/webmasters/tools/ping?sitemap=http://www.example.com/sitemap.gz
    (change that HKKP to HTTP:))
    after a while (for me it took two days) google ok's the sitemap
     
    elvirs, Aug 20, 2007 IP
  19. inatmo

    inatmo Active Member

    Messages:
    338
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    58
    #19
    Hi
    Did that. Will see what I get)
     
    inatmo, Aug 21, 2007 IP
  20. trichnosis

    trichnosis Prominent Member

    Messages:
    13,785
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #20
    trichnosis, Aug 21, 2007 IP