Network unreachable: robots.txt unreachable

I see the following message a lot and have searched around but haven't found a direct answer on what to do on the server side to allow Google's IP to be allowed:

Network unreachable: robots.txt unreachable
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.

What do you all recommend?

 

I'm not sure. I'm also seeing the same error message - my robots.txt file is in the root of my website (as are my sitemaps) and I can browse it no problem, but when Google tries to read my robots.txt it's coming back with this error.

 

You really shouldn't have to do anything for this to work -- it should work by default.

 

i also get such message when i use certain tools to check for broken links. I don't think it is any kind of error but it is related to the server or host of our site.

 

Webmaster Tools gives the impression that it can't read the robots.txt file,

http://www.neforum2.co.uk/robots.txt

Network unreachable: robots.txt unreachable
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit. Found: Jun 16, 2008


and that there are no urls in the sitemap.

Property Status
Sitemap type Web
Format –
Submitted Jun 15, 2008
Last downloaded by Google Jun 16, 2008
Status Errors
Total URLs in Sitemap 0
Indexed URLs in Sitemap –


http://www.neforum2.co.uk/ccsharps.xml

If you follow these links you'll see that not the case, so why doesn't it work? The domain is verified correctly and is very simple in structure. The ccsharps site is in a subdirectory immediately below the root.

 

Exactly, so it is from the server end. What most like I said have mentioned is, something on the server is blocking Google's IP's that go around looking for your sitemap.xml , but I wanted to know how do you figure out what IP's Google uses so you can whitelist them all, or what can you do to configure on your server so that your server will not block Google's IP?

 

Be default, your server should not be blocking anyone.

To whitelist Google, you can whitelist 66.249.*.*

Wow, look at the "Similar Threads" at the bottom of this page. A lot of people run into this error.

 

Could you explain the correct way to whitelist please? Is it done in the .htaccess file?

 

Whitelisting is a "concept' and as such there is no "one way" to implement it.

If you are using .htaccess to block, then you would use .htaccess to whitelist.

If you are using a firewall to block, then you would whitelist in that firewall's configuration.

Here's the whitelist from one of my .htaccess files:

Allow from 127.0.0.1
Allow from 65.55                # MSN
Allow from 66.249               # Google
Allow from 67.195               # Yahoo!
Allow from 72.30                # Yahoo!
Allow from 74.6                 # Yahoo!
Allow from 122.152.129.15       # Baidu

Code (markup):

 

Thanks Will.

I had "allow from all" set in my .htaccess file anyway, so no problem there. It's a bit perplexing this one.

 

I seem to have solved this.

I was previously verifying the sites by creating the blank google html key file, but have since changed to putting the meta tag verification key in my index page header instead. I also added my domain in Webmaster Tools as neforum2.co.uk instead of www.neforum2.co.uk. The domain verifies and the sitemaps submit without any errors.