Debt Consolidation - Bet365 bonus - Debt Consolidation - Find jobs - Credit Card Consolidation
PDA

View Full Version : We were unable to access the URL you provided due to a restriction in robots.txt.

NewComputer
Jun 20th 2005, 11:06 pm
Please make sure the robots.txt file and the Sitemap URL are correct and resubmit your Sitemap.

Anyone seen this when submitting a sitemap? I have a new site that has some wierd things going on. I am not sure what it is. Adsense ads are not displaying, this site map is not working... I don't know what is going on...
jbw
Jun 21st 2005, 2:21 am
So whats in your robots.txt file?
wrmineo
Jun 21st 2005, 4:45 am
First make sure your robots.txt file is accessible, then make sure it's not excluding Google or others that you want crawling your site.

If that is okay, check to make sure your robots.txt file is not being "protected" via firewall software.
NewComputer
Jun 21st 2005, 5:24 am
So whats in your robots.txt file?

It was blank...
crazyhorse
Jun 21st 2005, 5:41 am
There are probably plenty tools that allow you to validate your text file but this Robot.txt validator (http://www.searchengineworld.com/cgi-bin/robotcheck.cgi ) works for sure. Still weird that the text file wasnt accessible, did you set the wrong CHMOD rights or doesnt that make any difference?
NewComputer
Jun 21st 2005, 5:43 am
The robots.txt file validates... My host just said that it may be a problem with my custom 404 page?
NewComputer
Jun 21st 2005, 6:08 am
Now this is getting wierd...

I removed the www from the site map submission and it worked, why would that be? Also, Google ads are not displaying on the www url but they are on the non-www url.
minstrel
Jun 22nd 2005, 10:26 pm
If you're going to have a robots.txt file, don't leave it blank.

Have it contain the following:

User-agent: *
Disallow:

Are you using any rewrites or redirects in the .htaccess file?

What did you use to upload the files to the server?
NewComputer
Jun 22nd 2005, 10:48 pm
I upload using WS_FTP, I am not using any htaccess rewrites or redirects...
nevetS
Jun 22nd 2005, 11:10 pm
I wonder if there isn't some sort of problem with your apache configuration - maybe at your host that lets spiders get to one url, but not the other. There's a firefox plugin to change your user agent. Try accessing your robots.txt file as a googlebot and see if you don't end up either getting rejected or getting redirected.
minstrel
Jun 23rd 2005, 7:14 am
Good suggestion, nevetS.

You can also check to see what headers are being returned by either robots.txt or the page you're trying to access:

There are several header checkers around -- for example, you can check the headers returned for your pages by entering your URL here:

http://www.seoconsultants.com/tools/headers.asp
http://www.webrankinfo.com/english/t...ver-header.php
http://www.kenkai.com/seo-tools-page-header.php