Please make sure the robots.txt file and the Sitemap URL are correct and resubmit your Sitemap. Anyone seen this when submitting a sitemap? I have a new site that has some wierd things going on. I am not sure what it is. Adsense ads are not displaying, this site map is not working... I don't know what is going on...
First make sure your robots.txt file is accessible, then make sure it's not excluding Google or others that you want crawling your site. If that is okay, check to make sure your robots.txt file is not being "protected" via firewall software.
There are probably plenty tools that allow you to validate your text file but this Robot.txt validator works for sure. Still weird that the text file wasnt accessible, did you set the wrong CHMOD rights or doesnt that make any difference?
Now this is getting wierd... I removed the www from the site map submission and it worked, why would that be? Also, Google ads are not displaying on the www url but they are on the non-www url.
If you're going to have a robots.txt file, don't leave it blank. Have it contain the following: User-agent: * Disallow: Code (markup): Are you using any rewrites or redirects in the .htaccess file? What did you use to upload the files to the server?
I wonder if there isn't some sort of problem with your apache configuration - maybe at your host that lets spiders get to one url, but not the other. There's a firefox plugin to change your user agent. Try accessing your robots.txt file as a googlebot and see if you don't end up either getting rejected or getting redirected.
Good suggestion, nevetS. You can also check to see what headers are being returned by either robots.txt or the page you're trying to access: There are several header checkers around -- for example, you can check the headers returned for your pages by entering your URL here: http://www.seoconsultants.com/tools/headers.asp http://www.webrankinfo.com/english/t...ver-header.php http://www.kenkai.com/seo-tools-page-header.php