Hi guys, I will keep it short as much as I can .. I was creating a blog throug wordpress account. As the blog was in the beginnings of creating, I didnt want to be visible in the search engines, so I checked the option not my blog to be visible for all search engines inside wordpess (privacy option). After some days, I chenged the option --> to be visible for search engines Then inside my google webmaster account, I entered in sitemap section in order to submit the sitemap.xml. However, I received the same error after a lot of tries resubmitting sitemap: Googlebot restricted URL: domain.com. An error occured while trying to access your url. Please ensure sitemap is according our guidelines and that your site is accesible, then resubmit it .. Then I check my google robots.txt which states always this: User-agent: * Disallow: / But when clicking on it, then I see in a new webpage which comes up, that the content of robots.txt is this: User-agent: * Disallow: Sitemap: domain.name Strange doesnt it ?? I cant remove the dash "/" after word disallow, as Google does not permit/save it. So Im currently with the Sitempa error, and I think that as robots cant access my domain name, maybe its a sign that it wont be indexed. Do u have any idea or solution how to fix this sitemap error? How to change permision of the current robots.txt file and make appear in google text field the same, as it is in reality (as appears when clicking the link "http://domain.com.robots.txt") Looking forward to hearing from you
do you upload your robots.txt in root directory of your domain ? i think robots.txt is the first file that spider read when it entering your web.
thanks asadiyah I did it, however the same error. In general for other wordpress sites its ok, the sitemap has NO errors without uploading any robots.txt file on the server. That may be caused due to making the site not visible for the search engines in the beginnings. But I changed the option and I was expected that robots.txt would change too. I have uploaded the robots.txt file on server as it should be: domain-name/robots.txt so to include all pages..
hey, its fixed! Seems was needed some time for Google to discover that robots.txt permits all pages. I dont know, but I suppose that having a robots.txt file which permits all pages on the appropriate root server or not having it at all, should be the SAME thing. I dont know why happened this to be honest..
I ran across this the other day and was wondering if robots.txt would prevent your sitemap from being grabbed by google.
make sure that your robots.txt file has without error..you want to varify your robots.txt then go to http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php here and check your robots.txt is ok or not.. User-Agent: * Allow: / Sitemap:yourdomain name/sitemap.xml if u have any problem with creation of robots.txt then go to the http://www.searchenginepromotionhelp.com/m/robots-text-creator/simple-robots-creator.php site and create your robots.txt and upload into your root folder..