My site is not indexed since last 6 days and when submit sitemap then it says, URL restricted by robots.txt We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit. While i do not create any robots file
Seems strange. Why don't you have a robots.txt file? Maybe you should create one and allow access to your entire site, see if that helps! Cheers James.
If you haven't created a robot.txt by yourself, still check if your domain has a robots.txt Many of the webhosts create robots.txt by default and prohibit crawling of few inner dirs / specific files. Also check out http://groups.google.com/group/Google_Webmaster_Help-Sitemap/browse_thread/thread/a09d6a406ddb9ac1 - Similar problem happened to me.
Delete everything on your site and re-upload. You can even start with only uploading a few files and move from there. This will help you clarify what is going on.
Similar is the problem with two of my new blogs, i am about to start a thread. its been 3 weeks since my blogs are live but didn't yet indexed by Google, it is 1 year old domains with establish PR but not showing any results in wayback "archive.org" & give me this error. When i contacted my hosting provider they replied I check the root directory but there is no robots.txt file, so i write to the hosting provider again & tell them there is no such file in the root directory..today i got a reply from them that Here i want to mention that, when i buy the domains all its previous record was available in archive.org. so there is no issue with the past owner of the domain but when i switched this domain to a new host. it stop giving me details in archive.org not its indexing at Google yet..is there any problem with the hosting? or i need to wait more. please help
What? I thought I heard it all but that's a new one. I suppose what you'd be trying to do is change the "last modified" date. That can be changed in the sitemaps, just by specifying a new date. Or I suppose you could modify the pages on your site slightly to change the date returned by the server. Deleting everything is bad, bad, bad, because the age of URL's is VERY important. The older the better, just like domains.
Yeah the best bet would be to manually create a robots.txt file and then allow all access, by doing this: User-agent: * Disallow: Sitemap: YOUR SITEMAP URL
if it is the site listed as your Recent Blog post on your profile... you need to correct this. http://www.hungamaonline.net/robots.txt Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Code (markup):
My site index few days before after submitting sitemap , building few links ( my old sites ) but first time it takes time in indexing.
You have to first add robot.txt file inside your website and then follow all neccesary things to add in robot.txt file. Robot.txt file is very important to index in Google or any search engine. Please folllow everything which is needed to index in Search engine and I think if you do google search about robot.txt file than you will get everything about robot.txt and how input in. Or you try anysite likee domain.com/robot.txt, try the top website and see how they do it. And don't forget about Meta tags things.. I hope this works for you. Thanks Alam