I have problems when i try to fetch my site with google webmaster tools. I created a website on webs.com. Need some help..
You should learn to use a search engine. It was pretty easy to find the answer with a quick search regarding your "host" and robots.txt: http://support.webs.com/webs/topics...cking_google_from_being_able_to_crawl_my_site
I don't know when i try to fetch, it say blocking by robots.txt . Thanks ''Ryan_uk'' but allready viewed that topic, and it's not helpful -_-.
I don't think you did or you would have done what the official rep mentions - to contact them if robots.txt Disallow: / is still there after seven days.
Just create a robots.txt file that says User-Agent: Googlebot Allow: / User-Agent: * Disallow: / User-Agent: Googlebot Disallow: /privatstuff Code (markup): Add anything you don't want indexed to the fourth line or add more pairs of lines as Shown above
Looking back, I can't think of a good reason. For the OP's problem all needed is User-Agent: Googlebot Allow: / Code (markup):
Check out your robots.txt file and try to remove that link from robots.txt file and then fetch your websites link in Google Webmaster tools & I believe it will work.
What is different between allow and disallow in robots.txt and can i make two different file in robots.txt one for allow and other one is for disallow? Also please give me example so that i can apply for my website.