Hi, I made some posts on my blog recently and went to do a search but I wasn't coming up for any terms. Then I tried a load of old terms that I was number 1/2 for in google, and theres no sign of my site at all. I went to my Google Webmasters account and under the Crawler Access section it said the following: Does this mean that the robots.txt file was preventing the spider from accessing my site? I clicked on the "Generate robots.txt" tab and downloaded my file which contains: User-agent: * Allow: / And I uploaded it to the root directory of my site. However on Google it still says that the last downloaded file contains the original: User-agent: * Disallow: I would really really appreciate any help. Any idea whats caused my site to disappear from Google for a load of search terms? Is it the robots.txt file? And there was no robots.txt file on my site (when I uploaded one earlier I wasn't over-writing anything), so where was Google getting the original one from? Would Wordpress have an option for this? Thanks for any help offered...
just keep only one robots.txt file and it contents are:- User-agent: * Disallow: Sitemap: http://www.yourdomainname.com/sitemap.xml save it with the only name robots(not with robots.txt) and remove all other robots.txt files in your folder and submit it. hope this helps!! thanks
It has nothing to do with your robots.txt. Your robots.txt is ok. Might be your new posts have not been indexed yet by Google. Trying to make backlinks for these posts and Google will index it as soon as finds the backlinks.