Hi all, I have an issue with my robot, can anyone help? For a short period my robots.txt file instructed search engines not to index my site. Later i changed this so search engines would be able to index my site, but no spiders have been back?! Will the search bots ever come back and update my robots.txt file instructions?
If you have told the search engine spiders to not index your site, they would be in no rush to come back and check if you changed those instructions. You may need to help the search engines find your site again. This can be done by getting some backlinks from quality sites. Try commenting on a few good blogs or do some social bookmarking. That should help get your site indexed faster. Another thing you can do is submit a sitemap to the major search engines. You can even specify your sitemap in your robots.txt file.
Cheers for that vansterdam. I have already done that and google has been back, however one page still isn't indexed. No doubt it'll be the finish of the job in a week or 2. Cheers, Cookie.