I setup a new site. week later was looking at 4-5 pages indexed, then got a notice that google could not spider my robots.txt and the gbot spider would no longer be visting.... was not long until those 4-5 pages vanished, gone like the wind. Now that I got the spider issue fixed, its started to index again... It really sounds odd if gbot cant spider your site then it pretty much just wipes out what has already been indexed..... and I still have not seen gbot for most of the day.... does gbot not visit like it used too... What do you guys think?
What actually happened to your robots.txt then? Google does not have a problem if there is no robots.txt but they may consider a site with an 'invalid' robots.txt as not useful and remove it from their indexes. It's just a guess. I never had an issue like that.
Does not appear to be invalid, google says it is good. When I have the robots.txt in place, google simply says that is unable to fetch the robots.txt then says it is stopping spiders. When I remove the robots.txt then google says it is 404 (of course, I removed it) at this point, google proceeds to spider any and everything as there are no rules in place. So far have not found a solution other then to just remove the robots.txt and let google spider up the place....
Hi, maybe at the time gbot went to your site, the server was down. I have received a similar notice when my site was down for part of the day when I neglected to pay my host. Not saying you didnt pay your hosting bill, I'm just saying that there may have been a temporary outage at your host right at the time the crawler went through.
It's likely he didn't pay the bill, his site is suspended, assuming the problem website is that of his signature link.
this happened to my site,i have ranked the site to top 108049 thousand in 7weeks, suddenly after4month,my rank dropped to a million plus,before then,something happened,my useless host ipage shut one of my site claiming it has malware of which it never did which affected others,so server was down coz i hosted all site on one host.i have to work hard now to get things in other but google index wasnt like before anymore on my site,ranking is getting back up but very slow.advice is use reliable hosting and avoid domain expiring before renewal,also dont host all site or important website on one hosting plan.
It could actually take a few months to see results. Make sure that you are consistently promoting your campaign by honoring search engine rules.