I was working on a website and i had all kind of stupid things in there,like everybody uploades when they are working directly on the FTP,as I like to do. I used robots.txt,a file placed in the root of the FTP,here is the exact content of my robots.txt : What did this happen?this is really sad
Yous should have used User-agent: Googlebot Disallow: / # This website is temporary I believe, as you need to specify who exactly your are disallowing.
The original commnad would normally block Google as well. Google no longer reads the robots.txt file every time they visit. They only read it periodically. It's in one of Matt Cutts' blogs. None of the search engines have been real good at obeying the robots.txt file. I've seen all three of the big guys blow right past blocked subdirectories. The noindex meta tag on each page tends to work better if you want to block spiders. Google seems to be getting better at hitting new sites rather quickly. They started to index a couple of my sites with in a day or two of setting them up and before I set up any links to the site. That's not always a good thing, because I've also seen Google hit an incomplete site and then toss it into supplemental right away. Registrars must have a way of knowing when DNS changes are made to new sites. I suspect that triggers a spider visit. Google is a registrar. Either that, or they are tapped into the propagation updates for the DNS servers.
Okay,this is not good,i wasn't expecting this.So let's say that my website was about 25% complete,and i just abandonedit because it's a very big website and I can't do it right now because of personal issues and I want to get back to it in 2 months.My hosting will expire so there will not even be a website for this.what are the implications of this,how bad is this situation exacly?