using robots.txt during development

[dshah]

I am working on my site and its not READY for the whole world yet. Is it good idea to block search engines spider the in-dev site using robots.txt

During dev, lot of URLs may change often and I don't want search engines hitting on those URL later resulting in 404s

Is it okay to have robots.txt block search engine spiders, but later remove robots.txt. Does blocking spiders using robots.txt has any long term ill-effects (like permenantly marked not be indexed, crawled etc)

any inputs are welcome.

[minstrel]

I'm really not certain how long a spider block normally lasts although in another thread (about WebMasterWorld blocking Googlebot) it was suggested that normally the time period is 6 months. Unless your site is going to be in development for 6 months, that's a heavy price to pay for a few broken links and under construction pages.

[dshah]

whoa, 6 months is AGES. Seems that I surely can't pay such price. I have to locally develop the whole site and put it on at once, or at least have phased release.

Thanks for your inputs