using robots.txt during development

Discussion in 'robots.txt' started by dshah, Dec 15, 2005.

0
  1. #1
    I am working on my site and its not READY for the whole world yet. Is it good idea to block search engines spider the in-dev site using robots.txt

    During dev, lot of URLs may change often and I don't want search engines hitting on those URL later resulting in 404s

    Is it okay to have robots.txt block search engine spiders, but later remove robots.txt. Does blocking spiders using robots.txt has any long term ill-effects (like permenantly marked not be indexed, crawled etc)

    any inputs are welcome.
     
    dshah, Dec 15, 2005 IP
  2. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #2
    I'm really not certain how long a spider block normally lasts although in another thread (about WebMasterWorld blocking Googlebot) it was suggested that normally the time period is 6 months. Unless your site is going to be in development for 6 months, that's a heavy price to pay for a few broken links and under construction pages.
     
    minstrel, Dec 17, 2005 IP
    dshah likes this.
  3. dshah

    dshah Well-Known Member

    Messages:
    1,840
    Likes Received:
    69
    Best Answers:
    0
    Trophy Points:
    115
    #3
    whoa, 6 months is AGES. Seems that I surely can't pay such price. I have to locally develop the whole site and put it on at once, or at least have phased release.

    Thanks for your inputs
     
    dshah, Dec 18, 2005 IP