Archive.org

Discussion in 'Products & Tools' started by Neil McRae, Jun 26, 2011.

  1. #1
    Does anyone know any tricks for getting a more recent entry to appear for a site that hasn't been crawled since 2006? It seems there are a lot of websites out there that were crawled more frequently up until then. I contacted IA asking them if they take payments to update a website's entry. They wrote back saying that if the site is well linked it is likely to be crawled again and they do not charge for their services. I don't currently have a robots.txt file in the root directory on my homepage. Do I need to have one or could I possibly have something else enaled that might be preventing crawlers from retreiving copies of my website?

    Thanks for any information you are able to give me about this.

    -Neil
     
    Neil McRae, Jun 26, 2011 IP