how much damage to it's value have we done by not having a site up for this domain? what do you think we'd get for it?
I don't think any damage happens to a domain by not having a site on it. Damage happens when you put up a site, find that its not working, then switch the niche and put up a completely different site on it. This is when domain starts losing credibility in SEO. This is not your case. Your domain was unused, so its still a good domain with age factor behind it! Domain age might be able to get you a higher price, but not sure how much.