I recently popped up towards the top of Google for a certain phrase, which triggered the site owner in slot number one to email me asking for a link exchange. He said he had over 7 million visitors in the last 3 months, so I popped over to his site, and there he had an image from Alexa proclaiming his site rank to be 7 million... Searching Google, I found a post on a BBS from three years ago where he made the same ludicrous claim. I'm in the process of writing a Squidoo page and an article to permanently bump him off the number one spot
Them sort of link exchanges are gold. Your missing a valuable point here, you want YOUR website associated with the top ranking sites for your keywords, that is a good thing.