I read that a big benefit to the site's popularity bring the links relevant to your website, from the same domain of activity. It is clear that a site which sell shoes is not relevant with a jokes site, they are from different domains. But the site which sell shoes might be somehow related with sites which sell clothes. Both sites have sells but the objects are differently. You may think that site relates, but in other opinions might not. Let's say you have a proposal to swap links with a site which is not quite from the same domain like your site; how would you resolve this problem? Take this decision to how would it be better for the end user and not for big search engines; the popularity comes most of visitors, if they like what they read they will come again and will recommend your site to others. But not guide your link exchange process after this rule, pay attention to seo rules to
When Google goes to each site they spider all the content. They will know the relevance or "commonality" between sites and keywords.
Exchange link with the website like yours, google will give a site some keywords to marked, so as yours. It's like the related articles with tags.
I believe there is degree of relevancy and maybe Google is measuring them, too. Your scenario could be clothes site & shoes site are in the same "fashion & apparel" category, while clothes site & coats site are much related.
i believe its based on the content of the site... google can only index the the text content of a site which is believe the only way to recognize its niche..