It is allowed to add the anchor text of link exchange partner in sitemap page? Does google evaluate it as backlinks of your site?
Using your sitemap for link exchange seems odd. The sitemap ought to list URLs on your site or use footer links... below the nav links...standard templated links. Googles XML format does not allow links to other domains at all, but I guess it is possible on the HTML sitemap.