How about if you have many links from same IP address (hosting provider) as your site? It that trouble?
I don't think unrelated links will hurt your site in anyway. If that was the case one could easily damage its competitor site.
My understanding is that it is possible to do this by suddenly having loads of links from poor sites. Matt Cutts said that its unlikely but not impossible. But this is a side issue from the original question. In answer to the original question - those links from less relevant (but not bad neigbourhood) sites *will* increase your PR (which is keyword blind) but not your SERPs ranking