I think if you link to 'bad' websites that can harm your ranking. If they link to you I don't think there is a problem as people who link to you are not in your control but all your outbound links certainly are.
not sure if incomming links will affect though but i've read somewhere were they said outbound links may cause harm to your site especially if you have adult links in you site that has adsense in it.