I kinda gave up on my James McAvoy site after I found out he might not be the next Hobbit actor. Interestingly it's starting to get hits and ranking on Google and Yahoo after a few months. Thought this was just natural though I haven't put any effort into it. But I look at the referring links and the top one is a porn site with sex toys in the URL and the others some site selling a variety of cigarettes and discount meds. These are all the referring sites I have listed for visitors coming to my site. Looks like a competitor might have been trying to blackhat me into oblivion but instead, is helping my SERPs... Guess i should counter this attack by building quality links from good sites.
No they wouldn't be able to harm your site just by linking to it from a shady neighborhood. The search engines don't want to start a webmaster war with everyone sabotaging each other. I have had other random sites show up in my referral history. Usually I cannot find their link anywhere on their site. I believe they are doing this simply to get listed in your website traffic statistic pages that may not be password protected. Try blocking any stat pages via robots.txt and add a password if possible. As for your rankings, perhaps you just got out of the sandbox and Google is showing some trust in your site now. It might not have anything to do with the weird links.
you can not be affected by any action that some one else can take. Now if you linked to inappropriate sites then you would see a negative effect.