As stated - nobody has any control over who links to their website. It would therefore be inappropriate for SEs to penalise sites for any of the inbound links that they might have. If, however, they were to reciprocate and link to a known 'bad neighbourhood' that can definitely have a detrimental effect on a sites potential ranking.
Yeah, view as of most of you. You cant control who links to you! So google arent going to punish you because someone running a dodgey site links to you!
FYI i know someone who 3 years ago hired a link builder to throw 3k of assorted links into his website. The muppet link builder built them over a week, instead of the contracted 2 months, and the website dropped from decent rankins to round about page 8 for all important terms. Recently as matt cutts announced some old penalties were been removed, the same key terms jumped from the same page 8-11 upto page 1-3. Links do have the potential to harm your website
Too many, too fast is harmful, but otherwise I think the more the better. Check the top ranking sites, you will find many irrellevent links pointing to those sites, but they have a high ranking because they have many many in-bound links.
IMHO no, inbound links will not hurt your SEO. However make sure you are getting you links from a variety of sources - if something goes wrong with one of your sources then you may loose the link benefit you were getting and that will hurt you (not penalise you, but you will loose PR etc.). Your competitor could use automated link building to hurt your reputation (giving bad answers etc.) or by optimizing you for negative keywords (also known as Google Bombing). If this happens to you consider contacting Google.
Also, another thing that might happen if they constantly bombard you with low quality links is that Google will discredit/sandbox some of the real links for longer if they are low quality -I have no proof of this, it's just a theory. The solution to this is to make sure you are getting strong links from some other source.