Pretty easy: 1. Check if the site is penalized in Google or not by typing site's URL (example: siteurl.com) 2. Check if the linking site is new. 3. Check the backlinks of that website. If the majority of backlinks are low quality/spammy backlinks, then your link is a spam. 4. Check external links of the site it is giving to other sites, if the majority of external links are to new sites, then spam.
I personally use a paid tool which one of their features shows the risk factor of each link. They do have a free trial though, its called SEO Powersuite and its the SEO Spyglass part that gathers back-links and shows you the risk level per link. if its high risk you should check high risk links yourself to see if the source looks spammy. combine the above answer from oahson with your gut from viewing the back-link then take action. Remember if it looks fishy it probably is.