I think your site is affected by the sandbox effect. One of my sites have the same problem like yours. But the sandbox effect is very annoying because even if my site has backlinks double than the site on #3 it's ranking very very low. (and yes, i have the correct anchor tag, no bad neighbours, no nothing... it's no black hat)
it's not quite as simple as saying backlinks may be to blame, it could be for a variety of reasons. But you do need to ensure that you are not buying backlinks as this can hurt your rankings
I would like to know how backlinks paid or otherwise could hurt your ranking? wouldn't companies hire people do do bad SEO on their competitors? It does not make sense logically that backlinks could cause harm... maybe its one of the other 100 factors used to determine rank that is hurting you? Just a thought
If you concentrate on creating good anchor back links then your search engine ranking also will be in a better position.