What do you mean text links? Like in signatures on forums and such then i say no. Also how far back did you go. If it's a couple spots don't sweat it.
From what I understand is that Google only gives you points for what it conceder as good links but will not penalize you for any links
if too many links to your site appear quickly all containing the same anchor text and description then the links can be seen as not natural. It's easy for google to notice such links eg My Shop - (links to site) Come to my shop and buy loads of cheap stuff and products online. Now say this link appears on a few hundred webpages within a few weeks, then obviously the link is not natural, obviously it has been somehow placed by the website owner, either bought links, reciprocal links manager, mass submission, automated etc...so google can come to whatever conclusion it wants to ! But it doesn't take a genuis to see the link is not natural, if it was natural then each link would be unique. Another possiblilty is that google has earmarked certain sites and directories that are full of links to bad websites, many directories don't care who they link to, they list anyone, so their directory is flagged as being of low quality..if your links appear on lots of low quality sites then this can also affect your sites ranking. This is usually the case for websites losing ranking....lazy webmasters taking shortcuts...like using a directory submission service, you get what you pay for..no infact you get worse,, hundreds of links on bad directories all using the same anchor text and description .....you are trying to get links to get ranked and you end up losing ranking.
I think by text links you mean the keyword ranking of your website on Google. Yes, it's possible to go down on ranking due to growing competition and your competitors would have got more backlinks than your website so that's the reason for shift down of your ranking