I also had this problem, I don't think you should be worried. I actually lost links to my site that were giving it a boost in serps with anchor text and my site actually went up in serps for some reason??