I am one of those negatively affected by Google's changes in the last few days. Look at the following example and see if you agree with me that there is a new level of Google "search what I mean, not what I type". My site is a limousine service site in a Big City. My site ranks #2 for "Big City Limousine". A competitor ranks #1. A secondary phrase that my site is tuned for is "Big City Car Service", a term used in East Coast cities to describe black car services that are taxi like. Last week we ranked #1 for "Big City Car Service". After the weekend my limousine competitor ranked # 1 for "Big City Car Service and I slid to # 8. Here's the catch. The phrase "car service" never appears on the competitior's home page or anywhere in his site. Google has made the connection that people looking for a car service are looking for a limousine service and has ranked a more popular limousine site over one that focuses on car service.
Google doesn't report all the backlinks so they still may be using this anchor text. The other possibility is Google's latent semantic indexing which uses semantics or meaning instead of just syntax. I thought this has been implemented for a while but I could be wrong. Check the cache of your competitor's site in the SERP and you might see it say at the top that the search phrase only appears in links pointing to the page. If you could post the domains, I'd be interested in checking it out.