I notice that if you have more than 1 link one some site, at first the crawler will count them all, but later it will remove the unnessesary links and leave only 1-2 links from a site. For example if you have a sitewide link on some site, first will count all the links, later remove some, but the result is the same anyway.
i have saw this with a couple of my sites, I useually track my backlinks in yahoo too. My rankings are staying pretty stable. I'm thinking maybe yahoo is making some changes?