Well, I think a lot of it was buying/selling links. Directories tended to be worse offenders than most, and they were widely known as places to get links, so G took steps. We haven't heard much about directories that are squeeky clean getting hit. At least not me.
Right, I'm aware of that one. I don't deny that Google is 'keeping an eye' on directories, I just don't think there's some sort of auto-penatly if you have one. The JoeAnt slap appeared very manual.