Directory submission & article distribution can not count in spamming because they are not approving at a time. When they approve and will come under google cache at same day these number of resources then it may come under any kind of penalization. If google did not maintain database for your submission then how can you think your work is under consideration of penalization. When this cache then your work have some worth and can be decided some thing.
Too many backlinks in a month or two is the main cause of your drop down in ranking. wait for sometime as everything will take time to resolve.