Am I Being Penalized for SEO via Yahoo?

I have a domain that I've not really done much with until lately. It was at the very top of the SERPs in Yahoo for its keyword until yesterday. Now, I can't find it anywhere on the first two pages.

There are only a few things I'd done to the site. I added a CMFADs unit. Could this have killed my SEO? I've since removed it, but nothing's changed (it may be a little too soon.)
I also got a few back links from various blogs.

This isn't a very old domain and had been at the top since its inception about 6 months ago and now it's SEO effort has been pretty much killed.

It never was high in Google's search results.

 

Google keeps on moving things up and down. I had a site on the top of keywords now its on 11th position for the same keyword. But again those CMFAD's Units must have caused it

 

I know how to SEO for Google, but Yahoo is the problem in this case.

 

Im not familiar with CMFADs. But if they sell seo text links in their ads this may have hurt you with yahoo.

 

I hear that Yahoo has a paid site submit thing. Would this be of any help to me?

 

I doubt that being added to the Yahoo directory will do much for you...its reallt expensive and I never really saw much benefit from it.