I think you should wait it out, but, in meantime, become very aggressive in providing unique relative phrases and easy to read content for your viewers so that you'll see a rise in popularity once again.
Agreed off page links are best answer but try to get page rank to deep pages and get the rank to circulate through site with good structure...
Hi Warrior. Google is notorious for penalizing honest, hard working business owners and they say it's to keep the browsing experience good for their customers. Personally, I wonder if it's more so they can force people to invest in PPC advertising so they can stay on page one of their results. I know that may sound a bit like a conspiracy theory, but I really wouldn't put it past them. As far as what you can do, there are a few things that may help. Try to get listed in some high PR authority sites. I know it takes a while to do this, but it's worth the effort. If you do a search for directories in your niche, you will find quite a few. You can also do a search to find out which of them do follow. Make sure you don't build all the links too fast, otherwise Google will penalize you for that too. You can also post comments at some high PR blogs in your niche and even start a free blog of your own at Wordpress or Blogger.com. Google seems to like them. A Squidoo lens wouldn't hurt and some social bookmarking to round things out. I know it all seems like a lot of work, but if you plan the work out you can get it done within a reasonable amount of time. There is also software available to help with many of the more mundane tasks. I hope this helps some. Best wishes.