Dont panic, just work your way through your back links, any very poor links from pages with no page rank, you should try and remove the links, ask nicely. Also check your content, is it spammy?, if so sort that out too.