You could actually drop hundreds of search positions without even knowing what you've done wrong! That is why it's very important to keep a close eye on who you're linking to. If you a do a link exchange with a website, check back to that website every once in awhile to make sure that they haven't been "gray barred" by Google. This is one of the top reasons that people get penalized by Google, and the sad part is, they have no idea what they've done wrong because they haven't actually CHANGED anything on their website! Any easy way to check on your link partners is to use the tool. You can actually use SEO Elite to see if you are linking out to any bad neighbors. If you are, then delete their link immediately and I can almost guarantee you will see an increase in your ranking.