I think backlinks are the most important factor for google when it comes to SERPs. Backlinks will improve PR and SERPs.
Based on my experience on last month, I have builded link from many high PR site but the topic is various and does not relevant to my site. Google throw my keyword on SERP from the 1st and 2nd page. I realize that and remove all non-relevant link from my site then 3 days later my SERP was back. So, my conclusion is that backlink is absolutely impact to SERP and the most important is unique content.
Ya same thing happened to my site, I have a good serp for my key words(1 st or 2nd place), but in the last few days I got some link back,and I exchanged link with some related sites. But now my serp is just drooping to (5 or 6).dont know what is the reason behind this!!