I think it depends highly on bounce rate and time spend on site for a particular keyword. It is possible that a site starts ranking on some particular search terms, but when people come to the site they tend to bounce back without finding the site useful. Therefore google drops ranking.