Maybe if your site is down for 1-2 weeks, it would. My site was down for about 2 weeks, and it disappeared from the SERPS (#1 position). It's because the domain expired, I forgot to renew it. A little over a week later I got my #1 position back. In some cases, just getting your site back up and running is enough to get it back to its old position in the SERPS.
The site was initially down for 24 hours but the host then suffered a DDOS attack (from the same hacker) and it was out for a further 3 or 4 hours. Rankings are still ok so far, touch wood.