No! Dont confuse that with downtime, AFA speed goes, it will not affect the SERPS.. However, if things go the timeout way, cralwer re-schedules itself for another visit at a later stage, you should figure out the reason and correct it.
You may benefit if your page loads quickly though. A small page size will mean you can be read faster
Server Speed is not a factor as regards positioning, however slow response times will affect the User experience, and will cause them in most cases to go elsewhere.