Hosting that have often of problem will reduce the number of visitor, So if the New Hosting provider is better, I think that's will Increase your Visitor
Yes hosting can affect you site traffic...its depend where you wants to host your site. If your targeted visitors is in the same region where you wants to host then this will not affect the site traffic. For exp: Your website targeted keyword is "football" and the site is hosted in USA. You will get the traffic for football but here football means rugby. So your site will rank on rugby related keywords. If you change the hosting to UK or India..then here Football means Soccer. This will cause a huge traffic loss or unwanted visitors. Hope this will help you understand the importance of hosting!!!!!
Move to better host, it will compensate your loss (you inured while shifting) in near future. Don't worry about short term loss while keeping eye on long term gain with better host.