Naturally if you want to target a specific location like Australia, you should use Australian hosting to increase the website's speed. But what about if you're just wanting to target the world. Would you consider hosting on servers outside of America if America doesn't make up a large portion of the traffic base?
Basically I select hosting server where my maximum number of visitors exists. In your case, you need to find first what part of world constitutes maximum number of visitors; based on that select your hosting server.
A good stable USA host is ok for worldwide. But even better for worldwide traffic is a good and stable host in The Netherlands, Amsterdam in most cases. Laws there are similar to USA, it is a good choice for hosting a website.
Website visitors from Asia or Australia, by my experience, have better server response from USA than from Europe. Hence, for a website with worldwide traffic, the best website hosting server location is north USA, on east coast preferably.
The solution to your problem is to use a CDN (content delivery network) + a caching solution. A CDN has web servers across the world and will deliver your site's static content from the server that's closest to the location of the visitor. A content delivery network can only deliver static content (images, CSS files, JavaScript, HTML, Fonts, etc.) and that's the reason why you'll also need a way of caching your site's content (turning your pages from dynamic pages into static pages). Then, the cached pages can also be delivered by the CDN. If you use WordPress, it's effortless to cache the content of your site and integrate a content delivery network with the help of a plugin. I use W3 Total Cache for my WordPress site and is very good. In the case you use another CMS (like Joomla, Drupal, etc.), I'm sure you will find an alternative for W3 Total Cache.