Does a server physical location have any negative SEO affects for regional domains?

We host about 20 separate web applications and counting, each having atleast one unique top level domain name. Almost all of which are .com.au. We also manage our own DNS name servers. All on Australian soil.
A couple times I have suggested moving our Australian VPS instances to overseas providers namely in the US as they provide much better value for money. We have decent enough caching strategies in place that the increased latency won’t be a real issue.

However I’m often met with the response, “having our servers overseas affects SEO”. We’ve since had a major issue that will lead to infrastructure changes and I would like to suggest once again that we move our servers to the US.

My question is, is there any weight to this statement? And if so, are there any ways around this affect?

I would think the weight of .com.au would out rank the server’s locations as it is common place for servers to be located off shore.

Possible reverse proxying all requests through an Australian server? Just throwing ideas out there.

I can say for our sites which are .ca domains, having our server in Texas has had no effect on our SEO in Google.ca whatsoever. We rank on page one for our primary keywords for the various sites. All those sites have been SEO’d from within Canada too if that helps.