In my experience, PHP is very fast, so I don't see a problem in using it ... as long as it's not over used. The hello world example probably is overkill. I mainly use PHP for "includes", where common information like the HTML for your header, footer and menu is stored in one place and pulled onto each page. Then you can modify them in one place and not have to update each page.
for things that you can do in either HTML or PHP, is it preferable to use HTML in such cases instead of PHP? And if so, would this reason be because using HTML rather than PHP would be faster for the visitor as they won't have to wait for processing from the server, as well as being easier on the server?