As Paul said, you'll have to make the "new" elements display:block. Firefox 4 will support HTML5 as far as, it will know that things like <header> are blocks, so you won't need to do anything special for them then. Similarly, Opera will surely start supporting as well... maybe when the specs are more stable. Remember what happened when IE jumped the gun when the CSS specs were unfinished?
Anyway FF and Opera just see elements they don't recognise, and set them to display: inline by default.
Any back-end script you write to serve pages based on user agent will run into the problem of browsers lying. Browsers can have their user agent strings set by the users to anything else, and Opera is set to IE by default (it also incorrectly calls itself Opera 9.8 even when it's Opera10, because people write crappy browser sniffers that think 10 == 1 (because the sniffers did not expect 2-digit version numbers!).
Second, you'll have the problem of updating your list. As FF 4 comes out, for example, you won't want to give that browser the HTML4 version. So now your script will be complicated as it will have to have these separate lists of browsers and every request, the script will have to read the user agent string, check a list you've made, and then tell the server what to do. Will make your pages slower.