Perhaps I wasn’t being clear about what this meant, so let me to illustrate with a concrete example. Assume that your legacy, latin1-only application uses the superglobals (Eg. $_GET, $_POST etc.) directly. To assure BC, you would have to stick something like this to the top of your script:

$_POST = array_map(’utf8_decode’, $_POST);

Obviously the above implementation is naïve, but I hope it conveys my point.

Now, since we’re adding a piece of UTF-8 aware code into the application, we need this to be able to retrieve the raw, un-decoded input in those places. So let’s add this:

$GLOBALS['POST_UTF8'] = $_POST;
$_POST = array_map(’utf8_decode’, $_POST);

Now, the UTF-8 aware code can use $GLOBALS['POST_UTF8'], while still keeping full BC with the legacy code (Since $_POST will only contain latin1).

Converting UTF-8 to Latin-1 means that you loose all chars not in Latin-1. That way you’re not able to handle cyrillic, greek, chinese or … chars.

I’m not sure if this comment was meant for me, but in that case I think you missed the whole point. The idea of embedding utf-8 encoded strings within a sea of latin1, is exactly to preserve the full range of unicode, that these strings have. There is no conversion to latin1 in this recipe.

By using accept-charset=”utf-8″ you can instruct browsers to send the data encoded with UTF-8. As far as I know, browser support for this is quite decent.

Yes. In this case, it’s redundant though, since browsers are also pretty consistent in sending back in the same encoding as they receive. It wouldn’t hurt though, and it’s better to be safe than sorry.

You just should ensure you use the same encoding throughout your application (mind the backend connections) and there won’t be any real problems.

You’re absolutely right. The premise however, is that I have a legacy application in latin1, and now need to use utf-8. Porting the entire application to utf-8 is a major undertaking, so it’s out of the question. I suspect that a lot of people are in a similar situation. What I described, is a technique for coping with an imperfect world.

You should not try to parse such potentially recursive markup with regular expressions – and even using it in examples encourages people to follow this example.

Good point. If someone actually uses the delimiter as part of data, then the parser would choke on it. This risk can be reduced by chosing a more unique delimiter, but it can never be solved completely. With a sufficiently unique identifier, the risk is very low, so I think I’ll be bold and brush this off as an academic issue; Thanks for pointing it out though. If the problem does arise, there is a single place in the application, where the delimiter can be changed to something better than charset:utf8, which – arguably – isn’t very unique.

I suggest using UTF-8 internally. If you’re fetching data from old parts of the application that still use Latin-1 you can convert them to UTF-8 without loosing informations.

On input you can check for the used charset and convert this to UTF-8, too.

An encoding maps the characters of a specific character set to a sequence of bytes. Of course strings in PHP (<6) may contain strings encoded with each encoding and each character set. But there is no character set associated, and all string functions in PHP (<6) just operate on bytes – which actually do not much differ from characters in single-byte encodings. But there is *no* information if it is Latin1 or ISO-8859-*, or similar.

Strings in PHP do not have any charset or encoding information associated. They are just binary, like described here.

LOL. The binary representation of characters is exactly what an encoding is. The problem with PHP is that the string functions in the standard library assumes one byte per character. There are multi-byte string functions available, but then you have to choose which encoding to use.

Java and JavaScript, on the other hand, internally use 16 bits to represent each character. That means they can at least handle the BMP (basic multilingual plane) in Unicode.

a) Strings in PHP do not have any charset or encoding information associated. They are just binary, like described here.

b) Why do you want to convert to Latin1 anyways? It might only be relevant, if you need to process strings character-wise, what should not be necessary in “normal” applications. If you need to do so, take a look here.

c) For charset and encoding conversions you might want to use the iconv() functions. It does not only handle more encodings, but can also handle character set incompatibilities between encondings (transliteration, ignore).

d) The charsets/encodings browsers send can be influenced either by the encoding information given in the Content-Type headers (HTTP, HTML-meta-tags) or the form attribute already mentioned in another comment. Not mentioning, that all clients of course may send garbage which needs to be sanitized.

e) You should not try to parse such potentially recursive markup with regular expressions – and even using it in examples encourages people to follow this example. This will never work.

You just should ensure you use the same encoding throughout your application (mind the backend connections) and there won’t be any real problems.

You may or may not know this, but when submitting a form, browsers send back data in the same encoding as the page was served.

This is the default behaviour if the accept-charset attribute is omitted from the <form> tag. By using accept-charset="utf-8" you can instruct browsers to send the data encoded with UTF-8. As far as I know, browser support for this is quite decent.