Comments on: Character Encodings and Input

By: kanga

kanga — Wed, 26 Mar 2008 20:04:49 +0000

One of the techniques that works for just about everything regardless of encoding is the following:
[1] Copy the line with the error from the source file
[2] paste into notepad
[3] copy from notepad
[4] paste into source file and re-upload

notepad will strip the errors. I learned this on an emergency alert project I was working on: http://www.kangalert.com

By: Célio Santana

Célio Santana — Thu, 10 Nov 2005 13:37:16 +0000

I’ve a lot of problems with encodings i start to use utf-8 and the problems continues, there´s no effective way to get some data from my SQL server and
show to my users. I’m Brazilian and we use a lot of á, é, ç so these characters
aren´t correctly treated by PHP. I don´t know i used mbstrings and doesn´t work
too.

What should i do?

By: Anonymous

Anonymous — Wed, 19 Oct 2005 12:17:22 +0000

By: Mike

Mike — Wed, 31 Dec 1969 19:00:00 +0000

Huh? Never heard of iconv?

By: kaklz

kaklz — Wed, 31 Dec 1969 19:00:00 +0000

The answer is just plain simple - use UTF-8 and forget about the encodings.
There are countries out there, where you have to use more than one character set, even three of them or more.
One of them is my country, Latvia, where you have to work with latin, cyrillic and baltic character sets.
As soon as I started working with UTF-8, I don't have to care about the character sets anymore. So if you are using any other character set than latin, I would suggest you to move on to UTF-8.

By: BerislavLopac

BerislavLopac — Wed, 31 Dec 1969 19:00:00 +0000

I agree with kaklz above. Browsers use encoding specified in header to properly display HTML output and convert form entries, and databases store whatever comes from PHP — there is no need to convert strings internally. Even if you need to replace multibyte strings with other values, sprintf works like charm.

By: HarryF

HarryF — Wed, 31 Dec 1969 19:00:00 +0000

FYI, the WACT team is slowly putting together some resources on i18n;

http://wact.sourceforge.net/docs/doku.php?id=php:i18n

Also something on charsets, with the emphasis on UTF-8:
http://wact.sourceforge.net/docs/doku.php?id=php:i18n:charsets

Not complete yet and needs some heavy revision - very much in “note form” right now but getting there

By: Glen

Glen — Wed, 31 Dec 1969 19:00:00 +0000

I have found that most encoding problems can easily be by-passed by using UTF-8 encoding for all script generated HTML. i.e. In your , include the following:

That way, the client will render UTF-8 correctly, and as a bonus will send back UTF-8 data from forms.

By: mmj

mmj — Wed, 31 Dec 1969 19:00:00 +0000

That way, the client will render UTF-8 correctly, and as a bonus will send back UTF-8 data from forms.

Unfortunately, for many reasons, ranging from browser bugs to browsers blatantly ignoring the spec to users changing the character encoding setting in their browser, you cannot rely on submitted data to be in any particular character encoding. If you are expecting UTF-8 encoding and you receive something else, that something else may just break your CMS, or at least ensure that it will not validate as HTML.

Thus you will arrive at the very problem described in this blog post.

This article contains more information about why you can’t rely on input to be in any character encoding.
http://ppewww.ph.gla.ac.uk/~flavell/charset/form-i18n.html

By: mmj

mmj — Wed, 31 Dec 1969 19:00:00 +0000

Huh? Never heard of iconv?

Hi Mike,
I lumped iconv in with other third party libraries when I said "another third-party library". There is a PHP extension, however, which interfaces with it. The extension is an alternative to mbstring. Like mbstring, it is disabled by default in PHP and must be explictly enabled. Unfortunately, not all PHP users are able to take advantage of these extensions as they might not be able to compile or configure PHP themselves. However, if you can, then it would be a good solution.