Help please. Darn foreign chars.

[MarketJunction]

Hi,

I have a string with foreign chars in it. Looks like:

------
Ask many of his listeners what they think about his sermons and they’ll quickly respond with only words of acclamation. Follow that questions with a request for what the sermon was about and you’re met with only blank stares.
------

I have tried everything under the sun to remove the foreign chars, but no luck. The latest thing I tried was this function--and it does nothing.

function unaccent($text) {
  static $search, $replace;
  if (!$search) {
    $search = $replace = array();
    // Get the HTML entities table into an array
    $trans = get_html_translation_table(HTML_ENTITIES);
    // Go through the entity mappings one-by-one
    foreach ($trans as $literal => $entity) {
      // Make sure we don't process any other characters such as fractions, quotes etc:
      if (ord($literal) >= 192) {
        // Get the accented form of the letter
        $search[] = $literal;
        // Get e.g. 'E' from the string '&Eacute'
        $replace[] = "";//$entity[1];
      }
    }
  }
  return str_replace($search, $replace, $text);
} 

I don't know what to do now.

Any help appreciated.

Thanks

[sid egg]

echo ord("the character")

of each character you want to replace.
then check if ord('the character') = that number
if it does, replace it with a predefined replacement.


It's a bit of a hack, I'm sure there's a better way.

[MarketJunction]

What number are you referring to?

The string is about 4000 chars long. I tried looping per char and making them all match a-z 0-9 or a couple punctuations, but that failed.

[aamonkey]

here's something to try:

 $str = '~`!@#$%^&*()_+}|{":\?><,./\';\][=-and they’ll quickly respond with only words of acclamation. Follow that questions with a request for what the sermon was about and you’re';
  
  $allow_chars = '~`!@#$%^&*\(\)_\-\+=|\}\{\[\]"\'\:;\?\/><\,\.\\\\';
  
  $new_str = preg_replace("/[^\w\s" . $allow_chars . "]+/", "", $str);
  
  echo $new_str; 

just a side note--i was having a somewhat similar problem awhile back with people copying and pasting from microsoft word into a text input in a form on my site, which in turn got stored in the database. My validation did not include checking for these special unicode characters. Once in the (mysql) database, I tried everything I could to convert the special characters to display properly, but for some reason once they were stored in the table text column, the characters did not match up to any extended ASCII tables, and even doing a str_replace() COPYING AND PASTING the offending characters did not work. The only solution I could find (and which I should have done in the first place) was to get rid of or convert these characters BEFORE they reached the database.

Of course, if you're not storing this text in a DB, then ignore the last paragraph

[MarketJunction]

yea, I am storing in Mysql.

So you are saying that I should run a check on my post form and convert first? I was not aware it mattered either way.

In your example would [=-and they blah blah

be the same as:

[=-$string

[aamonkey]

So you are saying that I should run a check on my post form and convert first? I was not aware it mattered either way.

I wasn't aware of that either, and I still have not found any documentation on it...all I know is that I was able to replace the characters before they were entered in the database, but not after.

In your example would [=-and they blah blah

be the same as:

[=-$string

You can ignore that whole $str=......
I just put all the special characters in there so to illustrate that they would be left alone while the non-ASCII characters would be deleted. Just add or delete any special characters from the $allow_chars variable that you do/don't want to accept. Note that all Perl regular expression special characters are excaped with a backslash.

[MarketJunction]

Ok thanks. Think I will try alter before and screw everything else I have. Too many hours wasted.

I was able to get out word stuff, but the foreign chars won't budge.

Oh well.

[paulyG]

@seamonkey

Ditto - on the lost sanity and time. Came to the same conclusion as you.

theres a nice function strictify() on the user notes in the man on www.php.net/chr

That deals with most Word bad chars like curly quotes - but theres lots more really...