iconv
If the string //TRANSLIT is appended to to_encoding , then transliteration is activated. This means that when a character can’t be represented in the target charset, it may be approximated through one or several similarly looking characters. If the string //IGNORE is appended, characters that cannot be represented in the target charset are silently discarded. Otherwise, E_NOTICE is generated and the function will return false .
If and how //TRANSLIT works exactly depends on the system’s iconv() implementation (cf. ICONV_IMPL ). Some implementations are known to ignore //TRANSLIT , so the conversion is likely to fail for characters which are illegal for the to_encoding .
The string to be converted.
Return Values
Returns the converted string, or false on failure.
Examples
Example #1 iconv() example
echo ‘Original : ‘ , $text , PHP_EOL ;
echo ‘TRANSLIT : ‘ , iconv ( «UTF-8» , «ISO-8859-1//TRANSLIT» , $text ), PHP_EOL ;
echo ‘IGNORE : ‘ , iconv ( «UTF-8» , «ISO-8859-1//IGNORE» , $text ), PHP_EOL ;
echo ‘Plain : ‘ , iconv ( «UTF-8» , «ISO-8859-1» , $text ), PHP_EOL ;
The above example will output something similar to:
Original : This is the Euro symbol '€'. TRANSLIT : This is the Euro symbol 'EUR'. IGNORE : This is the Euro symbol ''. Plain : Notice: iconv(): Detected an illegal character in input string in .\iconv-example.php on line 7
Notes
Note:
The character encodings and options available depend on the installed implementation of iconv. If the argument to from_encoding or to_encoding is not supported on the current system, false will be returned.
See Also
- mb_convert_encoding() — Convert a string from one character encoding to another
- UConverter::transcode() — Convert a string from one character encoding to another
User Contributed Notes 39 notes
Please note that iconv(‘UTF-8’, ‘ASCII//TRANSLIT’, . ) doesn’t work properly when locale category LC_CTYPE is set to C or POSIX. You must choose another locale otherwise all non-ASCII characters will be replaced with question marks. This is at least true with glibc 2.5.
Example:
setlocale ( LC_CTYPE , ‘POSIX’ );
echo iconv ( ‘UTF-8’ , ‘ASCII//TRANSLIT’ , «Žluťoučký kůň\n» );
// ?lu?ou?k? k??
setlocale ( LC_CTYPE , ‘cs_CZ’ );
echo iconv ( ‘UTF-8’ , ‘ASCII//TRANSLIT’ , «Žluťoučký kůň\n» );
// Zlutoucky kun
?>
The «//ignore» option doesn’t work with recent versions of the iconv library. So if you’re having trouble with that option, you aren’t alone.
That means you can’t currently use this function to filter invalid characters. Instead it silently fails and returns an empty string (or you’ll get a notice but only if you have E_NOTICE enabled).
This has been a known bug with a known solution for at least since 2009 years but no one seems to be willing to fix it (PHP must pass the -c option to iconv). It’s still broken as of the latest release 5.4.3.
[UPDATE 15-JUN-2012]Here’s a workaround.
ini_set(‘mbstring.substitute_character’, «none»);
$text= mb_convert_encoding($text, ‘UTF-8’, ‘UTF-8’);
That will strip invalid characters from UTF-8 strings (so that you can insert it into a database, etc.). Instead of «none» you can also use the value 32 if you want it to insert spaces in place of the invalid characters.
Interestingly, setting different target locales results in different, yet appropriate, transliterations. For example:
//some German
$utf8_sentence = ‘Weiß, Goldmann, Göbel, Weiss, Göthe, Goethe und Götz’ ;
//UK
setlocale ( LC_ALL , ‘en_GB’ );
//transliterate
$trans_sentence = iconv ( ‘UTF-8’ , ‘ASCII//TRANSLIT’ , $utf8_sentence );
//gives [Weiss, Goldmann, Gobel, Weiss, Gothe, Goethe und Gotz]//which is our original string flattened into 7-bit ASCII as
//an English speaker would do it (ie. simply remove the umlauts)
echo $trans_sentence . PHP_EOL ;
//Germany
setlocale ( LC_ALL , ‘de_DE’ );
$trans_sentence = iconv ( ‘UTF-8’ , ‘ASCII//TRANSLIT’ , $utf8_sentence );
//gives [Weiss, Goldmann, Goebel, Weiss, Goethe, Goethe und Goetz]//which is exactly how a German would transliterate those
//umlauted characters if forced to use 7-bit ASCII!
//(because really ä = ae, ö = oe and ü = ue)
echo $trans_sentence . PHP_EOL ;
to test different combinations of convertions between charsets (when we don’t know the source charset and what is the convenient destination charset) this is an example :
$tab = array( «UTF-8» , «ASCII» , «Windows-1252» , «ISO-8859-15» , «ISO-8859-1» , «ISO-8859-6» , «CP1256» );
$chain = «» ;
foreach ( $tab as $i )
<
foreach ( $tab as $j )
<
$chain .= » $i$j » . iconv ( $i , $j , » $my_string » );
>
>
echo $chain ;
?>
then after displaying, you use the $i$j that shows good displaying.
NB: you can add other charsets to $tab to test other cases.
Like many other people, I have encountered massive problems when using iconv() to convert between encodings (from UTF-8 to ISO-8859-15 in my case), especially on large strings.
The main problem here is that when your string contains illegal UTF-8 characters, there is no really straight forward way to handle those. iconv() simply (and silently!) terminates the string when encountering the problematic characters (also if using //IGNORE), returning a clipped string. The
$newstring = html_entity_decode ( htmlentities ( $oldstring , ENT_QUOTES , ‘UTF-8’ ), ENT_QUOTES , ‘ISO-8859-15’ );
?>
workaround suggested here and elsewhere will also break when encountering illegal characters, at least dropping a useful note («htmlentities(): Invalid multibyte sequence in argument in. «)
I have found a lot of hints, suggestions and alternative methods (it’s scary and in my opinion no good sign how many ways PHP natively provides to convert the encoding of strings), but none of them really worked, except for this one:
$newstring = mb_convert_encoding ( $oldstring , ‘ISO-8859-15’ , ‘UTF-8’ );
If you are getting question-marks in your iconv output when transliterating, be sure to ‘setlocale’ to something your system supports.
Some PHP CMS’s will default setlocale to ‘C’, this can be a problem.
use the «locale» command to find out a list..
setlocale ( LC_CTYPE , ‘en_AU.utf8’ );
$str = iconv ( ‘UTF-8’ , ‘ASCII//TRANSLIT’ , «Côte d’Ivoire» );
?>
Here is how to convert UCS-2 numbers to UTF-8 numbers in hex:
function ucs2toutf8 ( $str )
<
for ( $i = 0 ; $i < strlen ( $str ); $i += 4 )
<
$substring1 = $str [ $i ]. $str [ $i + 1 ];
$substring2 = $str [ $i + 2 ]. $str [ $i + 3 ];
if ( $substring1 == «00» )
<
$byte1 = «» ;
$byte2 = $substring2 ;
>
else
<
$substring = $substring1 . $substring2 ;
$byte1 = dechex ( 192 +( hexdec ( $substring )/ 64 ));
$byte2 = dechex ( 128 +( hexdec ( $substring )% 64 ));
>
$utf8 .= $byte1 . $byte2 ;
>
return $utf8 ;
>
echo strtoupper ( ucs2toutf8 ( «06450631062D0020» ));
?>
Input:
06450631062D
Output:
D985D8B1D8AD
Be aware that iconv in PHP uses system implementations of locales and languages, what works under linux, normally doesn’t in windows.
Also, you may notice that recent versions of linux (debian, ubuntu, centos, etc) the //TRANSLIT option doesn’t work. since most distros doesn’t include the intl packages (example: php5-intl and icuxx (where xx is a number) in debian) by default. And this because the intl package conflicts with another package needed for international DNS resolution.
Problem is that configuration is dependent of the sysadmin of the machine where you’re hosted, so iconv is pretty much useless by default, depending on what configuration is used by your distro or the machine’s admin.
iconv with //IGNORE works as expected: it will skip the character if this one does not exist in the $out_charset encoding.
If a character is missing from the $in_charset encoding (eg byte \x81 from CP1252 encoding), then iconv will return an error, whether with //IGNORE or not.
There may be situations when a new version of a web site, all in UTF-8, has to display some old data remaining in the database with ISO-8859-1 accents. The problem is iconv(«ISO-8859-1», «UTF-8», $string) should not be applied if $string is already UTF-8 encoded.
I use this function that does’nt need any extension :
function convert_utf8( $string ) <
if ( strlen(utf8_decode($string)) == strlen($string) ) <
// $string is not UTF-8
return iconv(«ISO-8859-1», «UTF-8», $string);
> else // already UTF-8
return $string;
>
>
I have not tested it extensively, hope it may help.
If you want to convert to a Unicode encoding without the byte order mark (BOM), add the endianness to the encoding, e.g. instead of «UTF-16» which will add a BOM to the start of the string, use «UTF-16BE» which will convert the string without adding a BOM.
iconv ( ‘CP1252’ , ‘UTF-16’ , $text ); // with BOM
iconv ( ‘CP1252’ , ‘UTF-16BE’ , $text ); // without BOM
On some systems there may be no such function as iconv(); this is due to the following reason: a constant is defined named `iconv` with the value `libiconv`. So, the string PHP_FUNCTION(iconv) transforms to PHP_FUNCTION(libiconv), and you have to call libiconv() function instead of iconv().
I had seen this on FreeBSD, but I am sure that was a rather special build.
If you’d want not to be dependent on this behaviour, add the following to your script:
if (! function_exists ( ‘iconv’ ) && function_exists ( ‘libiconv’ )) function iconv ( $input_encoding , $output_encoding , $string ) return libiconv ( $input_encoding , $output_encoding , $string );
>
>
?>
Thanks to tony2001 at phpclub.net for explaining this behaviour.
In my case, I had to change:
setlocale ( LC_CTYPE , ‘cs_CZ’ );
?>
to
setlocale ( LC_CTYPE , ‘cs_CZ.UTF-8’ );
?>
Otherwise it returns question marks.
When I asked my linux for locale (by locale command) it returns «cs_CZ.UTF-8», so there is maybe correlation between it.
iconv (GNU libc) 2.6.1
glibc 2.3.6
Didn’t know its a feature or not but its works for me (PHP 5.0.4)
test it to convert from windows-1251 (stored in DB) to UTF-8 (which i use for web pages).
BTW i convert each array i fetch from DB with array_walk_recursive.
As orrd101 said, there is a bug with //IGNORE in recent PHP versions (we use 5.6.5) where we couldn’t convert some strings (i.e. «∙» from UTF8 to CP1251 with //IGNORE).
But we have found a workaround and now we use both //TRANSLIT and //IGNORE flags:
$text=»∙»;
iconv(«UTF8», «CP1251//TRANSLIT//IGNORE», $text);
I have used iconv to convert from cp1251 into UTF-8. I spent a day to investigate why a string with Russian capital ‘Р’ (sounds similar to ‘r’) at the end cannot be inserted into a database.
The problem is not in iconv. But ‘Р’ in cp1251 is chr(208) and ‘Р’ in UTF-8 is chr(208).chr(106). chr(106) is one of the space symbol which match ‘\s’ in regex. So, it can be taken by a greedy ‘+’ or ‘*’ operator. In that case, you loose ‘Р’ in your string.
For example, ‘ГР ‘ (Russian, UTF-8). Function preg_match. Regex is ‘(.+?)[\s]*’. Then ‘(.+?)’ matches ‘Г’.chr(208) and ‘[\s]*’ matches chr(106).’ ‘.
Although, it is not a bug of iconv, but it looks like it very much. That’s why I put this comment here.
I just found out today that the Windows and *NIX versions of PHP use different iconv libraries and are not very consistent with each other.
Here is a repost of my earlier code that now works on more systems. It converts as much as possible and replaces the rest with question marks:
if (! function_exists ( ‘utf8_to_ascii’ )) setlocale ( LC_CTYPE , ‘en_AU.utf8’ );
if (@ iconv ( «UTF-8» , «ASCII//IGNORE//TRANSLIT» , ‘é’ ) === false ) // PHP is probably using the glibc library (*NIX)
function utf8_to_ascii ( $text ) return iconv ( «UTF-8» , «ASCII//TRANSLIT» , $text );
>
>
else // PHP is probably using the libiconv library (Windows)
function utf8_to_ascii ( $text ) if ( is_string ( $text )) // Includes combinations of characters that present as a single glyph
$text = preg_replace_callback ( ‘/\X/u’ , __FUNCTION__ , $text );
>
elseif ( is_array ( $text ) && count ( $text ) == 1 && is_string ( $text [ 0 ])) // IGNORE characters that can’t be TRANSLITerated to ASCII
$text = iconv ( «UTF-8» , «ASCII//IGNORE//TRANSLIT» , $text [ 0 ]);
// The documentation says that iconv() returns false on failure but it returns »
if ( $text === » || ! is_string ( $text )) $text = ‘?’ ;
>
elseif ( preg_match ( ‘/\w/’ , $text )) < // If the text contains any letters.
$text = preg_replace ( ‘/\W+/’ , » , $text ); // . then remove all non-letters
>
>
else < // $text was not a string
$text = » ;
>
return $text ;
>
>
>
Here is how to convert UTF-8 numbers to UCS-2 numbers in hex:
function utf8toucs2 ( $str )
for ( $i = 0 ; $i < strlen ( $str ); $i += 2 )
$substring1 = $str [ $i ]. $str [ $i + 1 ];
$substring2 = $str [ $i + 2 ]. $str [ $i + 3 ];
if ( hexdec ( $substring1 ) < 127 )
$results = «00» . $str [ $i ]. $str [ $i + 1 ];
else
$results = dechex (( hexdec ( $substring1 )- 192 )* 64 + ( hexdec ( $substring2 )- 128 ));
if ( $results < 1000 ) $results = "0" . $results ;
$i += 2 ;
>
$ucs2 .= $results ;
>
return $ucs2 ;
>
echo strtoupper ( utf8toucs2 ( «D985D8B1D8AD» )). «\n» ;
echo strtoupper ( utf8toucs2 ( «456725» )). «\n» ;
Here is a code to convert ISO 8859-1 to UTF-8 and vice versa without using iconv.