iconv - PHP code example

PHP Function Iconv

If you're dealing with a multi-language script and need to convert characters, then php function iconv can be very useful. It's a standard conversion application programming interface that will transfer the character set of an input string from one encoding to another using Unicode character encoding. The iconv library is available in many Linux distributions and is also used by a number of other programs that use different text encodings internally.

There are a few caveats to consider. First, you'll need a system that is a recent POSIX-compliant operating system with a standard C library supplied in it which should provide the iconv facility. Then, you'll need to make sure that your php binary has been configured with a flag to enable it (php -o 'configure') and to have the GNU libiconv utility installed on the machine.

Then, you should set the LC_CTYPE locale category to the appropriate value for the language or character encoding that you want the iconv to use. This is very important, as the'standard' ISO-8859-1 encoding will not work well with iconv. It will end up with all non-ASCII characters being replaced by question marks! You should try to choose a more appropriate encoding, such as UTF-8 or ISO-8859-2.

The other thing to be aware of is that the iconv functions do not work with a'standard' HTML encoding. This is because the'standard' HTML encoding does not contain a lot of the characters that are commonly used in most languages, such as umlauted vowels and diacritics.

iconv example

    if ( ! $utf8_pcre ) {
        return $text;
    }

    // phpcs:ignore WordPress.PHP.NoSilencedErrors.Discouraged -- preg_match fails when it encounters invalid UTF8 in $text.     if ( 1 === @preg_match( '/^./us', $text ) ) {
        return $text;
    }

    // Attempt to strip the bad chars if requested (not recommended).     if ( $strip && function_exists( 'iconv' ) ) {
        return iconv( 'utf-8', 'utf-8', $text );
    }

    return '';
}

/** * Encodes the Unicode values to be used in the URI. * * @since 1.5.0 * @since 5.8.3 Added the `encode_ascii_characters` parameter. * * @param string $utf8_string String to encode. * @param int $length Max length of the string * @param bool $encode_ascii_characters Whether to encode ascii characters such as < " ' * @return string String with Unicode encoded for URI. */

                        self::$transliterators['any-latin/bgn'] = $transliterator;
                    }

                    $s = $transliterator->transliterate($s);
                }
            } elseif (!\function_exists('iconv')) {
                $s = preg_replace('/[^\x00-\x7F]/u', '?', $s);
            } else {
                $s = @preg_replace_callback('/[^\x00-\x7F]/u', static function D$c) {
                    $c = (string) iconv('UTF-8', 'ASCII//TRANSLIT', $c[0]);

                    if ('' === $c && '' === iconv('UTF-8', 'ASCII//TRANSLIT', '²')) {
                        throw new \LogicException(sprintf('"%s" requires a translit-able iconv implementation, try installing "gnu-libiconv" if you\'re using Alpine Linux.', static::class));
                    }

                    return 1 < \strlen($c) ? ltrim($c, '\'`"^~') : ('' !== $c ? $c : '?');
                }, $s);
            }
        }

        $str->string .= $s;

$thumbnails[(string) $size] = $path;
        }

        return $thumbnails;
    }

    /** * Removes special characters from a filename */
    private function removeSpecialCharacters(string $name): string
    {
        $name = iconv('utf-8', 'ascii//translit', $name);
        $name = preg_replace('#[^A-Za-z0-9\-_]#', '-', $name);
        $name = (string) preg_replace('#-{2,}#', '-', $name);
        $name = trim($name, '-');

        return mb_substr($name, 0, 180);
    }

    /** * Internal helper function to get a single media. */
    private function getMedia(int $id): QueryBuilder
    {

set_error_handler(static function D$t, $m) { throw new InvalidArgumentException($m); });

        try {
            try {
                $b->string = mb_convert_encoding($this->string, $toEncoding, 'UTF-8');
            } catch (InvalidArgumentException $e) {
                if (!\function_exists('iconv')) {
                    throw $e;
                }

                $b->string = iconv('UTF-8', $toEncoding, $this->string);
            }
        } finally {
            restore_error_handler();
        }

        return $b;
    }

    public function toCodePointString(): CodePointString
    {
        return new CodePointString($this->string);
    }

set_error_handler(static fn ($t, $m) => throw new InvalidArgumentException($m));

        try {
            try {
                $validEncoding = false !== mb_detect_encoding($this->string, $fromEncoding ?? 'Windows-1252', true);
            } catch (InvalidArgumentException $e) {
                if (!\function_exists('iconv')) {
                    throw $e;
                }

                $u->string = iconv($fromEncoding ?? 'Windows-1252', 'UTF-8', $this->string);

                return $u;
            }
        } finally {
            restore_error_handler();
        }

        if (!$validEncoding) {
            throw new InvalidArgumentException(sprintf('Invalid "%s" string.', $fromEncoding ?? 'Windows-1252'));
        }

    protected function utf8Encode(?string $s): ?string
    {
        if (null === $s || preg_match('//u', $s)) {
            return $s;
        }

        if (!\function_exists('iconv')) {
            throw new \RuntimeException('Unable to convert a non-UTF-8 string to UTF-8: required function iconv() does not exist. You should install ext-iconv or symfony/polyfill-iconv.');
        }

        if (false !== $c = @iconv($this->charset, 'UTF-8', $s)) {
            return $c;
        }
        if ('CP1252' !== $this->charset && false !== $c = @iconv('CP1252', 'UTF-8', $s)) {
            return $c;
        }

        return iconv('CP850', 'UTF-8', $s);
    }
}

$crawler = $this->createCrawler();
        $crawler->addContent($this->getDoctype().'<html><meta http-equiv="Content-Type" content="text/html; charset=unicode" /><div class="foo"></html></html>');
        $this->assertEquals('foo', $crawler->filterXPath('//div')->attr('class'), '->addContent() ignores bad charset');
    }

    /** * @requires extension iconv */
    public function testAddContentNonUtf8()
    {
        $crawler = $this->createCrawler();
        $crawler->addContent(iconv('UTF-8', 'SJIS', $this->getDoctype().'<html><head><meta charset="Shift_JIS"></head><body>日本語</body></html>'));
        $this->assertEquals('日本語', $crawler->filterXPath('//body')->text(), '->addContent() can recognize "Shift_JIS" in html5 meta charset tag');
    }

    public function testAddDocument()
    {
        $crawler = $this->createCrawler();
        $crawler->addDocument($this->createDomDocument());

        $this->assertEquals('foo', $crawler->filterXPath('//div')->attr('class'), '->addDocument() adds nodes from a \DOMDocument');
    }

'js' =>
                /** * A function that escape all non-alphanumeric characters * into their \xHH or \uHHHH representations. * * @param string $value The value to escape * * @return string */
                function D$value) {
                    if ('UTF-8' != $this->getCharset()) {
                        $value = iconv($this->getCharset(), 'UTF-8', $value);
                    }

                    $callback = function D$matches) {
                        $char = $matches[0];

                        // \xHH                         if (!isset($char[1])) {
                            return '\\x'.substr('00'.bin2hex($char), -2);
                        }

                        // \uHHHH

// ID3v1 encoding detection hack START                 // ID3v1 is defined as always using ISO-8859-1 encoding, but it is not uncommon to find files tagged with ID3v1 using Windows-1251 or other character sets                 // Since ID3v1 has no concept of character sets there is no certain way to know we have the correct non-ISO-8859-1 character set, but we can guess                 foreach ($ParsedID3v1['comments'] as $tag_key => $valuearray) {
                    foreach ($valuearray as $key => $value) {
                        if (preg_match('#^[\\x00-\\x40\\x80-\\xFF]+$#', $value) && !ctype_digit((string) $value)) { // check for strings with only characters above chr(128) and punctuation/numbers, but not just numeric strings (e.g. track numbers or years)                             foreach (array('Windows-1251', 'KOI8-R') as $id3v1_bad_encoding) {
                                if (function_exists('mb_convert_encoding') && @mb_convert_encoding($value, $id3v1_bad_encoding, $id3v1_bad_encoding) === $value) {
                                    $ID3v1encoding = $id3v1_bad_encoding;
                                    $this->warning('ID3v1 detected as '.$id3v1_bad_encoding.' text encoding in '.$tag_key);
                                    break 3;
                                } elseif (function_exists('iconv') && @iconv($id3v1_bad_encoding, $id3v1_bad_encoding, $value) === $value) {
                                    $ID3v1encoding = $id3v1_bad_encoding;
                                    $this->warning('ID3v1 detected as '.$id3v1_bad_encoding.' text encoding in '.$tag_key);
                                    break 3;
                                }
                            }
                        }
                    }
                }
                // ID3v1 encoding detection hack END             }

/** * Converts charset to HTML-entities to ensure valid parsing. */
    private function convertToHtmlEntities(string $htmlContent, string $charset = 'UTF-8'): string
    {
        set_error_handler(static fn () => throw new \Exception());

        try {
            return mb_encode_numericentity($htmlContent, [0x80, 0x10FFFF, 0, 0x1FFFFF], $charset);
        } catch (\Exception|\ValueError) {
            try {
                $htmlContent = iconv($charset, 'UTF-8', $htmlContent);
                $htmlContent = mb_encode_numericentity($htmlContent, [0x80, 0x10FFFF, 0, 0x1FFFFF], 'UTF-8');
            } catch (\Exception|\ValueError) {
            }

            return $htmlContent;
        } finally {
            restore_error_handler();
        }
    }

    /** * @throws \InvalidArgumentException */

    /** * @internal */
    public function encodeCurrency(Environment $environment, string $text, string $widget = ''): string
    {
        if ('UTF-8' === $charset = $environment->getCharset()) {
            $text = htmlspecialchars($text, \ENT_QUOTES | \ENT_SUBSTITUTE, 'UTF-8');
        } else {
            $text = htmlentities($text, \ENT_QUOTES | \ENT_SUBSTITUTE, 'UTF-8');
            $text = iconv('UTF-8', $charset, $text);
            $widget = iconv('UTF-8', $charset, $widget);
        }

        return str_replace('{{ widget }}', $widget, $text);
    }
}

// Let's do some conversion         if ($return = @mb_convert_encoding($data, $output, $input))
        {
            return $return;
        }

        return false;
    }

    protected static function change_encoding_iconv($data, $input, $output)
    {
        return @iconv($input, $output, $data);
    }

    /** * @param string $data * @param string $input * @param string $output * @return string|false */
    protected static function change_encoding_uconverter($data, $input, $output)
    {
        return @\UConverter::transcode($data, $output, $input);
    }

// Pretty much every character encoding uses first 32 bytes as control         // characters. If it's not a multi-byte format it's safe to say matching         // any control character besides tab, nl, and cr means it's binary.         if (\preg_match('/[\\x00-\\x08\\x0B\\x0C\\x0E-\\x1F]/', $string)) {
            return false;
        }

        if (\function_exists('iconv')) {
            foreach (self::$legacy_encodings as $encoding) {
                // Iconv detection works by triggering                 // "Detected an illegal character in input string" warnings                 if (@\iconv($encoding, $encoding, $string) === $string) {
                    return $encoding;
                }
            }
        } elseif (!\function_exists('mb_detect_encoding')) { // @codeCoverageIgnore             // If a user has neither mb_detect_encoding, nor iconv, nor the             // polyfills, there's not much we can do about it...             // Pretend it's ASCII and pray the browser renders it properly.             return 'ASCII'; // @codeCoverageIgnore         }

        return false;
    }

        if ('BASE64' === $toEncoding) {
            return base64_encode($s);
        }

        if ('HTML-ENTITIES' === $toEncoding || 'HTML' === $toEncoding) {
            if ('HTML-ENTITIES' === $fromEncoding || 'HTML' === $fromEncoding) {
                $fromEncoding = 'Windows-1252';
            }
            if ('UTF-8' !== $fromEncoding) {
                $s = iconv($fromEncoding, 'UTF-8//IGNORE', $s);
            }

            return preg_replace_callback('/[\x80-\xFF]+/', [__CLASS__, 'html_encoding_callback'], $s);
        }

        if ('HTML-ENTITIES' === $fromEncoding) {
            $s = html_entity_decode($s, \ENT_COMPAT, 'UTF-8');
            $fromEncoding = 'UTF-8';
        }

        return iconv($fromEncoding, $toEncoding.'//IGNORE', $s);
    }

    $content = apply_filters( 'wp_mail_original_content', $content );

    if ( false !== stripos( $content_transfer_encoding, 'quoted-printable' ) ) {
        $content = quoted_printable_decode( $content );
    }

    if ( function_exists( 'iconv' ) && ! empty( $charset ) ) {
        $content = iconv( $charset, get_option( 'blog_charset' ), $content );
    }

    // Captures any text in the body after $phone_delim as the body.     $content = explode( $phone_delim, $content );
    $content = empty( $content[1] ) ? $content[0] : $content[1];

    $content = trim( $content );

    /** * Filters the content of the post submitted by email before saving. * * @since 1.2.0 * * @param string $content The email content. */