token_get_all - PHP code example

Changing the PHP Error Mode to PDO:ERRMODE_EXCEPTION

PHP tries to be smart when it parses code but sometimes mistakes can happen. In those cases it should show proper errors, not just warnings or notices. This rfc will change the default error mode to PDO::ERRMODE_EXCEPTION so it should be easier for developers to get more information about mistakes they made while writing code.

PHP inbuilt function token_get_all() parses source string into PHP language tokens using Zend engine lexical scanner. It can recognize some of the PHP keywords, some functions and identifiers like class names. It also recognizes HTML code, comments and composite symbols. However it misses out to parse some other tokens which are important for working with PHP code. For example it doesn't recognize arithmetic or bitwise operations on arrays, resources and objects. It also fails to recognize the class names of private methods.

To solve these issues a new class PhpToken is introduced in PHP 8.0. It's a higher level abstraction than the token_get_all() and provides a better support for working with PHP code. It has public properties like $token->id, $token->text, $token->line and $token->pos that return useful information about the token. It can be used to unify tokens and support other PHP features which are not supported by the standard tokenizer of PHP. It also provides a function get_debug_type which is similar to the PHP inbuilt function gettype() but returns more useful output for arrays, strings, anonymous classes and objects. The corresponding RFC details all of these features.

token_get_all example

function wp_doc_link_parse( $content ) {
    if ( ! is_string( $content ) || empty( $content ) ) {
        return array();
    }

    if ( ! function_exists( 'token_get_all' ) ) {
        return array();
    }

    $tokens           = token_get_all( $content );
    $count            = count( $tokens );
    $functions        = array();
    $ignore_functions = array();

    for ( $t = 0; $t < $count - 2; $t++ ) {
        if ( ! is_array( $tokens[ $t ] ) ) {
            continue;
        }

        if ( T_STRING === $tokens[ $t ][0] && ( '(' === $tokens[ $t + 1 ] || '(' === $tokens[ $t + 2 ] ) ) {
            // If it's a function or class defined locally, there's not going to be any docs available.

        return (string) $this->text;
    }

    /** * @return static[] */
    public static function tokenize(string $code, int $flags = 0): array
    {
        $line = 1;
        $position = 0;
        $tokens = token_get_all($code, $flags);
        foreach ($tokens as $index => $token) {
            if (\is_string($token)) {
                $id = \ord($token);
                $text = $token;
            } else {
                [$id, $text, $line] = $token;
            }
            $tokens[$index] = new static($id, $text, $line, $position);
            $position += \strlen($text);
        }

self::MESSAGE_TOKEN,
        ],
    ];

    /** * @return void */
    public function extract(string|iterable $resource, MessageCatalogue $catalog)
    {
        $files = $this->extractFiles($resource);
        foreach ($files as $file) {
            $this->parseTokens(token_get_all(file_get_contents($file)), $catalog, $file);

            gc_mem_caches();
        }
    }

    /** * @return void */
    public function setPrefix(string $prefix)
    {
        $this->prefix = $prefix;
    }

        return false;
    }

    /** * Examines a file and returns the fully qualified class name. */
    public function getClassname(string $file): string
    {
        $php       = file_get_contents($file);
        $tokens    = token_get_all($php);
        $dlm       = false;
        $namespace = '';
        $className = '';

        foreach ($tokens as $i => $token) {
            if ($i < 2) {
                continue;
            }

            if ((isset($tokens[$i - 2][1]) && ($tokens[$i - 2][1] === 'phpnamespace' || $tokens[$i - 2][1] === 'namespace')) || ($dlm && $tokens[$i - 1][0] === T_NS_SEPARATOR && $token[0] === T_STRING)) {
                if (! $dlm) {

/** * The current array pointer. * * @var int */
    private $pointer = 0;

    /** @param string $contents */
    public function __construct($contents)
    {
        $this->tokens = token_get_all($contents);

        // The PHP parser sets internal compiler globals for certain things. Annoyingly, the last docblock comment it         // saw gets stored in doc_comment. When it comes to compile the next thing to be include()d this stored         // doc_comment becomes owned by the first thing the compiler sees in the file that it considers might have a         // docblock. If the first thing in the file is a class without a doc block this would cause calls to         // getDocBlock() on said class to return our long lost doc_comment. Argh.         // To workaround, cause the parser to parse an empty docblock. Sure getDocBlock() will return this, but at least         // it's harmless to us.         token_get_all("<?php\n/**\n *\n */");

        $this->numTokens = count($this->tokens);
    }

    $contents = file_get_contents($settings_file);
    if ($contents !== FALSE) {
      // Initialize the contents for the settings.php file if it is empty.       if (trim($contents) === '') {
        $contents = "<?php\n";
      }
      // Step through each token in settings.php and replace any variables that       // are in the passed-in array.       $buffer = '';
      $state = 'default';
      foreach (token_get_all($contents) as $token) {
        if (is_array($token)) {
          [$type, $value] = $token;
        }
        else {
          $type = -1;
          $value = $token;
        }
        // Do not operate on whitespace.         if (!in_array($type, [T_WHITESPACE, T_COMMENT, T_DOC_COMMENT], TRUE)) {
          switch ($state) {
            case 'default':

self::$operator[T_MATCH] = true;
            self::$strip[T_NULLSAFE_OBJECT_OPERATOR] = true;
            self::$classcalls[T_NULLSAFE_OBJECT_OPERATOR] = true;
            self::$namespace[T_NAME_FULLY_QUALIFIED] = true;
            self::$namespace[T_NAME_QUALIFIED] = true;
            self::$namespace[T_NAME_RELATIVE] = true;
            $identifier[T_NAME_FULLY_QUALIFIED] = true;
            $identifier[T_NAME_QUALIFIED] = true;
            $identifier[T_NAME_RELATIVE] = true;
        }

        $tokens = \token_get_all($source);
        $cursor = 1;
        $function_calls = [];

        // Performance optimization preventing backwards loops         /** @psalm-var array<PhpToken|null> */
        $prev_tokens = [null, null, null];

        if (\is_array($function)) {
            $class = \explode('\\', $function[0]);
            $class = \strtolower(\end($class));
            $function = \strtolower($function[1]);
        }

        return \is_string($resource) && 'php' === pathinfo($resource, \PATHINFO_EXTENSION) && (!$type || \in_array($type, ['annotation', 'attribute'], true));
    }

    /** * Returns the full class name for the first class in the file. */
    protected function findClass(string $file): string|false
    {
        $class = false;
        $namespace = false;
        $tokens = token_get_all(file_get_contents($file));

        if (1 === \count($tokens) && \T_INLINE_HTML === $tokens[0][0]) {
            throw new \InvalidArgumentException(sprintf('The file "%s" does not contain PHP code. Did you forgot to add the "<?php" start tag at the beginning of the file?', $file));
        }

        $nsTokens = [\T_NS_SEPARATOR => true, \T_STRING => true];
        if (\defined('T_NAME_QUALIFIED')) {
            $nsTokens[\T_NAME_QUALIFIED] = true;
        }
        for ($i = 0; isset($tokens[$i]); ++$i) {
            $token = $tokens[$i];

    public static function stripComments(string $source): string
    {
        trigger_deprecation('symfony/http-kernel', '6.4', 'Method "%s()" is deprecated without replacement.', __METHOD__);

        if (!\function_exists('token_get_all')) {
            return $source;
        }

        $rawChunk = '';
        $output = '';
        $tokens = token_get_all($source);
        $ignoreSpace = false;
        for ($i = 0; isset($tokens[$i]); ++$i) {
            $token = $tokens[$i];
            if (!isset($token[1]) || 'b"' === $token) {
                $rawChunk .= $token;
            } elseif (\T_START_HEREDOC === $token[0]) {
                $output .= $rawChunk.$token[1];
                do {
                    $token = $tokens[++$i];
                    $output .= isset($token[1]) && 'b"' !== $token ? $token[1] : $token;
                } while (\T_END_HEREDOC !== $token[0]);

    private static function stripComments(string $source): string
    {
        if (!\function_exists('token_get_all')) {
            return $source;
        }

        $rawChunk = '';
        $output = '';
        $tokens = token_get_all($source);
        $ignoreSpace = false;
        for ($i = 0; isset($tokens[$i]); ++$i) {
            $token = $tokens[$i];
            if (!isset($token[1]) || 'b"' === $token) {
                $rawChunk .= $token;
            } elseif (\T_START_HEREDOC === $token[0]) {
                $output .= $rawChunk.$token[1];
                do {
                    $token = $tokens[++$i];
                    $output .= isset($token[1]) && 'b"' !== $token ? $token[1] : $token;
                } while (\T_END_HEREDOC !== $token[0]);

    public static function stripComments(string $source): string
    {
        if (!\function_exists('token_get_all')) {
            return $source;
        }

        $rawChunk = '';
        $output = '';
        $tokens = token_get_all($source);
        $ignoreSpace = false;
        for ($i = 0; isset($tokens[$i]); ++$i) {
            $token = $tokens[$i];
            if (!isset($token[1]) || 'b"' === $token) {
                $rawChunk .= $token;
            } elseif (\T_START_HEREDOC === $token[0]) {
                $output .= $rawChunk.$token[1];
                do {
                    $token = $tokens[++$i];
                    $output .= isset($token[1]) && 'b"' !== $token ? $token[1] : $token;
                } while (\T_END_HEREDOC !== $token[0]);