| Previous | Table of Contents | Next |
There are five kinds of tokens: identifiers, keywords, literals, operators, and other separators. Blanks, horizontal and vertical
tabs, newlines, formfeeds, and comments (collective, “white space?), as described below, are ignored except as they serve
to separate tokens. Some white space is required to separate otherwise adjacent identifiers, keywords, and constants.
If the input stream has been parsed into tokens up to a given character, the next token is taken to be the longest string
of characters that could possibly constitute a token.