Module pl.lexer
Lexical scanner for creating a sequence of tokens from text.
lexer.scan(s) returns an iterator over all tokens found in the
string s. This iterator returns two values, a token type string
(such as 'string' for quoted string, 'iden' for identifier) and the value of the
token.
Versions specialized for Lua and C are available; these also handle block comments and classify keywords as 'keyword' tokens. For example:
> s = 'for i=1,n do' > for t,v in lexer.lua(s) do print(t,v) end keyword for iden i = = number 1 , , iden n keyword do
See the Guide for further discussion
Functions
| scan (s, matches[, filter[, options]]) | create a plain token iterator from a string or file-like object. |
| insert (tok, a1, a2) | insert tokens into a stream. |
| getline (tok) | get everything in a stream upto a newline. |
| lineno (tok) | get current line number. |
| getrest (tok) | get the rest of the stream. |
| get_keywords () | get the Lua keywords as a set-like table. |
| lua (s[, filter[, options]]) | create a Lua token iterator from a string or file-like object. |
| cpp (s[, filter[, options]]) | create a C/C++ token iterator from a string or file-like object. |
| get_separated_list (tok[, endtoken=')'[, delim=']]) | get a list of parameters separated by a delimiter from a stream. |
| skipws (tok) | get the next non-space token from the stream. |
| expecting (tok, expected_type, no_skip_ws) | get the next token, which must be of the expected type. |
Functions
- scan (s, matches[, filter[, options]])
-
create a plain token iterator from a string or file-like object.
Parameters:
- s
string or file
a string or a file-like object with
:read()method returning lines. - matches
table
an optional match table - array of token descriptions.
A token is described by a
{pattern, action}pair, wherepatternshould match token body andactionis a function called when a token of described type is found. - filter
table
a table of token types to exclude, by default
{space=true}(optional) - options
table
a table of options; by default,
{number=true,string=true}, which means convert numbers and strip string quotes. (optional)
- s
string or file
a string or a file-like object with
- insert (tok, a1, a2)
-
insert tokens into a stream.
Parameters:
- tok a token stream
- a1 a string is the type, a table is a token list and a function is assumed to be a token-like iterator (returns type & value)
- a2 string a string is the value
- getline (tok)
-
get everything in a stream upto a newline.
Parameters:
- tok a token stream
Returns:
-
a string
- lineno (tok)
-
get current line number.
Parameters:
- tok a token stream
Returns:
-
the line number.
if the input source is a file-like object,
also return the column.
- getrest (tok)
-
get the rest of the stream.
Parameters:
- tok a token stream
Returns:
-
a string
- get_keywords ()
-
get the Lua keywords as a set-like table.
So
res["and"]etc would betrue.Returns:
-
a table
- lua (s[, filter[, options]])
-
create a Lua token iterator from a string or file-like object.
Will return the token type and value.
Parameters:
- cpp (s[, filter[, options]])
-
create a C/C++ token iterator from a string or file-like object.
Will return the token type type and value.
Parameters:
- get_separated_list (tok[, endtoken=')'[, delim=']])
-
get a list of parameters separated by a delimiter from a stream.
Parameters:
- tok the token stream
- endtoken string end of list. Can be '\n' (default ')')
- delim string separator (default ')
Returns:
-
a list of token lists.
- skipws (tok)
-
get the next non-space token from the stream.
Parameters:
- tok the token stream.
- expecting (tok, expected_type, no_skip_ws)
-
get the next token, which must be of the expected type.
Throws an error if this type does not match!
Parameters:
- tok the token stream
- expected_type string the token type
- no_skip_ws boolean whether we should skip whitespace