Penlight Documentation

Additional Libraries

Libraries in this section are no longer considered to be part of the Penlight core, but still provide specialized functionality when needed.

Simple Input Patterns

Lua string pattern matching is very powerful, and usually you will not need a traditional regular expression library. Even so, sometimes Lua code ends up looking like Perl, which happens because string patterns are not always the easiest things to read, especially for the casual reader. Here is a program which needs to understand three distinct date formats:

-- parsing dates using Lua string patterns
months={Jan=1,Feb=2,Mar=3,Apr=4,May=5,Jun=6,
Jul=7,Aug=8,Sep=9,Oct=10,Nov=11,Dec=12}

function check_and_process(d,m,y)
    d = tonumber(d)
    m = tonumber(m)
    y = tonumber(y)
    ....
end

for line in f:lines() do
    -- ordinary (English) date format
    local d,m,y = line:match('(%d+)/(%d+)/(%d+)')
    if d then
        check_and_process(d,m,y)
    else -- ISO date??
        y,m,d = line:match('(%d+)%-(%d+)%-(%d+)')
        if y then
            check_and_process(d,m,y)
        else -- <day> <month-name> <year>?
            d,mm,y = line:match('%(d+)%s+(%a+)%s+(%d+)')
            m = months[mm]
            check_and_process(d,m,y)
        end
    end
end

These aren't particularly difficult patterns, but already typical issues are appearing, such as having to escape '-'. Also, string.match returns its captures, so that we're forced to use a slightly awkward nested if-statement.

Verification issues will further cloud the picture, since regular expression people try to enforce constraints (like year cannot be more than four digits) using regular expressions, on the usual grounds that you shouldn't stop using a hammer when you are enjoying yourself.

pl.sip provides a simple, intuitive way to detect patterns in strings and extract relevant parts.

> sip = require 'pl.sip'
> dump = require('pl.pretty').dump
> res = {}
> c = sip.compile 'ref=$S{file}:$d{line}'
> = c('ref=hello.c:10',res)
true
> dump(res)
{
  line = 10,
  file = "hello.c"
}
> = c('ref=long name, no line',res)
false

sip.compile creates a pattern matcher function, which takes a string and a table as arguments. If the string matches the pattern, then true is returned and the table is populated according to the captures within the pattern.

Here is another version of the date parser:

-- using SIP patterns
function check(t)
    check_and_process(t.day,t.month,t.year)
end

shortdate = sip.compile('$d{day}/$d{month}/$d{year}')
longdate = sip.compile('$d{day} $v{mon} $d{year}')
isodate = sip.compile('$d{year}-$d{month}-$d{day}')

for line in f:lines() do
    local res = {}
    if shortdate(str,res) then
        check(res)
    elseif isodate(str,res) then
        check(res)
    elseif longdate(str,res) then
        res.month = months[res.mon]
        check(res)
    end
end

SIP captures start with '$', then a one-character type, and then an optional variable name in curly braces.

Type      Meaning
v         identifier
i         possibly signed integer
f         floating-point number
r         rest of line
q         quoted string (quoted using either ' or ")
p         a path name
(         anything inside balanced parentheses
[         anything inside balanced brackets
{         anything inside balanced curly brackets
<         anything inside balanced angle brackets

If a type is not one of the above, then it's assumed to be one of the standard Lua character classes, and will match one or more repetitions of that class. Any spaces you leave in your pattern will match any number of spaces, including zero, unless the spaces are between two identifier characters or patterns matching them; in that case, at least one space will be matched.

SIP captures (like $v{mon}) do not have to be named. You can use just $v, but you have to be consistent; if a pattern contains unnamed captures, then all captures must be unnamed. In this case, the result table is a simple list of values.

sip.match is a useful shortcut if you want to compile and match in one call, without saving the compiled pattern. It caches the result, so it is not much slower than explicitly using sip.compile.

> sip.match('($q{first},$q{second})','("john","smith")',res)
true
> res
{second='smith',first='john'}
> res = {}
> sip.match('($q,$q)','("jan","smit")',res)  -- unnamed captures
true
> res
{'jan','smit'}
> sip.match('($q,$q)','("jan", "smit")',res)
false   ---> oops! Can't handle extra space!
> sip.match('( $q , $q )','("jan", "smit")',res)
true

As a general rule, allow for whitespace in your patterns.

Finally, putting a '$' at the end of a pattern means 'capture the rest of the line, starting at the first non-space'. It is a shortcut for '$r{rest}', or just '$r' if no named captures are used.

> sip.match('( $q , $q ) $','("jan", "smit") and a string',res)
true
> res
{'jan','smit','and a string'}
> res = {}
> sip.match('( $q{first} , $q{last} ) $','("jan", "smit") and a string',res)
true
> res
{first='jan',rest='and a string',last='smit'}

Command-line Programs with Lapp

pl.lapp is a small and focused Lua module which aims to make standard command-line parsing easier and intuitive. It implements the standard GNU style, i.e. short flags with one letter start with '-', and there may be an additional long flag which starts with '--'. Generally options which take an argument expect to find it as the next parameter (e.g. 'gcc test.c -o test') but single short options taking a value can dispense with the space (e.g. 'head -n4 test.c' or gcc -I/usr/include/lua/5.1 ...)

As far as possible, Lapp will convert parameters into their equivalent Lua types, i.e. convert numbers and convert filenames into file objects. If any conversion fails, or a required parameter is missing, an error will be issued and the usage text will be written out. So there are two necessary tasks, supplying the flag and option names and associating them with a type.

For any non-trivial script, even for personal consumption, it's necessary to supply usage text. The novelty of Lapp is that it starts from that point and defines a loose format for usage strings which can specify the names and types of the parameters.

An example will make this clearer:

-- scale.lua
  lapp = require 'pl.lapp'
  local args = lapp [[
  Does some calculations
    -o,--offset (default 0.0)  Offset to add to scaled number
    -s,--scale  (number)  Scaling factor
    <number> (number)  Number to be scaled
  ]]

  print(args.offset + args.scale * args.number)

Here is a command-line session using this script:

$ lua scale.lua
scale.lua:missing required parameter: scale

Does some calculations
 -o,--offset (default 0.0)  Offset to add to scaled number
 -s,--scale  (number)  Scaling factor
  <number> (number )  Number to be scaled

$ lua scale.lua -s 2.2 10
22

$ lua scale.lua -s 2.2 x10
scale.lua:unable to convert to number: x10

....(usage as before)

There are two kinds of lines in Lapp usage strings which are meaningful; option and parameter lines. An option line gives the short option, optionally followed by the corresponding long option. A type specifier in parentheses may follow. Similarly, a parameter line starts with '', followed by a type specifier.

Type specifiers usually start with a type name: one of 'boolean', 'string','number','file-in' or 'file-out'. You may leave this out, but then must say 'default' followed by a value. If a flag or parameter has a default, it is not required and is set to the default. The actual type is deduced from this value (number, string, file or boolean) if not provided directly. 'Deduce' is a fancy word for 'guess' and it can be wrong, e.g '(default 1)' will always be a number. You can say '(string default 1)' to override the guess. There are file values for the predefined console streams: stdin, stdout, stderr.

The boolean type is the default for flags. Not providing the type specifier is equivalent to '(boolean default false)`. If the flag is meant to be 'turned off' then either the full '(boolean default true)` or the shortcut '(default true)' will work.

An alternative to default is optional:

local lapp = require 'pl.lapp'
local args = lapp [[
   --cmd (optional string) Command to run.
]]

if args.cmd then
  os.execute(args.cmd)
end

Here we're implying that cmd need not be specified (just as with default) but if not present, then args.cmd is nil, which will always test false.

The rest of the line is ignored and can be used for explanatory text.

This script shows the relation between the specified parameter names and the fields in the output table.

-- simple.lua
local args = require ('pl.lapp') [[
Various flags and option types
  -p          A simple optional flag, defaults to false
  -q,--quiet  A simple flag with long name
  -o  (string)  A required option with argument
  -s  (default 'save') Optional string with default 'save' (single quotes ignored)
  -n  (default 1) Optional numerical flag with default 1
  -b  (string default 1)  Optional string flag with default '1' (type explicit)
  <input> (default stdin)  Optional input file parameter, reads from stdin
]]

for k,v in pairs(args) do
    print(k,v)
end

I've just dumped out all values of the args table; note that args.quiet has become true, because it's specified; args.p defaults to false. If there is a long name for an option, that will be used in preference as a field name. A type or default specifier is not necessary for simple flags, since the default type is boolean.

$ simple -o test -q simple.lua
p       false
input   file (781C1BD8)
quiet   true
o       test
input_name      simple.lua
D:\dev\lua\lapp>simple -o test simple.lua one two three
1       one
2       two
3       three
p       false
quiet   false
input   file (781C1BD8)
o       test
input_name      simple.lua

The parameter input has been set to an open read-only file object - we know it must be a read-only file since that is the type of the default value. The field input_name is automatically generated, since it's often useful to have access to the original filename.

Notice that any extra parameters supplied will be put in the result table with integer indices, i.e. args[i] where i goes from 1 to #args.

Files don't really have to be closed explicitly for short scripts with a quick well-defined mission, since the result of garbage-collecting file objects is to close them.

Enforcing a Range and Enumerations

The type specifier can also be of the form '(' MIN '..' MAX ')' or a set of strings separated by '|'.

local lapp = require 'pl.lapp'
local args = lapp [[
    Setting ranges
    <x> (1..10)  A number from 1 to 10
    <y> (-5..1e6) Bigger range
    <z> (slow|medium|fast)
]]

print(args.x,args.y)

Here the meaning of ranges is that the value is greater or equal to MIN and less or equal to MAX. An 'enum' is a string that can only have values from a specified set.

Custom Types

There is no builti-in way to force a parameter to be a whole number, but you may define a custom type that does this:

lapp = require ('pl.lapp')

lapp.add_type('integer','number',
    function(x)
        lapp.assert(math.ceil(x) == x, 'not an integer!')
    end
)

local args =  lapp [[
    <ival> (integer) Process PID
]]

print(args.ival)

lapp.add_type takes three parameters, a type name, a converter and a constraint function. The constraint function is expected to throw an assertion if some condition is not true; we use lapp.assert because it fails in the standard way for a command-line script. The converter argument can either be a type name known to Lapp, or a function which takes a string and generates a value.

Here's a useful custom type that allows dates to be input as pl.Date values:

local df = Date.Format()

lapp.add_type('date',
    function(s)
        local d,e = df:parse(s)
        lapp.assert(d,e)
        return d
    end
)

'varargs' Parameter Arrays

lapp = require 'pl.lapp'
local args = lapp [[
Summing numbers
    <numbers...> (number) A list of numbers to be summed
]]

local sum = 0
for i,x in ipairs(args.numbers) do
    sum = sum + x
end
print ('sum is '..sum)

The parameter number has a trailing '...', which indicates that this parameter is a 'varargs' parameter. It must be the last parameter, and args.number will be an array.

Consider this implementation of the head utility from Mac OS X:

-- implements a BSD-style head
-- (see http://www.manpagez.com/man/1/head/osx-10.3.php)

lapp = require ('pl.lapp')

local args = lapp [[
Print the first few lines of specified files
   -n         (default 10)    Number of lines to print
   <files...> (default stdin) Files to print
]]

-- by default, lapp converts file arguments to an actual Lua file object.
-- But the actual filename is always available as <file>_name.
-- In this case, 'files' is a varargs array, so that 'files_name' is
-- also an array.
local nline = args.n
local nfile = #args.files
for i = 1,nfile do
    local file = args.files[i]
    if nfile > 1 then
        print('==> '..args.files_name[i]..' <==')
    end
    local n = 0
    for line in file:lines() do
        print(line)
        n = n + 1
        if n == nline then break end
    end
end

Note how we have access to all the filenames, because the auto-generated field files_name is also an array!

(This is probably not a very considerate script, since Lapp will open all the files provided, and only close them at the end of the script. See the xhead.lua example for another implementation.)

Flags and options may also be declared as vararg arrays, and can occur anywhere. If there is both a short and long form, then the trailing "..." must happen after the long form, for example "-x,--network... (string)...",

Bear in mind that short options can be combined (like 'tar -xzf'), so it's perfectly legal to have '-vvv'. But normally the value of args.v is just a simple true value.

local args = require ('pl.lapp') [[
   -v...  Verbosity level; can be -v, -vv or -vvv
]]
vlevel = not args.v[1] and 0 or #args.v
print(vlevel)

The vlevel assignment is a bit of Lua voodoo, so consider the cases:

* No -v flag, v is just { false }
* One -v flags, v is { true }
* Two -v flags, v is { true, true }
* Three -v flags, v is { true, true, true }

Defining a Parameter Callback

If a script implements lapp.callback, then Lapp will call it after each argument is parsed. The callback is passed the parameter name, the raw unparsed value, and the result table. It is called immediately after assignment of the value, so the corresponding field is available.

lapp = require ('pl.lapp')

function lapp.callback(parm,arg,args)
    print('+',parm,arg)
end

local args = lapp [[
Testing parameter handling
    -p               Plain flag (defaults to false)
    -q,--quiet       Plain flag with GNU-style optional long name
    -o  (string)     Required string option
    -n  (number)     Required number option
    -s (default 1.0) Option that takes a number, but will default
    <start> (number) Required number argument
    <input> (default stdin)  A parameter which is an input file
    <output> (default stdout) One that is an output file
]]
print 'args'
for k,v in pairs(args) do
    print(k,v)
end

This produces the following output:

$ args -o name -n 2 10 args.lua
+       o       name
+       n       2
+       start   10
+       input   args.lua
args
p       false
s       1
input_name      args.lua
quiet   false
output  file (781C1B98)
start   10
input   file (781C1BD8)
o       name
n       2

Callbacks are needed when you want to take action immediately on parsing an argument.

Slack Mode

If you'd like to use a multi-letter 'short' parameter you need to set the lapp.slack variable to true.

In the following example we also see how default false and default true flags can be used and how to overwrite the default -h help flag (--help still works fine) - this applies to non-slack mode as well.

-- Parsing the command line ----------------------------------------------------
-- test.lua
local lapp = require 'pl.lapp'
local pretty = require 'pl.pretty'
lapp.slack = true
local args = lapp [[
Does some calculations
   -v, --video              (string)             Specify input video
   -w, --width              (default 256)        Width of the video
   -h, --height             (default 144)        Height of the video
   -t, --time               (default 10)         Seconds of video to process
   -sk,--seek               (default 0)          Seek number of seconds
   -f1,--flag1                                   A false flag
   -f2,--flag2                                   A false flag
   -f3,--flag3              (default true)       A true flag
   -f4,--flag4              (default true)       A true flag
]]

pretty.dump(args)

And here we can see the output of test.lua:

$> lua test.lua -v abc --time 40 -h 20 -sk 15 --flag1 -f3
---->
{
  width = 256,
  flag1 = true,
  flag3 = false,
  seek = 15,
  flag2 = false,
  video = abc,
  time = 40,
  height = 20,
  flag4 = true
}

Simple Test Framework

pl.test was originally developed for the sole purpose of testing Penlight itself, but you may find it useful for your own applications. (There are many other options.)

Most of the goodness is in test.asserteq. It uses tablex.deepcompare on its two arguments, and by default quits the test application with a non-zero exit code, and an informative message printed to stderr:

local test = require 'pl.test'

test.asserteq({10,20,30},{10,20,30.1})

--~ test-test.lua:3: assertion failed
--~ got:    {
--~  [1] = 10,
--~  [2] = 20,
--~  [3] = 30
--~ }
--~ needed:    {
--~  [1] = 10,
--~  [2] = 20,
--~  [3] = 30.1
--~ }
--~ these values were not equal

This covers most cases but it's also useful to compare strings using string.match

-- must start with bonzo the dog
test.assertmatch ('bonzo the dog is here','^bonzo the dog')
-- must end with an integer
test.assertmatch ('hello 42','%d+$')

Since Lua errors are usually strings, this matching strategy is used to test 'exceptions':

test.assertraise(function()
    local t = nil
    print(t.bonzo)
end,'nil value')

(Some care is needed to match the essential part of the thrown error if you care for portability, since in Lua 5.2 the exact error is "attempt to index local 't' (a nil value)" and in Lua 5.3 the error is "attempt to index a nil value (local 't')")

There is an extra optional argument to these test functions, which is helpful when writing test helper functions. There you want to highlight the failed line, not the actual call to asserteq or assertmatch - line 33 here is the call to is_iden

function is_iden(str)
    test.assertmatch(str,'^[%a_][%w_]*$',1)
end

is_iden 'alpha_dog'
is_iden '$dollars'

--~ test-test.lua:33: assertion failed
--~ got:    "$dollars"
--~ needed:    "^[%a_][%w_]*$"
--~ these strings did not match

Useful Lua functions often return multiple values, and test.tuple is a convenient way to capture these values, whether they contain nils or not.

T = test.tuple

--- common error pattern
function failing()
    return nil,'failed'
end

test.asserteq(T(failing()),T(nil,'failed'))

Penlight

Contents

Manual

Libraries

Classes

Examples