LPeg - Parsing Expression Grammars For Lua

LPeg
Parsing Expression Grammars For Lua, version 0.6

## Introduction

LPeg is a new pattern-matching library for Lua, based on Parsing Expression Grammars (PEGs). In this text, I assume you are familiar with PEGs. If you are not, you can get a quick start reading the Wikipedia Entry for PEGs or Section 2 of Parsing Expression Grammars: A Recognition-Based Syntactic Foundation (the section has only one page). The nice thing about PEGs is that it has a formal basis (instead of being an ad-hoc set of features), allows an efficient and simple implementation, and does most things we expect from a pattern-matching library (and more, as we can define entire grammars).

Following the Snobol tradition, LPeg defines patterns as first-class objects. That is, patterns are regular Lua values (represented by userdata). The library offers several functions to create and compose patterns. With the use of metamethods, several of these functions are provided as infix or prefix operators. On the one hand, the result is usually much more verbose than the typical encoding of patterns using the so called regular expressions (which typically are not regular expressions in the formal sense). On the other hand, first-class patterns allow much better documentation (as it is easy to comment the code, to use auxiliary variables to break complex definitions, etc.) and are extensible, as we can define new functions to create and compose patterns.

For a quick glance of the library, the following table summarizes its basic operations for creating patterns:

Operator | Description
`lpeg.P(string)` | Matches `string` literally
`lpeg.P(number)` | Matches exactly `number` characters
`lpeg.S(string)` | Matches any character in `string` (set)
`lpeg.R("xy")` | Matches any character between x and y (range)
`patt^n` | Matches at least `n` repetitions of `patt`
`patt^-n` | Matches at most `n` repetitions of `patt`
`patt1 * patt2` | Matches `patt1` followed by `patt2`
`patt1 + patt2` | Matches `patt1` or `patt2` (ordered choice)
`patt1 - patt2` | Matches `patt1` if `patt2` does not match
`-patt` | Equivalent to `"" - patt`

As a very simple example, `lpeg.R("09")^1` matches a non-empty sequence of digits. As a not so simple example, `-lpeg.P(1)` (which can be written as `lpeg.P(-1)` or simply `-1` for operations expecting a pattern) matches an empty string only if it cannot match a single character; so, it succeeds only at the subject's end.

Those not convinced by the previous syntax can try the `re` module, which implements patterns following a regular-expression style (e.g., `[09]+`). (This module is less than 100 lines of Lua code, and of course uses LPeg to parse regular expressions.) This library currently supports only very basic captures. But it provides a good source for those looking for extended examples of LPeg definitions.

## Basic Constructions

Most of the following operations build patterns. All operations that expect a pattern as an argument may receive also strings, tables, numbers, booleans, or functions, which are translated to patterns according to the rules of function `lpeg.P`.

### `lpeg.match (pattern, subject [, init])`

The matching function. It attempts to match the given pattern against the subject string. If the match succeeds, returns the index in the subject of the first character after the match, or the values of captured values (if the pattern captured any value).

An optional numeric argument `init` makes the match starts at that position in the subject string. As usual in Lua libraries, a negative value counts from the end.

Unlike typical pattern-matching functions, `match` works only in anchored mode; that is, it tries to match the pattern with a prefix of the given subject string (at position `init`), not with an arbitrary substring of the subject. So, if we want to find a pattern anywhere in a string, we must either write a loop in Lua or write a pattern that matches anywhere. This second approach is easy and quite efficient; see examples.

### `lpeg.P (value)`

Converts the given value into a proper pattern, according to the following rules:

- If the argument is a pattern, it is returned unmodified.

- If the argument is a string, it is translated to a pattern that matches literally the string.

- If the argument is a number, it is translated as follows. A non-negative number n gives a pattern that matches exactly n characters; a negative number -n gives a pattern that succeeds only if the input string does not have n characters. It is (as expected) equivalent to the unary minus operation (see below) applied over the absolute value of n.

- If the argument is a boolean, the result is a pattern that always succeeds or always fails (according to the boolean value), without consuming any input.

- If the argument is a table, it is interpreted as a grammar (see Grammars).

- If the argument is a function, returns a pattern that matches as follows. Each time there is an attempt for a match against this pattern, the function is called, always with two arguments: the original subject string, and the current position in the subject. If the call returns a valid number, the match succeeds and the returned number becomes the new current position. If the call returns false, nil, or an invalid number, the match fails.

  If the function is called with parameters s and i, its result is valid if i <= result <= len(s) + 1.

### `lpeg.R ({range})`

Returns a pattern that matches any single character belonging to one of the given ranges. Each `range` is a string xy of length 2, representing all characters with code between the codes of x and y (both inclusive).

As an example, the pattern `lpeg.R("09")` matches any digit, and `lpeg.R("az", "AZ")` matches any ASCII letter.

### `lpeg.S (string)`

Returns a pattern that matches any single character that appears in the given string. (The `S` stands for Set.)

As an example, the pattern `lpeg.S("+-*/")` matches any arithmetic operator.

Note that, if `s` is a character (that is, a string of length 1), then `lpeg.P(s)` is equivalent to `lpeg.S(s)` which is equivalent to `lpeg.R(s..s)`. Note also that both `lpeg.S("")` and `lpeg.R()` are patterns that always fail.

### `lpeg.V (v)`

This operation creates a non-terminal (a variable) for a grammar. The created non-terminal refers to the rule indexed by `v` in the enclosing grammar. (See Grammars for details.)

### `#patt`

Returns a pattern equivalent to &patt in the original PEG notation. This is a pattern that matches only if the input string does match `patt`, but without consuming any input, independently of success or failure.

### `-patt`

Returns a pattern equivalent to !patt in the original PEG notation. This pattern matches only if the input string does not match `patt`. It does not consume any input, independently of success or failure.

As an example, the pattern `-1` matches only the end of string.

### `patt1 + patt2`

Returns a pattern equivalent to an ordered choice of `patt1` and `patt2`. (This is denoted by patt1 / patt2 in the original PEG notation, not to be confused with the `/` operation in LPeg.) It matches either `patt1` or `patt2` (with no backtracking once one of them succeeds). The identity element for this operation is the pattern `lpeg.P(false)`, which always fails.

If both `patt1` and `patt2` are character sets, this operation is equivalent to set union:

    lower = lpeg.R("az")
    upper = lpeg.R("AZ")
    letter = lower + upper

### `patt1 - patt2`

Returns a pattern equivalent to !patt2 patt1. This pattern asserts that the input does not match `patt2` and then matches `patt1`.

If both `patt1` and `patt2` are character sets, this operation is equivalent to set difference. Note that `-patt` is equivalent to `"" - patt` (or `0 - patt`). If `patt` is a character set, `1 - patt` is its complement.

### `patt1 * patt2`

Returns a pattern that matches `patt1` and then matches `patt2`, starting where `patt1` finished. The identity element for this operation is the pattern `lpeg.P(true)`, which always succeeds.

(LPeg uses the `*` operator [instead of the more obvious `..`] both because it has the right priority and because in formal languages it is common to use a dot for denoting concatenation.)

### `patt^n`

If `n` is nonnegative, this pattern is equivalent to pattn patt*. It matches at least `n` occurrences of `patt`.

Otherwise, when `n` is negative, this pattern is equivalent to (patt?)-n. That is, it matches at most `-n` occurrences of `patt`.

In particular, `patt^0` is equivalent to patt*, `patt^1` is equivalent to patt+, and `patt^-1` is equivalent to patt? in the original PEG notation.

In all cases, the resulting pattern is greedy with no backtracking. That is, it matches only the longest possible sequence of matches for `patt`.

## Grammars

With the use of Lua variables, it is possible to define patterns incrementally, with each new pattern using previously defined ones. However, this technique does not allow the definition of recursive patterns. For recursive patterns, we need real grammars.

LPeg represents grammars with tables, where each entry is a rule.

The call `lpeg.V(v)` creates a pattern that represents the nonterminal (or variable) with index `v` in a grammar. Because the grammar still does not exist when this function is evaluated, the result is an open reference to the respective rule.

A table is fixed when it is converted to a pattern (either by calling `lpeg.P` or by using it wherein a pattern is expected). Then every open reference created by `lpeg.V(v)` is corrected to refer to the rule indexed by `v` in the table.

When a table is fixed, the result is a pattern that matches its initial rule. The entry with index 1 in the table defines its initial rule. If that entry is a string, it is assumed to be the name of the initial rule. Otherwise, LPeg assumes that the entry 1 itself is the initial rule.

As an example, the following grammar matches strings of a's and b's that have the same number of a's and b's:

    equalcount = lpeg.P{
      "S";   -- initial rule name
      S = "a" * lpeg.V"B" + "b" * lpeg.V"A" + "",
      A = "a" * lpeg.V"S" + "b" * lpeg.V"A" * lpeg.V"A",
      B = "b" * lpeg.V"S" + "a" * lpeg.V"B" * lpeg.V"B",
    } * -1

## Captures

Captures specify what a match operation should return (the so called semantic information). LPeg offers several kinds of captures, which build values based on matches and combine them to create new values.

The following table summarizes the basic captures:

Operation | What is Captured
`lpeg.C(patt)` | the match for `patt`
`lpeg.Cc(value)` | the given value (matches the empty string)
`lpeg.Cp()` | the current position (matches the empty string)
`lpeg.Cs(patt)` | the match for `patt` with nested captures replacing their matches
`lpeg.Ct(patt)` | a table with all captures from `patt`
`lpeg.Ca(patt)` | an accumulation (or folding) of the captures from `patt`
`patt / function` | the returns of `function` applied to the captures of `patt`
`patt / table` | `table[c]`, where `c` is the (first) capture of `patt`
`patt / string` | `string`, with some marks replaced by captures of `patt`

A capture pattern captures a value every time it succeeds. For instance, a capture inside a loop generates as many values as matched by the loop. A capture generates a value only when it succeeds. For instance, the pattern `lpeg.C(lpeg.P'a'^-1)` captures the empty string when there is no `'a'` (because the pattern `'a'?` succeeds), while the pattern `lpeg.C('a')^-1` does not capture any value when there is no `'a'` (because the pattern `'a'` fails).

### `lpeg.C (patt)`

Creates a simple capture, which captures the substring of the subject that matches `patt`. The captured value is a string. If `patt` has other captures, their values are returned after this one.

### `lpeg.Ca (patt)`

Creates an accumulator capture. This capture assumes that `patt` should produce at least one captured value of any kind, which becomes the initial value of an accumulator. Pattern `patt` then may produce zero or more function captures. Each of these functions in these captures is called having the accumulator as its first argument (followed by any other arguments provided by its own pattern), and the value returned by the function becomes the new value of the accumulator. The final value of this accumulator is the sole result of the whole capture.

As an example, the following pattern matches a list of numbers separated by commas and returns their addition:

    -- matches a numeral and captures its value
    local number = lpeg.R"09"^1 / tonumber

    -- auxiliary function to add two numbers
    local function add (acc, newvalue) return acc + newvalue end

    list = lpeg.Ca(number * ("," * number / add)^0)

    -- example of use
    print(list:match("10,30,43"))   --> 83

### `lpeg.Cc (value)`

Creates a constant capture. This pattern matches the empty string and produces `value` as its captured value.

### `lpeg.Cs (patt)`

Creates a substitution capture, which captures the substring of the subject that matches `patt`, with substitutions. For any capture inside `patt`, the substring that matched the capture is replaced by the capture value (which should be a string). The capture values from `patt` are not returned independently (only as substrings in the resulting string).

### `lpeg.Cp ()`

Creates a position capture. It matches the empty string and captures the position in the subject where the match occurs. The captured value is a number.

### `lpeg.Ct (patt)`

Creates a table capture. This capture creates a table and puts all captures made by `patt` inside this table in successive integer keys, starting at 1.

The captured value is only this table. The captures made by `patt` are not returned independently (only as table elements).

### `patt / function`

Creates a function capture. It calls the given function passing all captures made by `patt` as arguments, or the whole match if `patt` made no capture. The values returned by the function are the final values of the capture. (This capture may create multiple values.) In particular, if `function` returns no value, there is no captured value; everything works as if there was no capture.

### `patt / string`

Creates a string capture. It creates a capture string based on `string`. The captured value is a copy of `string`, except that the character `%` works as an escape character: any sequence in `string` of the form `%n`, with n between 1 and 9, stands for the match of the n-th capture in `patt`. (Currently these nested captures can be only simple captures.) The sequence `%0` stands for the whole match. The sequence `%%` stands for a single `%`.

### `patt / table`

Creates a query capture. It indexes the given table using as key the value of the first capture of `patt`, or the whole match if `patt` made no capture. The value at that index is the final value of the capture.

If the table does not have that key, there is no captured value. Everything works as if there was no capture.

## Some Examples

### Splitting a String

The following code splits a string using a given pattern `sep` as a separator:

    function split (s, sep)
      sep = lpeg.P(sep)
      local elem = lpeg.C((1 - sep)^0)
      local p = elem * (sep * elem)^0
      return lpeg.match(p, s)
    end

First the function ensures that `sep` is a proper pattern. The pattern `elem` is a repetition of zero of more arbitrary characters as long as there is not a match against the separator. It also captures its result. The pattern `p` matches a list of elements separated by `sep`.

If the split results in too many values, it may overflow the maximum number of values that can be returned by a Lua function. In this case, we should collect these values in a table:

    function split (s, sep)
      sep = lpeg.P(sep)
      local elem = lpeg.C((1 - sep)^0)
      local p = lpeg.Ct(elem * (sep * elem)^0)   -- make a table capture
      return lpeg.match(p, s)
    end

### Searching for a Pattern

The primitive `match` works only in anchored mode. If we want to find a pattern anywhere in a string, we must write a pattern that matches anywhere.

Because patterns are composable, we can write a function that, given any arbitrary pattern `p`, returns a new pattern that searches for `p` anywhere in a string. There are several ways to do the search. one way is like this:

    function anywhere (p)
      return lpeg.P{ p + 1 * lpeg.V(1) }
    end

This grammar has a straight reading: it matches `p` or skips one character and tries again.

If we want to know where the pattern is in the string (instead of knowing only that it is there somewhere), we can add position captures to the pattern:

    local I = lpeg.Cp()
    function anywhere (p)
      return lpeg.P{ I * p * I + 1 * lpeg.V(1) }
    end

Another option for the search is like this:

    local I = lpeg.Cp()
    function anywhere (p)
      return (1 - lpeg.P(p))^0 * I * p * I
    end

Again the pattern has a straight reading: it skips as many characters as possible while not matching `p`, and then matches `p` (plus appropriate captures).

If we want to look for a pattern only at word boundaries, we can use the following transformer:

    local wordletter = lpeg.R("AZ", "az")

    function atwordboundary (p)
      return lpeg.P{
        [1] = p + wordletter^0 * (1 - wordletter)^1 * lpeg.V(1)
      }
    end

### Balanced Parentheses

The following pattern matches only strings with balanced parentheses:

    b = lpeg.P{ "(" * ((1 - lpeg.S"()") + lpeg.V(1))^0 * ")" }

Reading the first (and only) rule of the given grammar, we have that a balanced string is an open parenthesis, followed by zero or more repetitions of either a non-parenthesis character or a balanced string (`lpeg.V(1)`), followed by a closing parenthesis.

### Global Substitution

The next example does a job somewhat similar to `string.gsub`. It receives a pattern and a replacement value, and substitutes the replacement value for all occurrences of the pattern in a given string:

    function gsub (s, patt, repl)
      patt = lpeg.P(patt)
      patt = lpeg.Cs((patt / repl + 1)^0)
      return lpeg.match(patt, s)
    end

As in `string.gsub`, the replacement value can be a string, a function, or a table.

### Comma-Separated Values (CSV)

This example breaks a string into comma-separated values, returning all fields:

    local field = '"' * lpeg.Cs(((lpeg.P(1) - '"') + lpeg.P'""' / '"')^0) * '"' +
                        lpeg.C((1 - lpeg.S',\n"')^0)

    local record = field * (',' * field)^0 * (lpeg.P'\n' + -1)

    function csv (s)
      return lpeg.match(record, s)
    end

A field is either a quoted field (which may contain any character except an individual quote, which may be written as two quotes that are replaced by one) or an unquoted field (which cannot contain commas, newlines, or quotes). A record is a list of fields separated by commas, ending with a newline or the string end (-1).

### Lua's Long Strings

This example matches long strings in Lua. Probably there is no pure PEG grammar that matches that syntax (as there is no Context-Free Grammar for it), so we resort to Lua:

    local start = "[" * lpeg.P"="^0 * "["
    longstring = lpeg.P(function (s, i)
      local l = lpeg.match(start, s, i)
      if not l then return nil end
      local p = lpeg.P("]" .. string.rep("=", l - i - 2) .. "]")
      p = (1 - p)^0 * p
      return lpeg.match(p, s, l)
    end)

The function first checks whether there is a long string starting at position `i`, using the `start` pattern. If so, it builds a pattern `p` that matches an ending bracket with the correct number of equal signs and then looks for this pattern.

Lua-function patterns are not efficient. So, it may be worth the use of an and-predicate to pre-analyze the subject before calling Lua:

    longstring = #("[" * lpeg.S"[=") * longstring

In this example, the pattern only calls Lua if the current position matches an open bracket followed by either another open bracket or an equal sign (so that the only valid token there is a long string).

### UTF-8 and Latin 1

It is not difficult to use LPeg to convert a string from utf-8 encoding to Latin 1 (ISO 8859-1):

    -- convert a two-byte utf8 sequence to a Latin 1 character
    local function f2 (s)
      local c1, c2 = string.byte(s, 1, 2)
      return string.char(c1 * 64 + c2 - 12416)
    end

    local utf8 = lpeg.R("\0\127")
               + lpeg.R("\194\195") * lpeg.R("\128\191") / f2

    local decode_pattern = lpeg.Cs(utf8^0) * -1

In this code, the definition of utf-8 is already restricted to the Latin 1 range (from 0 to 255). Any encoding outside this range (as well as any invalid encoding) will not match that pattern.

As the definition of `decode_pattern` demands that the pattern matches the whole input (because of the -1 at its end), any invalid string will simply fail to match, without any useful information about the problem. We can improve this situation redefining `decode_pattern` as follows:

    local function er (_, i) error("invalid encoding at position " .. i) end

    local decode_pattern = lpeg.Cs(utf8^0) * (-1 + lpeg.P(er))

Now, if the pattern `utf8^0` stops before the end of the string, an appropriate error function is called.

### UTF-8 and Unicode

We can extend the previous patterns to handle all Unicode code points. Of course, we cannot translate them to Latin 1 or any other one-byte encoding. Instead, our translation results in a array with the code points represented as numbers. The full code is here:

    -- decode a two-byte utf8 sequence
    local function f2 (s)
      local c1, c2 = string.byte(s, 1, 2)
      return c1 * 64 + c2 - 12416
    end

    -- decode a three-byte utf8 sequence
    local function f3 (s)
      local c1, c2, c3 = string.byte(s, 1, 3)
      return (c1 * 64 + c2) * 64 + c3 - 925824
    end

    -- decode a four-byte utf8 sequence
    local function f4 (s)
      local c1, c2, c3, c4 = string.byte(s, 1, 4)
      return ((c1 * 64 + c2) * 64 + c3) * 64 + c4 - 63447168
    end

    local cont = lpeg.R("\128\191")   -- continuation byte

    local utf8 = lpeg.R("\0\127") / string.byte
               + lpeg.R("\194\223") * cont / f2
               + lpeg.R("\224\239") * cont * cont / f3
               + lpeg.R("\240\244") * cont * cont * cont / f4

    local decode_pattern = lpeg.Ct(utf8^0) * -1

### Arithmetic Expressions

This example is a complete parser and evaluator for simple arithmetic expressions. We write it in two styles. The first approach first builds a syntax tree and then traverses this tree to compute the expression value:

    -- Lexical Elements
    local Space = lpeg.S(" \n\t")^0
    local Number = lpeg.C(lpeg.P"-"^-1 * lpeg.R("09")^1) * Space
    local FactorOp = lpeg.C(lpeg.S("+-")) * Space
    local TermOp = lpeg.C(lpeg.S("*/")) * Space
    local Open = "(" * Space
    local Close = ")" * Space

    -- Grammar
    local V = lpeg.V
    local Exp, Term, Factor = 1, 2, 3
    G = lpeg.P{ "Exp",
      Exp = lpeg.Ct(V"Factor" * (FactorOp * V"Factor")^0);
      Factor = lpeg.Ct(V"Term" * (TermOp * V"Term")^0);
      Term = Number + Open * V"Exp" * Close;
    }

    G = Space * G * -1

    -- Evaluator
    function eval (x)
      if type(x) == "string" then
        return tonumber(x)
      else
        local op1 = eval(x[1])
        for i = 2, #x, 2 do
          local op = x[i]
          local op2 = eval(x[i + 1])
          if (op == "+") then op1 = op1 + op2
          elseif (op == "-") then op1 = op1 - op2
          elseif (op == "*") then op1 = op1 * op2
          elseif (op == "/") then op1 = op1 / op2
          end
        end
        return op1
      end
    end

    -- Parser/Evaluator
    function evalExp (s)
      local t = lpeg.match(G, s)
      if not t then error("syntax error", 2) end
      return eval(t)
    end

    -- small example
    print(evalExp"3 + 5*9 / (1+1) - 12")

The second style computes the expression value on the fly, without building the syntax tree. The following grammar takes this approach. (It assumes the same lexical elements as before.)

    -- Auxiliary function
    function eval (v1, op, v2)
      if (op == "+") then return v1 + v2
      elseif (op == "-") then return v1 - v2
      elseif (op == "*") then return v1 * v2
      elseif (op == "/") then return v1 / v2
      end
    end

    -- Grammar
    local V = lpeg.V
    G = lpeg.P{ "Exp",
      Exp = lpeg.Ca(V"Factor" * (FactorOp * V"Factor" / eval)^0);
      Factor = lpeg.Ca(V"Term" * (TermOp * V"Term" / eval)^0);
      Term = Number / tonumber + Open * V"Exp" * Close;
    }

    -- small example
    print(lpeg.match(G, "3 + 5*9 / (1+1) - 12"))

Note the use of the accumulator capture. To compute the value of an expression, the accumulator starts with the value of the first factor, and then applies `eval` over the accumulator, the operator, and the new factor for each repetition.

## The `re` Module

The `re` Module (provided by file `re.lua` in the distribution) supports a conventional regular-expression syntax for pattern construction.

If offers two functions:

- `re.compile(string)` compiles the given string and returns an equivalent LPeg pattern. The given string may define either an expression or a grammar. (It memoizes its results, so that there is no penalty in calling it multiple times with the same string.)

- `re.match(subject,pattern)` compiles the given pattern (a string) and matches it against the given subject.

The syntax for patterns for this module follows closely the original PEG notation. Like in the original PEG notation, spaces have no meaning, and literal strings must be enclosed between quotes (but `re` allows both double and single quotes). Anywhere any special character may be escaped by a prefixed `%` (not `\` like in original PEG).

Like traditional regular expressions (but unlike original PEG), a character class may start with a `~` to complement its meaning. Curly brackets around a pattern capture its corresponding match. Empty curly brackets capture the current position. (Other captures are not supported in this version.)

As a simple example, the following call will produce the same pattern produced by the Lua expression in the balanced parentheses example:

    b = re.compile[[  balanced <- '(' ([^()] / balanced)* ')'  ]]

## Download

LPeg source code.

## License

Copyright © 2007 Lua.org, PUC-Rio.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

$Id: doc.html,v 1.33 2007/04/12 14:38:05 roberto Exp $