Syntax

Both abstract and concrete syntaxes of parsing expressions are seen in the literature, and in this article. The abstract syntax is essentially a mathematical formula and primarily used in theoretical contexts, whereas concrete syntax parsing expressions could be used directly to control a parser. The primary concrete syntax is that defined by Ford^[1]: Fig.1, although many tools have their own dialect of this. Other tools^[3] can be closer to using a programming-language native encoding of abstract syntax parsing expressions as their concrete syntax.

  "terminal" Nonterminal 'another terminal'

  Identifier LEFTARROW Expression

Example

This is a PEG that recognizes mathematical formulas that apply the basic five operations to non-negative integers.

Expr    ← Sum
Sum     ← Product (('+' / '-') Product)*
Product ← Power (('*' / '/') Power)*
Power   ← Value ('^' Power)?
Value   ← [0-9]+ / '(' Expr ')'

In the above example, the terminal symbols are characters of text, represented by characters in single quotes, such as '(' and ')'. The range [0-9] is a shortcut for the ten characters from '0' to '9'. (This range syntax is the same as the syntax used by regular expressions.) The nonterminal symbols are the ones that expand to other rules: Value, Power, Product, Sum, and Expr. Note that rules Sum and Product don't lead to desired left-associativity of these operations (they don't deal with associativity at all, and it has to be handled in post-processing step after parsing), and the Power rule (by referring to itself on the right) results in desired right-associativity of exponent. Also note that a rule like Sum ← Sum (('+' / '-') Product)? (with intention to achieve left-associativity) would cause infinite recursion, so it cannot be used in practice even though it can be expressed in the grammar.

Semantics

The fundamental difference between context-free grammars and parsing expression grammars is that the PEG's choice operator is ordered. If the first alternative succeeds, the second alternative is ignored. Thus ordered choice is not commutative, unlike unordered choice as in context-free grammars. Ordered choice is analogous to soft cut operators available in some logic programming languages.

The consequence is that if a CFG is transliterated directly to a PEG, any ambiguity in the former is resolved by deterministically picking one parse tree from the possible parses. By carefully choosing the order in which the grammar alternatives are specified, a programmer has a great deal of control over which parse tree is selected.

Parsing expression grammars also add the and- and not- syntactic predicates. Because they can use an arbitrarily complex sub-expression to "look ahead" into the input string without actually consuming it, they provide a powerful syntactic lookahead and disambiguation facility, in particular when reordering the alternatives cannot specify the exact parse tree desired.

Operational interpretation of parsing expressions

Each nonterminal in a parsing expression grammar essentially represents a parsing function in a recursive descent parser, and the corresponding parsing expression represents the "code" comprising the function. Each parsing function conceptually takes an input string as its argument, and yields one of the following results:

• success, in which the function may optionally move forward or consume one or more characters of the input string supplied to it, or
• failure, in which case no input is consumed.

An atomic parsing expression consisting of a single terminal (i.e. literal) succeeds if the first character of the input string matches that terminal, and in that case consumes the input character; otherwise the expression yields a failure result. An atomic parsing expression consisting of the empty string always trivially succeeds without consuming any input.

An atomic parsing expression consisting of a nonterminal A represents a recursive call to the nonterminal-function A. A nonterminal may succeed without actually consuming any input, and this is considered an outcome distinct from failure.

The sequence operator e₁ e₂ first invokes e₁, and if e₁ succeeds, subsequently invokes e₂ on the remainder of the input string left unconsumed by e₁, and returns the result. If either e₁ or e₂ fails, then the sequence expression e₁ e₂ fails (consuming no input).

The choice operator e₁ / e₂ first invokes e₁, and if e₁ succeeds, returns its result immediately. Otherwise, if e₁ fails, then the choice operator backtracks to the original input position at which it invoked e₁, but then calls e₂ instead, returning e₂'s result.

The zero-or-more, one-or-more, and optional operators consume zero or more, one or more, or zero or one consecutive repetitions of their sub-expression e, respectively. Unlike in context-free grammars and regular expressions, however, these operators always behave greedily, consuming as much input as possible and never backtracking. (Regular expression matchers may start by matching greedily, but will then backtrack and try shorter matches if they fail to match.) For example, the expression a* will always consume as many a's as are consecutively available in the input string, and the expression (a* a) will always fail because the first part (a*) will never leave any a's for the second part to match.

The and-predicate expression &e invokes the sub-expression e, and then succeeds if e succeeds and fails if e fails, but in either case never consumes any input.

The not-predicate expression !e succeeds if e fails and fails if e succeeds, again consuming no input in either case.

More examples

The following recursive rule matches standard C-style if/then/else statements in such a way that the optional "else" clause always binds to the innermost "if", because of the implicit prioritization of the '/' operator. (In a context-free grammar, this construct yields the classic dangling else ambiguity.)

S ← 'if' C 'then' S 'else' S / 'if' C 'then' S

The following recursive rule matches Pascal-style nested comment syntax, (* which can (* nest *) like this *). Recall that . matches any single character.

C     ← Begin N* End
Begin ← '(*'
End   ← '*)'
N     ← C / (!Begin !End .)

The parsing expression foo &(bar) matches and consumes the text "foo" but only if it is followed by the text "bar". The parsing expression foo !(bar) matches the text "foo" but only if it is not followed by the text "bar". The expression !(a+ b) a matches a single "a" but only if it is not part of an arbitrarily long sequence of a's followed by a b.

The parsing expression ('a'/'b')* matches and consumes an arbitrary-length sequence of a's and b's. The production rule S ← 'a' ''S''? 'b' describes the simple context-free "matching language" ${\displaystyle \{a^{n}b^{n}:n\geq 1\}}$. The following parsing expression grammar describes the classic non-context-free language ${\displaystyle \{a^{n}b^{n}c^{n}:n\geq 1\}}$:

S ← &(A 'c') 'a'+ B !.
A ← 'a' A? 'b'
B ← 'b' B? 'c'

Bottom-up PEG parsing

A pika parser^[7] uses dynamic programming to apply PEG rules bottom-up and right to left, which is the inverse of the normal recursive descent order of top-down, left to right. Parsing in reverse order solves the left recursion problem, allowing left-recursive rules to be used directly in the grammar without being rewritten into non-left-recursive form, and also conveys optimal error recovery capabilities upon the parser, which historically proved difficult to achieve for recursive descent parsers.

No compilation required

Many parsing algorithms require a preprocessing step where the grammar is first compiled into an opaque executable form, often some sort of automaton. Parsing expressions can be executed directly (even if it is typically still advisable to transform the human-readable PEGs shown in this article into a more native format, such as S-expressions, before evaluating them).

Compared to regular expressions

Compared to pure regular expressions (i.e., describing a language recognisable using a finite automaton), PEGs are vastly more powerful. In particular they can handle unbounded recursion, and so match parentheses down to an arbitrary nesting depth; regular expressions can at best keep track of nesting down to some fixed depth, because a finite automaton (having a finite set of internal states) can only distinguish finitely many different nesting depths. In more theoretical terms, ${\displaystyle \{a^{n}b^{n}\}_{n\geqslant 0}}$ (the language of all strings of zero or more ${\displaystyle a}$'s, followed by an equal number of ${\displaystyle b}$s) is not a regular language, but it is easily seen to be a parsing expression language, matched by the grammar

start   ← AB !.
AB ← ('a' AB 'b')?

Here AB !. is the starting expression. The !. part enforces that the input ends after the AB, by saying “there is no next character”; unlike regular expressions, which have magic constraints $ or \Z for this, parsing expressions can express the end of input using only the basic primitives.

The *, +, and ? of parsing expressions are similar to those in regular expressions, but a difference is that these operate strictly in a greedy mode. This is ultimately due to / being an ordered choice. A consequence is that something can match as a regular expression which does not match as parsing expression:

[ab]?[bc][cd]

is both a valid regular expression and a valid parsing expression. As regular expression, it matches bc, but as parsing expression it does not match, because the [ab]? will match the b, then [bc] will match the c, leaving nothing for the [cd], so at that point matching the sequence fails. "Trying again" with having [ab]? match the empty string is explicitly against the semantics of parsing expressions; this is not an edge case of a particular matching algorithm, instead it is the sought behaviour.

Even regular expressions that depend on nondeterminism can be compiled into a parsing expression grammar, by having a separate nonterminal for every state of the corresponding DFA and encoding its transition function into the definitions of these nonterminals —

A  ← 'x' B / 'y' C

is effectively saying "from state A transition to state B if the next character is x, but to state C if the next character is y" — but this works because nondeterminism can be eliminated within the realm of regular languages. It would not make use of the parsing expression variants of the repetition operations.

Compared to context-free grammars

PEGs can comfortably be given in terms of characters, whereas context-free grammars (CFGs) are usually given in terms of tokens, thus requiring an extra step of tokenisation in front of parsing proper.^[8] An advantage of not having a separate tokeniser is that different parts of the language (for example embedded mini-languages) can easily have different tokenisation rules.

In the strict formal sense, PEGs are likely incomparable to CFGs, but practically there are many things that PEGs can do which pure CFGs cannot, whereas it is difficult to come up with examples of the contrary. In particular PEGs can be crafted to natively resolve ambiguities, such as the "dangling else" problem in C, C++, and Java, whereas CFG-based parsing often needs a rule outside of the grammar to resolve them. Moreover any PEG can be parsed in linear time by using a packrat parser, as described above, whereas parsing according to a general CFG is asymptotically equivalent^[9] to boolean matrix multiplication (thus likely between quadratic and cubic).

One classical example of a formal language which is provably not context-free is the language ${\displaystyle \{a^{n}b^{n}c^{n}\}_{n\geqslant 0}}$: an arbitrary number of ${\displaystyle a}$s are followed by an equal number of ${\displaystyle b}$s, which in turn are followed by an equal number of ${\displaystyle c}$s. This, too, is a parsing expression language, matched by the grammar

start ← AB 'c'*
AB    ← 'a' AB 'b' / &(BC !.)
BC    ← ('b' BC 'c')?

For AB to match, the first stretch of ${\displaystyle a}$s must be followed by an equal number of ${\displaystyle b}$s, and in addition BC has to match where the ${\displaystyle a}$s switch to ${\displaystyle b}$s, which means those ${\displaystyle b}$s are followed by an equal number of ${\displaystyle c}$s.

Memory consumption

PEG parsing is typically carried out via packrat parsing, which uses memoization^[10][11] to eliminate redundant parsing steps. Packrat parsing requires internal storage proportional to the total input size, rather than to the depth of the parse tree as with LR parsers. Whether this is a significant difference depends on circumstances; if parsing is a service provided as a function then the parser will have stored the full parse tree up until returning it, and already that parse tree will typically be of size proportional to the total input size. If parsing is instead provided as a generator then one might get away with only keeping parts of the parse tree in memory, but the feasibility of this depends on the grammar. A parsing expression grammar can be designed so that only after consuming the full input will the parser discover that it needs to backtrack to the beginning,^[12] which again could require storage proportional to total input size.

For recursive grammars and some inputs, the depth of the parse tree can be proportional to the input size,^[13] so both an LR parser and a packrat parser will appear to have the same worst-case asymptotic performance. However in many domains, for example hand-written source code, the expression nesting depth has an effectively constant bound quite independent of the length of the program, because expressions nested beyond a certain depth tend to get refactored. When it is not necessary to keep the full parse tree, a more accurate analysis would take the depth of the parse tree into account separately from the input size.^[14]

Computational model

In order to attain linear overall complexity, the storage used for memoization must furthermore provide amortized constant time access to individual data items memoized. In practice that is no problem — for example a dynamically sized hash table attains this – but that makes use of pointer arithmetic, so it presumes having a random-access machine. Theoretical discussions of data structures and algorithms have an unspoken tendency to presume a more restricted model (possibly that of lambda calculus, possibly that of Scheme), where a sparse table rather has to be built using trees, and data item access is not constant time. Traditional parsing algorithms such as the LL parser are not affected by this, but it becomes a penalty for the reputation of packrat parsers: they rely on operations of seemingly ill repute.

Viewed the other way around, this says packrat parsers tap into computational power readily available in real life systems, that older parsing algorithms do not understand to employ.

Indirect left recursion

A PEG is called well-formed^[1] if it contains no left-recursive rules, i.e., rules that allow a nonterminal to expand to an expression in which the same nonterminal occurs as the leftmost symbol. For a left-to-right top-down parser, such rules cause infinite regress: parsing will continually expand the same nonterminal without moving forward in the string. Therefore, to allow packrat parsing, left recursion must be eliminated.

Sum   → Term | Sum '+' Term | Sum '-' Term
Args  → Arg | Arg ',' Args

Sum   ← Term ( '+' Term / '-' Term )*
Args  ← Arg ( ',' Arg )*

Value   ← [0-9.]+ / '(' Expr ')'
Product ← Expr (('*' / '/') Expr)+
Sum     ← Expr (('+' / '-') Expr)+
Expr    ← Sum / Product / Value

string-of-a ← string-of-a 'a' | 'a'

string-of-a ← 'a'+

The midpoint problem

PEG packrat parsers cannot recognize some unambiguous nondeterministic CFG rules, such as the following:^[2]

S ← 'x' S 'x' | 'x'

Neither LL(k) nor LR(k) parsing algorithms are capable of recognizing this example. However, this grammar can be used by a general CFG parser like the CYK algorithm. However, the language in question can be recognised by all these types of parser, since it is in fact a regular language (that of strings of an odd number of x's).

It is instructive to work out exactly what a PEG parser does when attempting to match

S ← 'x' S 'x' / 'x'

against the string xxxxxq. As expected, it recursively tries to match the nonterminal S at increasing positions in this string, until failing the match against the q, and after that begins to backtrack. This goes as follows:

Position:  123456
  String:  xxxxxq
 Results:       ↑ Pos.6: Neither branch of S matches
               ↑ Pos.5: First branch of S fails, second branch succeeds, yielding match of length 1.
              ↑ Pos.4: First branch of S fails, second branch succeeds, yielding match of length 1.
             ↑ Pos.3: First branch of S succeeds, yielding match of length 3.
            ↑ Pos.2: First branch of S fails, because after the S match at 3 comes a q.
              Second branch succeeds, yielding match of length 1.
           ↑ Pos.1: First branch of S succeeds, yielding match of length 3.

Matching against a parsing expression is greedy, in the sense that the first success encountered is the only one considered. Even if locally the choices are ordered longest first, there is no guarantee that this greedy matching will find the globally longest match.

Ambiguity detection and influence of rule order on language that is matched

LL(k) and LR(k) parser generators will fail to complete when the input grammar is ambiguous. This is a feature in the common case that the grammar is intended to be unambiguous but is defective. A PEG parser generator will resolve unintended ambiguities earliest-match-first, which may be arbitrary and lead to surprising parses.

The ordering of productions in a PEG grammar affects not only the resolution of ambiguity, but also the language matched. For example, consider the first PEG example in Ford's paper^[1] (example rewritten in pegjs.org/online notation, and labelled ${\displaystyle G_{1}}$ and ${\displaystyle G_{2}}$):

• ${\displaystyle G_{1}}$:  A = "a" "b" / "a"
• ${\displaystyle G_{2}}$:  A = "a" / "a" "b"

Ford notes that The second alternative in the latter PEG rule will never succeed because the first choice is always taken if the input string ... begins with 'a'..^[1] Specifically, ${\displaystyle L(G_{1})}$ (i.e., the language matched by ${\displaystyle G_{1}}$) includes the input "ab", but ${\displaystyle L(G_{2})}$ does not. Thus, adding a new option to a PEG grammar can remove strings from the language matched, e.g. ${\displaystyle G_{2}}$ is the addition of a rule to the single-production grammar A = "a" "b", which contains a string not matched by ${\displaystyle G_{2}}$. Furthermore, constructing a grammar to match ${\displaystyle L(G_{1})\cup L(G_{2})}$ from PEG grammars ${\displaystyle G_{1}}$ and ${\displaystyle G_{2}}$ is not always a trivial task. This is in stark contrast to CFG's, in which the addition of a new production cannot remove strings (though, it can introduce problems in the form of ambiguity), and a (potentially ambiguous) grammar for ${\displaystyle L(G_{1})\cup L(G_{2})}$ can be constructed

S → start(G1) | start(G2)

Undecidability of emptiness

In stark contrast to the case for context-free grammars, it is not possible to generate elements of a parsing expression language from its grammar. Indeed, it is algorithmically undecidable whether the language recognised by a parsing expression grammar is empty! One reason for this is that any instance of the Post correspondence problem reduces to an instance of the problem of deciding whether a parsing expression language is empty.

Recall that an instance of the Post correspondence problem consists of a list ${\displaystyle (\alpha _{1},\beta _{1}),(\alpha _{2},\beta _{2}),\dotsc ,(\alpha _{n},\beta _{n})}$ of pairs of strings (of terminal symbols). The problem is to determine whether there exists a sequence ${\displaystyle \{k_{i}\}_{i=1}^{m}}$ of indices in the range ${\displaystyle \{1,\dotsc ,n\}}$ such that ${\displaystyle \alpha _{k_{1}}\alpha _{k_{2}}\dotsb \alpha _{k_{m}}=\beta _{k_{1}}\beta _{k_{2}}\dotsb \beta _{k_{m}}}$. To reduce this to a parsing expression grammar, let ${\displaystyle \gamma _{0},\gamma _{1},\dotsc ,\gamma _{n}}$ be arbitrary pairwise distinct equally long strings of terminal symbols (already with ${\displaystyle 2}$ distinct symbols in the terminal symbol alphabet, length ${\displaystyle \lceil \log _{2}(n+1)\rceil }$ suffices) and consider the parsing expression grammar

$${\displaystyle {\begin{aligned}S&\leftarrow \&(A\,!.)\&(B\,!.)(\gamma _{1}/\dotsb /\gamma _{n})^{+}\gamma _{0}\\A&\leftarrow \gamma _{0}/\gamma _{1}A\alpha _{1}/\dotsb /\gamma _{n}A\alpha _{n}\\B&\leftarrow \gamma _{0}/\gamma _{1}B\beta _{1}/\dotsb /\gamma _{n}B\beta _{n}\end{aligned}}}$$

Any string matched by the nonterminal ${\displaystyle A}$ has the form ${\displaystyle \gamma _{k_{m}}\dotsb \gamma _{k_{2}}\gamma _{k_{1}}\gamma _{0}\alpha _{k_{1}}\alpha _{k_{2}}\dotsb \alpha _{k_{m}}}$ for some indices ${\displaystyle k_{1},k_{2},\dotsc ,k_{m}}$. Likewise any string matched by the nonterminal ${\displaystyle B}$ has the form ${\displaystyle \gamma _{k_{m}}\dotsb \gamma _{k_{2}}\gamma _{k_{1}}\gamma _{0}\beta _{k_{1}}\beta _{k_{2}}\dotsb \beta _{k_{m}}}$. Thus any string matched by ${\displaystyle S}$ will have the form ${\displaystyle \gamma _{k_{m}}\dotsb \gamma _{k_{2}}\gamma _{k_{1}}\gamma _{0}\rho }$ where ${\displaystyle \rho =\alpha _{k_{1}}\alpha _{k_{2}}\dotsb \alpha _{k_{m}}=\beta _{k_{1}}\beta _{k_{2}}\dotsb \beta _{k_{m}}}$.