PLY (Python Lex-Yacc) is an open‑source parsing tool for the Python programming language that implements the lexical analysis and parsing components of the classic Unix tools lex and yacc. Written entirely in Python, PLY provides a straightforward way to generate lexical analyzers (tokenizers) and parsers using Python functions and data structures.
Key Features
| Feature | Description |
|---|---|
| Lexical analysis | Defines tokens with regular expressions using Python functions or strings, mirroring the interface of the original lex utility. |
| LR(1) parsing | Implements an LALR(1) parser generator compatible with yacc specifications, allowing the construction of deterministic bottom‑up parsers. |
| Pure‑Python | No external C or compiled extensions are required; PLY runs on any platform that supports Python 2.7, Python 3.5+, and PyPy. |
| Error handling | Provides detailed syntax error reporting and facilities for custom error recovery strategies. |
| Extensibility | Allows integration with other Python libraries and can be combined with abstract syntax tree (AST) frameworks. |
| Documentation | Includes a comprehensive user guide, examples, and a test suite that demonstrates typical usage patterns. |
| License | Distributed under the BSD 3‑Clause License, permitting unrestricted use, modification, and redistribution. |
History and Development
PLY was created by David M. Beazley, a professor of computer science known for his contributions to Python education and software development tools. The initial public release occurred in 2004, with subsequent updates addressing compatibility with newer Python versions, bug fixes, and enhancements to error diagnostics. The project is hosted on GitHub, where contributors can submit pull requests and report issues.
Typical Use Cases
- Domain‑specific language (DSL) creation – Rapid prototyping of custom scripting languages or configuration file formats.
- Compiler front‑ends – Tokenizing and parsing source code as a first stage in compilation pipelines.
- Data extraction – Parsing structured text files (e.g., log files, custom reports) into Python data structures.
- Educational purposes – Demonstrating parsing concepts and compiler construction techniques in academic settings.
Example Overview
A minimal PLY lexer and parser might be defined as follows:
import ply.lex as lex
import ply.yacc as yacc
# Token list
tokens = ('NUMBER', 'PLUS', 'MINUS')
t_PLUS = r'\+'
t_MINUS = r'-'
t_NUMBER = r'\d+'
t_ignore = ' \t
'
def t_error(t):
print(f"Illegal character '{t.value[0]}'")
t.lexer.skip(1)
lexer = lex.lex()
# Parsing rules
def p_expression_plus(p):
'expression : expression PLUS term'
p[0] = p[1] + p[3]
def p_expression_minus(p):
'expression : expression MINUS term'
p[0] = p[1] - p[3]
def p_expression_term(p):
'expression : term'
p[0] = p[1]
def p_term_number(p):
'term : NUMBER'
p[0] = int(p[1])
def p_error(p):
print("Syntax error")
parser = yacc.yacc()
result = parser.parse("3 + 4 - 2")
print(result) # Outputs: 5
This snippet illustrates the definition of tokens with regular expressions, lexer construction, and the specification of grammar productions using docstring syntax.
Community and Support
PLY remains actively maintained, with the primary repository receiving periodic updates. Users can seek assistance through the project's issue tracker, mailing lists, and community forums such as Stack Overflow, where tags like python-ply are frequently used.
References
- Beazley, D. M. PLY (Python Lex-Yacc) Documentation. Available at https://github.com/dabeaz/ply
- BSD 3‑Clause License text, included with the distribution.
