Grammar Modification Research¶

What you'll learn¶

• How Python's grammar becomes a parser
• Build a tiny parser to understand the pipeline
• What "modifying the grammar" actually involves
• Realistic alternatives that don't fork Python
• Why new syntax is rarely the right answer

From grammar to parser¶

How a grammar definition becomes code that parses source.

Every language is defined by a grammar — formal rules describing valid syntax. Python's grammar lives in Grammar/python.gram in the CPython source, written in PEG (Parsing Expression Grammar) form since Python 3.9. At build time, a parser generator turns that grammar file into the C parser that reads your .py files.

   Grammar/python.gram  ──generator──▶  C parser  ──parses──▶  AST  ──compiles──▶  bytecode
   (the rules)                          (the code)

"Modifying the grammar" means editing those rules and rebuilding CPython — a genuine fork of the language. Before we get there, let's build a tiny parser so the pipeline is concrete.

A recursive-descent parser you can run¶

A small hand-written parser demonstrating the technique.

You can understand grammars by implementing one. Here's a parser+evaluator for a small expression grammar — fully runnable:

import re

# grammar:  expr := term (('+' | '-') term)*
#           term := NUMBER
def tokenize(s: str) -> list[str]:
    return re.findall(r'\d+|[+\-]', s.replace(" ", ""))

def evaluate(s: str) -> int:
    toks = tokenize(s)
    pos = 0
    def term() -> int:
        nonlocal pos
        v = int(toks[pos]); pos += 1
        return v
    def expr() -> int:
        nonlocal pos
        v = term()
        while pos < len(toks) and toks[pos] in "+-":
            op = toks[pos]; pos += 1
            rhs = term()
            v = v + rhs if op == "+" else v - rhs
        return v
    return expr()

print(evaluate("1 + 2 + 3"))     # -> 6
print(evaluate("10 - 3 + 1"))    # -> 8

Output:

6
8

Each grammar rule (expr, term) became a function; the structure of the code mirrors the structure of the grammar. This is recursive descent, the same style CPython's PEG parser uses (generated automatically rather than hand-written). Adding a rule for * and / would mean adding a factor function — that's what "extending a grammar" feels like in the small.

Modifying Python's actual grammar¶

Editing python.gram and regenerating the parser to add syntax.

To add real syntax to Python (say, a new operator or keyword), the steps are:

1. Edit Grammar/python.gram — add or change a PEG rule.
2. Regenerate the parser — run CPython's build, which invokes the parser generator (pegen).
3. Update the AST — new syntax usually needs new AST node types (Grammar/Python.asdl).
4. Update the compiler — teach compile.c how to turn the new AST nodes into bytecode.
5. Rebuild CPython — you now have a forked interpreter that understands your syntax.

Realistic alternatives (no fork needed)¶

Import hooks, AST transforms, and preprocessors instead of forking CPython.

Almost always, you want new behavior, not new syntax — and Python gives you powerful ways to get it without touching the grammar:

• AST transformation at import time — use an import hook to rewrite the AST of modules as they load (see AST Manipulation and Import System). This lets you change semantics while keeping valid Python syntax. Libraries like pytest (assertion rewriting) and MacroPy did exactly this.
• Operator overloading & dunder methods — a huge amount of "custom syntax" is really just __add__, __matmul__, __getitem__, context managers, and decorators. The @ operator was added to the language specifically so libraries like NumPy could express matrix multiply without a fork.
• A DSL parsed at runtime — parse your custom mini-language from strings (like the expression evaluator above), rather than embedding it in Python's grammar.

Practice exercises¶

1. Extend the expression evaluator with * and / at correct precedence (add a factor rule/function).
2. Add parentheses support: factor := NUMBER | '(' expr ')'.
3. Explain, step by step, what breaks in your toolchain if you add a new keyword by forking the grammar.
4. Research how pytest rewrites assert statements via an import hook — and why that needed no grammar change.
5. Describe a feature you might think needs new syntax, and design it instead with operator overloading or a decorator.