Lib/ast.py
cpython 3.14 @ ab2d84fe1023/Lib/ast.py
ast.py is a thin pure-Python layer over the _ast C extension. The C
extension defines all the node types (Module, FunctionDef, Call,
etc.) as Python classes and exposes compile(). ast.py imports those
types with from _ast import * and adds the utilities that operate on the
resulting trees.
The file is organized into four groups: the parse/compile entry points,
the visitor and transformer base classes, the tree-query and printing
helpers (walk, dump, unparse), and the safe expression evaluator
literal_eval. In 3.14 the module also gained _Unparser improvements
for PEP 695 type-parameter syntax and PEP 701 f-string grammar changes.
Map
| Lines | Symbol | Role | gopy |
|---|---|---|---|
| 1-50 | Imports, PyCF_* flag re-exports | Pulls node classes from _ast into the ast namespace; re-exports compile flags used by IDEs and type checkers. | (stdlib pending) |
| 51-150 | parse, compile, literal_eval (stub) | parse calls the built-in compile() with ast.PyCF_ONLY_AST; returns a Module, Expression, or Interactive node depending on mode. | (stdlib pending) |
| 151-250 | NodeVisitor | Base class with visit dispatch and generic_visit fallback; visit calls visit_ClassName if defined, else generic_visit. | (stdlib pending) |
| 251-350 | NodeTransformer | Subclass of NodeVisitor; generic_visit rebuilds child lists in-place, replacing nodes with the return value of their visitor and removing None returns. | (stdlib pending) |
| 351-450 | walk, iter_fields, iter_child_nodes, get_docstring | walk is a BFS over the tree using a deque; iter_fields yields (fieldname, value) pairs from node._fields; get_docstring checks whether the first statement is a Constant string. | (stdlib pending) |
| 451-550 | fix_missing_locations, copy_location, increment_lineno | Location-propagation helpers; fix_missing_locations walks the tree and fills in lineno, col_offset, end_lineno, end_col_offset from the parent where missing. | (stdlib pending) |
| 551-750 | literal_eval | Safe expression evaluator; walks a restricted subset of the AST and raises ValueError for any node that is not a literal. | (stdlib pending) |
| 751-1000 | _Unparser, unparse | Converts an AST back to source text; each visit_X method writes tokens to an internal buffer and manages operator precedence and parenthesization. | (stdlib pending) |
Reading
NodeVisitor.generic_visit (lines 151 to 350)
cpython 3.14 @ ab2d84fe1023/Lib/ast.py#L151-350
class NodeVisitor:
def visit(self, node):
"""Visit a node."""
method = 'visit_' + node.__class__.__name__
visitor = getattr(self, method, self.generic_visit)
return visitor(node)
def generic_visit(self, node):
"""Called if no explicit visitor function exists for a node."""
for field, value in iter_fields(node):
if isinstance(value, list):
for item in value:
if isinstance(item, AST):
self.visit(item)
elif isinstance(value, AST):
self.visit(value)
visit resolves the method name at runtime using getattr, so adding a
visit_FunctionDef method to a subclass is sufficient to intercept all
FunctionDef nodes. generic_visit recurses into all child nodes by
iterating node._fields through iter_fields. For list-valued fields it
recurses into each element that is itself an AST node.
NodeTransformer overrides generic_visit to rebuild the child lists:
class NodeTransformer(NodeVisitor):
def generic_visit(self, node):
for field, old_value in iter_fields(node):
if isinstance(old_value, list):
new_values = []
for old_node in old_value:
if isinstance(old_node, AST):
value = self.visit(old_node)
if value is None:
continue
elif not isinstance(value, AST):
new_values.extend(value)
continue
new_values.append(value)
old_value[:] = new_values
elif isinstance(old_value, AST):
new_value = self.visit(old_value)
if new_value is None:
delattr(node, field)
else:
setattr(node, field, new_value)
return node
Returning None from a visitor removes the node from its parent list.
Returning a list from a visitor splices multiple replacement nodes into
the parent list in its place. Returning the node unchanged (or a
different AST node) replaces the original. The in-place slice
assignment old_value[:] = new_values mutates the existing list so that
any other references to the field value see the update.
literal_eval safe evaluation (lines 551 to 750)
cpython 3.14 @ ab2d84fe1023/Lib/ast.py#L551-750
def literal_eval(node_or_string):
"""
Evaluate an expression node or a string containing only a Python
literal or container display.
"""
if isinstance(node_or_string, str):
try:
node_or_string = parse(node_or_string.lstrip(" \t"), mode='eval')
except SyntaxError:
node_or_string = parse(node_or_string, mode='eval')
if isinstance(node_or_string, Expression):
node_or_string = node_or_string.body
def _convert(node):
if isinstance(node, Constant):
return node.value
elif isinstance(node, Tuple):
return tuple(map(_convert, node.elts))
elif isinstance(node, List):
return list(map(_convert, node.elts))
elif isinstance(node, Set):
return set(map(_convert, node.elts))
elif isinstance(node, Dict):
if len(node.keys) != len(node.values):
_raise_malformed_node(node)
return dict(zip(map(_convert, node.keys),
map(_convert, node.values)))
elif isinstance(node, BinOp) and isinstance(node.op, (Add, Sub)):
# allow +/- for complex literals: 1+2j
left = _convert(node.left)
right = _convert(node.right)
if isinstance(left, (int, float, complex)):
if isinstance(node.op, Add):
return left + right
else:
return left - right
_raise_malformed_node(node)
return _convert(node_or_string)
literal_eval accepts strings or pre-parsed Expression nodes. It
handles Constant (all scalar literals), Tuple, List, Set, Dict,
and one special case: a BinOp with Add or Sub whose operands are
numbers. The +/- case exists solely to evaluate complex literals such
as 1+2j or 3.14-0.5j, which the parser represents as binary
expressions rather than single constants.
Any node type not in this list causes _raise_malformed_node, which
raises ValueError with a message that includes the problematic source
segment. This ensures that literal_eval cannot execute arbitrary code
regardless of what the input string contains.
ast.unparse and _Unparser (lines 751 to 1000)
cpython 3.14 @ ab2d84fe1023/Lib/ast.py#L751-1000
class _Unparser(NodeVisitor):
def __init__(self, *, _avoid_backslashes=False):
self._source = []
self._precedences = {}
self._type_ignores = {}
self._indent = 0
self._avoid_backslashes = _avoid_backslashes
self._in_try_star = False
def visit_BinOp(self, node):
operator = self.binop[node.op.__class__.__name__]
operator_precedence = self.binop_precedence[node.op.__class__.__name__]
with self.require_parens(operator_precedence, node):
self.set_precedence(operator_precedence.next(), node.left, node.right)
self.traverse(node.left)
self.write(f" {operator} ")
self.traverse(node.right)
_Unparser is a NodeVisitor that accumulates text in self._source.
Operator precedence is tracked via _precedences: set_precedence marks
a node with the minimum precedence its parent allows; require_parens is
a context manager that writes ( and ) around the sub-expression when
the node's actual precedence is lower than what the parent requires.
unparse creates an _Unparser, calls visit on the root node, and
joins self._source into a string. The result is a valid Python
expression or statement sequence but not necessarily identical to the
original source (whitespace and comments are lost).
gopy mirror
ast is pending as a stdlib module. The port will need the full _ast
node type hierarchy, which gopy's compiler already constructs internally.
literal_eval, NodeVisitor, and NodeTransformer are pure Python and
can be ported directly. _Unparser is large but also pure Python; it is
not required for the interpreter runtime but is useful for debug output.