Lib/ast.py (part 5)

Source:

cpython 3.14 @ ab2d84fe1023/Lib/ast.py ↗

This annotation covers AST transformation and unparsing. See lib_ast4_detail for ast.NodeVisitor, ast.parse, and the AST node hierarchy.

Map

Lines	Symbol	Role
1-80	ast.NodeTransformer	Rewrite AST nodes in-place
81-180	ast.fix_missing_locations	Fill in lineno/col_offset
181-280	ast.literal_eval	Safely evaluate a literal expression
281-400	ast.unparse	Round-trip AST back to source string
401-500	ast.dump	Debug dump of AST tree

Reading

ast.NodeTransformer

# CPython: Lib/ast.py:610 NodeTransformer
class NodeTransformer(NodeVisitor):
    def generic_visit(self, node):
        for field, old_value in ast.iter_fields(node):
            if isinstance(old_value, list):
                new_values = []
                for old_node in old_value:
                    if isinstance(old_node, AST):
                        value = self.visit(old_node)
                        if value is None:
                            continue  # delete node
                        elif not isinstance(value, AST):
                            new_values.extend(value)  # replace with list
                            continue
                    new_values.append(value)
                old_value[:] = new_values
            elif isinstance(old_value, AST):
                new_value = self.visit(old_value)
                if new_value is None:
                    delattr(node, field)
                else:
                    setattr(node, field, new_value)
        return node

NodeTransformer.visit calls visit_ClassName on each node, replacing it with the return value. Returning None deletes the node; returning a list of nodes expands it (e.g., replacing one statement with multiple). Used by tools like pytest (assertion rewriting) and numba (IR lowering).

ast.fix_missing_locations

# CPython: Lib/ast.py:235 fix_missing_locations
def fix_missing_locations(node):
    def _fix(node, lineno, col_offset, end_lineno, end_col_offset):
        if 'lineno' in node._attributes:
            if not hasattr(node, 'lineno'):
                node.lineno = lineno
            else:
                lineno = node.lineno
        for child in ast.iter_child_nodes(node):
            _fix(child, lineno, col_offset, end_lineno, end_col_offset)
        return node
    _fix(node, 1, 0, 1, 0)
    return node

When constructing AST nodes manually, lineno and col_offset are required by compile(). fix_missing_locations fills them in from the nearest parent node that has location info. This is the standard post-processing step after programmatic AST construction.

ast.literal_eval

# CPython: Lib/ast.py:62 literal_eval
def literal_eval(node_or_string):
    if isinstance(node_or_string, str):
        node_or_string = parse(node_or_string, mode='eval')
    if isinstance(node_or_string, Expression):
        node_or_string = node_or_string.body
    def _convert(node):
        if isinstance(node, Constant):
            return node.value
        elif isinstance(node, Tuple):
            return tuple(map(_convert, node.elts))
        elif isinstance(node, List):
            return list(map(_convert, node.elts))
        elif isinstance(node, Set):
            return set(map(_convert, node.elts))
        elif isinstance(node, Dict):
            return {_convert(k): _convert(v) for k, v in zip(node.keys, node.values)}
        elif isinstance(node, UnaryOp) and isinstance(node.op, (UAdd, USub)):
            operand = _convert(node.operand)
            if isinstance(operand, (int, float, complex)):
                return +operand if isinstance(node.op, UAdd) else -operand
        raise ValueError(f'malformed node or string: {node!r}')
    return _convert(node_or_string)

ast.literal_eval("{'a': 1, 'b': [2, 3]}") returns {'a': 1, 'b': [2, 3]}. Only literal constants, containers, and unary +/- on numbers are accepted. No function calls, attribute access, or variable references — safe for untrusted input.

ast.unparse

# CPython: Lib/ast.py:840 unparse
def unparse(ast_obj):
    unparser = _Unparser()
    return unparser.visit(ast_obj)

class _Unparser(NodeVisitor):
    def visit_BinOp(self, node):
        ...
        self.traverse(node.left)
        self.write(f" {BINOP_SYMBOLS[type(node.op)]} ")
        self.traverse(node.right)

ast.unparse(ast.parse('x + y * 2')) returns 'x + y * 2'. The _Unparser visitor reconstructs valid Python source from the AST. Parentheses are inserted only where needed based on operator precedence. unparse(parse(s)) is not guaranteed to equal s (comments and whitespace are lost) but compile(unparse(parse(s))) is semantically equivalent.

gopy notes

ast.NodeTransformer is pure Python and runs directly. ast.fix_missing_locations uses ast.iter_child_nodes which routes to objects.ASTIterChildNodes. ast.literal_eval calls compile.Parse in eval mode, then recursively converts objects.ASTConstant nodes. ast.unparse uses _Unparser which writes to a strings.Builder equivalent.