Skip to main content

1625. gopy v0.5 testing strategy

This spec is the test-side companion to 1620 and 1665. The checklist in 1620 names every Go file that must land for v0.5; this spec names the test that proves each file matches CPython 3.14.

The rule is one-to-one: every implementation checkbox in 1620 has a test checkbox here. A v0.5 release ships only when every box on both specs is ticked.

Layered gates​

The v0.5 test pyramid has four layers, each strictly stricter than the one above. Code lands at the bottom and climbs.

  1. Unit: package-local Go tests against hand-built inputs. Catches bad refactors and obvious arity errors.
  2. Cross-check: Go output diffed against CPython output for the same input, where the input is built in-process (no parser).
  3. Marshal parity: gopy emits co_code, co_linetable, co_exceptiontable byte-equal to CPython for the same source.
  4. Run parity: a future v0.6 layer; out of scope for v0.5.

The cross-check layer needs a host CPython 3.14 on $PATH. Tests that need it are tagged with //go:build cpython and skipped on CI without it; the gate jobs run with the tag.

How to test each kind of port​

Pure data types (Seq, Pos, Token)​

Table-driven equality tests. No CPython invocation. Cover empty, single, many, mutation, out-of-bounds. Locks shape; the per-field parity is in 1601.

Hand-written nodes (ast/nodes.go)​

Construction, field access, Position() round-trip, IsDocString truth table. No CPython needed; the asdl source-of-truth is in 1620.

Generated nodes (ast/nodes_gen.go)​

Generator-output parity: a tools/asdl_go_test round-trips Python.asdl and asserts the emitted Go file is byte-equal to the checked-in nodes_gen.go. This catches generator drift without re-running the generator on every CI.

Validators (ast/validate.go, ast/preprocess.go)​

Two-axis tests:

  • Acceptance: every node kind that CPython accepts must round-trip nil.
  • Rejection: every error string in Python/ast.c matches verbatim. The test asserts both errors.Is-style class and err.Error() substring.

For PEP 765 (try/finally with break/continue/return), the rejection panel mirrors Lib/test/test_syntax.py line-for-line.

Future flags (future/future.go)​

One test per __future__ name (recognised, no-op, or rejected). Plus the four wrong-shape rejections (braces, unknown, after non-import, relative). Plus location tracking on the from keyword.

Symtable​

Cross-check against symtable.symtable(src, '<test>', 'exec') in CPython. The Go test reads CPython's JSON dump of the symbol table for each fixture and asserts identical scope tree, identical name set per scope, identical flags per name. Fixtures live in symtable/testdata/*.py and symtable/testdata/*.json; the JSON is regenerated by tools/symtable_golden.

Coverage panel:

  • Module / function / class / lambda / comprehension scopes.
  • Free variable capture (closure cells).
  • global and nonlocal declarations.
  • Walrus inside comprehension.
  • import a.b.c name binding.
  • PEP 695 type-parameter scope.

Codegen (compile/codegen.go)​

Per statement and per expression. Each test:

  1. Builds the AST in Go (no parser).
  2. Runs codegen.
  3. Asserts the emitted opcode sequence matches a hand-curated golden.
  4. Cross-checks (with //go:build cpython) that CPython emits the same sequence for the equivalent source.

Statement panel mirrors the codegen visitor list in 1620 §6.

Flowgraph (compile/flowgraph.go)​

One test per optimization pass. Each pass has:

  • A "minimal positive" case (input that triggers it, output that applies it).
  • A "negative" case (input where the pass must not fire).
  • A "fixed-point" case (running the pass twice equals once).

Plus a property test: random instruction sequences round-trip through FromSequence -> Optimize -> ToSequence without losing reachable instructions.

Assemble (compile/assemble.go)​

Byte-exact tests. For each artefact (co_code, co_linetable, co_exceptiontable):

  • Hand-built varint encodings for every form (short, one-line, long, no-location for PEP 626; full form for PEP 657).
  • Round-trip: assemble then disassemble the bytes back into the in-memory representation.
  • Cross-check (cpython tag): identical bytes from CPython compile() for a panel of source files.

Compiler driver (compile/compiler.go)​

End-to-end on hand-built ASTs. The test panel here is the union of every other panel: assignment, if/while, try/except, def, class, comprehension, async function, match, type alias.

Tokenize (1665)​

Iterator contract: New(src, false).Next() ends in io.EOF, never panics on empty input, never yields a zero-Type token.

Token-stream parity (cpython tag): for every fixture in tokenize/testdata/*.py, the Go iterator's full output (Type, Value, Start, End, Line) matches tokenize.tokenize(io.BytesIO(src).readline) in the host CPython.

Error parity: each lexer error string matches CPython verbatim.

Per-checkbox test mapping​

The table below mirrors the order of 1620 §1 through §14 plus 1665. Every implementation checkbox has its test counterpart here. When the implementation lands, both boxes are ticked in the same commit.

1. ast package: asdl runtime​

  • Seq[T].Len/Get/Set table test.
  • NewSeq(0) returns non-nil empty sequence.

2. ast package: hand-written nodes​

  • Module/Interactive/Expression/FunctionType construction.
  • Position() returns the embedded Pos for every node.
  • IsDocString truth table (string Constant yes; non-string no; non-ExprStmt no).

3. ast package: validate​

  • Negative position rejected; NoPos accepted.
  • end_lineno < lineno rejected.
  • end_col_offset < col_offset on same line rejected.
  • Negative ImportFrom level rejected with exact CPython text.
  • Constant: every accepted scalar kind, plus tuple, plus frozenset, plus invalid-type rejection.
  • Nil module / nil statement / nil expression rejected.

4. ast package: preprocess (constant fold + PEP 765)​

  • UnaryOp folding panel: -1, not True, ~0xFF.
  • BinOp folding panel: every accepted (op, lhs-type, rhs-type) triple.
  • BoolOp folding (True and X, False or X).
  • Compare folding when both sides are constants.
  • Tuple/Frozenset/Set literal folding.
  • PEP 765: break / continue / return inside finally rejected with verbatim string.

5. ast package: unparse​

  • Per node kind: round-trip Unparse(parser.Parse(src)) == normalised(src) for a curated panel.
  • Operator precedence parens preserved.
  • f-string and t-string round-trip.

6. ast package: asdl-driven generator​

  • tools/asdl_go parses Python.asdl without error.
  • Generator output is byte-equal to checked-in nodes_gen.go (the regen-and-diff test).

7. future package​

  • Annotations recognised; bit set; location captured.
  • barry_as_FLUFL recognised.
  • No-op features (division, generators, ...) accepted.
  • from __future__ import braces rejected with "not a chance".
  • Unknown name rejected.
  • Docstring is skipped before scanning.
  • Stop at first non-future statement.
  • Relative from . import x ignored.
  • Expression mode: no future flags.

8. symtable​

  • Module scope name set parity with CPython.
  • Function scope: locals, args, free vars.
  • Class scope: __class__ cell, MRO entries.
  • Lambda scope.
  • Comprehension scope (free var capture).
  • global / nonlocal flag bits.
  • Walrus rebinds outer scope.
  • PEP 695 type-parameter scope.
  • All symtable error strings verbatim.

9. compile/instrseq​

  • NewLabel IDs are 1-based and monotonic.
  • Addop rejects opcode > MaxOpcode and oparg >= MaxOparg.
  • Insert shifts existing labels.
  • ApplyLabelMap is idempotent.
  • Non-jump opargs are not rewritten.
  • ExceptHandlerInfo.Label is resolved.
  • SetAnnotationsCode panics on second call.
  • AddNested attaches child sequence.

10. compile/opcodes_gen​

  • Opcode numbers cross-check against CPython 3.14 opmap.
  • Per-flag predicates for a sample (LOAD_CONST, LOAD_FAST, LOAD_NAME, JUMP_FORWARD, NOP).
  • HasTarget matches HasJump for the jump opcodes.
  • Out-of-range Name returns "".
  • Generator round-trip: regenerate and diff against checked-in opcodes_gen.go.

11. compile/codegen​

Statements:

  • Assign / AugAssign / AnnAssign.
  • If / While / For / AsyncFor.
  • Try / TryStar / Raise.
  • With / AsyncWith.
  • FunctionDef / AsyncFunctionDef.
  • ClassDef.
  • Match (per pattern kind: literal, capture, sequence, mapping, class, or, as, value).
  • Import / ImportFrom.
  • Global / Nonlocal.
  • Return / Break / Continue.
  • Pass.
  • PEP 695 TypeAlias / TypeVar.
  • Assert.
  • Delete.

Expressions:

  • BoolOp / BinOp / UnaryOp / Lambda / IfExp.
  • Dict / Set / List / Tuple displays.
  • ListComp / SetComp / DictComp / GeneratorExp.
  • Await / Yield / YieldFrom.
  • Compare (chained).
  • Call (star, starstar, keyword).
  • FormattedValue / Interpolation / JoinedStr / TemplateStr.
  • Constant.
  • Attribute (LOAD_SUPER_ATTR super-instruction).
  • Subscript (load/store/delete).
  • Starred.
  • Name (LOAD_FAST/DEREF/GLOBAL/NAME by scope).
  • Slice.

Block / unwind:

  • fblock_push / pop / unwind equivalents.
  • Try/except/finally exception-table emission.
  • With unwinding calls exit.
  • Async-with unwinding awaits aexit.
  • Generator return-value path.
  • Coroutine close path.

12. compile/flowgraph​

  • LOAD_CONST chain folding (positive, negative, fixed-point).
  • Jump threading.
  • Conditional-jump propagation.
  • Unreachable block elimination.
  • Dead-code after unconditional terminator.
  • Stack-effect verification: every block's net push - pop matches the recorded entry depth.
  • RESUME insertion at entry.
  • Implicit LOAD_CONST None/RETURN_VALUE for fall-through.
  • EXTENDED_ARG insertion for wide opargs.
  • Push-null fix-up at super-instruction call sites.
  • Property test: random sequences round-trip through Optimize.

13. compile/assemble​

  • EXTENDED_ARG widening for opargs > 0xFF, > 0xFFFF, > 0xFFFFFF.
  • PEP 626 location-table varint encoder per form.
  • PEP 657 exception-table varint encoder.
  • Constants table dedup parity.
  • Names / Varnames / Freevars / Cellvars population.
  • co_flags per kind: function, async, generator, coroutine, async generator.
  • co_qualname for nested defs.
  • co_code byte emission.
  • cpython-tag: byte-equal co_code for assignment, if, while, try/except, def, comprehension, async function.

14. compile/compiler​

  • End-to-end Compile(mod, "<test>", 0) for the full statement panel.
  • optimize levels -1, 0, 1, 2 each suppress the right opcodes.
  • cpython-tag: marshal.dumps(gopy_code) == marshal.dumps(cpython_code) for the v05test gate panel.

15. compile/dis​

  • Format output line-for-line equal to CPython 3.14 dis.dis() on the v05test panel.

16. tokenize (1665)​

  • Type constant numeric values match CPython 3.14 token.h.
  • New("", false).Next() returns io.EOF immediately.
  • Iterator never yields a zero Type.
  • cpython-tag: token-stream byte-parity on every fixture in tokenize/testdata/.
  • Error parity: every documented SyntaxError string is verbatim.

17. v05test cross-cut gate​

  • TestGateAssign: a = 1 + 2.
  • TestGateIfWhile: if/while panel.
  • TestGateTryExcept: try/except, finally, exception table.
  • TestGateDef: def f(x): return x + 1.
  • TestGateComprehension: [x*x for x in range(3)].
  • TestGateAsyncFunction: async def f(): pass.
  • TestGateMarshalRoundtrip: byte-equal co_code, co_linetable, co_exceptiontable against CPython marshal output.
  • tools/golden/: regeneration script for golden blobs under v05test/golden/.

Test-only utilities​

  • internal/parityhelpers (out of scope per "no internal/" rule; instead a flat testparity/ package): wrappers around exec.Command("python3.14", "-c", ...) for cpython-tag tests.
  • tools/golden/: regenerate captured CPython output. Stamps the CPython version into the file header so a host upgrade does not silently invalidate the golden.
  • tools/symtable_golden/: dump CPython symtables to JSON.

Coverage policy​

Every Go file under v0.5 has its own _test.go covering at minimum:

  • One success path per exported symbol.
  • One rejection path per exported error.
  • One regression hook for every CPython quirk preserved in the port.

A symbol with no tests is a release blocker. Lint rule (manual): grep for ^func [A-Z] in implementation files; cross-reference against _test.go in the same package; unmatched names fail the gate.

Out of scope for v0.5 testing​

  • Run-parity against CPython under the real VM. v0.6.
  • JIT / specialization parity. v0.11+.
  • Stdlib import behaviour. v0.8.