Skip to main content

Include/internal/pycore_import.h

cpython 3.14 @ ab2d84fe1023/Include/internal/pycore_import.h

pycore_import.h collects the internal interfaces for CPython's import machinery that are not exposed through the public Python/import.h API. It declares the per-interpreter recursive import lock, the low-level sys.modules dict accessors used inside import.c, the frozen module registry (_PyImport_FrozenBootstrap and friends), and the extension module cache that tracks .so-backed modules across sub-interpreters.

The import lock (_PyImport_AcquireLock) is a per-interpreter recursive mutex. It is taken at the start of PyImport_ImportModuleLevelObject and released after the module is fully initialized. Recursive imports by the same thread (which happen constantly during importlib bootstrap) are allowed: an internal depth counter tracks the recursion level, and the lock is only released when the depth returns to zero.

The sys.modules dict is stored on PyInterpreterState.modules (declared in pycore_interp_structs.h). The accessors here (_PyImport_GetModule, _PyImport_SetModule) wrap the raw dict with the interpreter argument so that sub-interpreter isolation is maintained: each sub-interpreter has its own sys.modules dict and cannot see modules cached by another sub-interpreter.

Map​

LinesSymbolRolegopy
1-30include guard + _PyImport_ImportLock structPer-interpreter lock: mutex, thread owner id, level depth counter.imp/sysmodules.go sysModulesMu
31-80_PyImport_AcquireLock / _PyImport_ReleaseLock / _PyImport_IsInitializedLock/unlock with recursion depth; initialized predicate.imp/import.go
81-120_PyImport_GetModule / _PyImport_SetModule / _PyImport_RemoveModulesys.modules dict wrappers taking an PyInterpreterState *.imp/sysmodules.go GetModule / AddModule / RemoveModule
121-160_PyImport_GetModuleAttr / _PyImport_GetModuleAttrStringFetch an attribute from a named module in sys.modules; used by _io, warnings, etc.n/a
161-200_PyImport_FrozenBootstrap / _PyImport_FrozenStdlib / _PyImport_FrozenTestNull-terminated arrays of struct _frozen for the three frozen module sets.imp/frozen.go
201-240_PyImport_FindExtensionObject / _PyImport_FixupExtensionObjectCache and retrieve PyModuleDef-backed .so modules; keyed by (name, path).n/a (no .so loading in gopy)

Reading​

Import lock protocol (lines 1 to 80)​

cpython 3.14 @ ab2d84fe1023/Include/internal/pycore_import.h#L1-80 

struct _import_lock {
    PyThread_type_lock lock;
    unsigned long thread;   /* OS thread id of the current owner, or 0 */
    int level;              /* recursion depth for the current owner */
};

int
_PyImport_AcquireLock(PyInterpreterState *interp)
{
    unsigned long me = PyThread_get_thread_ident();
    struct _import_lock *import_lock = &interp->imports.lock;

    if (import_lock->thread == me) {
        /* Recursive acquire by the same thread: just bump the depth. */
        import_lock->level++;
        return 1;
    }

    /* Blocking acquire by a different thread. */
    if (PyThread_acquire_lock(import_lock->lock, WAIT_LOCK) == PY_LOCK_FAILURE) {
        return -1;
    }
    assert(import_lock->thread == 0 && import_lock->level == 0);
    import_lock->thread = me;
    import_lock->level = 1;
    return 1;
}

int
_PyImport_ReleaseLock(PyInterpreterState *interp)
{
    struct _import_lock *import_lock = &interp->imports.lock;
    if (import_lock->thread != PyThread_get_thread_ident() || import_lock->level < 1) {
        return -1;  /* not owner */
    }
    import_lock->level--;
    if (import_lock->level == 0) {
        import_lock->thread = 0;
        PyThread_release_lock(import_lock->lock);
    }
    return 0;
}

The import lock prevents two threads from simultaneously initializing the same module. Without it, a second thread that reaches import foo while the first thread is mid-way through executing foo.py would see a partially-initialized module in sys.modules and proceed to use it.

The level counter allows the same thread to re-enter the lock during bootstrap. The importlib bootstrap itself does import _frozen_importlib which recursively calls into PyImport_ImportModuleLevelObject. Without the recursion check, importing any module would deadlock immediately.

In gopy, the imp package uses a sync.RWMutex (sysModulesMu in imp/sysmodules.go) to serialize writes to sys.modules. There is no separate import lock struct; Go's sync.RWMutex already permits recursive read-locks, and the single-interpreter constraint means only one goroutine runs Python bytecode at a time in v0.12.

sys.modules dict access pattern (lines 81 to 160)​

cpython 3.14 @ ab2d84fe1023/Include/internal/pycore_import.h#L81-160 

PyObject *
_PyImport_GetModule(PyInterpreterState *interp, PyObject *name)
{
    PyObject *modules = interp->modules;
    if (modules == NULL) {
        return NULL;
    }
    PyObject *m;
    (void)PyDict_GetItemRef(modules, name, &m);
    return m;  /* new reference, or NULL if not found */
}

int
_PyImport_SetModule(PyInterpreterState *interp, PyObject *name, PyObject *module)
{
    PyObject *modules = interp->modules;
    if (modules == NULL) {
        PyErr_SetString(PyExc_RuntimeError, "no sys.modules");
        return -1;
    }
    return PyObject_SetItem(modules, name, module);
}

Every import path bottlenecks through these two functions rather than accessing interp->modules directly. The indirection means that a sub-interpreter can swap out its modules dict (e.g. to run with an isolated import state) without any call-site changes.

_PyImport_GetModuleAttr and _PyImport_GetModuleAttrString are convenience wrappers used heavily by the standard library. For example, the warnings module implementation calls _PyImport_GetModuleAttrString(interp, "warnings", "_filters_mutated") to notify filters rather than importing warnings itself, avoiding a circular import risk.

In gopy, imp/sysmodules.go provides GetModule, AddModule, and RemoveModule as direct ports, using the package-level sysModules *Dict rather than interp->modules because gopy has only one interpreter in v0.12. The vm/eval_import.go IMPORT_NAME arm calls imp.ImportModuleLevel which checks GetModule as its first cache hit step.

Frozen module registry (lines 161 to 200)​

cpython 3.14 @ ab2d84fe1023/Include/internal/pycore_import.h#L161-200 

/* Each entry describes one frozen module. */
struct _frozen {
    const char *name;
    const unsigned char *code;  /* marshalled PyCodeObject bytes */
    int size;
    int is_package;
    const char *get_code;       /* optional thunk name for lazy loading */
};

/* Three separate arrays, all NULL-terminated. */
extern const struct _frozen _PyImport_FrozenBootstrap[];
extern const struct _frozen _PyImport_FrozenStdlib[];
extern const struct _frozen _PyImport_FrozenTest[];

CPython ships three sets of frozen modules:

  • _PyImport_FrozenBootstrap: the importlib bootstrap modules (_frozen_importlib, _frozen_importlib_external, zipimport). These are imported before the file-system finder is available.
  • _PyImport_FrozenStdlib: selected stdlib modules frozen for startup speed (e.g. abc, codecs, encodings.utf_8).
  • _PyImport_FrozenTest: test helpers frozen only in debug builds.

The import machinery in import.c searches all three arrays in order during _PyImport_FindFrozenObject. A frozen module is loaded by unmarshalling the embedded code bytes with PyMarshal_ReadObjectFromString, then executing the resulting code object with PyImport_ExecCodeModule.

In gopy, imp/frozen.go holds the equivalent FrozenModule registry and imp/bootstrap.go registers the importlib bootstrap. The frozen module bytes are stored as Go []byte constants generated by the stdlibinit package during the MANIFEST.txt-driven build step.

gopy mirror​

The gopy import pipeline in imp/ maps to this header as follows:

  • _PyImport_AcquireLock / _PyImport_ReleaseLock: replaced by sysModulesMu sync.RWMutex in imp/sysmodules.go.
  • _PyImport_GetModule / _PyImport_SetModule: GetModule / AddModule in imp/sysmodules.go.
  • _PyImport_FrozenBootstrap: FindFrozen in imp/frozen.go.
  • _PyImport_FindExtensionObject: not ported; gopy has no .so loader.

The IMPORT_NAME bytecode arm in vm/eval_import.go mirrors PyImport_ImportModuleLevelObject (Python/import.c:1561) by checking the sys.modules cache first, then delegating to imp.ImportModuleLevel.