Skip to main content

multiprocessing.connection

cpython 3.14 @ ab2d84fe1023/Lib/multiprocessing/connection.py

IPC connection objects for passing Python values between processes. The module wraps file descriptors and sockets with a framing protocol: send() pickles an object and writes a 4-byte length prefix, recv() reads that length then unpickles. Pipe() returns a pair of connected Connection objects backed by OS pipes or sockets. Client() and Listener() layer a socket-based rendezvous on top. Authentication between client and server uses HMAC-SHA256 over a random nonce produced by os.urandom.

Reading​

Connection: send, recv, and the framing protocol​

Every Connection wraps a single file descriptor. send() pickles the caller's object with pickle.dumps, prepends a 4-byte big-endian length, and calls the internal _send_bytes helper, which loops until the entire buffer is written. recv() mirrors this: _recv_bytes reads exactly 4 bytes for the header, then reads exactly that many payload bytes, and recv() unpickles the result.

# Lib/multiprocessing/connection.py (CPython 3.14, simplified)
def send(self, obj):
    self._check_closed()
    self._check_writable()
    self._send_bytes(_ForkingPickler.dumps(obj))

def recv(self):
    self._check_closed()
    self._check_readable()
    buf = self._recv_bytes()
    return _ForkingPickler.loads(buf.getbuffer())

def _send_bytes(self, buf):
    n = len(buf)
    if n > 0x7fffffff:
        raise ValueError("message too long")
    # 4-byte length header followed by payload
    self._send(struct.pack("!i", n))
    self._send(buf)

The !i format sends a signed 32-bit big-endian integer, so the maximum single message size is 2 GiB. Anything larger raises ValueError before any bytes are written.

Listener and Client: socket-based rendezvous​

Listener binds a server socket and accepts one connection at a time via accept(). The address family is inferred from the address type: a string that starts with / (or any string on Windows) uses AF_UNIX; a (host, port) tuple uses AF_INET. Client() connects to an existing Listener address and returns a Connection.

# Listener.accept() -- waits for one incoming connection
def accept(self):
    self._check_closed()
    s, self._last_accepted = self._listener.accept()
    s.setblocking(True)
    c = Connection(s.detach())
    if self._authkey is not None:
        answer_challenge(c, self._authkey)
        deliver_challenge(c, self._authkey)
    return c

If an authkey was supplied to Listener, both sides run the challenge-response protocol immediately after the TCP/Unix handshake, before the caller's accept() call returns.

Authentication: HMAC challenge-response​

The authentication protocol is symmetric. The server sends a random 20-byte nonce (os.urandom(20)). The client signs the nonce with HMAC-SHA256 keyed by the shared authkey and sends the digest back. The server verifies it, then the roles reverse so the client authenticates the server.

MESSAGE_LENGTH = 20
CHALLENGE = b"#CHALLENGE#"
WELCOME    = b"#WELCOME#"
FAILURE    = b"#FAILURE#"

def deliver_challenge(connection, authkey):
    import hmac
    assert isinstance(authkey, bytes)
    message = os.urandom(MESSAGE_LENGTH)
    connection.send_bytes(CHALLENGE + message)
    digest = hmac.new(authkey, message, "md5").digest()
    response = connection.recv_bytes(256)        # reject oversized replies
    if response == digest:
        connection.send_bytes(WELCOME)
    else:
        connection.send_bytes(FAILURE)
        raise AuthenticationError("digest mismatch")

The nonce prevents replay attacks. Both sides must complete the exchange or the connection is torn down before any user data flows.

gopy mirror​

Not yet ported. The Go standard library has net.Conn and os.Pipe, which cover the transport layer. A port would need to implement the framing protocol (_send_bytes / _recv_bytes), the HMAC challenge-response handshake, and wait() (a cross-platform select/poll wrapper).

CPython 3.14 changes​

  • wait() on Windows now uses WaitForMultipleObjects directly rather than a thread-per-handle workaround introduced in 3.3.
  • Connection.send_bytes / recv_bytes accept a memoryview without copying when the underlying fd supports sendmsg.
  • The default pickle protocol used by _ForkingPickler tracks pickle.DEFAULT_PROTOCOL, which is 5 in 3.14.