How do you parallelize CPU-bound work across 8 cores in Python without getting killed by `pickle` IPC overhead or `fork()` multithreaded deadlocks?

A strong answer

Separate processes do not share a virtual address space by default, so passing arguments to ProcessPoolExecutor.submit(fn, data) serializes data with pickle.dumps(), writes the bytes into an OS pipe/socket, and deserializes with pickle.loads() in the worker! Python 3.8+ solves this with multiprocessing.shared_memory.SharedMemory: the parent allocates a named POSIX shared memory block (/dev/shm on Linux) once, wraps it in a zero-copy memoryview or numpy.ndarray(..., buffer=shm.buf), and passes only the tiny string shm.name (20 bytes), shape, and dtype to the worker processes! Each worker attaches to SharedMemory(name=shm_name) and maps the exact same physical RAM pages into its own address space in $O(1)$ time with 0 bytes copied!

Passing a SharedMemory block across $W$ worker processes transfers only the segment metadata (name, shape, dtype, $O(1)$ bytes) over IPC pipes while all $W$ processes map the single shared physical RAM allocation. Because the OS kernel maps the underlying POSIX shm_open page frames into each worker's page table with MAP_SHARED, reading or writing the memoryview / ndarray buffer accesses physical RAM directly without pipe serialization.

Pickling and piping a 500 MB array to 8 workers copies 4,000 MB across pipes (~6–12 seconds); attaching 8 workers to SharedMemory transfers <200 bytes and takes <0.5 milliseconds! In production systems (PyTorch DataLoader(num_workers=8, pin_memory=True) (which uses torch.multiprocessing shared memory tensors), Ray's Plasma shared-memory object store, and Gunicorn/Celery multi-worker pools all rely on POSIX shared memory.), engineers monitor these exact trade-offs to keep p99 latency predictable and prevent resource exhaustion under peak load.

What the interviewer is listening for

• Traces how zero-copy multiprocessing.shared_memory and spawn vs fork process start methods operates in memory step by step rather than giving a vague textbook definition.
• States the core correctness invariant and explains why edge cases do not violate it.
• Quantifies time, space, and hardware costs (Per-Task Pickle (pool.map with 100MB arg), Pool initializer=load_data) with concrete numbers.
• Identifies real-world failure modes (Using fork start method after initializing multithreaded C libraries (PyTorch CUDA, gRPC, OpenMP, ssl)) and explains how production systems mitigate them.

Common mistakes

• Treating zero-copy multiprocessing.shared_memory and spawn vs fork process start methods as a black box without explaining the underlying memory or state transitions.
• Ignoring worst-case tail latency or memory overhead (Forgetting to call shm.unlink() in a finally: block after creating a SharedMemory segment).
• Forgetting boundary conditions and error recovery (Sending 1,000,000 tiny tasks one by one to ProcessPoolExecutor.map(fn, items) with default chunksize=1).