What is the exact difference between an Iterable and an Iterator in Python, how does a generator preserve local variables across `yield` without using the C.

A strong answer

In Python, every for x in obj: loop compiles into bytecode that calls it = iter(obj) (GET_ITER, invoking __iter__) and repeatedly calls x = next(it) (FOR_ITER, invoking __next__) until __next__ raises StopIteration. An Iterable (like a list) can produce a fresh Iterator each time iter() is called, whereas an Iterator (including any Generator) returns self from __iter__ and maintains an internal cursor that is exhausted after one pass! When you call a generator function (any function containing yield), CPython does not run the body—it returns a PyGenObject wrapping a suspended PyFrameObject on the heap. Each next(gen) resumes that frame from its saved instruction pointer (f_lasti), runs until the next yield val, suspends the frame with its local variables intact, and returns val—using $O(1)$ constant memory regardless of stream length!

A generator's heap frame holds only a single item's local variables at any point in time ($O(1)$ space), and once a generator or iterator raises StopIteration, every subsequent next() call immediately raises StopIteration (single-pass exhaustion). Because yield suspends the frame and hands ownership of the current result to the caller before advancing to the next iteration, once the caller finishes with item $i$, item $i$'s ob_refcnt drops to 0 and its memory is reclaimed before item $i+1$ is even computed.

Replacing an eager list comprehension [f(x) for x in stream] of 10M items with a generator expression (f(x) for x in stream) drops peak memory from ~800 MB down to ~200 bytes (the single PyGenObject frame). In production systems (FastAPI StreamingResponse (streaming LLM SSE chunks), SQLAlchemy chunked result cursors (yield_per), PyTorch IterableDataset, and itertools pipelines all rely on generator state machines.), engineers monitor these exact trade-offs to keep p99 latency predictable and prevent resource exhaustion under peak load.

What the interviewer is listening for

• Traces how the Python Iterator Protocol and lazy generator frames operates in memory step by step rather than giving a vague textbook definition.
• States the core correctness invariant and explains why edge cases do not violate it.
• Quantifies time, space, and hardware costs (List Comprehension [x for x in 10M], Generator Expression (x for x in 10M)) with concrete numbers.
• Identifies real-world failure modes (Passing a generator to a function that checks if len(items) == 0: or loops twice) and explains how production systems mitigate them.

Common mistakes

• Treating the Python Iterator Protocol and lazy generator frames as a black box without explaining the underlying memory or state transitions.
• Ignoring worst-case tail latency or memory overhead (Writing except Exception: around yield is fine, but writing bare except: around yield and yielding again).
• Forgetting boundary conditions and error recovery (Opening a file inside a generator without with open(...) and abandonar the generator mid-stream).