Generators, Iterators & Context Managers

Every for loop in this entire curriculum has relied on Python's iterator protocol without you ever seeing it directly. This session opens that hood, and shows you how with actually works too.

1. Learning Objective

2. Pre-Coding Quiz

Answer these before reading the concept explanation or writing any code. It is fine to get these wrong — that is the point. You need 4/5 to proceed to the lab.

3. The Concept — Iterators, Generators, and Context Managers

What every for loop has secretly relied on

Since Session 06, every for item in collection: has worked because the collection implements the iterator protocol — broadly, Python can repeatedly ask it "what's next?" until there is nothing left. Lists, dicts, strings, and files all support this. This session shows you the mechanism, and lets you build your own.

yield — writing a generator function

A function containing yield becomes a generator function. Calling it does not run the body — it returns a generator object that produces one value at a time, pausing at each yield until the next value is requested.

def countdown(n):
    while n > 0:
        yield n
        n -= 1

for number in countdown(3):
    print(number)
# 3
# 2
# 1

# Calling countdown(3) alone does NOT print anything — it returns a generator object
gen = countdown(3)
print(gen)          # <generator object countdown at 0x...>
print(next(gen))     # 3 — pulls the first value
print(next(gen))     # 2 — resumes exactly where it left off

Why lazy evaluation matters — memory efficiency

A function that builds and returns a full list must hold every value in memory before returning anything. A generator produces values one at a time and never holds the whole sequence at once — the same memory-efficiency idea Session 41 applied to reading huge files line by line.

# Builds the ENTIRE list in memory before returning anything
def all_populations_list(countries):
    return [c["population"] for c in countries]

# Produces ONE value at a time — never holds the whole sequence in memory
def all_populations_gen(countries):
    for c in countries:
        yield c["population"]

# For a huge dataset, the generator version scales far better

Writing your own context manager

A class becomes usable with with by implementing __enter__ (runs at the start of the block) and __exit__ (runs at the end, guaranteed — even if an exception occurred inside, exactly like Session 41's file-closing guarantee).

class Timer:
    def __enter__(self):
        import time
        self.start = time.perf_counter()
        return self   # this becomes "as timer" in the with statement

    def __exit__(self, exc_type, exc_value, traceback):
        import time
        elapsed = time.perf_counter() - self.start
        print(f"Elapsed: {elapsed:.6f}s")
        return False  # don't suppress any exception that occurred

with Timer() as timer:
    total = sum(range(1000000))
print(total)

open() was a context manager all along

Session 37 and 41 used with open(path) as f: without ever seeing why it worked. The file object open() returns implements exactly the __enter__/__exit__ protocol shown above — it was never special-cased magic.

# This was always just an ordinary context manager, working exactly like Timer above:
with open("countries.json") as f:
    data = f.read()
# f.__exit__ runs here automatically — closing the file, guaranteed

4. Lab

What you will build

A file called generators_lab.py.

Step-by-step instructions

# generators_lab.py
countries = [
    {"name": "Kenya", "region": "Africa", "population": 54000000},
    {"name": "Ghana", "region": "Africa", "population": 31000000},
    {"name": "Peru", "region": "Americas", "population": 33000000},
]

def large_countries(countries, threshold):
    for c in countries:
        if c["population"] > threshold:
            yield c

for c in large_countries(countries, 40000000):
    print(c["name"])

gen = large_countries(countries, 40000000)
print(gen)          # generator object, nothing printed from inside yet
print(next(gen))     # NOW the body runs up to the first yield

import time

class Timer:
    def __enter__(self):
        self.start = time.perf_counter()
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        self.elapsed = time.perf_counter() - self.start
        print(f"Elapsed: {self.elapsed:.6f}s")
        return False

with Timer() as t:
    total = sum(c["population"] for c in countries)
print("Total:", total)

try:
    with Timer() as t:
        raise ValueError("something went wrong inside the block")
except ValueError as e:
    print("Caught after Timer printed its elapsed time:", e)

5. Expected Files Changed

6. Commit Checkpoint

Once the lab is complete and you can explain every line, make this exact commit:

git add generators_lab.py docs/sessions/session-46/index.html
git commit -m "session-46: write a generator function and a custom __enter__/__exit__ context manager"

7. Code Review Checklist

Go through your code line by line and check each item:

8. Post-Coding Quiz

Same 5 questions. Take it again now that you have written and run the code. You need 4/5 to mark this session complete.

9. Reflection Questions

Think through these after the post-quiz. No right answer — they are for discussion.

10. What Breaks If This Knowledge Is Missing?

11. What We Learned