Multiprocessing

Python Multiprocessing: Many Helpers, One Big Job

The Kitchen Analogy 🍳

Imagine you’re making a huge feast for 100 guests. You have lots of dishes to prepare. If you cook alone, it takes forever. But what if you had 10 chefs, each with their own kitchen, working at the same time?

That’s multiprocessing in Python! Each chef is a process. Each kitchen is a separate workspace in memory. They all work together to finish the big job faster.

What is a Process?

A process is like a separate worker with its own:

• 🧠 Brain (memory)
• 🍳 Kitchen (workspace)
• 📋 Recipe list (code to run)

Each process runs independently. If one chef burns a dish, the others keep cooking!

Why Not Just Threads?

Think of threads like helpers in the same kitchen. They share the same stove, fridge, and counter. Sometimes they bump into each other (that’s a “race condition”).

Processes are in separate kitchens. No bumping! Each has its own everything.

1. Process Creation: Hiring New Chefs

The Simplest Way

from multiprocessing import Process

def cook_dish(dish_name):
    print(f"Cooking {dish_name}...")

# Create a process (hire a chef)
chef = Process(target=cook_dish,
               args=("pasta",))

# Start the process
chef.start()

# Wait for it to finish
chef.join()

What happens?

1. We create a new chef (Process)
2. We tell them to make “pasta”
3. start() sends them to their kitchen
4. join() waits until they’re done

Passing Arguments

def make_order(item, quantity):
    print(f"Making {quantity} {item}s")

p = Process(
    target=make_order,
    args=("cookie", 12)  # tuple!
)
p.start()
p.join()

Remember: args must be a tuple. Even for one item: args=("pasta",) with that comma!

2. Managing Processes: Running the Kitchen

Starting Multiple Chefs

from multiprocessing import Process

def cook(chef_id, dish):
    print(f"Chef {chef_id}: {dish}")

# Create a team of chefs
chefs = []
dishes = ["soup", "salad", "bread"]

for i, dish in enumerate(dishes):
    p = Process(target=cook,
                args=(i, dish))
    chefs.append(p)

# Start all at once
for p in chefs:
    p.start()

# Wait for all to finish
for p in chefs:
    p.join()

Checking Process Status

p = Process(target=cook_dish,
            args=("steak",))
p.start()

# Is the chef still cooking?
print(p.is_alive())  # True or False

# What's their process ID?
print(p.pid)  # Like a badge number

# Wait with timeout
p.join(timeout=5)  # Wait max 5 secs

Naming Your Processes

p = Process(
    target=cook_dish,
    args=("pizza",),
    name="Pizza-Chef"  # Give a name!
)
print(p.name)  # "Pizza-Chef"

Stopping a Process

p.terminate()  # Stop immediately!
p.kill()       # Force stop (Python 3.7+)

Warning: terminate() is like shouting “STOP!” mid-recipe. The dish won’t be finished!

3. Inter-Process Communication: Chefs Talking

Our chefs are in separate kitchens. How do they share information?

Method 1: Queue (The Order Window)

Think of a Queue like a restaurant order window. One chef puts orders in, another takes them out.

from multiprocessing import (
    Process, Queue
)

def chef(q):
    while True:
        order = q.get()  # Take order
        if order == "DONE":
            break
        print(f"Making: {order}")

# Create the order window
orders = Queue()

# Start the chef
p = Process(target=chef,
            args=(orders,))
p.start()

# Send orders
orders.put("burger")
orders.put("fries")
orders.put("DONE")  # Signal to stop

p.join()

Queue Rules:

• put() = add an item
• get() = take an item (waits if empty)
• Works like a line: first in, first out

Method 2: Pipe (Direct Phone Line)

A Pipe is like a phone between two chefs. They can talk back and forth!

from multiprocessing import (
    Process, Pipe
)

def chef(conn):
    order = conn.recv()  # Listen
    conn.send(f"{order} is ready!")
    conn.close()

# Create phone connection
main_phone, chef_phone = Pipe()

p = Process(target=chef,
            args=(chef_phone,))
p.start()

# Send order, get response
main_phone.send("pizza")
response = main_phone.recv()
print(response)  # "pizza is ready!"

p.join()

Method 3: Value & Array (Shared Chalkboard)

Sometimes chefs need to share a single number or list. Like a shared chalkboard in the hallway!

from multiprocessing import (
    Process, Value, Array
)

def count_orders(counter, items):
    counter.value += 1
    items[0] = 99  # Change first item

# 'i' means integer, 'd' means decimal
counter = Value('i', 0)
items = Array('i', [1, 2, 3])

p = Process(target=count_orders,
            args=(counter, items))
p.start()
p.join()

print(counter.value)  # 1
print(items[:])       # [99, 2, 3]

Comparison: Which to Use?

graph TD
    A["Need to Share Data?"] --> B{What kind?}
    B -->|Stream of items| C["Queue"]
    B -->|Two-way chat| D["Pipe"]
    B -->|Single value| E["Value"]
    B -->|List of numbers| F["Array"]

4. ProcessPoolExecutor: The Restaurant Manager

Managing many chefs manually is hard. What if we had a manager who handles everything?

That’s ProcessPoolExecutor! It:

• Creates a team of workers (pool)
• Assigns tasks automatically
• Collects results for you

The Easy Way: map()

from concurrent.futures import (
    ProcessPoolExecutor
)

def cook(dish):
    return f"{dish} is ready!"

dishes = ["soup", "salad", "steak"]

# Manager creates 3 workers
with ProcessPoolExecutor(
    max_workers=3
) as manager:
    # Cook all dishes in parallel
    results = manager.map(cook, dishes)

for r in results:
    print(r)
# "soup is ready!"
# "salad is ready!"
# "steak is ready!"

Submit Individual Tasks

from concurrent.futures import (
    ProcessPoolExecutor
)

def slow_cook(dish):
    import time
    time.sleep(1)
    return f"{dish} done!"

with ProcessPoolExecutor() as pool:
    # Submit tasks one by one
    future1 = pool.submit(slow_cook,
                          "roast")
    future2 = pool.submit(slow_cook,
                          "pie")

    # Get results when ready
    print(future1.result())
    print(future2.result())

as_completed: First Done, First Out

from concurrent.futures import (
    ProcessPoolExecutor,
    as_completed
)

def cook(dish):
    import time
    import random
    time.sleep(random.uniform(0.5, 2))
    return dish

dishes = ["soup", "pasta", "bread"]

with ProcessPoolExecutor() as pool:
    futures = {
        pool.submit(cook, d): d
        for d in dishes
    }

    # Get results as they finish
    for future in as_completed(futures):
        dish = futures[future]
        result = future.result()
        print(f"{dish} finished!")

Error Handling

def risky_cook(dish):
    if dish == "disaster":
        raise ValueError("Burned!")
    return f"{dish} OK"

with ProcessPoolExecutor() as pool:
    future = pool.submit(risky_cook,
                         "disaster")
    try:
        result = future.result()
    except ValueError as e:
        print(f"Oops: {e}")

Quick Reference Table

The Big Picture

graph TD
    A["Main Program"] --> B["Create Processes"]
    B --> C["Process 1"]
    B --> D["Process 2"]
    B --> E["Process 3"]
    C --> F["Do Work"]
    D --> F
    E --> F
    F --> G["Communicate via Queue/Pipe"]
    G --> H["Collect Results"]
    H --> I["Done!"]

When to Use Multiprocessing?

✅ Use it when:

• Heavy calculations (math, images)
• Processing many files
• Tasks that don’t share data often

❌ Don’t use when:

• Waiting for web requests (use threads)
• Tasks need to share lots of data
• Starting many short tasks (overhead)

You Did It! 🎉

You now understand:

1. Process Creation - Making new workers
2. Managing Processes - Starting, stopping, waiting
3. Communication - Queue, Pipe, shared memory
4. ProcessPoolExecutor - The easy manager

Like running a kitchen with many chefs, multiprocessing lets your Python code do many things at once. Each process has its own space, works independently, and together they get the big job done faster!

Now go build something amazing! 🚀