Parallel computing in Python (in 60 seconds or less)

Hey there,

If your Python programs are slower than you'd like you can often speed them up by *parallelizing* them.

Basically, parallel computing allows you to carry out many calculations at the same time, thus reducing the amount of time it takes to run your program to completion.

I know, this sounds fairly vague and complicated somehow...but bear with me for the next 50 seconds or so.

Here's an end-to-end example of parallel computing in Python 2/3, using only tools built into the Python standard library—

Ready? Go!

First, we need to do some setup work. We'll import the "collections" and the "multiprocessing" module so we can use Python's parallel computing facilities and define the data structure we'll work with:

import collections
import multiprocessing

Second, we'll use "collections.namedtuple" to define a new (immutable) data type we can use to represent our data set, a collection of scientists:

Scientist = collections.namedtuple('Scientist', [
    'name',
    'born',
])

scientists = (
    Scientist(name='Ada Lovelace', born=1815),
    Scientist(name='Emmy Noether', born=1882),
    Scientist(name='Marie Curie', born=1867),
    Scientist(name='Tu Youyou', born=1930),
    Scientist(name='Ada Yonath', born=1939),
    Scientist(name='Vera Rubin', born=1928),
    Scientist(name='Sally Ride', born=1951),
)

Third, we'll write a "data processing function" that accepts a scientist object and returns a dictionary containing the scientist's name and their calculated age: 

def process_item(item):
    return {
        'name': item.name,
        'age': 2017 - item.born
    }

The process_item() function just represents a simple data transformation to keep this example short and sweet—but you could swap it out with a much more complex computation no problem. 

(20 seconds remaining)

Fourth, and this is where the real parallelization magic happens, we'll set up a "multiprocessing pool" that allows us to spread our calculations across all available CPU cores. 

Then we call the pool's map() method to apply our process_item() function to all scientist objects, in parallel batches:

pool = multiprocessing.Pool()
result = pool.map(process_item, scientists)

Note how batching and distributing the work across multiple CPU cores, performing the work, and collecting the results are all handled by the multiprocessing pool. How great is that?

Fifth, we're all done here with 5 seconds remaining—

Let's print the results of our data transformation to the console so we can make sure the program did what it was supposed to:

print(tuple(result))

That's the end of our little program. And here's what you should expect to see printed out on your console:

({'name': 'Ada Lovelace', 'age': 202},
 {'name': 'Emmy Noether', 'age': 135},
 {'name': 'Marie Curie', 'age': 150},
 {'name': 'Tu Youyou', 'age': 87},
 {'name': 'Ada Yonath', 'age': 78},
 {'name': 'Vera Rubin', 'age': 89},
 {'name': 'Sally Ride', 'age': 66})

Isn't Python just lovely?

Now, obviously I took some shortcuts here and picked an example that made parallelization seem effortless—

But, I stand by the lessons learned here:

- If you know how to structure and represent your data, parallelization is convenient and feels completely natural. As a Pythonista, you should pick up the basics of functional programming for this reason.

- Python is a joy to work with and eminently suitable for these kinds of programming tasks.

— Dan Bader

Older messages

Python side projects & getting a job

Sunday, December 20, 2020

Hey there, Where do you find inspiration for Python side projects? It's a question I often hear from Pythonistas looking to build up a portfolio they can show to potential employers. So here's

[Sublime + Python Setup] Sublime Text is just a blank canvas…

Saturday, December 19, 2020

Hey there, When I became serious about optimizing Sublime Text with plugins, it was hard for me to separate the wheat from the chaff. Without a real guideline or roadmap I resorted to installing *any*

[PythonistaCafe] Q&A

Saturday, December 19, 2020

Hey there, At this point you should have a pretty good idea of what PythonistaCafe is about and what makes it special. In this email I want to answer some common questions that I get asked about the

[Sublime + Python Setup] Grumpy old greybeard with a whitespace problem

Friday, December 18, 2020

One fateful day, the Agile Gods that be decided to “add some firepower” to my little team… And so, developer Paul joined (name changed to protect the guilty). Before I dive into this story, let me ask

[PythonistaCafe] What's in PythonistaCafe for you?

Friday, December 18, 2020

Hey there, A couple of years ago I'd become quite interested in martial arts. Hours upon hours of watching "The Karate Kid" growing up must've taken their toll on me... And so, I

You Might Also Like

Import AI 399: 1,000 samples to make a reasoning model; DeepSeek proliferation; Apple's self-driving car simulator

Friday, February 14, 2025

What came before the golem? ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏

Defining Your Paranoia Level: Navigating Change Without the Overkill

Friday, February 14, 2025

We've all been there: trying to learn something new, only to find our old habits holding us back. We discussed today how our gut feelings about solving problems can sometimes be our own worst enemy

5 ways AI can help with taxes 🪄

Friday, February 14, 2025

Remotely control an iPhone; 💸 50+ early Presidents' Day deals -- ZDNET ZDNET Tech Today - US February 10, 2025 5 ways AI can help you with your taxes (and what not to use it for) 5 ways AI can help

Recurring Automations + Secret Updates

Friday, February 14, 2025

Smarter automations, better templates, and hidden updates to explore 👀 ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏

The First Provable AI-Proof Game: Introducing Butterfly Wings 4

Friday, February 14, 2025

Top Tech Content sent at Noon! Boost Your Article on HackerNoon for $159.99! Read this email in your browser How are you, @newsletterest1? undefined The Market Today #01 Instagram (Meta) 714.52 -0.32%

GCP Newsletter #437

Friday, February 14, 2025

Welcome to issue #437 February 10th, 2025 News BigQuery Cloud Marketplace Official Blog Partners BigQuery datasets now available on Google Cloud Marketplace - Google Cloud Marketplace now offers

Charted | The 1%'s Share of U.S. Wealth Over Time (1989-2024) 💰

Friday, February 14, 2025

Discover how the share of US wealth held by the top 1% has evolved from 1989 to 2024 in this infographic. View Online | Subscribe | Download Our App Download our app to see thousands of new charts from

The Great Social Media Diaspora & Tapestry is here

Friday, February 14, 2025

Apple introduces new app called 'Apple Invites', The Iconfactory launches Tapestry, beyond the traditional portfolio, and more in this week's issue of Creativerly. Creativerly The Great

Daily Coding Problem: Problem #1689 [Medium]

Friday, February 14, 2025

Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Google. Given a linked list, sort it in O(n log n) time and constant space. For example,

📧 Stop Conflating CQRS and MediatR

Friday, February 14, 2025

​ Stop Conflating CQRS and MediatR Read on: m​y website / Read time: 4 minutes The .NET Weekly is brought to you by: Step right up to the Generative AI Use Cases Repository! See how MongoDB powers your