Parallel computing in Python (in 60 seconds or less)

Hey there,

If your Python programs are slower than you'd like you can often speed them up by *parallelizing* them.

Basically, parallel computing allows you to carry out many calculations at the same time, thus reducing the amount of time it takes to run your program to completion.

I know, this sounds fairly vague and complicated somehow...but bear with me for the next 50 seconds or so.

Here's an end-to-end example of parallel computing in Python 2/3, using only tools built into the Python standard library—

Ready? Go!

First, we need to do some setup work. We'll import the "collections" and the "multiprocessing" module so we can use Python's parallel computing facilities and define the data structure we'll work with:

import collections
import multiprocessing

Second, we'll use "collections.namedtuple" to define a new (immutable) data type we can use to represent our data set, a collection of scientists:

Scientist = collections.namedtuple('Scientist', [
    'name',
    'born',
])

scientists = (
    Scientist(name='Ada Lovelace', born=1815),
    Scientist(name='Emmy Noether', born=1882),
    Scientist(name='Marie Curie', born=1867),
    Scientist(name='Tu Youyou', born=1930),
    Scientist(name='Ada Yonath', born=1939),
    Scientist(name='Vera Rubin', born=1928),
    Scientist(name='Sally Ride', born=1951),
)

Third, we'll write a "data processing function" that accepts a scientist object and returns a dictionary containing the scientist's name and their calculated age: 

def process_item(item):
    return {
        'name': item.name,
        'age': 2017 - item.born
    }

The process_item() function just represents a simple data transformation to keep this example short and sweet—but you could swap it out with a much more complex computation no problem. 

(20 seconds remaining)

Fourth, and this is where the real parallelization magic happens, we'll set up a "multiprocessing pool" that allows us to spread our calculations across all available CPU cores. 

Then we call the pool's map() method to apply our process_item() function to all scientist objects, in parallel batches:

pool = multiprocessing.Pool()
result = pool.map(process_item, scientists)

Note how batching and distributing the work across multiple CPU cores, performing the work, and collecting the results are all handled by the multiprocessing pool. How great is that?

Fifth, we're all done here with 5 seconds remaining—

Let's print the results of our data transformation to the console so we can make sure the program did what it was supposed to:

print(tuple(result))

That's the end of our little program. And here's what you should expect to see printed out on your console:

({'name': 'Ada Lovelace', 'age': 202},
 {'name': 'Emmy Noether', 'age': 135},
 {'name': 'Marie Curie', 'age': 150},
 {'name': 'Tu Youyou', 'age': 87},
 {'name': 'Ada Yonath', 'age': 78},
 {'name': 'Vera Rubin', 'age': 89},
 {'name': 'Sally Ride', 'age': 66})

Isn't Python just lovely?

Now, obviously I took some shortcuts here and picked an example that made parallelization seem effortless—

But, I stand by the lessons learned here:

- If you know how to structure and represent your data, parallelization is convenient and feels completely natural. As a Pythonista, you should pick up the basics of functional programming for this reason.

- Python is a joy to work with and eminently suitable for these kinds of programming tasks.

— Dan Bader