Graceful Handling of Keyboard Interrupts in Python Multiprocessing Pools

When working with Python's multiprocessing Pool, handling KeyboardInterrupt events is not always straightforward. In this article, we will explore how to handle such interrupts and ensure that the processes exit gracefully.

The provided code sample demonstrates the challenge. Despite having a catch block for KeyboardInterrupt, it does not execute when control-C is pressed. Instead, the program hangs until it is terminated externally.

The root of this issue lies in a Python bug related to blocking on a condition in threading.Condition.wait(). In this context, KeyboardInterrupt is never sent. As a result, it prevents the interrupt from being handled within the Pool.

One solution to this problem is to specify a timeout when waiting for the results. The map_async() method with a timeout parameter can be used instead of map(). This approach allows the KeyboardInterrupt to be recognized and processed:

results = pool.map_async(slowly_square, range(40)).get(9999999)

By setting a large timeout value, we essentially tell the Pool to continue waiting for the results until interrupted by the user.

It is important to note that there are some limitations to this workaround. If the timeout expires before all tasks are complete, the incomplete results will be discarded. Therefore, it is crucial to choose an appropriate timeout value that balances responsiveness with the risk of incomplete results.