Handling Keyboard Interrupts in Python's Multiprocessing Pool

In Python's multiprocessing module, the Pool class provides a convenient way to distribute tasks across multiple processes. However, handling KeyboardInterrupt events in Pools can be challenging, as demonstrated by the code snippet:

from multiprocessing import Pool
from time import sleep
from sys import exit

def slowly_square(i):
    sleep(1)
    return i*i

def go():
    pool = Pool(8)
    try:
        results = pool.map(slowly_square, range(40))
    except KeyboardInterrupt:
        # **** THIS PART NEVER EXECUTES. ****
        pool.terminate()
        print "You cancelled the program!"
        sys.exit(1)
    print "\nFinally, here are the results: ", results

if __name__ == "__main__":
    go()

When running this code, pressing Ctrl C will not trigger the cleanup process, leaving the subprocesses running indefinitely. To address this issue, consider the following workaround:

The behavior observed in the code is a consequence of a Python bug. KeyboardInterrupt is not sent when waiting for a condition in threading.Condition.wait(). As Pool.map() uses a condition wait internally, the interrupt is never received.

A solution is to use Pool.map_async(), which allows specifying a timeout. By setting a sufficiently long timeout (e.g., 9999999), we can ensure that the interrupt will be triggered within a reasonable time.

Therefore, replace:

    results = pool.map(slowly_square, range(40))

with:

    results = pool.map_async(slowly_square, range(40)).get(9999999)

This workaround provides a way to gracefully handle KeyboardInterrupt events in multiprocessing Pools, allowing for the termination of all subprocesses when the user cancels the program.