I have recently discovered a bug in Python (both in the 2.x and 3.x families) and offered
a patch to solve the issue.
When imap() or imap_unordered() are called with the iterable parameter set as a generator function, and when that generator function raises an exception, then the _task_handler thread (running the method _handle_tasks) dies immediately, without causing the other threads to stop and without reporting the exception to the main thread (that one that called imap()).
I saw this issue in Python 2.7.8, 2.7.9 and 3.4.2. Didn’t check other versions, but I assume this is a bug in all Python versions since 2.6.
I reported this bug here and attached examples that reproduce this issue, as well as patches for both Python 2.7 and Python 3.4.
The patches I attached do 2 things:
- A deadlock is prevented, wherein the main thread waits forever for the Pool thread/s to finish their execution, while they wait for instructions to terminate from the _task_handler thread which has died. Instead, the exception are caught and handled and termination of the pool execution is performed.
- The exception that was raised is caught and passed to the main thread, and is re-thrown in the context of the main thread – hence the user catch it and handle it, or – at the very least – be aware of the issue.
Now I’m waiting for a review.
Update, 2015-03-06: patch was reviewed, tests were added, and it was merged into all
Python branches.