gh-101267: [bug] ProcessPoolExecutor should not share one BrokenProcessPool exception among all failed futures#101268
Merged
Merged
Conversation
|
Most changes to Python require a NEWS entry. Please add it using the blurb_it web app or the blurb command-line tool. |
|
All commit authors signed the Contributor License Agreement. |
…among all failed futures (python#101267)
daniel-shields
force-pushed
the
fix-101267
branch
from
2b2e712 to
4d97a88
Compare
|
Most changes to Python require a NEWS entry. Please add it using the blurb_it web app or the blurb command-line tool. |
daniel-shields
changed the title
Contributor
Author
|
Could you please take a look @gpshead? |
arhadthedev
added
stdlib
|
This PR is stale because it has been open for 30 days with no activity. |
1 similar comment
|
This PR is stale because it has been open for 30 days with no activity. |
Adapt the pythongh-101267 fix to the current code shape: * concurrent.futures.process.ProcessPoolExecutor._terminate_broken() was refactored (renamed from terminate_broken(), and gained the exit-code reporting from pythongh-139462). Reapply the fix on top: compute the broken-pool cause once, then create a fresh BrokenProcessPool exception per pending future inside the loop. Preserve the InvalidStateError guard around set_exception(). * Lib/test/test_concurrent_futures.py was split into a package. Move the new regression test into test_concurrent_futures/test_process_pool.py.
…ting Two refinements to the broken-pool handling: * submit() raises BrokenProcessPool(self._broken) so the message says why the pool broke, rather than a generic fixed string. * Build the cause traceback string once before the per-future loop instead of rebuilding the same f-string on every iteration.
…king * Block the workers on a never-set Event (via the harness create_event()) instead of submitting time.sleep(3): the sleep could finish before the worker is terminated on a loaded machine, completing the future and raising no BrokenProcessPool. * Use try/except/else with self.fail() rather than seeding count = None, so a missing exception is reported clearly. assertRaises() can't be used here because it strips the traceback the test needs to inspect.
Describe the actual failure (re-raising one shared exception appends a traceback to it on each Future.result()) and cross-reference the API names.
gpshead
added
needs backport to 3.13
github-actions
Bot
removed
the
stale
gpshead
approved these changes
Jun 13, 2026
|
Thanks @daniel-shields for the PR, and @gpshead for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14, 3.15. |
|
Sorry, @daniel-shields and @gpshead, I could not cleanly backport this to |
|
GH-151430 is a backport of this pull request to the 3.15 branch. |
bedevere-app
Bot
removed
the
needs backport to 3.15
gpshead
added a commit
to miss-islington/cpython
that referenced
this pull request
Jun 13, 2026
…Pool exception among all failed futures (pythonGH-101268) (cherry picked from commit 3c00ebc) Co-authored-by: Daniel Shields <daniel.shields@twosigma.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
gpshead
added a commit
to miss-islington/cpython
that referenced
this pull request
Jun 13, 2026
…Pool exception among all failed futures (pythonGH-101268) (cherry picked from commit 3c00ebc) Co-authored-by: Daniel Shields <daniel.shields@twosigma.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
|
GH-151431 is a backport of this pull request to the 3.14 branch. |
gpshead
added a commit
that referenced
this pull request
Jun 13, 2026
…sPool exception among all failed futures (GH-101268) (#151430) gh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcessPool exception among all failed futures (GH-101268) (cherry picked from commit 3c00ebc) Co-authored-by: Daniel Shields <daniel.shields@twosigma.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
gpshead
added a commit
that referenced
this pull request
Jun 13, 2026
…sPool exception among all failed futures (GH-101268) (#151431) * gh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcessPool exception among all failed futures (GH-101268) (cherry picked from commit 3c00ebc) Co-authored-by: Daniel Shields <daniel.shields@twosigma.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Gregory P. Smith <greg@krypto.org> (cherry picked from commit 4e8c9c6) * Drop the abrupt-exit-code reporting from the 3.14 backport Reporting the exit codes of processes that died without a known cause is a new feature, not part of the gh-101267 bugfix. Keep only the bugfix on 3.14: each failed future gets its own BrokenProcessPool exception instead of one shared instance. --------- Co-authored-by: Daniel Shields <daniel.shields@twosigma.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
gpshead
added a commit
that referenced
this pull request
Jun 13, 2026
…sPool exception among all failed futures (GH-101268) (GH-151431) (#151435) [3.14] gh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcessPool exception among all failed futures (GH-101268) (GH-151431) * gh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcessPool exception among all failed futures (GH-101268) (cherry picked from commit 3c00ebc) (cherry picked from commit 4e8c9c6) * Drop the abrupt-exit-code reporting from the 3.14 backport Reporting the exit codes of processes that died without a known cause is a new feature, not part of the gh-101267 bugfix. Keep only the bugfix on 3.14: each failed future gets its own BrokenProcessPool exception instead of one shared instance. --------- (cherry picked from commit 27ff2c8) Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com> Co-authored-by: Daniel Shields <daniel.shields@twosigma.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please see associated issue for details.