Skip to content

[Test] Double the MPI ring job timeout to absorb cold cluster starts for stabilizing test_efa - #7570

Merged
hehe7318 merged 1 commit into
aws:developfrom
hehe7318:wip/add-warmup-to-test-efa
Aug 19, 2026
Merged

[Test] Double the MPI ring job timeout to absorb cold cluster starts for stabilizing test_efa#7570
hehe7318 merged 1 commit into
aws:developfrom
hehe7318:wip/add-warmup-to-test-efa

Conversation

@hehe7318

Copy link
Copy Markdown
Contributor

Description of changes

The timeout handed to mpi_submit_openmpi.sh was 20 seconds on 48-slot instances, which left no headroom over the observed runtime. The first MPI launch in a freshly created cluster reads its binaries and libraries from an EBS root volume whose blocks are still lazy-loading from the AMI snapshot, inflating that one launch by an order of magnitude and making test_efa fail on grounds that have nothing to do with EFA.

Checklist

  • Make sure you are pointing to the right branch.
  • If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
  • Check all commits' messages are clear, describing what and why vs how.
  • Make sure to have added unit tests or integration tests to cover the new/modified code.
  • Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

The timeout handed to mpi_submit_openmpi.sh was 20 seconds on 48-slot
instances, which left no headroom over the observed runtime. The first MPI
launch in a freshly created cluster reads its binaries and libraries from an
EBS root volume whose blocks are still lazy-loading from the AMI snapshot,
inflating that one launch by an order of magnitude and making test_efa fail
on grounds that have nothing to do with EFA.
@hehe7318
hehe7318 requested review from a team as code owners August 19, 2026 16:13
@hehe7318 hehe7318 added skip-changelog-update Disables the check that enforces changelog updates in PRs 3.x labels Aug 19, 2026
@hehe7318 hehe7318 changed the title [Test] Double the MPI ring job timeout to absorb cold cluster starts. [Test] Double the MPI ring job timeout to absorb cold cluster starts for stabilizing test_efa Aug 19, 2026
@hehe7318
hehe7318 enabled auto-merge (squash) August 19, 2026 16:58
@hehe7318
hehe7318 disabled auto-merge August 19, 2026 16:58
@hehe7318
hehe7318 enabled auto-merge (squash) August 19, 2026 16:59
@hehe7318
hehe7318 merged commit a8e5c32 into aws:develop Aug 19, 2026
24 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.x skip-changelog-update Disables the check that enforces changelog updates in PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants