Wip/mgiacomo/3160/release316 with test fix 0819 1 - #7573
Closed
gmarciani wants to merge 19 commits into
Closed
Conversation
… develop.yaml Reason: 1. Easier for maintenance 2. Less likely of insufficient capacity because we are consolidating capacity reservation
test_trainium doesn't exist in released.yaml. So only one removal is needed
Fractional-GPU instances require the NVIDIA GRID/vGPU driver. ParallelCluster installs the data-center driver instead, so it cannot bind to the vGPU and cluster creation fails.
1. use dynamic capacity reservation to reduce the risk of ICE. 2. reduce the number of compute nodes to the strictly required ones. 3. reduce the risk of dpkg locking failure by disabling boot-time apt jobs.
… is disabled (aws#7560) * Fix cluster creation error when login nodes are used and cloudwatch is disabled * Add unit test for dna json when cloudwatch is disabled * Fix tox --------- Co-authored-by: Helena Greebe <hgreebe@amazon>
… are missing
PinVersion installs linux-headers/linux-modules-extra for the running kernel
but ignored the exit code of apt-get.
The apt index could be outdated:
```
E: Failed to fetch .../wireless-regdb_2025.10.07-0ubuntu1~24.04.1_all.deb
404 Not Found
E: Unable to fetch some archives, maybe run apt-get update or try with
--fix-missing?
CmdExecution: ExitCode 0
```
`apt` aborted the transaction, so linux-modules-extra was never installed, the
step reported success.
Changes, limited to Ubuntu:
- run `apt-get update` before the install
- check the install exit code and abort the build with a clear message
…revent failures in test_build_image_no_internet. This is what a user would do to support the build-image in proxied environments. The proxy configuration is also validated by the proxy client, which verifies that every mirror is allowlisted. In this way the build instance can reach whichever EPEL mirror it is assigned.
…n on all OSS, but when DLAMI is used.
This commit partially revert aws#7563, while keeping the `apt-get -y update` addition This is to maximize compatibility with different Ubuntu versions. For example, [Ubuntu 24 with kernel 7 deprecated linux-modules-extra](https://discourse.ubuntu.com/t/kernel-development-release-cadence-and-deprecation-of-linux-modules-extra/65176)
…ailing the test (aws#7565) 27% of v3.16.0 CI runs have at least one failing OSU benchmark, concentrated in the 32-node c5n.18xlarge collectives. It is not a performance regression: on 38 fresh 32-instance draws, the first collective in a job intermittently spikes by an order of magnitude at the small packet sizes while the median of that same sweep stays at ~1.0x baseline, and it is gone from the next sweep on. The trip repeated on the immediately following sweep only once in twelve. Measure once, as before. Only when the check trips, measure that benchmark twice more inside a single job on the same nodes and take the verdict on the median per packet size. Per-draw failure rate goes from 11/16 to 1/16, and the one that remains genuinely reproduced. Detection is unaffected: a uniform 1.25x, 1.30x or 1.50x regression injected into real sweeps is still caught 13 of 13 times. Every repetition is recorded in the results table, not just the median. The verdict uses the median, but the spike that triggered the re-measurement only exists in the repetition it happened in, and that is the signal a future investigation would need. A re-measured benchmark therefore contributes three rows instead of one, each in the shape a single run produces, so the historical report is unaffected in shape and the two extra points are the clean ones. The extra time is paid only on runs that trip, so 73% of runs are unchanged and the p95 is 11.9 min, almost all of it alltoall.
…el modules on every head node to prevent false alarms.
…es via COMMON_HEAD_NODE_LAZY_MODULES to avoid code duplication.
…ut by decoupling ProxyVerificationWaitCondition from the Proxy instance. Also reduced the risk of proxy infra deployment failure by increasing the timeout of the proxy instance to not fail on slow user data.
…all the instances required by the test and let the cluster use the specific subnet where they have been reserved.
…ple_efs. IAM policy evaluation can transiently delay the mount on head node reboot, causing flaky failures. Retry the mount check only for IAM-authorized EFS.
…ue deterministically The TERMINATE branch snapshotted the transient CONFIGURING state via a single-shot assert_job_state after a fixed sleep(10). When replacement nodes boot quickly the whole node swap finishes inside the update-cluster --wait window, so the job is already RUNNING again by the time the test samples it, causing false failures. Assert the deterministic outcome instead: the job was requeued (wait_job_requeued) and is running again, and the original instance it ran on was terminated (wait_instance_replaced_or_terminating). Add SlurmCommands.get_job_instance_id to resolve the job's instance via Slurm, and wait_job_requeued helper.
…ed by dpkg user prompts. Such reconfiguration may occur when efs-utils get upgraded as part of the patching.
… from cluster config, which is not required for this test.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #7573 +/- ##
===========================================
+ Coverage 89.94% 90.07% +0.12%
===========================================
Files 180 180
Lines 16254 16391 +137
===========================================
+ Hits 14620 14764 +144
+ Misses 1634 1627 -7
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
Tests
References
Checklist
developadd the branch name as prefix in the PR title (e.g.[release-3.6]).Please review the guidelines for contributing and Pull Request Instructions.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.