[Test] Stabilize test_multiple_efs, test_queue_parameter_update, test_proxy, test_build_no_internet, test_cluster_with_gpu_health_checks - #7569
Draft
gmarciani wants to merge 8 commits into
Conversation
…el modules on every head node to prevent false alarms.
…es via COMMON_HEAD_NODE_LAZY_MODULES to avoid code duplication.
…ut by decoupling ProxyVerificationWaitCondition from the Proxy instance. Also reduced the risk of proxy infra deployment failure by increasing the timeout of the proxy instance to not fail on slow user data.
…all the instances required by the test and let the cluster use the specific subnet where they have been reserved.
…ple_efs. IAM policy evaluation can transiently delay the mount on head node reboot, causing flaky failures. Retry the mount check only for IAM-authorized EFS.
…ue deterministically The TERMINATE branch snapshotted the transient CONFIGURING state via a single-shot assert_job_state after a fixed sleep(10). When replacement nodes boot quickly the whole node swap finishes inside the update-cluster --wait window, so the job is already RUNNING again by the time the test samples it, causing false failures. Assert the deterministic outcome instead: the job was requeued (wait_job_requeued) and is running again, and the original instance it ran on was terminated (wait_instance_replaced_or_terminating). Add SlurmCommands.get_job_instance_id to resolve the job's instance via Slurm, and wait_job_requeued helper.
…ed by dpkg user prompts. Such reconfiguration may occur when efs-utils get upgraded as part of the patching.
… from cluster config, which is not required for this test.
gmarciani
force-pushed
the
wip/mgiacomo/3160/fix-tests-0818-1
branch
from
August 19, 2026 17:38
09111cf to
f7a5e26
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
Stabilize tests:
The TERMINATE branch snapshotted the transient CONFIGURING state via a single-shot after 10 seconds. If the comphute node boots quicker than the update, then the job goes running, not configuring. With this fix we assert on what matters, i.e. the job gets requeued then goes running on the replaced node.
ProxyReadyWaitCondition.crc32_generic.Tests
PENDING
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.