Skip to content

[Test] Stabilize test_multiple_efs, test_queue_parameter_update, test_proxy, test_build_no_internet, test_cluster_with_gpu_health_checks - #7569

Draft
gmarciani wants to merge 8 commits into
aws:integ-tests-3.16.0from
gmarciani:wip/mgiacomo/3160/fix-tests-0818-1
Draft

[Test] Stabilize test_multiple_efs, test_queue_parameter_update, test_proxy, test_build_no_internet, test_cluster_with_gpu_health_checks#7569
gmarciani wants to merge 8 commits into
aws:integ-tests-3.16.0from
gmarciani:wip/mgiacomo/3160/fix-tests-0818-1

Conversation

@gmarciani

Copy link
Copy Markdown
Contributor

Description of changes

Stabilize tests:

  • test_multiple_efs: Retry EFS mount assertion for IAM-authorized EFS in test_multiple_efs. The IAM auth can introduce not impactful delays in mounting on reboot.
  • test_queue_parameter_update: fix flakiness by asserting the TERMINATE requeue deterministically.
    The TERMINATE branch snapshotted the transient CONFIGURING state via a single-shot after 10 seconds. If the comphute node boots quicker than the update, then the job goes running, not configuring. With this fix we assert on what matters, i.e. the job gets requeued then goes running on the replaced node.
  • test_proxy, test_build_no_internet: fixed cfn resource dependency issue that could cause the timeout of ProxyReadyWaitCondition.
  • test_cluster_with_gpu_health_checks: reduced the risk of ICEs by ensuring the test consume the reserved capacity.
  • test_patching_cluster: prevent false alarm by tolerating the lazily loaded kernel module crc32_generic.

Tests

PENDING

test-suites:
  storage:
    # Cluster: head + queue-0 (MinCount 1) + queue-1 (MinCount 1) = 3 static c5.xlarge nodes.
    test_efs.py::test_multiple_efs:
      dimensions:
        - regions: [{{ c5_xlarge_CAPACITY_RESERVATION_4_INSTANCES_3_HOURS_NOPG_alinux2023 }}]
          instances: ["c5.xlarge"]
          oss: ["alinux2023", "ubuntu2404"]
          schedulers: ["slurm"]
        - regions: [{{ c5_xlarge_CAPACITY_RESERVATION_4_INSTANCES_3_HOURS_NOPG_rhel9 }}]
          instances: ["c5.xlarge"]
          oss: ["rhel9"]
          schedulers: ["slurm"]
  update:
    # Cluster: head + queue1 (MinCount 1, MaxCount 2) + queue2 (MinCount 1, MaxCount 2). The resize
    # path can reach 5 c5.xlarge nodes, so the reservation is sized to 5.
    test_update.py::test_queue_parameters_update:
      dimensions:
        - regions: [{{ c5_xlarge_CAPACITY_RESERVATION_5_INSTANCES_3_HOURS_NOPG_alinux2023 }}]
          instances: ["c5.xlarge"]
          oss: ["alinux2023", "ubuntu2404"]
          schedulers: ["slurm"]
        - regions: [{{ c5_xlarge_CAPACITY_RESERVATION_5_INSTANCES_3_HOURS_NOPG_rhel9 }}]
          instances: ["c5.xlarge"]
          oss: ["rhel9"]
          schedulers: ["slurm"]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…el modules on every head node to prevent false alarms.
…es via COMMON_HEAD_NODE_LAZY_MODULES to avoid code duplication.
…ut by decoupling ProxyVerificationWaitCondition from the Proxy instance.

Also reduced the risk of proxy infra deployment failure by increasing the timeout of the proxy instance to not fail on slow user data.
…all the instances required by the test and let the cluster use the specific subnet where they have been reserved.
…ple_efs.

IAM policy evaluation can transiently delay the mount on head node reboot, causing flaky failures. Retry the mount check only for IAM-authorized EFS.
…ue deterministically

The TERMINATE branch snapshotted the transient CONFIGURING state via a single-shot
assert_job_state after a fixed sleep(10). When replacement nodes boot quickly the whole
node swap finishes inside the update-cluster --wait window, so the job is already RUNNING
again by the time the test samples it, causing false failures.

Assert the deterministic outcome instead: the job was requeued (wait_job_requeued) and is
running again, and the original instance it ran on was terminated
(wait_instance_replaced_or_terminating). Add SlurmCommands.get_job_instance_id to resolve
the job's instance via Slurm, and wait_job_requeued helper.
@gmarciani
gmarciani changed the base branch from develop to integ-tests-3.16.0 August 18, 2026 22:02
@gmarciani gmarciani added skip-changelog-update Disables the check that enforces changelog updates in PRs 3.x Test labels Aug 18, 2026
…ed by dpkg user prompts.

Such reconfiguration may occur when efs-utils get upgraded as part of the patching.
… from cluster config, which is not required for this test.
@gmarciani
gmarciani force-pushed the wip/mgiacomo/3160/fix-tests-0818-1 branch from 09111cf to f7a5e26 Compare August 19, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.x skip-changelog-update Disables the check that enforces changelog updates in PRs Test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant