[Test] Stabilize test_multiple_efs, test_queue_parameter_update, test_proxy, test_build_no_internet, test_cluster_with_gpu_health_checks

[gmarciani]

Description of changes

Stabilize tests:

• test_multiple_efs: Retry EFS mount assertion for IAM-authorized EFS in test_multiple_efs. The IAM auth can introduce not impactful delays in mounting on reboot.
• test_queue_parameter_update: fix flakiness by asserting the TERMINATE requeue deterministically.
  The TERMINATE branch snapshotted the transient CONFIGURING state via a single-shot after 10 seconds. If the comphute node boots quicker than the update, then the job goes running, not configuring. With this fix we assert on what matters, i.e. the job gets requeued then goes running on the replaced node.
• test_proxy, test_build_no_internet: fixed cfn resource dependency issue that could cause the timeout of ProxyReadyWaitCondition.
• test_cluster_with_gpu_health_checks: reduced the risk of ICEs by ensuring the test consume the reserved capacity.
• test_patching_cluster: prevent false alarm by tolerating the lazily loaded kernel module crc32_generic.

Tests

PENDING

test-suites:
  storage:
    # Cluster: head + queue-0 (MinCount 1) + queue-1 (MinCount 1) = 3 static c5.xlarge nodes.
    test_efs.py::test_multiple_efs:
      dimensions:
        - regions: [{{ c5_xlarge_CAPACITY_RESERVATION_4_INSTANCES_3_HOURS_NOPG_alinux2023 }}]
          instances: ["c5.xlarge"]
          oss: ["alinux2023", "ubuntu2404"]
          schedulers: ["slurm"]
        - regions: [{{ c5_xlarge_CAPACITY_RESERVATION_4_INSTANCES_3_HOURS_NOPG_rhel9 }}]
          instances: ["c5.xlarge"]
          oss: ["rhel9"]
          schedulers: ["slurm"]
  update:
    # Cluster: head + queue1 (MinCount 1, MaxCount 2) + queue2 (MinCount 1, MaxCount 2). The resize
    # path can reach 5 c5.xlarge nodes, so the reservation is sized to 5.
    test_update.py::test_queue_parameters_update:
      dimensions:
        - regions: [{{ c5_xlarge_CAPACITY_RESERVATION_5_INSTANCES_3_HOURS_NOPG_alinux2023 }}]
          instances: ["c5.xlarge"]
          oss: ["alinux2023", "ubuntu2404"]
          schedulers: ["slurm"]
        - regions: [{{ c5_xlarge_CAPACITY_RESERVATION_5_INSTANCES_3_HOURS_NOPG_rhel9 }}]
          instances: ["c5.xlarge"]
          oss: ["rhel9"]
          schedulers: ["slurm"]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.07%. Comparing base (abee849) to head (98e70dc).
⚠️ Report is 4 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #7568      +/-   ##
===========================================
+ Coverage    89.94%   90.07%   +0.12%     
===========================================
  Files          180      180              
  Lines        16255    16391     +136     
===========================================
+ Hits         14621    14764     +143     
+ Misses        1634     1627       -7     

Flag	Coverage Δ
unittests	90.07% <ø> (+0.12%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.