[Integ test] Modify test_efa to verify EFA with FsxLustre tutorial

[himani2411]

Description of changes

• Modify test_efa to test tutorial https://docs.aws.amazon.com/parallelcluster/latest/ug/tutorial-efa-enabled-fsx-lustre.html
• Run EFA with FSxL tutorail only when  supported OS and Instance type are being used
• Provision EFA FSx Lustre via shared storage stack
• Dump LNet state for future debugging
• Assert Lustre rides EFA by LNet send counters, not the import paramters; write 4 GiB of direct I/O to the mount, sum again, then require @efa to gain at least half the expected bulk RPCs and more than @tcp gained. Metadata RPCs and pings ride @tcp by design, so the second check is an ordering rather than tcp_delta == 0.
• All the Nics for trn1.32xlareg instance are supposed to be bound unlike what Fsx specifies in their doc https://docs.aws.amazon.com/fsx/latest/LustreGuide/configure-efa-clients.html#add-efa-interfaces

Tests

• Integ test with [Pcluster-diag] Modify FSx checks to remove unwanted checks aws-parallelcluster-cookbook#3267

efa:
    test_efa.py::test_efa:
      dimensions:
        - regions: [{{ c6gn_16xlarge_CAPACITY_RESERVATION_2_INSTANCES_1_HOURS_YESPG_OS_ARM_0 }}]
          instances: ["c6gn.16xlarge"] # Single Nic
          oss: [{{ OS_ARM_0 }}]
          schedulers: ["slurm"]
        - regions: ["usw2-az4"]
          schedulers: ["slurm"]
          oss: ["ubuntu2404"]
          instances: ["trn1.32xlarge"] # Multi Nic
        - regions: [{{ c5n_18xlarge_CAPACITY_RESERVATION_2_INSTANCES_1_HOURS_YESPG_OS_X86_0 }}]
          instances: ["c5n.18xlarge"]  # No support for efa+FsxL feature
          oss: [{{ OS_X86_0 }}]
          schedulers: ["slurm"]

References

• Link to impacted open issues.
• Link to related PRs in other packages (i.e. cookbook, node).
• Link to documentation useful to understand the changes.

Checklist

• Make sure you are pointing to the right branch.
• If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
• Check all commits' messages are clear, describing what and why vs how.
• Make sure to have added unit tests or integration tests to cover the new/modified code.
• Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[himani2411]

NOTE FOR REVIWER: we need to unload the lnet module as latest FSX script configure-efa-fsx-lustre-client.zip fails with not being able to bind all the EFA devices. This fails our pcluster-diag checks and configure-efa-fsx-lustre-client.zip errors out.

[himani2411]

NOTE FOR REVIWER: Confirmed and verified by looking at FSX script that the table here is stale with respect to Other instances with multiple network cards entry. https://docs.aws.amazon.com/fsx/latest/LustreGuide/configure-efa-clients.html#add-efa-interfaces. The Fsx script binds all the EFA devices instead of the previously default 2 efa devices.

[himani2411]

NOTE FOR REVIWER: I am logging these commands for debugging purpose which are helpful in understanding the state of node. These commands were helpful in finding that I was polluting the peer table and hence led to improvement in pcluster-diag checks.