[Test] Stabilize test_osu by confirming a failing collective before failing the test

[hehe7318]

Description of changes

27% of v3.16.0 CI runs have at least one failing OSU benchmark, concentrated in the 32-node c5n.18xlarge collectives. It is not a performance regression: on 38 fresh 32-instance draws, the first collective in a job intermittently spikes by an order of magnitude at the small packet sizes while the median of that same sweep stays at ~1.0x baseline, and it is gone from the next sweep on. The trip repeated on the immediately following sweep only once in twelve.

Measure once, as before. Only when the check trips, measure that benchmark twice more inside a single job on the same nodes and take the verdict on the median per packet size. Per-draw failure rate goes from 11/16 to 1/16, and the one that remains genuinely reproduced. Detection is unaffected: a uniform 1.25x, 1.30x or 1.50x regression injected into real sweeps is still caught 13 of 13 times.

The extra time is paid only on runs that trip, so 73% of runs are unchanged and the p95 is 11.9 min, almost all of it alltoall.

Tests

• Replayed the gate over 40 back-to-back sweeps on one unchanged cluster, and over 38 fresh 32-instance draws (301 sweeps). For intelmpi osu_allreduce the per-draw failure rate goes from 11/16 to 1/16; the one that remains reproduced on later sweeps. Injected uniform 1.25x, 1.30x and 1.50x regressions are still caught 13 of 13 times.
• Ongoing: a re-run of full test_osu

Checklist

• Make sure you are pointing to the right branch.
• If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
• Check all commits' messages are clear, describing what and why vs how.
• Make sure to have added unit tests or integration tests to cover the new/modified code.
• Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.