Add premium data-disk performance validation against rated SKU maximum

[SrikanthMyakam (SRIKKANTH)]

Summary

Adds a post-run validation step for the premium data-disk fio performance tests that compares the measured random-read results against the Azure VM SKU's rated disk maximum, so the tests fail when a VM under-performs its published disk limits instead of only recording numbers.

Changes

lisa/microsoft/testsuites/performance/common.py

• Add _get_azure_sku_capabilities() to resolve the Azure SKU capability name/value map for the node's VM size (returns None on non-Azure platforms or when capabilities are unavailable).
• Add check_premium_datadisks_performance() which:
  • Ignores IO depths at or below the saturation threshold (PREMIUM_DATADISK_MIN_IODEPTH = 8).
  • Averages the random-read value per IO depth across iterations.
  • Compares against UncachedDiskIOPS for 4K (IOPS-bound) runs and UncachedDiskBytesPerSecond for 1024K (bandwidth-bound) runs, converting IOPS to bytes/sec via block size where needed.
  • Requires every qualifying IO depth to reach at least PREMIUM_DATADISK_PASS_RATIO (95%) of the rated maximum.
• Raise SkippedException (not a failure) when rated disk performance or the required capability is not published for the VM size/platform.
• Derive the pass percentage in log/assertion messages from PREMIUM_DATADISK_PASS_RATIO instead of a hardcoded 95%.

lisa/microsoft/testsuites/performance/storageperf.py

• Capture the messages returned by perf_premium_datadisks(...) and invoke check_premium_datadisks_performance() in the 4K, 1024K, and io_uring (4K/1024K) premium data-disk test cases.

Notes

• The rated-max validation only applies on Azure; other platforms are skipped.

Description

Related Issue

Type of Change

• Bug fix
• New feature
• Breaking change
• Refactoring
• Documentation update

Checklist

• Description is filled in above
• No credentials, secrets, or internal details are included
• Peer review requested (if not, add required peer reviewers after raising PR)
• Tests executed and results posted below

Test Validation

Key Test Cases:

perf_tcp_ntttcp_sriov

Impacted LISA Features:

Tested Azure Marketplace Images:

• Canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest

Test Results

Image	VM Size	Result
Canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest	Standard_E32ds_v6	PASSED

[Copilot]

Pull request overview

This PR adds an Azure-only post-run validation step for premium data-disk fio performance tests, comparing measured random-read performance against the VM SKU’s published disk limits so underperforming SKUs fail the test (or are skipped when SKU capabilities aren’t available).

Changes:

• Updated perf_disk() / perf_premium_datadisks() to return the emitted DiskPerformanceMessage list so callers can perform additional validation.
• Added Azure SKU capability lookup and premium data-disk rated-max validation logic in common.py.
• Wired the new validation into the premium data-disk test cases (4K, 1024K, and io_uring variants).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
lisa/microsoft/testsuites/performance/storageperf.py	Captures fio perf messages and invokes the new rated-max validation for premium data-disk test cases.
lisa/microsoft/testsuites/performance/common.py	Returns fio perf messages from helpers and introduces Azure SKU capability lookup + premium data-disk rated-max validation logic/constants.

Suppressed comments (1)

lisa/microsoft/testsuites/performance/common.py:1107

• The validation loop claims to average random-read across iterations per IO depth, but it currently uses the maximum ("best") sample at each depth. This makes the pass/fail decision more permissive than the stated rule and the PR description.

Compute an average per IO depth (or update the doc/logic consistently) so the check matches the documented behavior.

        # The best observed value at this depth (max across iterations).
        best = max(randread_by_iodepth[iodepth])
        percent_of_rated = best / rated_max * 100
        per_depth_percentages.append(percent_of_rated)
        log.info(
            f"premium data-disk check on {node.name} at iodepth {iodepth}: "
            f"best random-read {best:.1f} {rated_unit} = "
            f"{percent_of_rated:.1f}% of rated maximum {rated_max:.1f} "
            f"{rated_unit} (required >= {pass_percent:.0f}%)."
        )
        if best < required:
            failing_iodepths.append(
                f"iodepth {iodepth}: {best:.1f} {rated_unit} "
                f"({percent_of_rated:.1f}%)"
            )

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/microsoft/testsuites/performance/common.py:1103

• The validation says it averages random-read per IO depth across iterations, but the implementation currently uses the max ("best") sample. This makes the check more permissive than described (and than the PR description) and also contradicts the comment directly above the aggregation loop.

    for iodepth in sorted(randread_by_iodepth):
        # The best observed value at this depth (max across iterations).
        best = max(randread_by_iodepth[iodepth])
        percent_of_rated = best / rated_max * 100
        per_depth_percentages.append(percent_of_rated)

lisa/microsoft/testsuites/performance/common.py:1031

• Using a bare assert for test_result.environment is unsafe because Python can remove asserts with optimization flags, and this would turn a missing environment into an unhandled downstream error. Prefer an explicit exception with a clear message so the failure mode is consistent in all runs.

    environment = test_result.environment
    assert environment, "fail to get environment from testresult"
    node = cast(RemoteNode, environment.nodes[0])

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/microsoft/testsuites/performance/common.py:1100

• The function docstring/comment says the per-IO-depth random-read value is averaged across iterations, but the implementation takes max(...) (“best”). If multiple samples per IO depth are ever present, this can incorrectly pass underperforming depths. Either compute the mean (and log it as such) or update the docstring/requirements to explicitly validate against the best value.

        # The best observed value at this depth (max across iterations).
        best = max(randread_by_iodepth[iodepth])
        percent_of_rated = best / rated_max * 100

lisa/microsoft/testsuites/performance/common.py:1029

• Using a bare assert for the environment check is unsafe (asserts can be stripped with -O) and inconsistent with the repo’s general assert_that() usage. Prefer an explicit assert_that(...).is_not_none() check and then cast to Environment before indexing nodes.

    environment = test_result.environment
    assert environment, "fail to get environment from testresult"
    node = cast(RemoteNode, environment.nodes[0])

[LiliDeng]

AI Test Case Selection

Selected 4 test case(s): perf_premium_datadisks_4k,perf_premium_datadisks_1024k,perf_premium_datadisks_4k_io_uring,perf_premium_datadisks_1024k_io_uring

Marketplace image: canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest

Result: Canceled

View full logs in Azure DevOps

[LiliDeng]

AI Test Case Selection

Selected 4 test case(s): perf_premium_datadisks_4k,perf_premium_datadisks_1024k,perf_premium_datadisks_4k_io_uring,perf_premium_datadisks_1024k_io_uring

Marketplace image: canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest

Result: Canceled

View full logs in Azure DevOps

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/microsoft/testsuites/performance/common.py:1136

• The docstring/PR description say the random-read value is averaged per IO depth across iterations, but the implementation uses max(...) (best across iterations). This makes the check less strict than described and the log message is also labeled as "best". Consider using an average to match the stated behavior (or update the docs if "best" is intended).

            measured = randread_iops * block_size_bytes
        else:
            measured = randread_iops
        randread_by_iodepth.setdefault(message.iodepth, []).append(measured)

lisa/microsoft/testsuites/performance/common.py:1056

• Avoid using a bare assert for runtime validation here; assertions can be stripped with Python optimization flags and will turn this into an AttributeError later. Prefer an explicit check that raises LisaException with context.

    The validation can be turned off from the runbook with the shared
    ``enable_perf_check`` variable.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (3)

lisa/microsoft/testsuites/performance/common.py:1152

• The per-IO-depth value is documented as an average across iterations, but the implementation uses the maximum sample (max(...)) and logs it as "best". This can hide under-performance (passes if any single iteration spikes) and contradicts both the docstring and the PR description. Compute the mean for each IO depth and use that for the comparison/logging.

    for iodepth in sorted(randread_by_iodepth):
        # The best observed value at this depth (max across iterations).
        best = max(randread_by_iodepth[iodepth])
        percent_of_rated = best / rated_max * 100
        per_depth_percentages.append(percent_of_rated)

lisa/microsoft/testsuites/performance/common.py:1073

• This uses a bare assert to validate test_result.environment. Python assertions can be stripped with optimization flags and don't provide a great failure message for test runs. Prefer an explicit exception (or an assertpy check) so the failure is always enforced and actionable.

    environment = test_result.environment
    assert environment, "fail to get environment from testresult"
    node = cast(RemoteNode, environment.nodes[0])

lisa/microsoft/testsuites/core/azure_image_standard.py:858

• The assertion message currently interpolates the Python list (repo_urls) directly, which produces a noisy representation (brackets/quotes) in failure output. Joining into a human-readable string makes failures easier to interpret.

                assert_that(
                    is_repository_configured_correctly,
                    f"one of `{repo_urls}`, `security`, "
                    "`updates` should be in `apt-get "
                    "update` output",

[LiliDeng]

AI Test Case Selection

Selected 1 test case(s): verify_repository_installed

Marketplace image: debian debian-12 12-arm64 latest

Result: Succeeded

View full logs in Azure DevOps