First Version of E2E Validation Infrastructure by theelliotm · Pull Request #809 · microsoft/mxc

Elliot (theelliotm) · 2026-08-11T18:28:59Z

📖 Description

This PR introduces an E2E validation testing matrix that uses self-hosted runners to run tests across multiple OS versions. Currently includes support for:

x64

Windows: Prerelease, 25H2, 24H2, 23H2
MacOS: 26, 15
Linux: Ubuntu 26.04, Ubuntu 24.04, Debian 13, RHEL 10

arm64
As of now, there are no Azure VMs that support nested virt on ARM CPUs.

Current Backend Support

Process-T1: ✅ GOOD. Supported on prerelease Windows. Test failures are either a fault of the testing suite or legitimate issues with MXC.
Process-T3: ✅ GOOD. The test suite currently can't be manually run with T1 or T3, so testing T3 on T1 systems doesn't work. Other test failures are either a fault of the testing suite or legitimate issues with MXC.
Sandbox: ⛔ NOT IMPLEMENTED
WSLC: ⚠️✅ (MOSTLY) GOOD. There is a chance of hitting a rate-limit when updating WSL and downloading WSLC images in the test script itself. Other test failures are either a fault of the testing suite or legitimate issues with MXC. Future fix: Split WSLC into multiple jobs spaced 15 minutes apart.
Isolation Session: ⚠️ ISSUE: Feature_AgentSessionsBaseSupport is not enabled on the VM. Should be enabled by default soon!
Bubblewrap: ✅ GOOD.
LXC: ⚠️✅ (MOSTLY) GOOD: Some networking tests fail on versions besides Ubuntu 24. I'm not sure if it's the VM's fault or an issue with MXC.
MicroVM: ⛔ NOT WORKING: (Windows) Both cold and warm starts freeze and timeout. (Linux) No E2E test suite available.
Hyperlight: ⛔ NOT IMPLEMENTED
Seatbelt: ✅ GOOD. Currently no official E2E test suite, but a temporary one I had Copilot write didn't have any issues running or using Seatbelt APIs.

Schedule

The full test matrix runs nightly on main. Once we get bugs fixed and a clearer idea of cost, the schedule will be adjusted.

The testing matrixes and all self-hosted runners are fully configurable in /scripts/ci/validation-test-matrix.json.
The schedule timing is configurable in /.github/workflows/Validation.Tests.Scheduled.yml.

🔗 References

🔍 Validation

No MXC code was modified. All newly created workflows can run and provide feedback on resolving issues.

✅ Checklist

Signed the Contributor License Agreement
Updated documentation (if applicable)
Updated Copilot instructions (if build, architecture, or conventions changed)

📋 Issue Type

Feature

Microsoft Reviewers: Open in CodeFlow

…ames

…com/microsoft/mxc into user/emichlin/validation-infra-lite

Verifies microsoft/mxc can dispatch a job to the 1ES Hosted GitHub Runner pool 1es-mxc-e2e-windows-25h2-pro-x64 (org-level, runner group 145). Pickup-only; nested-virt/WHP checks intentionally omitted. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- Trigger only on push to this throwaway branch (not pull_request) so it doesn't re-run the full mxc CI matrix. - Omit 1ES.ImageOverride (single-image pool defaults) to isolate an image-match failure from an org-level authorization failure. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Multiple recent microsoft-org migrations (vscode, msquic, netperf, ebpf-for-windows) add a unique JobId=<run_id> label to their 1ES runs-on. Some pool configs require it to route/schedule the job; a missing JobId can leave the job queued with no runner. Testing this cheaper hypothesis before escalating the org-level authorization theory. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Root cause of the eternal queue: the pool has tag UseScaleSet=true, so it uses the 1ES Scale Set API, which is targeted by the pool NAME ALONE. The prior webhook-mode labels (self-hosted + 1ES.Pool= + JobId=) never match a scale-set pool, so jobs queued forever. Per 1ES docs, scale-set pools use: runs-on: <PoolName> with no other labels. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Pool tag UseScaleSet=true was removed (reverts pool to webhook mode), so target it with the standard proven format: self-hosted + 1ES.Pool= + JobId=. Benchmark: ebpf 1ES pool picks up in ~2.5-3 min. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

New pool 1es-mxc-e2e-windows-25h2-pro-x64-wh created cleanly via the 1ES Hub wizard: webhook mode (no UseScaleSet tag), SKU Standard_D8s_v7, its own auto-created runner group 146. Tests whether the original pool's eternal queue was due to its scale-set/half-registered state. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

1es-mxc-test-2 is a REPOSITORY-level 1ES pool (organizationProfile level=Repository, repositoryName=mxc) bound directly to microsoft/mxc. Repo-level runners register straight to the repo, bypassing the org runner-group public-repo gate that org-level pools depend on. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ol CONFIRMED working Fresh-eyes RCA: the repo-level pool 1es-mxc-test-2 DID pick up run 30931840357 (runner 21 / bf347e45c000000, group 1/Default) after a ~10.5-min cold start. Prior 'eternal queue' was a too-short (3-4 min) patience window vs ebpf's ~2.5-min WARM-pool benchmark. Only failure was pwsh not found: the raw windows-25h2-pro-x64 client image ships PS 5.1 (powershell.exe), not PS7 (pwsh.exe). Switched both steps to powershell. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ng to powershell 7 commands.

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

scripts/ci/resolve-validation-test-matrix.mjs:66

A missing pool currently passes validation because null/undefined bypasses this condition, then expandPlan silently skips that architecture at line 138. A typo or omitted pool therefore removes scheduled coverage instead of producing the promised early catalog-validation failure. Require the field to be a string; an explicit empty string can still represent a dormant entry.

      if (platform.family === 'macos') {
        assertNonEmptyString(details.runner, `${platform.id}.${architecture}.runner`);
      } else if (details.pool != null && typeof details.pool !== 'string') {
        throw new Error(`${platform.id}.${architecture}.pool must be a string`);

sdk/node/tests/integration/test-helpers.ts:52

These binaries are now mandatory npm package contents, but the local build.bat SDK staging block copies neither one (build.bat:94-120). Consequently, a local full build produces sdk/node/bin/<arch> without these newly required files and the package integration checks fail, while CI passes only because its packaging job downloads the entire Windows artifact. Add both binaries to the local SDK staging path as well, or do not make them package requirements.

  'wxc-ui-probe.exe',
  'wxc-test-driver.exe',

+          & ./scripts/ci/prepare-windows-host.ps1 `
+            -Backend '${{ matrix.backend }}' `
+            -BinaryDirectory (Join-Path $env:GITHUB_WORKSPACE 'artifacts\bin') *>&1 |
+            Tee-Object -FilePath (Join-Path $env:RUNNER_TEMP 'mxc-ci.log') -Append


Branden Bonaby (bbonaby) · 2026-08-13T01:13:55Z

+      ####################
+      # E2E Tests are being moved to the Validation.Tests.Matrix.Job.yml workflow, which runs cross-platform validation
+      ###############


issue: lets remove these since we'll have the PR as paper trail of where they went.

Branden Bonaby (bbonaby) · 2026-08-13T01:14:12Z

        run: cargo test --locked --release --target ${{ matrix.target }}
             -p wxc_e2e_tests

+      ###############


note: same here

Branden Bonaby (bbonaby) · 2026-08-13T01:19:55Z

        run: cargo build --locked --release --target ${{ matrix.target }}
             -p unix_test_proxy

+      # MICROVM NOT WORKING CURRENTLY


issue: By not working what do we mean here? If it's something to do with the tests then I don't think we need to comment out the whole block.

MicroVM currently isn't working correctly on the test machines, so I took it out of the test matrix but left in the code that packages the microvm binaries with MXC. After re-enabling the Cargo Test and SDK jobs, I was getting a lot of errors about "unexpected binaries". Option 1 was comment out everything until MicroVM issues are resolved, and option 2 was add the MicroVM binaries to the Cargo Test and SDK jobs for no actual benefit (and could lead to regressions). I suppose option 3 is just delete this for now 🙂

Branden Bonaby (bbonaby) · 2026-08-13T22:17:43Z

+    if ! sudo "$package_manager" install -y epel-release; then
+        sudo "$package_manager" install -y \
+            https://dl.fedoraproject.org/pub/epel/epel-release-latest-10.noarch.rpm
+    fi


issue: So from a supply chain attack perspective we'd want to download the package from the package manager as our first option and then fallback to downloading directly from fedora.

We'd also want verify the signature of this download (see https://docs.fedoraproject.org/en-US/security/cryptography/signatures/). In this case we should probably add the pgp file to our repo instead of downloading so it can be used during the verification.

Branden Bonaby (bbonaby) · 2026-08-13T22:22:03Z

+        test -x "$binary_directory/lxc-exec"
+        test -f "$binary_directory/unix-test-proxy"
+        mkdir -p "$release_directory"
+        cp -a "$binary_directory/." "$release_directory/"
+        chmod +x "$release_directory/lxc-exec" "$release_directory/unix-test-proxy"
+        bash "$script_root/run_lxc_all_tests.sh"


note: one thing to keep in mind is whether we want artifacts from one run to stay on the runner. I assume we don't and that we'd want artifacts to be removed per run. Let me know if this happens automatically, but if not then we'd want to place them into temporary directories that get blown away after the run.

It happens automatically. Runners use a fresh image for each run.

Branden Bonaby (bbonaby) · 2026-08-13T22:31:53Z

+**Skip semantics.** Several suites degrade gracefully on an unsupported host:
+the IsolationSession suites decide availability from a single `wxc-exec --probe`
+read of `probes.isolationSessionAvailable` (covering both a host that cannot
+activate the API and a binary built without `--features isolation_session`),
+print `SKIPPED`, and exit 0.


question (non-blocking): Anyway for us to surface that some of the tests were skipped? Just want to make sure even if we're all green we still have that extra info.

In my view, the test suite is responsible for how it wants to show pass/fail/skip. Currently Isolation Session gives a green check but sends a "SKIPPED: . . ." message in stdout (viewable in logs). Also, future plans include a more robust way to view testing results internally.

Branden Bonaby (bbonaby) · 2026-08-13T22:33:40Z

+    if command -v systemctl >/dev/null 2>&1; then
+        if systemctl list-unit-files lxc-net.service >/dev/null 2>&1 &&
+            systemctl cat lxc-net.service >/dev/null 2>&1; then
+            sudo systemctl start lxc-net


question: If this fails would this cause the script to abort? Just wondering if we need an error message here.

Won't abort but yes, it's missing a warning.

Copilot

Pull request overview

Copilot reviewed 153 out of 271 changed files in this pull request and generated no new comments.

Suppressed comments (1)

scripts/ci/validation-test-matrix.json:330

The PR description says the full matrix runs nightly and lists macOS 26/15 with Seatbelt support, but this plan contains no macOS trigger. Since the Seatbelt dispatcher also exits 2, the resolver emits no macOS validation jobs at all. Either wire the Seatbelt suite and add these platforms to nightly, or update the PR description to state that macOS entries are declared but dormant.

    "nightly": [

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

Copilot

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

scripts/ci/resolve-validation-test-matrix.mjs:65

A missing pool currently passes validation because only non-null values are type-checked, but expandPlan then silently drops that architecture at line 149. An explicit empty string is the documented dormant state; omitting the field is malformed and likely a catalog typo. Require every non-macOS architecture to declare a string pool (which may still be empty).

      } else if (details.pool != null && typeof details.pool !== 'string') {
        throw new Error(`${platform.id}.${architecture}.pool must be a string`);

.github/workflows/Validation.Tests.Scheduled.yml:6

The PR description says the schedule is fully configurable in validation-test-matrix.json, but the cron expressions and dispatch choices are hardcoded in this workflow (and docs/ci-validation-infrastructure.md:285-290 correctly says schedule changes belong here). Please update the PR description so it does not direct maintainers to the catalog for schedule changes.

  schedule:
    - cron: '0 8 * * 1-6'
    - cron: '0 8 * * 0'

Copilot

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (3)

sdk/node/tests/integration/test-helpers.ts:52

These new required package binaries are only added to the GitHub artifact manifest. The Azure build still copies Windows files exclusively through .azure-pipelines/templates/Rust.Build.Job.yml's signPattern, which omits both executables, while its SDK integration job runs this same package test on Windows. Official Azure builds will therefore fail with both files missing. Add them to the Azure signing/copy manifest as well, or keep validation-only binaries out of the SDK package expectations consistently.

  'wxc-ui-probe.exe',
  'wxc-test-driver.exe',

scripts/ci/resolve-validation-test-matrix.mjs:25

The new catalog validator and expansion logic has no automated tests, despite controlling every scheduled combination and containing important branches for malformed catalogs, dormant pools, architecture suppression, and empty family matrices. Similar JavaScript infrastructure under scripts/versioning/tests/ is unit-tested. Add Node tests for validateCatalog, expandPlan, and resolvePlan, and run them in CI so matrix regressions are caught before a scheduled workflow consumes them.

export function validateCatalog(catalog) {

.github/workflows/Validation.Tests.Scheduled.yml:6

The PR description says the schedule is fully configurable in validation-test-matrix.json, but the actual timing is hard-coded here (and the repository documentation correctly says schedule changes belong in this workflow). Please update the PR description to distinguish the cron schedule from the catalog-controlled plans, pools, and backend combinations.

    - cron: '0 8 * * 1-6'
    - cron: '0 8 * * 0'

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

sdk/node/tests/integration/test-helpers.ts:52

These new required package binaries are only added to the GitHub artifact manifest. The official Azure build still copies its Windows artifact from signPattern in .azure-pipelines/templates/Rust.Build.Job.yml:49-58,185-191, which omits both files, while Package.NpmSdk.Job.yml:51-57 packages that whole artifact. GitHub and official Azure npm tarballs will therefore have different payloads. Add both executables to the Azure build/signing manifest as part of this change (or keep them out of the SDK expectation).

  'wxc-ui-probe.exe',
  'wxc-test-driver.exe',

tests/scripts/run_ci_backend_tests.sh:51

MXC_LXC_TESTS_REQUIRE_EXECUTION is not read anywhere by run_lxc_all_tests.sh (the only repository occurrence is this assignment). That suite counts exit-77 cases as skips, and even its all-skipped path only warns before exiting 0 (run_lxc_all_tests.sh:73-85), so an LXC matrix job can still be green without the required coverage. Implement the flag in the aggregate suite (and fail on the intended skip condition), or have this dispatcher inspect the summary and return nonzero.

        MXC_LXC_TESTS_REQUIRE_EXECUTION=1 bash "$script_root/run_lxc_all_tests.sh"

…epel key verification.

…com/microsoft/mxc into user/emichlin/validation-infra-lite

…epel key verification

Copilot

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

scripts/ci/resolve-validation-test-matrix.mjs:65

A missing pool is accepted here even though it is required for Windows/Linux. expandPlan then treats the missing value like a dormant empty pool and silently emits no job, so a typo in this field can make requested validation disappear while the resolver succeeds. Require the field to be a string; keep "" as the explicit dormant value.

      if (platform.family === 'macos') {
        assertNonEmptyString(details.runner, `${platform.id}.${architecture}.runner`);
      } else if (details.pool != null && typeof details.pool !== 'string') {
        throw new Error(`${platform.id}.${architecture}.pool must be a string`);
      }

scripts/ci/prepare-linux-host.sh:177

With set -euo pipefail, this unguarded assignment exits host preparation when ip fails (notably after start_lxc_bridge deliberately warns and returns because the bridge is absent). That contradicts this function's diagnostic-only contract and prevents the backend suite from reporting the actual failure. Make the probe tolerate an absent bridge.

    local bridge_ip
    bridge_ip="$(ip -4 -o addr show "$bridge" 2>/dev/null |
        awk '{print $4}' | cut -d/ -f1 | head -n 1)"

.github/workflows/Build.Windows.Job.yml:119

These test-only executables are now part of the GitHub product artifact, and Package.NpmSdk.Job.yml:22-49 copies that whole artifact into sdk/node/bin, so GitHub-built npm packages ship internal harnesses that consumers do not need. The Azure artifact excludes them via .azure-pipelines/templates/Rust.Build.Job.yml:49-58, making package contents producer-dependent. Publish a separate validation artifact or explicitly strip these files before npm pack.

            src/target/${{ matrix.target }}/release/wxc-ui-probe.exe
            src/target/${{ matrix.target }}/release/wxc-test-driver.exe

scripts/ci/resolve-validation-test-matrix.mjs:25

This scheduling-critical resolver has no automated tests; resolving the checked-in nightly catalog only exercises one happy path. Add Node tests for malformed catalogs, empty versus missing pools, ARM64 suppression, duplicate requests, and per-family output, following the existing scripts/versioning/tests pattern.

export function validateCatalog(catalog) {

Elliot (theelliotm) and others added 30 commits July 30, 2026 17:16

lite implementation of validation infra with placeholders for image n…

332bc6e

…ames

aliging with similar projects over naming of prerelease

03fc97e

changing what's enabled for testing

cc38e54

test run with one windows and one linux runner

a8040c1

removed tests from build workflows

04a6cf8

lite implementation of validation infra with placeholders for image n…

1b88616

…ames

aliging with similar projects over naming of prerelease

94cf6d7

changing what's enabled for testing

01996fe

test run with one windows and one linux runner

5de032e

removed tests from build workflows

39c63f8

Merge branch 'user/emichlin/validation-infra-lite' of https://github.…

723caa2

…com/microsoft/mxc into user/emichlin/validation-infra-lite

Fix naming + re-running failed action

c8db517

migrating APIs

c52e17d

changing format of runs-on to match similar projects

b004f5e

ci: retrigger 1ES smoke after pool re-save (webhook mode)

2750006

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

updated powershell to use 5.1

af27bed

adding bubblewrap install for linux test prereq. for windows, reverti…

1903465

…ng to powershell 7 commands.

triggering linux test scripts via bash

0954172

bubblewrap test now owns files it delegates

2c05316

diagnosing interactive desktop issues

d932a42

diagnostics should continue on error

6af23d8

Removing diagnostics - intended behavior.

8a22379

Copilot AI review requested due to automatic review settings August 12, 2026 21:18

microsoft-github-policy-service Bot added the Copilot-Instructions PR modifies Copilot instruction files (.github/copilot-instructions.md or .github/instructions/) label Aug 12, 2026

Copilot started reviewing on behalf of Elliot (theelliotm) August 12, 2026 21:19 View session

Copilot AI reviewed Aug 12, 2026

View reviewed changes

Branden Bonaby (bbonaby) reviewed Aug 13, 2026

View reviewed changes

Addressed PR feedback, removed handler-backend distinction

f18c791

Copilot AI review requested due to automatic review settings August 14, 2026 22:15

Copilot started reviewing on behalf of Elliot (theelliotm) August 14, 2026 22:15 View session

Elliot (theelliotm) requested a review from a team August 14, 2026 22:21

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 14, 2026 22:21

Copilot AI reviewed Aug 14, 2026

Elliot (theelliotm) force-pushed the user/emichlin/validation-infra-lite branch from f2d996c to f18c791 Compare August 14, 2026 22:29

Copilot AI review requested due to automatic review settings August 14, 2026 22:34

Copilot started reviewing on behalf of Elliot (theelliotm) August 14, 2026 22:34 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Comment thread tests/scripts/run_ci_backend_tests.sh Outdated

Comment thread sdk/node/tests/integration/test-helpers.ts Outdated

Copilot AI review requested due to automatic review settings August 14, 2026 22:45

Elliot (theelliotm) force-pushed the user/emichlin/validation-infra-lite branch from 7438d24 to f18c791 Compare August 14, 2026 22:45

Copilot started reviewing on behalf of Elliot (theelliotm) August 14, 2026 22:45 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Don't greenlight prereq skips

f5c161b

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings August 14, 2026 22:51

Copilot started reviewing on behalf of Elliot (theelliotm) August 14, 2026 22:52 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Comment thread scripts/ci/prepare-linux-host.sh Outdated

Elliot (theelliotm) added 3 commits August 14, 2026 16:31

moved wxc-ui-probe and wxc-test-driver to optional binaries. removed …

8752bed

…epel key verification.

Merge branch 'user/emichlin/validation-infra-lite' of https://github.…

63193e0

…com/microsoft/mxc into user/emichlin/validation-infra-lite

moved wxc-ui probe and wxc-test-driver to optional binaries. removed …

d53b85b

…epel key verification

Copilot AI review requested due to automatic review settings August 14, 2026 23:32

Copilot started reviewing on behalf of Elliot (theelliotm) August 14, 2026 23:33 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Comment thread tests/scripts/run_ci_backend_tests.sh

Uh oh!

Conversation

Elliot (theelliotm) commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📖 Description

Current Backend Support

Schedule

🔗 References

🔍 Validation

✅ Checklist

📋 Issue Type

Microsoft Reviewers: Open in CodeFlow

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Elliot (theelliotm) commented Aug 11, 2026 •

edited

Loading