diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 8925b4a3c..575f4e7f6 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -43,6 +43,60 @@ build.bat --with-microvm # Include NanVix micro-VM binaries Requires Xcode Command Line Tools and Rust. Produces an unsigned `mxc-exec-mac` binary (codesigning + notarization happen at release time). Schema `0.7.0-alpha` or later required for macOS/Seatbelt backend. +### GitHub Actions + +`.github/workflows/Build.yml` is the PR/CI entry point. It fans out to the +workflow-call-only `Build.Windows.Job.yml`, `Build.Linux.Job.yml`, and +`Build.MacOS.Job.yml`, which build and upload the per-target artifacts in +parallel, then to the lint / versioning / SDK jobs. + +**Validation (E2E) test infrastructure.** Fully documented in +[`docs/ci-validation-infrastructure.md`](../docs/ci-validation-infrastructure.md) +(matrix contents, job names, per-backend coverage and status, and the runbook +for adding/removing an OS, backend, or plan). Backend E2E tests run from those +same build artifacts — never from a fresh build — so artifact production and +consumption stay in one workflow run: + +- `.github/workflows/Validation.Tests.Scheduled.yml` — scheduled entry point. + The `nightly` plan runs Mon–Sat; Sunday runs `nightly` *and* `weekly`. + `workflow_dispatch` takes a `plan` input to run one on demand. +- `.github/workflows/Validation.Tests.Matrix.Job.yml` — workflow-call-only, + takes a `plan` input. Its `resolve` job expands the plan into per-family + matrices, then the `windows` / `linux` / `macos` jobs each download the + artifact, prepare the host, and run the backend suite. + +An entry point must build the artifacts (call the three `Build.*.Job.yml` +workflows) before calling the matrix job. + +**The matrix is declarative:** + +- `scripts/ci/validation-test-matrix.json` is the catalog: `platforms` (each + with per-architecture target/artifact/1ES pool and the backends that platform + supports) and `triggers` (which OS/backend pairs each plan runs). The + `triggers` keys *are* the plan list — the resolver reads them at run time, so + adding a plan needs no script change. +- `scripts/ci/resolve-validation-test-matrix.mjs` validates that catalog and + expands a plan (currently `pr`, `nightly`, `weekly`, `enabled`) into GitHub + Actions matrices. It rejects an invalid catalog before any specialized test + runner is allocated, so add a backend to a trigger only where the platform + declares it. +- A non-macOS platform architecture with an empty `pool` is never scheduled, + which is how a catalog entry stays declared but dormant. macOS entries use a + GitHub-hosted `runner` instead of a 1ES `pool`. + +**Host preparation** happens in the matrix job before the tests, keyed by the +matrix `backend` id: `scripts/ci/prepare-windows-host.ps1` and +`scripts/ci/prepare-linux-host.sh`. A backend with no prerequisites is an +explicit no-op, so the step runs unconditionally for every entry. + +**Test dispatch** goes through `tests/scripts/run_ci_backend_tests.ps1` +(Windows) and `tests/scripts/run_ci_backend_tests.sh` (Linux/macOS), which map +the matrix `backend` id to the repository's existing backend suite. Ids that +share a suite get their own case (`process-t1` and `process-t3` both run +`WinProcessContainer-Tests.ps1`, which derives the tier it expects from the +host's own `--probe`). A backend with no wired suite fails loudly rather than +reporting a false success. + ### Individual components ``` @@ -98,6 +152,18 @@ tests\scripts\run_bwrap_all_tests.sh # All Bubblewrap tests (Linux, req # E2E test crate — Rust executor integration tests (from src/) cargo test -p wxc_e2e_tests # Invokes MXC binaries directly cargo test -p wxc_e2e_tests -- --ignored # Include stress tests (run_on_repeat) + +# WSLC has no cargo E2E suite — it is covered by tests\scripts\run_wslc_all_tests.ps1, +# which the validation matrix runs via tests\scripts\run_ci_backend_tests.ps1. + +# CI validation entry points — run a backend suite against a downloaded artifact +# the way the validation matrix does. Take the matrix backend id exactly as it +# appears in scripts/ci/validation-test-matrix.json. +tests\scripts\run_ci_backend_tests.ps1 -Backend process-t1 -BinaryDirectory -Architecture x64 +tests\scripts\run_ci_backend_tests.sh + +# Resolve a plan locally to see exactly what CI would schedule +node scripts/ci/resolve-validation-test-matrix.mjs --plan nightly ``` ## Architecture @@ -154,6 +220,7 @@ Core references: - `docs/authoring-a-new-feature.md` — step-by-step guide for adding experimental features (which files to touch, in what order) - `docs/examples.md` — annotated configuration examples (see also `tests/examples/` and `tests/configs/`) - `docs/diagnostics.md` — diagnostic logging knobs (env vars, log file format) +- `docs/ci-validation-infrastructure.md` — validation (E2E) test matrix: workflows and job names, catalog format, per-backend coverage and status, and the runbook for adding/removing an OS, backend, or plan - `docs/host-prep.md` — `wxc-host-prep.exe` host setup binary (`prepare-system-drive` / `unprepare-system-drive` for the AppContainer ACEs on the system-drive root, plus `prepare-null-device` / `verify-null-device` / `dump-null-device` for the `\Device\Null` security descriptor that AppContainer-based backends require). Owns elevation via embedded `requireAdministrator` manifest — `wxc-exec.exe` no longer self-elevates. - `docs/sandbox-policy/0.7.0/policy.md` — sandbox policy 0.7.0 specification diff --git a/.github/workflows/Build.Linux.Job.yml b/.github/workflows/Build.Linux.Job.yml index c6fa9b999..66c804c46 100644 --- a/.github/workflows/Build.Linux.Job.yml +++ b/.github/workflows/Build.Linux.Job.yml @@ -13,9 +13,11 @@ jobs: - arch: x64 runner: ubuntu-latest target: x86_64-unknown-linux-gnu + features: hyperlight - arch: arm64 runner: ubuntu-24.04-arm target: aarch64-unknown-linux-gnu + features: '' runs-on: ${{ matrix.runner }} defaults: run: @@ -54,12 +56,26 @@ jobs: run: sudo apt-get update && sudo apt-get install -y --no-install-recommends gcc-aarch64-linux-gnu - name: Build lxc - run: cargo build --locked --release --target ${{ matrix.target }} - --no-default-features --features hyperlight + shell: bash + run: | + set -euo pipefail + features=() + if [[ -n "${{ matrix.features }}" ]]; then + features=(--features "${{ matrix.features }}") + fi + cargo build --locked --release --target "${{ matrix.target }}" \ + --no-default-features "${features[@]}" - name: Test lxc - run: cargo test --locked --release --target ${{ matrix.target }} - --no-default-features --features hyperlight + shell: bash + run: | + set -euo pipefail + features=() + if [[ -n "${{ matrix.features }}" ]]; then + features=(--features "${{ matrix.features }}") + fi + cargo test --locked --release --target "${{ matrix.target }}" \ + --no-default-features "${features[@]}" # Bubblewrap is required to run the executor characterization tests in # wxc_e2e_tests (they skip via has_bwrap() when it is absent). lxc-exec @@ -81,7 +97,7 @@ jobs: working-directory: src run: cargo test --locked --release --target ${{ matrix.target }} -p wxc_e2e_tests - + # PLM (Permissive Learning Mode) is functionally Windows-only, but the # crate builds cross-platform: the lib's helper modules compile on every # target, and the binary has a no-op stub `fn main()` for non-Windows so diff --git a/.github/workflows/Build.MacOS.Job.yml b/.github/workflows/Build.MacOS.Job.yml index cf0ab2aab..ce97d3596 100644 --- a/.github/workflows/Build.MacOS.Job.yml +++ b/.github/workflows/Build.MacOS.Job.yml @@ -6,8 +6,7 @@ on: jobs: build: name: arm64 - # macos-14 / macos-latest are Apple Silicon (arm64); older labels are Intel. - runs-on: macos-14 + runs-on: macos-15 defaults: run: working-directory: src @@ -55,6 +54,18 @@ jobs: run: cargo test --locked --release --target aarch64-apple-darwin -p mxc_darwin -p seatbelt_common -p wxc_common -p wxc_e2e_tests + - name: Verify artifact payload + shell: bash + run: | + set -euo pipefail + bin_dir="target/aarch64-apple-darwin/release" + for file in mxc-exec-mac unix-test-proxy; do + if [[ ! -f "$bin_dir/$file" ]]; then + echo "Missing artifact file: $file" >&2 + exit 1 + fi + done + - name: Upload binaries uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: diff --git a/.github/workflows/Build.Windows.Job.yml b/.github/workflows/Build.Windows.Job.yml index d859e3b5a..03b9579a6 100644 --- a/.github/workflows/Build.Windows.Job.yml +++ b/.github/workflows/Build.Windows.Job.yml @@ -18,8 +18,8 @@ jobs: - arch: arm64 runner: windows-11-arm target: aarch64-pc-windows-msvc - # nanvixd.exe is x64-only, so microvm tests can't run on arm64. - features: hyperlight isolation_session wslc + # Hyperlight and MicroVM runtimes are x64-only. + features: isolation_session wslc runs-on: ${{ matrix.runner }} defaults: run: @@ -86,5 +86,7 @@ jobs: src/target/${{ matrix.target }}/release/wxc-test-proxy.exe src/target/${{ matrix.target }}/release/mxc-diagnostic-console.exe src/target/${{ matrix.target }}/release/wslcsdk.dll - if-no-files-found: warn + src/target/${{ matrix.target }}/release/wxc-ui-probe.exe + src/target/${{ matrix.target }}/release/wxc-test-driver.exe + if-no-files-found: error retention-days: 1 diff --git a/.github/workflows/Build.yml b/.github/workflows/Build.yml index fbcd36f0a..6b3a8cc52 100644 --- a/.github/workflows/Build.yml +++ b/.github/workflows/Build.yml @@ -18,6 +18,7 @@ concurrency: cancel-in-progress: true permissions: + actions: read contents: read jobs: diff --git a/.github/workflows/Lint.Job.yml b/.github/workflows/Lint.Job.yml index 251bacf13..6ca9abc63 100644 --- a/.github/workflows/Lint.Job.yml +++ b/.github/workflows/Lint.Job.yml @@ -24,7 +24,7 @@ jobs: clippy-args: --locked --all-targets --all-features --release - os_label: macos component: MAC - runner: macos-14 + runner: macos-26 target: aarch64-apple-darwin working-directory: src # Workspace has Windows-only crates, so --all-features won't build on macOS. diff --git a/.github/workflows/SDK.Integration.Test.Job.yml b/.github/workflows/SDK.Integration.Test.Job.yml index 7d3a4b513..fc8495e37 100644 --- a/.github/workflows/SDK.Integration.Test.Job.yml +++ b/.github/workflows/SDK.Integration.Test.Job.yml @@ -15,7 +15,7 @@ jobs: - os_label: linux runner: ubuntu-latest - os_label: macos - runner: macos-14 + runner: macos-26 runs-on: ${{ matrix.runner }} defaults: run: diff --git a/.github/workflows/Validation.Tests.Matrix.Job.yml b/.github/workflows/Validation.Tests.Matrix.Job.yml new file mode 100644 index 000000000..9a958d94a --- /dev/null +++ b/.github/workflows/Validation.Tests.Matrix.Job.yml @@ -0,0 +1,173 @@ +name: Create Validation Test Matrix + +on: + workflow_call: + inputs: + plan: + description: Test plan to resolve. + required: true + type: string + +permissions: + actions: read + contents: read + +jobs: + resolve: + name: resolve "${{ inputs.plan }}" test matrix + runs-on: ubuntu-latest + outputs: + windows: ${{ steps.matrix.outputs.windows }} + linux: ${{ steps.matrix.outputs.linux }} + macos: ${{ steps.matrix.outputs.macos }} + has_windows: ${{ steps.matrix.outputs.has_windows }} + has_linux: ${{ steps.matrix.outputs.has_linux }} + has_macos: ${{ steps.matrix.outputs.has_macos }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 + + - name: Resolve test combinations + id: matrix + run: node scripts/ci/resolve-validation-test-matrix.mjs --plan "${{ inputs.plan }}" + + windows: + name: ${{ matrix.os }}, ${{ matrix.architecture }}, ${{ matrix.backend }} + needs: resolve + if: needs.resolve.outputs.has_windows == 'true' + strategy: + fail-fast: false + matrix: ${{ fromJSON(needs.resolve.outputs.windows) }} + runs-on: [ self-hosted, "1ES.Pool=${{ matrix.pool }}", "JobId=mxc-e2e-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}" ] + timeout-minutes: 60 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 + + - name: Download ${{ matrix.target }} artifact + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c + with: + name: ${{ matrix.artifact }} + path: artifacts/bin + + - name: Prepare backend prerequisites + timeout-minutes: 15 + shell: pwsh + run: | + & ./scripts/ci/prepare-windows-host.ps1 ` + -Backend '${{ matrix.backend }}' ` + -BinaryDirectory (Join-Path $env:GITHUB_WORKSPACE 'artifacts\bin') *>&1 | + Tee-Object -FilePath (Join-Path $env:RUNNER_TEMP 'mxc-ci.log') -Append + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + - name: Run backend tests + # A backend that hangs (rather than failing) would otherwise burn the + # full job timeout. Fail fast enough to keep the log useful. + timeout-minutes: 45 + shell: pwsh + run: | + & ./tests/scripts/run_ci_backend_tests.ps1 ` + -Backend '${{ matrix.backend }}' ` + -BinaryDirectory (Join-Path $env:GITHUB_WORKSPACE 'artifacts\bin') ` + -Architecture '${{ matrix.architecture }}' *>&1 | + Tee-Object -FilePath (Join-Path $env:RUNNER_TEMP 'mxc-ci.log') -Append + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + - name: Upload failure logs + if: failure() || cancelled() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a + with: + name: logs-${{ inputs.plan }}-${{ matrix.os }}-${{ matrix.architecture }}-${{ matrix.backend }}-${{ github.run_attempt }} + path: | + ${{ runner.temp }}/mxc-ci.log + ${{ runner.temp }}/mxc-wpc-tests/logs + ${{ runner.temp }}/WinProcessContainer-Tests.results.* + ${{ runner.temp }}/mxc_concurrent_oneshot + if-no-files-found: ignore + retention-days: 7 + + linux: + name: ${{ matrix.os }}, ${{ matrix.architecture }}, ${{ matrix.backend }} + needs: resolve + if: needs.resolve.outputs.has_linux == 'true' + strategy: + fail-fast: false + matrix: ${{ fromJSON(needs.resolve.outputs.linux) }} + runs-on: [ self-hosted, "1ES.Pool=${{ matrix.pool }}", "JobId=mxc-e2e-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}" ] + timeout-minutes: 60 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 + + - name: Download ${{ matrix.target }} artifact + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c + with: + name: ${{ matrix.artifact }} + path: artifacts/bin + + - name: Prepare backend prerequisites + timeout-minutes: 15 + shell: bash + run: | + set -euo pipefail + bash scripts/ci/prepare-linux-host.sh \ + '${{ matrix.backend }}' "$GITHUB_WORKSPACE/artifacts/bin" 2>&1 | + tee -a "$RUNNER_TEMP/mxc-ci.log" + + - name: Run backend tests + timeout-minutes: 45 + shell: bash + run: | + set -euo pipefail + if [[ '${{ matrix.backend }}' == 'lxc' ]]; then + sudo --preserve-env=RUNNER_TEMP bash tests/scripts/run_ci_backend_tests.sh \ + '${{ matrix.backend }}' "$GITHUB_WORKSPACE/artifacts/bin" 2>&1 | + tee -a "$RUNNER_TEMP/mxc-ci.log" + else + bash tests/scripts/run_ci_backend_tests.sh \ + '${{ matrix.backend }}' "$GITHUB_WORKSPACE/artifacts/bin" 2>&1 | + tee -a "$RUNNER_TEMP/mxc-ci.log" + fi + + - name: Upload failure logs + if: failure() || cancelled() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a + with: + name: logs-${{ inputs.plan }}-${{ matrix.os }}-${{ matrix.architecture }}-${{ matrix.backend }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/mxc-ci.log + if-no-files-found: ignore + retention-days: 7 + + macos: + name: ${{ matrix.os }}, ${{ matrix.backend }} + needs: resolve + if: needs.resolve.outputs.has_macos == 'true' + strategy: + fail-fast: false + matrix: ${{ fromJSON(needs.resolve.outputs.macos) }} + runs-on: ${{ matrix.runner }} + timeout-minutes: 60 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 + + - name: Download ${{ matrix.target }} artifact + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c + with: + name: ${{ matrix.artifact }} + path: artifacts/bin + + - name: Run backend tests + timeout-minutes: 60 + shell: bash + run: | + set -euo pipefail + chmod +x artifacts/bin/mxc-exec-mac artifacts/bin/unix-test-proxy + bash tests/scripts/run_ci_backend_tests.sh \ + '${{ matrix.backend }}' "$GITHUB_WORKSPACE/artifacts/bin" 2>&1 | + tee "$RUNNER_TEMP/mxc-ci.log" + + - name: Upload failure logs + if: failure() || cancelled() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a + with: + name: logs-${{ inputs.plan }}-${{ matrix.os }}-${{ matrix.architecture }}-${{ matrix.backend }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/mxc-ci.log + if-no-files-found: ignore + retention-days: 7 diff --git a/.github/workflows/Validation.Tests.Scheduled.yml b/.github/workflows/Validation.Tests.Scheduled.yml new file mode 100644 index 000000000..9bc3bf9ad --- /dev/null +++ b/.github/workflows/Validation.Tests.Scheduled.yml @@ -0,0 +1,54 @@ +name: Scheduled Validation Tests + +on: + schedule: + - cron: '0 8 * * 1-6' + - cron: '0 8 * * 0' + workflow_dispatch: + inputs: + plan: + description: Test plan to run. + required: true + default: nightly + type: choice + options: + - nightly + - weekly + +concurrency: + group: test-matrix-e2e-${{ github.ref }}-${{ github.event.schedule || inputs.plan }} + cancel-in-progress: false + +permissions: + actions: read + contents: read + +jobs: + dependency-feed-check: + uses: ./.github/workflows/Dependency.Feed.Check.Job.yml + + windows: + needs: dependency-feed-check + uses: ./.github/workflows/Build.Windows.Job.yml + + linux: + needs: dependency-feed-check + uses: ./.github/workflows/Build.Linux.Job.yml + + macos: + needs: dependency-feed-check + uses: ./.github/workflows/Build.MacOS.Job.yml + + test-nightly: + needs: [windows, linux, macos] + if: github.event_name == 'schedule' || inputs.plan == 'nightly' + uses: ./.github/workflows/Validation.Tests.Matrix.Job.yml + with: + plan: nightly + + test-weekly: + needs: [windows, linux, macos] + if: (github.event_name == 'schedule' && github.event.schedule == '0 8 * * 0') || inputs.plan == 'weekly' + uses: ./.github/workflows/Validation.Tests.Matrix.Job.yml + with: + plan: weekly \ No newline at end of file diff --git a/.github/workflows/hyperlight-e2e.yml b/.github/workflows/hyperlight-e2e.yml index b78d62cc8..7396fba99 100644 --- a/.github/workflows/hyperlight-e2e.yml +++ b/.github/workflows/hyperlight-e2e.yml @@ -1,5 +1,7 @@ name: Hyperlight E2E Tests +# Retained until the unified test matrix has equivalent Hyperlight coverage. + on: push: branches: [main] diff --git a/.github/workflows/microvm-e2e.yml b/.github/workflows/microvm-e2e.yml index aaf0cbc95..3f602116c 100644 --- a/.github/workflows/microvm-e2e.yml +++ b/.github/workflows/microvm-e2e.yml @@ -1,5 +1,7 @@ name: Integration Tests +# Retained until the unified test matrix has equivalent MicroVM coverage. + on: push: branches: [main] diff --git a/docs/ci-validation-infrastructure.md b/docs/ci-validation-infrastructure.md new file mode 100644 index 000000000..ec0e99486 --- /dev/null +++ b/docs/ci-validation-infrastructure.md @@ -0,0 +1,366 @@ +# Validation (E2E) test infrastructure + +How MXC runs its backend end-to-end suites across real operating systems, what +each job covers today, and what to change when you need to add, remove, or +retire something. + +This document describes the GitHub Actions validation matrix only. PR-time +build/lint/SDK validation is covered by [`pull-requests.md`](pull-requests.md); +the individual local test scripts are documented in +[`tests/scripts/README.md`](../tests/scripts/README.md). + +## At a glance + +- Validation tests **never build from source**. They download the artifacts + produced by `Build.Windows.Job.yml` / `Build.Linux.Job.yml` / + `Build.MacOS.Job.yml` in the same workflow run, so what gets tested is exactly + what got built. +- The matrix is **declarative**. `scripts/ci/validation-test-matrix.json` is the + only file you edit to change *what runs where*; + `scripts/ci/resolve-validation-test-matrix.mjs` validates it and expands a + plan into GitHub Actions matrices. +- Validation runs **on a schedule, not on PRs**. + +## Moving parts + +| File | Role | +|------|------| +| `.github/workflows/Validation.Tests.Scheduled.yml` | Scheduled entry point. Builds artifacts, then calls the matrix job. | +| `.github/workflows/Validation.Tests.Matrix.Job.yml` | `workflow_call`-only. Resolves the plan and runs the per-family test jobs. | +| `scripts/ci/validation-test-matrix.json` | The matrix: OS versions, backends, triggers. | +| `scripts/ci/resolve-validation-test-matrix.mjs` | Matrix validator + plan expander. Emits the GitHub Actions matrices. | +| `scripts/ci/prepare-windows-host.ps1` | Per-backend Windows host preparation / prerequisite assertions. | +| `scripts/ci/prepare-linux-host.sh` | Per-backend Linux package install and service startup (distro-aware). | +| `tests/scripts/run_ci_backend_tests.ps1` | Windows dispatcher: backend id → existing backend suite. | +| `tests/scripts/run_ci_backend_tests.sh` | Linux/macOS dispatcher: backend id → existing backend suite. | + +### Flow + +``` +Validation.Tests.Scheduled.yml + └─ dependency-feed-check + ├─ windows / linux / macos → Build.*.Job.yml (upload artifacts) + └─ test-nightly / test-weekly → Validation.Tests.Matrix.Job.yml + └─ resolve → resolve-validation-test-matrix.mjs --plan + ├─ windows job (matrix) → download artifact → prepare-windows-host.ps1 → run_ci_backend_tests.ps1 + ├─ linux job (matrix) → download artifact → prepare-linux-host.sh → run_ci_backend_tests.sh + └─ macos job (matrix) → download artifact → run_ci_backend_tests.sh +``` + +An entry point **must** build the artifacts before calling the matrix job — the +test jobs only ever `download-artifact`. + +## Jobs + +### `Validation.Tests.Scheduled.yml` — "Scheduled Validation Tests" + +| Job | What it does | +|-----|--------------| +| `dependency-feed-check` | Resolves the locked crate graph through the public `MxcDependencies` feed. Gates the builds. | +| `windows` | `Build.Windows.Job.yml` — x64 + arm64 release build, unit tests, uploads `wxc-binaries-`. | +| `linux` | `Build.Linux.Job.yml` — x64 + arm64 release build, unit tests, `wxc_e2e_tests`, uploads `lxc-binaries-`. | +| `macos` | `Build.MacOS.Job.yml` — arm64 release build, unit + `wxc_e2e_tests`, uploads `mxc-binaries-aarch64-apple-darwin`. | +| `test-nightly` | Calls the matrix job with `plan: nightly`. Runs on every schedule tick and on a `nightly` dispatch. | +| `test-weekly` | Calls the matrix job with `plan: weekly`. Runs only on the Sunday cron and on a `weekly` dispatch. | + +Build artifacts are kept for 1 day — they exist only to feed these jobs. + +### `Validation.Tests.Matrix.Job.yml` — "Create Validation Test Matrix" + +| Job | Runner | What it does | +|-----|--------|--------------| +| `resolve` | `ubuntu-latest` | Runs the resolver, emits one matrix per OS family plus `has_` flags so an empty family is skipped rather than failing on an empty matrix. | +| `windows` | `[self-hosted, 1ES.Pool=, JobId=mxc-e2e-…]` | Download artifact → `prepare-windows-host.ps1 -Backend ` → `run_ci_backend_tests.ps1 -Backend `. | +| `linux` | `[self-hosted, 1ES.Pool=, JobId=mxc-e2e-…]` | Download artifact → `prepare-linux-host.sh ` → `run_ci_backend_tests.sh ` (under `sudo` for LXC). | +| `macos` | GitHub-hosted `${{ matrix.runner }}` | Download artifact → `chmod +x` → `run_ci_backend_tests.sh `. No host-prep step. | + +Per-job display name: `, , ` (macOS omits +the architecture). Job timeout 60 min; host prep 15 min; the test step 45 min +(60 on macOS), so a hung backend fails while the log is still useful. On failure +or cancellation the job uploads `mxc-ci.log` plus the Process Container log +directories as `logs-----`, kept 7 days. + +## The catalog + +`scripts/ci/validation-test-matrix.json` has two sections. + +### `platforms` + +Declares an OS image and, per architecture, the build it consumes, the host pool +it runs on, and **which backends that platform is capable of running**. This is +a capability declaration, not a schedule. + +| Field | Meaning | +|-------|---------| +| `id` | Stable key referenced by `triggers`. Also the value shown in job names. | +| `displayName` | Human label (emitted as `os_name`). | +| `family` | `windows` \| `linux` \| `macos` — selects the matrix job and the dispatcher. | +| `prerelease` | `true` marks an unreleased Windows image. Its `id` must be a neutral alias matching `windows-prerelease-`, because the id is public in job names. | +| `architectures..target` | Rust target triple. | +| `architectures.<…>.artifact` | Build artifact name to download. | +| `architectures.<…>.pool` | 1ES pool name (Windows/Linux). **An empty string means "declared but never scheduled"** — the entry stays documented but dormant. | +| `architectures.<…>.runner` | GitHub-hosted runner label (macOS only; required there). | +| `architectures.<…>.backends` | Backend ids this platform/arch can run. | + +Current platforms: + +| Platform id | Family | x64 pool | arm64 pool | Declared backends (x64) | +|-------------|--------|----------|------------|--------------------------| +| `windows-prerelease-process-container` | windows | `1es-mxc-windows-prerelease-t1-x64` | *(dormant)* | process-t1, process-t3, isolation-session, wslc, windows-sandbox, microvm, hyperlight | +| `windows-prerelease-isolation-session` | windows | *(dormant)* | *(dormant)* | same as above | +| `windows-canary` | windows | *(dormant)* | *(dormant)* | same as above | +| `windows-25h2` | windows | `1es-mxc-e2e-windows-25h2-pro-x64` | *(dormant)* | process-t3, wslc, windows-sandbox, microvm, hyperlight | +| `windows-24h2` | windows | `1es-mxc-e2e-windows-24h2-pro-x64` | *(dormant)* | process-t3, wslc, windows-sandbox, microvm, hyperlight | +| `windows-23h2` | windows | `1es-mxc-e2e-windows-23h2-enterprise-x64` | *(dormant)* | process-t3, wslc, windows-sandbox, microvm, hyperlight | +| `ubuntu-26.04` | linux | `1es-mxc-e2e-ubuntu-26.04-x64` | *(dormant)* | bubblewrap, hyperlight, lxc | +| `ubuntu-24.04` | linux | `1es-mxc-e2e-ubuntu-24.04-x64` | *(dormant)* | bubblewrap, microvm, hyperlight, lxc | +| `rhel-10` | linux | `1es-mxc-e2e-rhel-10-x64` | *(dormant)* | bubblewrap, hyperlight, lxc | +| `debian-13` | linux | `1es-mxc-e2e-debian-13-x64` | *(dormant)* | bubblewrap, hyperlight, lxc | +| `macos-26` | macos | — | runner `macos-26` | seatbelt | +| `macos-15` | macos | — | runner `macos-15` | seatbelt | + +ARM64 is declared throughout but never emitted: no Azure VM SKU offers nested +virtualization on ARM CPUs yet, so the resolver filters Windows/Linux ARM64 out +after expansion (`suppressNonMacArm64`). macOS is ARM64-only. + +### Backend ids + +A backend id is passed straight through: the matrix job hands it to the host-prep +script and then to the dispatcher, which has one `switch`/`case` per id. Ids that +share a suite each keep their own case so they can diverge later without a +mapping table — `process-t1` and `process-t3` both run +`WinProcessContainer-Tests.ps1` today. Teaching the Process Container test suite to +accept an explicit tier (so a T1 host can also be exercised +at the T3 fallback) is a worthwhile future improvement; see +[Possible future improvements](#possible-future-improvements). + +An unwired backend fails loudly on purpose: adding it to a trigger produces a +red job ("write the tests or remove it"), never a green no-op. The dispatchers' +accepted-id lists (`ValidateSet` on Windows, the `case` arms on Unix) are what +catch a typo'd id in the catalog. + +### `triggers` + +Names the OS/backend pairs a plan runs. Entries are architecture-neutral: +expansion emits a job for every architecture of that platform that declares the +backend **and** has a non-empty pool. + +| Plan | Wired to | Contents today | +|------|----------|----------------| +| `nightly` | scheduled Mon–Sun | 4 Windows platforms, 4 Linux platforms | +| `weekly` | scheduled Sunday | empty | +| `pr` | *(nothing — `Build.yml` does not call the matrix job)* | empty; reserved for a potential future PR-time subset | +| `enabled` | *(nothing — resolvable locally only)* | reserved for testing this infrastructure and rapid iteration | + +Resolved `nightly` today = **17 jobs**: 9 Windows (prerelease × process-t1, +isolation-session, wslc; 25H2/24H2/23H2 × process-t3 + wslc) and +8 Linux (each of the four distros × bubblewrap + lxc). macOS resolves empty +because Seatbelt has no wired suite. + +## Backend status + +Snapshot of what the matrix actually proves today. Update this table as backends +get fixed or wired. + +| Backend | Status | Notes | +|---------|--------|-------| +| Process T1 | ✅ Good | Prerelease Windows only. Remaining failures are genuine MXC bugs or harness limitations. | +| Process T3 | ✅ Good | Non-prerelease Windows builds only, until the testing suite is updated. | +| Bubblewrap | ✅ Good | | +| LXC | ⚠️ Mostly good | Some networking tests fail on distros other than Ubuntu 24.04; host-vs-MXC cause not yet isolated. | +| WSLC | ⚠️ Mostly good | Can hit a download rate limit while updating WSL / pulling container images. Planned fix: split into several jobs spaced ~15 min apart. | +| IsolationSession | ⚠️ Blocked | `Feature_AgentSessionsBaseSupport` is not enabled on the pool image yet. | +| Windows Sandbox | ⛔ Not scheduled | Dispatcher case is wired; no trigger entry yet. | +| MicroVM | ⛔ Not working | Windows cold and warm starts hang; no Linux suite. The artifact payload is currently commented out in the build jobs. | +| Hyperlight | ⛔ Not implemented | No suite on any platform. | +| Seatbelt | ⛔ Not implemented | The backend itself is healthy; there is no official E2E suite to dispatch to. | + +## Host preparation + +Preparation runs before the tests, keyed by the matrix backend id. A backend +with no prerequisites is an explicit no-op, so the step runs unconditionally for +every entry. + +`prepare-windows-host.ps1`: + +- `process-t3` — runs `wxc-host-prep.exe prepare-system-drive` and + `prepare-null-device --no-sacl`. +- `microvm` — asserts the NanVix payload is in the artifact, adds a Defender + exclusion for the binary directory, and requires the Windows Hypervisor + Platform feature *and* a running hypervisor. +- `wslc` — asserts `wslcsdk.dll` shipped, requires the WSL and + VirtualMachinePlatform optional features to be baked into the image, then + installs/updates the WSL runtime (including the pre-release ring) up to the + minimum version parsed from `WSLC_SDK_VERSION` in + `src/backends/wslc/common/build.rs`. +- everything else — prints a "no prerequisites yet" line. + +Windows optional features are **verified, never enabled**: turning one on needs a +reboot the job cannot take, so a mis-imaged pool fails here with a pointed +message instead of surfacing later as an opaque backend error. + +`prepare-linux-host.sh`: + +- `bubblewrap` — installs `bwrap` (apt/dnf/yum/microdnf) and relaxes + `kernel.apparmor_restrict_unprivileged_userns` (ephemeral CI hosts only). +- `lxc` — installs the LXC stack, reloads the AppArmor profile, starts and waits + for `lxcbr0`, and prints network diagnostics. On RHEL-likes it first needs + EPEL, because Red Hat dropped LXC after RHEL 7 and ships no replacement. +- `microvm` — asserts the NanVix payload exists. +- `hyperlight` — no-op. + +macOS has no preparation step. + +## Runbook + +Always finish with a local resolve, which runs the full catalog validation: + +```bash +node scripts/ci/resolve-validation-test-matrix.mjs --plan nightly +``` + +An invalid catalog fails here and in the `resolve` job — before any specialized +test runner is allocated. + +### Schedule an existing backend on an existing OS + +1. Add the backend id to that platform/arch's `backends` list in + `validation-test-matrix.json` if it isn't already declared. +2. Add it to the platform's entry under the plan you want in `triggers`, + creating the `{ "os": …, "backends": [] }` entry if the platform isn't listed. +3. Confirm the platform/arch has a non-empty `pool` (or `runner` on macOS) — + otherwise it silently resolves to nothing. +4. Resolve locally and check the new combination appears. + +### Stop running something + +- **Temporarily, one backend:** remove it from the `triggers` entry. The + platform keeps declaring the capability. +- **Temporarily, a whole platform/arch:** blank its `pool` (`""`). It stays + documented but is never scheduled. +- **Permanently:** remove the trigger entry, then the `backends` entries, then + the platform. If that leaves a backend id declared nowhere, decide whether to + keep its dispatcher and host-prep branches (harmless) or delete them too. + +### Add a new backend + +1. **Catalog:** add the id to the `backends` list of every platform/arch that + can run it. There is no separate registration step — the id *is* the + dispatcher argument. +2. **Dispatcher:** add a case to `run_ci_backend_tests.ps1` (`ValidateSet` + + `switch`) or `run_ci_backend_tests.sh` (`usage` + `case`), pointing at the + suite. Until a suite exists, leave the explicit throw / `exit 2` so + accidental activation fails loudly. +3. **Host prep:** add a branch to `prepare-windows-host.ps1` (`ValidateSet` + + `switch`) or `prepare-linux-host.sh` (`usage` + `case`). Skip only if there + is genuinely nothing to install or assert. +4. **Artifact:** make sure everything the suite needs is in the + `Upload binaries` list of the relevant `Build.*.Job.yml`, and that the build + enables the backend's cargo feature. +5. **Trigger:** add the OS/backend pair to a plan. + +If two ids should run the same suite, give each its own `case` and have both call +the shared function — that is how `process-t1` and `process-t3` are wired. Keep +that split in the dispatcher, not in the workflow YAML, so a case can start +passing a distinguishing argument later without touching the matrix. + +### Add a new OS image + +1. Stand up the 1ES pool (Windows/Linux) with the required optional features + already baked into the image — the jobs verify but never enable them. +2. Add a `platforms` entry: `id`, `displayName`, `family`, and per-architecture + `target`, `artifact`, `pool`/`runner`, and `backends`. +3. For a Windows prerelease image set `"prerelease": true` and use a neutral + `windows-prerelease-` id — the id appears in public job names. +4. For a new Linux distro, check that `prepare-linux-host.sh` handles its + package manager and service layout. +5. Add it to a plan's `triggers`, then resolve locally. + +### Wire an unwired backend to a suite + +Replace the explicit failure in the dispatcher with the suite invocation, add +any host prerequisites, then add the OS/backend pair to a trigger. Always verify +by testing it ahead of time. + +### Change the schedule + +Everything schedule-related lives in `Validation.Tests.Scheduled.yml`: the two +`cron` entries, the `if:` conditions on `test-nightly` / `test-weekly`, and the +`workflow_dispatch` `plan` choices. Keep the three in sync — a new plan needs a +cron *and* a job condition *and* a dispatch choice. + +### Add a new plan + +1. Add the key to `triggers` in the catalog. That is what defines the plan — + `resolve-validation-test-matrix.mjs` derives its plan list from these keys, + so it needs no edit. +2. Add a job that calls `Validation.Tests.Matrix.Job.yml` with that plan, plus a + `workflow_dispatch` choice if it should be runnable on demand. + +### Enable ARM64 + +Set the ARM64 `pool` for the platform *and* remove or narrow +`suppressNonMacArm64` in the resolver. Note that the resolver rejects +`hyperlight` and `microvm` on ARM64 outright (x64-only runtimes), and the WSLC +dispatcher still refuses non-x64. + +## Testing Your Changes to the Validation Infrastructure + +1. Pick a pre-existing trigger or make a custom trigger with the tests you plan + to run in `scripts/ci/validation-test-matrix.json`. +2. Create a workflow file in your branch with the following code, replacing the + branch name and plan name with your branch name and trigger name respectively. +3. Push your changes. + +```yml +name: Validation Infrastructure Testing + +on: + push: + branches: + - # BRANCH NAME HERE + +concurrency: + group: validation-infra-pr-tests-${{ github.ref }} + cancel-in-progress: true + +permissions: + actions: read + contents: read + +jobs: + dependency-feed-check: + uses: ./.github/workflows/Dependency.Feed.Check.Job.yml + + windows: + needs: dependency-feed-check + uses: ./.github/workflows/Build.Windows.Job.yml + + linux: + needs: dependency-feed-check + uses: ./.github/workflows/Build.Linux.Job.yml + + macos: + needs: dependency-feed-check + uses: ./.github/workflows/Build.MacOS.Job.yml + + test: + needs: [windows, linux, macos] + uses: ./.github/workflows/Validation.Tests.Matrix.Job.yml + with: + plan: # YOUR PLAN HERE +``` + +## Important to Note + +- **A green job does not prove a suite ran.** Several suites (notably + IsolationSession) print `SKIPPED` and exit 0 on an unsupported host, and the + dispatchers propagate only the exit code. A matrix entry asserts the host + *should* support the backend, so a silent skip there is a coverage gap — check + the `SKIPPED` line or the executed count in the log, not just the exit status. +- **Empty pool = invisible.** A trigger entry pointing at a platform whose pool + is blank resolves to zero jobs and reports nothing. Resolve locally after any + catalog edit. +- **All OS build jobs must pass** before validation testing happens. +- **Artifacts live one day.** Re-running a test job long after the build has + expired fails at download; re-run the whole workflow instead. diff --git a/docs/pull-requests.md b/docs/pull-requests.md index 617ae9188..ac1407e40 100644 --- a/docs/pull-requests.md +++ b/docs/pull-requests.md @@ -3,9 +3,11 @@ ## GitHub Actions (automatic) Every PR is validated automatically by the GitHub Actions workflows under -`.github/workflows/` (entry point: `Build.yml`). This is the primary PR -signal — it builds and tests on native Windows x64/arm64, Linux x64/arm64, -and macOS arm64 hosts in parallel. +`.github/workflows/` (entry point: `Build.yml`). This is the primary PR signal — +it fans out to the reusable `Build.Windows.Job.yml`, `Build.Linux.Job.yml`, and +`Build.MacOS.Job.yml` workflows, which build and test on native Windows +x64/arm64, Linux x64/arm64, and macOS arm64 hosts, then runs the lint, +versioning, and SDK jobs. ## Azure Pipelines (optional on PRs, required on `main`) diff --git a/scripts/ci/prepare-linux-host.sh b/scripts/ci/prepare-linux-host.sh new file mode 100644 index 000000000..16eafcfeb --- /dev/null +++ b/scripts/ci/prepare-linux-host.sh @@ -0,0 +1,220 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Prepares a Linux host for a backend's artifact-only test suite by installing +# the packages and starting the services it needs. Distro-aware so the same +# matrix entry works on Ubuntu, Debian, and RHEL images. + +usage() { + echo "Usage: $0 " >&2 +} + +if [[ $# -ne 2 ]]; then + usage + exit 2 +fi + +backend="$1" +binary_directory="$2" + +apt_update() { + # Unrelated third-party repositories on the pool images can fail to + # refresh; the package install below still decides success. + if ! sudo apt-get update; then + echo "WARNING: apt-get update reported repository errors; continuing with available package indexes." >&2 + fi +} + +# Red Hat ships no third-party content, so epel-release is not in RHEL's own +# repos; the documented install is the release RPM straight from Fedora. +install_epel() { + local package_manager="$1" + + if command -v subscription-manager >/dev/null 2>&1; then + sudo subscription-manager repos \ + --enable "codeready-builder-for-rhel-10-$(arch)-rpms" || + echo "WARNING: could not enable the CRB repository; EPEL packages that depend on it may fail to install." >&2 + fi + + sudo "$package_manager" install -y \ + https://dl.fedoraproject.org/pub/epel/epel-release-latest-10.noarch.rpm +} + +install_bubblewrap() { + if command -v bwrap >/dev/null 2>&1; then + return + fi + if command -v apt-get >/dev/null 2>&1; then + apt_update + sudo apt-get install -y --no-install-recommends bubblewrap + elif command -v dnf >/dev/null 2>&1; then + sudo dnf install -y bubblewrap + elif command -v yum >/dev/null 2>&1; then + sudo yum install -y bubblewrap + elif command -v microdnf >/dev/null 2>&1; then + sudo microdnf install -y bubblewrap + else + echo "No supported package manager found to install bubblewrap." >&2 + exit 1 + fi +} + +install_lxc() { + if command -v lxc-start >/dev/null 2>&1; then + return + fi + if command -v apt-get >/dev/null 2>&1; then + apt_update + # Debian dropped lxc-utils; Ubuntu still ships it. + local packages=(lxc dnsmasq-base iptables bridge-utils) + if apt-cache show lxc-utils >/dev/null 2>&1; then + packages+=(lxc-utils) + fi + sudo apt-get install -y --no-install-recommends "${packages[@]}" + elif command -v dnf >/dev/null 2>&1; then + install_epel dnf + sudo dnf install -y lxc lxc-templates dnsmasq iptables + elif command -v yum >/dev/null 2>&1; then + install_epel yum + sudo yum install -y lxc lxc-templates dnsmasq iptables + elif command -v microdnf >/dev/null 2>&1; then + install_epel microdnf + sudo microdnf install -y lxc lxc-templates dnsmasq iptables + else + echo "No supported package manager found to install LXC." >&2 + exit 1 + fi +} + +# Start the LXC bridge and wait until it can actually serve containers. +# A freshly installed lxc-net needs a moment before lxcbr0 has its IPv4 and +# dnsmasq is answering DHCP/DNS. Without this wait a container can boot into a +# bridge that has no lease to give, which surfaces much later as an unrelated +# name-resolution failure inside the guest. +start_lxc_bridge() { + local bridge="${LXC_BRIDGE:-lxcbr0}" + + if command -v systemctl >/dev/null 2>&1; then + if systemctl list-unit-files lxc-net.service >/dev/null 2>&1 && + systemctl cat lxc-net.service >/dev/null 2>&1; then + if ! sudo systemctl start lxc-net; then + echo "WARNING: failed to start lxc-net; container networking may be unavailable." >&2 + fi + else + echo "No lxc-net unit on this distribution; skipping bridge startup." + fi + fi + + if ! ip link show "$bridge" >/dev/null 2>&1; then + echo "WARNING: bridge $bridge does not exist; container networking may be unavailable." >&2 + return 0 + fi + + local deadline=$((SECONDS + 30)) + while (( SECONDS < deadline )); do + if ip -4 addr show "$bridge" 2>/dev/null | grep -q 'inet '; then + echo "Bridge $bridge is up:" + ip -4 addr show "$bridge" | sed -n 's/^[[:space:]]*\(inet .*\)$/ \1/p' + if pgrep -f "dnsmasq.*$bridge" >/dev/null 2>&1; then + echo " dnsmasq is serving $bridge" + else + echo " WARNING: no dnsmasq bound to $bridge; DHCP and DNS may fail." >&2 + fi + return 0 + fi + sleep 1 + done + + echo "WARNING: $bridge did not receive an IPv4 address within 30s." >&2 + ip addr show "$bridge" || true +} + +# Report the host-side state that container networking depends on. Purely +# diagnostic: never fails the job, so a networking problem still surfaces as +# the backend test failure rather than as a prerequisite error. +report_lxc_network_diagnostics() { + local bridge="${LXC_BRIDGE:-lxcbr0}" + + echo "--- LXC network diagnostics (host) ---" + + echo "net.ipv4.ip_forward: $(cat /proc/sys/net/ipv4/ip_forward 2>/dev/null || echo unknown)" + + echo "Bridge $bridge:" + ip -4 addr show "$bridge" 2>/dev/null | sed 's/^/ /' || echo " (absent)" + + echo "dnsmasq processes:" + pgrep -af dnsmasq 2>/dev/null | sed 's/^/ /' || echo " (none)" + + echo "NAT rules for the bridge subnet:" + sudo iptables -t nat -S POSTROUTING 2>/dev/null | grep -E '10\.0\.3|MASQUERADE' | + sed 's/^/ /' || echo " (none found)" + + echo "FORWARD policy and bridge rules:" + sudo iptables -S FORWARD 2>/dev/null | grep -E "policy|$bridge" | sed 's/^/ /' || + echo " (none found)" + + echo "Host /etc/resolv.conf nameservers:" + grep '^nameserver' /etc/resolv.conf 2>/dev/null | sed 's/^/ /' || echo " (none)" + + echo "lxc-net configuration:" + grep -E '^(USE_LXC_BRIDGE|LXC_ADDR|LXC_NETMASK|LXC_DHCP_RANGE|LXC_DHCP_CONFILE)' \ + /etc/default/lxc-net 2>/dev/null | sed 's/^/ /' || echo " (no /etc/default/lxc-net)" + + # Prove the host itself can resolve the name the network test uses. If this + # fails, the container was never going to succeed. + if command -v getent >/dev/null 2>&1; then + echo "Host resolution of api.github.com:" + getent ahostsv4 api.github.com 2>/dev/null | head -n 2 | sed 's/^/ /' || + echo " FAILED - the host cannot resolve it either" + fi + + # Ask the bridge's own resolver, which is what a container is handed via + # DHCP. This isolates "dnsmasq is broken" from "the host is fine". + local bridge_ip + bridge_ip="$(ip -4 -o addr show "$bridge" 2>/dev/null | + awk '{print $4}' | cut -d/ -f1 | head -n 1)" + if [[ -n "$bridge_ip" ]] && command -v nslookup >/dev/null 2>&1; then + echo "Resolution via bridge resolver ($bridge_ip):" + nslookup api.github.com "$bridge_ip" 2>&1 | tail -n 4 | sed 's/^/ /' || + echo " FAILED - dnsmasq on $bridge is not answering" + fi + + echo "--- end diagnostics ---" +} + +chmod +x "$binary_directory/lxc-exec" +case "$backend" in + bubblewrap) + install_bubblewrap + command -v bwrap + # disabled AppArmor restrictions on unprivileged user namespaces, which bubblewrap needs to create a new namespace. + # should only be used on ephemeral CI runners, not on persistent hosts. + if sysctl kernel.apparmor_restrict_unprivileged_userns >/dev/null 2>&1; then + sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 + fi + ;; + lxc) + install_lxc + command -v lxc-start + sudo -n true + # Package installs may not activate the AppArmor profile, which + # lxc-start needs. + if command -v apparmor_parser >/dev/null 2>&1; then + sudo apparmor_parser -rT /etc/apparmor.d/lxc* 2>/dev/null || true + fi + start_lxc_bridge + report_lxc_network_diagnostics + ;; + microvm) + for file in nanvixd.elf nanvix_rootfs.img python3.initrd bin/kernel.elf; do + test -f "$binary_directory/$file" + done + ;; + hyperlight) + echo "Hyperlight has no artifact-only Linux test prerequisites yet." + ;; + *) + usage + exit 2 + ;; +esac diff --git a/scripts/ci/prepare-windows-host.ps1 b/scripts/ci/prepare-windows-host.ps1 new file mode 100644 index 000000000..76955478d --- /dev/null +++ b/scripts/ci/prepare-windows-host.ps1 @@ -0,0 +1,322 @@ +#Requires -Version 7.0 + +<# +.SYNOPSIS + Prepares a Windows host for a backend's artifact-only test suite. + +.PARAMETER Backend + Matrix backend id. + +.PARAMETER BinaryDirectory + Directory holding the downloaded build artifact. + +.EXAMPLE + ./scripts/ci/prepare-windows-host.ps1 -Backend process-t3 -BinaryDirectory artifacts/bin +#> + +[CmdletBinding()] +param( + [Parameter(Mandatory)] + [ValidateSet( + 'process-t1', + 'process-t3', + 'isolation-session', + 'wslc', + 'windows-sandbox', + 'microvm', + 'hyperlight' + )] + [string]$Backend, + + [Parameter(Mandatory)] + [string]$BinaryDirectory +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +function Exit-WithError { + param([Parameter(Mandatory)][string]$Message) + + Write-Host "::error::$Message" + exit 1 +} + +function Assert-RequiredFile { + param( + [Parameter(Mandatory)][string[]]$RelativePath + ) + + $missing = $RelativePath | Where-Object { -not (Test-Path (Join-Path $BinaryDirectory $_)) } + if ($missing) { + Exit-WithError "Missing binaries: $($missing -join ', ')" + } + + $leaves = $RelativePath | ForEach-Object { Split-Path $_ -Leaf } + Get-ChildItem $BinaryDirectory -Include $leaves -Recurse | Format-Table FullName, Length +} + +# Read a Windows optional feature's state without throwing, so both the +# diagnostic and assertion paths can share one query. A host that cannot answer +# (querying needs elevation) reports the reason as its state rather than +# aborting, which keeps the failure message actionable. +function Get-OptionalFeatureState { + param([Parameter(Mandatory)][string]$Name) + + try { + $feature = Get-WindowsOptionalFeature -Online -FeatureName $Name -ErrorAction Stop + } catch { + return "query-failed: $($_.Exception.Message.Trim())" + } + + if ($null -eq $feature) { + return 'unknown' + } + return [string]$feature.State +} + +# Require every named optional feature to be Enabled. Enabling one needs a +# reboot the runner cannot take mid-job, so this verifies rather than installs: +# a mis-imaged pool fails here with a pointed message instead of surfacing +# later as an opaque backend error. $Remedy names the image-level fix. +function Assert-RequiredFeature { + param( + [Parameter(Mandatory)][string[]]$Name, + [Parameter(Mandatory)][string]$Remedy + ) + + $notEnabled = @() + foreach ($feature in $Name) { + $state = Get-OptionalFeatureState -Name $feature + Write-Host " $feature = $state" + if ($state -ne 'Enabled') { + $notEnabled += "$feature ($state)" + } + } + + if ($notEnabled) { + Exit-WithError "Required Windows optional feature(s) not enabled: $($notEnabled -join '; '). $Remedy" + } +} + +# Report the hypervisor state a VM-backed backend depends on. Purely +# diagnostic: never fails, so a hypervisor problem surfaces as the explicit +# check below rather than as an unexplained collection error. +function Write-HypervisorDiagnostic { + Write-Host '=== Hypervisor Diagnostics ===' + Write-Host "OS: $([System.Environment]::OSVersion)" + + $computerSystem = Get-CimInstance -ClassName Win32_ComputerSystem -ErrorAction SilentlyContinue + $hypervisorPresent = if ($null -eq $computerSystem) { 'unknown' } else { $computerSystem.HypervisorPresent } + Write-Host "HypervisorPresent: $hypervisorPresent" + Write-Host "WinHvPlatform.dll exists: $(Test-Path "$env:SystemRoot\System32\WinHvPlatform.dll")" + Write-Host '=== end diagnostics ===' +} + +# The feature can be enabled while the hypervisor is not actually running (for +# example when a host reboot is still pending), so both are required. +function Assert-HypervisorPlatform { + Assert-RequiredFeature -Name 'HypervisorPlatform' ` + -Remedy 'This backend requires Windows Hypervisor Platform on the runner image.' + + $computerSystem = Get-CimInstance -ClassName Win32_ComputerSystem -ErrorAction SilentlyContinue + if ($null -eq $computerSystem -or -not $computerSystem.HypervisorPresent) { + Exit-WithError 'HypervisorPresent is false - WHP feature is enabled but hypervisor is not running.' + } + + Write-Host 'WHP is enabled and hypervisor is present.' +} + +function Initialize-ProcessContainerHost { + $hostPrep = Join-Path $BinaryDirectory 'wxc-host-prep.exe' + if (-not (Test-Path $hostPrep)) { + Exit-WithError "wxc-host-prep.exe not found in $BinaryDirectory" + } + + # The AppContainer tier needs the system-drive ACEs and the \Device\Null + # security descriptor. --no-sacl keeps the descriptor within what a CI host + # can grant without SeSecurityPrivilege. + & $hostPrep prepare-system-drive + if ($LASTEXITCODE -ne 0) { + Exit-WithError "wxc-host-prep prepare-system-drive failed with exit code $LASTEXITCODE" + } + + & $hostPrep prepare-null-device --no-sacl + if ($LASTEXITCODE -ne 0) { + Exit-WithError "wxc-host-prep prepare-null-device failed with exit code $LASTEXITCODE" + } +} + +function Initialize-MicroVmHost { + # Staged next to wxc-exec.exe by the --features microvm build, so their + # absence means a broken artifact rather than a host problem. Snapshots are + # excluded: they are a warm-start cache the runner regenerates on demand. + Assert-RequiredFile @( + 'wxc-exec.exe', + 'nanvixd.exe', + 'nanvix_rootfs.img', + 'python3.initrd', + 'bin\kernel.elf' + ) + + # NanVix boots a VM from these images on every invocation; Defender scanning + # them can push boot past its timeout. + Add-MpPreference -ExclusionPath $BinaryDirectory + Write-Host "Added Defender exclusion for $BinaryDirectory" + + Write-HypervisorDiagnostic + Assert-HypervisorPlatform +} + +# The optional features must be baked into the pool image (enabling one needs a +# reboot this job cannot take), but the WSL runtime package is installed here if +# missing. Container images are pulled by the suite itself +# (tests/scripts/run_wslc_all_tests.ps1). +function Initialize-WslcHost { + # wslcsdk.dll ships beside wxc-exec.exe only in a --features wslc build. + Assert-RequiredFile @('wxc-exec.exe', 'wslcsdk.dll') + + Assert-RequiredFeature -Name 'Microsoft-Windows-Subsystem-Linux', 'VirtualMachinePlatform' ` + -Remedy 'WSL2 must be baked into the runner image; enabling these features requires a host reboot this job cannot take.' + + Write-Host "=== wsl.exe presence + status ===" + $wsl = Get-Command wsl.exe -ErrorAction SilentlyContinue + if ($wsl) { + Write-Host "wsl.exe: $($wsl.Source)" + } else { + Write-Host "wsl.exe NOT found on PATH" + Exit-WithError 'WSL2 is not installed on this runner. The runner image must include WSL2 for this backend.' + } + + $previousEncoding = [Console]::OutputEncoding + [Console]::OutputEncoding = [System.Text.Encoding]::Unicode + $output = wsl.exe --status 2>&1 | Out-String + Write-Host $output + [Console]::OutputEncoding = $previousEncoding + + # # if unicode output mentions wsl.exe --install, skip version check for now + if ($output -match 'wsl.exe --install') { + Exit-WithError 'WSL2 is not installed on this runner. The runner image must include WSL2 for this backend.' + } + + if ((Invoke-Wsl @('--version') -Quiet) -ne 0) { + Write-Host 'version command failed, so WSL2 is installed but not updated.' + Write-Host "=== updating inbox WSL to modern version ===" + + if ((Invoke-Wsl @('--update', '--web-download') -Quiet) -ne 0 -and + (Invoke-Wsl @('--update')) -ne 0) { + Exit-WithError 'wsl --update failed; the WSL2 runtime could not be installed on this runner.' + } + + if ((Invoke-Wsl @('--version') -Quiet) -ne 0) { + Exit-WithError 'wsl --version failed after updating; the WSL2 runtime is not usable on this runner.' + } + + Write-Host 'WSL2 is installed and updated (not prerelease, yet)' + + } + + # WSLC needs a runtime at least as new as the pinned WSLC SDK, and those + # builds ship only on the pre-release ring — the stable ring lands well + # behind it. Without this the SDK fails at run time with + # "WSLC runtime unavailable. Missing components: WslPackage". + $required = Get-RequiredWslVersion + $installed = Get-InstalledWslVersion + if ($null -ne $required -and ($null -eq $installed -or $installed -lt $required)) { + Write-Host "WSL $installed is older than the $required WSLC requires; updating to pre-release..." + if ((Invoke-Wsl @('--update', '--pre-release', '--web-download') -Quiet) -ne 0 -and + (Invoke-Wsl @('--update', '--pre-release')) -ne 0) { + Exit-WithError "wsl --update --pre-release failed; WSLC requires WSL $required or newer." + } + $installed = Get-InstalledWslVersion + } + + if ($null -eq $installed) { + Exit-WithError 'wsl --version failed after updating; the WSL2 runtime is not usable on this runner.' + } + if ($null -ne $required -and $installed -lt $required) { + Exit-WithError "WSL $installed is installed, but WSLC requires $required or newer." + } + Write-Host "WSL runtime $installed is ready (WSLC requires $required or newer)." + + Write-Host "=== done. ===" +} + +# Minimum WSL runtime for WSLC, read from the pinned SDK version so the two +# cannot drift. The SDK's own runtime error names this same version. +function Get-RequiredWslVersion { + $buildScript = Join-Path $PSScriptRoot '..\..\src\backends\wslc\common\build.rs' + if (-not (Test-Path $buildScript)) { + Write-Host "WARNING: $buildScript not found; skipping the WSL version gate." + return $null + } + + $match = [regex]::Match((Get-Content $buildScript -Raw), 'WSLC_SDK_VERSION:\s*&str\s*=\s*"([0-9]+(?:\.[0-9]+)+)"') + if (-not $match.Success) { + Write-Host 'WARNING: could not parse WSLC_SDK_VERSION; skipping the WSL version gate.' + return $null + } + return [version]$match.Groups[1].Value +} + +# Installed modern-runtime version, or $null when wsl.exe is the legacy inbox +# build (no --version) or otherwise unusable. +function Get-InstalledWslVersion { + $result = Invoke-WslCapture -Arguments @('--version') + if ($result.ExitCode -ne 0) { + return $null + } + + $match = [regex]::Match($result.Output, '(?im)^\s*WSL version:\s*([0-9]+(?:\.[0-9]+)+)') + if (-not $match.Success) { + return $null + } + return [version]$match.Groups[1].Value +} + + +# wsl.exe emits UTF-16LE, which the default console encoding renders as +# null-separated garbage. Returns @{ ExitCode; Output } with the output decoded. +function Invoke-WslCapture { + param([Parameter(Mandatory)][string[]]$Arguments) + + $previousEncoding = [Console]::OutputEncoding + try { + [Console]::OutputEncoding = [System.Text.Encoding]::Unicode + $output = & wsl.exe @Arguments 2>&1 | Out-String + return @{ ExitCode = $LASTEXITCODE; Output = $output } + } catch { + return @{ ExitCode = 1; Output = "wsl.exe could not be run: $($_.Exception.Message)" } + } finally { + [Console]::OutputEncoding = $previousEncoding + } +} + +# Run wsl.exe and return its exit code. -Quiet suppresses output for probes, +# where the legacy wsl.exe dumps its whole usage text on an unknown switch. +function Invoke-Wsl { + param( + [Parameter(Mandatory)][string[]]$Arguments, + [switch]$Quiet + ) + + $result = Invoke-WslCapture -Arguments $Arguments + if (-not $Quiet -and $result.Output.Trim()) { + Write-Host $result.Output.Trim() + } + return $result.ExitCode +} + +if (-not (Test-Path $BinaryDirectory)) { + Exit-WithError "Binary directory not found: $BinaryDirectory" +} +$BinaryDirectory = (Resolve-Path $BinaryDirectory).Path + +Write-Host "Preparing Windows host for backend '$Backend' using $BinaryDirectory" + +switch ($Backend) { + 'process-t3' { Initialize-ProcessContainerHost } + 'microvm' { Initialize-MicroVmHost } + 'wslc' { Initialize-WslcHost } + default { Write-Host "$Backend has no artifact-only Windows test prerequisites yet." } +} diff --git a/scripts/ci/resolve-validation-test-matrix.mjs b/scripts/ci/resolve-validation-test-matrix.mjs new file mode 100644 index 000000000..9c73382d2 --- /dev/null +++ b/scripts/ci/resolve-validation-test-matrix.mjs @@ -0,0 +1,257 @@ +#!/usr/bin/env node + +// Validates the declarative test catalog and emits GitHub Actions matrices. +// Keeping expansion here makes the workflow YAML small and lets CI reject an +// invalid catalog before allocating any specialized test runners. + +import fs from 'node:fs'; +import path from 'node:path'; +import process from 'node:process'; +import { fileURLToPath } from 'node:url'; + +const FAMILIES = ['windows', 'linux', 'macos']; +const ARM64_UNSUPPORTED_BACKENDS = new Set(['hyperlight', 'microvm']); + +function assertNonEmptyString(value, label) { + if (typeof value !== 'string' || value.trim() === '') { + throw new Error(`${label} must be a non-empty string`); + } +} + +export function readCatalog(catalogPath) { + return JSON.parse(fs.readFileSync(catalogPath, 'utf8')); +} + +export function validateCatalog(catalog) { + if (catalog.schemaVersion !== 1) { + throw new Error(`unsupported catalog schemaVersion: ${catalog.schemaVersion}`); + } + + const platforms = new Map(); + const targets = new Set(); + for (const platform of catalog.platforms ?? []) { + assertNonEmptyString(platform.id, 'platform.id'); + assertNonEmptyString(platform.displayName, `${platform.id}.displayName`); + if (!FAMILIES.includes(platform.family)) { + throw new Error(`${platform.id} has unsupported family ${platform.family}`); + } + if (platforms.has(platform.id)) { + throw new Error(`duplicate platform id: ${platform.id}`); + } + if (platform.prerelease === true) { + // Prerelease platforms use neutral IDs in public matrix fields. + if (!/^windows-prerelease-[a-z-]+$/.test(platform.id)) { + throw new Error(`${platform.id} must use a neutral prerelease-platform alias`); + } + } + + const architectures = Object.entries(platform.architectures ?? {}); + if (architectures.length === 0) { + throw new Error(`${platform.id} has no architectures`); + } + + for (const [architecture, details] of architectures) { + if (!['x64', 'arm64'].includes(architecture)) { + throw new Error(`${platform.id} has unsupported architecture ${architecture}`); + } + assertNonEmptyString(details.target, `${platform.id}.${architecture}.target`); + assertNonEmptyString(details.artifact, `${platform.id}.${architecture}.artifact`); + targets.add(details.target); + + if (platform.family === 'macos') { + assertNonEmptyString(details.runner, `${platform.id}.${architecture}.runner`); + } else if (details.pool != null && typeof details.pool !== 'string') { + throw new Error(`${platform.id}.${architecture}.pool must be a string`); + } + + const backends = new Set(); + for (const backend of details.backends ?? []) { + assertNonEmptyString(backend, `${platform.id}.${architecture}.backend`); + if (backends.has(backend)) { + throw new Error(`duplicate backend ${backend} on ${platform.id}/${architecture}`); + } + if (architecture === 'arm64' && ARM64_UNSUPPORTED_BACKENDS.has(backend)) { + throw new Error(`${backend} cannot be scheduled on arm64 (${platform.id})`); + } + backends.add(backend); + } + } + platforms.set(platform.id, platform); + } + + const expectedTargets = new Set([ + 'aarch64-apple-darwin', + 'aarch64-pc-windows-msvc', + 'aarch64-unknown-linux-gnu', + 'x86_64-pc-windows-msvc', + 'x86_64-unknown-linux-gnu' + ]); + if (targets.size !== expectedTargets.size + || [...expectedTargets].some(target => !targets.has(target))) { + throw new Error(`catalog targets do not match the five required build targets`); + } + + // The catalog's `triggers` keys are the plan list: a plan exists because it + // is declared there. + const triggers = catalog.triggers; + if (triggers == null || typeof triggers !== 'object' || Array.isArray(triggers)) { + throw new Error('catalog triggers must be an object keyed by plan name'); + } + const plans = Object.keys(triggers); + if (plans.length === 0) { + throw new Error('catalog declares no plans under triggers'); + } + + // Trigger entries name an OS/backend pair. Architecture expansion happens + // later, so a backend is valid here when at least one OS architecture has it. + for (const plan of plans) { + assertNonEmptyString(plan, 'trigger plan name'); + if (!Array.isArray(triggers[plan])) { + throw new Error(`${plan} must be an array of trigger requests`); + } + + const seenRequests = new Set(); + for (const request of triggers[plan]) { + const platform = platforms.get(request.os); + if (!platform) { + throw new Error(`${plan} references unknown platform ${request.os}`); + } + for (const backend of request.backends ?? []) { + const requestKey = `${request.os}|${backend}`; + if (seenRequests.has(requestKey)) { + throw new Error(`duplicate ${plan} request ${requestKey}`); + } + const supported = Object.values(platform.architectures) + .some(details => details.backends.includes(backend)); + if (!supported) { + throw new Error(`${plan} requests unsupported ${request.os}/${backend}`); + } + seenRequests.add(requestKey); + } + } + } + + return { platforms, plans }; +} + +export function expandPlan(catalog, plan) { + const { platforms, plans } = validateCatalog(catalog); + if (!plans.includes(plan)) { + throw new Error(`unsupported plan: ${plan} (catalog declares: ${plans.join(', ')})`); + } + const combinations = []; + + for (const request of catalog.triggers[plan]) { + const platform = platforms.get(request.os); + // A trigger is architecture-neutral. Expand it only where the platform's + // capability declaration supports the requested backend. + for (const [architecture, details] of Object.entries(platform.architectures)) { + if (platform.family !== 'macos' && !details.pool?.trim()) { + continue; + } + for (const backend of request.backends) { + if (!details.backends.includes(backend)) { + continue; + } + combinations.push({ + plan, + os: platform.id, + os_name: platform.displayName, + family: platform.family, + architecture, + target: details.target, + artifact: details.artifact, + pool: details.pool, + runner: details.runner, + backend + }); + } + } + } + + return combinations; +} + +export function resolvePlan(catalog, plan) { + // expandPlan validates the catalog and rejects an unknown plan name. + const matrices = Object.fromEntries(FAMILIES.map(family => [family, []])); + + for (const combination of expandPlan(catalog, plan)) { + // A trigger entry means "run this". A backend without a test script fails + // in the dispatcher, which is an actionable result: write the tests or + // remove the backend from the trigger. + const { family, ...matrixEntry } = combination; + matrices[family].push(matrixEntry); + } + + suppressNonMacArm64(matrices); + sortMatrices(matrices); + return matrices; +} + +// Windows and Linux ARM64 hosted VMs currently lack nested virtualization. +// Keep their catalog entries intact for future enablement, but never emit them +// until suitable test hosts are available. macOS remains ARM64-only. +function suppressNonMacArm64(matrices) { + for (const family of ['windows', 'linux']) { + matrices[family] = matrices[family] + .filter(entry => entry.architecture !== 'arm64'); + } +} + +function sortMatrices(matrices) { + for (const family of FAMILIES) { + // Stable ordering keeps local output and workflow diagnostics reproducible. + matrices[family].sort((left, right) => ( + `${left.os}|${left.architecture}|${left.backend}` + .localeCompare(`${right.os}|${right.architecture}|${right.backend}`) + )); + } +} + +function parseArguments(argv) { + const args = { plan: undefined, catalog: undefined }; + for (let index = 0; index < argv.length; index += 1) { + const argument = argv[index]; + if (argument === '--plan') { + args.plan = argv[++index]; + } else if (argument === '--catalog') { + args.catalog = argv[++index]; + } else { + throw new Error(`unknown argument: ${argument}`); + } + } + if (!args.plan) { + throw new Error('--plan is required'); + } + return args; +} + +function writeOutputs(matrices) { + const lines = []; + for (const family of FAMILIES) { + // Empty-matrix flags let the reusable workflow skip an OS-family job + // instead of asking GitHub Actions to evaluate an empty matrix. + lines.push(`${family}=${JSON.stringify({ include: matrices[family] })}`); + lines.push(`has_${family}=${matrices[family].length > 0}`); + } + + if (process.env.GITHUB_OUTPUT) { + fs.appendFileSync(process.env.GITHUB_OUTPUT, `${lines.join('\n')}\n`); + } else { + process.stdout.write(`${lines.join('\n')}\n`); + } +} + +const currentFile = fileURLToPath(import.meta.url); +if (process.argv[1] && path.resolve(process.argv[1]) === currentFile) { + try { + const args = parseArguments(process.argv.slice(2)); + const defaultCatalog = path.join(path.dirname(currentFile), 'validation-test-matrix.json'); + const catalog = readCatalog(path.resolve(args.catalog ?? defaultCatalog)); + writeOutputs(resolvePlan(catalog, args.plan)); + } catch (error) { + process.stderr.write(`resolve-validation-test-matrix: ${error.message}\n`); + process.exitCode = 1; + } +} diff --git a/scripts/ci/validation-test-matrix.json b/scripts/ci/validation-test-matrix.json new file mode 100644 index 000000000..bf7b2fc62 --- /dev/null +++ b/scripts/ci/validation-test-matrix.json @@ -0,0 +1,392 @@ +{ + "schemaVersion": 1, + "platforms": [ + { + "id": "windows-prerelease-process-container", + "displayName": "Windows Prerelease T1 Process Container", + "family": "windows", + "prerelease": true, + "architectures": { + "x64": { + "target": "x86_64-pc-windows-msvc", + "artifact": "wxc-binaries-x86_64-pc-windows-msvc", + "pool": "1es-mxc-windows-prerelease-t1-x64", + "backends": [ + "process-t1", + "process-t3", + "isolation-session", + "wslc", + "windows-sandbox", + "microvm", + "hyperlight" + ] + }, + "arm64": { + "target": "aarch64-pc-windows-msvc", + "artifact": "wxc-binaries-aarch64-pc-windows-msvc", + "pool": "", + "backends": [ + "process-t1", + "process-t3", + "isolation-session", + "wslc", + "windows-sandbox" + ] + } + } + }, + { + "id": "windows-prerelease-isolation-session", + "displayName": "Windows Pre-release Isolation Session", + "family": "windows", + "prerelease": true, + "architectures": { + "x64": { + "target": "x86_64-pc-windows-msvc", + "artifact": "wxc-binaries-x86_64-pc-windows-msvc", + "pool": "", + "backends": [ + "process-t1", + "process-t3", + "isolation-session", + "wslc", + "windows-sandbox", + "microvm", + "hyperlight" + ] + }, + "arm64": { + "target": "aarch64-pc-windows-msvc", + "artifact": "wxc-binaries-aarch64-pc-windows-msvc", + "pool": "", + "backends": [ + "process-t1", + "process-t3", + "isolation-session", + "wslc", + "windows-sandbox" + ] + } + } + }, + { + "id": "windows-canary", + "displayName": "Windows canary", + "family": "windows", + "architectures": { + "x64": { + "target": "x86_64-pc-windows-msvc", + "artifact": "wxc-binaries-x86_64-pc-windows-msvc", + "pool": "", + "backends": [ + "process-t1", + "process-t3", + "isolation-session", + "wslc", + "windows-sandbox", + "microvm", + "hyperlight" + ] + }, + "arm64": { + "target": "aarch64-pc-windows-msvc", + "artifact": "wxc-binaries-aarch64-pc-windows-msvc", + "pool": "", + "backends": [ + "process-t1", + "process-t3", + "isolation-session", + "wslc", + "windows-sandbox" + ] + } + } + }, + { + "id": "windows-25h2", + "displayName": "Windows 25H2", + "family": "windows", + "architectures": { + "x64": { + "target": "x86_64-pc-windows-msvc", + "artifact": "wxc-binaries-x86_64-pc-windows-msvc", + "pool": "1es-mxc-e2e-windows-25h2-pro-x64", + "backends": [ + "process-t3", + "wslc", + "windows-sandbox", + "microvm", + "hyperlight" + ] + }, + "arm64": { + "target": "aarch64-pc-windows-msvc", + "artifact": "wxc-binaries-aarch64-pc-windows-msvc", + "pool": "", + "backends": [ + "process-t3", + "wslc", + "windows-sandbox" + ] + } + } + }, + { + "id": "windows-24h2", + "displayName": "Windows 24H2", + "family": "windows", + "architectures": { + "x64": { + "target": "x86_64-pc-windows-msvc", + "artifact": "wxc-binaries-x86_64-pc-windows-msvc", + "pool": "1es-mxc-e2e-windows-24h2-pro-x64", + "backends": [ + "process-t3", + "wslc", + "windows-sandbox", + "microvm", + "hyperlight" + ] + }, + "arm64": { + "target": "aarch64-pc-windows-msvc", + "artifact": "wxc-binaries-aarch64-pc-windows-msvc", + "pool": "", + "backends": [ + "process-t3", + "wslc", + "windows-sandbox" + ] + } + } + }, + { + "id": "windows-23h2", + "displayName": "Windows 23H2", + "family": "windows", + "architectures": { + "x64": { + "target": "x86_64-pc-windows-msvc", + "artifact": "wxc-binaries-x86_64-pc-windows-msvc", + "pool": "1es-mxc-e2e-windows-23h2-enterprise-x64", + "backends": [ + "process-t3", + "wslc", + "windows-sandbox", + "microvm", + "hyperlight" + ] + }, + "arm64": { + "target": "aarch64-pc-windows-msvc", + "artifact": "wxc-binaries-aarch64-pc-windows-msvc", + "pool": "", + "backends": [ + "process-t3", + "wslc", + "windows-sandbox" + ] + } + } + }, + { + "id": "ubuntu-26.04", + "displayName": "Ubuntu 26.04", + "family": "linux", + "architectures": { + "x64": { + "target": "x86_64-unknown-linux-gnu", + "artifact": "lxc-binaries-x86_64-unknown-linux-gnu", + "pool": "1es-mxc-e2e-ubuntu-26.04-x64", + "backends": [ + "bubblewrap", + "hyperlight", + "lxc" + ] + }, + "arm64": { + "target": "aarch64-unknown-linux-gnu", + "artifact": "lxc-binaries-aarch64-unknown-linux-gnu", + "pool": "", + "backends": [ + "bubblewrap", + "lxc" + ] + } + } + }, + { + "id": "ubuntu-24.04", + "displayName": "Ubuntu 24.04", + "family": "linux", + "architectures": { + "x64": { + "target": "x86_64-unknown-linux-gnu", + "artifact": "lxc-binaries-x86_64-unknown-linux-gnu", + "pool": "1es-mxc-e2e-ubuntu-24.04-x64", + "backends": [ + "bubblewrap", + "microvm", + "hyperlight", + "lxc" + ] + }, + "arm64": { + "target": "aarch64-unknown-linux-gnu", + "artifact": "lxc-binaries-aarch64-unknown-linux-gnu", + "pool": "", + "backends": [ + "bubblewrap", + "lxc" + ] + } + } + }, + { + "id": "rhel-10", + "displayName": "RHEL 10", + "family": "linux", + "architectures": { + "x64": { + "target": "x86_64-unknown-linux-gnu", + "artifact": "lxc-binaries-x86_64-unknown-linux-gnu", + "pool": "1es-mxc-e2e-rhel-10-x64", + "backends": [ + "bubblewrap", + "hyperlight", + "lxc" + ] + }, + "arm64": { + "target": "aarch64-unknown-linux-gnu", + "artifact": "lxc-binaries-aarch64-unknown-linux-gnu", + "pool": "", + "backends": [ + "bubblewrap", + "lxc" + ] + } + } + }, + { + "id": "debian-13", + "displayName": "Debian 13", + "family": "linux", + "architectures": { + "x64": { + "target": "x86_64-unknown-linux-gnu", + "artifact": "lxc-binaries-x86_64-unknown-linux-gnu", + "pool": "1es-mxc-e2e-debian-13-x64", + "backends": [ + "bubblewrap", + "hyperlight", + "lxc" + ] + }, + "arm64": { + "target": "aarch64-unknown-linux-gnu", + "artifact": "lxc-binaries-aarch64-unknown-linux-gnu", + "pool": "", + "backends": [ + "bubblewrap", + "lxc" + ] + } + } + }, + { + "id": "macos-26", + "displayName": "macOS 26", + "family": "macos", + "architectures": { + "arm64": { + "target": "aarch64-apple-darwin", + "artifact": "mxc-binaries-aarch64-apple-darwin", + "runner": "macos-26", + "backends": [ + "seatbelt" + ] + } + } + }, + { + "id": "macos-15", + "displayName": "macOS 15", + "family": "macos", + "architectures": { + "arm64": { + "target": "aarch64-apple-darwin", + "artifact": "mxc-binaries-aarch64-apple-darwin", + "runner": "macos-15", + "backends": [ + "seatbelt" + ] + } + } + } + ], + "triggers": { + "pr": [], + "nightly": [ + { + "os": "windows-prerelease-process-container", + "backends": [ + "process-t1", + "isolation-session", + "wslc" + ] + }, + { + "os": "windows-25h2", + "backends": [ + "process-t3", + "wslc" + ] + }, + { + "os": "windows-24h2", + "backends": [ + "process-t3", + "wslc" + ] + }, + { + "os": "windows-23h2", + "backends": [ + "process-t3", + "wslc" + ] + }, + { + "os": "ubuntu-26.04", + "backends": [ + "bubblewrap", + "lxc" + ] + }, + { + "os": "ubuntu-24.04", + "backends": [ + "bubblewrap", + "lxc" + ] + }, + { + "os": "rhel-10", + "backends": [ + "bubblewrap", + "lxc" + ] + }, + { + "os": "debian-13", + "backends": [ + "bubblewrap", + "lxc" + ] + } + ], + "weekly": [], + "enabled": [] + } +} diff --git a/sdk/node/tests/integration/test-helpers.ts b/sdk/node/tests/integration/test-helpers.ts index 1046d8dc8..8240cf8f7 100644 --- a/sdk/node/tests/integration/test-helpers.ts +++ b/sdk/node/tests/integration/test-helpers.ts @@ -71,6 +71,16 @@ export const EXPECTED_MACOS_BINARIES = [ const OPTIONAL_BINARIES = [ 'wslcsdk.dll', // Only built with --with-wslc 'wxc-wslc-daemon.exe', // Only built with --with-wslc + 'plm.exe', // Permissive Learning Mode helper (Windows-only); staged + // only when the plm crate is included in the build. + // Test-only binaries. The GitHub build artifact carries them so the + // validation matrix can run the Windows suites from a downloaded artifact, + // and the npm packager copies that whole artifact into bin/ — so they show + // up here. They are not required: no SDK consumer needs them, and the ADO + // package producer filters its artifact through signPattern, which + // deliberately ships only product binaries. + 'wxc-ui-probe.exe', // WinProcessContainer-Tests.ps1 + 'wxc-test-driver.exe', // run_test_configs.ps1 ]; // Combined list of all known binaries across platforms. The npm package diff --git a/tests/scripts/README.md b/tests/scripts/README.md index e4ff96dfa..f3e3d534a 100644 --- a/tests/scripts/README.md +++ b/tests/scripts/README.md @@ -1,7 +1,7 @@ # Test Scripts -This directory contains PowerShell convenience scripts for running MXC end-to-end -tests locally on Windows. The primary Rust executor E2E path is +This directory contains convenience scripts for running MXC end-to-end tests +locally and in CI. The primary Rust executor E2E path is `cargo test -p wxc_e2e_tests`, which invokes the MXC binaries directly instead of shelling through these scripts. @@ -9,11 +9,21 @@ All scripts accept a `-Release` switch to use the release build (default: debug) ## Prerequisites -- Windows 11 +Shared: + - Rust toolchain installed (`rustup`, `cargo`) - Built binaries (`cargo build` from `src/`) + +Windows (`.ps1`): + +- Windows 11 - PowerShell 7+ (`pwsh`) +Linux / macOS (`.sh`): + +- Bash, plus the per-backend prerequisites listed in the backend's doc (for + example `bwrap` for Bubblewrap, the LXC stack for LXC) + ## Scripts | Script | Description | Extra prerequisites | @@ -33,22 +43,71 @@ All scripts accept a `-Release` switch to use the release build (default: debug) | `run_processcontainer_proxy_tests.ps1` | Process container proxy tests | `wxc-exec.exe` | | `run_on_repeat.ps1` | Stress test (loops core tests) | `wxc-exec.exe` | -These scripts are local helpers. Not every script is run by CI because several -depend on local OS features such as Windows Sandbox, WHP, proxy setup, or stress -test duration. +### Linux suites + +| Script | Description | Extra prerequisites | +|--------|-------------|---------------------| +| `run_bwrap_all_tests.sh` | All Bubblewrap tests | `lxc-exec`, `bwrap` | +| `run_lxc_all_tests.sh` | All LXC tests | `lxc-exec`, LXC stack, root | + +Individual `run_bwrap_*.sh` / `run_lxc_*.sh` scripts run one case each; the +aggregate scripts above are what CI dispatches to. + +Not every script runs in CI: several depend on local OS features such as +Windows Sandbox, WHP, proxy setup, or stress-test duration. The ones CI does +run are reached through the dispatchers below rather than being invoked +directly. + +### CI dispatch + +The validation matrix (see `scripts/ci/validation-test-matrix.json` and +`.github/workflows/Validation.Tests.Matrix.Job.yml`, documented end to end in +[`docs/ci-validation-infrastructure.md`](../../docs/ci-validation-infrastructure.md)) +never builds from source. +It downloads a build artifact, prepares the host, and then hands off to one of +these dispatchers, which map a matrix backend id to the suites above: + +| Dispatcher | Platforms | Backend ids | +|------------|-----------|-------------| +| `run_ci_backend_tests.ps1` | Windows | `process-t1`, `process-t3`, `isolation-session`, `windows-sandbox`, `wslc`, `microvm`, `hyperlight` | +| `run_ci_backend_tests.sh` | Linux, macOS | `bubblewrap`, `lxc`, `seatbelt`, `microvm`, `hyperlight` | + +Pass the backend id exactly as it appears in the catalog — there is no separate +handler name. Ids that share a suite have their own case in the dispatcher: +`process-t1` and `process-t3` both run `WinProcessContainer-Tests.ps1`, which +determines the tier it expects from the host's own `wxc-exec --probe`. + +```powershell +tests\scripts\run_ci_backend_tests.ps1 -Backend process-t1 ` + -BinaryDirectory -Architecture x64 +``` + +```bash +tests/scripts/run_ci_backend_tests.sh bubblewrap +``` + +A backend with no wired suite exits non-zero on purpose, so accidentally +enabling it in a trigger fails loudly instead of reporting a false success. + +To see exactly what a plan would schedule without pushing: + +```bash +node scripts/ci/resolve-validation-test-matrix.mjs --plan nightly +``` -**Skip semantics for the IsolationSession suites.** Availability is decided by a -single `wxc-exec --probe` call reading `probes.isolationSessionAvailable`, which -covers both a host that cannot activate the API and a binary built without -`--features isolation_session`. When unavailable the suite prints `SKIPPED` and -exits 0, so running on an unsupported host degrades gracefully rather than -reporting failures. +**Skip semantics.** Several suites degrade gracefully on an unsupported host: +the IsolationSession suites decide availability from a single `wxc-exec --probe` +read of `probes.isolationSessionAvailable` (covering both a host that cannot +activate the API and a binary built without `--features isolation_session`), +print `SKIPPED`, and exit 0. -Because a skip exits 0, **a caller that only reads the exit code cannot tell a -skipped suite from a passing one.** Any automated runner that treats these -suites as validation evidence must therefore check the `SKIPPED` line or the -executed count, not just the exit status. Independently, a run that reaches the -summary having executed zero tests always fails, since it substantiates nothing. +Because a skip exits 0 and the dispatchers propagate only the exit code, **a +green CI job does not by itself prove the suite ran.** Anything treating these +suites as validation evidence must check the `SKIPPED` line or the executed +count, not just the exit status — the matrix entry says the host is expected to +support the backend, so a silent skip there is a gap in coverage rather than a +graceful degradation. Independently, a run that reaches the summary having +executed zero tests always fails, since it substantiates nothing. ### Manual smoke tests @@ -80,9 +139,11 @@ itself and takes a `-ComputerName` / `-VMName` plus `-Credential`. | `push_batch_and_config_files_to_vm.ps1` | `tests\configs\`, `examples\`, runner batch files, helper scripts | TShell (active `Open-Device` session) | | `push_sdk_integration_tests_to_vm.ps1` | SDK integration test artifacts (`sdk\bin\x64`, compiled tests, `node_modules`, `package.json`, `run-tests.js`) | PowerShell Remoting (`-ComputerName`/`-VMName` + `-Credential`) | -CI currently runs the MicroVM Rust E2E suite when WHP is available. Other -executor E2E tests are local/prerequisite-gated and should be run on machines -with the required Windows features and binaries. +Backend E2E coverage runs on a schedule (not on PRs) through the validation +matrix described under [CI dispatch](#ci-dispatch), against binaries downloaded +from the build artifacts. Suites whose backend is not yet wired into a trigger — +and any test needing a Windows feature or hardware the pool images lack — remain +local/prerequisite-gated and should be run on a machine that has them. ## Test ownership diff --git a/tests/scripts/run_ci_backend_tests.ps1 b/tests/scripts/run_ci_backend_tests.ps1 new file mode 100644 index 000000000..adcb916a9 --- /dev/null +++ b/tests/scripts/run_ci_backend_tests.ps1 @@ -0,0 +1,141 @@ +<# +.SYNOPSIS +Runs a Windows backend test from a downloaded CI artifact. + +.DESCRIPTION +Takes the matrix backend id straight from the catalog, so there is no +id-to-command mapping to keep in sync. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory)] + [ValidateSet( + 'process-t1', + 'process-t3', + 'isolation-session', + 'windows-sandbox', + 'wslc', + 'microvm', + 'hyperlight' + )] + [string]$Backend, + + [Parameter(Mandatory)] + [string]$BinaryDirectory, + + [Parameter(Mandatory)] + [ValidateSet('x64', 'arm64')] + [string]$Architecture +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +$scriptRoot = Split-Path -Parent $MyInvocation.MyCommand.Path +$binaryDirectoryPath = (Resolve-Path -LiteralPath $BinaryDirectory).Path +$wxc = Join-Path $binaryDirectoryPath 'wxc-exec.exe' + +function Assert-File { + param([Parameter(Mandatory)][string]$Path) + + if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) { + throw "Required CI artifact file is missing: $Path" + } +} + +function Invoke-TestScript { + param( + [Parameter(Mandatory)][string]$Path, + # Splat a hashtable, not an array. Array splatting binds elements + # positionally, so '-BinDir' would be passed as the first positional + # value rather than naming the parameter. + [hashtable]$Arguments = @{} + ) + + # PowerShell scripts do not always replace a previous native exit code. + # Reset it so a successful script cannot inherit a stale failure. + $global:LASTEXITCODE = 0 + & $Path @Arguments + if ($LASTEXITCODE -ne 0) { + throw "Backend test failed with exit code $LASTEXITCODE`: $Path" + } +} + +Assert-File -Path $wxc + +function Invoke-ProcessContainerTests { + # The existing harness expects separate debug and release layouts. CI + # intentionally tests one release artifact, so stage it in both slots. + $debugDirectory = Join-Path $binaryDirectoryPath 'debug' + $releaseDirectory = Join-Path $binaryDirectoryPath 'release' + New-Item -ItemType Directory -Force -Path $debugDirectory, $releaseDirectory | Out-Null + Copy-Item -LiteralPath $wxc -Destination (Join-Path $debugDirectory 'wxc-exec.exe') -Force + Copy-Item -LiteralPath $wxc -Destination (Join-Path $releaseDirectory 'wxc-exec.exe') -Force + + $uiProbe = Join-Path $binaryDirectoryPath 'wxc-ui-probe.exe' + Assert-File -Path $uiProbe + Copy-Item -LiteralPath $uiProbe -Destination (Join-Path $debugDirectory 'wxc-ui-probe.exe') -Force + Copy-Item -LiteralPath $uiProbe -Destination (Join-Path $releaseDirectory 'wxc-ui-probe.exe') -Force + + $script = Join-Path $scriptRoot 'WinProcessContainer-Tests.ps1' + # Skip build and Cargo phases because this job consumes a previously + # built artifact; retain the host and containment behavior phases. + $phases = @( + 'Probes', + 'T3Forced', + 'T1DenyForced', + 'UiMitigationMatrix', + 'GlobalAtomIsolation', + 'DaclDisabled', + 'CrashRecovery' + ) + $global:LASTEXITCODE = 0 + & $script ` + -SkipBuild ` + -SkipReleaseLane ` + -WxcDebug (Join-Path $debugDirectory 'wxc-exec.exe') ` + -WxcRelease (Join-Path $releaseDirectory 'wxc-exec.exe') ` + -UiProbeDebug (Join-Path $debugDirectory 'wxc-ui-probe.exe') ` + -UiProbeRelease (Join-Path $releaseDirectory 'wxc-ui-probe.exe') ` + -Phases $phases + if ($LASTEXITCODE -ne 0) { + throw "Process Container tests failed with exit code $LASTEXITCODE." + } +} + +switch ($Backend) { + 'process-t1' { + Invoke-ProcessContainerTests + } + 'process-t3' { + Invoke-ProcessContainerTests + } + 'isolation-session' { + Invoke-TestScript -Path (Join-Path $scriptRoot 'run_isolation_session_tests.ps1') -Arguments @{ + WxcExePath = $wxc + } + } + 'windows-sandbox' { + Invoke-TestScript -Path (Join-Path $scriptRoot 'run_windows_sandbox_one_shot_tests.ps1') -Arguments @{ + BinDir = $binaryDirectoryPath + } + } + 'wslc' { + # The current WSLC helper hardcodes the x64 target when locating assets. + if ($Architecture -ne 'x64') { + throw 'The existing WSLC test harness is not architecture-portable yet.' + } + Invoke-TestScript -Path (Join-Path $scriptRoot 'run_wslc_all_tests.ps1') -Arguments @{ + WxcExecPath = $wxc + } + } + 'microvm' { + Invoke-TestScript -Path (Join-Path $scriptRoot 'run_microvm_tests.ps1') -Arguments @{ + BinDir = $binaryDirectoryPath + } + } + 'hyperlight' { + # Keep unwired backends explicit so accidental activation fails loudly. + throw 'The Hyperlight CI backend is not wired to an existing test entry point yet.' + } +} diff --git a/tests/scripts/run_ci_backend_tests.sh b/tests/scripts/run_ci_backend_tests.sh new file mode 100644 index 000000000..dd03f1245 --- /dev/null +++ b/tests/scripts/run_ci_backend_tests.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Dispatches a downloaded Unix artifact to the repository's existing backend +# test suites, keyed by the matrix backend id. Unsupported backends fail +# explicitly rather than reporting a false-success placeholder job. + +usage() { + echo "Usage: $0 " >&2 +} + +if [[ $# -ne 2 ]]; then + usage + exit 2 +fi + +backend="$1" +binary_directory="$(cd "$2" && pwd)" +script_root="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +repo_root="$(cd "$script_root/../.." && pwd)" +release_directory="$repo_root/src/target/release" + +case "$backend" in + microvm) + # Keep unwired commands explicit so accidental activation fails loudly. + # Future test script: run_microvm_tests.sh + echo "The MicroVM CI backend is not wired to an artifact-only Linux test entry point yet." >&2 + exit 2 + ;; + hyperlight) + # Keep unwired commands explicit so accidental activation fails loudly. + # Future test script: run_hyperlight_tests.sh + echo "The Hyperlight CI backend is not wired to an existing test entry point yet." >&2 + exit 2 + ;; + bubblewrap) + # Existing Linux shell tests locate binaries under src/target/release. + test -x "$binary_directory/lxc-exec" + test -f "$binary_directory/unix-test-proxy" + mkdir -p "$release_directory" + cp -a "$binary_directory/." "$release_directory/" + chmod +x "$release_directory/lxc-exec" "$release_directory/unix-test-proxy" + bash "$script_root/run_bwrap_all_tests.sh" + ;; + lxc) + test -x "$binary_directory/lxc-exec" + test -f "$binary_directory/unix-test-proxy" + mkdir -p "$release_directory" + cp -a "$binary_directory/." "$release_directory/" + chmod +x "$release_directory/lxc-exec" "$release_directory/unix-test-proxy" + MXC_LXC_TESTS_REQUIRE_EXECUTION=1 bash "$script_root/run_lxc_all_tests.sh" + ;; + seatbelt) + test -x "$binary_directory/mxc-exec-mac" + test -x "$binary_directory/unix-test-proxy" + echo "The Seatbelt CI backend is not wired to an existing test entry point yet." >&2 + exit 2 + ;; + *) + usage + exit 2 + ;; +esac diff --git a/tests/scripts/run_lxc_network_test.sh b/tests/scripts/run_lxc_network_test.sh index c2d3f93ea..751b07964 100644 --- a/tests/scripts/run_lxc_network_test.sh +++ b/tests/scripts/run_lxc_network_test.sh @@ -15,6 +15,5 @@ if [ ! -f "$LXC_EXEC" ]; then exit 1 fi -echo "Running LXC network test..." "$LXC_EXEC" "$REPO_DIR/tests/configs/lxc_network_test.json" echo "LXC network test complete." diff --git a/tests/scripts/run_microvm_tests.ps1 b/tests/scripts/run_microvm_tests.ps1 index 1b32c94ee..dcf6002b5 100644 --- a/tests/scripts/run_microvm_tests.ps1 +++ b/tests/scripts/run_microvm_tests.ps1 @@ -88,7 +88,14 @@ $wxcExe = Resolve-Path $WxcExePath # -- Verify MicroVM binaries -------------------------------------------------- -$requiredBinaries = @("nanvixd.exe", "kernel.elf", "python3.12", "nanvix_rootfs.img") +$requiredBinaries = @( + "nanvixd.exe", + "nanvix_rootfs.img", + "python3.initrd", + "bin\kernel.elf", + "snapshots\kernel.vmem", + "snapshots\kernel.whp.cbor" +) $binDir = Split-Path $wxcExe $missing = $requiredBinaries | Where-Object { -not (Test-Path (Join-Path $binDir $_)) } @@ -153,8 +160,10 @@ foreach ($test in $tests) { $reason = "expected exit=$expectedExit, got exit=$actualExit" } - # Check stdout content if OutputContains is specified - if ($pass -and $test.OutputContains) { + # Check stdout content if OutputContains is specified. Not every test + # defines the key, and StrictMode makes a missing hashtable key throw, so + # test for its presence rather than accessing it directly. + if ($pass -and $test.ContainsKey('OutputContains')) { $combined = "$stdout`n$stderr" if ($combined -notmatch [regex]::Escape($test.OutputContains)) { $pass = $false diff --git a/tests/scripts/run_wslc_all_tests.ps1 b/tests/scripts/run_wslc_all_tests.ps1 index 9c2a6cb49..694bdd0e0 100644 --- a/tests/scripts/run_wslc_all_tests.ps1 +++ b/tests/scripts/run_wslc_all_tests.ps1 @@ -97,6 +97,17 @@ if (-not $SkipSetup) { } } +# Helper: StrictMode-safe property read; returns $null when the property (or the +# object) is absent. Lets the optional-config-field reads below work under the +# Set-StrictMode -Version Latest that run_ci_backend_tests.ps1 imposes. +function Get-JsonProperty { + param($Object, [Parameter(Mandatory)][string]$Name) + if ($null -eq $Object) { return $null } + $prop = $Object.PSObject.Properties[$Name] + if ($null -eq $prop) { return $null } + return $prop.Value +} + # Helper: run a single WSLC test config function Run-WslcTest { param( @@ -114,9 +125,12 @@ function Run-WslcTest { return @{ Name = $ConfigFile; Pass = $true; Skipped = $true; Reason = "File not found" } } - # Skip if the config references a tar file that doesn't exist locally + # Skip if the config references a tar file that doesn't exist locally. + # Read the chain defensively: this suite inherits Set-StrictMode -Version + # Latest from run_ci_backend_tests.ps1, under which touching a missing + # property is a terminating error, and most configs have no wslc.imageTarPath. $configJson = Get-Content $configPath -Raw | ConvertFrom-Json - $tarPath = $configJson.experimental.wslc.imageTarPath + $tarPath = Get-JsonProperty (Get-JsonProperty (Get-JsonProperty $configJson 'experimental') 'wslc') 'imageTarPath' if ($tarPath -and -not (Test-Path $tarPath)) { Write-Host " $ConfigFile ... " -NoNewline Write-Host "SKIP (tar not found: $tarPath)" -ForegroundColor Yellow @@ -169,7 +183,7 @@ function Run-WslcTest { # PostExitCheck runs after exit/output gates pass. Receives ($id, $output) # and must return truthy. Use for externally-observable state assertions. if ($pass -and $PostExitCheck) { - $containerId = $configJson.containerId + $containerId = Get-JsonProperty $configJson 'containerId' try { $checkResult = & $PostExitCheck $containerId $output if (-not $checkResult) {