[experiment] ci: try 1es-ubuntu-24.04-min for Linux jobs instead of disk cleanup - #893
Closed
Ahmed Muhsin (ahmedmuhsin) wants to merge 5 commits into
Closed
[experiment] ci: try 1es-ubuntu-24.04-min for Linux jobs instead of disk cleanup#893Ahmed Muhsin (ahmedmuhsin) wants to merge 5 commits into
Ahmed Muhsin (ahmedmuhsin) wants to merge 5 commits into
Conversation
added 5 commits
August 13, 2026 16:14
EXPERIMENT (do not merge as-is). The 1es-ubuntu-22.04 image boots ~92% full on /, which forced the 'Free up disk space' rm -rf/prune steps added in #883. Try the leaner 1es-ubuntu-24.04-min image instead and drop the cleanup, keeping a df/du report so we can see the new baseline headroom.
First run died at 'Check Maven is installed' (exit 127) so only the mvn gap is known. Replace the disk-only report with a probe that reports every tool these jobs need (mvn, java/javac, docker + daemon, python3, node/npm, dotnet) plus the tool cache and JDK dirs, so one run reveals the full gap. Always exits 0.
The -min image has the headroom (62G free vs 6.5G on 22.04) but ships no mvn/java/node/npm and an empty /usr/lib/jvm, so JavaToolInstaller PreInstalled has nothing to find. Try the non-min 24.04 image, which should carry the full toolchain and may still be leaner than 22.04. Probe step reports the result.
Plain 1es-ubuntu-24.04 does not exist in the pool, so -min is the only leaner option. Scope it to the emulated job, which is where the disk exhaustion actually killed agents. -min starts at 15% used (62G free) vs 22.04's 92% (6.5G), so the rm -rf/prune cleanup is no longer needed here. -min ships no Maven and no Node, so install both explicitly. The PreInstalled JDK step is dropped: /usr/lib/jvm is empty on -min, and it only set a default before the matrix JDK is installed from JDK_DOWNLOAD_LINK a few steps later. Docker integration tests stay on 1es-ubuntu-22.04 with the existing cleanup: that job resolves JDKs 8/11/17/21 via PreInstalled and uses UsePythonVersion, which -min cannot satisfy without a much larger change.
The build script failed with '/opt/hostedtoolcache/NuGet/7.9.0/x64/nuget: 2: mono: not found' — the nuget CLI is a .NET Framework exe needing mono, which 1es-ubuntu-24.04-min does not ship. The emulated tests never consume the .nupkg: setup-tests-pipeline.ps1 copies the worker from target/ and only build-artifacts.yml publishes packages. So pass -skipNuget (as the docker test job already does) instead of installing mono, and drop the now-unused NuGetToolInstaller step.
Contributor
Author
|
Superseded by #895, which now carries this change together with the Docker Hub packaging fix. The emulated Linux job on 1es-ubuntu-24.04-min is unchanged from the version validated here (all 5 legs green, 62G free with no cleanup step); the docker legs that were red on this PR were the pre-existing dev breakage that #895 fixes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment — not for merge as-is. Opened as a draft to get a CI signal.
What
Switch the two Linux CI jobs (emulated tests, docker integration tests) from
1es-ubuntu-22.04to1es-ubuntu-24.04-min, and drop theFree up disk spacecleanup steps that #883 added. Adf/dureport step is kept so the run tells us the new image's baseline headroom.Why
The
1es-ubuntu-22.04image boots ~92% full on/(67G used of 73G), almost entirely from toolsets these jobs never touch — Android SDK and Haskell under/usr/local(~24G), pre-cached docker images (~12G), Swift/miniconda under/usr/share(~9.5G), CodeQL/PyPy in the tool cache (~6.7G). That left so little headroom thatdotnet testfilled the disk, the agent process aborted with exit 134, and ADO reported the job as canceled.#883 worked around it by
rm -rf-ing those toolsets at job start (/92% → 54%, ~28G freed), which has held: 17 of 18 nightlydevbuilds since have passed with zero cancellations.That workaround is still deleting things out from under a hosted image, which is fragile — it silently breaks if paths move in a future image. If the
-minimage ships lean enough, we can drop the cleanup entirely and rely on the image instead.What to look for
-minimages carry fewer preinstalled toolsets, so the likely failure mode is a missing dependency (Maven, preinstalled JDKs forJavaToolInstaller, docker, Python) rather than disk.Report disk spacestep output — how much free space does-minactually start with?If it provisions cleanly with comfortable headroom, the follow-up is to drop the cleanup steps for good. If it needs extra tool installs, we compare that cost against keeping the
rm -rf.Scope
CI only, Linux only. Untouched: the Windows jobs,
java-version-check.yml, and the officialrun-e2e-tests-linux.yml— those stay on1es-ubuntu-22.04until this experiment reports back.