fix(ci): unbreak Linux agents (no Docker Hub, emulated tests on 24.04-min) - #895
Open
Ahmed Muhsin (ahmedmuhsin) wants to merge 6 commits into
Open
fix(ci): unbreak Linux agents (no Docker Hub, emulated tests on 24.04-min)#895Ahmed Muhsin (ahmedmuhsin) wants to merge 6 commits into
Ahmed Muhsin (ahmedmuhsin) wants to merge 6 commits into
Conversation
Docker integration tests have failed on dev every night since 2026-08-15 at 'Build and package test apps': docker: Get https://registry-1.docker.io/v2/: context deadline exceeded. The agents can no longer reach Docker Hub, and build-apps.ps1 pulled ubuntu:22.04 purely to run mksquashfs. No code change caused this; dev has not moved since 08-12. Run mksquashfs on the agent instead, and install squashfs-tools in the job. This also drops the apt-get that ran inside the container. The Docker path is kept as a fallback for machines without squashfs-tools (e.g. Windows dev boxes) and now points at the MCR mirror rather than Docker Hub. Everything else in dockertests already used MCR (azurite, mesh); ubuntu:22.04 was the only Docker Hub dependency.
The 22.04 image boots ~92% full on / (6.5G free), which is why dotnet test kept filling the disk and killing agents (exit 134, reported as a cancelled job). The workaround was to rm -rf Android/Haskell/Swift/CodeQL at job start. 1es-ubuntu-24.04-min starts at 15% used (62G free), so that cleanup is no longer needed here. -min ships no Maven and no Node, so install both. The PreInstalled JDK step is dropped (/usr/lib/jvm is empty on -min, and the matrix JDK is downloaded a few steps later anyway), and packaging runs with -skipNuget because the nuget CLI needs mono: these tests read the worker from target/, never the .nupkg. Docker integration tests stay on 22.04: that job resolves JDKs 8/11/17/21 via PreInstalled and uses UsePythonVersion, which -min cannot satisfy without a much larger change.
added 4 commits
August 19, 2026 17:43
The script cloned https://github.com/ahmedmuhsin/azure-maven-plugins (a personal fork) at branch sdk-types and ran mvn clean install, publishing the result into the local ~/.m2 repository that subsequent worker builds resolve from. That is untrusted build code landing in the build environment (CWE-829). Nothing references it: no pipeline, script, or doc mentions installMavenPluginLocally, so it is not reachable from CI and removing it changes no build behaviour. The sibling installAdditionsLocally.ps1 shows the intended convention, cloning the official Azure/azure-functions-java-additions repo.
Same reasoning as the Linux job: setup-tests-pipeline.ps1 copies the worker from target/, so the .nupkg these tests build is never read. Only build-artifacts.yml publishes packages. Drops the NuGet tool installer that existed to serve the pack step; 'Authenticate NuGet to CFS' and the dotnet test restore do not depend on it, as the Linux job already demonstrates.
Removes the last disk-cleanup step. On the docker legs that step cost 75-119s each (~100s avg, roughly 12% of a 10-16 min leg, ~6.7 min of agent time per build) purely to make room on an image that ships 67G of toolsets these tests never use. -min starts at 15% used, so nothing needs deleting. Adapting to the leaner image: install Maven, squashfs-tools and python3-venv up front; download the matrix JDK and install it with JavaToolInstaller LocalDirectory, since /usr/lib/jvm is empty on -min and PreInstalled resolves nothing; and replace UsePythonVersion (no Python in the -min tool cache) with the system python3, which satisfies the test kit's requires-python >= 3.8. pip installs into a venv because the system interpreter on 24.04 is externally managed (PEP 668). The docker daemon is present on -min, so the tests themselves are unaffected.
Temurin for 8, Microsoft OpenJDK from 11 up, matching production and the emulated jobs. Worth recording because the previous PreInstalled lookup read JAVA_HOME_<n>_X64 from the agent image, which supplied Temurin for every version, so the 11+ legs had quietly diverged from what we ship.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes for the Linux CI agents, plus a related build-script cleanup. Supersedes #893.
1. Test apps can no longer be packaged via Docker Hub
The Docker integration tests have failed on
devevery night since 2026-08-15 (builds 298300, 298418, 298540, 298743). All four legs fail at Build and package test apps:No code change caused this:
devhas not moved since 08-12, and 08-11 through 08-14 were green. The agents can no longer pull from Docker Hub.build-apps.ps1pulledubuntu:22.04only to getmksquashfs, then ranapt-get install squashfs-toolsinside that container — two dependencies on public network access.Fix: run
mksquashfson the agent and installsquashfs-toolsin the job, which removes the image pull and the in-containerapt-gettogether. The Docker path stays as a fallback for machines withoutsquashfs-tools(Windows dev boxes) and now points at the MCR mirror. Everything else underdockertests/already used MCR (azure-storage/azurite,azure-functions/mesh);ubuntu:22.04was the last Docker Hub reference in the suite.2. Emulated Linux tests move to
1es-ubuntu-24.04-min1es-ubuntu-22.04boots ~92% full on/(67G used of 73G), almost all of it toolsets these jobs never touch: Android SDK and Haskell under/usr/local(~24G),/var/libincluding cached docker images (~12G), Swift and miniconda under/usr/share(~9.5G), CodeQL and PyPy in the tool cache (~6.7G). With that little headroomdotnet testfilled the disk, the agent aborted with exit 134, and ADO reported the job as cancelled rather than failed.#883 worked around it by
rm -rf-ing those toolsets at job start. That held, but it means deleting things out from under a hosted image, which breaks silently whenever paths move.1es-ubuntu-24.04-minstarts at 15% used (62G free) — more headroom than the cleanup produced (34G), with nothing to delete. So the cleanup step is gone here.Adapting to the leaner image:
-minapt-get install -y mavennpm install -g azurite)NodeTool@0/usr/lib/jvmempty, soJavaToolInstallerPreInstalledfinds nothingnugetCLI cannot run-skipNuget; see belowAnything added to this job from now on has to install what it needs — that is the tradeoff for the headroom.
Docker integration tests stay on
1es-ubuntu-22.04. With the cleanup step they no longer report low-disk warnings, so they are not short of space, and moving them would mean replacing fourPreInstalledJDK installs andUsePythonVersionfor no current benefit.Plain
1es-ubuntu-24.04is not an option — it does not exist in the pool (Image 1es-ubuntu-24.04 doesn't exist in pool 1es-pool-azfunc-public).3. Emulated tests stop building a NuGet package
Both emulated jobs ran
package-pipeline.ps1without-skipNuget, building a.nupkgneither of them reads:setup-tests-pipeline.ps1copies the worker out oftarget/, and onlybuild-artifacts.ymlpublishes packages.On Linux this became a hard failure, because the
nugetCLI needs mono and-mindoes not ship it. On Windows it was only wasted work. Both now pass-skipNugetand drop the NuGet tool installer that existed to serve the pack step;Authenticate NuGet to CFSand thedotnet testrestore do not depend on it.4. Remove
installMavenPluginLocally.ps1The script cloned
https://github.com/ahmedmuhsin/azure-maven-plugins(a personal fork) at branchsdk-typesand ranmvn clean install, publishing the result into the local~/.m2repository that later worker builds resolve from. That puts untrusted build code into the build environment (CWE-829).Nothing references it — no pipeline, script, or doc mentions
installMavenPluginLocally— so it is not reachable from CI and removing it changes no build behaviour. The siblinginstallAdditionsLocally.ps1shows the intended convention, cloning the officialAzure/azure-functions-java-additions.Validation
-min(9.8–13.7 min), disk steady at 62G free with no cleanup step.dev.official-build.ymlandpublic-build.ymlconsume these job templates, so the fixes apply to both pipelines.CI and local build scripts only, no product change.
Follow-up (not in this PR)
integration-tests.ymlstill runs its E2E Linux job on1es-ubuntu-22.04and still packs a NuGet package. It is unaffected by either problem here — it never invokes docker, and mono is present on 22.04 — so it is left alone. Moving it to-minwould need the same Maven, Node, and-skipNugettreatment.