Skip to content

fix(ci): unbreak Linux agents (no Docker Hub, emulated tests on 24.04-min) - #895

Open
Ahmed Muhsin (ahmedmuhsin) wants to merge 6 commits into
devfrom
fix/squashfs-no-dockerhub
Open

fix(ci): unbreak Linux agents (no Docker Hub, emulated tests on 24.04-min)#895
Ahmed Muhsin (ahmedmuhsin) wants to merge 6 commits into
devfrom
fix/squashfs-no-dockerhub

Conversation

@ahmedmuhsin

@ahmedmuhsin Ahmed Muhsin (ahmedmuhsin) commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Fixes for the Linux CI agents, plus a related build-script cleanup. Supersedes #893.

1. Test apps can no longer be packaged via Docker Hub

The Docker integration tests have failed on dev every night since 2026-08-15 (builds 298300, 298418, 298540, 298743). All four legs fail at Build and package test apps:

Unable to find image 'ubuntu:22.04' locally
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/":
        context deadline exceeded (Client.Timeout exceeded while awaiting headers)

No code change caused this: dev has not moved since 08-12, and 08-11 through 08-14 were green. The agents can no longer pull from Docker Hub.

build-apps.ps1 pulled ubuntu:22.04 only to get mksquashfs, then ran apt-get install squashfs-tools inside that container — two dependencies on public network access.

Fix: run mksquashfs on the agent and install squashfs-tools in the job, which removes the image pull and the in-container apt-get together. The Docker path stays as a fallback for machines without squashfs-tools (Windows dev boxes) and now points at the MCR mirror. Everything else under dockertests/ already used MCR (azure-storage/azurite, azure-functions/mesh); ubuntu:22.04 was the last Docker Hub reference in the suite.

2. Emulated Linux tests move to 1es-ubuntu-24.04-min

1es-ubuntu-22.04 boots ~92% full on / (67G used of 73G), almost all of it toolsets these jobs never touch: Android SDK and Haskell under /usr/local (~24G), /var/lib including cached docker images (~12G), Swift and miniconda under /usr/share (~9.5G), CodeQL and PyPy in the tool cache (~6.7G). With that little headroom dotnet test filled the disk, the agent aborted with exit 134, and ADO reported the job as cancelled rather than failed.

#883 worked around it by rm -rf-ing those toolsets at job start. That held, but it means deleting things out from under a hosted image, which breaks silently whenever paths move.

1es-ubuntu-24.04-min starts at 15% used (62G free) — more headroom than the cleanup produced (34G), with nothing to delete. So the cleanup step is gone here.

Adapting to the leaner image:

Gap on -min Handling
no Maven apt-get install -y maven
no Node/npm (needed for npm install -g azurite) NodeTool@0
/usr/lib/jvm empty, so JavaToolInstaller PreInstalled finds nothing dropped that step; the matrix JDK is downloaded a few steps later anyway
no mono, so the nuget CLI cannot run -skipNuget; see below

Anything added to this job from now on has to install what it needs — that is the tradeoff for the headroom.

Docker integration tests stay on 1es-ubuntu-22.04. With the cleanup step they no longer report low-disk warnings, so they are not short of space, and moving them would mean replacing four PreInstalled JDK installs and UsePythonVersion for no current benefit.

Plain 1es-ubuntu-24.04 is not an option — it does not exist in the pool (Image 1es-ubuntu-24.04 doesn't exist in pool 1es-pool-azfunc-public).

3. Emulated tests stop building a NuGet package

Both emulated jobs ran package-pipeline.ps1 without -skipNuget, building a .nupkg neither of them reads: setup-tests-pipeline.ps1 copies the worker out of target/, and only build-artifacts.yml publishes packages.

On Linux this became a hard failure, because the nuget CLI needs mono and -min does not ship it. On Windows it was only wasted work. Both now pass -skipNuget and drop the NuGet tool installer that existed to serve the pack step; Authenticate NuGet to CFS and the dotnet test restore do not depend on it.

4. Remove installMavenPluginLocally.ps1

The script cloned https://github.com/ahmedmuhsin/azure-maven-plugins (a personal fork) at branch sdk-types and ran mvn clean install, publishing the result into the local ~/.m2 repository that later worker builds resolve from. That puts untrusted build code into the build environment (CWE-829).

Nothing references it — no pipeline, script, or doc mentions installMavenPluginLocally — so it is not reachable from CI and removing it changes no build behaviour. The sibling installAdditionsLocally.ps1 shows the intended convention, cloning the official Azure/azure-functions-java-additions.

Validation

  • All 5 emulated Linux legs pass on -min (9.8–13.7 min), disk steady at 62G free with no cleanup step.
  • All 4 Docker integration legs pass again with the squashfs fix, after four consecutive nightly failures on dev.
  • Both official-build.yml and public-build.yml consume these job templates, so the fixes apply to both pipelines.

CI and local build scripts only, no product change.

Follow-up (not in this PR)

integration-tests.yml still runs its E2E Linux job on 1es-ubuntu-22.04 and still packs a NuGet package. It is unaffected by either problem here — it never invokes docker, and mono is present on 22.04 — so it is left alone. Moving it to -min would need the same Maven, Node, and -skipNuget treatment.

Docker integration tests have failed on dev every night since 2026-08-15 at 'Build and package test apps': docker: Get https://registry-1.docker.io/v2/: context deadline exceeded. The agents can no longer reach Docker Hub, and build-apps.ps1 pulled ubuntu:22.04 purely to run mksquashfs. No code change caused this; dev has not moved since 08-12.

Run mksquashfs on the agent instead, and install squashfs-tools in the job. This also drops the apt-get that ran inside the container. The Docker path is kept as a fallback for machines without squashfs-tools (e.g. Windows dev boxes) and now points at the MCR mirror rather than Docker Hub. Everything else in dockertests already used MCR (azurite, mesh); ubuntu:22.04 was the only Docker Hub dependency.
The 22.04 image boots ~92% full on / (6.5G free), which is why dotnet test kept filling the disk and killing agents (exit 134, reported as a cancelled job). The workaround was to rm -rf Android/Haskell/Swift/CodeQL at job start. 1es-ubuntu-24.04-min starts at 15% used (62G free), so that cleanup is no longer needed here.

-min ships no Maven and no Node, so install both. The PreInstalled JDK step is dropped (/usr/lib/jvm is empty on -min, and the matrix JDK is downloaded a few steps later anyway), and packaging runs with -skipNuget because the nuget CLI needs mono: these tests read the worker from target/, never the .nupkg.

Docker integration tests stay on 22.04: that job resolves JDKs 8/11/17/21 via PreInstalled and uses UsePythonVersion, which -min cannot satisfy without a much larger change.
@ahmedmuhsin Ahmed Muhsin (ahmedmuhsin) changed the title fix(ci): package test apps without pulling from Docker Hub fix(ci): unbreak Linux agents (no Docker Hub, emulated tests on 24.04-min) Aug 19, 2026
AzureFunctionsJava added 4 commits August 19, 2026 17:43
The script cloned https://github.com/ahmedmuhsin/azure-maven-plugins (a personal fork) at branch sdk-types and ran mvn clean install, publishing the result into the local ~/.m2 repository that subsequent worker builds resolve from. That is untrusted build code landing in the build environment (CWE-829).

Nothing references it: no pipeline, script, or doc mentions installMavenPluginLocally, so it is not reachable from CI and removing it changes no build behaviour. The sibling installAdditionsLocally.ps1 shows the intended convention, cloning the official Azure/azure-functions-java-additions repo.
Same reasoning as the Linux job: setup-tests-pipeline.ps1 copies the worker from target/, so the .nupkg these tests build is never read. Only build-artifacts.yml publishes packages. Drops the NuGet tool installer that existed to serve the pack step; 'Authenticate NuGet to CFS' and the dotnet test restore do not depend on it, as the Linux job already demonstrates.
Removes the last disk-cleanup step. On the docker legs that step cost 75-119s each (~100s avg, roughly 12% of a 10-16 min leg, ~6.7 min of agent time per build) purely to make room on an image that ships 67G of toolsets these tests never use. -min starts at 15% used, so nothing needs deleting.

Adapting to the leaner image: install Maven, squashfs-tools and python3-venv up front; download the matrix JDK and install it with JavaToolInstaller LocalDirectory, since /usr/lib/jvm is empty on -min and PreInstalled resolves nothing; and replace UsePythonVersion (no Python in the -min tool cache) with the system python3, which satisfies the test kit's requires-python >= 3.8. pip installs into a venv because the system interpreter on 24.04 is externally managed (PEP 668). The docker daemon is present on -min, so the tests themselves are unaffected.
Temurin for 8, Microsoft OpenJDK from 11 up, matching production and the emulated jobs. Worth recording because the previous PreInstalled lookup read JAVA_HOME_<n>_X64 from the agent image, which supplied Temurin for every version, so the 11+ legs had quietly diverged from what we ship.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant