Skip to content

[deckhouse-cli] fix mirror pull silently loses image on fail#393

Open
Glitchy-Sheep wants to merge 3 commits into
mainfrom
fix/mirror-pull-lost-image
Open

[deckhouse-cli] fix mirror pull silently loses image on fail#393
Glitchy-Sheep wants to merge 3 commits into
mainfrom
fix/mirror-pull-lost-image

Conversation

@Glitchy-Sheep

Copy link
Copy Markdown
Contributor

[mirror] Fix silent image loss on pull retry after failed write

Summary

Network failure during mirror pull could cause a silent drop of an image from bundle.
But summary of the pull reports the success.

A cluster therefore could have ImagePullBackOff on the missing digest.

Problem

ImageLayout.AddImage recorded metaByTag[tag] before AppendImage wrote
the image.

When the blob download broke during write, the (tag, digest) pair
stayed in the map, so the retry hit the idempotency guard and returned
success without writing anything
:

[92 / 363] Pulling ...@sha256:ae6870...  failed, next retry in 10s (but it won't actually retry)
[92 / 363] Pulling ...@sha256:ae6870...   <- "succeeds" in 151 ms, writes nothing

Fix

  • Record metaByTag only after AppendImage succeeds: a failed write leaves
    no trace and the retry performs a real write.
  • Defense in depth: after PullImageSet, verify every planned image is
    present in the layout. An incomplete bundle now fails the pull with the
    list of missing images instead of exiting 0.

Before / After

Before: the pull looks successful, the loss is invisible:

[92 / 363] Pulling ee@sha256:ae6870...  failed, next retry in 10s
[92 / 363] Pulling ee@sha256:ae6870...          <- "success" in 151 ms
...
╔══ Pull summary ═══════════════════════
║ Platform:   v1.76.2 (2 channels)
║   platform.tar                11.6 GiB
╚═══════════════════════════════════════   exit code 0

The mismatch surfaces only as [362 / 362] Pushing ... vs 363 pulled, and days
later in the air-gapped cluster: 404 error and ImagePullBackoff.

After: the same network blip heals itself - the retry re-downloads the
image and the bundle is complete:

[92 / 363] Pulling ee@sha256:ae6870...  failed, next retry in 10s
[92 / 363] Pulling ee@sha256:ae6870...          <- real re-download
...
╚═══════════════════════════════════════   exit code 0, 363 images in bundle

If an image still fails to land for any reason, the pull fails on the spot,
while the user still has network access and can simply re-run it:

ERROR ... pulled layout is incomplete: 1 of 363 planned images missing:
registry.deckhouse.ru/deckhouse/ee@sha256:ae6870...
                                                exit code 1

Tests

  • TestRepro_AddImage_RetryAfterFailedWrite_ImageMustLand - unit: failed
    write + retry must land the image (red before the fix).
  • TestRepro_AddImage_NoFailure_Control - guard still dedups repeated adds.
  • TestRepro_MirrorPullThenPush_TransientBlobFailureLosesImage - e2e: pull
    through a registry that resets one blob download, push, verify every digest
    resolves in the target registry.
  • TestVerifyPlannedImagesLanded_ReportsMissingImages - the completeness
    check reports exactly the missing references.
  • TestPullImageSet_CompletePullPassesVerification - a healthy pull of
    digest- and tag-style references does not trip the check.

… retry

AddImage recorded metaByTag before AppendImage. A transient network
failure during blob download left the (tag, digest) pair in the map,
so the idempotency guard treated the retry as already-done and returned
nil without writing anything. The pull finished with exit 0 while the
image was missing from the bundle, surfacing later as ImagePullBackOff
in the cluster.

Record metaByTag only after AppendImage succeeds and add regression
tests covering both the unit-level retry and the full pull/push flow
with an injected transient blob failure.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
… layout

- `PullImageSet` now cross-checks the pull plan against the layout: every planned image must be present in `index.json` under its short tag.
- An incomplete bundle fails the pull with the list of missing references, while the user still has network access.
- Matching by `short_tag` annotation avoids false positives on multi-arch digest rewrites; nil metas (AllowMissingTags) are skipped.
- Tests cover both branches: a missing image trips the check with exact references, a healthy digest- and tag-style pull passes.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep Glitchy-Sheep self-assigned this Jul 6, 2026
@Glitchy-Sheep Glitchy-Sheep added the bug Something isn't working label Jul 6, 2026
@Glitchy-Sheep Glitchy-Sheep marked this pull request as ready for review July 6, 2026 07:15
@Glitchy-Sheep Glitchy-Sheep requested a review from ldmonster as a code owner July 6, 2026 07:15
Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant