Skip to content

feat(dvcr): add per-namespace authorization#2586

Open
danilrwx wants to merge 30 commits into
mainfrom
dvcr-v3-upgrade
Open

feat(dvcr): add per-namespace authorization#2586
danilrwx wants to merge 30 commits into
mainfrom
dvcr-v3-upgrade

Conversation

@danilrwx

@danilrwx danilrwx commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

This PR upgrades DVCR (the Deckhouse Virtualization container registry) and adds optional per-namespace authorization for it.

DVCR registry upgrade (behavior-preserving):

  • Upgrade the embedded distribution registry from 2.8.3 to 3.1.1, migrating the build from GOPATH to a Go module (github.com/distribution/distribution/v3). Config schema, on-disk storage layout, dvcr-cleaner and garbage collection are unchanged.

Per-namespace DVCR authorization (new, behind dvcr.tenantRegistryAuthorization, default false):

  • A new in-registry dvcr-k8s AccessController is compiled into the registry binary. It authorizes by scoped token, not by caller identity: the controller mints a short-lived JWT granting access to exactly the repositories a given Pod uses, and the registry verifies the signature (reusing distribution's own registry/auth/token backend) and allows only the operations listed in the token. Fail-closed.
  • The token format is distribution's native JWT (access claim), so verification reuses upstream code; go-jose is already in distribution's dependency graph.
  • The signing key is an asymmetric ES256 keypair generated into dvcr-secrets (private → controller, public → registry). A compromised registry cannot mint tokens.
  • importer/uploader Pods and CDI DataVolumes authenticate to DVCR with the scoped token presented as ordinary Basic credentials. The shared read-write credential is no longer copied into tenant namespaces, and no CDI fork or TokenReview is required.
  • Nodes (containerd) use a dedicated pull-only node-puller credential; the controller keeps the static admin credential.
  • When the flag is off, rendered manifests and behavior are identical to today (htpasswd).

Why do we need it, and what problem does it solve?

DVCR is a single cluster-wide registry shared by all namespaces, and image import/upload previously relied on one shared read-write credential (copied into tenant namespaces) with no per-repository authorization. A tenant able to read Secrets in its own namespace could exfiltrate that credential and read or overwrite any other namespace's images. This change adds optional multi-tenant isolation so a Pod can access only the repositories it legitimately needs.

Authorizing by a signed scoped token (rather than by the caller's namespace) is what makes this correct even for imports whose Pod must run in another namespace to mount a source PVC (e.g. a ClusterVirtualImage built from a VirtualDisk): the token names the exact repository, independent of where the Pod runs.

The v3 upgrade is a prerequisite: the Go-module build is what allows compiling the in-registry authorization backend and reusing its token verification.

What is the expected result?

  • Flag off (default): identical to current behavior (htpasswd) — verified with helm template for both states.
  • Flag on: each import/upload Pod gets a token scoped to its own repositories; cross-repository access is denied (403); no read-write DVCR credential Secret is copied into tenant namespaces; nodes still pull VM images (pull-only); the controller still operates.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

Changelog entries

section: module
type: feature
summary: Optional per-namespace authorization for DVCR (`dvcr.tenantRegistryAuthorization`) isolating container image access between namespaces.

@danilrwx danilrwx added this to the v1.10.0 milestone Jul 3, 2026
@danilrwx danilrwx changed the title Dvcr v3 upgrade feat(dvcr): upgrade distribution to v3 and add per-namespace authorization Jul 3, 2026
@danilrwx danilrwx changed the title feat(dvcr): upgrade distribution to v3 and add per-namespace authorization feat(dvcr): add per-namespace authorization Jul 3, 2026
@danilrwx danilrwx marked this pull request as ready for review July 3, 2026 15:48
danilrwx added 17 commits July 3, 2026 18:24
@danilrwx
chore(dvcr): upgrade distribution registry 2.8.3 to 3.0.0
45bce80 
Migrate the DVCR build from the GOPATH-based build of distribution v2.8.3
to a Go-module build of github.com/distribution/distribution/v3 at v3.0.0.

- bump core.distribution to 3.0.0 in build/components/versions.yml
- drop the GO111MODULE=off GOPATH shim in images/dvcr/werf.inc.yaml and
  build ./cmd/registry directly from the module root
- update version ldflags to the v3 package path and lowercase symbols
  (version.version/revision/mainpkg under
  github.com/distribution/distribution/v3/version)

Config schema is unchanged: version, log, storage.cache.blobdescriptor,
http.debug.prometheus and health.storagedriver are all still valid in v3,
so templates/dvcr/configmap.yaml needs no changes.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
feat(dvcr): add multi-tenant authorization policy core
7918445 
Add the pure, dependency-free authorization decision logic for the DVCR
in-registry AccessController: namespace-scoped access for tenant Pods,
pull-only for nodes, full access for the controller, fail-closed default.

The policy anchors repository-name normalization with path.Clean (not
HasPrefix) so that names like vi/nsA-evil or vi/nsA/../nsB/x cannot be
mistaken for another namespace, denies the registry catalog to tenants,
and authorizes cross-repo blob mounts per-access (a mount from a foreign
namespace is denied via its pull access on the source repo).

Covered by table unit tests. The distribution auth.AccessController glue
(request parsing, TokenReview) and build wiring are added separately.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
feat(dvcr): add distribution v3 AccessController glue and TokenReview
f525193 
Wire the authorization policy into a distribution v3 auth.AccessController:

- access.go registers the "dvcr-k8s" controller, parses Basic credentials,
  classifies them (constant-time static admin/node-puller match, else a
  ServiceAccount token verified via TokenReview) and maps the result to the
  policy Subject; denies map to a 401 challenge (bad credential) or a plain
  403 (authenticated but not permitted).
- tokenreview.go verifies SA tokens through the Kubernetes TokenReview API
  using only the standard library (no client-go), with positive/negative
  result caching to keep the push path off the apiserver; the registry's own
  SA token is re-read per call to tolerate projected-token rotation.

Both files are behind the dvcr_registry build tag and compiled only inside
the werf DVCR build, so the dependency-free policy stays unit-testable. The
glue was type-checked against distribution v3.0.0.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
feat(dvcr): build the dvcr-k8s auth plugin into the registry binary
e7895ae 
Add a custom registry entrypoint (images/dvcr/registry-main) equal to
upstream cmd/registry plus a blank import of the dvcr-k8s auth plugin, and
graft the plugin sources into the distribution module during the werf build.

The binary is still emitted as "registry" so the existing consumers
(dvcr-cleaner's `registry garbage-collect`, the final image includePaths)
are unchanged. Built with -tags dvcr_registry so the distribution-dependent
plugin glue is compiled in; htpasswd stays compiled in too, so the same
image serves both auth modes and the feature flag only flips config.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
feat(dvcr): add node-puller credential and tenant-authz feature flag
2d4b625 
- generate a pull-only node-puller password (passwordPuller) in the
  generate-secret-for-dvcr hook, alongside the existing admin passwordRW;
  it needs no htpasswd entry since it carries no push rights
- expose internal.dvcr.passwordPuller in the values schema
- add the dvcr.tenantRegistryAuthorization config flag (default false)
  that gates per-namespace DVCR authorization, with a Russian doc-ru entry

Hook unit tests updated and passing.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
feat(dvcr): wire dvcr-k8s auth backend behind the tenant-authz flag
8113b69 
When dvcr.tenantRegistryAuthorization is enabled:
- _helpers.tpl drops the htpasswd REGISTRY_AUTH env (auth comes from the
  ConfigMap instead) and mounts the passwordRW/passwordPuller files into /auth
- configmap.yaml renders the auth.dvcr-k8s block (realm, admin and node-puller
  password files, TokenReview cache TTLs)
- nodegroupconfiguration.yaml gives containerd the pull-only node-puller
  credential instead of admin:passwordRW
- secret.yaml stores passwordPuller in dvcr-secrets
- rbac-for-us.yaml binds the registry ServiceAccount to system:auth-delegator
  so the backend can create TokenReviews

When disabled (default) the rendered output is unchanged (htpasswd). Verified
with helm template for both flag states.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
feat(dvcr): authenticate importer/uploader Pods with SA token
1c887c8 
Cut the importer/uploader Pod path over to per-namespace DVCR authorization
when dvcr.tenantRegistryAuthorization is enabled:

- plumb DVCR_TENANT_AUTHZ_ENABLED into dvcr.Settings.TenantAuthzEnabled
- ShouldCopyDVCRAuthSecret no longer copies the shared read-write credential
  into the tenant namespace when the flag is on
- importer/uploader ApplyDVCRDestinationSettings mounts a projected
  ServiceAccount token (600s, default audience) instead of the dockerconfigjson
  Secret, and passes its path via *_DESTINATION_TOKEN_FILE
- dvcr-artifact reads the token fresh on every request via a custom
  authn.Authenticator, so kubelet token rotation does not break long pushes

The CDI DataVolume path (VirtualDisk provisioning via the upstream CDI
importer) still uses the shared credential and is not yet covered; tracked
separately. Both modules build; existing unit tests pass; helm template
verified for flag on/off.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
feat(dvcr): use SA-token auth for VirtualDisk CDI imports
1e2f3ea 
Close the CDI/VirtualDisk gap in per-namespace DVCR authorization. When
dvcr.tenantRegistryAuthorization is enabled, EnsureForDataVolume no longer
copies the shared read-write credential into the tenant namespace; instead it
writes a CDI-compatible Secret whose accessKeyId is the ServiceAccount-token
sentinel. The patched CDI importer reads that sentinel and authenticates to
DVCR with the importer Pod's own projected ServiceAccount token, which the
registry authorizes for the tenant namespace only.

Requires the matching CDI fork change (feat(importer): support ServiceAccount
token auth for registry source).

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
chore(cdi): point 3p-containerized-data-importer to SA-token auth branch
43e767a 
Reference the CDI fork branch feat/dvcr-sa-token, which adds ServiceAccount-token
authentication for the registry source. Required so the VirtualDisk CDI import
path authenticates to DVCR with the importer Pod's namespace-scoped identity
under per-namespace authorization.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(dvcr): grant admin to ServiceAccounts in the module namespace
ebb0a96 
Control-plane components run in the module namespace (d8-virtualization) as
its default ServiceAccount: the controller, the ClusterVirtualImage importer
and garbage collection. They legitimately read images across namespaces and
write cluster-scoped cvi repositories. Classifying them as namespace-scoped
tenants broke ClusterVirtualImage imports (cross-namespace source pull and
cvi push returned 400).

Add a privilegednamespace option to the dvcr-k8s auth backend: a ServiceAccount
whose TokenReview namespace equals it is granted admin. Tenants cannot run Pods
in that namespace, so this is safe. Rendered from configmap.yaml as
d8-<chart>.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
refactor(dvcr): verify scoped JWT tokens instead of TokenReview
7d768bb 
Replace the TokenReview-based per-namespace classification with signed
scoped tokens: importer/uploader Pods present a JWT (minted by the
controller) carrying an access claim for exactly the repository they use.
The registry plugin reuses distribution's own token verification
(signature, issuer, audience, nbf/exp) and authorizes each requested
access against the token's grants.

- policy.go: RoleScoped carrying Grants; drop tenant/TokenReview roles
- access.go: verify JWT via registry/auth/token; drop namespace derivation
- remove tokenreview.go (no apiserver TokenReview on the push path)

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
feat(controller): add scoped DVCR token signer
378c801 
Mint ES256-signed JWTs whose access claim (matching distribution's
ClaimSet/ResourceActions) grants exactly the DVCR repositories a Pod
uses. Promote golang-jwt/v5 to a direct dependency.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
feat(controller): mint scoped DVCR tokens for importer/uploader/CDI
28ca6b1 
When per-namespace authorization is on, the controller mints a token
scoped to exactly the repositories each Pod uses and delivers it through
the existing destination-auth-secret path — a dockerconfigjson for the
dvcr-artifact importer/uploader, an Opaque accessKeyId/secretKey for the
CDI DataVolume. No shared read-write credential is copied into tenant
namespaces, and no projected ServiceAccount token / TokenReview is needed.

- dvcr.Settings: TokenSigner + RepoPath helper; load private key from env
- supplements: EnsureForPod/EnsureForDataVolume take the token scope
- service: compute per-Pod scope (dest push+pull, source pull, CDI pull)
- drop projected SA token volumes and their consts

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
refactor(dvcr-artifact): drop projected SA token auth
38a521c 
The scoped DVCR token now arrives as ordinary Basic credentials in the
destination auth config, so the per-request ServiceAccount token file
reader is no longer needed. Destination auth is plain Basic again.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
feat(hooks): generate ECDSA keypair for scoped DVCR tokens
cf22e4f 
Add tokenPrivateKey/tokenPublicKey (ECDSA P-256, PKCS8/PKIX PEM) to
dvcr-secrets. The controller signs scoped tokens with the private key;
the registry verifies them with the public key. Regenerated only when the
private key is missing or unparseable.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
feat(dvcr): wire scoped-token JWT auth into manifests
9705a6d 
- dvcr-secrets: ship the ECDSA keypair when tenant authz is on
- registry configmap: dvcr-k8s auth verifies JWT (issuer/audience/kid,
  public key file) instead of TokenReview
- mount tokenPublicKey into the registry; inject tokenPrivateKey into the
  controller
- drop the system:auth-delegator binding (no TokenReview)
- revert the CDI fork pin to upstream (no fork needed)

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
refactor(dvcr): simplify scoped-token plumbing
cff2b2f 
- revert dvcr-artifact to upstream: the scoped token arrives as plain
  Basic creds, so the destAuthenticator indirection added earlier is gone
  (the package is now byte-identical to main)
- drop the unused ttl parameter from Signer.Sign (always DefaultTTL)
- correct the feature-flag docs: scoped token, not ServiceAccount token

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
danilrwx added 3 commits July 3, 2026 18:24
@danilrwx
docs(dvcr): reword token TTL comment
2b24a96 
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
refactor(dvcr): drop redundant comments
a68b12e 
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
chore(dvcr): bump distribution to v3.1.1
1dabbb4 
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
danilrwx added 10 commits July 3, 2026 18:56
@danilrwx
fix(dvcr): reject x5c/jwk-signed scoped tokens in registry authz
a63ab20 
distribution's token.Verify tries a token's embedded x5c/jwk certificate
chain before the pinned kid->TrustedKeys lookup. With VerifyOptions.Roots
unset the chain validates against the system CA pool, so any publicly
trusted ECDSA leaf certificate could forge a token and bypass per-namespace
authorization. Pass a non-nil empty cert pool so no external chain verifies
and only the pinned public key (matched by kid) is trusted.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(dvcr): mint scoped DataVolume token independent of AuthSecret
0bae887 
CreateScopedTokenCDI signs from TokenSigner and does not read the source
AuthSecret, but it was nested under `if AuthSecret != ""`. With tenant
authorization on and AuthSecret empty, the DataVolume referenced an auth
secret that was never created, stalling the import. Switch on
TenantAuthzEnabled first, matching EnsureForPod.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(dvcr): generate registry passwords with crypto/rand
6f81b3d 
alphaNum seeded math/rand/v2 with the wall-clock time, a non-cryptographic
PRNG, to produce the shared registry passwordRW and the new node-puller
password. Use crypto/rand so these long-lived credentials are not derivable
from the generation time.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(dvcr): regenerate token keypair when public key is bad or mismatched
04a3bb2 
Keypair regeneration was gated only on the private key parsing. A corrupt,
missing or mismatched tokenPublicKey (shipped to the registry) would then be
kept, making the registry reject every scoped token with no self-heal.
Validate that the public key parses and matches the private key, else
regenerate the pair.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
docs(dvcr): note go-jose dependency in registry graft comment
3d69e25 
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(dvcr): refresh scoped token secret before it expires
6fe5203 
The scoped token was minted once and stored with a create-only Secret, so a
reconcile never re-issued it. An import that outlived the 24h token TTL, or a
Pod rescheduled past it, read an expired token and got a permanent 403 with no
self-heal. Record the token expiry in an annotation and re-mint the Secret on
reconcile only when less than half the TTL remains, keeping churn low while
guaranteeing a valid token for long-running imports.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
test(dvcr): cover JWT verifier against real distribution backend
05e1e8d 
The verifyJWT path (signature, issuer, audience, expiry, and the empty-Roots
x5c/jwk defense) had no test — access.go sits behind the dvcr_registry build
tag with distribution imports absent from the standalone module. Add
access_test.go under the same tag plus the distribution/go-jose/golang-jwt test
deps, minting tokens as the controller does and as an x5c attacker would, then
verifying through distribution's own token backend. Asserts the x5c-signed
token is rejected, guarding the load-bearing Roots line against regression.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
test(dvcr): cover token keypair validation on mismatch
3b4a98b 
hook_test.go claimed keypair-regeneration coverage lived elsewhere, but
validateECKeypair was never exercised. Add cases for a matching pair, a
mismatched public key, and corrupt/missing keys.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
refactor(dvcr): refresh scoped token in its last hour, not at half TTL
b5a271f 
Reconciles run far more often than an hour, so waiting until under an hour of
validity remains keeps a valid token available while cutting re-mint churn
roughly in half versus the TTL/2 threshold.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
refactor(dvcr): tighten comments on scoped-token code
ecae47f 
Trim the verbose comments added with the token refresh and JWT verifier work to
concise why-only notes; also drop a stale "half its TTL" reference.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@universal-itengineer universal-itengineer Awaiting requested review from universal-itengineer universal-itengineer is a code owner
@nevermarine nevermarine Awaiting requested review from nevermarine nevermarine is a code owner
@Isteb4k Isteb4k Awaiting requested review from Isteb4k Isteb4k is a code owner
@yaroslavborbat yaroslavborbat Awaiting requested review from yaroslavborbat yaroslavborbat is a code owner
@fl64 fl64 Awaiting requested review from fl64 fl64 is a code owner
@prismagod prismagod Awaiting requested review from prismagod prismagod is a code owner
@goganat goganat Awaiting requested review from goganat goganat is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@danilrwx danilrwx

Labels

None yet

Projects

None yet

Milestone

v1.10.0

Development

Successfully merging this pull request may close these issues.

1 participant

@danilrwx