From 364f19d151ddc0791422a5e0bde4d49575a80b04 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 17 Jun 2026 14:31:57 +0500 Subject: [PATCH 1/5] [docs] Add GitOps configuration surfaces design proposal Proposes a single GitOps primitive (GitRepository + Kustomization + a scoped ServiceAccount) instantiated at two trust tiers: platform GitOps for admins (cluster scope) and tenant GitOps for tenants (namespace scope), plus a packageOverrides admin override layer. Generalizes cozystack/cozystack#2731 as the first increment. Signed-off-by: Myasnikov Daniil --- design-proposals/gitops-surfaces/README.md | 213 +++++++++++++++++++++ 1 file changed, 213 insertions(+) create mode 100644 design-proposals/gitops-surfaces/README.md diff --git a/design-proposals/gitops-surfaces/README.md b/design-proposals/gitops-surfaces/README.md new file mode 100644 index 0000000..e7d3632 --- /dev/null +++ b/design-proposals/gitops-surfaces/README.md @@ -0,0 +1,213 @@ +# GitOps Configuration Surfaces for Cozystack + +- **Title:** `GitOps Configuration Surfaces: admin and tenant self-service from git` +- **Author(s):** `@myasnikovdaniil` +- **Date:** `2026-06-17` +- **Status:** Draft + +## Overview + +Cozystack ships a complete Flux control plane but exposes almost none of it to the people running the cluster. Admins configure the platform by hand-patching Package CRs and applying raw manifests with `kubectl`; tenants configure their apps through the dashboard or one-off `kubectl apply`. Both paths produce configuration drift and neither is reproducible. Meanwhile the source-controller and kustomize-controller needed to reconcile a git repository are *already running* in the management cluster — they are just not offered as a product surface. + +This proposal defines a single primitive — **a reconciled GitOps surface** (`GitRepository` + `Kustomization` + a scoped `ServiceAccount`) — and instantiates it at two trust tiers: a **platform GitOps** surface for admins (cluster scope) and a **tenant GitOps** surface for tenant users (namespace scope). It also proposes an **admin override layer** so that the most common admin task — changing component parameters — does not require fighting Helm over field ownership. The first increment (the admin surface) is already prototyped in [cozystack/cozystack#2731](https://github.com/cozystack/cozystack/pull/2731). + +## Scope and related proposals + +- **First increment — admin surface:** [cozystack/cozystack#2731](https://github.com/cozystack/cozystack/pull/2731) (`feat(platform): add cozystack.customizer system package`) implements Tool 1, Phase 1. This proposal generalizes that PR and gives it a home in the larger model. +- **Deferred to follow-up proposals** (intentionally not designed in full here): field-ownership enforcement via `ValidatingAdmissionPolicy` / admission webhook on `packages.cozystack.io` (sketched under Security; full design is separate); and a delegated mid-tier between cluster-admin and tenant (team-scoped admins). +- This proposal should land before the tenant surface (Tool 2) is implemented, because the unifying primitive and the multitenancy-lockdown decision are shared. + +## Context + +Cozystack delivers everything through Flux. The relevant pieces today: + +- **Management Flux is real and complete.** `internal/fluxinstall` (embedded in the cozystack-operator) installs source-controller, kustomize-controller, helm-controller, and notification-controller into the `cozy-fluxcd` namespace, watching all namespaces. An admin-created `GitRepository` + `Kustomization` in `cozy-system` reconciles today with no new controllers. API versions in use: `source.toolkit.fluxcd.io/v1`, `kustomize.toolkit.fluxcd.io/v1`, `helm.toolkit.fluxcd.io/v2`. +- **The Package CRD is the platform's config unit.** `api/v1alpha1/package_types.go` (`scope=Cluster`). A `Package` selects a variant and carries component values. Two very different lifecycles exist: + - **The bootstrap Package** `cozystack.cozystack-platform` is **created by hand** at install time — operators copy `packages/core/installer/example/platform.yaml`, edit `spec.variant` and `spec.components.platform.values`, and `kubectl apply` it. No controller owns its spec. + - **Downstream Packages** (`cozystack.linstor`, `cozystack.cilium`, …) are **rendered by the platform chart** (`packages/core/platform`) with `helm.sh/resource-policy: keep` and are owned by helm-controller. +- **Most admin config is Package values, not raw resources.** VM golden images live in `packages/system/vm-default-images` values; LINSTOR parameters in `packages/system/linstor` values; cert-manager `ClusterIssuer`s are driven by `packages/core/platform` values → the `cozystack-values` Secret in `cozy-system` → the `cert-manager-issuers` chart. +- **Tenants cannot self-GitOps.** A `Tenant` (`apps.cozystack.io`) produces a `tenant-` namespace, a ServiceAccount named ``, and `cozy:tenant:*` ClusterRoles. Tenants create curated **Application** CRs (e.g. Postgres, Kafka, VM) which the aggregated apiserver converts into HelmReleases reconciled by a sharded helm-controller (`sharding.fluxcd.io/key: tenants`). Tenants have **no RBAC for `source`/`kustomize` CRDs** and there is **no tenant-scoped source/kustomize reconciliation**. The shipped `flux-instance` sets `multitenant: false` (`packages/system/fluxcd/charts/flux-instance/values.yaml`). + +### The problem + +- *"I changed the LINSTOR auto-diskful timeout on a node by hand last month and now I can't remember which clusters got it."* — admin config is imperative and drifts. +- *"To set up DR I'd need to reconstruct every `kubectl patch` I ever ran."* — there is no single source of truth for cluster configuration. +- *"My team already keeps our app manifests in git. I want Cozystack to reconcile them into my tenant, but all I can do is click in the dashboard or `kubectl apply` and hope nobody else changed it."* — tenants have no GitOps path even though Flux is right there. +- *"I patched a downstream Package's values and Helm silently took the field back on the next reconcile (or I silently took Helm's)."* — patching chart-owned Packages races helm-controller because kustomize-controller hardcodes `client.ForceOwnership`. + +## Goals + +- Provide an opt-in, declarative GitOps surface for **admins** that reconciles an admin-owned repo into the management cluster. +- Provide an opt-in, declarative GitOps surface for **tenants** that reconciles a tenant-owned repo into that tenant's namespace, isolated to that tenant's existing RBAC. +- Make the common admin task — overriding component parameters (versions, golden-image values, LINSTOR options) — possible **without** racing helm-controller for field ownership. +- Define one shared primitive and reuse it across tiers, rather than shipping unrelated one-off tools. +- Keep all surfaces **off by default**; an operator opts in per cluster / per tenant. +- Establish an honest, documented trust model for each tier. + +### Non-goals + +- This proposal does **not** reduce the capability of an admin below what they already have by hand. The admin surface is a cluster-admin-equivalent credential by design. +- It does **not** fully specify the field-ownership admission policy (separate proposal). +- It does **not** introduce a per-tenant Flux *instance*; tenant reconciliation reuses the shared controllers with isolation enforced by RBAC + lockdown. +- It does **not** add multi-cluster / fleet management. + +## Design + +### The unifying primitive + +Every use case is the same atom: + +``` +GitRepository + Kustomization + a scoped ServiceAccount → reconciled by cozy-fluxcd +``` + +Only two things vary: **(a)** the trust level of the ServiceAccount and **(b)** the isolation guarantees enforced on its Kustomization. We therefore build one parameterized chart and bind it to different SAs at different scopes. + +```mermaid +flowchart TD + subgraph repo_admin[Admin git repo] + A1[Package overrides
raw cluster resources
policies, issuers, BYO HelmReleases] + end + subgraph repo_tenant[Tenant git repo] + T1[Application CRs
namespaced manifests] + end + + A1 -->|GitRepository + Kustomization
SA: cozystack-customizer| KC[(kustomize-controller
cozy-fluxcd)] + T1 -->|GitRepository + Kustomization
SA: tenant-name| KCT[(kustomize-controller
tenant shard)] + + KC -->|cluster scope| CLUSTER[(Management cluster
Packages, cluster resources)] + KCT -->|namespace scope only| NS[(tenant-name namespace)] +``` + +### Tool 1 — Platform GitOps (admin, cluster scope) + +Generalizes [#2731](https://github.com/cozystack/cozystack/pull/2731). A system package (`cozystack.customizer`) provisions, in `cozy-system`: a `GitRepository` pointing at an admin-owned repo (auth Secret pre-created by the admin); a `Kustomization` reconciled by a curated `cozystack-customizer` ServiceAccount; and the RBAC that SA needs. + +**Trust model (explicit):** write access to the admin repo is equivalent to cluster-admin. This is not a weakening of the platform — an operator who can `kubectl patch` already has this power. The surface *legitimizes and records* what was previously done by hand. Documentation must state this plainly, so the repo and its git credentials are protected like a kubeconfig. + +This tool covers admin use cases directly: changing platform parameters (UC1), managing arbitrary cluster resources outside the core API — custom `ClusterIssuer`s, LINSTOR `StoragePool`s, BYO `HelmRelease`s, `NetworkPolicy`, custom CRDs (UC3) — plus secrets-as-code, policy-as-code, environment overlays, and ultimately whole-cluster reproducibility / DR. + +#### Package ownership: two classes, two stories + +The single most important design point. RBAC is object-level; the contract we need is field-level. The two Package lifecycles diverge here: + +| | bootstrap `cozystack.cozystack-platform` | downstream `cozystack.linstor`, … | +|---|---|---| +| Created by | hand (`kubectl apply` from the installer example) | the platform chart (`resource-policy: keep`) | +| Spec field manager | the operator — **no controller** | **helm-controller** | +| Force-ownership risk | **none** — nothing competes | **real** — claiming a chart-owned field steals it | + +- **Bootstrap Package:** because no controller owns its spec, the admin repo can simply **own it**. The recommended flow is *self-adoption*: bootstrap once by hand with the customizer enabled, then commit the same Package manifest to the repo; from then on the repo is its source of truth (the standard `flux bootstrap` pattern). Use Server-Side Apply with a *partial* Package manifest to own only the fields you manage. +- **Downstream Packages:** patching them races helm-controller. We do **not** solve this by denying access (that would re-introduce hand-management of exactly the parameters admins care about). We solve it with an override layer (below) and, later, an admission policy. + +#### Admin override layer (for UC1b — component parameters) + +Today, changing a downstream component's parameters (e.g. `linstor.autoDiskful.minutes`, a golden-image URL, a pinned version) means patching a *chart-owned* Package — the contested case. Instead, introduce a **clean override surface the platform chart reads but never owns**, extending the existing `cozystack-values` mechanism: + +```yaml +# carried on the hand-owned bootstrap Package (or a sibling cozystack-overrides Secret) +spec: + components: + platform: + values: + packageOverrides: + linstor: + autoDiskful: + minutes: 10 + vm-default-images: + images: + - name: ubuntu-24.04 + url: https://example/custom.img +``` + +Downstream package HelmReleases merge `packageOverrides.` over their chart defaults. The admin then **only ever edits hand-owned objects** — no contention with helm-controller — and all admin config collapses to a single GitOps-managed surface, which is also the foundation for whole-cluster reproducibility (UC5). + +### Tool 2 — Tenant GitOps (tenant, namespace scope) + +Delivered as a **catalog app**, not a `Tenant` API change — it is opt-in per tenant, requires no operator change, and fits the existing app model. + +- New app package `packages/apps/gitops` + ApplicationDefinition in `packages/system/gitops-rd`. When a tenant instantiates it, the chart renders, in the tenant namespace, a `GitRepository` + `Kustomization` with `spec.serviceAccountName: ` — the tenant's **existing** SA, which already holds `cozy:tenant:admin`. +- Because reconciliation runs as the tenant SA, isolation is **RBAC-enforced by construction**: a tenant can only create what `cozy:tenant:*` permits, inside its own namespace subtree. Tenants point the surface at a repo of Application CRs and namespaced manifests. + +#### Isolation hardening + +A tenant must not be able to (a) reconcile into another namespace, or (b) borrow a more privileged SA. Two options, recommendation first: + +1. **Dedicated tenant kustomize-controller shard** (recommended) — mirror the existing `flux-tenants` helm-controller shard. Run a kustomize-controller selecting `sharding.fluxcd.io/key: tenants`, started with `--no-cross-namespace-refs` and a default-service-account lockdown, so the restrictions apply *only* to tenant reconciliation and platform Kustomizations are unaffected. This matches a pattern Cozystack already uses. +2. **Global Flux multitenancy flags** — flip the `flux-instance` `multitenant` setting and apply `--no-cross-namespace-refs` cluster-wide. Simpler, but it constrains the platform's own Kustomizations and is a larger blast radius. + +### Cross-cutting capabilities + +- **Secrets-as-code:** SOPS/age decryption (the customizer already exposes a `decryption` block) for admins; per-namespace decryption keys (or external-secrets) for tenants. One design, two scopes. +- **Status surfacing:** expose `Kustomization` / `GitRepository` sync status in the dashboard via a `TenantModule`-style read-only view, so admins and tenants see drift without raw Flux access. + +## User-facing changes + +- **Admins:** a single `customizer.enabled` flag (plus `source`/`kustomization`/`rbac` blocks) in the platform values, off by default. When enabled, the admin manages the cluster from a git repo; component parameters are set via `packageOverrides`. +- **Tenants:** a new **GitOps** entry in the app catalog. Instantiating it lets a tenant register a git repo that reconciles into their namespace. Sync status appears in the dashboard. +- **Docs:** a new `operations/customizer` (admin) page and a tenant GitOps how-to, including the explicit trust model and the bootstrap-vs-downstream ownership rules. + +## Upgrade and rollback compatibility + +- Fully **additive and opt-in**; existing clusters are unaffected until an operator enables a surface. No migration required. +- The `packageOverrides` layer is backward compatible: absent the key, charts use their current defaults. +- **Rollback:** disabling a surface stops emitting its resources. Note that platform Packages carry `helm.sh/resource-policy: keep`, so the customizer Package and its child objects are **not** auto-deleted on disable — teardown is a documented manual step (this is existing platform behavior, called out here so it is not surprising). +- Self-adoption of the bootstrap Package is reversible: stop reconciling it from the repo and resume hand-management; nothing about the object changes. + +## Security + +This is the heart of the proposal; each tier has a distinct, **explicit** trust boundary. + +- **Tool 1 (admin):** repo write == cluster-admin, by design and by documentation. The curated ClusterRole in the prototype (patch-not-delete on Packages, no CRDs, etc.) is *defense against accident, not against intent* — anyone with repo write can escalate, so the docs must not over-claim a boundary. New trust surface introduced: an admin git repo + its credentials (protect like a kubeconfig), and a standing, self-healing reconciling credential whose blast radius equals an admin's. +- **Tool 1 footguns to narrow** (they cost nothing functionally): source access should not let the customizer rewrite the platform's *own* supply chain (the `cozystack-packages` OCIRepository in `cozy-system`), so scope write away from `cozy-system`; and field ownership on downstream Packages should eventually be enforced by a `ValidatingAdmissionPolicy` scoped to the customizer SA that rejects changes to fields currently managed by helm-controller (detected via `managedFields`), leaving the hand-owned bootstrap Package free. Until then, the `packageOverrides` layer keeps admins off chart-owned objects entirely, so the contract is mostly avoided rather than merely documented. +- **Tool 2 (tenant):** the surface runs as the tenant's own SA, so it can do nothing the tenant cannot already do via the API. The new risk is **cross-namespace / SA-borrowing**, closed by the tenant shard lockdown (`--no-cross-namespace-refs`, forced `serviceAccountName`). Tenant-supplied input = the contents of a tenant-owned repo, bounded by `cozy:tenant:*`. +- **Secrets:** both tiers may pull git credentials and SOPS keys. These are admin/tenant pre-created Secrets; the platform never generates or owns them. + +## Failure and edge cases + +- Missing `source.url` / `kustomization.path` when enabled → chart `fail`s at render; Flux surfaces the error on HelmRelease status (already implemented in the prototype). +- Private repo without a `secretRef` → GitRepository reports auth failure on its status; no partial apply. +- Admin manifest declares a chart-owned Package field directly (bypassing `packageOverrides`) → silently claimed from helm-controller until the admission policy lands; documented as advisory in the interim. +- Tenant Kustomization references a source in another namespace or a foreign SA → rejected by the tenant shard lockdown. +- Customizer disabled then re-enabled → resources re-adopt cleanly (SSA, `resource-policy: keep`); no duplicate-creation errors. +- `packageOverrides` set for a non-existent component → no-op; the override map is read only by the components that look for their key. + +## Testing + +- **Helm unit tests** (`make unit-tests`): bundle wiring for the customizer (enabled / disabled / `disabledPackages`) — already in the prototype; add assertions that lock the RBAC surface (no `delete` on Packages, source write excluded from `cozy-system`). +- **Helm unit tests** for `packageOverrides` merge precedence on at least one downstream package (e.g. linstor). +- **Helm unit tests** for the tenant `gitops` app: SA is forced to ``, sourceRef is in-namespace. +- **e2e (BATS, `hack/e2e-apps/`):** enable the admin surface against a dev cluster, push a `packageOverrides` change, assert the downstream HelmRelease re-renders; create a tenant `gitops` app, assert it reconciles an Application CR and is denied a cross-namespace ref. +- **Manual:** `kubectl auth can-i --as=...` matrix for both SAs (already done for the admin SA in #2731); verify the tenant SA cannot reconcile outside its namespace. + +## Rollout + +1. **Tool 1, Phase 1** — admin customizer system package + ownership-model docs ([#2731](https://github.com/cozystack/cozystack/pull/2731), in review). Narrow the two source/field footguns noted under Security. +2. **Tool 1, Phase 2** — `packageOverrides` admin override layer. Highest leverage: unblocks safe component-parameter changes (UC1b) and whole-cluster reproducibility (UC5). +3. **Tool 2, Phases 1–2** — tenant `gitops` app + tenant kustomize-controller shard with lockdown. The largest user-visible win. +4. **Cross-cutting** — secrets-as-code + dashboard sync-status views. +5. **Hardening** — field-ownership `ValidatingAdmissionPolicy` (separate proposal) and policy-as-code guardrails. + +## Open questions + +1. **`packages/system/fluxcd` vs `internal/fluxinstall`** — is the flux-operator-managed `FluxInstance` (`multitenant: false`) a separate reconciliation path from the operator-embedded controllers? This determines exactly which instance serves tenant Kustomizations and where the lockdown flags belong. +2. **Tenant raw resources** — should the tenant SA ever hold raw `HelmRelease` / CR create rights, or stay confined to the curated Application abstraction? Affects how much of UC2 is "BYO manifests" vs "catalog apps from git." +3. **Override layer shape** — `packageOverrides` on the bootstrap Package vs a dedicated `cozystack-overrides` Secret. The former keeps a single object; the latter decouples override lifecycle from the bootstrap Package. +4. **Tenant GitOps delivery** — catalog app (recommended) vs a `spec.gitops` field on the `Tenant` CR. The app avoids an API change; the field is more discoverable. + +## Alternatives considered + +- **Runtime mechanism — give admins/tenants raw Flux CRDs via RBAC, no chart.** Rejected: no curated trust tiers, no isolation defaults, and tenants would need source/kustomize RBAC plus a cross-namespace lockdown anyway. The primitive-as-a-package gives a consistent UX and safe defaults for the same underlying objects. +- **Field ownership — solve UC1b with RBAC (`resourceNames` allow/deny on Packages).** Rejected: RBAC is object-level, but the contract is field-level (admin owns `spec.components.*.values`, chart owns `spec.variant` and structural fields). Denying the object re-introduces hand-management of the parameters admins most want; allowing it races helm-controller. The override layer plus a field-level admission policy is the correct axis. +- **Field ownership — bespoke admission webhook now.** Deferred in favor of a `ValidatingAdmissionPolicy` (native, CEL, no webhook server) and, before either, the `packageOverrides` layer that avoids the contested objects entirely. +- **Tenant isolation — flip global Flux multitenancy.** Rejected as the default: it constrains the platform's own Kustomizations. A dedicated tenant shard matches the existing `flux-tenants` pattern and limits the blast radius. +- **Tenant delivery — extend the `Tenant` CRD with a `gitops` field.** Viable, kept as an open question; the catalog app is preferred to avoid a core API change and to make the surface opt-in per tenant. +- **One monolithic tool for all cases.** Rejected: admin and tenant surfaces have fundamentally different trust models, RBAC, and isolation requirements. They share a primitive, not an implementation. + +--- + + From 9faf7e01c4cf2d60d0164fa4aa35aec549953adf Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Fri, 19 Jun 2026 14:57:00 +0500 Subject: [PATCH 2/5] docs(gitops-surfaces): make PackageValues the primary admin override layer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Recast Tool 1's override layer from a `packageOverrides` map on the bootstrap Package (merged into downstream Packages by the platform chart) to a dedicated `PackageValues` API kind that the operator merges into each HelmRelease. `packageOverrides` is kept as a lighter-weight chart-only alternative/interim. PackageValues removes the helm-controller force-ownership race structurally (the chart never writes PackageValues; the admin never writes the chart-owned Package), makes ownership enforceable by object-level RBAC, and is fully additive — absent a PackageValues, the operator builds HelmReleases as today. Updates the admin override layer, user-facing / upgrade / security / failure / testing / rollout sections, open question 3 (now PackageValues precedence and binding), and alternatives considered. Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Myasnikov Daniil --- design-proposals/gitops-surfaces/README.md | 56 +++++++++++++--------- 1 file changed, 33 insertions(+), 23 deletions(-) diff --git a/design-proposals/gitops-surfaces/README.md b/design-proposals/gitops-surfaces/README.md index e7d3632..06cfd00 100644 --- a/design-proposals/gitops-surfaces/README.md +++ b/design-proposals/gitops-surfaces/README.md @@ -102,25 +102,34 @@ The single most important design point. RBAC is object-level; the contract we ne #### Admin override layer (for UC1b — component parameters) -Today, changing a downstream component's parameters (e.g. `linstor.autoDiskful.minutes`, a golden-image URL, a pinned version) means patching a *chart-owned* Package — the contested case. Instead, introduce a **clean override surface the platform chart reads but never owns**, extending the existing `cozystack-values` mechanism: +Today, changing a downstream component's parameters (e.g. `linstor.autoDiskful.minutes`, a golden-image URL, a pinned version) means patching a *chart-owned* Package — the contested case. We solve it by **splitting the override layer out of the Package into its own resource**, extending the existing API with a third kind: + +| kind | owner | role | +|---|---|---| +| `PackageSource` | platform (unchanged) | charts + variants | +| `Package` | the platform chart (`resource-policy: keep`) | the reconciled, system-owned layer | +| `PackageValues` (new) | the admin repo | the human-owned override layer | + +A `PackageValues` object carries per-package admin overrides; the operator merges it over the matching Package's values when it builds the HelmRelease: ```yaml -# carried on the hand-owned bootstrap Package (or a sibling cozystack-overrides Secret) +apiVersion: cozystack.io/v1alpha1 +kind: PackageValues +metadata: + name: cozystack.linstor # 1:1 with the target Package by name spec: components: - platform: + linstor: values: - packageOverrides: - linstor: - autoDiskful: - minutes: 10 - vm-default-images: - images: - - name: ubuntu-24.04 - url: https://example/custom.img + autoDiskful: + minutes: 10 ``` -Downstream package HelmReleases merge `packageOverrides.` over their chart defaults. The admin then **only ever edits hand-owned objects** — no contention with helm-controller — and all admin config collapses to a single GitOps-managed surface, which is also the foundation for whole-cluster reproducibility (UC5). +Because the chart never writes `PackageValues` and the admin never writes the chart-owned `Package`, **there is no field for the admin and helm-controller to fight over** — the force-ownership race is removed structurally, not merely sidestepped. This is *not* a sibling `cozystack-values`-style Secret (the operator hardcodes a single `valuesFrom` and resets others); `PackageValues` is a first-class CR the operator itself merges into `hr.Spec.Values`, so it cooperates with the existing reconcile path. + +The split is **additive**: with no `PackageValues` present, the operator path is byte-identical to today, so existing clusters are unaffected (see Upgrade compatibility). It also makes ownership enforceable by ordinary object-level RBAC — the admin GitOps SA gets write on `PackageValues` and read-only on `Package` — which is the discipline that keeps admins off chart-owned objects entirely, the foundation for whole-cluster reproducibility (UC5). + +*Alternative — `packageOverrides` on the bootstrap Package.* The same overrides can instead be carried as a `packageOverrides.` map under `spec.components.platform.values` on the hand-owned bootstrap Package, with the platform chart merging them into each downstream Package at render time. This ships with no API change (pure chart templating) and collapses all admin config into one object, but it sidesteps the race rather than removing it, gives all-or-nothing RBAC on one god-object, and pushes list-vs-map merge correctness into per-package templates. Kept as a lighter-weight fallback; since `PackageValues` is additive, `packageOverrides` could even ship first as an interim. ### Tool 2 — Tenant GitOps (tenant, namespace scope) @@ -143,14 +152,14 @@ A tenant must not be able to (a) reconcile into another namespace, or (b) borrow ## User-facing changes -- **Admins:** a single `customizer.enabled` flag (plus `source`/`kustomization`/`rbac` blocks) in the platform values, off by default. When enabled, the admin manages the cluster from a git repo; component parameters are set via `packageOverrides`. +- **Admins:** a single `customizer.enabled` flag (plus `source`/`kustomization`/`rbac` blocks) in the platform values, off by default. When enabled, the admin manages the cluster from a git repo; component parameters are set via `PackageValues` resources. - **Tenants:** a new **GitOps** entry in the app catalog. Instantiating it lets a tenant register a git repo that reconciles into their namespace. Sync status appears in the dashboard. - **Docs:** a new `operations/customizer` (admin) page and a tenant GitOps how-to, including the explicit trust model and the bootstrap-vs-downstream ownership rules. ## Upgrade and rollback compatibility - Fully **additive and opt-in**; existing clusters are unaffected until an operator enables a surface. No migration required. -- The `packageOverrides` layer is backward compatible: absent the key, charts use their current defaults. +- The `PackageValues` resource is additive: with none present, the operator builds HelmReleases exactly as today, so charts use their current defaults. (The `packageOverrides` alternative is equally backward compatible — absent the key, charts use their current defaults.) - **Rollback:** disabling a surface stops emitting its resources. Note that platform Packages carry `helm.sh/resource-policy: keep`, so the customizer Package and its child objects are **not** auto-deleted on disable — teardown is a documented manual step (this is existing platform behavior, called out here so it is not surprising). - Self-adoption of the bootstrap Package is reversible: stop reconciling it from the repo and resume hand-management; nothing about the object changes. @@ -159,7 +168,7 @@ A tenant must not be able to (a) reconcile into another namespace, or (b) borrow This is the heart of the proposal; each tier has a distinct, **explicit** trust boundary. - **Tool 1 (admin):** repo write == cluster-admin, by design and by documentation. The curated ClusterRole in the prototype (patch-not-delete on Packages, no CRDs, etc.) is *defense against accident, not against intent* — anyone with repo write can escalate, so the docs must not over-claim a boundary. New trust surface introduced: an admin git repo + its credentials (protect like a kubeconfig), and a standing, self-healing reconciling credential whose blast radius equals an admin's. -- **Tool 1 footguns to narrow** (they cost nothing functionally): source access should not let the customizer rewrite the platform's *own* supply chain (the `cozystack-packages` OCIRepository in `cozy-system`), so scope write away from `cozy-system`; and field ownership on downstream Packages should eventually be enforced by a `ValidatingAdmissionPolicy` scoped to the customizer SA that rejects changes to fields currently managed by helm-controller (detected via `managedFields`), leaving the hand-owned bootstrap Package free. Until then, the `packageOverrides` layer keeps admins off chart-owned objects entirely, so the contract is mostly avoided rather than merely documented. +- **Tool 1 footguns to narrow** (they cost nothing functionally): source access should not let the customizer rewrite the platform's *own* supply chain (the `cozystack-packages` OCIRepository in `cozy-system`), so scope write away from `cozy-system`; and field ownership on downstream Packages should eventually be enforced by a `ValidatingAdmissionPolicy` scoped to the customizer SA that rejects changes to fields currently managed by helm-controller (detected via `managedFields`), leaving the hand-owned bootstrap Package free. Until then, the `PackageValues` resource keeps admins off chart-owned objects entirely — they write `PackageValues`, never the chart-owned `Package` — so the contract is structurally avoided rather than merely documented. - **Tool 2 (tenant):** the surface runs as the tenant's own SA, so it can do nothing the tenant cannot already do via the API. The new risk is **cross-namespace / SA-borrowing**, closed by the tenant shard lockdown (`--no-cross-namespace-refs`, forced `serviceAccountName`). Tenant-supplied input = the contents of a tenant-owned repo, bounded by `cozy:tenant:*`. - **Secrets:** both tiers may pull git credentials and SOPS keys. These are admin/tenant pre-created Secrets; the platform never generates or owns them. @@ -167,23 +176,23 @@ This is the heart of the proposal; each tier has a distinct, **explicit** trust - Missing `source.url` / `kustomization.path` when enabled → chart `fail`s at render; Flux surfaces the error on HelmRelease status (already implemented in the prototype). - Private repo without a `secretRef` → GitRepository reports auth failure on its status; no partial apply. -- Admin manifest declares a chart-owned Package field directly (bypassing `packageOverrides`) → silently claimed from helm-controller until the admission policy lands; documented as advisory in the interim. +- Admin manifest declares a chart-owned Package field directly (bypassing `PackageValues`) → silently claimed from helm-controller until the admission policy lands; documented as advisory in the interim. - Tenant Kustomization references a source in another namespace or a foreign SA → rejected by the tenant shard lockdown. - Customizer disabled then re-enabled → resources re-adopt cleanly (SSA, `resource-policy: keep`); no duplicate-creation errors. -- `packageOverrides` set for a non-existent component → no-op; the override map is read only by the components that look for their key. +- `PackageValues` for a non-existent Package → no-op; the operator has no Package to merge it into, and merge keys for absent components are ignored. ## Testing - **Helm unit tests** (`make unit-tests`): bundle wiring for the customizer (enabled / disabled / `disabledPackages`) — already in the prototype; add assertions that lock the RBAC surface (no `delete` on Packages, source write excluded from `cozy-system`). -- **Helm unit tests** for `packageOverrides` merge precedence on at least one downstream package (e.g. linstor). +- **Operator unit tests** for `PackageValues` merge precedence on at least one downstream package (e.g. linstor): map keys merge deep, list keys follow the documented policy, and an absent `PackageValues` leaves `hr.Spec.Values` byte-identical to today. - **Helm unit tests** for the tenant `gitops` app: SA is forced to ``, sourceRef is in-namespace. -- **e2e (BATS, `hack/e2e-apps/`):** enable the admin surface against a dev cluster, push a `packageOverrides` change, assert the downstream HelmRelease re-renders; create a tenant `gitops` app, assert it reconciles an Application CR and is denied a cross-namespace ref. +- **e2e (BATS, `hack/e2e-apps/`):** enable the admin surface against a dev cluster, push a `PackageValues` change, assert the downstream HelmRelease re-renders; create a tenant `gitops` app, assert it reconciles an Application CR and is denied a cross-namespace ref. - **Manual:** `kubectl auth can-i --as=...` matrix for both SAs (already done for the admin SA in #2731); verify the tenant SA cannot reconcile outside its namespace. ## Rollout 1. **Tool 1, Phase 1** — admin customizer system package + ownership-model docs ([#2731](https://github.com/cozystack/cozystack/pull/2731), in review). Narrow the two source/field footguns noted under Security. -2. **Tool 1, Phase 2** — `packageOverrides` admin override layer. Highest leverage: unblocks safe component-parameter changes (UC1b) and whole-cluster reproducibility (UC5). +2. **Tool 1, Phase 2** — the `PackageValues` admin override resource (new API kind + operator merge). Highest leverage: unblocks safe component-parameter changes (UC1b) and whole-cluster reproducibility (UC5). If the new kind needs to wait, the `packageOverrides` chart-only alternative can land first as an interim. 3. **Tool 2, Phases 1–2** — tenant `gitops` app + tenant kustomize-controller shard with lockdown. The largest user-visible win. 4. **Cross-cutting** — secrets-as-code + dashboard sync-status views. 5. **Hardening** — field-ownership `ValidatingAdmissionPolicy` (separate proposal) and policy-as-code guardrails. @@ -192,14 +201,15 @@ This is the heart of the proposal; each tier has a distinct, **explicit** trust 1. **`packages/system/fluxcd` vs `internal/fluxinstall`** — is the flux-operator-managed `FluxInstance` (`multitenant: false`) a separate reconciliation path from the operator-embedded controllers? This determines exactly which instance serves tenant Kustomizations and where the lockdown flags belong. 2. **Tenant raw resources** — should the tenant SA ever hold raw `HelmRelease` / CR create rights, or stay confined to the curated Application abstraction? Affects how much of UC2 is "BYO manifests" vs "catalog apps from git." -3. **Override layer shape** — `packageOverrides` on the bootstrap Package vs a dedicated `cozystack-overrides` Secret. The former keeps a single object; the latter decouples override lifecycle from the bootstrap Package. +3. **`PackageValues` merge precedence and binding** — the precedence chain (chart defaults < `cozystack-values` < `Package` < `PackageValues`) and the list-vs-map merge policy need to be fixed and documented; and `PackageValues` must bind to its target Package (1:1 by name, as drafted, vs an explicit `spec.packageRef`). Secondary: whether to also keep the `packageOverrides` chart layer as a documented convenience or drop it once `PackageValues` lands. 4. **Tenant GitOps delivery** — catalog app (recommended) vs a `spec.gitops` field on the `Tenant` CR. The app avoids an API change; the field is more discoverable. ## Alternatives considered - **Runtime mechanism — give admins/tenants raw Flux CRDs via RBAC, no chart.** Rejected: no curated trust tiers, no isolation defaults, and tenants would need source/kustomize RBAC plus a cross-namespace lockdown anyway. The primitive-as-a-package gives a consistent UX and safe defaults for the same underlying objects. -- **Field ownership — solve UC1b with RBAC (`resourceNames` allow/deny on Packages).** Rejected: RBAC is object-level, but the contract is field-level (admin owns `spec.components.*.values`, chart owns `spec.variant` and structural fields). Denying the object re-introduces hand-management of the parameters admins most want; allowing it races helm-controller. The override layer plus a field-level admission policy is the correct axis. -- **Field ownership — bespoke admission webhook now.** Deferred in favor of a `ValidatingAdmissionPolicy` (native, CEL, no webhook server) and, before either, the `packageOverrides` layer that avoids the contested objects entirely. +- **Field ownership — solve UC1b with RBAC (`resourceNames` allow/deny on Packages).** Rejected: RBAC is object-level, but on a single Package the contract is field-level (admin owns `spec.components.*.values`, chart owns `spec.variant` and structural fields), and RBAC cannot express that. Splitting overrides into a separate `PackageValues` resource turns the field-level contract back into an object-level one — write `PackageValues`, read-only `Package` — which RBAC *can* enforce. Denying the chart-owned Package object outright would instead re-introduce hand-management of the parameters admins most want. +- **Override layer — `packageOverrides` on the bootstrap Package (chart-only, no API change).** The override map lives under `spec.components.platform.values` and the platform chart merges it into each downstream Package at render time. Considered as the primary mechanism and kept as a lighter-weight alternative (see the Admin override layer section): it ships without touching the API but sidesteps the force-ownership race rather than removing it, gives all-or-nothing RBAC on one god-object, and pushes merge correctness into per-package templates. `PackageValues` is preferred; `packageOverrides` remains a valid interim because it is additive and independent. +- **Field ownership — bespoke admission webhook now.** Deferred in favor of a `ValidatingAdmissionPolicy` (native, CEL, no webhook server) and, before either, the `PackageValues` split that keeps admins off the contested objects entirely. - **Tenant isolation — flip global Flux multitenancy.** Rejected as the default: it constrains the platform's own Kustomizations. A dedicated tenant shard matches the existing `flux-tenants` pattern and limits the blast radius. - **Tenant delivery — extend the `Tenant` CRD with a `gitops` field.** Viable, kept as an open question; the catalog app is preferred to avoid a core API change and to make the surface opt-in per tenant. - **One monolithic tool for all cases.** Rejected: admin and tenant surfaces have fundamentally different trust models, RBAC, and isolation requirements. They share a primitive, not an implementation. From 771ebec49621a8e150df681950698da677f3b82f Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 6 Jul 2026 18:08:04 +0500 Subject: [PATCH 3/5] docs(gitops-surfaces): refactor tenant isolation onto flux-shard-operator The flux-tenants Deployment is retired; helm-controller sharding is now driven by flux-shard-operator (an auto-scaled shard pool), and there is no kustomize-controller sharding. Rework Tool 2 and Context accordingly: - Context: describe the embedded flux-aio control plane and flux-shard-operator; note tenants hold no Flux-CRD RBAC today, so Tool 2 is what creates the surface (and thus the isolation problem). - Tool 2 isolation: a dedicated kustomize-controller + SA per gitops-enabled tenant (a separate sharding path, not the helm-controller pool), a fail-closed mutating webhook that stamps the shard label + serviceAccountName, and a ValidatingAdmissionPolicy backstop. Enforced by construction and shipped with the surface, not deferred hardening. - Resolve Open Question 1: management-plane tenant reconciliation is the embedded flux-aio; the FluxInstance/multitenant lever governs child Kubernetes clusters only (migration 21 retired it here). - Disambiguate the deferred Tool 1 Package field-ownership VAP from the in-scope Tool 2 Kustomization admission enforcement. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Myasnikov Daniil --- design-proposals/gitops-surfaces/README.md | 39 ++++++++++++---------- 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/design-proposals/gitops-surfaces/README.md b/design-proposals/gitops-surfaces/README.md index 06cfd00..6c4e368 100644 --- a/design-proposals/gitops-surfaces/README.md +++ b/design-proposals/gitops-surfaces/README.md @@ -14,7 +14,7 @@ This proposal defines a single primitive — **a reconciled GitOps surface** (`G ## Scope and related proposals - **First increment — admin surface:** [cozystack/cozystack#2731](https://github.com/cozystack/cozystack/pull/2731) (`feat(platform): add cozystack.customizer system package`) implements Tool 1, Phase 1. This proposal generalizes that PR and gives it a home in the larger model. -- **Deferred to follow-up proposals** (intentionally not designed in full here): field-ownership enforcement via `ValidatingAdmissionPolicy` / admission webhook on `packages.cozystack.io` (sketched under Security; full design is separate); and a delegated mid-tier between cluster-admin and tenant (team-scoped admins). +- **Deferred to follow-up proposals** (intentionally not designed in full here): **Package** field-ownership enforcement via `ValidatingAdmissionPolicy` / admission webhook on `packages.cozystack.io` (Tool 1; sketched under Security, full design separate) — this is distinct from the tenant-`Kustomization` admission enforcement in Tool 2, which is **in-scope** and ships with that surface; and a delegated mid-tier between cluster-admin and tenant (team-scoped admins). - This proposal should land before the tenant surface (Tool 2) is implemented, because the unifying primitive and the multitenancy-lockdown decision are shared. ## Context @@ -26,7 +26,8 @@ Cozystack delivers everything through Flux. The relevant pieces today: - **The bootstrap Package** `cozystack.cozystack-platform` is **created by hand** at install time — operators copy `packages/core/installer/example/platform.yaml`, edit `spec.variant` and `spec.components.platform.values`, and `kubectl apply` it. No controller owns its spec. - **Downstream Packages** (`cozystack.linstor`, `cozystack.cilium`, …) are **rendered by the platform chart** (`packages/core/platform`) with `helm.sh/resource-policy: keep` and are owned by helm-controller. - **Most admin config is Package values, not raw resources.** VM golden images live in `packages/system/vm-default-images` values; LINSTOR parameters in `packages/system/linstor` values; cert-manager `ClusterIssuer`s are driven by `packages/core/platform` values → the `cozystack-values` Secret in `cozy-system` → the `cert-manager-issuers` chart. -- **Tenants cannot self-GitOps.** A `Tenant` (`apps.cozystack.io`) produces a `tenant-` namespace, a ServiceAccount named ``, and `cozy:tenant:*` ClusterRoles. Tenants create curated **Application** CRs (e.g. Postgres, Kafka, VM) which the aggregated apiserver converts into HelmReleases reconciled by a sharded helm-controller (`sharding.fluxcd.io/key: tenants`). Tenants have **no RBAC for `source`/`kustomize` CRDs** and there is **no tenant-scoped source/kustomize reconciliation**. The shipped `flux-instance` sets `multitenant: false` (`packages/system/fluxcd/charts/flux-instance/values.yaml`). +- **Management-plane Flux is the embedded flux-aio, and helm-controller is already sharded by an operator.** The management cluster runs a single all-in-one `flux` Deployment installed from `internal/fluxinstall`; its source-, kustomize-, and helm-controllers carry `--watch-label-selector=!sharding.fluxcd.io/key`, so any object *labelled* with that key is skipped by the unrestricted main plane. **flux-shard-operator** provisions auto-scaled `helm-controller-shard` Deployments (each `--watch-label-selector=sharding.fluxcd.io/key=shard`), assigns each tenant to a shard, and, via a mutating webhook, rewrites the birth label `sharding.fluxcd.io/key: tenants` on tenant HelmReleases to the assigned `shard`. It has **retired the old hand-rolled `flux-tenants` Deployment** (migration 44) — so tenant HelmReleases are no longer one-controller-per-tenant but packed onto a shared, auto-scaled pool. Crucially, **there is no kustomize-controller sharding**: a `Kustomization` *without* the shard label is reconciled by the unrestricted main kustomize-controller. +- **Tenants cannot self-GitOps.** A `Tenant` (`apps.cozystack.io`) produces a `tenant-` namespace, a ServiceAccount named ``, and `cozy:tenant:*` ClusterRoles. Tenants create curated **Application** CRs (e.g. Postgres, Kafka, VM) which the aggregated apiserver converts into the HelmReleases above. Tenants have **no RBAC for `source`/`kustomize`/`helm` toolkit CRDs** and there is **no tenant-scoped source/kustomize reconciliation** — the only isolation a tenant has today is precisely that it cannot author Flux objects at all. (The flux-operator `FluxInstance` with `multitenant: false` in `packages/system/fluxcd` governs **child Kubernetes clusters**, not this management plane — migration 21 retired the FluxInstance path here in favor of the embedded flux-aio. This resolves Open Question 1 below.) ### The problem @@ -72,8 +73,8 @@ flowchart TD T1[Application CRs
namespaced manifests] end - A1 -->|GitRepository + Kustomization
SA: cozystack-customizer| KC[(kustomize-controller
cozy-fluxcd)] - T1 -->|GitRepository + Kustomization
SA: tenant-name| KCT[(kustomize-controller
tenant shard)] + A1 -->|GitRepository + Kustomization
SA: cozystack-customizer| KC[(main kustomize-controller
cozy-fluxcd)] + T1 -->|GitRepository + Kustomization
SA: tenant-name| KCT[(dedicated kustomize-controller
per gitops-enabled tenant
forced SA + no-cross-ns-refs)] KC -->|cluster scope| CLUSTER[(Management cluster
Packages, cluster resources)] KCT -->|namespace scope only| NS[(tenant-name namespace)] @@ -133,17 +134,18 @@ The split is **additive**: with no `PackageValues` present, the operator path is ### Tool 2 — Tenant GitOps (tenant, namespace scope) -Delivered as a **catalog app**, not a `Tenant` API change — it is opt-in per tenant, requires no operator change, and fits the existing app model. +Delivered as a **catalog app**, not a `Tenant` API change — it is opt-in per tenant and fits the existing app model. Unlike Tool 1 it is *not* a no-op on the control plane: tenants hold **no** RBAC for Flux CRDs today, so the surface must grant Kustomization write, and granting it is exactly what opens the isolation problem below. The enforcement therefore ships **with** the surface, not as later hardening. -- New app package `packages/apps/gitops` + ApplicationDefinition in `packages/system/gitops-rd`. When a tenant instantiates it, the chart renders, in the tenant namespace, a `GitRepository` + `Kustomization` with `spec.serviceAccountName: ` — the tenant's **existing** SA, which already holds `cozy:tenant:admin`. -- Because reconciliation runs as the tenant SA, isolation is **RBAC-enforced by construction**: a tenant can only create what `cozy:tenant:*` permits, inside its own namespace subtree. Tenants point the surface at a repo of Application CRs and namespaced manifests. +- New app package `packages/apps/gitops` + ApplicationDefinition in `packages/system/gitops-rd`. When a tenant instantiates it, the chart renders, in the tenant namespace, a `GitRepository` + `Kustomization` bound to a tenant-scoped GitOps `ServiceAccount` (carrying `cozy:tenant:*`), and the tenant is provisioned **its own dedicated kustomize-controller**. +- Tenants point the surface at a repo of Application CRs and namespaced manifests. Because every reconcile runs as the tenant's SA, a tenant can only create what `cozy:tenant:*` permits inside its own namespace subtree — but that guarantee holds only if nothing lets a tenant Kustomization escape onto the unrestricted main controller or borrow a foreign SA (see below). -#### Isolation hardening +#### Isolation: a dedicated controller + SA per tenant, enforced (not by discipline) -A tenant must not be able to (a) reconcile into another namespace, or (b) borrow a more privileged SA. Two options, recommendation first: +The helm-controller shard pool can pack many tenants onto a few auto-scaled shards because it *impersonates* nothing — tenants never author HelmReleases directly. Kustomizations are different: the surface hands tenants the right to create them, and a kustomize-controller reconciles their contents **as a named SA**. A single shared, pooled controller would therefore have to impersonate every tenant's SA — a fat, cross-tenant privilege and a single blast radius for all tenants. So the Kustomization path is deliberately **not** the helm-controller pool; it is a **separate sharding path with its own provisioning and its own logic**, running alongside (never merged into) the helm-controller shard machinery. flux-shard-operator already owns an equivalent placement/provisioning/webhook skeleton for helm-controller, so this reuses the pattern rather than inventing one. -1. **Dedicated tenant kustomize-controller shard** (recommended) — mirror the existing `flux-tenants` helm-controller shard. Run a kustomize-controller selecting `sharding.fluxcd.io/key: tenants`, started with `--no-cross-namespace-refs` and a default-service-account lockdown, so the restrictions apply *only* to tenant reconciliation and platform Kustomizations are unaffected. This matches a pattern Cozystack already uses. -2. **Global Flux multitenancy flags** — flip the `flux-instance` `multitenant` setting and apply `--no-cross-namespace-refs` cluster-wide. Simpler, but it constrains the platform's own Kustomizations and is a larger blast radius. +1. **A dedicated kustomize-controller + SA per gitops-enabled tenant.** Enabling the app provisions a kustomize-controller scoped to that tenant, watching only that tenant's shard label, started with `--no-cross-namespace-refs`, pinned so every Kustomization reconciles as the tenant's own SA. It is opt-in, so the controller count tracks only the tenants who use GitOps — not every tenant — which is what makes a per-tenant (rather than pooled) controller affordable here. +2. **A separate mutating webhook** (distinct from the helm-controller shard webhook) stamps the tenant's shard label **and** forces `spec.serviceAccountName` on Kustomizations created in tenant namespaces. Because the surface grants tenants Kustomization write, this webhook is a **security boundary**, so it runs **fail-closed** (`failurePolicy: Fail`) — unlike the helm-controller webhook, which can fail-open safely since tenants cannot author HelmReleases. +3. **A `ValidatingAdmissionPolicy` as an independent backstop.** The VAP rejects any tenant-namespace Kustomization missing the shard label or naming a foreign SA, so the boundary does not rest on a single webhook. Webhook (mutate → correct) plus VAP (validate → reject) close the "omit the label to fall through to the unrestricted controller" and "borrow a privileged SA" evasions **by construction**, rather than documenting them as advisory. ### Cross-cutting capabilities @@ -169,7 +171,7 @@ This is the heart of the proposal; each tier has a distinct, **explicit** trust - **Tool 1 (admin):** repo write == cluster-admin, by design and by documentation. The curated ClusterRole in the prototype (patch-not-delete on Packages, no CRDs, etc.) is *defense against accident, not against intent* — anyone with repo write can escalate, so the docs must not over-claim a boundary. New trust surface introduced: an admin git repo + its credentials (protect like a kubeconfig), and a standing, self-healing reconciling credential whose blast radius equals an admin's. - **Tool 1 footguns to narrow** (they cost nothing functionally): source access should not let the customizer rewrite the platform's *own* supply chain (the `cozystack-packages` OCIRepository in `cozy-system`), so scope write away from `cozy-system`; and field ownership on downstream Packages should eventually be enforced by a `ValidatingAdmissionPolicy` scoped to the customizer SA that rejects changes to fields currently managed by helm-controller (detected via `managedFields`), leaving the hand-owned bootstrap Package free. Until then, the `PackageValues` resource keeps admins off chart-owned objects entirely — they write `PackageValues`, never the chart-owned `Package` — so the contract is structurally avoided rather than merely documented. -- **Tool 2 (tenant):** the surface runs as the tenant's own SA, so it can do nothing the tenant cannot already do via the API. The new risk is **cross-namespace / SA-borrowing**, closed by the tenant shard lockdown (`--no-cross-namespace-refs`, forced `serviceAccountName`). Tenant-supplied input = the contents of a tenant-owned repo, bounded by `cozy:tenant:*`. +- **Tool 2 (tenant):** the surface runs as the tenant's own SA, so it can do nothing the tenant cannot already do via the API — *once the routing is enforced*. The subtlety is that tenants hold **no** Flux-CRD RBAC today, so Tool 2 introduces the write path; the new risk is a tenant **omitting the shard label** (falling through to the unrestricted main kustomize-controller) or **naming a foreign SA**. Both are closed at admission by the per-tenant dedicated controller (`--no-cross-namespace-refs`, pinned SA), a fail-closed mutating webhook that stamps the label + `serviceAccountName`, and a `ValidatingAdmissionPolicy` backstop. This enforcement is a **prerequisite of the surface, not deferred hardening** — it must land in the same increment that grants tenants Kustomization write. Tenant-supplied input = the contents of a tenant-owned repo, bounded by `cozy:tenant:*`. - **Secrets:** both tiers may pull git credentials and SOPS keys. These are admin/tenant pre-created Secrets; the platform never generates or owns them. ## Failure and edge cases @@ -177,7 +179,7 @@ This is the heart of the proposal; each tier has a distinct, **explicit** trust - Missing `source.url` / `kustomization.path` when enabled → chart `fail`s at render; Flux surfaces the error on HelmRelease status (already implemented in the prototype). - Private repo without a `secretRef` → GitRepository reports auth failure on its status; no partial apply. - Admin manifest declares a chart-owned Package field directly (bypassing `PackageValues`) → silently claimed from helm-controller until the admission policy lands; documented as advisory in the interim. -- Tenant Kustomization references a source in another namespace or a foreign SA → rejected by the tenant shard lockdown. +- Tenant Kustomization omits the shard label, references a source in another namespace, or names a foreign SA → the mutating webhook stamps the label + SA and the VAP rejects a violation, so it never reaches the unrestricted main kustomize-controller; the per-tenant controller's `--no-cross-namespace-refs` blocks the cross-namespace ref. - Customizer disabled then re-enabled → resources re-adopt cleanly (SSA, `resource-policy: keep`); no duplicate-creation errors. - `PackageValues` for a non-existent Package → no-op; the operator has no Package to merge it into, and merge keys for absent components are ignored. @@ -185,21 +187,21 @@ This is the heart of the proposal; each tier has a distinct, **explicit** trust - **Helm unit tests** (`make unit-tests`): bundle wiring for the customizer (enabled / disabled / `disabledPackages`) — already in the prototype; add assertions that lock the RBAC surface (no `delete` on Packages, source write excluded from `cozy-system`). - **Operator unit tests** for `PackageValues` merge precedence on at least one downstream package (e.g. linstor): map keys merge deep, list keys follow the documented policy, and an absent `PackageValues` leaves `hr.Spec.Values` byte-identical to today. -- **Helm unit tests** for the tenant `gitops` app: SA is forced to ``, sourceRef is in-namespace. -- **e2e (BATS, `hack/e2e-apps/`):** enable the admin surface against a dev cluster, push a `PackageValues` change, assert the downstream HelmRelease re-renders; create a tenant `gitops` app, assert it reconciles an Application CR and is denied a cross-namespace ref. +- **Helm / webhook / policy unit tests** for the tenant `gitops` app: the rendered `Kustomization` carries the tenant shard label and `spec.serviceAccountName: `, and sourceRef is in-namespace; the mutating webhook stamps a label-less tenant Kustomization; the VAP rejects one that omits the shard label or names a foreign SA. +- **e2e (BATS, `hack/e2e-apps/`):** enable the admin surface against a dev cluster, push a `PackageValues` change, assert the downstream HelmRelease re-renders; create a tenant `gitops` app, assert it reconciles an Application CR, is denied a cross-namespace ref, and — the key isolation test — assert a Kustomization created without the shard label or with a foreign `serviceAccountName` is rejected at admission (never reaching the unrestricted main controller). - **Manual:** `kubectl auth can-i --as=...` matrix for both SAs (already done for the admin SA in #2731); verify the tenant SA cannot reconcile outside its namespace. ## Rollout 1. **Tool 1, Phase 1** — admin customizer system package + ownership-model docs ([#2731](https://github.com/cozystack/cozystack/pull/2731), in review). Narrow the two source/field footguns noted under Security. 2. **Tool 1, Phase 2** — the `PackageValues` admin override resource (new API kind + operator merge). Highest leverage: unblocks safe component-parameter changes (UC1b) and whole-cluster reproducibility (UC5). If the new kind needs to wait, the `packageOverrides` chart-only alternative can land first as an interim. -3. **Tool 2, Phases 1–2** — tenant `gitops` app + tenant kustomize-controller shard with lockdown. The largest user-visible win. +3. **Tool 2, Phase 1** — tenant `gitops` app shipped **together with** its enforcement: the per-tenant dedicated kustomize-controller + SA (a separate sharding path from the helm-controller pool), the fail-closed mutating webhook (label + `serviceAccountName`), and the `ValidatingAdmissionPolicy` backstop. This is not a later phase — granting tenants Kustomization write *without* it is the isolation hole, so they ship as one unit. The largest user-visible win. 4. **Cross-cutting** — secrets-as-code + dashboard sync-status views. 5. **Hardening** — field-ownership `ValidatingAdmissionPolicy` (separate proposal) and policy-as-code guardrails. ## Open questions -1. **`packages/system/fluxcd` vs `internal/fluxinstall`** — is the flux-operator-managed `FluxInstance` (`multitenant: false`) a separate reconciliation path from the operator-embedded controllers? This determines exactly which instance serves tenant Kustomizations and where the lockdown flags belong. +1. **Resolved (during review).** Management-plane tenant reconciliation is served by the **embedded flux-aio** (`internal/fluxinstall`), whose main controllers already carry `--watch-label-selector=!sharding.fluxcd.io/key`; the flux-operator-managed `FluxInstance` / `multitenant: false` in `packages/system/fluxcd` applies to **child Kubernetes clusters** only (migration 21 retired the FluxInstance path on the management cluster). Lockdown flags for the tenant surface therefore live on the **dedicated per-tenant kustomize-controller** the operator provisions (Tool 2), not on a global `FluxInstance`. 2. **Tenant raw resources** — should the tenant SA ever hold raw `HelmRelease` / CR create rights, or stay confined to the curated Application abstraction? Affects how much of UC2 is "BYO manifests" vs "catalog apps from git." 3. **`PackageValues` merge precedence and binding** — the precedence chain (chart defaults < `cozystack-values` < `Package` < `PackageValues`) and the list-vs-map merge policy need to be fixed and documented; and `PackageValues` must bind to its target Package (1:1 by name, as drafted, vs an explicit `spec.packageRef`). Secondary: whether to also keep the `packageOverrides` chart layer as a documented convenience or drop it once `PackageValues` lands. 4. **Tenant GitOps delivery** — catalog app (recommended) vs a `spec.gitops` field on the `Tenant` CR. The app avoids an API change; the field is more discoverable. @@ -210,7 +212,8 @@ This is the heart of the proposal; each tier has a distinct, **explicit** trust - **Field ownership — solve UC1b with RBAC (`resourceNames` allow/deny on Packages).** Rejected: RBAC is object-level, but on a single Package the contract is field-level (admin owns `spec.components.*.values`, chart owns `spec.variant` and structural fields), and RBAC cannot express that. Splitting overrides into a separate `PackageValues` resource turns the field-level contract back into an object-level one — write `PackageValues`, read-only `Package` — which RBAC *can* enforce. Denying the chart-owned Package object outright would instead re-introduce hand-management of the parameters admins most want. - **Override layer — `packageOverrides` on the bootstrap Package (chart-only, no API change).** The override map lives under `spec.components.platform.values` and the platform chart merges it into each downstream Package at render time. Considered as the primary mechanism and kept as a lighter-weight alternative (see the Admin override layer section): it ships without touching the API but sidesteps the force-ownership race rather than removing it, gives all-or-nothing RBAC on one god-object, and pushes merge correctness into per-package templates. `PackageValues` is preferred; `packageOverrides` remains a valid interim because it is additive and independent. - **Field ownership — bespoke admission webhook now.** Deferred in favor of a `ValidatingAdmissionPolicy` (native, CEL, no webhook server) and, before either, the `PackageValues` split that keeps admins off the contested objects entirely. -- **Tenant isolation — flip global Flux multitenancy.** Rejected as the default: it constrains the platform's own Kustomizations. A dedicated tenant shard matches the existing `flux-tenants` pattern and limits the blast radius. +- **Tenant isolation — flip global Flux multitenancy.** Rejected as the default: on the management plane it would constrain the platform's own Kustomizations (and the `FluxInstance` `multitenant` lever governs child clusters, not this plane anyway). A dedicated per-tenant kustomize-controller + SA, with a fail-closed webhook and a VAP backstop, confines the blast radius to the opted-in tenant and enforces isolation at admission. +- **Tenant isolation — reuse the helm-controller shard pool for Kustomizations.** Rejected: a shared, pooled kustomize-controller would need to impersonate every tenant's SA (helm-controller impersonates none, because tenants never author HelmReleases). A per-tenant controller keeps a 1:1 SA binding; opt-in keeps the count bounded. - **Tenant delivery — extend the `Tenant` CRD with a `gitops` field.** Viable, kept as an open question; the catalog app is preferred to avoid a core API change and to make the surface opt-in per tenant. - **One monolithic tool for all cases.** Rejected: admin and tenant surfaces have fundamentally different trust models, RBAC, and isolation requirements. They share a primitive, not an implementation. From 1620d3bc58627558ef376bf9dbe31724ebaf9c84 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 6 Jul 2026 18:26:36 +0500 Subject: [PATCH 4/5] docs(gitops-surfaces): keep admin overrides permissive, drop packageOverrides MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address the remaining review points as deliberate design decisions: - Package field-ownership: the direct-edit path on chart-owned Packages stays open by design (it is a cluster-admin tool; locking it risks removing tuning knobs admins can reach today). PackageValues is the recommended non-racing path, not a cage; the field-ownership VAP is an optional opt-out guard, never a prerequisite. Reconcile the "structurally removed" vs "advisory" wording toward this. - cozy-system: do not scope customizer writes away from cozy-system — same logic as Flux managing its own manifests. Supply-chain guards are opt-out, not a default namespace exclusion. - PackageValues merge policy fixed (maps deep-merge, lists replace wholesale; the vm-default-images image list is the footgun). packageOverrides is dropped in favor of PackageValues as the single override mechanism. - Resolve Open Question 4: tenant GitOps ships as a catalog app, not a Tenant CRD field. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Myasnikov Daniil --- design-proposals/gitops-surfaces/README.md | 30 +++++++++++----------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/design-proposals/gitops-surfaces/README.md b/design-proposals/gitops-surfaces/README.md index 6c4e368..dd5fa36 100644 --- a/design-proposals/gitops-surfaces/README.md +++ b/design-proposals/gitops-surfaces/README.md @@ -99,7 +99,7 @@ The single most important design point. RBAC is object-level; the contract we ne | Force-ownership risk | **none** — nothing competes | **real** — claiming a chart-owned field steals it | - **Bootstrap Package:** because no controller owns its spec, the admin repo can simply **own it**. The recommended flow is *self-adoption*: bootstrap once by hand with the customizer enabled, then commit the same Package manifest to the repo; from then on the repo is its source of truth (the standard `flux bootstrap` pattern). Use Server-Side Apply with a *partial* Package manifest to own only the fields you manage. -- **Downstream Packages:** patching them races helm-controller. We do **not** solve this by denying access (that would re-introduce hand-management of exactly the parameters admins care about). We solve it with an override layer (below) and, later, an admission policy. +- **Downstream Packages:** patching them races helm-controller. We do **not** solve this by denying access — that would re-introduce hand-management of exactly the parameters admins care about, and this is an admin tool. We solve it with an override layer (below); a field-ownership admission policy is an *optional* guard for teams that want one, never a wall that removes reachable knobs. #### Admin override layer (for UC1b — component parameters) @@ -126,11 +126,11 @@ spec: minutes: 10 ``` -Because the chart never writes `PackageValues` and the admin never writes the chart-owned `Package`, **there is no field for the admin and helm-controller to fight over** — the force-ownership race is removed structurally, not merely sidestepped. This is *not* a sibling `cozystack-values`-style Secret (the operator hardcodes a single `valuesFrom` and resets others); `PackageValues` is a first-class CR the operator itself merges into `hr.Spec.Values`, so it cooperates with the existing reconcile path. +Because the chart never writes `PackageValues`, and an admin using it never needs to touch the chart-owned `Package`, **there is no field for the admin and helm-controller to fight over on the override path** — the force-ownership race is removed for that path, not merely sidestepped. Admins are not *prevented* from editing a chart-owned `Package` directly — that stays possible by design (see the security posture and edge cases); `PackageValues` simply offers a first-class path that does not race. This is *not* a sibling `cozystack-values`-style Secret (the operator hardcodes a single `valuesFrom` and resets others); `PackageValues` is a first-class CR the operator itself merges into `hr.Spec.Values`, so it cooperates with the existing reconcile path. -The split is **additive**: with no `PackageValues` present, the operator path is byte-identical to today, so existing clusters are unaffected (see Upgrade compatibility). It also makes ownership enforceable by ordinary object-level RBAC — the admin GitOps SA gets write on `PackageValues` and read-only on `Package` — which is the discipline that keeps admins off chart-owned objects entirely, the foundation for whole-cluster reproducibility (UC5). +The split is **additive**: with no `PackageValues` present, the operator path is byte-identical to today, so existing clusters are unaffected (see Upgrade compatibility). It also makes ownership *expressible* with ordinary object-level RBAC — the admin GitOps SA can be granted write on `PackageValues` and read-only on `Package` — an opt-in discipline for teams that want their overrides confined to the human-owned layer, and the foundation for whole-cluster reproducibility (UC5). It is a discipline, not a cage: an admin who needs a knob that lives only on the chart-owned `Package` can still be granted it. -*Alternative — `packageOverrides` on the bootstrap Package.* The same overrides can instead be carried as a `packageOverrides.` map under `spec.components.platform.values` on the hand-owned bootstrap Package, with the platform chart merging them into each downstream Package at render time. This ships with no API change (pure chart templating) and collapses all admin config into one object, but it sidesteps the race rather than removing it, gives all-or-nothing RBAC on one god-object, and pushes list-vs-map merge correctness into per-package templates. Kept as a lighter-weight fallback; since `PackageValues` is additive, `packageOverrides` could even ship first as an interim. +**Merge policy (fixed).** `PackageValues` merges over the Package's values with the precedence `chart defaults < cozystack-values < Package < PackageValues`. Maps merge deep; **lists replace wholesale** (standard Helm semantics) — so an override that sets one entry of a list (e.g. a single image under `vm-default-images`) replaces the entire list, and a caller who means to *add* one must restate the full list. Keyed or append-merge for a specific list is an explicit per-package opt-in, never implicit. `PackageValues` is the **single** override mechanism: the earlier `packageOverrides`-on-the-bootstrap-Package sketch is dropped in its favor (it collapsed all config onto one god-object with all-or-nothing RBAC and pushed this list-merge correctness into per-package templates — see Alternatives). ### Tool 2 — Tenant GitOps (tenant, namespace scope) @@ -161,7 +161,7 @@ The helm-controller shard pool can pack many tenants onto a few auto-scaled shar ## Upgrade and rollback compatibility - Fully **additive and opt-in**; existing clusters are unaffected until an operator enables a surface. No migration required. -- The `PackageValues` resource is additive: with none present, the operator builds HelmReleases exactly as today, so charts use their current defaults. (The `packageOverrides` alternative is equally backward compatible — absent the key, charts use their current defaults.) +- The `PackageValues` resource is additive: with none present, the operator builds HelmReleases exactly as today, so charts use their current defaults. - **Rollback:** disabling a surface stops emitting its resources. Note that platform Packages carry `helm.sh/resource-policy: keep`, so the customizer Package and its child objects are **not** auto-deleted on disable — teardown is a documented manual step (this is existing platform behavior, called out here so it is not surprising). - Self-adoption of the bootstrap Package is reversible: stop reconciling it from the repo and resume hand-management; nothing about the object changes. @@ -170,7 +170,7 @@ The helm-controller shard pool can pack many tenants onto a few auto-scaled shar This is the heart of the proposal; each tier has a distinct, **explicit** trust boundary. - **Tool 1 (admin):** repo write == cluster-admin, by design and by documentation. The curated ClusterRole in the prototype (patch-not-delete on Packages, no CRDs, etc.) is *defense against accident, not against intent* — anyone with repo write can escalate, so the docs must not over-claim a boundary. New trust surface introduced: an admin git repo + its credentials (protect like a kubeconfig), and a standing, self-healing reconciling credential whose blast radius equals an admin's. -- **Tool 1 footguns to narrow** (they cost nothing functionally): source access should not let the customizer rewrite the platform's *own* supply chain (the `cozystack-packages` OCIRepository in `cozy-system`), so scope write away from `cozy-system`; and field ownership on downstream Packages should eventually be enforced by a `ValidatingAdmissionPolicy` scoped to the customizer SA that rejects changes to fields currently managed by helm-controller (detected via `managedFields`), leaving the hand-owned bootstrap Package free. Until then, the `PackageValues` resource keeps admins off chart-owned objects entirely — they write `PackageValues`, never the chart-owned `Package` — so the contract is structurally avoided rather than merely documented. +- **Tool 1 posture (deliberately permissive — it is an admin tool):** the customizer is a cluster-admin-equivalent credential, so by design it manages `cozy-system` and the platform's own supply chain the same way Flux manages its own manifests — this is intentional, not a boundary to wall off. Optional, **opt-out** guards exist for teams that want them: a `ValidatingAdmissionPolicy` scoped to the customizer SA could protect specific supply-chain objects (the `cozystack-packages` OCIRepository) or reject changes to helm-controller-managed `Package` fields (detected via `managedFields`). These are **defense against accident, not prerequisites** — locking chart-owned `Package` fields by default would risk removing tuning knobs admins can reach today, so the direct-edit path stays open and `PackageValues` is the recommended non-racing path rather than a wall. - **Tool 2 (tenant):** the surface runs as the tenant's own SA, so it can do nothing the tenant cannot already do via the API — *once the routing is enforced*. The subtlety is that tenants hold **no** Flux-CRD RBAC today, so Tool 2 introduces the write path; the new risk is a tenant **omitting the shard label** (falling through to the unrestricted main kustomize-controller) or **naming a foreign SA**. Both are closed at admission by the per-tenant dedicated controller (`--no-cross-namespace-refs`, pinned SA), a fail-closed mutating webhook that stamps the label + `serviceAccountName`, and a `ValidatingAdmissionPolicy` backstop. This enforcement is a **prerequisite of the surface, not deferred hardening** — it must land in the same increment that grants tenants Kustomization write. Tenant-supplied input = the contents of a tenant-owned repo, bounded by `cozy:tenant:*`. - **Secrets:** both tiers may pull git credentials and SOPS keys. These are admin/tenant pre-created Secrets; the platform never generates or owns them. @@ -178,14 +178,14 @@ This is the heart of the proposal; each tier has a distinct, **explicit** trust - Missing `source.url` / `kustomization.path` when enabled → chart `fail`s at render; Flux surfaces the error on HelmRelease status (already implemented in the prototype). - Private repo without a `secretRef` → GitRepository reports auth failure on its status; no partial apply. -- Admin manifest declares a chart-owned Package field directly (bypassing `PackageValues`) → silently claimed from helm-controller until the admission policy lands; documented as advisory in the interim. +- Admin manifest declares a chart-owned `Package` field directly (bypassing `PackageValues`) → the field is force-claimed from helm-controller, exactly as a hand-run `kubectl patch` does today. This stays available **by design** (it is an admin tool); teams that want it fenced can opt into the field-ownership VAP. `PackageValues` is the recommended non-racing path. - Tenant Kustomization omits the shard label, references a source in another namespace, or names a foreign SA → the mutating webhook stamps the label + SA and the VAP rejects a violation, so it never reaches the unrestricted main kustomize-controller; the per-tenant controller's `--no-cross-namespace-refs` blocks the cross-namespace ref. - Customizer disabled then re-enabled → resources re-adopt cleanly (SSA, `resource-policy: keep`); no duplicate-creation errors. - `PackageValues` for a non-existent Package → no-op; the operator has no Package to merge it into, and merge keys for absent components are ignored. ## Testing -- **Helm unit tests** (`make unit-tests`): bundle wiring for the customizer (enabled / disabled / `disabledPackages`) — already in the prototype; add assertions that lock the RBAC surface (no `delete` on Packages, source write excluded from `cozy-system`). +- **Helm unit tests** (`make unit-tests`): bundle wiring for the customizer (enabled / disabled / `disabledPackages`) — already in the prototype; add assertions for the RBAC surface (e.g. no `delete` on Packages). The surface is intentionally permissive on `cozy-system`, so if the optional supply-chain guard is enabled, assert it protects the `cozystack-packages` OCIRepository — it is opt-out, not a default `cozy-system` exclusion. - **Operator unit tests** for `PackageValues` merge precedence on at least one downstream package (e.g. linstor): map keys merge deep, list keys follow the documented policy, and an absent `PackageValues` leaves `hr.Spec.Values` byte-identical to today. - **Helm / webhook / policy unit tests** for the tenant `gitops` app: the rendered `Kustomization` carries the tenant shard label and `spec.serviceAccountName: `, and sourceRef is in-namespace; the mutating webhook stamps a label-less tenant Kustomization; the VAP rejects one that omits the shard label or names a foreign SA. - **e2e (BATS, `hack/e2e-apps/`):** enable the admin surface against a dev cluster, push a `PackageValues` change, assert the downstream HelmRelease re-renders; create a tenant `gitops` app, assert it reconciles an Application CR, is denied a cross-namespace ref, and — the key isolation test — assert a Kustomization created without the shard label or with a foreign `serviceAccountName` is rejected at admission (never reaching the unrestricted main controller). @@ -193,8 +193,8 @@ This is the heart of the proposal; each tier has a distinct, **explicit** trust ## Rollout -1. **Tool 1, Phase 1** — admin customizer system package + ownership-model docs ([#2731](https://github.com/cozystack/cozystack/pull/2731), in review). Narrow the two source/field footguns noted under Security. -2. **Tool 1, Phase 2** — the `PackageValues` admin override resource (new API kind + operator merge). Highest leverage: unblocks safe component-parameter changes (UC1b) and whole-cluster reproducibility (UC5). If the new kind needs to wait, the `packageOverrides` chart-only alternative can land first as an interim. +1. **Tool 1, Phase 1** — admin customizer system package + ownership-model docs ([#2731](https://github.com/cozystack/cozystack/pull/2731), in review). Ship the permissive default posture; the optional supply-chain / field-ownership guards (Security) are opt-in follow-ups. +2. **Tool 1, Phase 2** — the `PackageValues` admin override resource (new API kind + operator merge). Highest leverage: unblocks safe component-parameter changes (UC1b) and whole-cluster reproducibility (UC5). 3. **Tool 2, Phase 1** — tenant `gitops` app shipped **together with** its enforcement: the per-tenant dedicated kustomize-controller + SA (a separate sharding path from the helm-controller pool), the fail-closed mutating webhook (label + `serviceAccountName`), and the `ValidatingAdmissionPolicy` backstop. This is not a later phase — granting tenants Kustomization write *without* it is the isolation hole, so they ship as one unit. The largest user-visible win. 4. **Cross-cutting** — secrets-as-code + dashboard sync-status views. 5. **Hardening** — field-ownership `ValidatingAdmissionPolicy` (separate proposal) and policy-as-code guardrails. @@ -203,18 +203,18 @@ This is the heart of the proposal; each tier has a distinct, **explicit** trust 1. **Resolved (during review).** Management-plane tenant reconciliation is served by the **embedded flux-aio** (`internal/fluxinstall`), whose main controllers already carry `--watch-label-selector=!sharding.fluxcd.io/key`; the flux-operator-managed `FluxInstance` / `multitenant: false` in `packages/system/fluxcd` applies to **child Kubernetes clusters** only (migration 21 retired the FluxInstance path on the management cluster). Lockdown flags for the tenant surface therefore live on the **dedicated per-tenant kustomize-controller** the operator provisions (Tool 2), not on a global `FluxInstance`. 2. **Tenant raw resources** — should the tenant SA ever hold raw `HelmRelease` / CR create rights, or stay confined to the curated Application abstraction? Affects how much of UC2 is "BYO manifests" vs "catalog apps from git." -3. **`PackageValues` merge precedence and binding** — the precedence chain (chart defaults < `cozystack-values` < `Package` < `PackageValues`) and the list-vs-map merge policy need to be fixed and documented; and `PackageValues` must bind to its target Package (1:1 by name, as drafted, vs an explicit `spec.packageRef`). Secondary: whether to also keep the `packageOverrides` chart layer as a documented convenience or drop it once `PackageValues` lands. -4. **Tenant GitOps delivery** — catalog app (recommended) vs a `spec.gitops` field on the `Tenant` CR. The app avoids an API change; the field is more discoverable. +3. **`PackageValues` binding** — the merge precedence and list-vs-map policy are now fixed under *Admin override layer* (maps deep-merge, lists replace wholesale). The remaining question is binding: 1:1-by-name (as drafted) vs an explicit `spec.packageRef`. (The `packageOverrides` chart layer is dropped in favor of `PackageValues`, so it is no longer an open question.) +4. **Resolved (during review):** tenant GitOps is delivered as a **catalog app**, not a `Tenant` CRD field — it avoids core API surface and keeps the feature opt-in per tenant, decoupled from the `Tenant` lifecycle. (A `spec.gitops` field would be more discoverable but is not worth the API surface; see Alternatives.) ## Alternatives considered - **Runtime mechanism — give admins/tenants raw Flux CRDs via RBAC, no chart.** Rejected: no curated trust tiers, no isolation defaults, and tenants would need source/kustomize RBAC plus a cross-namespace lockdown anyway. The primitive-as-a-package gives a consistent UX and safe defaults for the same underlying objects. - **Field ownership — solve UC1b with RBAC (`resourceNames` allow/deny on Packages).** Rejected: RBAC is object-level, but on a single Package the contract is field-level (admin owns `spec.components.*.values`, chart owns `spec.variant` and structural fields), and RBAC cannot express that. Splitting overrides into a separate `PackageValues` resource turns the field-level contract back into an object-level one — write `PackageValues`, read-only `Package` — which RBAC *can* enforce. Denying the chart-owned Package object outright would instead re-introduce hand-management of the parameters admins most want. -- **Override layer — `packageOverrides` on the bootstrap Package (chart-only, no API change).** The override map lives under `spec.components.platform.values` and the platform chart merges it into each downstream Package at render time. Considered as the primary mechanism and kept as a lighter-weight alternative (see the Admin override layer section): it ships without touching the API but sidesteps the force-ownership race rather than removing it, gives all-or-nothing RBAC on one god-object, and pushes merge correctness into per-package templates. `PackageValues` is preferred; `packageOverrides` remains a valid interim because it is additive and independent. -- **Field ownership — bespoke admission webhook now.** Deferred in favor of a `ValidatingAdmissionPolicy` (native, CEL, no webhook server) and, before either, the `PackageValues` split that keeps admins off the contested objects entirely. +- **Override layer — `packageOverrides` on the bootstrap Package (chart-only, no API change).** The override map would live under `spec.components.platform.values` and the platform chart would merge it into each downstream Package at render time. **Rejected/dropped:** it collapses all admin config onto one god-object with all-or-nothing RBAC, sidesteps the force-ownership race rather than removing it for the override path, and pushes list-vs-map merge correctness into per-package templates. `PackageValues` supersedes it as the single override mechanism. +- **Field ownership — bespoke admission webhook now.** Deferred: a `ValidatingAdmissionPolicy` (native, CEL, no webhook server) is the better shape *if* a team opts into fencing, and `PackageValues` already gives admins a non-racing path off the contested objects without walling the direct-edit path off. - **Tenant isolation — flip global Flux multitenancy.** Rejected as the default: on the management plane it would constrain the platform's own Kustomizations (and the `FluxInstance` `multitenant` lever governs child clusters, not this plane anyway). A dedicated per-tenant kustomize-controller + SA, with a fail-closed webhook and a VAP backstop, confines the blast radius to the opted-in tenant and enforces isolation at admission. - **Tenant isolation — reuse the helm-controller shard pool for Kustomizations.** Rejected: a shared, pooled kustomize-controller would need to impersonate every tenant's SA (helm-controller impersonates none, because tenants never author HelmReleases). A per-tenant controller keeps a 1:1 SA binding; opt-in keeps the count bounded. -- **Tenant delivery — extend the `Tenant` CRD with a `gitops` field.** Viable, kept as an open question; the catalog app is preferred to avoid a core API change and to make the surface opt-in per tenant. +- **Tenant delivery — extend the `Tenant` CRD with a `gitops` field.** Viable and more discoverable, but rejected: it adds core API surface and couples the feature to the `Tenant` lifecycle. The catalog app keeps it opt-in per tenant with no API change. - **One monolithic tool for all cases.** Rejected: admin and tenant surfaces have fundamentally different trust models, RBAC, and isolation requirements. They share a primitive, not an implementation. --- From 4c47c08e2833d450e8d3824aeb7ca24a3ac00c27 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 6 Jul 2026 19:01:30 +0500 Subject: [PATCH 5/5] =?UTF-8?q?docs(gitops-surfaces):=20resolve=20Open=20Q?= =?UTF-8?q?uestion=202=20=E2=80=94=20tenants=20stay=20on=20curated=20apps?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tenants keep the curated Application abstraction: the GitOps surface grants the source/kustomize authoring needed to reconcile, but not raw HelmRelease or arbitrary-CR create rights. Since the per-tenant kustomize-controller runs as the tenant SA, "BYO from git" means Application CRs from git, not arbitrary HelmReleases — the tenant trust model is identical to today. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Myasnikov Daniil --- design-proposals/gitops-surfaces/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design-proposals/gitops-surfaces/README.md b/design-proposals/gitops-surfaces/README.md index dd5fa36..0eb4db8 100644 --- a/design-proposals/gitops-surfaces/README.md +++ b/design-proposals/gitops-surfaces/README.md @@ -202,7 +202,7 @@ This is the heart of the proposal; each tier has a distinct, **explicit** trust ## Open questions 1. **Resolved (during review).** Management-plane tenant reconciliation is served by the **embedded flux-aio** (`internal/fluxinstall`), whose main controllers already carry `--watch-label-selector=!sharding.fluxcd.io/key`; the flux-operator-managed `FluxInstance` / `multitenant: false` in `packages/system/fluxcd` applies to **child Kubernetes clusters** only (migration 21 retired the FluxInstance path on the management cluster). Lockdown flags for the tenant surface therefore live on the **dedicated per-tenant kustomize-controller** the operator provisions (Tool 2), not on a global `FluxInstance`. -2. **Tenant raw resources** — should the tenant SA ever hold raw `HelmRelease` / CR create rights, or stay confined to the curated Application abstraction? Affects how much of UC2 is "BYO manifests" vs "catalog apps from git." +2. **Resolved (during review):** tenants stay confined to the curated **Application** abstraction. The surface grants the `source`/`kustomize` authoring needed to run reconciliation, but **not** raw `HelmRelease` or arbitrary-CR create rights — `cozy:tenant:*` is unchanged there. Because the per-tenant kustomize-controller reconciles as the tenant's own SA, a tenant's git repo can only produce what `cozy:tenant:*` already permits (Application CRs plus the namespaced manifests it allows) — so "BYO from git" means Application CRs from git, not arbitrary HelmReleases. This keeps the tenant trust model identical to today; the surface only changes *how* Applications are submitted (git instead of dashboard/`kubectl`). 3. **`PackageValues` binding** — the merge precedence and list-vs-map policy are now fixed under *Admin override layer* (maps deep-merge, lists replace wholesale). The remaining question is binding: 1:1-by-name (as drafted) vs an explicit `spec.packageRef`. (The `packageOverrides` chart layer is dropped in favor of `PackageValues`, so it is no longer an open question.) 4. **Resolved (during review):** tenant GitOps is delivered as a **catalog app**, not a `Tenant` CRD field — it avoids core API surface and keeps the feature opt-in per tenant, decoupled from the `Tenant` lifecycle. (A `spec.gitops` field would be more discoverable but is not worth the API surface; see Alternatives.)