chore: Kustomize k8s restructure, API image v0.19

k8s/README.md

@@ -1,80 +1,100 @@
-# Kubernetes Deployment Files
+# High Command Kubernetes (Kustomize)
 
-This directory contains Kubernetes manifests for deploying the High Command stack (UI, API, MCP, Gateway, Cloudflare Tunnel).
+Kustomize-based deployment for the High Command stack: UI, API, Poller, PostgreSQL, Gateway, Cloudflare Tunnel.
 
-## Architecture
+## Structure
 
-Traffic flow: **Cloudflare Tunnel** → **Envoy Gateway** → **HTTPRoute** → UI/API/MCP. No nginx Ingress.
-
-## UI Files
-
-- `ui-deployment-blue.yaml`, `ui-deployment-green.yaml` - Blue/green deployments
-- `ui-service.yaml` - Service routing
-- `ui-pdb.yaml` - Pod Disruption Budget
-
-## Full Stack Files
+```
+k8s/
+├── base/                    # Base resources by component
+│   ├── api/                 # API deployments (blue/green), service, pdb
+│   ├── ui/                  # UI deployments (blue/green), service, pdb
+│   ├── poller/              # Data collector
+│   ├── postgres/            # CloudNativePG cluster, pooler
+│   ├── gateway/             # Envoy Gateway, HTTPRoute, certificate
+│   ├── mcp/                 # MCP service alias, ReferenceGrant
+│   └── cloudflare/          # Cloudflare Tunnel deployment
+├── overlays/
+│   └── default/             # Default overlay (no patches)
+├── docs/
+│   └── CLOUDFLARE_TUNNEL.md # Tunnel setup guide
+└── README.md
+```
 
-- `api-deployment-blue.yaml`, `api-deployment-green.yaml` - API deployments
-- `api-service.yaml`, `api-pdb.yaml` - API service
-- `httproute.yaml` - Gateway API HTTPRoute (/api, /claude, /mcp, /)
-- `gateway.yaml`, `gatewayclass.yaml`, `gateway-certificate.yaml` - Envoy Gateway
-- `gateway-tunnel-service.yaml` - Alias for Cloudflare Tunnel → Gateway
-- `cloudflared-tunnel-deployment.yaml` - Cloudflare Tunnel pod
-- `mcp-service.yaml`, `mcp-referencegrant.yaml`, `referencegrant.yaml` - MCP routing
+## Quick Start
 
-See `CLOUDFLARE_TUNNEL.md` for tunnel setup.
+```bash
+# 1. Create secrets (required before deploy)
+kubectl create secret generic high-command-postgres-credentials \
+  --from-literal=username=highcommand \
+  --from-literal=password='$(openssl rand -base64 32)' \
+  -n high-command
 
-## Secrets
+kubectl create secret generic high-command-api-secrets \
+  --from-literal=database-url='postgresql://user:pass@high-command-postgres-pooler.high-command.svc.cluster.local:5432/highcommand' \
+  --from-literal=claude-api-key='sk-ant-...' \
+  -n high-command
 
-**No secrets are stored in these files.**
+kubectl create secret generic high-command-poller-secrets \
+  --from-literal=database-url='postgresql://user:pass@high-command-postgres-pooler.high-command.svc.cluster.local:5432/highcommand' \
+  --from-literal=helldivers-api-base='https://api.helldivers2.dev/api/v1' \
+  --from-literal=helldivers-api-client-name='High Command' \
+  --from-literal=helldivers-api-contact='lee@fullmetal.dev' \
+  --from-literal=scrape-interval='300' \
+  -n high-command
 
-**API secrets** (required): `database-url` and optionally `claude-api-key`:
+# 2. Deploy
+kubectl apply -k overlays/default
 
-```bash
-kubectl create secret generic high-command-api-secrets \
-  --from-literal=database-url='postgresql://user:password@high-command-postgres-rw.high-command.svc.cluster.local:5432/highcommand' \
-  --from-literal=claude-api-key='sk-ant-api03-...' \
-  -n high-command
+# 3. Cloudflare Tunnel (optional, see docs/CLOUDFLARE_TUNNEL.md)
+kubectl create secret generic cloudflared-tunnel-credentials --from-literal=token='...' -n high-command
+kubectl apply -k overlays/default
 ```
 
-See `api-secrets-example.yaml` for details.
+## Commands
 
-## Cloudflare
+```bash
+# Build manifests (dry-run)
+kubectl kustomize overlays/default
 
-The `../cloudflare/` folder contains the tunnel Dockerfile. GitLab CI builds the cloudflared-tunnel image from `cloudflare/Dockerfile`.
+# Deploy
+kubectl apply -k overlays/default
 
-## Environment Variables
+# Delete
+kubectl delete -k overlays/default
+```
 
-The UI uses environment variables at build time (Vite). These are configured in:
-- `ui-deployment-blue.yaml`
-- `ui-deployment-green.yaml`
+## Secrets Examples
 
-Current environment variables:
-- `NODE_ENV=production`
-- `PORT=3000`
+Secret templates are in `base/*/` with `-example` suffix. Create secrets with `kubectl create`, do not apply example files:
 
-For build-time variables (e.g., `VITE_CLAUDE_API_KEY`), they must be set during the Docker build process, not at runtime.
+- `base/api/api-secrets-example.yaml`
+- `base/ui/ui-secrets-example.yaml`
+- `base/poller/poller-secrets-example.yaml`
+- `base/postgres/postgres-credentials-example.yaml`
+- `base/cloudflare/cloudflared-tunnel-secrets-example.yaml`
 
-## Deployment
+## Database: Use PgBouncer Pooler
 
-Deploy all resources:
+API and poller use `high-command-postgres-pooler` to avoid exhausting Postgres `max_connections`. The pooler must be deployed (`postgres-pooler-example.yaml`) before creating secrets.
 
+**To migrate existing deployment** from direct Postgres to pooler:
 ```bash
-kubectl apply -f k8s/
+kubectl delete secret high-command-api-secrets high-command-poller-secrets -n high-command
+# Recreate with pooler URL (see Quick Start above)
+kubectl rollout restart deployment/high-command-api-blue deployment/high-command-api-green deployment/high-command-poller -n high-command
 ```
 
-Switch between blue/green by updating the service selector in `ui-service.yaml`:
+## Prerequisites
 
-```yaml
-selector:
-  app: high-command-ui
-  version: blue  # or green
-```
+- **CloudNativePG operator** – for PostgreSQL cluster
+- **Envoy Gateway** – for Gateway API
+- **cert-manager** – for TLS certificates
+- **Namespace** – `kubectl create namespace high-command` (or let apply create it)
 
-Or use the annotation:
+## Blue/Green Switching
 
 ```bash
-kubectl annotate service high-command-ui \
-  deployment.kubernetes.io/active-version=green \
-  -n high-command
+kubectl patch svc high-command-ui -n high-command -p '{"spec":{"selector":{"version":"green"}}}'
+kubectl patch svc high-command-api -n high-command -p '{"spec":{"selector":{"version":"green"}}}'
 ```

k8s/api-deployment-blue.yaml → k8s/base/api/api-deployment-blue.yaml

@@ -20,7 +20,7 @@ spec:
     spec:
       containers:
       - name: api
-        image: harbor.dataknife.net/library/high-command-api:v0.11
+        image: harbor.dataknife.net/library/high-command-api:v0.21
         imagePullPolicy: Always
         env:
         - name: MODE
@@ -56,27 +56,27 @@ spec:
           protocol: TCP
         livenessProbe:
           httpGet:
-            path: /api/health
+            path: /api/livez
             port: http
           initialDelaySeconds: 30
           periodSeconds: 30
-          timeoutSeconds: 5
+          timeoutSeconds: 10
           failureThreshold: 3
         readinessProbe:
           httpGet:
             path: /api/health
             port: http
-          initialDelaySeconds: 10
+          initialDelaySeconds: 15
           periodSeconds: 10
-          timeoutSeconds: 3
-          failureThreshold: 2
+          timeoutSeconds: 10
+          failureThreshold: 3
         resources:
           requests:
             cpu: "200m"
             memory: "256Mi"
           limits:
             cpu: "1000m"
-            memory: "1Gi"
+            memory: "2Gi"
         securityContext:
           runAsNonRoot: true
           runAsUser: 1000

k8s/api-deployment-green.yaml → k8s/base/api/api-deployment-green.yaml

@@ -20,7 +20,7 @@ spec:
     spec:
       containers:
       - name: api
-        image: harbor.dataknife.net/library/high-command-api:v0.11
+        image: harbor.dataknife.net/library/high-command-api:v0.21
         imagePullPolicy: Always
         env:
         - name: MODE
@@ -56,27 +56,27 @@ spec:
           protocol: TCP
         livenessProbe:
           httpGet:
-            path: /api/health
+            path: /api/livez
             port: http
           initialDelaySeconds: 30
           periodSeconds: 30
-          timeoutSeconds: 5
+          timeoutSeconds: 10
           failureThreshold: 3
         readinessProbe:
           httpGet:
             path: /api/health
             port: http
-          initialDelaySeconds: 10
+          initialDelaySeconds: 15
           periodSeconds: 10
-          timeoutSeconds: 3
-          failureThreshold: 2
+          timeoutSeconds: 10
+          failureThreshold: 3
         resources:
           requests:
             cpu: "200m"
             memory: "256Mi"
           limits:
             cpu: "1000m"
-            memory: "1Gi"
+            memory: "2Gi"
         securityContext:
           runAsNonRoot: true
           runAsUser: 1000

k8s/api-secrets-example.yaml → k8s/base/api/api-secrets-example.yaml

@@ -1,11 +1,15 @@
 # API secrets - DATABASE_URL required, claude-api-key optional
 # Never commit real values. Create with kubectl:
 #
+# Use PgBouncer pooler (recommended - avoids exhausting Postgres max_connections):
 #   kubectl create secret generic high-command-api-secrets \
-#     --from-literal=database-url='postgresql://user:password@high-command-postgres-rw.high-command.svc.cluster.local:5432/highcommand' \
+#     --from-literal=database-url='postgresql://user:password@high-command-postgres-pooler.high-command.svc.cluster.local:5432/highcommand' \
 #     --from-literal=claude-api-key='sk-ant-api03-...' \
 #     -n high-command
 #
+# Direct Postgres (use only if pooler not deployed):
+#   --from-literal=database-url='postgresql://user:password@high-command-postgres-rw.high-command.svc.cluster.local:5432/highcommand'
+#
 # DATABASE_URL: Required. PostgreSQL connection string.
 # claude-api-key: Optional. For Claude UI integration via backend proxy.
 #

k8s/base/api/kustomization.yaml

@@ -0,0 +1,8 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - api-deployment-blue.yaml
+  - api-deployment-green.yaml
+  - api-service.yaml
+  - api-pdb.yaml

k8s/base/cloudflare/kustomization.yaml

@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - tunnel-deployment.yaml

k8s/base/gateway/kustomization.yaml

@@ -0,0 +1,9 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - gatewayclass.yaml
+  - gateway.yaml
+  - certificate.yaml
+  - httproute.yaml
+  - tunnel-service.yaml

k8s/base/kustomization.yaml

@@ -0,0 +1,13 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+# Namespace set per-resource (most high-command, mcp-referencegrant uses mcp-servers)
+
+resources:
+  - api
+  - ui
+  - poller
+  - postgres
+  - gateway
+  - mcp
+  - cloudflare

k8s/base/mcp/kustomization.yaml

@@ -0,0 +1,6 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - service.yaml
+  - referencegrant.yaml

k8s/mcp-service.yaml → k8s/base/mcp/service.yaml

@@ -1,3 +1,4 @@
+# Alias for HTTPRoute to reach MCP in mcp-servers namespace
 apiVersion: v1
 kind: Service
 metadata:
@@ -11,6 +12,4 @@ spec:
   ports:
   - name: http
     port: 8000
-    targetPort: 8000
     protocol: TCP
-  sessionAffinity: None

k8s/base/poller/deployment.yaml

@@ -0,0 +1,71 @@
+# Data collector - polls Hell Divers 2 API and writes to PostgreSQL.
+# Runs separately from API (API is read-only). One replica sufficient.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: high-command-poller
+  namespace: high-command
+  labels:
+    app: high-command-poller
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: high-command-poller
+  template:
+    metadata:
+      labels:
+        app: high-command-poller
+    spec:
+      containers:
+      - name: poller
+        image: harbor.dataknife.net/library/high-command-api:v0.22
+        imagePullPolicy: Always
+        env:
+        - name: MODE
+          value: "collector"
+        - name: PYTHONUNBUFFERED
+          value: "1"
+        - name: LOG_LEVEL
+          value: "INFO"
+        - name: POOL_MAX_CONN
+          value: "100"
+        - name: DATABASE_URL
+          valueFrom:
+            secretKeyRef:
+              name: high-command-poller-secrets
+              key: database-url
+        - name: HELLDIVERS_API_BASE
+          valueFrom:
+            secretKeyRef:
+              name: high-command-poller-secrets
+              key: helldivers-api-base
+        - name: HELLDIVERS_API_CLIENT_NAME
+          valueFrom:
+            secretKeyRef:
+              name: high-command-poller-secrets
+              key: helldivers-api-client-name
+        - name: HELLDIVERS_API_CONTACT
+          valueFrom:
+            secretKeyRef:
+              name: high-command-poller-secrets
+              key: helldivers-api-contact
+        - name: SCRAPE_INTERVAL
+          valueFrom:
+            secretKeyRef:
+              name: high-command-poller-secrets
+              key: scrape-interval
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "128Mi"
+          limits:
+            cpu: "500m"
+            memory: "512Mi"
+        securityContext:
+          runAsNonRoot: true
+          runAsUser: 1000
+          allowPrivilegeEscalation: false
+          capabilities:
+            drop:
+            - ALL

k8s/base/poller/kustomization.yaml

@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - deployment.yaml