UmbraDatastore/docs/DEPLOYMENT_GUIDE.md

UmbraTrans Kubernetes Deployment - Quick Start

Prerequisites

• Kubernetes cluster (v1.28+)
• NFS server accessible from cluster (192.168.123.14:/mnt/umbra-data)
• S3-compatible storage for backups
• cert-manager installed for TLS
• nginx-ingress controller
• NVIDIA GPU operator (for worker nodes)
• CloudNativePG operator

1. Install Operators

# CloudNativePG
kubectl apply -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.24/releases/cnpg-1.24.0.yaml

# NFS CSI Driver
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/v4.7.0/deploy/example/nfs-driver.yaml

# cert-manager (if not installed)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml

# NVIDIA GPU Operator (on GPU nodes)
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm install gpu-operator nvidia/gpu-operator -n gpu-operator --create-namespace

2. Configure Secrets

# Create namespace
kubectl create namespace umbratrans

# Database credentials (CNPG will also create these)
kubectl create secret generic umbratrans-db-credentials \
  -n umbratrans \
  --from-literal=username=umbratrans \
  --from-literal=password=$(openssl rand -base64 32) \
  --from-literal=dsn="postgresql://umbratrans:PASSWORD@umbratrans-db-rw:5432/umbratrans?sslmode=require"

# SMTP credentials
kubectl create secret generic umbratrans-smtp \
  -n umbratrans \
  --from-literal=host=smtp.example.com \
  --from-literal=port=587 \
  --from-literal=username=umbra@example.com \
  --from-literal=password=YOUR_SMTP_PASSWORD \
  --from-literal=sender=umbratrans@umbra.example.com

# Application secrets
kubectl create secret generic umbratrans-secrets \
  -n umbratrans \
  --from-literal=token_pepper=$(openssl rand -base64 32)

# Backup credentials
kubectl create secret generic umbratrans-backup-credentials \
  -n umbratrans \
  --from-literal=ACCESS_KEY_ID=YOUR_S3_ACCESS_KEY \
  --from-literal=SECRET_ACCESS_KEY=YOUR_S3_SECRET_KEY

# PgBouncer admin
kubectl create secret generic umbratrans-pgbouncer-admin \
  -n umbratrans \
  --from-literal=username=pgbouncer_admin \
  --from-literal=password=$(openssl rand -base64 32)

3. Update Configuration

Edit these files with your values:

• k8s/base/01-postgresql.yaml - S3 backup endpoint
• k8s/base/02-nfs-storage.yaml - NFS server IP (192.168.123.14)
• k8s/base/03-web-deployment.yaml - Ingress host (umbratrans.bdmurphy.me)
• k8s/base/05-ingress-network.yaml - TLS hosts

4. Build Images

# Build web image
docker build -f Dockerfile.web -t umbratrans-web:latest .
docker push your-registry/umbratrans-web:latest

# Build worker image
docker build -f Dockerfile.worker -t umbratrans-worker:latest .
docker push your-registry/umbratrans-worker:latest

# Update image references in k8s/base/kustomization.yaml

5. Deploy

# Development
kubectl apply -k k8s/overlays/dev

# Production
kubectl apply -k k8s/overlays/prod

6. Run Migrations

# Migrations run automatically via initContainer on web deployment
# Or run manually:
kubectl run migrate --rm -it --image=umbratrans-web:latest \
  --env="UMBRA_ENV=production" \
  --env="UMBRATRANS_DATABASE_DSN=$(kubectl get secret umbratrans-db-credentials -n umbratrans -o jsonpath='{.data.dsn}' | base64 -d)" \
  -- python -m umbratrans_web.cli migrate --allow-production

7. Import Existing Productions

# One-time import
kubectl apply -f k8s/base/06-import-job.yaml

# Or run manually:
kubectl run import --rm -it --image=umbratrans-worker:latest \
  --env="UMBRA_ENV=production" \
  --env="UMBRATRANS_DATABASE_DSN=..." \
  --env="UMBRATRANS_PRODUCTIONS_ROOT=/mnt/productions" \
  -- python -m umbratrans_web.cli import-productions \
    --organization umbra \
    --allow-production \
    --confirm

8. Verify Deployment

# Check pods
kubectl get pods -n umbratrans

# Check web health
curl -k https://umbratrans.bdmurphy.me/api/health

# Check database
kubectl exec -n umbratrans umbratrans-db-1 -- pg_isready -U umbratrans

# Check workers processing jobs
kubectl logs -n umbratrans -l app=umbratrans-worker-translation -f

9. Monitoring

# Port-forward Prometheus
kubectl port-forward -n monitoring svc/prometheus 9090

# Port-forward Grafana
kubectl port-forward -n monitoring svc/grafana 3000

10. Common Operations

Scale Workers

# Translation workers
kubectl scale deployment umbratrans-worker-translation -n umbratrans --replicas=10

# Audio workers
kubectl scale deployment umbratrans-worker-audio -n umbratrans --replicas=5

View Logs

# Web
kubectl logs -n umbratrans -l app=umbratrans-web -f

# Translation worker
kubectl logs -n umbratrans -l app=umbratrans-worker-translation -f

# Audio worker
kubectl logs -n umbratrans -l app=umbratrans-worker-audio -f

# Database
kubectl logs -n umbratrans -l app=umbratrans-db -f

Database Access

# Connect to primary
kubectl exec -it -n umbratrans umbratrans-db-1 -- psql -U umbratrans -d umbratrans

# Connect via PgBouncer (pooled)
kubectl run -it --rm pgclient --image=postgres:16 -- \
  psql "postgresql://umbratrans:PASSWORD@umbratrans-db-pgbouncer:6432/umbratrans"

Backup/Restore

# Create manual backup
kubectl apply -f - <<EOF
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
  name: manual-backup-$(date +%s)
  namespace: umbratrans
spec:
  cluster: umbratrans-db
EOF

# List backups
kubectl get backups -n umbratrans

# Restore to new cluster
kubectl apply -f - <<EOF
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: umbratrans-db-restored
  namespace: umbratrans
spec:
  instances: 1
  storage:
    size: 50Gi
  bootstrap:
    recovery:
      source: umbratrans-db
      backup:
        name: manual-backup-1234567890
EOF

Troubleshooting

Issue	Solution
Web pod CrashLoopBackOff	Check logs: kubectl logs -n umbratrans -l app=umbratrans-web
Workers not picking jobs	Verify DB connectivity, check job queue: SELECT * FROM jobs WHERE status='queued'
NFS mount failures	Check NFS server reachable, CSI driver pods running
GPU not allocated	Verify GPU operator, node labels nvidia.com/gpu.present=true
Migration failures	Check migration logs, run manually with --dry-run first
TLS certificate issues	Check cert-manager logs, verify DNS challenge

File Inventory

k8s/
├── base/
│   ├── 00-namespace.yaml
│   ├── 01-postgresql.yaml          # CNPG cluster
│   ├── 02-nfs-storage.yaml         # NFS CSI, PVs, PVCs
│   ├── 03-web-deployment.yaml      # Web API + UI
│   ├── 04-worker-deployments.yaml  # Translation + Audio workers
│   ├── 05-ingress-network.yaml     # Ingress + NetworkPolicies
│   ├── 06-import-job.yaml          # Production import job
│   ├── 07-monitoring-secrets.yaml  # Secrets, ServiceMonitors, Alerts
│   └── kustomization.yaml
├── overlays/
│   ├── dev/kustomization.yaml      # Dev overrides
│   ├── staging/kustomization.yaml  # Staging overrides
│   └── prod/kustomization.yaml     # Production overrides
Dockerfile.web                      # Web container
Dockerfile.worker                   # Worker container (GPU)
DATASTORE_SPEC.md                   # Complete datastore contract
umbratrans_web/cli.py               # CLI: migrate, import-productions, worker

CLI Commands Reference

# Bootstrap admin (dev only)
python -m umbratrans_web.cli bootstrap-admin --email admin@example.com

# Serve web (dev/prod)
python -m umbratrans_web.serve --host 0.0.0.0 --port 8442 --repository postgres --allow-production

# Run migrations
python -m umbratrans_web.cli migrate --allow-production

# Import productions
python -m umbratrans_web.cli import-productions --organization umbra --allow-production --confirm

# Run workers
python -m umbratrans_web.cli worker translation --allow-production
python -m umbratrans_web.cli worker audio --allow-production

# Import audio bundle
python -m umbratrans_web.cli import-audio --bundle /path/to/bundle --organization-id <uuid> --commit --confirm <hash>

External Service Integration

For UmbraAudio and UmbraFormat (running on separate machines), see:

→ INTEGRATION_SPEC.md

This document covers:

• PostgreSQL connection parameters (PgBouncer on 6432)
• NFS mount configuration (/mnt/umbra-data)
• Job queue protocol (claim/complete via FOR UPDATE SKIP LOCKED)
• Artifact storage protocol (content-addressed, NFS + PostgreSQL)
• Compute node registration & heartbeats
• Model registry
• Audio import bundle format
• Format metadata API
• Webhooks for job completion
• Required database roles and grants
• Health checks and monitoring metrics