UmbraDatastore/docs/DEPLOYMENT_GUIDE.md

# UmbraTrans Kubernetes Deployment - Quick Start

## Prerequisites

- Kubernetes cluster (v1.28+)
- NFS server accessible from cluster (`192.168.123.14:/mnt/umbra-data`)
- S3-compatible storage for backups
- cert-manager installed for TLS
- nginx-ingress controller
- NVIDIA GPU operator (for worker nodes)
- CloudNativePG operator

## 1. Install Operators

```bash
# CloudNativePG
kubectl apply -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.24/releases/cnpg-1.24.0.yaml

# NFS CSI Driver
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/v4.7.0/deploy/example/nfs-driver.yaml

# cert-manager (if not installed)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml

# NVIDIA GPU Operator (on GPU nodes)
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm install gpu-operator nvidia/gpu-operator -n gpu-operator --create-namespace
```

## 2. Configure Secrets

```bash
# Create namespace
kubectl create namespace umbratrans

# Database credentials (CNPG will also create these)
kubectl create secret generic umbratrans-db-credentials \
  -n umbratrans \
  --from-literal=username=umbratrans \
  --from-literal=password=$(openssl rand -base64 32) \
  --from-literal=dsn="postgresql://umbratrans:PASSWORD@umbratrans-db-rw:5432/umbratrans?sslmode=require"

# SMTP credentials
kubectl create secret generic umbratrans-smtp \
  -n umbratrans \
  --from-literal=host=smtp.example.com \
  --from-literal=port=587 \
  --from-literal=username=umbra@example.com \
  --from-literal=password=YOUR_SMTP_PASSWORD \
  --from-literal=sender=umbratrans@umbra.example.com

# Application secrets
kubectl create secret generic umbratrans-secrets \
  -n umbratrans \
  --from-literal=token_pepper=$(openssl rand -base64 32)

# Backup credentials
kubectl create secret generic umbratrans-backup-credentials \
  -n umbratrans \
  --from-literal=ACCESS_KEY_ID=YOUR_S3_ACCESS_KEY \
  --from-literal=SECRET_ACCESS_KEY=YOUR_S3_SECRET_KEY

# PgBouncer admin
kubectl create secret generic umbratrans-pgbouncer-admin \
  -n umbratrans \
  --from-literal=username=pgbouncer_admin \
  --from-literal=password=$(openssl rand -base64 32)
```

## 3. Update Configuration

Edit these files with your values:

- `k8s/base/01-postgresql.yaml` - S3 backup endpoint
- `k8s/base/02-nfs-storage.yaml` - NFS server IP (`192.168.123.14`)
- `k8s/base/03-web-deployment.yaml` - Ingress host (`umbratrans.bdmurphy.me`)
- `k8s/base/05-ingress-network.yaml` - TLS hosts

## 4. Build Images

```bash
# Build web image
docker build -f Dockerfile.web -t umbratrans-web:latest .
docker push your-registry/umbratrans-web:latest

# Build worker image
docker build -f Dockerfile.worker -t umbratrans-worker:latest .
docker push your-registry/umbratrans-worker:latest

# Update image references in k8s/base/kustomization.yaml
```

## 5. Deploy

```bash
# Development
kubectl apply -k k8s/overlays/dev

# Production
kubectl apply -k k8s/overlays/prod
```

## 6. Run Migrations

```bash
# Migrations run automatically via initContainer on web deployment
# Or run manually:
kubectl run migrate --rm -it --image=umbratrans-web:latest \
  --env="UMBRA_ENV=production" \
  --env="UMBRATRANS_DATABASE_DSN=$(kubectl get secret umbratrans-db-credentials -n umbratrans -o jsonpath='{.data.dsn}' | base64 -d)" \
  -- python -m umbratrans_web.cli migrate --allow-production
```

## 7. Import Existing Productions

```bash
# One-time import
kubectl apply -f k8s/base/06-import-job.yaml

# Or run manually:
kubectl run import --rm -it --image=umbratrans-worker:latest \
  --env="UMBRA_ENV=production" \
  --env="UMBRATRANS_DATABASE_DSN=..." \
  --env="UMBRATRANS_PRODUCTIONS_ROOT=/mnt/productions" \
  -- python -m umbratrans_web.cli import-productions \
    --organization umbra \
    --allow-production \
    --confirm
```

## 8. Verify Deployment

```bash
# Check pods
kubectl get pods -n umbratrans

# Check web health
curl -k https://umbratrans.bdmurphy.me/api/health

# Check database
kubectl exec -n umbratrans umbratrans-db-1 -- pg_isready -U umbratrans

# Check workers processing jobs
kubectl logs -n umbratrans -l app=umbratrans-worker-translation -f
```

## 9. Monitoring

```bash
# Port-forward Prometheus
kubectl port-forward -n monitoring svc/prometheus 9090

# Port-forward Grafana
kubectl port-forward -n monitoring svc/grafana 3000
```

## 10. Common Operations

### Scale Workers

```bash
# Translation workers
kubectl scale deployment umbratrans-worker-translation -n umbratrans --replicas=10

# Audio workers
kubectl scale deployment umbratrans-worker-audio -n umbratrans --replicas=5
```

### View Logs

```bash
# Web
kubectl logs -n umbratrans -l app=umbratrans-web -f

# Translation worker
kubectl logs -n umbratrans -l app=umbratrans-worker-translation -f

# Audio worker
kubectl logs -n umbratrans -l app=umbratrans-worker-audio -f

# Database
kubectl logs -n umbratrans -l app=umbratrans-db -f
```

### Database Access

```bash
# Connect to primary
kubectl exec -it -n umbratrans umbratrans-db-1 -- psql -U umbratrans -d umbratrans

# Connect via PgBouncer (pooled)
kubectl run -it --rm pgclient --image=postgres:16 -- \
  psql "postgresql://umbratrans:PASSWORD@umbratrans-db-pgbouncer:6432/umbratrans"
```

### Backup/Restore

```bash
# Create manual backup
kubectl apply -f - <<EOF
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
  name: manual-backup-$(date +%s)
  namespace: umbratrans
spec:
  cluster: umbratrans-db
EOF

# List backups
kubectl get backups -n umbratrans

# Restore to new cluster
kubectl apply -f - <<EOF
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: umbratrans-db-restored
  namespace: umbratrans
spec:
  instances: 1
  storage:
    size: 50Gi
  bootstrap:
    recovery:
      source: umbratrans-db
      backup:
        name: manual-backup-1234567890
EOF
```

## Troubleshooting

| Issue | Solution |
|-------|----------|
| Web pod CrashLoopBackOff | Check logs: `kubectl logs -n umbratrans -l app=umbratrans-web` |
| Workers not picking jobs | Verify DB connectivity, check job queue: `SELECT * FROM jobs WHERE status='queued'` |
| NFS mount failures | Check NFS server reachable, CSI driver pods running |
| GPU not allocated | Verify GPU operator, node labels `nvidia.com/gpu.present=true` |
| Migration failures | Check migration logs, run manually with `--dry-run` first |
| TLS certificate issues | Check cert-manager logs, verify DNS challenge |

## File Inventory

```
k8s/
├── base/
│   ├── 00-namespace.yaml
│   ├── 01-postgresql.yaml          # CNPG cluster
│   ├── 02-nfs-storage.yaml         # NFS CSI, PVs, PVCs
│   ├── 03-web-deployment.yaml      # Web API + UI
│   ├── 04-worker-deployments.yaml  # Translation + Audio workers
│   ├── 05-ingress-network.yaml     # Ingress + NetworkPolicies
│   ├── 06-import-job.yaml          # Production import job
│   ├── 07-monitoring-secrets.yaml  # Secrets, ServiceMonitors, Alerts
│   └── kustomization.yaml
├── overlays/
│   ├── dev/kustomization.yaml      # Dev overrides
│   ├── staging/kustomization.yaml  # Staging overrides
│   └── prod/kustomization.yaml     # Production overrides
Dockerfile.web                      # Web container
Dockerfile.worker                   # Worker container (GPU)
DATASTORE_SPEC.md                   # Complete datastore contract
umbratrans_web/cli.py               # CLI: migrate, import-productions, worker
```

## CLI Commands Reference

```bash
# Bootstrap admin (dev only)
python -m umbratrans_web.cli bootstrap-admin --email admin@example.com

# Serve web (dev/prod)
python -m umbratrans_web.serve --host 0.0.0.0 --port 8442 --repository postgres --allow-production

# Run migrations
python -m umbratrans_web.cli migrate --allow-production

# Import productions
python -m umbratrans_web.cli import-productions --organization umbra --allow-production --confirm

# Run workers
python -m umbratrans_web.cli worker translation --allow-production
python -m umbratrans_web.cli worker audio --allow-production

# Import audio bundle
python -m umbratrans_web.cli import-audio --bundle /path/to/bundle --organization-id <uuid> --commit --confirm <hash>
```

## External Service Integration

For **UmbraAudio** and **UmbraFormat** (running on separate machines), see:

**→ [INTEGRATION_SPEC.md](INTEGRATION_SPEC.md)**

This document covers:
- PostgreSQL connection parameters (PgBouncer on 6432)
- NFS mount configuration (/mnt/umbra-data)
- Job queue protocol (claim/complete via `FOR UPDATE SKIP LOCKED`)
- Artifact storage protocol (content-addressed, NFS + PostgreSQL)
- Compute node registration & heartbeats
- Model registry
- Audio import bundle format
- Format metadata API
- Webhooks for job completion
- Required database roles and grants
- Health checks and monitoring metrics