Initial commit: UmbraDatastore configuration and deployment scripts
- Database deployment script (scripts/deploy-database.sh) - Kubernetes manifests for CNPG PostgreSQL cluster (k8s/base/) - Documentation (docs/DEPLOYMENT_GUIDE.md, DATASTORE_SPEC.md, INTEGRATION_SPEC.md) - README with architecture overview and operations guide
This commit is contained in:
umbra audio
233
README.md
Normal file
233
README.md
Normal file
@@ -0,0 +1,233 @@
|
||||
# UmbraDatastore
|
||||
|
||||
Configuration, deployment scripts, and specifications for the UmbraTrans data layer.
|
||||
|
||||
## Overview
|
||||
|
||||
This repository contains the infrastructure-as-code for the UmbraTrans PostgreSQL database cluster and related data services. It uses CloudNativePG (CNPG) for PostgreSQL high availability and NFS CSI for shared storage.
|
||||
|
||||
## Components
|
||||
|
||||
### Database (PostgreSQL + CNPG)
|
||||
- **High Availability**: 3-node cluster (1 primary + 2 replicas)
|
||||
- **Connection Pooling**: PgBouncer (transaction mode)
|
||||
- **Backups**: Continuous WAL archiving + daily base backups to S3
|
||||
- **Monitoring**: Prometheus PodMonitor with custom queries
|
||||
- **Extensions**: uuid-ossp, citext, pgcrypto
|
||||
|
||||
### Shared Storage (NFS)
|
||||
- **Productions**: Source book data and translation runs (`/mnt/umbra-data/productions`)
|
||||
- **Artifacts**: Content-addressed audio/EPUB/PDF outputs (`/mnt/umbra-data/artifacts`)
|
||||
- **Covers**: Cover images (`/mnt/umbra-data/covers`)
|
||||
- **Audio**: Master audio files (`/mnt/umbra-data/audio`)
|
||||
- **Backups**: Database backup staging (`/mnt/umbra-data/backups`)
|
||||
|
||||
### Schema
|
||||
The database schema includes:
|
||||
- **Identity**: users, organizations, sessions, credentials, audit_events
|
||||
- **Projects**: projects (per-tenant schemas), format_metadata
|
||||
- **Compute**: compute_nodes, compute_devices, model_registry, node_models, jobs, job_events
|
||||
- **Artifacts**: artifacts, job_artifacts, project_source_refs
|
||||
- **Publish/Byline**: publishers, authors, project_publishers, project_authors
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
- Kubernetes cluster (v1.28+)
|
||||
- NFS server accessible from cluster (default: `192.168.123.14:/mnt/umbra-data`)
|
||||
- S3-compatible storage for backups
|
||||
- cert-manager + nginx-ingress (for web UI)
|
||||
|
||||
### Deploy
|
||||
|
||||
```bash
|
||||
# Make script executable
|
||||
chmod +x scripts/deploy-database.sh
|
||||
|
||||
# Run deployment
|
||||
./scripts/deploy-database.sh
|
||||
```
|
||||
|
||||
The script will:
|
||||
1. Install CNPG operator
|
||||
2. Create namespace and secrets
|
||||
3. Deploy PostgreSQL cluster
|
||||
4. Run migrations
|
||||
5. Create initial admin user
|
||||
|
||||
### Manual Steps
|
||||
|
||||
If you prefer manual deployment:
|
||||
|
||||
```bash
|
||||
# 1. Install CNPG operator
|
||||
kubectl apply -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.24.0/releases/cnpg-1.24.0.yaml
|
||||
|
||||
# 2. Install NFS CSI driver
|
||||
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/v4.7.0/deploy/example/nfs-driver.yaml
|
||||
|
||||
# 3. Create namespace
|
||||
kubectl create namespace umbratrans
|
||||
|
||||
# 4. Create secrets (see scripts/deploy-database.sh for details)
|
||||
|
||||
# 3. Deploy cluster
|
||||
kubectl apply -k k8s/base
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Database Cluster (`k8s/base/01-postgresql.yaml`)
|
||||
Key settings:
|
||||
- Instances: 3 (1 primary + 2 replicas)
|
||||
- Storage: 50Gi on NFS
|
||||
- Resources: 1-2Gi memory, 0.5-2 CPU
|
||||
- PgBouncer: 2 replicas, transaction pooling
|
||||
|
||||
### NFS Storage (`k8s/base/02-nfs-storage.yaml`)
|
||||
Update the NFS server IP:
|
||||
```yaml
|
||||
parameters:
|
||||
server: 192.168.123.14 # <-- Your NFS server IP
|
||||
share: /mnt/umbra-data
|
||||
```
|
||||
|
||||
### Monitoring (`k8s/base/07-monitoring-secrets.yaml`)
|
||||
- ServiceMonitors for Prometheus
|
||||
- Custom PostgreSQL queries
|
||||
- Alert rules for cluster health
|
||||
|
||||
## Operations
|
||||
|
||||
### Connect to Database
|
||||
```bash
|
||||
# Via port-forward
|
||||
kubectl port-forward -n umbratrans svc/umbratrans-db-rw 5432:5432
|
||||
psql -h localhost -U umbratrans -d umbratrans
|
||||
|
||||
# Direct (internal)
|
||||
psql -h umbratrans-db-rw.umbratrans.svc -U umbratrans -d umbratrans
|
||||
```
|
||||
|
||||
### Check Cluster Status
|
||||
```bash
|
||||
kubectl get cluster umbratrans-db -n umbratrans
|
||||
kubectl get pods -n umbratrans -l postgresql=umbratrans-db
|
||||
```
|
||||
|
||||
### View Logs
|
||||
```bash
|
||||
# Primary
|
||||
kubectl logs -n umbratrans -l postgresql=umbratrans-db,role=primary -f
|
||||
|
||||
# PgBouncer
|
||||
kubectl logs -n umbratrans -l app=umbratrans-db-pgbouncer -f
|
||||
```
|
||||
|
||||
### Backup/Restore
|
||||
```bash
|
||||
# Manual backup
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Backup
|
||||
metadata:
|
||||
name: manual-backup-$(date +%s)
|
||||
namespace: umbratrans
|
||||
spec:
|
||||
cluster: umbratrans-db
|
||||
EOF
|
||||
|
||||
# List backups
|
||||
kubectl get backups -n umbratrans
|
||||
|
||||
# Restore to new cluster
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Cluster
|
||||
metadata:
|
||||
name: umbratrans-db-restored
|
||||
namespace: umbratrans
|
||||
spec:
|
||||
instances: 1
|
||||
storage:
|
||||
size: 50Gi
|
||||
bootstrap:
|
||||
recovery:
|
||||
source: umbratrans-db
|
||||
backup:
|
||||
name: manual-backup-1234567890
|
||||
EOF
|
||||
```
|
||||
|
||||
### Scale Resources
|
||||
```bash
|
||||
# Scale cluster (instances)
|
||||
kubectl patch cluster umbratrans-db -n umbratrans \
|
||||
-p '{"spec":{"instances":5}}' --type=merge
|
||||
|
||||
# Scale PgBouncer
|
||||
kubectl patch cluster umbratrans-db -n umbratrans \
|
||||
-p '{"spec":{"proxy":{"pgBouncer":{"replicas":4}}}}' --type=merge
|
||||
```
|
||||
|
||||
## Secrets Management
|
||||
|
||||
All sensitive data is stored in Kubernetes Secrets:
|
||||
- `umbratrans-db-credentials` - Database user/password/DSN
|
||||
- `umbratrans-pgbouncer-admin` - PgBouncer admin credentials
|
||||
- `umbratrans-secrets` - Token pepper, etc.
|
||||
- `umbratrans-smtp` - SMTP credentials
|
||||
- `umbratrans-backup-credentials` - S3 backup credentials
|
||||
|
||||
Update placeholder values after deployment:
|
||||
```bash
|
||||
kubectl patch secret umbratrans-smtp -n umbratrans \
|
||||
-p '{"data":{"password":"'$(echo -n "real-password" | base64)'"}}'
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Deployment Guide](docs/DEPLOYMENT_GUIDE.md) - Full deployment instructions
|
||||
- [Datastore Specification](docs/DATASTORE_SPEC.md) - Complete schema and contracts
|
||||
- [Integration Spec](docs/INTEGRATION_SPEC.md) - UmbraAudio/UmbraFormat integration
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ KUBERNETES CLUSTER │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
||||
│ │ umbratrans │ │ umbratrans │ │ umbratrans │ │
|
||||
│ │ web (API) │ │ worker │ │ worker │ │
|
||||
│ └──────┬───────┘ └──────┬───────┘ └────────┬─────────┘ │
|
||||
│ │ │ │ │
|
||||
│ └─────────────────┼────────────────────┘ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────────────────────────────────────────┐ │
|
||||
│ │ POSTGRESQL CLUSTER (CNPG) │ │
|
||||
│ │ Primary ──── Replica 1 │ │
|
||||
│ │ │ │ │
|
||||
│ │ └──── Replica 2 │ │
|
||||
│ │ │ │
|
||||
│ │ PgBouncer (pool) ──► Primary │ │
|
||||
│ └──────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────────────────┼─────────────────┐ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
|
||||
│ │ Productions│ │ Artifacts │ │ Covers │ │
|
||||
│ │ (NFS) │ │ (NFS) │ │ (NFS) │ │
|
||||
│ └────────────┘ └────────────┘ └────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Related Repositories
|
||||
|
||||
- **UmbraTrans** - Main application (API, UI, workers)
|
||||
- **UmbraAudio** - Audio processing node
|
||||
- **UmbraFormat** - EPUB/PDF formatting node
|
||||
- **UmbraTools** - Core libraries and tools
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
493
docs/DATASTORE_SPEC.md
Normal file
493
docs/DATASTORE_SPEC.md
Normal file
@@ -0,0 +1,493 @@
|
||||
# UmbraTrans Datastore Specification
|
||||
|
||||
> **Version:** 1.0
|
||||
> **Status:** Active
|
||||
> **Last Updated:** 2026-10-31
|
||||
|
||||
This document defines the complete datastore contract that every UmbraTrans utility (web API, workers, CLI tools, importers) must understand. It covers PostgreSQL schema, NFS directory layout, and the invariants each component must respect.
|
||||
|
||||
---
|
||||
|
||||
## 1. PostgreSQL Schema (Authoritative Store)
|
||||
|
||||
### 1.1 Core Identity Tables (Phase 1)
|
||||
|
||||
| Table | Purpose | Key Columns |
|
||||
|-------|---------|-------------|
|
||||
| `public.users` | Canonical accounts | `id` (uuid, PK), `email` (citext, unique, not null), `username` (citext, unique, nullable), `display_name`, `role`, `status`, `organization_id`, `created_at` |
|
||||
| `public.user_credentials` | Auth credentials (one per user) | `id` (uuid, PK), `user_id` (uuid, unique, FK), `credential_type` (local_password/oidc), `password_hash` (argon2id), `must_change_password`, `failed_login_count`, `locked_until` |
|
||||
| `public.sessions` | Server-side sessions | `id` (uuid, PK), `user_id` (FK), `token_hash`, `created_at`, `last_seen_at`, `idle_deadline_at`, `absolute_deadline_at`, `revoked_at` |
|
||||
| `public.organizations` | Tenants | `id` (uuid, PK), `name`, `slug` (citext, unique), `status` (active/suspended/archived), `created_at` |
|
||||
| `public.user_roles` | Role assignments | `user_id`, `role_id`, `scope` (organization/project), `organization_id`, `project_id`, `status` |
|
||||
| `public.audit_events` | Immutable audit log | `id` (bigserial, PK), `occurred_at`, `action`, `outcome`, `actor_user_id`, `target_user_id`, `organization_id`, `detail` (jsonb) |
|
||||
|
||||
### 1.2 Project & Work Tables (Phase 2+)
|
||||
|
||||
| Table | Purpose | Key Columns |
|
||||
|-------|---------|-------------|
|
||||
| `tenant_<org>.projects` | Projects (per-tenant schema) | `id` (uuid, PK), `name`, `slug`, `status`, `created_by`, `created_at` |
|
||||
| `public.jobs` | Work queue | `id` (uuid, PK), `organization_id`, `project_id`, `lang`, `kind`, `status`, `priority`, `requires` (jsonb), `lease_owner`, `lease_expires_at`, `attempt`, `max_attempts`, `payload`, `result`, `error`, `created_at`, `started_at`, `finished_at` |
|
||||
| `public.job_events` | Job history | `id` (bigserial, PK), `job_id`, `node_id`, `event`, `detail` (jsonb), `at` |
|
||||
|
||||
### 1.3 Compute & Node Tables
|
||||
|
||||
| Table | Purpose | Key Columns |
|
||||
|-------|---------|-------------|
|
||||
| `public.compute_nodes` | Worker nodes | `id` (uuid, PK), `organization_id`, `name`, `is_controller`, `trust_domain`, `status` (provisioning/online/draining/offline/retired), `last_heartbeat_at` |
|
||||
| `public.compute_devices` | GPUs/CPUs per node | `id` (uuid, PK), `compute_node_id`, `kind` (cpu/gpu_discrete/...), `label`, `vram_total_mb`, `vram_free_mb`, `throughput_factor` |
|
||||
| `public.model_registry` | Known models | `id` (uuid, PK), `slug` (unique), `kind`, `vram_min_mb`, `vram_preferred_mb`, `requires_accelerators` |
|
||||
| `public.node_models` | Model residency | `compute_node_id`, `model_id`, `device_id`, `state` (loaded/warm/available/loading/failed), `vram_held_mb` |
|
||||
| `public.node_heartbeats` | Node health | `compute_node_id` (PK), `reported_at`, `cpu_busy_pct`, `ram_free_mb`, `vram_free_mb`, `active_jobs`, `resident_models` (jsonb) |
|
||||
|
||||
### 1.4 Artifact & Provenance Tables
|
||||
|
||||
| Table | Purpose | Key Columns |
|
||||
|-------|---------|-------------|
|
||||
| `public.artifacts` | Content-addressed artifacts | `id` (uuid, PK), `project_id`, `organization_id`, `lang`, `kind`, `status` (intermediate/final/superseded), `content_hash` (sha256), `byte_size`, `media_type`, `storage_key`, `storage_uri`, `storage_node_id`, `integrity` (unverified/verified), `verified_at`, `derivation` (jsonb), `parent_artifact_id`, `created_by_job_id`, `created_by_user_id`, `created_at` |
|
||||
| `public.job_artifacts` | Provenance links | `job_id`, `artifact_id`, `role` (consumed/produced), `created_at` (PK: job_id, artifact_id, role) |
|
||||
| `public.project_source_refs` | Audio import ledger | `project_id`, `organization_id`, `system`, `node_id`, `external_id`, `source_title`, `languages` (jsonb), `content_hash`, `imported_at` |
|
||||
|
||||
### 1.5 Publisher & Byline Tables
|
||||
|
||||
| Table | Purpose | Key Columns |
|
||||
|-------|---------|-------------|
|
||||
| `public.publishers` | Imprints/publishers | `id` (uuid, PK), `parent_id`, `name`, `slug` (unique), `owner_organization_id`, `kind` (imprint/publisher) |
|
||||
| `public.project_publishers` | Project→Publisher mapping | `project_id`, `publisher_id`, `lang` (PK: project_id, publisher_id, lang) |
|
||||
| `public.authors` | Bylines (authors, translators, narrators) | `id` (uuid, PK), `name`, `display_name`, `slug` (unique), `user_id`, `owner_organization_id` |
|
||||
| `public.project_authors` | Project→Author mapping | `project_id`, `author_id`, `credit_role`, `lang`, `position` (PK: project_id, author_id, credit_role, lang) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Tenancy Model
|
||||
|
||||
### 2.1 Schema-per-Tenant Isolation
|
||||
|
||||
- **Platform tables** (`users`, `organizations`, `jobs`, `artifacts`, `compute_nodes`, etc.) live in `public` schema
|
||||
- **Tenant tables** (`projects`, `chapters`, `utterances`, etc.) live in `tenant_<org_id>` schema
|
||||
- **Isolation mechanism:** `SET LOCAL search_path = tenant_<org_id>, public` on every transaction
|
||||
- **RLS policies** enforce organization membership on all platform tables
|
||||
|
||||
### 2.2 Connection Protocol
|
||||
|
||||
Every database operation MUST:
|
||||
```python
|
||||
with pool.connection() as conn:
|
||||
with conn.transaction():
|
||||
# 1. Set statement timeout
|
||||
conn.execute("SET LOCAL statement_timeout = 5000")
|
||||
|
||||
# 2. Set caller identity for RLS
|
||||
conn.execute("SELECT set_config('app.user_id', %s, true)", (str(user_id),))
|
||||
|
||||
# 3. Set tenant schema for isolation
|
||||
search_path = tenant_search_path(organization_id) # "tenant_xxx, public"
|
||||
conn.execute("SELECT set_config('search_path', %s, true)", (search_path,))
|
||||
|
||||
# 4. Execute queries - unqualified 'projects' resolves to tenant schema
|
||||
```
|
||||
|
||||
### 2.3 Helper Functions (Database-Side)
|
||||
|
||||
```sql
|
||||
-- Returns tenant schema name for an organization
|
||||
app.tenant_schema_name(org_id uuid) RETURNS text
|
||||
|
||||
-- Returns current user from session GUC
|
||||
app.current_user_id() RETURNS uuid
|
||||
|
||||
-- Returns search path for tenant
|
||||
app.tenant_search_path(org_id uuid) RETURNS text
|
||||
|
||||
-- Policy helpers
|
||||
app.is_platform_admin() RETURNS boolean
|
||||
app.is_organization_admin(org_id uuid) RETURNS boolean
|
||||
app.is_organization_member(org_id uuid) RETURNS boolean
|
||||
app.can_access_project(project_id uuid) RETURNS boolean
|
||||
app.validate_user_role_scope() -- trigger on user_roles
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. NFS Directory Layout (Shared Storage)
|
||||
|
||||
### 3.1 Root Structure
|
||||
|
||||
```
|
||||
/mnt/umbra-data/
|
||||
├── productions/ # Source productions (read/write by workers)
|
||||
│ ├── <project>-<lang>/
|
||||
│ │ ├── source/
|
||||
│ │ │ ├── book.json # Project metadata
|
||||
│ │ │ └── covers/ # Cover images
|
||||
│ │ ├── runs/
|
||||
│ │ │ ├── ch-<N>/
|
||||
│ │ │ │ ├── translation.json
|
||||
│ │ │ │ ├── ch-<N>-c<M>.analysis.json
|
||||
│ │ │ │ ├── ch-<N>-c<M>.draft.json
|
||||
│ │ │ │ └── ch-<N>-c<M>.critic.json
|
||||
│ │ ├── glossary/
|
||||
│ │ │ └── glossary.json
|
||||
│ │ └── bundle/ # Audio import bundle (after export)
|
||||
│ │ ├── bundle.json
|
||||
│ │ └── manifest.jsonl
|
||||
│ └── ...
|
||||
├── artifacts/ # Content-addressed artifacts (read/write)
|
||||
│ ├── <org_id>/
|
||||
│ │ ├── <project_id>/
|
||||
│ │ │ ├── <content_hash>[:<kind>][:<lang>]
|
||||
│ │ │ │ ├── data # Raw bytes (or symlink to object storage)
|
||||
│ │ │ │ └── meta.json # Artifact metadata
|
||||
│ │ └── ...
|
||||
│ └── ...
|
||||
├── covers/ # Cover images (read/write)
|
||||
│ └── <asset_id>/
|
||||
│ ├── original.png
|
||||
│ ├── thumbnail.jpg
|
||||
│ └── meta.json
|
||||
├── audio/ # Audio master files (read/write by audio workers)
|
||||
│ └── <project_id>/
|
||||
│ ├── <lang>/
|
||||
│ │ ├── chapters/
|
||||
│ │ │ └── ch-<N>/
|
||||
│ │ │ ├── track-<M>.wav
|
||||
│ │ │ └── track-<M>.mp3
|
||||
│ │ ├── assembled/
|
||||
│ │ │ └── book.mp3
|
||||
│ │ └── meta.json
|
||||
└── backups/ # Database backups (optional)
|
||||
└── postgresql/
|
||||
└── <timestamp>.dump
|
||||
```
|
||||
|
||||
### 3.2 Path Conventions
|
||||
|
||||
| Data Type | NFS Path | PostgreSQL Reference |
|
||||
|-----------|----------|---------------------|
|
||||
| Production source | `/mnt/umbra-data/productions/<project>-<lang>/source/book.json` | `project_source_refs.book_directory` |
|
||||
| Translation runs | `/mnt/umbra-data/productions/<project>-<lang>/runs/` | `artifacts.storage_uri` (file://) |
|
||||
| Audio renders | `/mnt/umbra-data/audio/<project_id>/<lang>/` | `artifacts.storage_uri` (file://) |
|
||||
| Cover images | `/mnt/umbra-data/covers/<asset_id>/` | `format_metadata.cover_front_asset_id` |
|
||||
| Artifact bytes | `/mnt/umbra-data/artifacts/<org>/<project>/<hash>/data` | `artifacts.storage_key` |
|
||||
|
||||
### 3.3 Access Patterns
|
||||
|
||||
| Component | Productions | Artifacts | Covers | Audio |
|
||||
|-----------|-------------|-----------|--------|-------|
|
||||
| Web API | Read | Read | Read/Write | Read |
|
||||
| Translation Worker | Read/Write | Read | Read | Read |
|
||||
| Audio Worker | Read | Read/Write | Read | Read/Write |
|
||||
| Import Job | Read | Write | Write | Write |
|
||||
|
||||
---
|
||||
|
||||
## 4. Invariants & Contracts
|
||||
|
||||
### 4.1 Content Addressing (Artifacts)
|
||||
|
||||
- **Identity = SHA256 hash of bytes** (`content_hash`)
|
||||
- **Storage key ≠ Identity** — `storage_key` is location, `content_hash` is identity
|
||||
- **Deduplication:** Same hash = same artifact (within project scope)
|
||||
- **Immutability:** `artifacts` table has `BEFORE UPDATE` trigger preventing `content_hash`/`storage_key` changes
|
||||
- **Supersession:** New version = new row with `parent_artifact_id` pointing to previous
|
||||
|
||||
### 4.2 Job Queue Semantics
|
||||
|
||||
```sql
|
||||
-- Claim a job (atomic, skip locked)
|
||||
UPDATE public.jobs
|
||||
SET status = 'running',
|
||||
lease_owner = $node_id,
|
||||
lease_expires_at = now() + interval '30 minutes',
|
||||
started_at = now(),
|
||||
attempt = attempt + 1
|
||||
WHERE id = (
|
||||
SELECT id FROM public.jobs
|
||||
WHERE status = 'queued'
|
||||
AND not_before <= now()
|
||||
AND (lease_expires_at IS NULL OR lease_expires_at <= now())
|
||||
AND organization_id = $org_id
|
||||
ORDER BY priority, not_before, created_at
|
||||
FOR UPDATE SKIP LOCKED
|
||||
LIMIT 1
|
||||
)
|
||||
RETURNING *;
|
||||
```
|
||||
|
||||
- **Lease expiration** returns job to `queued` (handled by controller or job timeout)
|
||||
- **Idempotency key** prevents duplicate submission
|
||||
- **Parent job** links for decomposition (one job → many sub-jobs)
|
||||
|
||||
### 4.3 Audio Import Idempotency
|
||||
|
||||
```python
|
||||
# Deterministic IDs
|
||||
project_id = uuid5(AUDIO_IMPORT_NAMESPACE, f"{org_id}:{system}:{external_id}")
|
||||
import_id = uuid5(AUDIO_IMPORT_NAMESPACE, f"{project_id}:{content_hash}")
|
||||
|
||||
# Import logic
|
||||
existing_hash = store.last_content_hash(external_id)
|
||||
if existing_hash == content_hash:
|
||||
return "no_change" # Idempotent
|
||||
# Else: DELETE old source rows, INSERT new snapshot, INSERT import ledger row
|
||||
```
|
||||
|
||||
### 4.4 Worker Job Contract
|
||||
|
||||
Each worker mode polls for specific job kinds:
|
||||
|
||||
| Worker Mode | Job Kinds | Required Resources |
|
||||
|-------------|-----------|-------------------|
|
||||
| `translation` | `translate`, `analyze`, `critic`, `backtranslate` | GPU (LLM), CPU |
|
||||
| `audio` | `tts`, `asr`, `assemble`, `encode`, `master` | GPU (TTS), CPU, Audio storage |
|
||||
|
||||
---
|
||||
|
||||
## 5. Component Responsibilities
|
||||
|
||||
### 5.1 Web API (`umbratrans-web`)
|
||||
|
||||
- **Reads:** Users, orgs, projects, jobs, artifacts, format metadata
|
||||
- **Writes:** Users, orgs, projects, format metadata, cover uploads, job queue (enqueue)
|
||||
- **NFS:** Reads productions for status, reads/writes covers, reads artifacts for download
|
||||
- **DB:** Uses `PostgresAccountRepository` + direct queries for project status
|
||||
|
||||
### 5.2 Translation Worker (`umbratrans-worker translation`)
|
||||
|
||||
- **Polls:** `jobs` WHERE `kind IN ('translate','analyze','critic','backtranslate')`
|
||||
- **Reads:** Source text from productions NFS, project metadata from DB
|
||||
- **Writes:** Translation artifacts to NFS + DB, job results to DB
|
||||
- **External:** Calls Ollama/LLM API for translation
|
||||
|
||||
### 5.3 Audio Worker (`umbratrans-worker audio`)
|
||||
|
||||
- **Polls:** `jobs` WHERE `kind LIKE 'audio%'`
|
||||
- **Reads:** Translation artifacts from NFS, speaker profiles from DB
|
||||
- **Writes:** Audio renders to NFS (`/audio/`) + artifact rows to DB
|
||||
- **External:** Calls TTS/ASR models (local or remote)
|
||||
|
||||
### 5.4 Import CLI (`umbratrans-import-productions`)
|
||||
|
||||
- **Reads:** Productions from NFS (`/productions/`)
|
||||
- **Writes:** Audio import tables (`project_source_refs`, `translation_chapters`, `translation_speakers`, `translation_utterances`, `artifacts`, `translation_imports`)
|
||||
- **NFS:** Reads bundle, writes artifacts to `/artifacts/`
|
||||
- **Idempotent:** Re-runnable, skips unchanged content
|
||||
|
||||
### 5.5 Migration CLI (`umbratrans-migrate`)
|
||||
|
||||
- **Applies:** All `.sql` files in `umbratrans_web/migrations/` (except rollback)
|
||||
- **Order:** Lexicographic by filename (0001, 0002, ...)
|
||||
- **Transaction:** Each migration in its own transaction
|
||||
- **Idempotent:** Uses `IF NOT EXISTS`, `ON CONFLICT`, guarded `ALTER TABLE`
|
||||
|
||||
---
|
||||
|
||||
## 6. Environment Variables
|
||||
|
||||
### 6.1 Required for All Components
|
||||
|
||||
| Variable | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `UMBRA_ENV` | Environment: development/test/staging/production | `production` |
|
||||
| `UMBRATRANS_DATABASE_DSN` | PostgreSQL connection string | `postgresql://user:pass@host:5432/db?sslmode=require` |
|
||||
| `UMBRATRANS_PRODUCTIONS_ROOT` | NFS productions mount | `/mnt/productions` |
|
||||
| `UMBRATRANS_TOKEN_PEPPER` | 32+ byte secret for token signing | `base64-encoded` |
|
||||
|
||||
### 6.2 Web API Specific
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `UMBRATRANS_REPOSITORY` | `postgres` (production) / `memory` / `json` (dev) |
|
||||
| `UMBRATRANS_ALLOWED_HOSTS` | Comma-separated host allowlist |
|
||||
| `UMBRATRANS_VERIFICATION_BASE_URL` | Absolute URL for email links |
|
||||
| `UMBRATRANS_COOKIE_SECURE` | `true` in production |
|
||||
| `UMBRATRANS_REQUIRE_CSRF` | `true` in production |
|
||||
| `UMBRATRANS_SMTP_*` | SMTP configuration |
|
||||
| `UMBRATRANS_STATIC_ROOT` | Static files path (`/app/web`) |
|
||||
|
||||
### 6.3 Worker Specific
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `OLLAMA_BASE_URL` | LLM API endpoint (`http://ollama:11434/v1`) |
|
||||
| `UMBRATRANS_WORKER_POLL_INTERVAL` | Queue poll seconds (default: 5) |
|
||||
|
||||
---
|
||||
|
||||
## 7. Kubernetes Deployment Contract
|
||||
|
||||
### 7.1 Required Resources
|
||||
|
||||
| Component | CPU | Memory | GPU | NFS Mounts |
|
||||
|-----------|-----|--------|-----|------------|
|
||||
| Web API | 250m-1000m | 256-512Mi | No | productions (ro), artifacts (rw) |
|
||||
| Translation Worker | 2000m-4000m | 2-4Gi | 1x GPU | productions (rw), artifacts (rw) |
|
||||
| Audio Worker | 2000m-4000m | 4-8Gi | 1x GPU | productions (ro), artifacts (rw), audio (rw) |
|
||||
|
||||
### 7.2 Service Discovery
|
||||
|
||||
| Service | DNS | Port | Purpose |
|
||||
|---------|-----|------|---------|
|
||||
| `umbratrans-web` | `umbratrans-web.umbratrans.svc` | 80 | Web API + UI |
|
||||
| `umbratrans-db-rw` | `umbratrans-db-rw.umbratrans.svc` | 5432 | Primary DB (writes) |
|
||||
| `umbratrans-db-ro` | `umbratrans-db-ro.umbratrans.svc` | 5432 | Replica DB (reads) |
|
||||
| `umbratrans-db-pgbouncer` | `umbratrans-db-pgbouncer.umbratrans.svc` | 6432 | Pooled connections |
|
||||
| `ollama` | `ollama.umbratrans.svc` | 11434 | LLM API |
|
||||
|
||||
### 7.3 Health Checks
|
||||
|
||||
```yaml
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /api/health
|
||||
port: 8442
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 30
|
||||
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /api/health
|
||||
port: 8442
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
```
|
||||
|
||||
Health endpoint returns:
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"service": "umbratrans-web",
|
||||
"version": "0.2.0",
|
||||
"environment": "production",
|
||||
"repository": "postgres",
|
||||
"cookie_secure": true,
|
||||
"csrf_required": true,
|
||||
"mail_deliverable": true,
|
||||
"mail_delivery_waived": false,
|
||||
"time": "2026-10-31T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Backup & Recovery
|
||||
|
||||
### 8.1 PostgreSQL (CNPG)
|
||||
|
||||
- **Continuous:** WAL archiving to S3
|
||||
- **Scheduled:** Daily base backup at 02:00 UTC (retention: 30 days)
|
||||
- **Point-in-time recovery:** Supported via `barman-cloud-wal-archive`
|
||||
- **Test:** Monthly restore to staging
|
||||
|
||||
### 8.2 NFS Data
|
||||
|
||||
- **Productions:** Rsync to offsite daily
|
||||
- **Artifacts:** Content-addressed, reproducible from source
|
||||
- **Covers:** Reproducible from source projects
|
||||
|
||||
### 8.3 Recovery Procedures
|
||||
|
||||
| Scenario | Procedure |
|
||||
|----------|-----------|
|
||||
| Web pod crash | Kubernetes restarts (liveness probe) |
|
||||
| Worker pod crash | Kubernetes restarts, job lease expires, re-queued |
|
||||
| DB primary failover | CNPG promotes replica (30-60s), connections retry |
|
||||
| NFS server down | Workers pause (NFS hard mount), resume on recovery |
|
||||
| Data corruption | Restore PG from backup, re-import productions |
|
||||
|
||||
---
|
||||
|
||||
## 9. Security Boundaries
|
||||
|
||||
| Boundary | Enforcement |
|
||||
|----------|-------------|
|
||||
| Tenant isolation | `search_path` + RLS policies on all platform tables |
|
||||
| API authentication | Argon2id password hash, Secure/HttpOnly/SameSite cookies, CSRF tokens |
|
||||
| Session management | Server-side sessions with token hash only, idle/absolute timeouts |
|
||||
| Worker auth | Node credentials (argon2id key hash), scoped to queue operations |
|
||||
| NFS access | Kubernetes `fsGroup=1000`, NFS `no_root_squash`, directory mode 0700 |
|
||||
| Database credentials | Stored in Kubernetes Secrets, never in images or configmaps |
|
||||
| Token pepper | 32+ bytes, in Secret, used for session/verification token HMAC |
|
||||
|
||||
---
|
||||
|
||||
## 10. Version Compatibility
|
||||
|
||||
| Component | Minimum Schema Version | Compatible With |
|
||||
|-----------|------------------------|-----------------|
|
||||
| Web API 0.2.x | Migrations 0001-0010 | Workers 0.2.x, Import 0.2.x |
|
||||
| Workers 0.2.x | Migrations 0001-0010 | Web API 0.2.x |
|
||||
| Import CLI 0.2.x | Migrations 0001-0010 | Web API 0.2.x |
|
||||
|
||||
**Breaking changes require:** New migration file, version bump, coordinated deploy.
|
||||
|
||||
---
|
||||
|
||||
## 11. Quick Reference: Key SQL Queries
|
||||
|
||||
```sql
|
||||
-- Get project with all languages status
|
||||
SELECT p.*,
|
||||
jsonb_agg(jsonb_build_object(
|
||||
'lang', l.language_tag,
|
||||
'translation', l.translation_status,
|
||||
'audio', l.audio_status
|
||||
)) as languages
|
||||
FROM tenant_<org>.projects p
|
||||
LEFT JOIN tenant_<org>.project_languages l ON l.project_id = p.id
|
||||
WHERE p.id = $1
|
||||
GROUP BY p.id;
|
||||
|
||||
-- Queue depth by kind
|
||||
SELECT kind, status, count(*)
|
||||
FROM public.jobs
|
||||
WHERE organization_id = $1
|
||||
GROUP BY kind, status;
|
||||
|
||||
-- Artifact storage usage
|
||||
SELECT organization_id, sum(byte_size) as total_bytes, count(*) as count
|
||||
FROM public.artifacts
|
||||
GROUP BY organization_id;
|
||||
|
||||
-- Node capacity
|
||||
SELECT cn.id, cn.name, cn.status,
|
||||
jsonb_agg(jsonb_build_object(
|
||||
'kind', cd.kind,
|
||||
'label', cd.label,
|
||||
'vram_free', cd.vram_free_mb,
|
||||
'throughput', cd.throughput_factor
|
||||
)) as devices
|
||||
FROM public.compute_nodes cn
|
||||
LEFT JOIN public.compute_devices cd ON cd.compute_node_id = cn.id
|
||||
WHERE cn.status = 'online'
|
||||
GROUP BY cn.id;
|
||||
|
||||
-- Import history
|
||||
SELECT pi.*, o.name as org_name
|
||||
FROM public.translation_imports pi
|
||||
JOIN public.organizations o ON o.id = pi.organization_id
|
||||
ORDER BY pi.imported_at DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. Appendix: Migration Order
|
||||
|
||||
```
|
||||
0001_phase1_identity.sql -- Users, credentials, sessions, organizations
|
||||
0002_password_reset_purpose.sql -- Password reset tokens
|
||||
0003_compute_and_queue.sql -- Compute nodes, devices, models, jobs, artifacts, publishers, authors
|
||||
0004_node_pairing.sql -- Node pairing protocol
|
||||
0005_tenant_schemas.sql -- Tenant schema creation trigger
|
||||
0006_node_removal_and_assignments.sql -- Node lifecycle, project assignments
|
||||
0007_external_artifacts.sql -- Artifact storage_uri, integrity
|
||||
0008_audio_source_import.sql -- Audio import tables (project_source_refs, etc.)
|
||||
0009_translation_import_runs.sql -- Translation import tracking
|
||||
0010_utterance_translations.sql -- Utterance translation tables
|
||||
```
|
||||
|
||||
**Apply with:** `kubectl apply -k k8s/overlays/prod` (includes migration init container)
|
||||
|
||||
---
|
||||
|
||||
*This specification is the source of truth for all datastore interactions. Any component that reads or writes data must conform to these contracts.*
|
||||
307
docs/DEPLOYMENT_GUIDE.md
Normal file
307
docs/DEPLOYMENT_GUIDE.md
Normal file
@@ -0,0 +1,307 @@
|
||||
# UmbraTrans Kubernetes Deployment - Quick Start
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Kubernetes cluster (v1.28+)
|
||||
- NFS server accessible from cluster (`192.168.123.14:/mnt/umbra-data`)
|
||||
- S3-compatible storage for backups
|
||||
- cert-manager installed for TLS
|
||||
- nginx-ingress controller
|
||||
- NVIDIA GPU operator (for worker nodes)
|
||||
- CloudNativePG operator
|
||||
|
||||
## 1. Install Operators
|
||||
|
||||
```bash
|
||||
# CloudNativePG
|
||||
kubectl apply -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.24/releases/cnpg-1.24.0.yaml
|
||||
|
||||
# NFS CSI Driver
|
||||
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/v4.7.0/deploy/example/nfs-driver.yaml
|
||||
|
||||
# cert-manager (if not installed)
|
||||
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml
|
||||
|
||||
# NVIDIA GPU Operator (on GPU nodes)
|
||||
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
|
||||
helm install gpu-operator nvidia/gpu-operator -n gpu-operator --create-namespace
|
||||
```
|
||||
|
||||
## 2. Configure Secrets
|
||||
|
||||
```bash
|
||||
# Create namespace
|
||||
kubectl create namespace umbratrans
|
||||
|
||||
# Database credentials (CNPG will also create these)
|
||||
kubectl create secret generic umbratrans-db-credentials \
|
||||
-n umbratrans \
|
||||
--from-literal=username=umbratrans \
|
||||
--from-literal=password=$(openssl rand -base64 32) \
|
||||
--from-literal=dsn="postgresql://umbratrans:PASSWORD@umbratrans-db-rw:5432/umbratrans?sslmode=require"
|
||||
|
||||
# SMTP credentials
|
||||
kubectl create secret generic umbratrans-smtp \
|
||||
-n umbratrans \
|
||||
--from-literal=host=smtp.example.com \
|
||||
--from-literal=port=587 \
|
||||
--from-literal=username=umbra@example.com \
|
||||
--from-literal=password=YOUR_SMTP_PASSWORD \
|
||||
--from-literal=sender=umbratrans@umbra.example.com
|
||||
|
||||
# Application secrets
|
||||
kubectl create secret generic umbratrans-secrets \
|
||||
-n umbratrans \
|
||||
--from-literal=token_pepper=$(openssl rand -base64 32)
|
||||
|
||||
# Backup credentials
|
||||
kubectl create secret generic umbratrans-backup-credentials \
|
||||
-n umbratrans \
|
||||
--from-literal=ACCESS_KEY_ID=YOUR_S3_ACCESS_KEY \
|
||||
--from-literal=SECRET_ACCESS_KEY=YOUR_S3_SECRET_KEY
|
||||
|
||||
# PgBouncer admin
|
||||
kubectl create secret generic umbratrans-pgbouncer-admin \
|
||||
-n umbratrans \
|
||||
--from-literal=username=pgbouncer_admin \
|
||||
--from-literal=password=$(openssl rand -base64 32)
|
||||
```
|
||||
|
||||
## 3. Update Configuration
|
||||
|
||||
Edit these files with your values:
|
||||
|
||||
- `k8s/base/01-postgresql.yaml` - S3 backup endpoint
|
||||
- `k8s/base/02-nfs-storage.yaml` - NFS server IP (`192.168.123.14`)
|
||||
- `k8s/base/03-web-deployment.yaml` - Ingress host (`umbratrans.bdmurphy.me`)
|
||||
- `k8s/base/05-ingress-network.yaml` - TLS hosts
|
||||
|
||||
## 4. Build Images
|
||||
|
||||
```bash
|
||||
# Build web image
|
||||
docker build -f Dockerfile.web -t umbratrans-web:latest .
|
||||
docker push your-registry/umbratrans-web:latest
|
||||
|
||||
# Build worker image
|
||||
docker build -f Dockerfile.worker -t umbratrans-worker:latest .
|
||||
docker push your-registry/umbratrans-worker:latest
|
||||
|
||||
# Update image references in k8s/base/kustomization.yaml
|
||||
```
|
||||
|
||||
## 5. Deploy
|
||||
|
||||
```bash
|
||||
# Development
|
||||
kubectl apply -k k8s/overlays/dev
|
||||
|
||||
# Production
|
||||
kubectl apply -k k8s/overlays/prod
|
||||
```
|
||||
|
||||
## 6. Run Migrations
|
||||
|
||||
```bash
|
||||
# Migrations run automatically via initContainer on web deployment
|
||||
# Or run manually:
|
||||
kubectl run migrate --rm -it --image=umbratrans-web:latest \
|
||||
--env="UMBRA_ENV=production" \
|
||||
--env="UMBRATRANS_DATABASE_DSN=$(kubectl get secret umbratrans-db-credentials -n umbratrans -o jsonpath='{.data.dsn}' | base64 -d)" \
|
||||
-- python -m umbratrans_web.cli migrate --allow-production
|
||||
```
|
||||
|
||||
## 7. Import Existing Productions
|
||||
|
||||
```bash
|
||||
# One-time import
|
||||
kubectl apply -f k8s/base/06-import-job.yaml
|
||||
|
||||
# Or run manually:
|
||||
kubectl run import --rm -it --image=umbratrans-worker:latest \
|
||||
--env="UMBRA_ENV=production" \
|
||||
--env="UMBRATRANS_DATABASE_DSN=..." \
|
||||
--env="UMBRATRANS_PRODUCTIONS_ROOT=/mnt/productions" \
|
||||
-- python -m umbratrans_web.cli import-productions \
|
||||
--organization umbra \
|
||||
--allow-production \
|
||||
--confirm
|
||||
```
|
||||
|
||||
## 8. Verify Deployment
|
||||
|
||||
```bash
|
||||
# Check pods
|
||||
kubectl get pods -n umbratrans
|
||||
|
||||
# Check web health
|
||||
curl -k https://umbratrans.bdmurphy.me/api/health
|
||||
|
||||
# Check database
|
||||
kubectl exec -n umbratrans umbratrans-db-1 -- pg_isready -U umbratrans
|
||||
|
||||
# Check workers processing jobs
|
||||
kubectl logs -n umbratrans -l app=umbratrans-worker-translation -f
|
||||
```
|
||||
|
||||
## 9. Monitoring
|
||||
|
||||
```bash
|
||||
# Port-forward Prometheus
|
||||
kubectl port-forward -n monitoring svc/prometheus 9090
|
||||
|
||||
# Port-forward Grafana
|
||||
kubectl port-forward -n monitoring svc/grafana 3000
|
||||
```
|
||||
|
||||
## 10. Common Operations
|
||||
|
||||
### Scale Workers
|
||||
|
||||
```bash
|
||||
# Translation workers
|
||||
kubectl scale deployment umbratrans-worker-translation -n umbratrans --replicas=10
|
||||
|
||||
# Audio workers
|
||||
kubectl scale deployment umbratrans-worker-audio -n umbratrans --replicas=5
|
||||
```
|
||||
|
||||
### View Logs
|
||||
|
||||
```bash
|
||||
# Web
|
||||
kubectl logs -n umbratrans -l app=umbratrans-web -f
|
||||
|
||||
# Translation worker
|
||||
kubectl logs -n umbratrans -l app=umbratrans-worker-translation -f
|
||||
|
||||
# Audio worker
|
||||
kubectl logs -n umbratrans -l app=umbratrans-worker-audio -f
|
||||
|
||||
# Database
|
||||
kubectl logs -n umbratrans -l app=umbratrans-db -f
|
||||
```
|
||||
|
||||
### Database Access
|
||||
|
||||
```bash
|
||||
# Connect to primary
|
||||
kubectl exec -it -n umbratrans umbratrans-db-1 -- psql -U umbratrans -d umbratrans
|
||||
|
||||
# Connect via PgBouncer (pooled)
|
||||
kubectl run -it --rm pgclient --image=postgres:16 -- \
|
||||
psql "postgresql://umbratrans:PASSWORD@umbratrans-db-pgbouncer:6432/umbratrans"
|
||||
```
|
||||
|
||||
### Backup/Restore
|
||||
|
||||
```bash
|
||||
# Create manual backup
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Backup
|
||||
metadata:
|
||||
name: manual-backup-$(date +%s)
|
||||
namespace: umbratrans
|
||||
spec:
|
||||
cluster: umbratrans-db
|
||||
EOF
|
||||
|
||||
# List backups
|
||||
kubectl get backups -n umbratrans
|
||||
|
||||
# Restore to new cluster
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Cluster
|
||||
metadata:
|
||||
name: umbratrans-db-restored
|
||||
namespace: umbratrans
|
||||
spec:
|
||||
instances: 1
|
||||
storage:
|
||||
size: 50Gi
|
||||
bootstrap:
|
||||
recovery:
|
||||
source: umbratrans-db
|
||||
backup:
|
||||
name: manual-backup-1234567890
|
||||
EOF
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|----------|
|
||||
| Web pod CrashLoopBackOff | Check logs: `kubectl logs -n umbratrans -l app=umbratrans-web` |
|
||||
| Workers not picking jobs | Verify DB connectivity, check job queue: `SELECT * FROM jobs WHERE status='queued'` |
|
||||
| NFS mount failures | Check NFS server reachable, CSI driver pods running |
|
||||
| GPU not allocated | Verify GPU operator, node labels `nvidia.com/gpu.present=true` |
|
||||
| Migration failures | Check migration logs, run manually with `--dry-run` first |
|
||||
| TLS certificate issues | Check cert-manager logs, verify DNS challenge |
|
||||
|
||||
## File Inventory
|
||||
|
||||
```
|
||||
k8s/
|
||||
├── base/
|
||||
│ ├── 00-namespace.yaml
|
||||
│ ├── 01-postgresql.yaml # CNPG cluster
|
||||
│ ├── 02-nfs-storage.yaml # NFS CSI, PVs, PVCs
|
||||
│ ├── 03-web-deployment.yaml # Web API + UI
|
||||
│ ├── 04-worker-deployments.yaml # Translation + Audio workers
|
||||
│ ├── 05-ingress-network.yaml # Ingress + NetworkPolicies
|
||||
│ ├── 06-import-job.yaml # Production import job
|
||||
│ ├── 07-monitoring-secrets.yaml # Secrets, ServiceMonitors, Alerts
|
||||
│ └── kustomization.yaml
|
||||
├── overlays/
|
||||
│ ├── dev/kustomization.yaml # Dev overrides
|
||||
│ ├── staging/kustomization.yaml # Staging overrides
|
||||
│ └── prod/kustomization.yaml # Production overrides
|
||||
Dockerfile.web # Web container
|
||||
Dockerfile.worker # Worker container (GPU)
|
||||
DATASTORE_SPEC.md # Complete datastore contract
|
||||
umbratrans_web/cli.py # CLI: migrate, import-productions, worker
|
||||
```
|
||||
|
||||
## CLI Commands Reference
|
||||
|
||||
```bash
|
||||
# Bootstrap admin (dev only)
|
||||
python -m umbratrans_web.cli bootstrap-admin --email admin@example.com
|
||||
|
||||
# Serve web (dev/prod)
|
||||
python -m umbratrans_web.serve --host 0.0.0.0 --port 8442 --repository postgres --allow-production
|
||||
|
||||
# Run migrations
|
||||
python -m umbratrans_web.cli migrate --allow-production
|
||||
|
||||
# Import productions
|
||||
python -m umbratrans_web.cli import-productions --organization umbra --allow-production --confirm
|
||||
|
||||
# Run workers
|
||||
python -m umbratrans_web.cli worker translation --allow-production
|
||||
python -m umbratrans_web.cli worker audio --allow-production
|
||||
|
||||
# Import audio bundle
|
||||
python -m umbratrans_web.cli import-audio --bundle /path/to/bundle --organization-id <uuid> --commit --confirm <hash>
|
||||
```
|
||||
|
||||
## External Service Integration
|
||||
|
||||
For **UmbraAudio** and **UmbraFormat** (running on separate machines), see:
|
||||
|
||||
**→ [INTEGRATION_SPEC.md](INTEGRATION_SPEC.md)**
|
||||
|
||||
This document covers:
|
||||
- PostgreSQL connection parameters (PgBouncer on 6432)
|
||||
- NFS mount configuration (/mnt/umbra-data)
|
||||
- Job queue protocol (claim/complete via `FOR UPDATE SKIP LOCKED`)
|
||||
- Artifact storage protocol (content-addressed, NFS + PostgreSQL)
|
||||
- Compute node registration & heartbeats
|
||||
- Model registry
|
||||
- Audio import bundle format
|
||||
- Format metadata API
|
||||
- Webhooks for job completion
|
||||
- Required database roles and grants
|
||||
- Health checks and monitoring metrics
|
||||
739
docs/INTEGRATION_SPEC.md
Normal file
739
docs/INTEGRATION_SPEC.md
Normal file
@@ -0,0 +1,739 @@
|
||||
# UmbraTrans Integration Specification for External Services
|
||||
|
||||
> **Version:** 1.0
|
||||
> **Audience:** UmbraAudio, UmbraFormat teams
|
||||
> **Status:** Active
|
||||
> **Last Updated:** 2026-10-31
|
||||
|
||||
This document defines the contract for external services (UmbraAudio, UmbraFormat) to integrate with the UmbraTrans datastore. It covers PostgreSQL access, NFS shared storage, and API boundaries.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ UMBRA ECOSYSTEM │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
|
||||
│ │ UmbraTrans │ │ UmbraAudio │ │ UmbraFormat │ │
|
||||
│ │ (this cluster) │◄───►│ (remote node) │◄───►│ (remote node) │ │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ │ • Web API/UI │ │ • TTS/ASR │ │ • EPUB/PDF │ │
|
||||
│ │ • Job Queue │ │ • Audio Master │ │ • Cover Gen │ │
|
||||
│ │ • PostgreSQL │ │ • Speaker DB │ │ • Print Layout │ │
|
||||
│ │ • NFS Server │ │ │ │ │ │
|
||||
│ └────────┬─────────┘ └────────┬─────────┘ └────────┬─────────┘ │
|
||||
│ │ │ │ │
|
||||
│ └────────────────────────┼────────────────────────┘ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ SHARED STORE │ │
|
||||
│ │ │ │
|
||||
│ │ PostgreSQL │ │
|
||||
│ │ (192.168.123.14)│ │
|
||||
│ │ │ │
|
||||
│ │ NFS │ │
|
||||
│ │ (192.168.123.14)│ │
|
||||
│ └──────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ WordPress │ (External - API only) │
|
||||
│ │ (sales site) │ │
|
||||
│ └──────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ Git Server │ (192.168.123.18 - Mini-Eq) │
|
||||
│ │ (source code) │ │
|
||||
│ └──────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Network Connectivity
|
||||
|
||||
### 2.1 Required Access from Remote Nodes
|
||||
|
||||
| Service | Host | Port | Protocol | Purpose |
|
||||
|---------|------|------|----------|---------|
|
||||
| PostgreSQL (PgBouncer) | `192.168.123.14` | `6432` | TCP | Pooled DB connections |
|
||||
| PostgreSQL (Direct) | `192.168.123.14` | `5432` | TCP | Admin/backup (optional) |
|
||||
| NFS | `192.168.123.14` | `2049` | TCP/UDP | Shared storage mount |
|
||||
| Web API | `umbra.bdmurphy.me` | `443` | HTTPS | REST API, webhooks |
|
||||
|
||||
### 2.2 Firewall Rules (on 192.168.123.14)
|
||||
|
||||
```bash
|
||||
# PostgreSQL (PgBouncer) - allow from cluster nodes
|
||||
iptables -A INPUT -s 192.168.123.0/24 -p tcp --dport 6432 -j ACCEPT
|
||||
|
||||
# PostgreSQL (Direct) - allow from cluster nodes
|
||||
iptables -A INPUT -s 192.168.123.0/24 -p tcp --dport 5432 -j ACCEPT
|
||||
|
||||
# NFS - allow from cluster nodes
|
||||
iptables -A INPUT -s 192.168.123.0/24 -p tcp --dport 2049 -j ACCEPT
|
||||
iptables -A INPUT -s 192.168.123.0/24 -p udp --dport 2049 -j ACCEPT
|
||||
# Also need: 111 (rpcbind), 20048 (mountd) - check rpcinfo -p
|
||||
```
|
||||
|
||||
### 2.3 DNS / Hosts
|
||||
|
||||
Add to `/etc/hosts` on remote machines (or configure DNS):
|
||||
|
||||
```
|
||||
192.168.123.14 umbra-db umbra-nfs umbra-api
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. PostgreSQL Access
|
||||
|
||||
### 3.1 Connection Parameters
|
||||
|
||||
```bash
|
||||
# PgBouncer (RECOMMENDED - pooled, 6432)
|
||||
host=192.168.123.14
|
||||
port=6432
|
||||
dbname=umbratrans
|
||||
user=umbraudio # or umbraformat
|
||||
password=<from secret>
|
||||
sslmode=require
|
||||
connect_timeout=10
|
||||
application_name=umbraudio-worker # or umbraformat-worker
|
||||
|
||||
# Direct (for admin/backup only - 5432)
|
||||
host=192.168.123.14
|
||||
port=5432
|
||||
dbname=umbratrans
|
||||
user=umbraudio
|
||||
password=<from secret>
|
||||
sslmode=require
|
||||
```
|
||||
|
||||
### 3.2 Database Roles
|
||||
|
||||
| Role | Permissions | Used By |
|
||||
|------|-------------|---------|
|
||||
| `umbraudio` | `SELECT, INSERT, UPDATE` on `jobs`, `artifacts`, `compute_nodes`, `node_heartbeats`, `model_registry`, `node_models`; `SELECT` on `projects`, `organizations` | UmbraAudio workers |
|
||||
| `umbraformat` | `SELECT, INSERT, UPDATE` on `jobs`, `artifacts`, `format_metadata`; `SELECT` on `projects`, `organizations` | UmbraFormat workers |
|
||||
| `umbra_web` | Full access to all tables (web API) | UmbraTrans web |
|
||||
|
||||
### 3.3 Required Extensions
|
||||
|
||||
```sql
|
||||
-- Already installed by migrations
|
||||
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
|
||||
CREATE EXTENSION IF NOT EXISTS "citext";
|
||||
CREATE EXTENSION IF NOT EXISTS "pgcrypto";
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. NFS Shared Storage
|
||||
|
||||
### 4.1 Mount Points (on Remote Nodes)
|
||||
|
||||
```bash
|
||||
# /etc/fstab
|
||||
192.168.123.14:/mnt/umbra-data/productions /mnt/umbra/productions nfs4 rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2 0 0
|
||||
192.168.123.14:/mnt/umbra-data/artifacts /mnt/umbra/artifacts nfs4 rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2 0 0
|
||||
192.168.123.14:/mnt/umbra-data/covers /mnt/umbra/covers nfs4 rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2 0 0
|
||||
192.168.123.14:/mnt/umbra-data/audio /mnt/umbra/audio nfs4 rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2 0 0
|
||||
```
|
||||
|
||||
### 4.2 Directory Permissions
|
||||
|
||||
```bash
|
||||
# On NFS server (192.168.123.14)
|
||||
mkdir -p /mnt/umbra-data/{productions,artifacts,covers,audio,backups}
|
||||
chown -R 1000:1000 /mnt/umbra-data
|
||||
chmod -R 700 /mnt/umbra-data
|
||||
|
||||
# On remote nodes - same UID/GID
|
||||
groupadd -g 1000 umbra
|
||||
useradd -u 1000 -g umbra -s /bin/bash umbra
|
||||
```
|
||||
|
||||
### 4.3 Access Patterns by Service
|
||||
|
||||
| Path | UmbraAudio | UmbraFormat | UmbraTrans Web |
|
||||
|------|------------|-------------|----------------|
|
||||
| `/productions/<project>-<lang>/source/` | Read (source text) | Read (metadata) | Read/Write |
|
||||
| `/productions/<project>-<lang>/runs/` | Read/Write (translation artifacts) | Read | Read |
|
||||
| `/artifacts/<org>/<project>/<hash>/` | Read/Write (audio renders) | Read/Write (EPUB/PDF) | Read |
|
||||
| `/covers/<asset_id>/` | Read | Read/Write (cover gen) | Read/Write |
|
||||
| `/audio/<project>/<lang>/` | Read/Write (master audio) | Read | Read |
|
||||
|
||||
---
|
||||
|
||||
## 5. Job Queue Integration
|
||||
|
||||
### 5.1 How Workers Claim Jobs
|
||||
|
||||
```sql
|
||||
-- Atomic job claim (run in transaction)
|
||||
BEGIN;
|
||||
SET LOCAL statement_timeout = 5000;
|
||||
SET LOCAL app.user_id = '<node-user-id>';
|
||||
SET LOCAL search_path = 'public';
|
||||
|
||||
UPDATE public.jobs
|
||||
SET status = 'running',
|
||||
lease_owner = '<compute_node_id>',
|
||||
lease_expires_at = now() + interval '30 minutes',
|
||||
started_at = now(),
|
||||
attempt = attempt + 1
|
||||
WHERE id = (
|
||||
SELECT id FROM public.jobs
|
||||
WHERE status = 'queued'
|
||||
AND not_before <= now()
|
||||
AND (lease_expires_at IS NULL OR lease_expires_at <= now())
|
||||
AND kind IN ('<allowed-kinds>')
|
||||
AND organization_id = '<org-id>'
|
||||
ORDER BY priority, not_before, created_at
|
||||
FOR UPDATE SKIP LOCKED
|
||||
LIMIT 1
|
||||
)
|
||||
RETURNING *;
|
||||
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
### 5.2 Job Kinds by Service
|
||||
|
||||
| Service | Job Kinds (claims) | Job Kinds (enqueues) |
|
||||
|---------|-------------------|---------------------|
|
||||
| **UmbraAudio** | `tts`, `asr`, `assemble`, `encode`, `master`, `speaker_profile` | `tts` (after translation), `assemble` (after tts) |
|
||||
| **UmbraFormat** | `build_epub`, `build_pdf`, `cover_generate`, `metadata_validate` | `build_epub` (after translation+audio), `cover_generate` |
|
||||
|
||||
### 5.3 Job Payload Structure
|
||||
|
||||
```json
|
||||
// TTS Job
|
||||
{
|
||||
"utterance_ids": ["utt-001", "utt-002"],
|
||||
"speaker_id": "spk-narrator",
|
||||
"voice_config": {
|
||||
"model": "xtts-v2",
|
||||
"language": "es",
|
||||
"speed": 1.0
|
||||
},
|
||||
"output_format": "wav"
|
||||
}
|
||||
|
||||
// EPUB Build Job
|
||||
{
|
||||
"project_id": "uuid",
|
||||
"language": "es",
|
||||
"include_audio": true,
|
||||
"cover_asset_id": "asset-uuid",
|
||||
"format_options": {
|
||||
"epub_version": "3.2",
|
||||
"drm": false,
|
||||
"embedded_fonts": ["Source Serif Pro"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.4 Job Result Structure
|
||||
|
||||
```json
|
||||
// TTS Result
|
||||
{
|
||||
"artifacts": [
|
||||
{
|
||||
"artifact_id": "uuid",
|
||||
"kind": "audio",
|
||||
"storage_uri": "file://192.168.123.14/mnt/umbra-data/audio/proj123/es/chapters/ch-001/track-001.wav",
|
||||
"content_hash": "sha256...",
|
||||
"byte_size": 1048576,
|
||||
"media_type": "audio/wav",
|
||||
"duration_ms": 5432
|
||||
}
|
||||
],
|
||||
"metrics": {
|
||||
"total_duration_ms": 5432,
|
||||
"rtf": 0.15
|
||||
}
|
||||
}
|
||||
|
||||
// EPUB Build Result
|
||||
{
|
||||
"artifacts": [
|
||||
{
|
||||
"artifact_id": "uuid",
|
||||
"kind": "epub",
|
||||
"storage_uri": "file://192.168.123.14/mnt/umbra-data/artifacts/org1/proj123/hash123.epub",
|
||||
"content_hash": "sha256...",
|
||||
"byte_size": 5242880,
|
||||
"media_type": "application/epub+zip"
|
||||
}
|
||||
],
|
||||
"validation": {
|
||||
"epubcheck": "passed",
|
||||
"page_count": 312
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Artifact Protocol (Content-Addressed Storage)
|
||||
|
||||
### 6.1 Writing Artifacts
|
||||
|
||||
```python
|
||||
# 1. Compute content hash
|
||||
import hashlib
|
||||
content_hash = hashlib.sha256(audio_bytes).hexdigest()
|
||||
|
||||
# 2. Write to NFS (atomic)
|
||||
artifact_path = f"/mnt/umbra/artifacts/{org_id}/{project_id}/{content_hash[:2]}/{content_hash}"
|
||||
os.makedirs(os.path.dirname(artifact_path), exist_ok=True)
|
||||
tmp_path = artifact_path + ".tmp"
|
||||
with open(tmp_path, "wb") as f:
|
||||
f.write(audio_bytes)
|
||||
os.rename(tmp_path, artifact_path)
|
||||
|
||||
# 3. Insert artifact row (PostgreSQL)
|
||||
INSERT INTO public.artifacts (
|
||||
id, project_id, organization_id, lang, kind, status,
|
||||
content_hash, byte_size, media_type, storage_key,
|
||||
storage_uri, integrity, derivation, created_at
|
||||
) VALUES (
|
||||
gen_random_uuid(), $project_id, $org_id, $lang, 'audio', 'final',
|
||||
$content_hash, $byte_size, 'audio/wav', $content_hash,
|
||||
$storage_uri, 'unverified', $derivation_json, now()
|
||||
)
|
||||
ON CONFLICT (project_id, content_hash, kind, lang) DO NOTHING;
|
||||
|
||||
# 4. Link to job
|
||||
INSERT INTO public.job_artifacts (job_id, artifact_id, role, created_at)
|
||||
VALUES ($job_id, $artifact_id, 'produced', now());
|
||||
```
|
||||
|
||||
### 6.2 Storage URI Format
|
||||
|
||||
```
|
||||
file://<node-id>/<relative-path-from-nfs-root>
|
||||
|
||||
Examples:
|
||||
file://umbra-audio-01/mnt/umbra-data/audio/proj123/es/chapters/ch-001/track-001.wav
|
||||
file://umbra-format-01/mnt/umbra-data/artifacts/org1/proj123/ab12cd34.epub
|
||||
```
|
||||
|
||||
### 6.3 Reading Artifacts
|
||||
|
||||
```python
|
||||
# Via storage_uri from artifact row
|
||||
storage_uri = artifact_row["storage_uri"] # "file://umbra-audio-01/mnt/umbra-data/..."
|
||||
|
||||
# Parse and read
|
||||
if storage_uri.startswith("file://"):
|
||||
# Extract path after hostname
|
||||
path = storage_uri.split("/", 3)[-1] # "/mnt/umbra-data/..."
|
||||
with open(path, "rb") as f:
|
||||
data = f.read()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Compute Node Registration
|
||||
|
||||
### 7.1 Node Registration (at startup)
|
||||
|
||||
```sql
|
||||
-- Register compute node
|
||||
INSERT INTO public.compute_nodes (
|
||||
id, organization_id, name, is_controller, trust_domain,
|
||||
locality, transport, status, agent_version, agent_started_at, created_at
|
||||
) VALUES (
|
||||
$node_id, $org_id, $hostname, false, 'remote',
|
||||
$datacenter, 'https', 'provisioning', $version, now(), now()
|
||||
)
|
||||
ON CONFLICT (organization_id, name) DO UPDATE SET
|
||||
status = 'provisioning',
|
||||
agent_version = EXCLUDED.agent_version,
|
||||
agent_started_at = EXCLUDED.agent_started_at,
|
||||
updated_at = now()
|
||||
RETURNING id;
|
||||
```
|
||||
|
||||
### 7.2 Device Registration
|
||||
|
||||
```sql
|
||||
-- Register GPUs/CPUs
|
||||
INSERT INTO public.compute_devices (
|
||||
id, compute_node_id, kind, label, vram_total_mb, vram_free_mb,
|
||||
ram_total_mb, ram_free_mb, compute_units, throughput_factor, supports
|
||||
) VALUES (
|
||||
gen_random_uuid(), $node_id, 'gpu_discrete', 'gpu:0',
|
||||
24576, 24576, 0, 0, 10496, 1.0, '{"cuda": "12.4"}'
|
||||
)
|
||||
ON CONFLICT (compute_node_id, label) DO UPDATE SET
|
||||
vram_total_mb = EXCLUDED.vram_total_mb,
|
||||
vram_free_mb = EXCLUDED.vram_free_mb,
|
||||
updated_at = now();
|
||||
```
|
||||
|
||||
### 7.3 Heartbeat (every 30 seconds)
|
||||
|
||||
```sql
|
||||
-- Update heartbeat (overwrites previous)
|
||||
INSERT INTO public.node_heartbeats (
|
||||
compute_node_id, reported_at, cpu_busy_pct, ram_free_mb,
|
||||
vram_free_mb, vram_total_mb, disk_free_mb, load_avg_1m,
|
||||
resident_models, active_jobs, agent_version, raw
|
||||
) VALUES (
|
||||
$node_id, now(), $cpu_pct, $ram_free, $vram_free, $vram_total,
|
||||
$disk_free, $load_avg, $resident_models_json, $active_jobs, $version, $raw_json
|
||||
)
|
||||
ON CONFLICT (compute_node_id) DO UPDATE SET
|
||||
reported_at = EXCLUDED.reported_at,
|
||||
cpu_busy_pct = EXCLUDED.cpu_busy_pct,
|
||||
ram_free_mb = EXCLUDED.ram_free_mb,
|
||||
vram_free_mb = EXCLUDED.vram_free_mb,
|
||||
vram_total_mb = EXCLUDED.vram_total_mb,
|
||||
disk_free_mb = EXCLUDED.disk_free_mb,
|
||||
load_avg_1m = EXCLUDED.load_avg_1m,
|
||||
resident_models = EXCLUDED.resident_models,
|
||||
active_jobs = EXCLUDED.active_jobs,
|
||||
agent_version = EXCLUDED.agent_version,
|
||||
raw = EXCLUDED.raw;
|
||||
```
|
||||
|
||||
### 7.4 Status Transitions
|
||||
|
||||
```
|
||||
provisioning → online → draining → offline → retired
|
||||
↘️ (heartbeat stops, >2min) ↗️
|
||||
```
|
||||
|
||||
- **online**: Heartbeat received, accepting jobs
|
||||
- **draining**: Finishing current jobs, not accepting new
|
||||
- **offline**: No heartbeat > 2 minutes
|
||||
- **retired**: Decommissioned permanently
|
||||
|
||||
---
|
||||
|
||||
## 8. Model Registry
|
||||
|
||||
### 8.1 Model Registration (by controller)
|
||||
|
||||
```sql
|
||||
INSERT INTO public.model_registry (
|
||||
id, slug, kind, vram_min_mb, vram_preferred_mb, ram_min_mb,
|
||||
quantisation, context_tokens, requires_accelerators,
|
||||
reference_throughput, load_seconds, capabilities, source_uri, license
|
||||
) VALUES (
|
||||
gen_random_uuid(), 'xtts-v2', 'tts', 6144, 8192, 2048,
|
||||
'fp16', 4096, ARRAY['cuda'], 1.0, 30,
|
||||
'{"languages": ["en","es","fr","de","it","pt","pl","tr","ru","nl","cs","ar","zh","ja","hu","ko"]}',
|
||||
'https://github.com/coqui-ai/TTS', 'CPML'
|
||||
)
|
||||
ON CONFLICT (slug) DO NOTHING;
|
||||
```
|
||||
|
||||
### 8.2 Model Residency (worker reports)
|
||||
|
||||
```sql
|
||||
-- When model loaded
|
||||
INSERT INTO public.node_models (
|
||||
compute_node_id, model_id, device_id, state, loaded_at, vram_held_mb
|
||||
) VALUES ($node_id, $model_id, $device_id, 'loaded', now(), 7168)
|
||||
ON CONFLICT (compute_node_id, model_id) DO UPDATE SET
|
||||
state = EXCLUDED.state,
|
||||
loaded_at = EXCLUDED.loaded_at,
|
||||
vram_held_mb = EXCLUDED.vram_held_mb,
|
||||
updated_at = now();
|
||||
|
||||
-- When model unloaded
|
||||
UPDATE public.node_models SET state = 'available', vram_held_mb = 0
|
||||
WHERE compute_node_id = $node_id AND model_id = $model_id;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Audio Import (UmbraAudio → UmbraTrans)
|
||||
|
||||
### 9.1 Bundle Export Format
|
||||
|
||||
UmbraAudio exports a bundle directory:
|
||||
|
||||
```
|
||||
/mnt/umbra/productions/<project>-<lang>/bundle/
|
||||
├── bundle.json # Envelope with metadata
|
||||
└── manifest.jsonl # Line-delimited JSON records
|
||||
```
|
||||
|
||||
**bundle.json:**
|
||||
```json
|
||||
{
|
||||
"bundle_format_version": 1,
|
||||
"content_hash": "sha256-of-manifest",
|
||||
"source": {
|
||||
"system": "umbraaudio",
|
||||
"node_id": "umbra-audio-01",
|
||||
"product_version": "1.2.0"
|
||||
},
|
||||
"scope": {
|
||||
"books_full_detail": ["book-dir-name"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**manifest.jsonl records:** `book`, `chapter`, `speaker`, `utterance`, `render`
|
||||
|
||||
### 9.2 Import Trigger
|
||||
|
||||
UmbraTrans imports via CLI (runs as CronJob or manual):
|
||||
|
||||
```bash
|
||||
python -m umbratrans_web.cli import-productions \
|
||||
--organization umbra \
|
||||
--allow-production \
|
||||
--confirm
|
||||
```
|
||||
|
||||
This reads bundles from `/mnt/umbra/productions/*/bundle/` and populates:
|
||||
- `project_source_refs`
|
||||
- `translation_chapters`
|
||||
- `translation_speakers`
|
||||
- `translation_utterances`
|
||||
- `artifacts` (audio renders)
|
||||
- `translation_imports` (ledger)
|
||||
|
||||
---
|
||||
|
||||
## 10. Format Metadata API (UmbraFormat ↔ UmbraTrans)
|
||||
|
||||
### 10.1 Get Format Metadata
|
||||
|
||||
```bash
|
||||
GET /api/projects/{project_id}/format-metadata?language_tag=es
|
||||
Authorization: Bearer <token>
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"project_id": "uuid",
|
||||
"language_tag": "es",
|
||||
"title": "El Cazador de Ofertas",
|
||||
"isbn13": "9788412345678",
|
||||
"cover_front_asset_id": "asset-uuid",
|
||||
"binding": "paperback",
|
||||
"trim_w_in": 6.0,
|
||||
"trim_h_in": 9.0,
|
||||
"page_count_total": 312,
|
||||
"font_primary": "Source Serif Pro",
|
||||
"font_primary_size_pt": 11.0,
|
||||
"line_height_pt": 14.5,
|
||||
"margin_inside_in": 0.75,
|
||||
"margin_outside_in": 0.625,
|
||||
"margin_top_in": 0.875,
|
||||
"margin_bottom_in": 0.875
|
||||
}
|
||||
```
|
||||
|
||||
### 10.2 Save Format Metadata
|
||||
|
||||
```bash
|
||||
PUT /api/projects/{project_id}/format-metadata
|
||||
Authorization: Bearer <token>
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"language_tag": "es",
|
||||
"isbn13": "9788412345678",
|
||||
"cover_front_asset_id": "asset-uuid",
|
||||
"binding": "paperback",
|
||||
"trim_w_in": 6.0,
|
||||
"trim_h_in": 9.0,
|
||||
"page_count_total": 312,
|
||||
"font_primary": "Source Serif Pro",
|
||||
"font_primary_size_pt": 11.0
|
||||
}
|
||||
```
|
||||
|
||||
### 10.3 Cover Upload
|
||||
|
||||
```bash
|
||||
POST /api/projects/{project_id}/cover?language_tag=es
|
||||
Authorization: Bearer <token>
|
||||
Content-Type: multipart/form-data
|
||||
|
||||
file: <cover-image.png>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. Webhooks / Event Notifications
|
||||
|
||||
### 11.1 Job Completion Webhook
|
||||
|
||||
UmbraTrans can POST to external services on job completion:
|
||||
|
||||
```json
|
||||
POST https://umbra-audio.internal/webhooks/job-complete
|
||||
Content-Type: application/json
|
||||
X-Umbra-Signature: sha256=...
|
||||
|
||||
{
|
||||
"event": "job.completed",
|
||||
"job_id": "uuid",
|
||||
"kind": "tts",
|
||||
"project_id": "uuid",
|
||||
"organization_id": "uuid",
|
||||
"status": "complete",
|
||||
"result": { ... },
|
||||
"artifacts": [
|
||||
{ "artifact_id": "uuid", "kind": "audio", "storage_uri": "..." }
|
||||
],
|
||||
"completed_at": "2026-10-31T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### 11.2 Configure Webhook URLs
|
||||
|
||||
```sql
|
||||
-- In organization or project settings
|
||||
UPDATE public.organizations
|
||||
SET webhook_urls = '{"job_complete": "https://umbra-audio.internal/webhooks/job-complete"}'
|
||||
WHERE id = $org_id;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. Credentials & Secrets
|
||||
|
||||
### 12.1 Service Accounts (Kubernetes)
|
||||
|
||||
```yaml
|
||||
# umbra-audio namespace on remote cluster
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: umbratrans-credentials
|
||||
namespace: umbra-audio
|
||||
type: Opaque
|
||||
stringData:
|
||||
DATABASE_DSN: "postgresql://umbraudio:PASS@192.168.123.14:6432/umbratrans?sslmode=require"
|
||||
NFS_SERVER: "192.168.123.14"
|
||||
NFS_PRODUCTIONS_PATH: "/mnt/umbra-data/productions"
|
||||
NFS_ARTIFACTS_PATH: "/mnt/umbra-data/artifacts"
|
||||
NODE_CREDENTIAL_PREFIX: "umbra-audio-"
|
||||
```
|
||||
|
||||
### 12.2 Node Credentials (for job queue auth)
|
||||
|
||||
```sql
|
||||
-- Created by UmbraTrans admin
|
||||
INSERT INTO public.node_credentials (
|
||||
id, compute_node_id, key_prefix, key_hash, agent_name, scopes, issued_at
|
||||
) VALUES (
|
||||
gen_random_uuid(), $node_id, 'umbra-audio-01-',
|
||||
'$argon2id$v=19$m=65536,t=3,p=4$salt$hash',
|
||||
'umbra-audio-worker', ARRAY['queue:claim', 'queue:complete', 'artifacts:write'],
|
||||
now()
|
||||
);
|
||||
```
|
||||
|
||||
Worker uses this credential to authenticate job queue operations.
|
||||
|
||||
---
|
||||
|
||||
## 13. Health Checks & Monitoring
|
||||
|
||||
### 13.1 Required Health Endpoints
|
||||
|
||||
Each service exposes:
|
||||
|
||||
```
|
||||
GET /health → { "status": "ok", "service": "umbra-audio", "version": "1.2.0" }
|
||||
GET /metrics → Prometheus metrics
|
||||
```
|
||||
|
||||
### 13.2 Key Metrics to Export
|
||||
|
||||
| Metric | Type | Labels |
|
||||
|--------|------|--------|
|
||||
| `umbra_jobs_claimed_total` | Counter | service, kind, status |
|
||||
| `umbra_job_duration_seconds` | Histogram | service, kind |
|
||||
| `umbra_artifact_bytes_written` | Counter | service, kind |
|
||||
| `umbra_node_heartbeat_age_seconds` | Gauge | node_id |
|
||||
| `umbra_gpu_utilization` | Gauge | node_id, device_id |
|
||||
| `umbra_queue_depth` | Gauge | kind, status |
|
||||
|
||||
---
|
||||
|
||||
## 14. Deployment Checklist for Remote Services
|
||||
|
||||
### UmbraAudio
|
||||
|
||||
- [ ] PostgreSQL role `umbraudio` created with correct grants
|
||||
- [ ] Node credentials inserted in `node_credentials`
|
||||
- [ ] NFS mounts configured in `/etc/fstab`
|
||||
- [ ] Compute node registered in `compute_nodes`
|
||||
- [ ] Devices registered in `compute_devices`
|
||||
- [ ] Models registered in `model_registry`
|
||||
- [ ] Heartbeat cron running (every 30s)
|
||||
- [ ] Job poller running for `tts`, `asr`, `assemble`, `encode`, `master`
|
||||
- [ ] Artifact writer writing to `/mnt/umbra/artifacts/` + `artifacts` table
|
||||
- [ ] Webhook endpoint for job completion (optional)
|
||||
|
||||
### UmbraFormat
|
||||
|
||||
- [ ] PostgreSQL role `umbraformat` created with correct grants
|
||||
- [ ] NFS mounts configured in `/etc/fstab`
|
||||
- [ ] Job poller running for `build_epub`, `build_pdf`, `cover_generate`, `metadata_validate`
|
||||
- [ ] Format metadata API integrated (GET/PUT `/format-metadata`)
|
||||
- [ ] Cover upload integrated (POST `/cover`)
|
||||
- [ ] Artifact writer for EPUB/PDF to `/mnt/umbra/artifacts/`
|
||||
- [ ] Webhook endpoint for job completion (optional)
|
||||
|
||||
---
|
||||
|
||||
## 15. Troubleshooting
|
||||
|
||||
| Issue | Diagnosis | Resolution |
|
||||
|-------|-----------|------------|
|
||||
| Job claim returns 0 rows | No queued jobs, or wrong `organization_id` | Check `SELECT * FROM jobs WHERE status='queued' AND organization_id=...` |
|
||||
| Lease expires mid-job | Job takes >30min | Increase `lease_expires_at` on claim, or heartbeat extends it |
|
||||
| NFS mount hangs | Network partition, server down | Use `hard,intr` options; check `rpcinfo -p 192.168.123.14` |
|
||||
| SSL connection failed | Cert mismatch | Use `sslmode=require` with valid CA, or `sslmode=verify-ca` |
|
||||
| Artifact conflict | Duplicate content hash | `ON CONFLICT DO NOTHING` - safe to ignore |
|
||||
| Heartbeat not updating | Clock skew | Ensure NTP sync on all nodes |
|
||||
|
||||
---
|
||||
|
||||
## 16. Contact & Escalation
|
||||
|
||||
| Role | Contact | Channel |
|
||||
|------|---------|---------|
|
||||
| UmbraTrans Platform | platform@umbra.example.com | Slack #umbra-platform |
|
||||
| Database Admin | dba@umbra.example.com | PagerDuty |
|
||||
| Network/Infra | infra@umbra.example.com | Slack #umbra-infra |
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Quick Reference Commands
|
||||
|
||||
```bash
|
||||
# Test DB connectivity
|
||||
psql "postgresql://umbraudio:PASS@192.168.123.14:6432/umbratrans?sslmode=require" -c "SELECT now();"
|
||||
|
||||
# Test NFS mount
|
||||
mount -t nfs4 192.168.123.14:/mnt/umbra-data/productions /mnt/test && ls /mnt/test
|
||||
|
||||
# Check job queue
|
||||
psql $DSN -c "SELECT kind, status, count(*) FROM jobs GROUP BY kind, status;"
|
||||
|
||||
# Check node status
|
||||
psql $DSN -c "SELECT id, name, status, last_heartbeat_at FROM compute_nodes;"
|
||||
|
||||
# View recent artifacts
|
||||
psql $DSN -c "SELECT kind, count(*), sum(byte_size) FROM artifacts GROUP BY kind;"
|
||||
```
|
||||
10
k8s/base/00-namespace.yaml
Normal file
10
k8s/base/00-namespace.yaml
Normal file
@@ -0,0 +1,10 @@
|
||||
# Copyright 2026 Umbra Tools (umbratools.com)
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: umbratrans
|
||||
labels:
|
||||
name: umbratrans
|
||||
environment: production
|
||||
97
k8s/base/01-postgresql.yaml
Normal file
97
k8s/base/01-postgresql.yaml
Normal file
@@ -0,0 +1,97 @@
|
||||
# Copyright 2026 Umbra Tools (umbratools.com)
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
# CloudNativePG Cluster for UmbraTrans
|
||||
# Requires CNPG operator installed: kubectl apply -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.24/releases/cnpg-1.24.0.yaml
|
||||
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Cluster
|
||||
metadata:
|
||||
name: umbratrans-db
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
app: umbratrans-db
|
||||
environment: production
|
||||
spec:
|
||||
instances: 3
|
||||
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
|
||||
primaryUpdateStrategy: unsupervised
|
||||
postgresql:
|
||||
parameters:
|
||||
max_connections: "200"
|
||||
shared_buffers: "256MB"
|
||||
effective_cache_size: "1GB"
|
||||
work_mem: "16MB"
|
||||
maintenance_work_mem: "128MB"
|
||||
max_worker_processes: "8"
|
||||
max_parallel_workers_per_gather: "4"
|
||||
max_parallel_workers: "8"
|
||||
wal_buffers: "16MB"
|
||||
min_wal_size: "1GB"
|
||||
max_wal_size: "4GB"
|
||||
checkpoint_completion_target: "0.9"
|
||||
random_page_cost: "1.1"
|
||||
effective_io_concurrency: "200"
|
||||
log_destination: "stderr"
|
||||
logging_collector: "off"
|
||||
log_min_duration_statement: "1000"
|
||||
log_line_prefix: "%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h "
|
||||
log_checkpoints: "on"
|
||||
log_connections: "on"
|
||||
log_disconnections: "on"
|
||||
log_lock_waits: "on"
|
||||
log_temp_files: "10MB"
|
||||
log_autovacuum_min_duration: "1000"
|
||||
statement_timeout: "300000"
|
||||
idle_in_transaction_session_timeout: "300000"
|
||||
bootstrap:
|
||||
initdb:
|
||||
database: umbratrans
|
||||
owner: umbratrans
|
||||
secret:
|
||||
name: umbratrans-db-credentials
|
||||
postInitApplicationSQL:
|
||||
- CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
|
||||
- CREATE EXTENSION IF NOT EXISTS "citext";
|
||||
- CREATE EXTENSION IF NOT EXISTS "pgcrypto";
|
||||
storage:
|
||||
storageClass: nfs-client
|
||||
size: 50Gi
|
||||
backup:
|
||||
barmanObjectStore:
|
||||
destinationPath: s3://umbra-backups/postgresql/umbratrans-db
|
||||
endpointURL: https://s3.example.com
|
||||
s3Credentials:
|
||||
accessKeyId:
|
||||
name: umbratrans-backup-credentials
|
||||
key: ACCESS_KEY_ID
|
||||
secretAccessKey:
|
||||
name: umbratrans-backup-credentials
|
||||
key: SECRET_ACCESS_KEY
|
||||
retentionPolicy: "30d"
|
||||
schedule: "0 2 * * *"
|
||||
proxy:
|
||||
pgBouncer:
|
||||
imageName: ghcr.io/cloudnative-pg/pgbouncer:1.23
|
||||
replicas: 2
|
||||
poolMode: transaction
|
||||
maxClientConn: 2000
|
||||
defaultPoolSize: 20
|
||||
maxDbConnections: 100
|
||||
admin:
|
||||
secret:
|
||||
name: umbratrans-pgbouncer-admin
|
||||
monitoring:
|
||||
enablePodMonitor: true
|
||||
customQueriesConfigMap:
|
||||
- name: umbratrans-db-queries
|
||||
resources:
|
||||
requests:
|
||||
memory: "1Gi"
|
||||
cpu: "500m"
|
||||
limits:
|
||||
memory: "2Gi"
|
||||
cpu: "2000m"
|
||||
affinity:
|
||||
podAntiAffinityType: preferred
|
||||
topologyKey: topology.kubernetes.io/zone
|
||||
150
k8s/base/02-nfs-storage.yaml
Normal file
150
k8s/base/02-nfs-storage.yaml
Normal file
@@ -0,0 +1,150 @@
|
||||
# Copyright 2026 Umbra Tools (umbratools.com)
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
# NFS CSI Driver installation (apply once per cluster)
|
||||
# kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/v4.7.0/deploy/example/nfs-driver.yaml
|
||||
|
||||
---
|
||||
# StorageClass for NFS
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: nfs-client
|
||||
annotations:
|
||||
storageclass.kubernetes.io/is-default-class: "true"
|
||||
provisioner: nfs.csi.k8s.io
|
||||
parameters:
|
||||
server: 192.168.123.14 # NFS server (this machine)
|
||||
share: /mnt/umbra-data
|
||||
reclaimPolicy: Retain
|
||||
volumeBindingMode: Immediate
|
||||
mountOptions:
|
||||
- hard
|
||||
- nfsvers=4.1
|
||||
- rsize=1048576
|
||||
- wsize=1048576
|
||||
- timeo=600
|
||||
- retrans=2
|
||||
|
||||
---
|
||||
# PersistentVolume for existing productions data
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: umbratrans-productions
|
||||
labels:
|
||||
app: umbratrans
|
||||
data: productions
|
||||
spec:
|
||||
capacity:
|
||||
storage: 100Gi
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: nfs-client
|
||||
csi:
|
||||
driver: nfs.csi.k8s.io
|
||||
volumeHandle: umbratrans-productions
|
||||
volumeAttributes:
|
||||
server: 192.168.123.14 # NFS server (this machine)
|
||||
share: /mnt/umbra-data/productions
|
||||
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: umbratrans-productions
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
app: umbratrans
|
||||
data: productions
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
storageClassName: nfs-client
|
||||
resources:
|
||||
requests:
|
||||
storage: 100Gi
|
||||
volumeName: umbratrans-productions
|
||||
|
||||
---
|
||||
# PersistentVolume for artifacts (audio files, covers, etc.)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: umbratrans-artifacts
|
||||
labels:
|
||||
app: umbratrans
|
||||
data: artifacts
|
||||
spec:
|
||||
capacity:
|
||||
storage: 500Gi
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: nfs-client
|
||||
csi:
|
||||
driver: nfs.csi.k8s.io
|
||||
volumeHandle: umbratrans-artifacts
|
||||
volumeAttributes:
|
||||
server: 192.168.123.14
|
||||
share: /mnt/umbra-data/artifacts
|
||||
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: umbratrans-artifacts
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
app: umbratrans
|
||||
data: artifacts
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
storageClassName: nfs-client
|
||||
resources:
|
||||
requests:
|
||||
storage: 500Gi
|
||||
volumeName: umbratrans-artifacts
|
||||
|
||||
---
|
||||
# PersistentVolume for database backups (optional, separate from PG backup)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: umbratrans-backups
|
||||
labels:
|
||||
app: umbratrans
|
||||
data: backups
|
||||
spec:
|
||||
capacity:
|
||||
storage: 200Gi
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: nfs-client
|
||||
csi:
|
||||
driver: nfs.csi.k8s.io
|
||||
volumeHandle: umbratrans-backups
|
||||
volumeAttributes:
|
||||
server: 192.168.123.14
|
||||
share: /mnt/umbra-data/backups
|
||||
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: umbratrans-backups
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
app: umbratrans
|
||||
data: backups
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
storageClassName: nfs-client
|
||||
resources:
|
||||
requests:
|
||||
storage: 200Gi
|
||||
volumeName: umbratrans-backups
|
||||
284
k8s/base/03-web-deployment.yaml
Normal file
284
k8s/base/03-web-deployment.yaml
Normal file
@@ -0,0 +1,284 @@
|
||||
# Copyright 2026 Umbra Tools (umbratools.com)
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
---
|
||||
# ServiceAccount for web
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: umbratrans-web
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
app: umbratrans-web
|
||||
|
||||
---
|
||||
# Role for web (minimal permissions)
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: umbratrans-web
|
||||
namespace: umbratrans
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["pods", "services", "configmaps"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
- apiGroups: ["postgresql.cnpg.io"]
|
||||
resources: ["clusters", "backups"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: umbratrans-web
|
||||
namespace: umbratrans
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: umbratrans-web
|
||||
namespace: umbratrans
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: umbratrans-web
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
|
||||
---
|
||||
# ConfigMap for static files (served by web)
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: umbratrans-static
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
app: umbratrans-web
|
||||
data:
|
||||
# Static files are built into the image, this is for runtime config
|
||||
nginx.conf: |
|
||||
# Not used - FastAPI serves static files directly
|
||||
|
||||
---
|
||||
# Secret for web configuration (non-sensitive)
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: umbratrans-web-config
|
||||
namespace: umbratrans
|
||||
type: Opaque
|
||||
stringData:
|
||||
UMBRA_ENV: "production"
|
||||
UMBRATRANS_REPOSITORY: "postgres"
|
||||
UMBRATRANS_PRODUCTIONS_ROOT: "/mnt/productions"
|
||||
UMBRATRANS_STATIC_ROOT: "/app/web"
|
||||
UMBRATRANS_ALLOWED_HOSTS: "umbratrans.bdmurphy.me,localhost"
|
||||
UMBRATRANS_VERIFICATION_BASE_URL: "https://umbratrans.bdmurphy.me"
|
||||
UMBRATRANS_COOKIE_SECURE: "true"
|
||||
UMBRATRANS_REQUIRE_CSRF: "true"
|
||||
UMBRATRANS_MAILER: "smtp"
|
||||
UMBRATRANS_SMTP_HOST: "smtp.example.com"
|
||||
UMBRATRANS_SMTP_PORT: "587"
|
||||
UMBRATRANS_SMTP_USERNAME: "umbra@example.com"
|
||||
UMBRATRANS_SMTP_SENDER: "umbratrans@umbra.example.com"
|
||||
UMBRATRANS_SMTP_STARTTLS: "true"
|
||||
# Sensitive values in separate secret
|
||||
# UMBRATRANS_DATABASE_DSN: from umbratrans-db-credentials
|
||||
# UMBRATRANS_SMTP_PASSWORD: from umbratrans-smtp
|
||||
# UMBRATRANS_TOKEN_PEPPER: from umbratrans-secrets
|
||||
|
||||
---
|
||||
# Deployment for web API + UI
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: umbratrans-web
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
app: umbratrans-web
|
||||
component: web
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: umbratrans-web
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxSurge: 1
|
||||
maxUnavailable: 0
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: umbratrans-web
|
||||
component: web
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "8442"
|
||||
prometheus.io/path: "/api/metrics"
|
||||
spec:
|
||||
serviceAccountName: umbratrans-web
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
initContainers:
|
||||
- name: migrate
|
||||
image: umbratrans-web:latest
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["python", "-m", "umbratrans_web.cli", "migrate"]
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: umbratrans-web-config
|
||||
- secretRef:
|
||||
name: umbratrans-db-credentials
|
||||
- secretRef:
|
||||
name: umbratrans-smtp
|
||||
- secretRef:
|
||||
name: umbratrans-secrets
|
||||
resources:
|
||||
requests:
|
||||
memory: "128Mi"
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: "256Mi"
|
||||
cpu: "500m"
|
||||
containers:
|
||||
- name: web
|
||||
image: umbratrans-web:latest
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["python", "-m", "umbratrans_web", "serve"]
|
||||
args:
|
||||
- "--host"
|
||||
- "0.0.0.0"
|
||||
- "--port"
|
||||
- "8442"
|
||||
- "--repository"
|
||||
- "postgres"
|
||||
- "--allow-production"
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8442
|
||||
protocol: TCP
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: umbratrans-web-config
|
||||
- secretRef:
|
||||
name: umbratrans-db-credentials
|
||||
- secretRef:
|
||||
name: umbratrans-smtp
|
||||
- secretRef:
|
||||
name: umbratrans-secrets
|
||||
volumeMounts:
|
||||
- name: productions
|
||||
mountPath: /mnt/productions
|
||||
readOnly: true
|
||||
- name: artifacts
|
||||
mountPath: /mnt/artifacts
|
||||
readOnly: false
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /api/health
|
||||
port: 8442
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 30
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /api/health
|
||||
port: 8442
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 3
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /api/health
|
||||
port: 8442
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 30
|
||||
resources:
|
||||
requests:
|
||||
memory: "256Mi"
|
||||
cpu: "250m"
|
||||
limits:
|
||||
memory: "512Mi"
|
||||
cpu: "1000m"
|
||||
volumes:
|
||||
- name: productions
|
||||
persistentVolumeClaim:
|
||||
claimName: umbratrans-productions
|
||||
- name: artifacts
|
||||
persistentVolumeClaim:
|
||||
claimName: umbratrans-artifacts
|
||||
- name: tmp
|
||||
emptyDir:
|
||||
sizeLimit: 1Gi
|
||||
|
||||
---
|
||||
# Service for web
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: umbratrans-web
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
app: umbratrans-web
|
||||
spec:
|
||||
selector:
|
||||
app: umbratrans-web
|
||||
ports:
|
||||
- name: http
|
||||
port: 80
|
||||
targetPort: 8442
|
||||
protocol: TCP
|
||||
type: ClusterIP
|
||||
|
||||
---
|
||||
# HorizontalPodAutoscaler for web
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: umbratrans-web
|
||||
namespace: umbratrans
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: umbratrans-web
|
||||
minReplicas: 2
|
||||
maxReplicas: 10
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 80
|
||||
behavior:
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 300
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 10
|
||||
periodSeconds: 60
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 60
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 50
|
||||
periodSeconds: 60
|
||||
- type: Pods
|
||||
value: 2
|
||||
periodSeconds: 60
|
||||
selectPolicy: Max
|
||||
338
k8s/base/04-worker-deployments.yaml
Normal file
338
k8s/base/04-worker-deployments.yaml
Normal file
@@ -0,0 +1,338 @@
|
||||
# Copyright 2026 Umbra Tools (umbratools.com)
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
---
|
||||
# ServiceAccount for workers
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: umbratrans-worker
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
app: umbratrans-worker
|
||||
|
||||
---
|
||||
# Role for workers
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: umbratrans-worker
|
||||
namespace: umbratrans
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["pods", "services", "configmaps", "secrets"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
- apiGroups: ["batch"]
|
||||
resources: ["jobs", "cronjobs"]
|
||||
verbs: ["get", "list", "watch", "create", "update", "patch"]
|
||||
- apiGroups: ["postgresql.cnpg.io"]
|
||||
resources: ["clusters", "backups"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: umbratrans-worker
|
||||
namespace: umbratrans
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: umbratrans-worker
|
||||
namespace: umbratrans
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: umbratrans-worker
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
|
||||
---
|
||||
# Secret for worker configuration
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: umbratrans-worker-config
|
||||
namespace: umbratrans
|
||||
type: Opaque
|
||||
stringData:
|
||||
UMBRA_ENV: "production"
|
||||
UMBRATRANS_REPOSITORY: "postgres"
|
||||
UMBRATRANS_PRODUCTIONS_ROOT: "/mnt/productions"
|
||||
UMBRATRANS_ALLOWED_HOSTS: "umbratrans.bdmurphy.me"
|
||||
OLLAMA_BASE_URL: "http://ollama:11434/v1"
|
||||
# UMBRATRANS_DATABASE_DSN: from umbratrans-db-credentials
|
||||
|
||||
---
|
||||
# Deployment for Translation Workers
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: umbratrans-worker-translation
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
app: umbratrans-worker-translation
|
||||
component: worker
|
||||
workload: translation
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: umbratrans-worker-translation
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxSurge: 1
|
||||
maxUnavailable: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: umbratrans-worker-translation
|
||||
component: worker
|
||||
workload: translation
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "9090"
|
||||
prometheus.io/path: "/metrics"
|
||||
spec:
|
||||
serviceAccountName: umbratrans-worker
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
# Node affinity for GPU nodes
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: nvidia.com/gpu.present
|
||||
operator: In
|
||||
values: ["true"]
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 100
|
||||
preference:
|
||||
matchExpressions:
|
||||
- key: workload
|
||||
operator: In
|
||||
values: ["translation"]
|
||||
tolerations:
|
||||
- key: "nvidia.com/gpu"
|
||||
operator: "Exists"
|
||||
effect: "NoSchedule"
|
||||
containers:
|
||||
- name: worker
|
||||
image: umbratrans-worker:latest
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["python", "-m", "umbratrans_web.cli", "worker", "translation"]
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: umbratrans-worker-config
|
||||
- secretRef:
|
||||
name: umbratrans-db-credentials
|
||||
- secretRef:
|
||||
name: umbratrans-secrets
|
||||
volumeMounts:
|
||||
- name: productions
|
||||
mountPath: /mnt/productions
|
||||
- name: artifacts
|
||||
mountPath: /mnt/artifacts
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
resources:
|
||||
requests:
|
||||
memory: "2Gi"
|
||||
cpu: "2000m"
|
||||
nvidia.com/gpu: "1"
|
||||
limits:
|
||||
memory: "4Gi"
|
||||
cpu: "4000m"
|
||||
nvidia.com/gpu: "1"
|
||||
livenessProbe:
|
||||
exec:
|
||||
command: ["python", "-c", "import psycopg; psycopg.connect('$UMBRATRANS_DATABASE_DSN', connect_timeout=3).close()"]
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 60
|
||||
timeoutSeconds: 10
|
||||
failureThreshold: 3
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["python", "-c", "import psycopg; psycopg.connect('$UMBRATRANS_DATABASE_DSN', connect_timeout=3).close()"]
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 30
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
volumes:
|
||||
- name: productions
|
||||
persistentVolumeClaim:
|
||||
claimName: umbratrans-productions
|
||||
- name: artifacts
|
||||
persistentVolumeClaim:
|
||||
claimName: umbratrans-artifacts
|
||||
- name: tmp
|
||||
emptyDir:
|
||||
sizeLimit: 2Gi
|
||||
|
||||
---
|
||||
# HorizontalPodAutoscaler for translation workers
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: umbratrans-worker-translation
|
||||
namespace: umbratrans
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: umbratrans-worker-translation
|
||||
minReplicas: 1
|
||||
maxReplicas: 20
|
||||
metrics:
|
||||
- type: External
|
||||
external:
|
||||
metric:
|
||||
name: umbratrans_translation_queue_depth
|
||||
target:
|
||||
type: AverageValue
|
||||
averageValue: "10"
|
||||
behavior:
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 600
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 60
|
||||
|
||||
---
|
||||
# Deployment for Audio Workers
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: umbratrans-worker-audio
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
app: umbratrans-worker-audio
|
||||
component: worker
|
||||
workload: audio
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: umbratrans-worker-audio
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxSurge: 1
|
||||
maxUnavailable: 0
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: umbratrans-worker-audio
|
||||
component: worker
|
||||
workload: audio
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "9090"
|
||||
prometheus.io/path: "/metrics"
|
||||
spec:
|
||||
serviceAccountName: umbratrans-worker
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: nvidia.com/gpu.present
|
||||
operator: In
|
||||
values: ["true"]
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 100
|
||||
preference:
|
||||
matchExpressions:
|
||||
- key: workload
|
||||
operator: In
|
||||
values: ["audio"]
|
||||
tolerations:
|
||||
- key: "nvidia.com/gpu"
|
||||
operator: "Exists"
|
||||
effect: "NoSchedule"
|
||||
containers:
|
||||
- name: worker
|
||||
image: umbratrans-worker:latest
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["python", "-m", "umbratrans_web.cli", "worker", "audio"]
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: umbratrans-worker-config
|
||||
- secretRef:
|
||||
name: umbratrans-db-credentials
|
||||
- secretRef:
|
||||
name: umbratrans-secrets
|
||||
volumeMounts:
|
||||
- name: productions
|
||||
mountPath: /mnt/productions
|
||||
- name: artifacts
|
||||
mountPath: /mnt/artifacts
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
resources:
|
||||
requests:
|
||||
memory: "4Gi"
|
||||
cpu: "2000m"
|
||||
nvidia.com/gpu: "1"
|
||||
limits:
|
||||
memory: "8Gi"
|
||||
cpu: "4000m"
|
||||
nvidia.com/gpu: "1"
|
||||
livenessProbe:
|
||||
exec:
|
||||
command: ["python", "-c", "import psycopg; psycopg.connect('$UMBRATRANS_DATABASE_DSN', connect_timeout=3).close()"]
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 60
|
||||
timeoutSeconds: 10
|
||||
failureThreshold: 3
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["python", "-c", "import psycopg; psycopg.connect('$UMBRATRANS_DATABASE_DSN', connect_timeout=3).close()"]
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 30
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
volumes:
|
||||
- name: productions
|
||||
persistentVolumeClaim:
|
||||
claimName: umbratrans-productions
|
||||
- name: artifacts
|
||||
persistentVolumeClaim:
|
||||
claimName: umbratrans-artifacts
|
||||
- name: tmp
|
||||
emptyDir:
|
||||
sizeLimit: 4Gi
|
||||
|
||||
---
|
||||
# HorizontalPodAutoscaler for audio workers
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: umbratrans-worker-audio
|
||||
namespace: umbratrans
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: umbratrans-worker-audio
|
||||
minReplicas: 1
|
||||
maxReplicas: 10
|
||||
metrics:
|
||||
- type: External
|
||||
external:
|
||||
metric:
|
||||
name: umbratrans_audio_queue_depth
|
||||
target:
|
||||
type: AverageValue
|
||||
averageValue: "5"
|
||||
behavior:
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 900
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 120
|
||||
169
k8s/base/05-ingress-network.yaml
Normal file
169
k8s/base/05-ingress-network.yaml
Normal file
@@ -0,0 +1,169 @@
|
||||
# Copyright 2026 Umbra Tools (umbratools.com)
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
---
|
||||
# Ingress for web UI and API
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: umbratrans-web
|
||||
namespace: umbratrans
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
|
||||
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
|
||||
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
|
||||
nginx.ingress.kubernetes.io/client-body-timeout: "300"
|
||||
nginx.ingress.kubernetes.io/ssl-redirect: "true"
|
||||
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
|
||||
nginx.ingress.kubernetes.io/hsts: "true"
|
||||
nginx.ingress.kubernetes.io/hsts-max-age: "31536000"
|
||||
nginx.ingress.kubernetes.io/hsts-include-subdomains: "true"
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
tls:
|
||||
- hosts:
|
||||
- umbratrans.bdmurphy.me
|
||||
secretName: umbratrans-tls
|
||||
rules:
|
||||
- host: umbratrans.bdmurphy.me
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: umbratrans-web
|
||||
port:
|
||||
number: 80
|
||||
|
||||
---
|
||||
# NetworkPolicy for web - allow ingress from nginx, egress to DB
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: umbratrans-web
|
||||
namespace: umbratrans
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: umbratrans-web
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
name: ingress-nginx
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8442
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
name: monitoring
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8442
|
||||
egress:
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: umbratrans-db
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
- protocol: TCP
|
||||
port: 6432 # PgBouncer
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
name: kube-system
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- to: []
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443 # External APIs (Ollama, SMTP, etc.)
|
||||
|
||||
---
|
||||
# NetworkPolicy for workers - egress to DB and NFS
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: umbratrans-worker
|
||||
namespace: umbratrans
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
component: worker
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: umbratrans-db
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
- protocol: TCP
|
||||
port: 6432
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
name: kube-system
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- to: []
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- protocol: TCP
|
||||
port: 11434 # Ollama
|
||||
- protocol: TCP
|
||||
port: 2049 # NFS
|
||||
|
||||
---
|
||||
# NetworkPolicy for database - allow from web and workers only
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: umbratrans-db
|
||||
namespace: umbratrans
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: umbratrans-db
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: umbratrans-web
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
component: worker
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
name: monitoring
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
- protocol: TCP
|
||||
port: 6432
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: umbratrans-db
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
148
k8s/base/06-import-job.yaml
Normal file
148
k8s/base/06-import-job.yaml
Normal file
@@ -0,0 +1,148 @@
|
||||
# Copyright 2026 Umbra Tools (umbratools.com)
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
---
|
||||
# Job to import existing productions data into PostgreSQL
|
||||
# Run once after initial deployment: kubectl apply -f 06-import-job.yaml
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: umbratrans-import-productions
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
app: umbratrans-import
|
||||
component: import
|
||||
annotations:
|
||||
# Allow job to run for a long time
|
||||
batch.kubernetes.io/job-tracking: "true"
|
||||
spec:
|
||||
ttlSecondsAfterFinished: 86400 # Keep job for 24h after completion
|
||||
backoffLimit: 3
|
||||
activeDeadlineSeconds: 7200 # 2 hours max
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: umbratrans-import
|
||||
component: import
|
||||
annotations:
|
||||
prometheus.io/scrape: "false"
|
||||
spec:
|
||||
serviceAccountName: umbratrans-worker
|
||||
restartPolicy: OnFailure
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
initContainers:
|
||||
- name: wait-for-db
|
||||
image: postgres:16-alpine
|
||||
command: ["sh", "-c", "until pg_isready -h umbratrans-db-rw -U umbratrans -d umbratrans; do sleep 5; done"]
|
||||
env:
|
||||
- name: PGPASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: umbratrans-db-credentials
|
||||
key: password
|
||||
containers:
|
||||
- name: import
|
||||
image: umbratrans-worker:latest
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["python", "-m", "umbratrans_web.cli", "import-productions"]
|
||||
args:
|
||||
- "--organization"
|
||||
- "umbra" # Organization slug
|
||||
- "--confirm"
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: umbratrans-worker-config
|
||||
- secretRef:
|
||||
name: umbratrans-db-credentials
|
||||
- secretRef:
|
||||
name: umbratrans-secrets
|
||||
volumeMounts:
|
||||
- name: productions
|
||||
mountPath: /mnt/productions
|
||||
readOnly: true
|
||||
- name: artifacts
|
||||
mountPath: /mnt/artifacts
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "500m"
|
||||
limits:
|
||||
memory: "2Gi"
|
||||
cpu: "2000m"
|
||||
volumes:
|
||||
- name: productions
|
||||
persistentVolumeClaim:
|
||||
claimName: umbratrans-productions
|
||||
- name: artifacts
|
||||
persistentVolumeClaim:
|
||||
claimName: umbratrans-artifacts
|
||||
|
||||
---
|
||||
# CronJob for periodic re-import (optional - for catching new productions)
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: umbratrans-import-productions-schedule
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
app: umbratrans-import
|
||||
component: import
|
||||
spec:
|
||||
schedule: "0 3 * * *" # Daily at 3 AM UTC
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
spec:
|
||||
ttlSecondsAfterFinished: 86400
|
||||
backoffLimit: 1
|
||||
activeDeadlineSeconds: 7200
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: umbratrans-worker
|
||||
restartPolicy: OnFailure
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
containers:
|
||||
- name: import
|
||||
image: umbratrans-worker:latest
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["python", "-m", "umbratrans_web.cli", "import-productions"]
|
||||
args:
|
||||
- "--organization"
|
||||
- "umbra"
|
||||
- "--incremental" # Only import new/changed
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: umbratrans-worker-config
|
||||
- secretRef:
|
||||
name: umbratrans-db-credentials
|
||||
- secretRef:
|
||||
name: umbratrans-secrets
|
||||
volumeMounts:
|
||||
- name: productions
|
||||
mountPath: /mnt/productions
|
||||
readOnly: true
|
||||
- name: artifacts
|
||||
mountPath: /mnt/artifacts
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "500m"
|
||||
limits:
|
||||
memory: "2Gi"
|
||||
cpu: "2000m"
|
||||
volumes:
|
||||
- name: productions
|
||||
persistentVolumeClaim:
|
||||
claimName: umbratrans-productions
|
||||
- name: artifacts
|
||||
persistentVolumeClaim:
|
||||
claimName: umbratrans-artifacts
|
||||
286
k8s/base/07-monitoring-secrets.yaml
Normal file
286
k8s/base/07-monitoring-secrets.yaml
Normal file
@@ -0,0 +1,286 @@
|
||||
# Copyright 2026 Umbra Tools (umbratools.com)
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
---
|
||||
# Secret for database credentials (created by CNPG, but we reference it)
|
||||
# This is a template - CNPG creates the actual secret
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: umbratrans-db-credentials
|
||||
namespace: umbratrans
|
||||
type: kubernetes.io/basic-auth
|
||||
stringData:
|
||||
username: "umbratrans"
|
||||
password: "CHANGE_ME_GENERATED_BY_CNPG"
|
||||
# DSN format: postgresql://user:pass@host:port/db?sslmode=require
|
||||
dsn: "postgresql://umbratrans:CHANGE_ME@umbratrans-db-rw:5432/umbratrans?sslmode=require"
|
||||
|
||||
---
|
||||
# Secret for SMTP credentials
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: umbratrans-smtp
|
||||
namespace: umbratrans
|
||||
type: Opaque
|
||||
stringData:
|
||||
host: "smtp.example.com"
|
||||
port: "587"
|
||||
username: "umbra@example.com"
|
||||
password: "CHANGE_ME_SMTP_PASSWORD"
|
||||
sender: "umbratrans@umbra.example.com"
|
||||
|
||||
---
|
||||
# Secret for application secrets (token pepper, etc.)
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: umbratrans-secrets
|
||||
namespace: umbratrans
|
||||
type: Opaque
|
||||
stringData:
|
||||
token_pepper: "CHANGE_ME_32_BYTES_MINIMUM_BASE64_ENCODED"
|
||||
|
||||
---
|
||||
# Secret for PgBouncer admin (generated by CNPG)
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: umbratrans-pgbouncer-admin
|
||||
namespace: umbratrans
|
||||
type: Opaque
|
||||
stringData:
|
||||
username: "pgbouncer_admin"
|
||||
password: "CHANGE_ME_GENERATED_BY_CNPG"
|
||||
|
||||
---
|
||||
# Secret for backup credentials (S3)
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: umbratrans-backup-credentials
|
||||
namespace: umbratrans
|
||||
type: Opaque
|
||||
stringData:
|
||||
ACCESS_KEY_ID: "CHANGE_ME"
|
||||
SECRET_ACCESS_KEY: "CHANGE_ME"
|
||||
|
||||
---
|
||||
# TLS certificate for ingress (managed by cert-manager)
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: umbratrans-tls
|
||||
namespace: umbratrans
|
||||
type: kubernetes.io/tls
|
||||
data:
|
||||
# cert-manager will populate this
|
||||
tls.crt: ""
|
||||
tls.key: ""
|
||||
|
||||
---
|
||||
# ServiceMonitor for Prometheus (requires Prometheus Operator)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: umbratrans-web
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
release: prometheus
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: umbratrans-web
|
||||
endpoints:
|
||||
- port: http
|
||||
path: /api/metrics
|
||||
interval: 30s
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- umbratrans
|
||||
|
||||
---
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: umbratrans-workers
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
release: prometheus
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
component: worker
|
||||
endpoints:
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
interval: 30s
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- umbratrans
|
||||
|
||||
---
|
||||
# PodMonitor for PostgreSQL (CNPG)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
name: umbratrans-db
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
release: prometheus
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: umbratrans-db
|
||||
podMetricsEndpoints:
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
interval: 30s
|
||||
|
||||
---
|
||||
# ConfigMap for custom PostgreSQL queries
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: umbratrans-db-queries
|
||||
namespace: umbratrans
|
||||
data:
|
||||
umbratrans_queries.yaml: |
|
||||
- name: umbratrans_projects_total
|
||||
query: |
|
||||
SELECT count(*) as value FROM projects;
|
||||
metrics:
|
||||
- value
|
||||
- name: umbratrans_active_jobs
|
||||
query: |
|
||||
SELECT count(*) as value FROM jobs WHERE status IN ('queued', 'running', 'blocked');
|
||||
metrics:
|
||||
- value
|
||||
- name: umbratrans_translation_queue_depth
|
||||
query: |
|
||||
SELECT count(*) as value FROM jobs WHERE kind = 'translate' AND status = 'queued';
|
||||
metrics:
|
||||
- value
|
||||
- name: umbratrans_audio_queue_depth
|
||||
query: |
|
||||
SELECT count(*) as value FROM jobs WHERE kind LIKE 'audio%' AND status = 'queued';
|
||||
metrics:
|
||||
- value
|
||||
- name: umbratrans_artifacts_total
|
||||
query: |
|
||||
SELECT count(*) as value FROM artifacts;
|
||||
metrics:
|
||||
- value
|
||||
- name: umbratrans_artifacts_size_bytes
|
||||
query: |
|
||||
SELECT sum(byte_size) as value FROM artifacts;
|
||||
metrics:
|
||||
- value
|
||||
- name: umbratrans_import_runs_total
|
||||
query: |
|
||||
SELECT count(*) as value FROM translation_imports;
|
||||
metrics:
|
||||
- value
|
||||
|
||||
---
|
||||
# PrometheusRule for alerts
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: umbratrans-alerts
|
||||
namespace: umbratrans
|
||||
labels:
|
||||
release: prometheus
|
||||
spec:
|
||||
groups:
|
||||
- name: umbratrans.rules
|
||||
rules:
|
||||
- alert: UmbraTransWebDown
|
||||
expr: up{job="umbratrans-web"} == 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "UmbraTrans web is down"
|
||||
description: "UmbraTrans web has been down for more than 2 minutes."
|
||||
|
||||
- alert: UmbraTransHighErrorRate
|
||||
expr: |
|
||||
sum(rate(http_requests_total{job="umbratrans-web",code=~"5.."}[5m]))
|
||||
/
|
||||
sum(rate(http_requests_total{job="umbratrans-web"}[5m])) > 0.05
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High error rate on UmbraTrans web"
|
||||
description: "Error rate is above 5% for the last 5 minutes."
|
||||
|
||||
- alert: UmbraTransDBConnectionsHigh
|
||||
expr: |
|
||||
pg_stat_database_numbackends / pg_settings_max_connections > 0.8
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "PostgreSQL connections near limit"
|
||||
description: "Database connections are above 80% of max_connections."
|
||||
|
||||
- alert: UmbraTransTranslationQueueBacklog
|
||||
expr: umbratrans_translation_queue_depth > 100
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Translation queue backlog"
|
||||
description: "Translation queue has more than 100 pending jobs for 10 minutes."
|
||||
|
||||
- alert: UmbraTransAudioQueueBacklog
|
||||
expr: umbratrans_audio_queue_depth > 50
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Audio queue backlog"
|
||||
description: "Audio queue has more than 50 pending jobs for 10 minutes."
|
||||
|
||||
- alert: UmbraTransDiskSpaceLow
|
||||
expr: |
|
||||
(kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes) < 0.1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Disk space low on UmbraTrans volumes"
|
||||
description: "Less than 10% disk space remaining on persistent volumes."
|
||||
|
||||
- alert: UmbraTransWorkerTranslationDown
|
||||
expr: |
|
||||
kube_deployment_status_replicas_available{deployment="umbratrans-worker-translation"} < 1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "No translation workers available"
|
||||
description: "All translation worker pods are down."
|
||||
|
||||
- alert: UmbraTransWorkerAudioDown
|
||||
expr: |
|
||||
kube_deployment_status_replicas_available{deployment="umbratrans-worker-audio"} < 1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "No audio workers available"
|
||||
description: "All audio worker pods are down."
|
||||
|
||||
- alert: UmbraTransImportJobFailed
|
||||
expr: |
|
||||
kube_job_status_failed{job=~"umbratrans-import.*"} > 0
|
||||
for: 1m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "UmbraTrans import job failed"
|
||||
description: "Production import job has failed."
|
||||
61
k8s/base/kustomization.yaml
Normal file
61
k8s/base/kustomization.yaml
Normal file
@@ -0,0 +1,61 @@
|
||||
# Copyright 2026 Umbra Tools (umbratools.com)
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: umbratrans
|
||||
|
||||
resources:
|
||||
- 00-namespace.yaml
|
||||
- 01-postgresql.yaml
|
||||
- 02-nfs-storage.yaml
|
||||
- 03-web-deployment.yaml
|
||||
- 04-worker-deployments.yaml
|
||||
- 05-ingress-network.yaml
|
||||
- 06-import-job.yaml
|
||||
- 07-monitoring-secrets.yaml
|
||||
|
||||
commonLabels:
|
||||
app.kubernetes.io/name: umbratrans
|
||||
app.kubernetes.io/part-of: umbratrans
|
||||
app.kubernetes.io/managed-by: kustomize
|
||||
|
||||
images:
|
||||
- name: umbratrans-web
|
||||
newTag: latest
|
||||
- name: umbratrans-worker
|
||||
newTag: latest
|
||||
|
||||
# Patches for different environments
|
||||
patches:
|
||||
- patch: |-
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: umbratrans-web
|
||||
spec:
|
||||
replicas: 1
|
||||
target:
|
||||
kind: Deployment
|
||||
name: umbratrans-web
|
||||
- patch: |-
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: umbratrans-worker-translation
|
||||
spec:
|
||||
replicas: 1
|
||||
target:
|
||||
kind: Deployment
|
||||
name: umbratrans-worker-translation
|
||||
- patch: |-
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: umbratrans-worker-audio
|
||||
spec:
|
||||
replicas: 0
|
||||
target:
|
||||
kind: Deployment
|
||||
name: umbratrans-worker-audio
|
||||
351
scripts/deploy-database.sh
Normal file
351
scripts/deploy-database.sh
Normal file
@@ -0,0 +1,351 @@
|
||||
#!/usr/bin/env bash
|
||||
# Copyright 2026 Umbra Tools (umbratools.com)
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
# Deploy UmbraTrans PostgreSQL Database
|
||||
# This script installs CNPG operator, creates secrets, deploys the cluster,
|
||||
# runs migrations, and verifies the deployment.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Configuration
|
||||
NAMESPACE="umbratrans"
|
||||
CLUSTER_NAME="umbratrans-db"
|
||||
CNPG_VERSION="1.24.0"
|
||||
POSTGRES_VERSION="16.4"
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
log_info() { echo -e "${BLUE}[INFO]${NC} $*"; }
|
||||
log_success() { echo -e "${GREEN}[SUCCESS]${NC} $*"; }
|
||||
log_warning() { echo -e "${YELLOW}[WARNING]${NC} $*"; }
|
||||
log_error() { echo -e "${RED}[ERROR]${NC} $*"; }
|
||||
|
||||
# Check prerequisites
|
||||
check_prerequisites() {
|
||||
log_info "Checking prerequisites..."
|
||||
|
||||
if ! command -v kubectl &> /dev/null; then
|
||||
log_error "kubectl not found. Please install kubectl."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! kubectl cluster-info &> /dev/null; then
|
||||
log_error "Cannot connect to Kubernetes cluster. Check kubeconfig."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log_success "Prerequisites check passed"
|
||||
}
|
||||
|
||||
# Install CNPG operator
|
||||
install_cnpg_operator() {
|
||||
log_info "Installing CloudNativePG operator v${CNPG_VERSION}..."
|
||||
|
||||
if kubectl get crd clusters.postgresql.cnpg.io &> /dev/null; then
|
||||
log_warning "CNPG operator already installed (CRD exists)"
|
||||
return 0
|
||||
fi
|
||||
|
||||
kubectl apply -f "https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-${CNPG_VERSION}/releases/cnpg-${CNPG_VERSION}.yaml"
|
||||
|
||||
log_info "Waiting for CNPG operator to be ready..."
|
||||
kubectl wait --for=condition=Available deployment/cnpg-controller-manager \
|
||||
-n cnpg-system --timeout=120s
|
||||
|
||||
log_success "CNPG operator installed"
|
||||
}
|
||||
|
||||
# Create namespace
|
||||
create_namespace() {
|
||||
log_info "Creating namespace: ${NAMESPACE}"
|
||||
kubectl create namespace "${NAMESPACE}" --dry-run=client -o yaml | kubectl apply -f -
|
||||
log_success "Namespace ready"
|
||||
}
|
||||
|
||||
# Generate secure passwords
|
||||
generate_password() {
|
||||
openssl rand -base64 32 | tr -d '\n'
|
||||
}
|
||||
|
||||
# Create secrets
|
||||
create_secrets() {
|
||||
log_info "Creating database secrets..."
|
||||
|
||||
# Main database credentials
|
||||
DB_PASSWORD=$(generate_password)
|
||||
DB_DSN="postgresql://umbratrans:${DB_PASSWORD}@${CLUSTER_NAME}-rw:5432/umbratrans?sslmode=require"
|
||||
|
||||
kubectl create secret generic umbratrans-db-credentials \
|
||||
-n "${NAMESPACE}" \
|
||||
--from-literal=username=umbratrans \
|
||||
--from-literal=password="${DB_PASSWORD}" \
|
||||
--from-literal=dsn="${DB_DSN}" \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
# PgBouncer admin credentials
|
||||
PGBOUNCER_PASSWORD=$(generate_password)
|
||||
kubectl create secret generic umbratrans-pgbouncer-admin \
|
||||
-n "${NAMESPACE}" \
|
||||
--from-literal=username=pgbouncer_admin \
|
||||
--from-literal=password="${PGBOUNCER_PASSWORD}" \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
# Application secrets (token pepper)
|
||||
TOKEN_PEPPER=$(generate_password)
|
||||
kubectl create secret generic umbratrans-secrets \
|
||||
-n "${NAMESPACE}" \
|
||||
--from-literal=token_pepper="${TOKEN_PEPPER}" \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
# SMTP credentials (placeholder - update with real values)
|
||||
kubectl create secret generic umbratrans-smtp \
|
||||
-n "${NAMESPACE}" \
|
||||
--from-literal=host=smtp.example.com \
|
||||
--from-literal=port=587 \
|
||||
--from-literal=username=umbra@example.com \
|
||||
--from-literal=password=CHANGE_ME \
|
||||
--from-literal=sender=umbratrans@umbra.example.com \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
# Backup credentials (placeholder - update with real S3 credentials)
|
||||
kubectl create secret generic umbratrans-backup-credentials \
|
||||
-n "${NAMESPACE}" \
|
||||
--from-literal=ACCESS_KEY_ID=CHANGE_ME \
|
||||
--from-literal=SECRET_ACCESS_KEY=CHANGE_ME \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
log_success "Secrets created"
|
||||
log_warning "Remember to update placeholder secrets (SMTP, S3) with real values!"
|
||||
}
|
||||
|
||||
# Deploy database cluster
|
||||
deploy_cluster() {
|
||||
log_info "Deploying PostgreSQL cluster: ${CLUSTER_NAME}"
|
||||
|
||||
kubectl apply -f k8s/base/01-postgresql.yaml
|
||||
|
||||
log_info "Waiting for cluster to be ready (this may take several minutes)..."
|
||||
kubectl wait --for=condition=Ready cluster/${CLUSTER_NAME} \
|
||||
-n "${NAMESPACE}" --timeout=600s
|
||||
|
||||
log_success "PostgreSQL cluster is ready"
|
||||
}
|
||||
|
||||
# Verify database connectivity
|
||||
verify_database() {
|
||||
log_info "Verifying database connectivity..."
|
||||
|
||||
# Get the primary pod
|
||||
PRIMARY_POD=$(kubectl get pods -n "${NAMESPACE}" \
|
||||
-l "postgresql=${CLUSTER_NAME},role=primary" \
|
||||
-o jsonpath='{.items[0].metadata.name}')
|
||||
|
||||
if [[ -z "${PRIMARY_POD}" ]]; then
|
||||
log_error "Could not find primary pod"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log_info "Primary pod: ${PRIMARY_POD}"
|
||||
|
||||
# Test connection
|
||||
DB_PASSWORD=$(kubectl get secret umbratrans-db-credentials -n "${NAMESPACE}" -o jsonpath='{.data.password}' | base64 -d)
|
||||
|
||||
kubectl exec -n "${NAMESPACE}" "${PRIMARY_POD}" -- \
|
||||
psql -U umbratrans -d umbratrans -c "SELECT version();" \
|
||||
PGPASSWORD="${DB_PASSWORD}"
|
||||
|
||||
log_success "Database connection verified"
|
||||
}
|
||||
|
||||
# Run database migrations
|
||||
run_migrations() {
|
||||
log_info "Running database migrations..."
|
||||
|
||||
# Create a job to run migrations
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: umbratrans-migrate-$(date +%s)
|
||||
namespace: ${NAMESPACE}
|
||||
spec:
|
||||
ttlSecondsAfterFinished: 3600
|
||||
backoffLimit: 3
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: OnFailure
|
||||
serviceAccountName: umbratrans-worker
|
||||
containers:
|
||||
- name: migrate
|
||||
image: umbratrans-web:latest
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["python", "-m", "umbratrans_web.cli", "migrate", "--allow-production"]
|
||||
env:
|
||||
- name: UMBRA_ENV
|
||||
value: "production"
|
||||
- name: UMBRATRANS_DATABASE_DSN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: umbratrans-db-credentials
|
||||
key: dsn
|
||||
- name: UMBRATRANS_REPOSITORY
|
||||
value: "postgres"
|
||||
EOF
|
||||
|
||||
log_info "Waiting for migration job to complete..."
|
||||
kubectl wait --for=condition=complete job -l job-name=umbratrans-migrate \
|
||||
-n "${NAMESPACE}" --timeout=300s
|
||||
|
||||
# Check job status
|
||||
JOB_STATUS=$(kubectl get jobs -n "${NAMESPACE}" -l job-name=umbratrans-migrate -o jsonpath='{.items[0].status.conditions[?(@.type=="Complete")].status}')
|
||||
|
||||
if [[ "${JOB_STATUS}" != "True" ]]; then
|
||||
log_error "Migration job failed"
|
||||
kubectl logs -n "${NAMESPACE}" -l job-name=umbratrans-migrate
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log_success "Migrations completed successfully"
|
||||
}
|
||||
|
||||
# Verify schema
|
||||
verify_schema() {
|
||||
log_info "Verifying database schema..."
|
||||
|
||||
PRIMARY_POD=$(kubectl get pods -n "${NAMESPACE}" \
|
||||
-l "postgresql=${CLUSTER_NAME},role=primary" \
|
||||
-o jsonpath='{.items[0].metadata.name}')
|
||||
|
||||
DB_PASSWORD=$(kubectl get secret umbratrans-db-credentials -n "${NAMESPACE}" -o jsonpath='{.data.password}' | base64 -d)
|
||||
|
||||
# Check key tables exist
|
||||
kubectl exec -n "${NAMESPACE}" "${PRIMARY_POD}" -- \
|
||||
psql -U umbratrans -d umbratrans -c "
|
||||
SELECT table_name FROM information_schema.tables
|
||||
WHERE table_schema = 'public'
|
||||
AND table_name IN ('users', 'organizations', 'projects', 'jobs', 'artifacts', 'compute_nodes');
|
||||
" PGPASSWORD="${DB_PASSWORD}"
|
||||
|
||||
log_success "Schema verification passed"
|
||||
}
|
||||
|
||||
# Create initial admin user (if needed)
|
||||
create_admin_user() {
|
||||
log_info "Creating initial admin user (if needed)..."
|
||||
|
||||
PRIMARY_POD=$(kubectl get pods -n "${NAMESPACE}" \
|
||||
-l "postgresql=${CLUSTER_NAME},role=primary" \
|
||||
-o jsonpath='{.items[0].metadata.name}')
|
||||
|
||||
DB_PASSWORD=$(kubectl get secret umbratrans-db-credentials -n "${NAMESPACE}" -o jsonpath='{.data.password}' | base64 -d)
|
||||
|
||||
# Check if admin exists
|
||||
ADMIN_EXISTS=$(kubectl exec -n "${NAMESPACE}" "${PRIMARY_POD}" -- \
|
||||
psql -U umbratrans -d umbratrans -t -c "
|
||||
SELECT COUNT(*) FROM users WHERE role = 'platform_admin';
|
||||
" PGPASSWORD="${DB_PASSWORD}" | tr -d ' ')
|
||||
|
||||
if [[ "${ADMIN_EXISTS}" -eq 0 ]]; then
|
||||
log_info "No admin user found. Creating bootstrap admin..."
|
||||
ADMIN_PASSWORD=$(generate_password)
|
||||
|
||||
# Run bootstrap admin creation
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: umbratrans-bootstrap-admin-$(date +%s)
|
||||
namespace: ${NAMESPACE}
|
||||
spec:
|
||||
ttlSecondsAfterFinished: 3600
|
||||
backoffLimit: 1
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: OnFailure
|
||||
containers:
|
||||
- name: bootstrap
|
||||
image: umbratrans-web:latest
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["python", "-m", "umbratrans_web.cli", "bootstrap-admin"]
|
||||
env:
|
||||
- name: UMBRA_ENV
|
||||
value: "production"
|
||||
- name: UMBRATRANS_DATABASE_DSN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: umbratrans-db-credentials
|
||||
key: dsn
|
||||
- name: UMBRATRANS_REPOSITORY
|
||||
value: "postgres"
|
||||
stdin: true
|
||||
stdinOnce: true
|
||||
EOF
|
||||
|
||||
log_warning "Bootstrap admin job created. Check logs for the one-time password."
|
||||
log_warning "Run: kubectl logs -n ${NAMESPACE} -l job-name=umbratrans-bootstrap-admin -f"
|
||||
else
|
||||
log_info "Admin user already exists"
|
||||
fi
|
||||
}
|
||||
|
||||
# Display connection info
|
||||
show_connection_info() {
|
||||
log_info "Database deployment complete!"
|
||||
echo
|
||||
echo "Connection Information:"
|
||||
echo " Cluster: ${CLUSTER_NAME}"
|
||||
echo " Namespace: ${NAMESPACE}"
|
||||
echo " Primary Service: ${CLUSTER_NAME}-rw.${NAMESPACE}.svc.cluster.local:5432"
|
||||
echo " Replica Service: ${CLUSTER_NAME}-ro.${NAMESPACE}.svc.cluster.local:5432"
|
||||
echo " PgBouncer Service: ${CLUSTER_NAME}-pgbouncer.${NAMESPACE}.svc.cluster.local:6432"
|
||||
echo
|
||||
echo "Database: umbratrans"
|
||||
echo "User: umbratrans"
|
||||
echo "Password: (stored in secret umbratrans-db-credentials)"
|
||||
echo
|
||||
echo "Connection string (internal):"
|
||||
echo " postgresql://umbratrans:<password>@${CLUSTER_NAME}-rw.${NAMESPACE}.svc.cluster.local:5432/umbratrans?sslmode=require"
|
||||
echo
|
||||
echo "Connection string (via PgBouncer):"
|
||||
echo " postgresql://umbratrans:<password>@${CLUSTER_NAME}-pgbouncer.${NAMESPACE}.svc.cluster.local:6432/umbratrans?sslmode=require"
|
||||
echo
|
||||
echo "To connect from outside the cluster:"
|
||||
echo " kubectl port-forward -n ${NAMESPACE} svc/${CLUSTER_NAME}-rw 5432:5432"
|
||||
echo " Then connect to localhost:5432"
|
||||
echo
|
||||
echo "To check cluster status:"
|
||||
echo " kubectl get cluster ${CLUSTER_NAME} -n ${NAMESPACE}"
|
||||
echo
|
||||
echo "To view logs:"
|
||||
echo " kubectl logs -n ${NAMESPACE} -l postgresql=${CLUSTER_NAME},role=primary -f"
|
||||
}
|
||||
|
||||
# Main deployment flow
|
||||
main() {
|
||||
log_info "Starting UmbraTrans database deployment"
|
||||
echo
|
||||
|
||||
check_prerequisites
|
||||
install_cnpg_operator
|
||||
create_namespace
|
||||
create_secrets
|
||||
deploy_cluster
|
||||
verify_database
|
||||
run_migrations
|
||||
verify_schema
|
||||
create_admin_user
|
||||
show_connection_info
|
||||
|
||||
log_success "Database deployment completed successfully!"
|
||||
}
|
||||
|
||||
# Run main if script is executed directly
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
Reference in New Issue
Block a user