cce2ce55dc
- Database deployment script (scripts/deploy-database.sh) - Kubernetes manifests for CNPG PostgreSQL cluster (k8s/base/) - Documentation (docs/DEPLOYMENT_GUIDE.md, DATASTORE_SPEC.md, INTEGRATION_SPEC.md) - README with architecture overview and operations guide
739 lines
23 KiB
Markdown
739 lines
23 KiB
Markdown
# UmbraTrans Integration Specification for External Services
|
|
|
|
> **Version:** 1.0
|
|
> **Audience:** UmbraAudio, UmbraFormat teams
|
|
> **Status:** Active
|
|
> **Last Updated:** 2026-10-31
|
|
|
|
This document defines the contract for external services (UmbraAudio, UmbraFormat) to integrate with the UmbraTrans datastore. It covers PostgreSQL access, NFS shared storage, and API boundaries.
|
|
|
|
---
|
|
|
|
## 1. Architecture Overview
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ UMBRA ECOSYSTEM │
|
|
├─────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
|
|
│ │ UmbraTrans │ │ UmbraAudio │ │ UmbraFormat │ │
|
|
│ │ (this cluster) │◄───►│ (remote node) │◄───►│ (remote node) │ │
|
|
│ │ │ │ │ │ │ │
|
|
│ │ • Web API/UI │ │ • TTS/ASR │ │ • EPUB/PDF │ │
|
|
│ │ • Job Queue │ │ • Audio Master │ │ • Cover Gen │ │
|
|
│ │ • PostgreSQL │ │ • Speaker DB │ │ • Print Layout │ │
|
|
│ │ • NFS Server │ │ │ │ │ │
|
|
│ └────────┬─────────┘ └────────┬─────────┘ └────────┬─────────┘ │
|
|
│ │ │ │ │
|
|
│ └────────────────────────┼────────────────────────┘ │
|
|
│ ▼ │
|
|
│ ┌──────────────────┐ │
|
|
│ │ SHARED STORE │ │
|
|
│ │ │ │
|
|
│ │ PostgreSQL │ │
|
|
│ │ (192.168.123.14)│ │
|
|
│ │ │ │
|
|
│ │ NFS │ │
|
|
│ │ (192.168.123.14)│ │
|
|
│ └──────────────────┘ │
|
|
│ │
|
|
│ ┌──────────────────┐ │
|
|
│ │ WordPress │ (External - API only) │
|
|
│ │ (sales site) │ │
|
|
│ └──────────────────┘ │
|
|
│ │
|
|
│ ┌──────────────────┐ │
|
|
│ │ Git Server │ (192.168.123.18 - Mini-Eq) │
|
|
│ │ (source code) │ │
|
|
│ └──────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Network Connectivity
|
|
|
|
### 2.1 Required Access from Remote Nodes
|
|
|
|
| Service | Host | Port | Protocol | Purpose |
|
|
|---------|------|------|----------|---------|
|
|
| PostgreSQL (PgBouncer) | `192.168.123.14` | `6432` | TCP | Pooled DB connections |
|
|
| PostgreSQL (Direct) | `192.168.123.14` | `5432` | TCP | Admin/backup (optional) |
|
|
| NFS | `192.168.123.14` | `2049` | TCP/UDP | Shared storage mount |
|
|
| Web API | `umbra.bdmurphy.me` | `443` | HTTPS | REST API, webhooks |
|
|
|
|
### 2.2 Firewall Rules (on 192.168.123.14)
|
|
|
|
```bash
|
|
# PostgreSQL (PgBouncer) - allow from cluster nodes
|
|
iptables -A INPUT -s 192.168.123.0/24 -p tcp --dport 6432 -j ACCEPT
|
|
|
|
# PostgreSQL (Direct) - allow from cluster nodes
|
|
iptables -A INPUT -s 192.168.123.0/24 -p tcp --dport 5432 -j ACCEPT
|
|
|
|
# NFS - allow from cluster nodes
|
|
iptables -A INPUT -s 192.168.123.0/24 -p tcp --dport 2049 -j ACCEPT
|
|
iptables -A INPUT -s 192.168.123.0/24 -p udp --dport 2049 -j ACCEPT
|
|
# Also need: 111 (rpcbind), 20048 (mountd) - check rpcinfo -p
|
|
```
|
|
|
|
### 2.3 DNS / Hosts
|
|
|
|
Add to `/etc/hosts` on remote machines (or configure DNS):
|
|
|
|
```
|
|
192.168.123.14 umbra-db umbra-nfs umbra-api
|
|
```
|
|
|
|
---
|
|
|
|
## 3. PostgreSQL Access
|
|
|
|
### 3.1 Connection Parameters
|
|
|
|
```bash
|
|
# PgBouncer (RECOMMENDED - pooled, 6432)
|
|
host=192.168.123.14
|
|
port=6432
|
|
dbname=umbratrans
|
|
user=umbraudio # or umbraformat
|
|
password=<from secret>
|
|
sslmode=require
|
|
connect_timeout=10
|
|
application_name=umbraudio-worker # or umbraformat-worker
|
|
|
|
# Direct (for admin/backup only - 5432)
|
|
host=192.168.123.14
|
|
port=5432
|
|
dbname=umbratrans
|
|
user=umbraudio
|
|
password=<from secret>
|
|
sslmode=require
|
|
```
|
|
|
|
### 3.2 Database Roles
|
|
|
|
| Role | Permissions | Used By |
|
|
|------|-------------|---------|
|
|
| `umbraudio` | `SELECT, INSERT, UPDATE` on `jobs`, `artifacts`, `compute_nodes`, `node_heartbeats`, `model_registry`, `node_models`; `SELECT` on `projects`, `organizations` | UmbraAudio workers |
|
|
| `umbraformat` | `SELECT, INSERT, UPDATE` on `jobs`, `artifacts`, `format_metadata`; `SELECT` on `projects`, `organizations` | UmbraFormat workers |
|
|
| `umbra_web` | Full access to all tables (web API) | UmbraTrans web |
|
|
|
|
### 3.3 Required Extensions
|
|
|
|
```sql
|
|
-- Already installed by migrations
|
|
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
|
|
CREATE EXTENSION IF NOT EXISTS "citext";
|
|
CREATE EXTENSION IF NOT EXISTS "pgcrypto";
|
|
```
|
|
|
|
---
|
|
|
|
## 4. NFS Shared Storage
|
|
|
|
### 4.1 Mount Points (on Remote Nodes)
|
|
|
|
```bash
|
|
# /etc/fstab
|
|
192.168.123.14:/mnt/umbra-data/productions /mnt/umbra/productions nfs4 rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2 0 0
|
|
192.168.123.14:/mnt/umbra-data/artifacts /mnt/umbra/artifacts nfs4 rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2 0 0
|
|
192.168.123.14:/mnt/umbra-data/covers /mnt/umbra/covers nfs4 rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2 0 0
|
|
192.168.123.14:/mnt/umbra-data/audio /mnt/umbra/audio nfs4 rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2 0 0
|
|
```
|
|
|
|
### 4.2 Directory Permissions
|
|
|
|
```bash
|
|
# On NFS server (192.168.123.14)
|
|
mkdir -p /mnt/umbra-data/{productions,artifacts,covers,audio,backups}
|
|
chown -R 1000:1000 /mnt/umbra-data
|
|
chmod -R 700 /mnt/umbra-data
|
|
|
|
# On remote nodes - same UID/GID
|
|
groupadd -g 1000 umbra
|
|
useradd -u 1000 -g umbra -s /bin/bash umbra
|
|
```
|
|
|
|
### 4.3 Access Patterns by Service
|
|
|
|
| Path | UmbraAudio | UmbraFormat | UmbraTrans Web |
|
|
|------|------------|-------------|----------------|
|
|
| `/productions/<project>-<lang>/source/` | Read (source text) | Read (metadata) | Read/Write |
|
|
| `/productions/<project>-<lang>/runs/` | Read/Write (translation artifacts) | Read | Read |
|
|
| `/artifacts/<org>/<project>/<hash>/` | Read/Write (audio renders) | Read/Write (EPUB/PDF) | Read |
|
|
| `/covers/<asset_id>/` | Read | Read/Write (cover gen) | Read/Write |
|
|
| `/audio/<project>/<lang>/` | Read/Write (master audio) | Read | Read |
|
|
|
|
---
|
|
|
|
## 5. Job Queue Integration
|
|
|
|
### 5.1 How Workers Claim Jobs
|
|
|
|
```sql
|
|
-- Atomic job claim (run in transaction)
|
|
BEGIN;
|
|
SET LOCAL statement_timeout = 5000;
|
|
SET LOCAL app.user_id = '<node-user-id>';
|
|
SET LOCAL search_path = 'public';
|
|
|
|
UPDATE public.jobs
|
|
SET status = 'running',
|
|
lease_owner = '<compute_node_id>',
|
|
lease_expires_at = now() + interval '30 minutes',
|
|
started_at = now(),
|
|
attempt = attempt + 1
|
|
WHERE id = (
|
|
SELECT id FROM public.jobs
|
|
WHERE status = 'queued'
|
|
AND not_before <= now()
|
|
AND (lease_expires_at IS NULL OR lease_expires_at <= now())
|
|
AND kind IN ('<allowed-kinds>')
|
|
AND organization_id = '<org-id>'
|
|
ORDER BY priority, not_before, created_at
|
|
FOR UPDATE SKIP LOCKED
|
|
LIMIT 1
|
|
)
|
|
RETURNING *;
|
|
|
|
COMMIT;
|
|
```
|
|
|
|
### 5.2 Job Kinds by Service
|
|
|
|
| Service | Job Kinds (claims) | Job Kinds (enqueues) |
|
|
|---------|-------------------|---------------------|
|
|
| **UmbraAudio** | `tts`, `asr`, `assemble`, `encode`, `master`, `speaker_profile` | `tts` (after translation), `assemble` (after tts) |
|
|
| **UmbraFormat** | `build_epub`, `build_pdf`, `cover_generate`, `metadata_validate` | `build_epub` (after translation+audio), `cover_generate` |
|
|
|
|
### 5.3 Job Payload Structure
|
|
|
|
```json
|
|
// TTS Job
|
|
{
|
|
"utterance_ids": ["utt-001", "utt-002"],
|
|
"speaker_id": "spk-narrator",
|
|
"voice_config": {
|
|
"model": "xtts-v2",
|
|
"language": "es",
|
|
"speed": 1.0
|
|
},
|
|
"output_format": "wav"
|
|
}
|
|
|
|
// EPUB Build Job
|
|
{
|
|
"project_id": "uuid",
|
|
"language": "es",
|
|
"include_audio": true,
|
|
"cover_asset_id": "asset-uuid",
|
|
"format_options": {
|
|
"epub_version": "3.2",
|
|
"drm": false,
|
|
"embedded_fonts": ["Source Serif Pro"]
|
|
}
|
|
}
|
|
```
|
|
|
|
### 5.4 Job Result Structure
|
|
|
|
```json
|
|
// TTS Result
|
|
{
|
|
"artifacts": [
|
|
{
|
|
"artifact_id": "uuid",
|
|
"kind": "audio",
|
|
"storage_uri": "file://192.168.123.14/mnt/umbra-data/audio/proj123/es/chapters/ch-001/track-001.wav",
|
|
"content_hash": "sha256...",
|
|
"byte_size": 1048576,
|
|
"media_type": "audio/wav",
|
|
"duration_ms": 5432
|
|
}
|
|
],
|
|
"metrics": {
|
|
"total_duration_ms": 5432,
|
|
"rtf": 0.15
|
|
}
|
|
}
|
|
|
|
// EPUB Build Result
|
|
{
|
|
"artifacts": [
|
|
{
|
|
"artifact_id": "uuid",
|
|
"kind": "epub",
|
|
"storage_uri": "file://192.168.123.14/mnt/umbra-data/artifacts/org1/proj123/hash123.epub",
|
|
"content_hash": "sha256...",
|
|
"byte_size": 5242880,
|
|
"media_type": "application/epub+zip"
|
|
}
|
|
],
|
|
"validation": {
|
|
"epubcheck": "passed",
|
|
"page_count": 312
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Artifact Protocol (Content-Addressed Storage)
|
|
|
|
### 6.1 Writing Artifacts
|
|
|
|
```python
|
|
# 1. Compute content hash
|
|
import hashlib
|
|
content_hash = hashlib.sha256(audio_bytes).hexdigest()
|
|
|
|
# 2. Write to NFS (atomic)
|
|
artifact_path = f"/mnt/umbra/artifacts/{org_id}/{project_id}/{content_hash[:2]}/{content_hash}"
|
|
os.makedirs(os.path.dirname(artifact_path), exist_ok=True)
|
|
tmp_path = artifact_path + ".tmp"
|
|
with open(tmp_path, "wb") as f:
|
|
f.write(audio_bytes)
|
|
os.rename(tmp_path, artifact_path)
|
|
|
|
# 3. Insert artifact row (PostgreSQL)
|
|
INSERT INTO public.artifacts (
|
|
id, project_id, organization_id, lang, kind, status,
|
|
content_hash, byte_size, media_type, storage_key,
|
|
storage_uri, integrity, derivation, created_at
|
|
) VALUES (
|
|
gen_random_uuid(), $project_id, $org_id, $lang, 'audio', 'final',
|
|
$content_hash, $byte_size, 'audio/wav', $content_hash,
|
|
$storage_uri, 'unverified', $derivation_json, now()
|
|
)
|
|
ON CONFLICT (project_id, content_hash, kind, lang) DO NOTHING;
|
|
|
|
# 4. Link to job
|
|
INSERT INTO public.job_artifacts (job_id, artifact_id, role, created_at)
|
|
VALUES ($job_id, $artifact_id, 'produced', now());
|
|
```
|
|
|
|
### 6.2 Storage URI Format
|
|
|
|
```
|
|
file://<node-id>/<relative-path-from-nfs-root>
|
|
|
|
Examples:
|
|
file://umbra-audio-01/mnt/umbra-data/audio/proj123/es/chapters/ch-001/track-001.wav
|
|
file://umbra-format-01/mnt/umbra-data/artifacts/org1/proj123/ab12cd34.epub
|
|
```
|
|
|
|
### 6.3 Reading Artifacts
|
|
|
|
```python
|
|
# Via storage_uri from artifact row
|
|
storage_uri = artifact_row["storage_uri"] # "file://umbra-audio-01/mnt/umbra-data/..."
|
|
|
|
# Parse and read
|
|
if storage_uri.startswith("file://"):
|
|
# Extract path after hostname
|
|
path = storage_uri.split("/", 3)[-1] # "/mnt/umbra-data/..."
|
|
with open(path, "rb") as f:
|
|
data = f.read()
|
|
```
|
|
|
|
---
|
|
|
|
## 7. Compute Node Registration
|
|
|
|
### 7.1 Node Registration (at startup)
|
|
|
|
```sql
|
|
-- Register compute node
|
|
INSERT INTO public.compute_nodes (
|
|
id, organization_id, name, is_controller, trust_domain,
|
|
locality, transport, status, agent_version, agent_started_at, created_at
|
|
) VALUES (
|
|
$node_id, $org_id, $hostname, false, 'remote',
|
|
$datacenter, 'https', 'provisioning', $version, now(), now()
|
|
)
|
|
ON CONFLICT (organization_id, name) DO UPDATE SET
|
|
status = 'provisioning',
|
|
agent_version = EXCLUDED.agent_version,
|
|
agent_started_at = EXCLUDED.agent_started_at,
|
|
updated_at = now()
|
|
RETURNING id;
|
|
```
|
|
|
|
### 7.2 Device Registration
|
|
|
|
```sql
|
|
-- Register GPUs/CPUs
|
|
INSERT INTO public.compute_devices (
|
|
id, compute_node_id, kind, label, vram_total_mb, vram_free_mb,
|
|
ram_total_mb, ram_free_mb, compute_units, throughput_factor, supports
|
|
) VALUES (
|
|
gen_random_uuid(), $node_id, 'gpu_discrete', 'gpu:0',
|
|
24576, 24576, 0, 0, 10496, 1.0, '{"cuda": "12.4"}'
|
|
)
|
|
ON CONFLICT (compute_node_id, label) DO UPDATE SET
|
|
vram_total_mb = EXCLUDED.vram_total_mb,
|
|
vram_free_mb = EXCLUDED.vram_free_mb,
|
|
updated_at = now();
|
|
```
|
|
|
|
### 7.3 Heartbeat (every 30 seconds)
|
|
|
|
```sql
|
|
-- Update heartbeat (overwrites previous)
|
|
INSERT INTO public.node_heartbeats (
|
|
compute_node_id, reported_at, cpu_busy_pct, ram_free_mb,
|
|
vram_free_mb, vram_total_mb, disk_free_mb, load_avg_1m,
|
|
resident_models, active_jobs, agent_version, raw
|
|
) VALUES (
|
|
$node_id, now(), $cpu_pct, $ram_free, $vram_free, $vram_total,
|
|
$disk_free, $load_avg, $resident_models_json, $active_jobs, $version, $raw_json
|
|
)
|
|
ON CONFLICT (compute_node_id) DO UPDATE SET
|
|
reported_at = EXCLUDED.reported_at,
|
|
cpu_busy_pct = EXCLUDED.cpu_busy_pct,
|
|
ram_free_mb = EXCLUDED.ram_free_mb,
|
|
vram_free_mb = EXCLUDED.vram_free_mb,
|
|
vram_total_mb = EXCLUDED.vram_total_mb,
|
|
disk_free_mb = EXCLUDED.disk_free_mb,
|
|
load_avg_1m = EXCLUDED.load_avg_1m,
|
|
resident_models = EXCLUDED.resident_models,
|
|
active_jobs = EXCLUDED.active_jobs,
|
|
agent_version = EXCLUDED.agent_version,
|
|
raw = EXCLUDED.raw;
|
|
```
|
|
|
|
### 7.4 Status Transitions
|
|
|
|
```
|
|
provisioning → online → draining → offline → retired
|
|
↘️ (heartbeat stops, >2min) ↗️
|
|
```
|
|
|
|
- **online**: Heartbeat received, accepting jobs
|
|
- **draining**: Finishing current jobs, not accepting new
|
|
- **offline**: No heartbeat > 2 minutes
|
|
- **retired**: Decommissioned permanently
|
|
|
|
---
|
|
|
|
## 8. Model Registry
|
|
|
|
### 8.1 Model Registration (by controller)
|
|
|
|
```sql
|
|
INSERT INTO public.model_registry (
|
|
id, slug, kind, vram_min_mb, vram_preferred_mb, ram_min_mb,
|
|
quantisation, context_tokens, requires_accelerators,
|
|
reference_throughput, load_seconds, capabilities, source_uri, license
|
|
) VALUES (
|
|
gen_random_uuid(), 'xtts-v2', 'tts', 6144, 8192, 2048,
|
|
'fp16', 4096, ARRAY['cuda'], 1.0, 30,
|
|
'{"languages": ["en","es","fr","de","it","pt","pl","tr","ru","nl","cs","ar","zh","ja","hu","ko"]}',
|
|
'https://github.com/coqui-ai/TTS', 'CPML'
|
|
)
|
|
ON CONFLICT (slug) DO NOTHING;
|
|
```
|
|
|
|
### 8.2 Model Residency (worker reports)
|
|
|
|
```sql
|
|
-- When model loaded
|
|
INSERT INTO public.node_models (
|
|
compute_node_id, model_id, device_id, state, loaded_at, vram_held_mb
|
|
) VALUES ($node_id, $model_id, $device_id, 'loaded', now(), 7168)
|
|
ON CONFLICT (compute_node_id, model_id) DO UPDATE SET
|
|
state = EXCLUDED.state,
|
|
loaded_at = EXCLUDED.loaded_at,
|
|
vram_held_mb = EXCLUDED.vram_held_mb,
|
|
updated_at = now();
|
|
|
|
-- When model unloaded
|
|
UPDATE public.node_models SET state = 'available', vram_held_mb = 0
|
|
WHERE compute_node_id = $node_id AND model_id = $model_id;
|
|
```
|
|
|
|
---
|
|
|
|
## 9. Audio Import (UmbraAudio → UmbraTrans)
|
|
|
|
### 9.1 Bundle Export Format
|
|
|
|
UmbraAudio exports a bundle directory:
|
|
|
|
```
|
|
/mnt/umbra/productions/<project>-<lang>/bundle/
|
|
├── bundle.json # Envelope with metadata
|
|
└── manifest.jsonl # Line-delimited JSON records
|
|
```
|
|
|
|
**bundle.json:**
|
|
```json
|
|
{
|
|
"bundle_format_version": 1,
|
|
"content_hash": "sha256-of-manifest",
|
|
"source": {
|
|
"system": "umbraaudio",
|
|
"node_id": "umbra-audio-01",
|
|
"product_version": "1.2.0"
|
|
},
|
|
"scope": {
|
|
"books_full_detail": ["book-dir-name"]
|
|
}
|
|
}
|
|
```
|
|
|
|
**manifest.jsonl records:** `book`, `chapter`, `speaker`, `utterance`, `render`
|
|
|
|
### 9.2 Import Trigger
|
|
|
|
UmbraTrans imports via CLI (runs as CronJob or manual):
|
|
|
|
```bash
|
|
python -m umbratrans_web.cli import-productions \
|
|
--organization umbra \
|
|
--allow-production \
|
|
--confirm
|
|
```
|
|
|
|
This reads bundles from `/mnt/umbra/productions/*/bundle/` and populates:
|
|
- `project_source_refs`
|
|
- `translation_chapters`
|
|
- `translation_speakers`
|
|
- `translation_utterances`
|
|
- `artifacts` (audio renders)
|
|
- `translation_imports` (ledger)
|
|
|
|
---
|
|
|
|
## 10. Format Metadata API (UmbraFormat ↔ UmbraTrans)
|
|
|
|
### 10.1 Get Format Metadata
|
|
|
|
```bash
|
|
GET /api/projects/{project_id}/format-metadata?language_tag=es
|
|
Authorization: Bearer <token>
|
|
```
|
|
|
|
Response:
|
|
```json
|
|
{
|
|
"project_id": "uuid",
|
|
"language_tag": "es",
|
|
"title": "El Cazador de Ofertas",
|
|
"isbn13": "9788412345678",
|
|
"cover_front_asset_id": "asset-uuid",
|
|
"binding": "paperback",
|
|
"trim_w_in": 6.0,
|
|
"trim_h_in": 9.0,
|
|
"page_count_total": 312,
|
|
"font_primary": "Source Serif Pro",
|
|
"font_primary_size_pt": 11.0,
|
|
"line_height_pt": 14.5,
|
|
"margin_inside_in": 0.75,
|
|
"margin_outside_in": 0.625,
|
|
"margin_top_in": 0.875,
|
|
"margin_bottom_in": 0.875
|
|
}
|
|
```
|
|
|
|
### 10.2 Save Format Metadata
|
|
|
|
```bash
|
|
PUT /api/projects/{project_id}/format-metadata
|
|
Authorization: Bearer <token>
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"language_tag": "es",
|
|
"isbn13": "9788412345678",
|
|
"cover_front_asset_id": "asset-uuid",
|
|
"binding": "paperback",
|
|
"trim_w_in": 6.0,
|
|
"trim_h_in": 9.0,
|
|
"page_count_total": 312,
|
|
"font_primary": "Source Serif Pro",
|
|
"font_primary_size_pt": 11.0
|
|
}
|
|
```
|
|
|
|
### 10.3 Cover Upload
|
|
|
|
```bash
|
|
POST /api/projects/{project_id}/cover?language_tag=es
|
|
Authorization: Bearer <token>
|
|
Content-Type: multipart/form-data
|
|
|
|
file: <cover-image.png>
|
|
```
|
|
|
|
---
|
|
|
|
## 11. Webhooks / Event Notifications
|
|
|
|
### 11.1 Job Completion Webhook
|
|
|
|
UmbraTrans can POST to external services on job completion:
|
|
|
|
```json
|
|
POST https://umbra-audio.internal/webhooks/job-complete
|
|
Content-Type: application/json
|
|
X-Umbra-Signature: sha256=...
|
|
|
|
{
|
|
"event": "job.completed",
|
|
"job_id": "uuid",
|
|
"kind": "tts",
|
|
"project_id": "uuid",
|
|
"organization_id": "uuid",
|
|
"status": "complete",
|
|
"result": { ... },
|
|
"artifacts": [
|
|
{ "artifact_id": "uuid", "kind": "audio", "storage_uri": "..." }
|
|
],
|
|
"completed_at": "2026-10-31T12:00:00Z"
|
|
}
|
|
```
|
|
|
|
### 11.2 Configure Webhook URLs
|
|
|
|
```sql
|
|
-- In organization or project settings
|
|
UPDATE public.organizations
|
|
SET webhook_urls = '{"job_complete": "https://umbra-audio.internal/webhooks/job-complete"}'
|
|
WHERE id = $org_id;
|
|
```
|
|
|
|
---
|
|
|
|
## 12. Credentials & Secrets
|
|
|
|
### 12.1 Service Accounts (Kubernetes)
|
|
|
|
```yaml
|
|
# umbra-audio namespace on remote cluster
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: umbratrans-credentials
|
|
namespace: umbra-audio
|
|
type: Opaque
|
|
stringData:
|
|
DATABASE_DSN: "postgresql://umbraudio:PASS@192.168.123.14:6432/umbratrans?sslmode=require"
|
|
NFS_SERVER: "192.168.123.14"
|
|
NFS_PRODUCTIONS_PATH: "/mnt/umbra-data/productions"
|
|
NFS_ARTIFACTS_PATH: "/mnt/umbra-data/artifacts"
|
|
NODE_CREDENTIAL_PREFIX: "umbra-audio-"
|
|
```
|
|
|
|
### 12.2 Node Credentials (for job queue auth)
|
|
|
|
```sql
|
|
-- Created by UmbraTrans admin
|
|
INSERT INTO public.node_credentials (
|
|
id, compute_node_id, key_prefix, key_hash, agent_name, scopes, issued_at
|
|
) VALUES (
|
|
gen_random_uuid(), $node_id, 'umbra-audio-01-',
|
|
'$argon2id$v=19$m=65536,t=3,p=4$salt$hash',
|
|
'umbra-audio-worker', ARRAY['queue:claim', 'queue:complete', 'artifacts:write'],
|
|
now()
|
|
);
|
|
```
|
|
|
|
Worker uses this credential to authenticate job queue operations.
|
|
|
|
---
|
|
|
|
## 13. Health Checks & Monitoring
|
|
|
|
### 13.1 Required Health Endpoints
|
|
|
|
Each service exposes:
|
|
|
|
```
|
|
GET /health → { "status": "ok", "service": "umbra-audio", "version": "1.2.0" }
|
|
GET /metrics → Prometheus metrics
|
|
```
|
|
|
|
### 13.2 Key Metrics to Export
|
|
|
|
| Metric | Type | Labels |
|
|
|--------|------|--------|
|
|
| `umbra_jobs_claimed_total` | Counter | service, kind, status |
|
|
| `umbra_job_duration_seconds` | Histogram | service, kind |
|
|
| `umbra_artifact_bytes_written` | Counter | service, kind |
|
|
| `umbra_node_heartbeat_age_seconds` | Gauge | node_id |
|
|
| `umbra_gpu_utilization` | Gauge | node_id, device_id |
|
|
| `umbra_queue_depth` | Gauge | kind, status |
|
|
|
|
---
|
|
|
|
## 14. Deployment Checklist for Remote Services
|
|
|
|
### UmbraAudio
|
|
|
|
- [ ] PostgreSQL role `umbraudio` created with correct grants
|
|
- [ ] Node credentials inserted in `node_credentials`
|
|
- [ ] NFS mounts configured in `/etc/fstab`
|
|
- [ ] Compute node registered in `compute_nodes`
|
|
- [ ] Devices registered in `compute_devices`
|
|
- [ ] Models registered in `model_registry`
|
|
- [ ] Heartbeat cron running (every 30s)
|
|
- [ ] Job poller running for `tts`, `asr`, `assemble`, `encode`, `master`
|
|
- [ ] Artifact writer writing to `/mnt/umbra/artifacts/` + `artifacts` table
|
|
- [ ] Webhook endpoint for job completion (optional)
|
|
|
|
### UmbraFormat
|
|
|
|
- [ ] PostgreSQL role `umbraformat` created with correct grants
|
|
- [ ] NFS mounts configured in `/etc/fstab`
|
|
- [ ] Job poller running for `build_epub`, `build_pdf`, `cover_generate`, `metadata_validate`
|
|
- [ ] Format metadata API integrated (GET/PUT `/format-metadata`)
|
|
- [ ] Cover upload integrated (POST `/cover`)
|
|
- [ ] Artifact writer for EPUB/PDF to `/mnt/umbra/artifacts/`
|
|
- [ ] Webhook endpoint for job completion (optional)
|
|
|
|
---
|
|
|
|
## 15. Troubleshooting
|
|
|
|
| Issue | Diagnosis | Resolution |
|
|
|-------|-----------|------------|
|
|
| Job claim returns 0 rows | No queued jobs, or wrong `organization_id` | Check `SELECT * FROM jobs WHERE status='queued' AND organization_id=...` |
|
|
| Lease expires mid-job | Job takes >30min | Increase `lease_expires_at` on claim, or heartbeat extends it |
|
|
| NFS mount hangs | Network partition, server down | Use `hard,intr` options; check `rpcinfo -p 192.168.123.14` |
|
|
| SSL connection failed | Cert mismatch | Use `sslmode=require` with valid CA, or `sslmode=verify-ca` |
|
|
| Artifact conflict | Duplicate content hash | `ON CONFLICT DO NOTHING` - safe to ignore |
|
|
| Heartbeat not updating | Clock skew | Ensure NTP sync on all nodes |
|
|
|
|
---
|
|
|
|
## 16. Contact & Escalation
|
|
|
|
| Role | Contact | Channel |
|
|
|------|---------|---------|
|
|
| UmbraTrans Platform | platform@umbra.example.com | Slack #umbra-platform |
|
|
| Database Admin | dba@umbra.example.com | PagerDuty |
|
|
| Network/Infra | infra@umbra.example.com | Slack #umbra-infra |
|
|
|
|
---
|
|
|
|
## Appendix: Quick Reference Commands
|
|
|
|
```bash
|
|
# Test DB connectivity
|
|
psql "postgresql://umbraudio:PASS@192.168.123.14:6432/umbratrans?sslmode=require" -c "SELECT now();"
|
|
|
|
# Test NFS mount
|
|
mount -t nfs4 192.168.123.14:/mnt/umbra-data/productions /mnt/test && ls /mnt/test
|
|
|
|
# Check job queue
|
|
psql $DSN -c "SELECT kind, status, count(*) FROM jobs GROUP BY kind, status;"
|
|
|
|
# Check node status
|
|
psql $DSN -c "SELECT id, name, status, last_heartbeat_at FROM compute_nodes;"
|
|
|
|
# View recent artifacts
|
|
psql $DSN -c "SELECT kind, count(*), sum(byte_size) FROM artifacts GROUP BY kind;"
|
|
``` |