# UmbraTrans Integration Specification for External Services > **Version:** 1.0 > **Audience:** UmbraAudio, UmbraFormat teams > **Status:** Active > **Last Updated:** 2026-10-31 This document defines the contract for external services (UmbraAudio, UmbraFormat) to integrate with the UmbraTrans datastore. It covers PostgreSQL access, NFS shared storage, and API boundaries. --- ## 1. Architecture Overview ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ UMBRA ECOSYSTEM │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ │ │ UmbraTrans │ │ UmbraAudio │ │ UmbraFormat │ │ │ │ (this cluster) │◄───►│ (remote node) │◄───►│ (remote node) │ │ │ │ │ │ │ │ │ │ │ │ • Web API/UI │ │ • TTS/ASR │ │ • EPUB/PDF │ │ │ │ • Job Queue │ │ • Audio Master │ │ • Cover Gen │ │ │ │ • PostgreSQL │ │ • Speaker DB │ │ • Print Layout │ │ │ │ • NFS Server │ │ │ │ │ │ │ └────────┬─────────┘ └────────┬─────────┘ └────────┬─────────┘ │ │ │ │ │ │ │ └────────────────────────┼────────────────────────┘ │ │ ▼ │ │ ┌──────────────────┐ │ │ │ SHARED STORE │ │ │ │ │ │ │ │ PostgreSQL │ │ │ │ (192.168.123.14)│ │ │ │ │ │ │ │ NFS │ │ │ │ (192.168.123.14)│ │ │ └──────────────────┘ │ │ │ │ ┌──────────────────┐ │ │ │ WordPress │ (External - API only) │ │ │ (sales site) │ │ │ └──────────────────┘ │ │ │ │ ┌──────────────────┐ │ │ │ Git Server │ (192.168.123.18 - Mini-Eq) │ │ │ (source code) │ │ │ └──────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────┘ ``` --- ## 2. Network Connectivity ### 2.1 Required Access from Remote Nodes | Service | Host | Port | Protocol | Purpose | |---------|------|------|----------|---------| | PostgreSQL (PgBouncer) | `192.168.123.14` | `6432` | TCP | Pooled DB connections | | PostgreSQL (Direct) | `192.168.123.14` | `5432` | TCP | Admin/backup (optional) | | NFS | `192.168.123.14` | `2049` | TCP/UDP | Shared storage mount | | Web API | `umbra.bdmurphy.me` | `443` | HTTPS | REST API, webhooks | ### 2.2 Firewall Rules (on 192.168.123.14) ```bash # PostgreSQL (PgBouncer) - allow from cluster nodes iptables -A INPUT -s 192.168.123.0/24 -p tcp --dport 6432 -j ACCEPT # PostgreSQL (Direct) - allow from cluster nodes iptables -A INPUT -s 192.168.123.0/24 -p tcp --dport 5432 -j ACCEPT # NFS - allow from cluster nodes iptables -A INPUT -s 192.168.123.0/24 -p tcp --dport 2049 -j ACCEPT iptables -A INPUT -s 192.168.123.0/24 -p udp --dport 2049 -j ACCEPT # Also need: 111 (rpcbind), 20048 (mountd) - check rpcinfo -p ``` ### 2.3 DNS / Hosts Add to `/etc/hosts` on remote machines (or configure DNS): ``` 192.168.123.14 umbra-db umbra-nfs umbra-api ``` --- ## 3. PostgreSQL Access ### 3.1 Connection Parameters ```bash # PgBouncer (RECOMMENDED - pooled, 6432) host=192.168.123.14 port=6432 dbname=umbratrans user=umbraudio # or umbraformat password= sslmode=require connect_timeout=10 application_name=umbraudio-worker # or umbraformat-worker # Direct (for admin/backup only - 5432) host=192.168.123.14 port=5432 dbname=umbratrans user=umbraudio password= sslmode=require ``` ### 3.2 Database Roles | Role | Permissions | Used By | |------|-------------|---------| | `umbraudio` | `SELECT, INSERT, UPDATE` on `jobs`, `artifacts`, `compute_nodes`, `node_heartbeats`, `model_registry`, `node_models`; `SELECT` on `projects`, `organizations` | UmbraAudio workers | | `umbraformat` | `SELECT, INSERT, UPDATE` on `jobs`, `artifacts`, `format_metadata`; `SELECT` on `projects`, `organizations` | UmbraFormat workers | | `umbra_web` | Full access to all tables (web API) | UmbraTrans web | ### 3.3 Required Extensions ```sql -- Already installed by migrations CREATE EXTENSION IF NOT EXISTS "uuid-ossp"; CREATE EXTENSION IF NOT EXISTS "citext"; CREATE EXTENSION IF NOT EXISTS "pgcrypto"; ``` --- ## 4. NFS Shared Storage ### 4.1 Mount Points (on Remote Nodes) ```bash # /etc/fstab 192.168.123.14:/mnt/umbra-data/productions /mnt/umbra/productions nfs4 rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2 0 0 192.168.123.14:/mnt/umbra-data/artifacts /mnt/umbra/artifacts nfs4 rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2 0 0 192.168.123.14:/mnt/umbra-data/covers /mnt/umbra/covers nfs4 rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2 0 0 192.168.123.14:/mnt/umbra-data/audio /mnt/umbra/audio nfs4 rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2 0 0 ``` ### 4.2 Directory Permissions ```bash # On NFS server (192.168.123.14) mkdir -p /mnt/umbra-data/{productions,artifacts,covers,audio,backups} chown -R 1000:1000 /mnt/umbra-data chmod -R 700 /mnt/umbra-data # On remote nodes - same UID/GID groupadd -g 1000 umbra useradd -u 1000 -g umbra -s /bin/bash umbra ``` ### 4.3 Access Patterns by Service | Path | UmbraAudio | UmbraFormat | UmbraTrans Web | |------|------------|-------------|----------------| | `/productions/-/source/` | Read (source text) | Read (metadata) | Read/Write | | `/productions/-/runs/` | Read/Write (translation artifacts) | Read | Read | | `/artifacts////` | Read/Write (audio renders) | Read/Write (EPUB/PDF) | Read | | `/covers//` | Read | Read/Write (cover gen) | Read/Write | | `/audio///` | Read/Write (master audio) | Read | Read | --- ## 5. Job Queue Integration ### 5.1 How Workers Claim Jobs ```sql -- Atomic job claim (run in transaction) BEGIN; SET LOCAL statement_timeout = 5000; SET LOCAL app.user_id = ''; SET LOCAL search_path = 'public'; UPDATE public.jobs SET status = 'running', lease_owner = '', lease_expires_at = now() + interval '30 minutes', started_at = now(), attempt = attempt + 1 WHERE id = ( SELECT id FROM public.jobs WHERE status = 'queued' AND not_before <= now() AND (lease_expires_at IS NULL OR lease_expires_at <= now()) AND kind IN ('') AND organization_id = '' ORDER BY priority, not_before, created_at FOR UPDATE SKIP LOCKED LIMIT 1 ) RETURNING *; COMMIT; ``` ### 5.2 Job Kinds by Service | Service | Job Kinds (claims) | Job Kinds (enqueues) | |---------|-------------------|---------------------| | **UmbraAudio** | `tts`, `asr`, `assemble`, `encode`, `master`, `speaker_profile` | `tts` (after translation), `assemble` (after tts) | | **UmbraFormat** | `build_epub`, `build_pdf`, `cover_generate`, `metadata_validate` | `build_epub` (after translation+audio), `cover_generate` | ### 5.3 Job Payload Structure ```json // TTS Job { "utterance_ids": ["utt-001", "utt-002"], "speaker_id": "spk-narrator", "voice_config": { "model": "xtts-v2", "language": "es", "speed": 1.0 }, "output_format": "wav" } // EPUB Build Job { "project_id": "uuid", "language": "es", "include_audio": true, "cover_asset_id": "asset-uuid", "format_options": { "epub_version": "3.2", "drm": false, "embedded_fonts": ["Source Serif Pro"] } } ``` ### 5.4 Job Result Structure ```json // TTS Result { "artifacts": [ { "artifact_id": "uuid", "kind": "audio", "storage_uri": "file://192.168.123.14/mnt/umbra-data/audio/proj123/es/chapters/ch-001/track-001.wav", "content_hash": "sha256...", "byte_size": 1048576, "media_type": "audio/wav", "duration_ms": 5432 } ], "metrics": { "total_duration_ms": 5432, "rtf": 0.15 } } // EPUB Build Result { "artifacts": [ { "artifact_id": "uuid", "kind": "epub", "storage_uri": "file://192.168.123.14/mnt/umbra-data/artifacts/org1/proj123/hash123.epub", "content_hash": "sha256...", "byte_size": 5242880, "media_type": "application/epub+zip" } ], "validation": { "epubcheck": "passed", "page_count": 312 } } ``` --- ## 6. Artifact Protocol (Content-Addressed Storage) ### 6.1 Writing Artifacts ```python # 1. Compute content hash import hashlib content_hash = hashlib.sha256(audio_bytes).hexdigest() # 2. Write to NFS (atomic) artifact_path = f"/mnt/umbra/artifacts/{org_id}/{project_id}/{content_hash[:2]}/{content_hash}" os.makedirs(os.path.dirname(artifact_path), exist_ok=True) tmp_path = artifact_path + ".tmp" with open(tmp_path, "wb") as f: f.write(audio_bytes) os.rename(tmp_path, artifact_path) # 3. Insert artifact row (PostgreSQL) INSERT INTO public.artifacts ( id, project_id, organization_id, lang, kind, status, content_hash, byte_size, media_type, storage_key, storage_uri, integrity, derivation, created_at ) VALUES ( gen_random_uuid(), $project_id, $org_id, $lang, 'audio', 'final', $content_hash, $byte_size, 'audio/wav', $content_hash, $storage_uri, 'unverified', $derivation_json, now() ) ON CONFLICT (project_id, content_hash, kind, lang) DO NOTHING; # 4. Link to job INSERT INTO public.job_artifacts (job_id, artifact_id, role, created_at) VALUES ($job_id, $artifact_id, 'produced', now()); ``` ### 6.2 Storage URI Format ``` file:/// Examples: file://umbra-audio-01/mnt/umbra-data/audio/proj123/es/chapters/ch-001/track-001.wav file://umbra-format-01/mnt/umbra-data/artifacts/org1/proj123/ab12cd34.epub ``` ### 6.3 Reading Artifacts ```python # Via storage_uri from artifact row storage_uri = artifact_row["storage_uri"] # "file://umbra-audio-01/mnt/umbra-data/..." # Parse and read if storage_uri.startswith("file://"): # Extract path after hostname path = storage_uri.split("/", 3)[-1] # "/mnt/umbra-data/..." with open(path, "rb") as f: data = f.read() ``` --- ## 7. Compute Node Registration ### 7.1 Node Registration (at startup) ```sql -- Register compute node INSERT INTO public.compute_nodes ( id, organization_id, name, is_controller, trust_domain, locality, transport, status, agent_version, agent_started_at, created_at ) VALUES ( $node_id, $org_id, $hostname, false, 'remote', $datacenter, 'https', 'provisioning', $version, now(), now() ) ON CONFLICT (organization_id, name) DO UPDATE SET status = 'provisioning', agent_version = EXCLUDED.agent_version, agent_started_at = EXCLUDED.agent_started_at, updated_at = now() RETURNING id; ``` ### 7.2 Device Registration ```sql -- Register GPUs/CPUs INSERT INTO public.compute_devices ( id, compute_node_id, kind, label, vram_total_mb, vram_free_mb, ram_total_mb, ram_free_mb, compute_units, throughput_factor, supports ) VALUES ( gen_random_uuid(), $node_id, 'gpu_discrete', 'gpu:0', 24576, 24576, 0, 0, 10496, 1.0, '{"cuda": "12.4"}' ) ON CONFLICT (compute_node_id, label) DO UPDATE SET vram_total_mb = EXCLUDED.vram_total_mb, vram_free_mb = EXCLUDED.vram_free_mb, updated_at = now(); ``` ### 7.3 Heartbeat (every 30 seconds) ```sql -- Update heartbeat (overwrites previous) INSERT INTO public.node_heartbeats ( compute_node_id, reported_at, cpu_busy_pct, ram_free_mb, vram_free_mb, vram_total_mb, disk_free_mb, load_avg_1m, resident_models, active_jobs, agent_version, raw ) VALUES ( $node_id, now(), $cpu_pct, $ram_free, $vram_free, $vram_total, $disk_free, $load_avg, $resident_models_json, $active_jobs, $version, $raw_json ) ON CONFLICT (compute_node_id) DO UPDATE SET reported_at = EXCLUDED.reported_at, cpu_busy_pct = EXCLUDED.cpu_busy_pct, ram_free_mb = EXCLUDED.ram_free_mb, vram_free_mb = EXCLUDED.vram_free_mb, vram_total_mb = EXCLUDED.vram_total_mb, disk_free_mb = EXCLUDED.disk_free_mb, load_avg_1m = EXCLUDED.load_avg_1m, resident_models = EXCLUDED.resident_models, active_jobs = EXCLUDED.active_jobs, agent_version = EXCLUDED.agent_version, raw = EXCLUDED.raw; ``` ### 7.4 Status Transitions ``` provisioning → online → draining → offline → retired ↘️ (heartbeat stops, >2min) ↗️ ``` - **online**: Heartbeat received, accepting jobs - **draining**: Finishing current jobs, not accepting new - **offline**: No heartbeat > 2 minutes - **retired**: Decommissioned permanently --- ## 8. Model Registry ### 8.1 Model Registration (by controller) ```sql INSERT INTO public.model_registry ( id, slug, kind, vram_min_mb, vram_preferred_mb, ram_min_mb, quantisation, context_tokens, requires_accelerators, reference_throughput, load_seconds, capabilities, source_uri, license ) VALUES ( gen_random_uuid(), 'xtts-v2', 'tts', 6144, 8192, 2048, 'fp16', 4096, ARRAY['cuda'], 1.0, 30, '{"languages": ["en","es","fr","de","it","pt","pl","tr","ru","nl","cs","ar","zh","ja","hu","ko"]}', 'https://github.com/coqui-ai/TTS', 'CPML' ) ON CONFLICT (slug) DO NOTHING; ``` ### 8.2 Model Residency (worker reports) ```sql -- When model loaded INSERT INTO public.node_models ( compute_node_id, model_id, device_id, state, loaded_at, vram_held_mb ) VALUES ($node_id, $model_id, $device_id, 'loaded', now(), 7168) ON CONFLICT (compute_node_id, model_id) DO UPDATE SET state = EXCLUDED.state, loaded_at = EXCLUDED.loaded_at, vram_held_mb = EXCLUDED.vram_held_mb, updated_at = now(); -- When model unloaded UPDATE public.node_models SET state = 'available', vram_held_mb = 0 WHERE compute_node_id = $node_id AND model_id = $model_id; ``` --- ## 9. Audio Import (UmbraAudio → UmbraTrans) ### 9.1 Bundle Export Format UmbraAudio exports a bundle directory: ``` /mnt/umbra/productions/-/bundle/ ├── bundle.json # Envelope with metadata └── manifest.jsonl # Line-delimited JSON records ``` **bundle.json:** ```json { "bundle_format_version": 1, "content_hash": "sha256-of-manifest", "source": { "system": "umbraaudio", "node_id": "umbra-audio-01", "product_version": "1.2.0" }, "scope": { "books_full_detail": ["book-dir-name"] } } ``` **manifest.jsonl records:** `book`, `chapter`, `speaker`, `utterance`, `render` ### 9.2 Import Trigger UmbraTrans imports via CLI (runs as CronJob or manual): ```bash python -m umbratrans_web.cli import-productions \ --organization umbra \ --allow-production \ --confirm ``` This reads bundles from `/mnt/umbra/productions/*/bundle/` and populates: - `project_source_refs` - `translation_chapters` - `translation_speakers` - `translation_utterances` - `artifacts` (audio renders) - `translation_imports` (ledger) --- ## 10. Format Metadata API (UmbraFormat ↔ UmbraTrans) ### 10.1 Get Format Metadata ```bash GET /api/projects/{project_id}/format-metadata?language_tag=es Authorization: Bearer ``` Response: ```json { "project_id": "uuid", "language_tag": "es", "title": "El Cazador de Ofertas", "isbn13": "9788412345678", "cover_front_asset_id": "asset-uuid", "binding": "paperback", "trim_w_in": 6.0, "trim_h_in": 9.0, "page_count_total": 312, "font_primary": "Source Serif Pro", "font_primary_size_pt": 11.0, "line_height_pt": 14.5, "margin_inside_in": 0.75, "margin_outside_in": 0.625, "margin_top_in": 0.875, "margin_bottom_in": 0.875 } ``` ### 10.2 Save Format Metadata ```bash PUT /api/projects/{project_id}/format-metadata Authorization: Bearer Content-Type: application/json { "language_tag": "es", "isbn13": "9788412345678", "cover_front_asset_id": "asset-uuid", "binding": "paperback", "trim_w_in": 6.0, "trim_h_in": 9.0, "page_count_total": 312, "font_primary": "Source Serif Pro", "font_primary_size_pt": 11.0 } ``` ### 10.3 Cover Upload ```bash POST /api/projects/{project_id}/cover?language_tag=es Authorization: Bearer Content-Type: multipart/form-data file: ``` --- ## 11. Webhooks / Event Notifications ### 11.1 Job Completion Webhook UmbraTrans can POST to external services on job completion: ```json POST https://umbra-audio.internal/webhooks/job-complete Content-Type: application/json X-Umbra-Signature: sha256=... { "event": "job.completed", "job_id": "uuid", "kind": "tts", "project_id": "uuid", "organization_id": "uuid", "status": "complete", "result": { ... }, "artifacts": [ { "artifact_id": "uuid", "kind": "audio", "storage_uri": "..." } ], "completed_at": "2026-10-31T12:00:00Z" } ``` ### 11.2 Configure Webhook URLs ```sql -- In organization or project settings UPDATE public.organizations SET webhook_urls = '{"job_complete": "https://umbra-audio.internal/webhooks/job-complete"}' WHERE id = $org_id; ``` --- ## 12. Credentials & Secrets ### 12.1 Service Accounts (Kubernetes) ```yaml # umbra-audio namespace on remote cluster apiVersion: v1 kind: Secret metadata: name: umbratrans-credentials namespace: umbra-audio type: Opaque stringData: DATABASE_DSN: "postgresql://umbraudio:PASS@192.168.123.14:6432/umbratrans?sslmode=require" NFS_SERVER: "192.168.123.14" NFS_PRODUCTIONS_PATH: "/mnt/umbra-data/productions" NFS_ARTIFACTS_PATH: "/mnt/umbra-data/artifacts" NODE_CREDENTIAL_PREFIX: "umbra-audio-" ``` ### 12.2 Node Credentials (for job queue auth) ```sql -- Created by UmbraTrans admin INSERT INTO public.node_credentials ( id, compute_node_id, key_prefix, key_hash, agent_name, scopes, issued_at ) VALUES ( gen_random_uuid(), $node_id, 'umbra-audio-01-', '$argon2id$v=19$m=65536,t=3,p=4$salt$hash', 'umbra-audio-worker', ARRAY['queue:claim', 'queue:complete', 'artifacts:write'], now() ); ``` Worker uses this credential to authenticate job queue operations. --- ## 13. Health Checks & Monitoring ### 13.1 Required Health Endpoints Each service exposes: ``` GET /health → { "status": "ok", "service": "umbra-audio", "version": "1.2.0" } GET /metrics → Prometheus metrics ``` ### 13.2 Key Metrics to Export | Metric | Type | Labels | |--------|------|--------| | `umbra_jobs_claimed_total` | Counter | service, kind, status | | `umbra_job_duration_seconds` | Histogram | service, kind | | `umbra_artifact_bytes_written` | Counter | service, kind | | `umbra_node_heartbeat_age_seconds` | Gauge | node_id | | `umbra_gpu_utilization` | Gauge | node_id, device_id | | `umbra_queue_depth` | Gauge | kind, status | --- ## 14. Deployment Checklist for Remote Services ### UmbraAudio - [ ] PostgreSQL role `umbraudio` created with correct grants - [ ] Node credentials inserted in `node_credentials` - [ ] NFS mounts configured in `/etc/fstab` - [ ] Compute node registered in `compute_nodes` - [ ] Devices registered in `compute_devices` - [ ] Models registered in `model_registry` - [ ] Heartbeat cron running (every 30s) - [ ] Job poller running for `tts`, `asr`, `assemble`, `encode`, `master` - [ ] Artifact writer writing to `/mnt/umbra/artifacts/` + `artifacts` table - [ ] Webhook endpoint for job completion (optional) ### UmbraFormat - [ ] PostgreSQL role `umbraformat` created with correct grants - [ ] NFS mounts configured in `/etc/fstab` - [ ] Job poller running for `build_epub`, `build_pdf`, `cover_generate`, `metadata_validate` - [ ] Format metadata API integrated (GET/PUT `/format-metadata`) - [ ] Cover upload integrated (POST `/cover`) - [ ] Artifact writer for EPUB/PDF to `/mnt/umbra/artifacts/` - [ ] Webhook endpoint for job completion (optional) --- ## 15. Troubleshooting | Issue | Diagnosis | Resolution | |-------|-----------|------------| | Job claim returns 0 rows | No queued jobs, or wrong `organization_id` | Check `SELECT * FROM jobs WHERE status='queued' AND organization_id=...` | | Lease expires mid-job | Job takes >30min | Increase `lease_expires_at` on claim, or heartbeat extends it | | NFS mount hangs | Network partition, server down | Use `hard,intr` options; check `rpcinfo -p 192.168.123.14` | | SSL connection failed | Cert mismatch | Use `sslmode=require` with valid CA, or `sslmode=verify-ca` | | Artifact conflict | Duplicate content hash | `ON CONFLICT DO NOTHING` - safe to ignore | | Heartbeat not updating | Clock skew | Ensure NTP sync on all nodes | --- ## 16. Contact & Escalation | Role | Contact | Channel | |------|---------|---------| | UmbraTrans Platform | platform@umbra.example.com | Slack #umbra-platform | | Database Admin | dba@umbra.example.com | PagerDuty | | Network/Infra | infra@umbra.example.com | Slack #umbra-infra | --- ## Appendix: Quick Reference Commands ```bash # Test DB connectivity psql "postgresql://umbraudio:PASS@192.168.123.14:6432/umbratrans?sslmode=require" -c "SELECT now();" # Test NFS mount mount -t nfs4 192.168.123.14:/mnt/umbra-data/productions /mnt/test && ls /mnt/test # Check job queue psql $DSN -c "SELECT kind, status, count(*) FROM jobs GROUP BY kind, status;" # Check node status psql $DSN -c "SELECT id, name, status, last_heartbeat_at FROM compute_nodes;" # View recent artifacts psql $DSN -c "SELECT kind, count(*), sum(byte_size) FROM artifacts GROUP BY kind;" ```