UmbraDatastore/docs/INTEGRATION_SPEC.md

UmbraTrans Integration Specification for External Services

Version: 1.0
Audience: UmbraAudio, UmbraFormat teams
Status: Active
Last Updated: 2026-10-31

This document defines the contract for external services (UmbraAudio, UmbraFormat) to integrate with the UmbraTrans datastore. It covers PostgreSQL access, NFS shared storage, and API boundaries.

---

1. Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                         UMBRA ECOSYSTEM                                      │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌──────────────────┐     ┌──────────────────┐     ┌──────────────────┐   │
│  │   UmbraTrans     │     │   UmbraAudio     │     │  UmbraFormat     │   │
│  │   (this cluster) │◄───►│   (remote node)  │◄───►│  (remote node)   │   │
│  │                  │     │                  │     │                  │   │
│  │  • Web API/UI    │     │  • TTS/ASR       │     │  • EPUB/PDF      │   │
│  │  • Job Queue     │     │  • Audio Master  │     │  • Cover Gen     │   │
│  │  • PostgreSQL    │     │  • Speaker DB    │     │  • Print Layout  │   │
│  │  • NFS Server    │     │                  │     │                  │   │
│  └────────┬─────────┘     └────────┬─────────┘     └────────┬─────────┘   │
│           │                        │                        │              │
│           └────────────────────────┼────────────────────────┘              │
│                                    ▼                                       │
│                         ┌──────────────────┐                              │
│                         │   SHARED STORE   │                              │
│                         │                  │                              │
│                         │  PostgreSQL      │                              │
│                         │  (192.168.123.14)│                              │
│                         │                  │                              │
│                         │  NFS             │                              │
│                         │  (192.168.123.14)│                              │
│                         └──────────────────┘                              │
│                                                                              │
│  ┌──────────────────┐                                                      │
│  │   WordPress      │  (External - API only)                              │
│  │   (sales site)   │                                                      │
│  └──────────────────┘                                                      │
│                                                                              │
│  ┌──────────────────┐                                                      │
│  │   Git Server     │  (192.168.123.18 - Mini-Eq)                         │
│  │   (source code)  │                                                      │
│  └──────────────────┘                                                      │
└─────────────────────────────────────────────────────────────────────────────┘

---

2. Network Connectivity

2.1 Required Access from Remote Nodes

Service	Host	Port	Protocol	Purpose
PostgreSQL (PgBouncer)	192.168.123.14	6432	TCP	Pooled DB connections
PostgreSQL (Direct)	192.168.123.14	5432	TCP	Admin/backup (optional)
NFS	192.168.123.14	2049	TCP/UDP	Shared storage mount
Web API	umbra.bdmurphy.me	443	HTTPS	REST API, webhooks

2.2 Firewall Rules (on 192.168.123.14)

# PostgreSQL (PgBouncer) - allow from cluster nodes
iptables -A INPUT -s 192.168.123.0/24 -p tcp --dport 6432 -j ACCEPT

# PostgreSQL (Direct) - allow from cluster nodes
iptables -A INPUT -s 192.168.123.0/24 -p tcp --dport 5432 -j ACCEPT

# NFS - allow from cluster nodes
iptables -A INPUT -s 192.168.123.0/24 -p tcp --dport 2049 -j ACCEPT
iptables -A INPUT -s 192.168.123.0/24 -p udp --dport 2049 -j ACCEPT
# Also need: 111 (rpcbind), 20048 (mountd) - check rpcinfo -p

2.3 DNS / Hosts

Add to /etc/hosts on remote machines (or configure DNS):

192.168.123.14    umbra-db umbra-nfs umbra-api

---

3. PostgreSQL Access

3.1 Connection Parameters

# PgBouncer (RECOMMENDED - pooled, 6432)
host=192.168.123.14
port=6432
dbname=umbratrans
user=umbraudio          # or umbraformat
password=<from secret>
sslmode=require
connect_timeout=10
application_name=umbraudio-worker  # or umbraformat-worker

# Direct (for admin/backup only - 5432)
host=192.168.123.14
port=5432
dbname=umbratrans
user=umbraudio
password=<from secret>
sslmode=require

3.2 Database Roles

Role	Permissions	Used By
umbraudio	SELECT, INSERT, UPDATE on jobs, artifacts, compute_nodes, node_heartbeats, model_registry, node_models; SELECT on projects, organizations	UmbraAudio workers
umbraformat	SELECT, INSERT, UPDATE on jobs, artifacts, format_metadata; SELECT on projects, organizations	UmbraFormat workers
umbra_web	Full access to all tables (web API)	UmbraTrans web

3.3 Required Extensions

-- Already installed by migrations
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "citext";
CREATE EXTENSION IF NOT EXISTS "pgcrypto";

---

4. NFS Shared Storage

4.1 Mount Points (on Remote Nodes)

# /etc/fstab
192.168.123.14:/mnt/umbra-data/productions  /mnt/umbra/productions  nfs4  rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2  0  0
192.168.123.14:/mnt/umbra-data/artifacts    /mnt/umbra/artifacts    nfs4  rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2  0  0
192.168.123.14:/mnt/umbra-data/covers       /mnt/umbra/covers       nfs4  rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2  0  0
192.168.123.14:/mnt/umbra-data/audio        /mnt/umbra/audio        nfs4  rw,hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=2  0  0

4.2 Directory Permissions

# On NFS server (192.168.123.14)
mkdir -p /mnt/umbra-data/{productions,artifacts,covers,audio,backups}
chown -R 1000:1000 /mnt/umbra-data
chmod -R 700 /mnt/umbra-data

# On remote nodes - same UID/GID
groupadd -g 1000 umbra
useradd -u 1000 -g umbra -s /bin/bash umbra

4.3 Access Patterns by Service

Path	UmbraAudio	UmbraFormat	UmbraTrans Web
/productions/<project>-<lang>/source/	Read (source text)	Read (metadata)	Read/Write
/productions/<project>-<lang>/runs/	Read/Write (translation artifacts)	Read	Read
/artifacts/<org>/<project>/<hash>/	Read/Write (audio renders)	Read/Write (EPUB/PDF)	Read
/covers/<asset_id>/	Read	Read/Write (cover gen)	Read/Write
/audio/<project>/<lang>/	Read/Write (master audio)	Read	Read

---

5. Job Queue Integration

5.1 How Workers Claim Jobs

-- Atomic job claim (run in transaction)
BEGIN;
SET LOCAL statement_timeout = 5000;
SET LOCAL app.user_id = '<node-user-id>';
SET LOCAL search_path = 'public';

UPDATE public.jobs
SET status = 'running',
    lease_owner = '<compute_node_id>',
    lease_expires_at = now() + interval '30 minutes',
    started_at = now(),
    attempt = attempt + 1
WHERE id = (
    SELECT id FROM public.jobs
    WHERE status = 'queued'
      AND not_before <= now()
      AND (lease_expires_at IS NULL OR lease_expires_at <= now())
      AND kind IN ('<allowed-kinds>')
      AND organization_id = '<org-id>'
    ORDER BY priority, not_before, created_at
    FOR UPDATE SKIP LOCKED
    LIMIT 1
)
RETURNING *;

COMMIT;

5.2 Job Kinds by Service

Service	Job Kinds (claims)	Job Kinds (enqueues)
UmbraAudio	tts, asr, assemble, encode, master, speaker_profile	tts (after translation), assemble (after tts)
UmbraFormat	build_epub, build_pdf, cover_generate, metadata_validate	build_epub (after translation+audio), cover_generate

5.3 Job Payload Structure

// TTS Job
{
  "utterance_ids": ["utt-001", "utt-002"],
  "speaker_id": "spk-narrator",
  "voice_config": {
    "model": "xtts-v2",
    "language": "es",
    "speed": 1.0
  },
  "output_format": "wav"
}

// EPUB Build Job
{
  "project_id": "uuid",
  "language": "es",
  "include_audio": true,
  "cover_asset_id": "asset-uuid",
  "format_options": {
    "epub_version": "3.2",
    "drm": false,
    "embedded_fonts": ["Source Serif Pro"]
  }
}

5.4 Job Result Structure

// TTS Result
{
  "artifacts": [
    {
      "artifact_id": "uuid",
      "kind": "audio",
      "storage_uri": "file://192.168.123.14/mnt/umbra-data/audio/proj123/es/chapters/ch-001/track-001.wav",
      "content_hash": "sha256...",
      "byte_size": 1048576,
      "media_type": "audio/wav",
      "duration_ms": 5432
    }
  ],
  "metrics": {
    "total_duration_ms": 5432,
    "rtf": 0.15
  }
}

// EPUB Build Result
{
  "artifacts": [
    {
      "artifact_id": "uuid",
      "kind": "epub",
      "storage_uri": "file://192.168.123.14/mnt/umbra-data/artifacts/org1/proj123/hash123.epub",
      "content_hash": "sha256...",
      "byte_size": 5242880,
      "media_type": "application/epub+zip"
    }
  ],
  "validation": {
    "epubcheck": "passed",
    "page_count": 312
  }
}

---

6. Artifact Protocol (Content-Addressed Storage)

6.1 Writing Artifacts

# 1. Compute content hash
import hashlib
content_hash = hashlib.sha256(audio_bytes).hexdigest()

# 2. Write to NFS (atomic)
artifact_path = f"/mnt/umbra/artifacts/{org_id}/{project_id}/{content_hash[:2]}/{content_hash}"
os.makedirs(os.path.dirname(artifact_path), exist_ok=True)
tmp_path = artifact_path + ".tmp"
with open(tmp_path, "wb") as f:
    f.write(audio_bytes)
os.rename(tmp_path, artifact_path)

# 3. Insert artifact row (PostgreSQL)
INSERT INTO public.artifacts (
    id, project_id, organization_id, lang, kind, status,
    content_hash, byte_size, media_type, storage_key,
    storage_uri, integrity, derivation, created_at
) VALUES (
    gen_random_uuid(), $project_id, $org_id, $lang, 'audio', 'final',
    $content_hash, $byte_size, 'audio/wav', $content_hash,
    $storage_uri, 'unverified', $derivation_json, now()
)
ON CONFLICT (project_id, content_hash, kind, lang) DO NOTHING;

# 4. Link to job
INSERT INTO public.job_artifacts (job_id, artifact_id, role, created_at)
VALUES ($job_id, $artifact_id, 'produced', now());

6.2 Storage URI Format

file://<node-id>/<relative-path-from-nfs-root>

Examples:
file://umbra-audio-01/mnt/umbra-data/audio/proj123/es/chapters/ch-001/track-001.wav
file://umbra-format-01/mnt/umbra-data/artifacts/org1/proj123/ab12cd34.epub

6.3 Reading Artifacts

# Via storage_uri from artifact row
storage_uri = artifact_row["storage_uri"]  # "file://umbra-audio-01/mnt/umbra-data/..."

# Parse and read
if storage_uri.startswith("file://"):
    # Extract path after hostname
    path = storage_uri.split("/", 3)[-1]  # "/mnt/umbra-data/..."
    with open(path, "rb") as f:
        data = f.read()

---

7. Compute Node Registration

7.1 Node Registration (at startup)

-- Register compute node
INSERT INTO public.compute_nodes (
    id, organization_id, name, is_controller, trust_domain,
    locality, transport, status, agent_version, agent_started_at, created_at
) VALUES (
    $node_id, $org_id, $hostname, false, 'remote',
    $datacenter, 'https', 'provisioning', $version, now(), now()
)
ON CONFLICT (organization_id, name) DO UPDATE SET
    status = 'provisioning',
    agent_version = EXCLUDED.agent_version,
    agent_started_at = EXCLUDED.agent_started_at,
    updated_at = now()
RETURNING id;

7.2 Device Registration

-- Register GPUs/CPUs
INSERT INTO public.compute_devices (
    id, compute_node_id, kind, label, vram_total_mb, vram_free_mb,
    ram_total_mb, ram_free_mb, compute_units, throughput_factor, supports
) VALUES (
    gen_random_uuid(), $node_id, 'gpu_discrete', 'gpu:0',
    24576, 24576, 0, 0, 10496, 1.0, '{"cuda": "12.4"}'
)
ON CONFLICT (compute_node_id, label) DO UPDATE SET
    vram_total_mb = EXCLUDED.vram_total_mb,
    vram_free_mb = EXCLUDED.vram_free_mb,
    updated_at = now();

7.3 Heartbeat (every 30 seconds)

-- Update heartbeat (overwrites previous)
INSERT INTO public.node_heartbeats (
    compute_node_id, reported_at, cpu_busy_pct, ram_free_mb,
    vram_free_mb, vram_total_mb, disk_free_mb, load_avg_1m,
    resident_models, active_jobs, agent_version, raw
) VALUES (
    $node_id, now(), $cpu_pct, $ram_free, $vram_free, $vram_total,
    $disk_free, $load_avg, $resident_models_json, $active_jobs, $version, $raw_json
)
ON CONFLICT (compute_node_id) DO UPDATE SET
    reported_at = EXCLUDED.reported_at,
    cpu_busy_pct = EXCLUDED.cpu_busy_pct,
    ram_free_mb = EXCLUDED.ram_free_mb,
    vram_free_mb = EXCLUDED.vram_free_mb,
    vram_total_mb = EXCLUDED.vram_total_mb,
    disk_free_mb = EXCLUDED.disk_free_mb,
    load_avg_1m = EXCLUDED.load_avg_1m,
    resident_models = EXCLUDED.resident_models,
    active_jobs = EXCLUDED.active_jobs,
    agent_version = EXCLUDED.agent_version,
    raw = EXCLUDED.raw;

7.4 Status Transitions

provisioning → online → draining → offline → retired
                    ↘️ (heartbeat stops, >2min) ↗️

• online: Heartbeat received, accepting jobs
• draining: Finishing current jobs, not accepting new
• offline: No heartbeat > 2 minutes
• retired: Decommissioned permanently

---

8. Model Registry

8.1 Model Registration (by controller)

INSERT INTO public.model_registry (
    id, slug, kind, vram_min_mb, vram_preferred_mb, ram_min_mb,
    quantisation, context_tokens, requires_accelerators,
    reference_throughput, load_seconds, capabilities, source_uri, license
) VALUES (
    gen_random_uuid(), 'xtts-v2', 'tts', 6144, 8192, 2048,
    'fp16', 4096, ARRAY['cuda'], 1.0, 30,
    '{"languages": ["en","es","fr","de","it","pt","pl","tr","ru","nl","cs","ar","zh","ja","hu","ko"]}',
    'https://github.com/coqui-ai/TTS', 'CPML'
)
ON CONFLICT (slug) DO NOTHING;

8.2 Model Residency (worker reports)

-- When model loaded
INSERT INTO public.node_models (
    compute_node_id, model_id, device_id, state, loaded_at, vram_held_mb
) VALUES ($node_id, $model_id, $device_id, 'loaded', now(), 7168)
ON CONFLICT (compute_node_id, model_id) DO UPDATE SET
    state = EXCLUDED.state,
    loaded_at = EXCLUDED.loaded_at,
    vram_held_mb = EXCLUDED.vram_held_mb,
    updated_at = now();

-- When model unloaded
UPDATE public.node_models SET state = 'available', vram_held_mb = 0
WHERE compute_node_id = $node_id AND model_id = $model_id;

---

9. Audio Import (UmbraAudio → UmbraTrans)

9.1 Bundle Export Format

UmbraAudio exports a bundle directory:

/mnt/umbra/productions/<project>-<lang>/bundle/
├── bundle.json          # Envelope with metadata
└── manifest.jsonl       # Line-delimited JSON records

bundle.json:

{
  "bundle_format_version": 1,
  "content_hash": "sha256-of-manifest",
  "source": {
    "system": "umbraaudio",
    "node_id": "umbra-audio-01",
    "product_version": "1.2.0"
  },
  "scope": {
    "books_full_detail": ["book-dir-name"]
  }
}

manifest.jsonl records: book, chapter, speaker, utterance, render

9.2 Import Trigger

UmbraTrans imports via CLI (runs as CronJob or manual):

python -m umbratrans_web.cli import-productions \
  --organization umbra \
  --allow-production \
  --confirm

This reads bundles from /mnt/umbra/productions/*/bundle/ and populates:

• project_source_refs
• translation_chapters
• translation_speakers
• translation_utterances
• artifacts (audio renders)
• translation_imports (ledger)

---

10. Format Metadata API (UmbraFormat ↔ UmbraTrans)

10.1 Get Format Metadata

GET /api/projects/{project_id}/format-metadata?language_tag=es
Authorization: Bearer <token>

Response:

{
  "project_id": "uuid",
  "language_tag": "es",
  "title": "El Cazador de Ofertas",
  "isbn13": "9788412345678",
  "cover_front_asset_id": "asset-uuid",
  "binding": "paperback",
  "trim_w_in": 6.0,
  "trim_h_in": 9.0,
  "page_count_total": 312,
  "font_primary": "Source Serif Pro",
  "font_primary_size_pt": 11.0,
  "line_height_pt": 14.5,
  "margin_inside_in": 0.75,
  "margin_outside_in": 0.625,
  "margin_top_in": 0.875,
  "margin_bottom_in": 0.875
}

10.2 Save Format Metadata

PUT /api/projects/{project_id}/format-metadata
Authorization: Bearer <token>
Content-Type: application/json

{
  "language_tag": "es",
  "isbn13": "9788412345678",
  "cover_front_asset_id": "asset-uuid",
  "binding": "paperback",
  "trim_w_in": 6.0,
  "trim_h_in": 9.0,
  "page_count_total": 312,
  "font_primary": "Source Serif Pro",
  "font_primary_size_pt": 11.0
}

10.3 Cover Upload

POST /api/projects/{project_id}/cover?language_tag=es
Authorization: Bearer <token>
Content-Type: multipart/form-data

file: <cover-image.png>

---

11. Webhooks / Event Notifications

11.1 Job Completion Webhook

UmbraTrans can POST to external services on job completion:

POST https://umbra-audio.internal/webhooks/job-complete
Content-Type: application/json
X-Umbra-Signature: sha256=...

{
  "event": "job.completed",
  "job_id": "uuid",
  "kind": "tts",
  "project_id": "uuid",
  "organization_id": "uuid",
  "status": "complete",
  "result": { ... },
  "artifacts": [
    { "artifact_id": "uuid", "kind": "audio", "storage_uri": "..." }
  ],
  "completed_at": "2026-10-31T12:00:00Z"
}

11.2 Configure Webhook URLs

-- In organization or project settings
UPDATE public.organizations
SET webhook_urls = '{"job_complete": "https://umbra-audio.internal/webhooks/job-complete"}'
WHERE id = $org_id;

---

12. Credentials & Secrets

12.1 Service Accounts (Kubernetes)

# umbra-audio namespace on remote cluster
apiVersion: v1
kind: Secret
metadata:
  name: umbratrans-credentials
  namespace: umbra-audio
type: Opaque
stringData:
  DATABASE_DSN: "postgresql://umbraudio:PASS@192.168.123.14:6432/umbratrans?sslmode=require"
  NFS_SERVER: "192.168.123.14"
  NFS_PRODUCTIONS_PATH: "/mnt/umbra-data/productions"
  NFS_ARTIFACTS_PATH: "/mnt/umbra-data/artifacts"
  NODE_CREDENTIAL_PREFIX: "umbra-audio-"

12.2 Node Credentials (for job queue auth)

-- Created by UmbraTrans admin
INSERT INTO public.node_credentials (
    id, compute_node_id, key_prefix, key_hash, agent_name, scopes, issued_at
) VALUES (
    gen_random_uuid(), $node_id, 'umbra-audio-01-',
    '$argon2id$v=19$m=65536,t=3,p=4$salt$hash',
    'umbra-audio-worker', ARRAY['queue:claim', 'queue:complete', 'artifacts:write'],
    now()
);

Worker uses this credential to authenticate job queue operations.

---

13. Health Checks & Monitoring

13.1 Required Health Endpoints

Each service exposes:

GET /health          → { "status": "ok", "service": "umbra-audio", "version": "1.2.0" }
GET /metrics         → Prometheus metrics

13.2 Key Metrics to Export

Metric	Type	Labels
umbra_jobs_claimed_total	Counter	service, kind, status
umbra_job_duration_seconds	Histogram	service, kind
umbra_artifact_bytes_written	Counter	service, kind
umbra_node_heartbeat_age_seconds	Gauge	node_id
umbra_gpu_utilization	Gauge	node_id, device_id
umbra_queue_depth	Gauge	kind, status

---

14. Deployment Checklist for Remote Services

UmbraAudio

• PostgreSQL role umbraudio created with correct grants
• Node credentials inserted in node_credentials
• NFS mounts configured in /etc/fstab
• Compute node registered in compute_nodes
• Devices registered in compute_devices
• Models registered in model_registry
• Heartbeat cron running (every 30s)
• Job poller running for tts, asr, assemble, encode, master
• Artifact writer writing to /mnt/umbra/artifacts/ + artifacts table
• Webhook endpoint for job completion (optional)

UmbraFormat

• PostgreSQL role umbraformat created with correct grants
• NFS mounts configured in /etc/fstab
• Job poller running for build_epub, build_pdf, cover_generate, metadata_validate
• Format metadata API integrated (GET/PUT /format-metadata)
• Cover upload integrated (POST /cover)
• Artifact writer for EPUB/PDF to /mnt/umbra/artifacts/
• Webhook endpoint for job completion (optional)

---

15. Troubleshooting

Issue	Diagnosis	Resolution
Job claim returns 0 rows	No queued jobs, or wrong organization_id	Check SELECT * FROM jobs WHERE status='queued' AND organization_id=...
Lease expires mid-job	Job takes >30min	Increase lease_expires_at on claim, or heartbeat extends it
NFS mount hangs	Network partition, server down	Use hard,intr options; check rpcinfo -p 192.168.123.14
SSL connection failed	Cert mismatch	Use sslmode=require with valid CA, or sslmode=verify-ca
Artifact conflict	Duplicate content hash	ON CONFLICT DO NOTHING - safe to ignore
Heartbeat not updating	Clock skew	Ensure NTP sync on all nodes

---

16. Contact & Escalation

Role	Contact	Channel
UmbraTrans Platform	platform@umbra.example.com	Slack #umbra-platform
Database Admin	dba@umbra.example.com	PagerDuty
Network/Infra	infra@umbra.example.com	Slack #umbra-infra

---

Appendix: Quick Reference Commands

# Test DB connectivity
psql "postgresql://umbraudio:PASS@192.168.123.14:6432/umbratrans?sslmode=require" -c "SELECT now();"

# Test NFS mount
mount -t nfs4 192.168.123.14:/mnt/umbra-data/productions /mnt/test && ls /mnt/test

# Check job queue
psql $DSN -c "SELECT kind, status, count(*) FROM jobs GROUP BY kind, status;"

# Check node status
psql $DSN -c "SELECT id, name, status, last_heartbeat_at FROM compute_nodes;"

# View recent artifacts
psql $DSN -c "SELECT kind, count(*), sum(byte_size) FROM artifacts GROUP BY kind;"