UmbraAudio/specs/31-utterance-language-and-voiced-flag.md

Spec 31 — Utterance source language and the voiced flag

Status: agreed 2026-10-17 (operator decisions recorded in §4), being implemented. Extends spec 30, which remains the target-language design.

---

1. The two axes, kept apart

The single most important thing in this spec. Conflating these two is the design error it exists to prevent.

	Source axis (this spec)	Target axis (spec 30)
Question	What language is this utterance's text written in?	What language are we rendering it into?
Example	A Spanish line inside an English book	The whole book in German
Where it lives	utterances.language_code	utterance_translations.language_code
Translated?	No — it is story content	Yes, that is the point
Cardinailty	one per utterance	one per (utterance, target language)

A Spanish phrase in a Chinese scene is not a translation target. It is source content that must be spoken in Spanish and must be excluded from translation. "Bob overhears them speaking in German" is the test case: the German is narrative payload. The reader is meant to read it. Translating it destroys the scene.

Both axes are needed. They do not conflict, and they are not the same column.

2. What exists today — measured, not assumed

Thing	State
utterances (Deal Hunter)	4,319 rows, no language column — one text, implicitly English
qwen_tts.py language	hardcoded language="english" at lines 494 and 522 — never varies per utterance
/v1/tts language	language: str = Form("english"), free-form, not validated against the supported set
utterance_translations	does not exist (spec 30 is a proposal; only a mockup route is built)
translation export/import	does not exist
language in config	none — the source language is the constant SOURCE_LANGUAGE = "en" in the mockup
characters.status = 'voiced'	already means "this speaker has a voice reference" — a different concept, at speaker grain
mockup "translated but not voiced"	pipeline progress (text delivered, audio not yet rendered) — also not this flag
sample translations	10 languages delivered as samples; the 5 gap languages have none

Three different things in this tree say "voiced" and mean different things. §6 gives this flag an unambiguous name for that reason.

3. The constraint that shapes the design

is_current() (umbraaudio/generation/staleness.py) compares the marker's recorded text against the utterance's text. Spec 30 §2 established that audio must be namespaced per language or two languages destroy each other's markers.

This spec adds a second, finer requirement: language must be in the marker, because an utterance's language can be corrected after its audio was made. A Spanish line voiced as English is byte-comparable (the text is unchanged) but acoustically wrong. Without language in the marker, correcting the tag would not invalidate the audio, and the wrong render would survive every subsequent check.

4. Decisions (operator, 2026-10-17)

1. The voiced flag is per-language. It lives with the translation row, not on the utterance. The source language always voices everything, so a single utterance-level flag cannot express "voiced in English, not in German" — which is exactly the sentence-collapse case.
2. The source language tag comes from UmbraTrans during translation. UmbraTrans is the authority on identification. See §9 for the sequencing consequence this creates.
3. A tag in an unsupported language is allowed and recorded; the render is refused. Never emit a degraded voice. Fail loud, never wrong.
4. Scope: source side + translation alignment.

5. Language registry

One module, one source of truth: umbraaudio/languages.py.

Class	Languages	Voiceable?
Qwen3-TTS (10)	Chinese, English, French, German, Italian, Japanese, Korean, Portuguese, Russian, Spanish	yes
Gap (5)	Dutch, Norwegian, Polish, Arabic, Hindi	no — tag recorded, render refused

sources — where a tag can originate — is recorded so a reviewer can tell a human correction from a machine identification:

origin	meaning
unknown	no tag yet (default)
umbratrans	identified by the controller during translation
operator	corrected by hand

The registry is the only place the supported set is written down. /v1/tts validation (spec 30 §8) and the export filter both read it, so they cannot disagree.

6. Schema

6.1 utterances gains the source tag

ALTER TABLE utterances ADD COLUMN language_code TEXT;      -- NULL = project source language
ALTER TABLE utterances ADD COLUMN language_origin TEXT;    -- 'unknown' | 'umbratrans' | 'operator'

NULL means "the project's source language", which keeps all 4,319 existing rows correct with no backfill and no ambiguity. A non-NULL value is a deliberate claim that the text is in another language.

Not named voiced. The per-language flag lives on the translation row (§6.2), and the word voiced is already taken twice in this tree (§2).

6.2 utterance_translations gains the flag

Spec 30 §3.2's table, verbatim, plus one column:

CREATE TABLE IF NOT EXISTS utterance_translations (
    id            INTEGER PRIMARY KEY,
    project_id    INTEGER NOT NULL,
    utterance_id  INTEGER NOT NULL REFERENCES utterances(id),
    language_code TEXT NOT NULL,          -- target language, 'de'
    language_name TEXT NOT NULL,          -- 'German'
    text          TEXT NOT NULL,
    status        TEXT NOT NULL,          -- 'draft' | 'confirmed'
    origin        TEXT NOT NULL,          -- 'umbratrans' | 'manual'
    source_ref    TEXT,
    delivered_at  TEXT,
    confirmed_at  TEXT,
    created_at    TEXT NOT NULL,
    updated_at    TEXT NOT NULL,
    voiced        INTEGER NOT NULL DEFAULT 1,   -- NEW: render audio for this row?
    review_reason TEXT,                          -- NEW: why it is not voiced
    UNIQUE(utterance_id, language_code)
);

voiced = 0 means this target language does not voice this utterance. The canonical case is collapse: the source used three sentences, the target needs two, so one source utterance has no target counterpart. It is not deleted — deleting it would lose the alignment and make the gap unreconstructable. It is marked not-voiced with a reason and surfaced for review.

The default is 1, so every ordinary translation is voiced and nothing changes for the common case.

6.3 Nothing else moves

English audio stays at .umbra/chapter_audio/. No existing column is renamed or dropped. Zero regression risk to the pipeline that works today — the same reasoning spec 30 §4 used.

7. Export contract (UmbraAudio → UmbraTrans)

One row per utterance. This extends spec 30 §7 rather than replacing it.

{utterance_id, chapter_id, chapter_index, utterance_index, speaker_name,
 source_text, context_before, context_after,
 language_code,          -- NEW: non-null only when already known
 book_language}          -- NEW: the project's primary language

language_code is sent when known so a re-delivery does not ask UmbraTrans to re-identify a line the operator already corrected. It is null on the first pass, which is normal.

8. Import contract (UmbraTrans → UmbraAudio)

POST /api/import/translations. Idempotent via UNIQUE(utterance_id, language_code). Refuses to silently overwrite a confirmed row.

Per utterance, UmbraTrans may return:

field	meaning
source_language	identified language of the original text. Stored on utterances.
text + language_code	a translation into language_code. Stored in utterance_translations.
voiced: false + reason	this utterance has no target counterpart (collapse).
(no text)	the utterance is in a non-primary language — not translated, by design.

The last row is the whole point of the source axis: a Spanish line comes back with source_language: "es" and no translation. That is a correct, expected response, not an omission to be chased.

An source_language in the gap set (Dutch, Norwegian, Polish, Arabic, Hindi) is stored and recorded (§4.3), and the render is refused. It is reported as an explicit refusal, not as a failure of the import.

9. The sequencing consequence (stated plainly)

Because the tag arrives during translation (§4.2), the source-language master is voiced before any tag exists. So today's English master of a bilingual book contains at least one wrong-voice line: a Spanish phrase read with English phonology.

That line is corrected only after a translation round-trip returns es. For the correction to actually happen, language must be in the staleness marker (§10) — otherwise the audio is never invalidated and the wrong render survives indefinitely.

This is a consequence of the decision, not a defect in it. It is recorded here so nobody later reads "the English master is complete" as "the English master is correct".

10. Staleness

write_marker() records language (the effective language the clip was synthesised in); is_current() compares it. A clip whose utterance language has changed since it was made is stale, even when its text is byte-identical.

This is the same class of bug the release work already fixed: a declared value (the marker says English) disagreeing with a measured one (the tag now says Spanish) must force a re-render, not pass silently.

11. Refuse-render is loud

An utterance whose language cannot be voiced is not silently skipped. It appears:

• in a count on the page (N utterances cannot be voiced in <language>),
• in a list, with the language and the reason (no Qwen3-TTS voice for 'nl'),
• and in the pipeline's completion report.

A deliberate hole that nobody can see is indistinguishable from a bug. The operator chose fail-loud (§4.3); this is the part that makes it actually loud.

12. What this does not do

• Does not translate anything. UmbraTrans does the translating; this stores what comes back and honours the source axis.
• Does not implement spec 30's UI language selector. That remains spec 30, which can now build on a schema that already knows what "the source language" means per utterance.
• Does not backfill a language onto existing rows. Every existing utterance is correctly NULL = source language. Backfilling would assert an identification nobody performed.
• Does not choose a voice for a foreign passage. A character keeps their own voice (spec 29); only the language of the phonology changes.

13. Open questions

1. Book-level source language. Currently a constant ("en"). Should it become a column on projects, set at parse time? Needed to know what "not the primary language" means per book. Proposed: yes, column on projects, defaulting to en.
2. Character voice for a foreign line. A character with an Australian accent reading Spanish — spec 29 says characters keep their accent. Does "accent" mean English-accented Spanish, or native Spanish in that character's timbre? Spec 29 leans to the latter for the narrator and the former for characters; unresolved for in-dialogue code-switching.
3. Detecting a missing language tag. If UmbraTrans never runs, no foreign line is ever tagged and the master is silently all-English. A cheap heuristic (non-Latin script, dictionary miss) could flag candidates for review. Out of scope; noted because §9's correction path never fires without a tag.