umbragit/UmbraAudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Narrator speaks the translation language; characters keep their own accent

Browse Source 
Specification change. Two different rules, and they need different
mechanisms.

Narrator: accent = translation language. Its reference was generated by
VoiceDesign from a description that hardcoded '{accent} English', baking an
American-English timbre into the WAV; cloning that into German gives German
phonology from an American voice. Ten per-language references now exist
(narrator_EN.wav ... narrator_ZH.wav), each designed from the narrator's
timbre plus 'Speaking {language} as a native {language} speaker'.

Characters: keep the accent on the speaker page. Their accents were already
encoded in their existing refs - what destroyed them was the language
parameter forced onto every utterance, which made every voice adopt the
target language's prosody. That is why Deal Hunter came out French-native.
Phonology and accent are now separate controls: language carries phonology,
an explicit voice_description carries accent character. Deal Hunter gets
'with a strong Australian accent' in every language.

Measured A/B, French line from Deal Hunter: with and without the Australian
instruction the waveforms differ by 152% of signal amplitude and duration
moves 4.24s -> 4.08s, so the instruction is reaching the model. Whether it is
convincingly Australian is an ear test, not something measurable here.

Re-rendered all 120 clips under the new spec, 0 failures. The previous
render is kept in Lang_test_preSpec for A/B. 24 speakers carry a non-default
accent; the narrator is deliberately excluded from that set.

Suite: 721 passed, 8 pre-existing.
This commit is contained in:
B.D. Murphy
2026-10-13 06:19:43 -07:00
parent f381c58160
commit d7fc0236bc
3 changed files with 301 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

162
scripts/generate_narrator_refs.py Normal file
View File

@@ -0,0 +1,162 @@
"""Generate a narrator reference voice for each target language.
Specification change (2026-10-13):
* The narrator speaks with the accent of the *translation language*. An Italian
book is narrated by an Italian-sounding narrator, not by an American reading
Italian.
* Every other character keeps whatever accent is defined on the speaker page -
Deal Hunter stays Australian, Duchess Victoria stays Canadian, and so on.
Why a reference per language is needed: the narrator reference is generated by
the VoiceDesign model from a description that used to hardcode
"{accent} English", which baked an American-English timbre into the ref. Cloning
that ref into German gives German phonology from an American voice. Passing the
target language fixes the phonology but not the identity.
Characters are a different problem and do NOT need new references: their accents
are already encoded in their existing refs. What was erasing them was the
`language` parameter forcing the target language onto every utterance. The
renderer now passes the character's accent as an explicit instruction alongside
the clone, so phonology follows the translation while prosody and accent
character follow the speaker page.
Usage:
python scripts/generate_narrator_refs.py --project DealHunterShip_4.0e_whole-to-audio
python scripts/generate_narrator_refs.py --languages DE FR --dry-run
"""
from __future__ import annotations
import argparse
import sys
import time
import urllib.request
import uuid
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from umbraaudio.config import load_config # noqa: E402
TTS_URL = "http://127.0.0.1:8881"
# Qwen3-TTS language names, from the model's own codec_language_id keys.
LANGUAGES = {
"EN": ("English", "english"),
"ZH": ("Chinese", "chinese"),
"JA": ("Japanese", "japanese"),
"KO": ("Korean", "korean"),
"DE": ("German", "german"),
"ES": ("Spanish", "spanish"),
"FR": ("French", "french"),
"RU": ("Russian", "russian"),
"IT": ("Italian", "italian"),
"PT": ("Portuguese", "portuguese"),
}
# The narrator line from the sample chapter, per language. The text must be in
# the target language: it is what the model reads while shaping the voice.
NARRATOR_TEXT = {
"EN": "Welcome to the audiobook Deal Hunter. I am your narrator.",
"DE": "Willkommen zum Hörbuch Deal Hunter. Ich bin Ihr Erzähler.",
"FR": "Bienvenue dans le livre audio Deal Hunter. Je suis votre narrateur.",
"ES": "Bienvenidos al audiolibro Deal Hunter. Soy su narrador.",
"IT": "Benvenuti all'audiolibro Deal Hunter. Sono il vostro narratore.",
"PT": "Bem-vindos ao audiolivro Deal Hunter. Sou o seu narrador.",
"RU": "Добро пожаловать в аудиокнигу «Deal Hunter». Я ваш рассказчик.",
"JA": "オードブック『Deal Hunter』へようこそ。ナレーターの私です。",
"KO": "오디오북 『Deal Hunter』에 오신 것을 환영합니다. 내레이션은 제가 맡고 있습니다.",
"ZH": "欢迎收听有声书《Deal Hunter》。我是您的旁白。",
}
# The narrator's character, held constant across every language. Only the
# language and its accent change, so the voice stays recognisably the narrator.
NARRATOR_TIMBRE = (
"A male narrator with a deep, resonant, authoritative baritone voice. "
"Calm, measured delivery, warm and trustworthy, unhurried pacing."
)
def build_instruct(language_name: str) -> str:
return (
f"{NARRATOR_TIMBRE} "
f"Speaking {language_name} as a native {language_name} speaker, with a "
f"natural {language_name} accent and native {language_name} prosody. "
f"This is an audiobook narration, not a translation being read aloud."
)
def post_design(text: str, instruct: str, language: str, timeout: int) -> bytes:
"""Ask the server for a designed voice. Returns raw WAV bytes."""
boundary = f"----umbra{uuid.uuid4().hex}"
parts: list[bytes] = []
def field(name: str, value: str) -> None:
parts.append(
f'--{boundary}\r\nContent-Disposition: form-data; name="{name}"\r\n\r\n'
f"{value}\r\n".encode()
)
field("text", text)
field("voice_description", instruct)
field("language", language)
parts.append(f"--{boundary}--\r\n".encode())
req = urllib.request.Request(
f"{TTS_URL}/v1/tts",
data=b"".join(parts),
headers={"Content-Type": f"multipart/form-data; boundary={boundary}"},
method="POST",
)
with urllib.request.urlopen(req, timeout=timeout) as resp:
body = resp.read()
if not body.startswith(b"RIFF"):
raise RuntimeError(f"expected WAV, got {body[:120]!r}")
return body
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--project")
ap.add_argument("--languages", help="comma-separated suffixes, default all")
ap.add_argument("--timeout", type=int, default=900)
ap.add_argument("--dry-run", action="store_true")
args = ap.parse_args()
config = load_config(args.project)
voices_dir = config.data_dir / "voices"
voices_dir.mkdir(parents=True, exist_ok=True)
wanted = (
{s.strip().upper() for s in args.languages.split(",")}
if args.languages
else set(LANGUAGES)
)
targets = [(k, v) for k, v in LANGUAGES.items() if k in wanted]
print(f" narrator references -> {voices_dir}")
for suffix, (name, code) in targets:
out = voices_dir / f"narrator_{suffix}.wav"
text = NARRATOR_TEXT.get(suffix, "")
if not text:
print(f" {suffix}: no narrator line available, skipped")
continue
instruct = build_instruct(name)
if args.dry_run:
print(f"\n {suffix} ({code}) -> {out.name}")
print(f" text : {text}")
print(f" instruct: {instruct}")
continue
t0 = time.time()
try:
wav = post_design(text, instruct, code, args.timeout)
out.write_bytes(wav)
kb = len(wav) / 1024
print(f" {suffix} {name:11s} {kb:6.0f} KB {time.time()-t0:5.1f}s -> {out.name}")
except Exception as e: # surfaced, not swallowed
print(f" {suffix} {name:11s} FAILED: {e}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

57
scripts/render_lang_test.py
View File

@@ -77,6 +77,45 @@ def speaker_refs(data_dir: Path) -> dict[str, Path]:
return {p.name[: -len("_ref.wav")]: p for p in voices.glob("*_ref.wav")}
# How the voice page's accent is phrased when handed to the model. The
# language parameter sets phonology; this sets accent character, which is how
# Deal Hunter stays Australian while speaking French.
ACCENT_INSTRUCT = {
"australian": "with a strong Australian accent",
"canadian": "with a Canadian accent",
"american": "with a neutral American accent",
"british": "with a British accent",
"indian": "with an Indian accent",
"nigerian": "with a Nigerian accent",
"kenyan": "with a Kenyan accent",
"hong_kong": "with a Hong Kong accent",
}
def accent_for(display: str, data_dir: Path) -> str:
"""The speaker page's accent, as an instruction phrase.
The narrator is excluded on purpose: its accent is the translation language
(see generate_narrator_refs.py), and restating "American" for the German
narrator is precisely the bug this replaces.
"""
import json as _json
if display.lower() == "narrator":
return ""
speakers = _json.loads((data_dir / "speakers.json").read_text())["speakers"]
by_lower = {s.get("primary_name", "").lower(): s for s in speakers}
rec = by_lower.get(display.lower())
if not rec:
for s in speakers:
aliases = {a.lower() for a in (s.get("aliases") or [])}
if display.lower() in aliases:
rec = s
break
accent = (rec or {}).get("accent") or ""
return ACCENT_INSTRUCT.get(accent.lower(), "")
def build_resolver(data_dir: Path, refs: dict[str, Path]) -> dict[str, str]:
"""Map the display name in a translation to the real voice slug.
@@ -111,6 +150,7 @@ def post_tts(
ref_text: str | None,
timeout: int,
language: str = "english",
voice_description: str = "",
) -> bytes:
"""POST multipart by hand; the boundary must match byte for byte."""
boundary = f"----umbra{uuid.uuid4().hex}"
@@ -134,6 +174,8 @@ def post_tts(
# Must be sent: the server used to hardcode language="english" in every
# branch, so German and French were synthesised with English phonology.
field("language", language)
if voice_description:
field("voice_description", voice_description)
if prompt_slug:
field("prompt_slug", prompt_slug)
if ref_audio:
@@ -281,6 +323,9 @@ def main() -> int:
out_dir = out_root / suffix
out_dir.mkdir(parents=True, exist_ok=True)
made: list[Path] = []
lang_ref = data_dir / "voices" / f"narrator_{suffix}.wav"
if not lang_ref.exists():
lang_ref = None
for u in doc["utterances"]:
slug = slugify(u["speaker"])
dest = out_dir / f"utt{u['index']:02d}_{u['speaker'].replace(' ', '_')}.wav"
@@ -291,13 +336,23 @@ def main() -> int:
t0 = time.time()
try:
use_prompt = args.voice_mode == "prompt" and slug in prompts
# The narrator's accent is the translation language, so it uses
# a reference designed for that language. Every other character
# keeps its own reference and carries its page accent through.
ref = refs[slug]
instruct = ""
if u["speaker"].lower() == "narrator" and lang_ref is not None:
ref = lang_ref
else:
instruct = accent_for(u["speaker"], data_dir)
audio = post_tts(
TTS_URL, u["text"],
prompt_slug=slug if use_prompt else None,
ref_audio=None if use_prompt else refs[slug],
ref_audio=None if use_prompt else ref,
ref_text=None if use_prompt else REF_TEXT,
timeout=args.timeout,
language=LANG_CODES.get(suffix, "english"),
voice_description=instruct,
)
dest.write_bytes(audio)
made.append(dest)

83
specs/29-accent-and-voice-spec.md Normal file
View File

@@ -0,0 +1,83 @@
# Accent and voice specification for translated books
Status: specification, agreed 2026-10-13. Implemented by
`scripts/generate_narrator_refs.py` + `scripts/render_lang_test.py`.
## The rule
Two rules, and they are different:
1. **The narrator speaks with the accent of the translation language.** An
Italian book is narrated by an Italian-sounding narrator. A German book by a
German one. Not an American reading Italian.
2. **Every other character keeps the accent defined on the speaker page.**
Deal Hunter stays Australian. Duchess Victoria stays Canadian. Xu Ming stays
Hong Kong. This holds in every target language.
The asymmetry is deliberate. The narrator is a narrator in *each* language and
is heard far more than any single character, so a foreign accent on the
narrating voice is the thing listeners notice. Characters are defined
individually on the speaker page, and that definition is the contract.
## Why this needed two different mechanisms
**The narrator** needed a new reference voice per language. Its reference was
generated by the VoiceDesign model from a description that hardcoded
`"{accent} English"`, which baked an American-English timbre into the WAV.
Cloning that into German gives German phonology from an American voice. Passing
the target language fixes phonology but not identity.
So the narrator now has one reference per language, generated by VoiceDesign from
the narrator's timbre plus "Speaking {language} as a native {language} speaker".
`narrator_EN.wav` … `narrator_ZH.wav`, ten in all, in
`<project>/.umbra/voices/`.
**Characters needed no new references.** Their accents are already encoded in
their existing reference WAVs. What was destroying them was the `language`
parameter: forced onto every utterance, it made every voice adopt the target
language's prosody, which is why Deal Hunter came out French-native.
Phonology and accent are separate controls, and the fix uses them separately:
| | language parameter | accent carried by |
|---|---|---|
| Narrator | target language | its per-language reference |
| Characters | target language | existing reference + explicit accent instruction |
The accent instruction is derived from the speaker page (`accent_for()` in the
renderer) and passed as `voice_description` alongside the clone. Measured A/B on
one French line from Deal Hunter: with and without her Australian instruction,
waveforms differ by 152% of signal amplitude and duration moves 4.24s -> 4.08s.
## Deal Hunter's current accents
24 speakers have a non-default accent. The narrator is deliberately *not* in
this list — its accent is the translation language.
australian Deal Hunter (strong) canadian 17 speakers incl.
nigerian Adonai Maul, Dealer Kainda, Nanny,
indian Alec, Interrogator Duchess Victoria
kenyan Warbots hong_kong Trader Wakam, Xu Ming
Any character whose accent is `none` simply gets no instruction and speaks the
target language plainly.
## What this does not fix
- **The ICL prompt path is still unusable.** One `prompt_slug` request takes the
server from 5.4 GB to 19.1 GB and never returns. The render path uses
`x_vector_only_mode=True`, which is stable. Cached prompts are effectively
bypassed.
- **Accent is verified by measurement, not by ear.** The instruction demonstrably
changes the output. Whether "strong Australian" is *convincingly* Australian
in French is a listening test for the reviewer, not something measurable here.
- **One pass per language.** Languages are rendered sequentially so the system
can reset between them; not all fifteen at once.
## Division of labour with the translation node
The translation node produces the text and the print paperback, including the
full analysis needed for correct translation. This system turns that text into
audio and knows nothing about translation. The renderer consumes
`(index, speaker, text)` per utterance and emits WAV.