c86e323d7a
Quality-first multi-pass audiobook generation from English manuscripts. Includes: - umbraaudio/ core package (pipeline, config, voices, web review UI) - scripts/ CLI utilities + model download script - tts_server/ remote TTS server - specs/ design docs, tests/, pyproject.toml + uv.lock - Qwen3-TTS git submodule (pinned 022e286) with local demo.py UI patch saved at scripts/qwen3_tts_demo_umbra_patch.patch - .env with runtime config/secrets Excluded (see .gitignore): models/ (downloaded via scripts/download_models.sh), output/, logs/, .venv, .umbra runtime data, and book projects (DealHunterShip_*), which are committed separately.
88 lines
6.5 KiB
Plaintext
88 lines
6.5 KiB
Plaintext
Add user management and login to system. plan for future sales. Users will have projects each is seperate.
|
|
|
|
|
|
Rules
|
|
The tags inform the system on how to process parts of the book.
|
|
All info before [book start] can be ignored.
|
|
[Book start] - TItle page of book. It will have Title, optional subtitle on a sperate lines with no punctionation useed. The system can use line breaks understand the utterances. The bottom of the page will have the author on one line. The next optional line will be Publisher on one line.
|
|
[copyright]
|
|
The copyright page will have a mix of content. The needs to be segmented based on line breaks. THere are lines and paragraphs that use sentance punctuation which must be used.
|
|
[author note] should be parsed with punctuations, ".?!", to segment utterances.
|
|
[TOC] - table of contents is not included in the audio book.
|
|
[prologue] - optional and will follow punctuation, ".?!", to segment utterance.
|
|
[chapter] - Every chapter will follow a patter. THere is a chapter title with no punctuation as the first line after [chapter]. The paragraphs will use punctuation to segment sentences for utterance. There can be scene breaks designated by "<<<>>>". The speakers will change with a scene, but utterances will use sentance punctuation.
|
|
[epilogue] - Will be treated as a chapter, after the main story.
|
|
[Back Matter] - Back matter will be a mixture of utterances and will need to be processed similar to the copyright page.
|
|
[req review] is focused on requeting a review and the utterance of a webpage URL isn't needed.
|
|
[about author] - will use best practices for utterances.
|
|
|
|
Parse runs.
|
|
# Scan to identify sections, chapters and scenes - based on tags. The book is too large for the AI, so it must parse by chapters. The tags can inform this parse and make it a simple program.
|
|
# AI Discover characters, speakers, aliases, and proper names of people and places. Speakers are stored in book speakers.json
|
|
# AI Parse, for each speaker to find their voice, any clues in the book about how they talk. These will be used to design unique voices for each speaker. Each speaker attributes are stored in the json file.
|
|
# The online system will show a page for speakers - reference already. This will allow the user to adjust or approve voices of each speaker. The results will be stored in the speaker.json.
|
|
# The book will be parced to find proper names, people, locations, etc. These will be shown on a web page where the user can listen to the generated name and alter the pronunciation to be correct. A corrected name will use that pronunciation by every speaker and the name reference in utterances will add the pronunciation.
|
|
# The book will be parced section by section with store meta data captured, emotional tone, speakers in this section - tension? THe output will be sentences / utterances with speaker, *** Narrator?
|
|
# Utterances will be generated and shown on the web for verification using the parcing for tone, emotion, etc.
|
|
# When the user has approved all the utterances in a chapter the chapter will be built for the final output and available for listening.
|
|
# The user can trigger the re-generation of voice sameples, name pronunciation and utterances after changes.
|
|
# Chapter pages with utterances will allow adding, editing, deleting utterances from the chapter. An added utterance can be moved into the flow at any point. This allows last minute fixes vs updating the manuscript and processing all the steps again.
|
|
# Utterances will be a sentance with an identified speaker, using any emotional modifiers, to say the words.
|
|
|
|
He/She said.
|
|
|
|
# Identify each person by their voice. Don't say bob says as the narrator then switch to bob's voice to speak. each voice is unique so the listener can tell from the voice who is speaking.
|
|
|
|
# Scan the book to identify the first time each speaker talks. Within two sentences someone else must say their name. The narrator will do this if no one else does. Every instanance that is identified will be flagged and shown for adjustments. A page for (speaker adjustments) will show five lines before and after the new speaker's first line.
|
|
|
|
|
|
|
|
|
|
4. Technical Constraints & Logic
|
|
Punctuation as Prosody
|
|
The system must interpret typography as direct instruction for the TTS engine:
|
|
|
|
Ellipses (...): Triggers trailing, thoughtful pauses.
|
|
|
|
Em-dashes (—): Triggers sudden shifts in tone or abrupt interruptions.
|
|
|
|
ALL-CAPS: Triggers increased volume and intensity markers.
|
|
|
|
Question Marks: Triggers terminal pitch rising.
|
|
|
|
Voice Consistency Invariants
|
|
Narrator: Fixed at a low-emphasis, calm cadence to maintain baseline stability.
|
|
|
|
Cast Distinctiveness: Voices must be audibly distinguishable via Qwen3-TTS "VoiceDesign" (instruct mode).
|
|
|
|
VRAM Management: The orchestrator.py must manage the lifecycle of LM Studio and Qwen3-TTS to prevent OOM errors, as both models are VRAM-intensive.
|
|
|
|
|
|
A. Phase I: Character & Voice DiscoveryPurpose: Extract entities and define their vocal "DNA" for Qwen3-TTS.File: CHARACTER_DISCOVERY_PROMPT.mdPlaintextAnalyze the provided manuscript text. Identify all speaking characters and the narrator.
|
|
For each character, generate a Qwen3-TTS 'instruct' string that captures their unique
|
|
vocal identity.
|
|
|
|
Return ONLY a JSON object:
|
|
{
|
|
"CharacterName": {
|
|
"instruct": "A [pitch] and [texture] [identity] who speaks with a [pacing] rhythm, [emotion] personality, and [accent] accent. Distinguishing mannerisms: [details].",
|
|
"evolution_markers": [
|
|
{
|
|
"trigger": "e.g., Chapter 7",
|
|
"instruction_update": "New instruction string reflecting changed state."
|
|
}
|
|
]
|
|
}
|
|
}
|
|
B. Phase II: Speaker AttributionPurpose: Resolve unattributed dialogue using contextual awareness.Prompt:PlaintextYou are an expert literary analyst. Review the provided segment of dialogue.
|
|
Using the character profiles provided in speakers.json, attribute the
|
|
unlabeled dialogue to the correct character.
|
|
|
|
Focus on:
|
|
1. Resolving pronouns based on preceding context.
|
|
2. Distinguishing between the narrator and active characters.
|
|
3. Maintaining consistency with established character traits.
|
|
|
|
Output the attribution as:
|
|
{"utterance_id": "ID", "character": "CharacterName", "confidence": 0.0-1.0}
|
|
3. Pipeline StagesPhaseDescriptionQuality Gate1a: ParseSegment text into chapters/scenes; strip non-audio elements.Tag verification1b: DiscoveryLLM extracts characters; generates voice_profiles.json.Human review of voices2: AttributionMechanical splitting + LLM-based dialogue attribution.0.05% error threshold3: GenerationQwen3-TTS synthesis using instruct and clone modes.Audio test pass4: ReviewWeb UI validation of utterances and character voices.Human sign-off |