Add a voice-regeneration queue with inline per-speaker state
Reviewing speaker exceptions meant clicking Regenerate, watching a modal that
blocked the page, then navigating back to find out what happened. Worse, the
route refused a second request with a 409 while one was running, and
/api/generation/status exposed only a single global `active` flag - so there
was nowhere to put a second speaker and nothing to render per row.
Now: POST returns as soon as the task is captured, the row shows a status tag
(queued / working / done / failed), and further speakers can be queued while one
generates. No navigation, no modal.
umbraaudio/voices/voice_queue.py
- Ordered, serial queue. TTS is strictly serial here, so items run one at a
time in enqueue order; the second speaker waits rather than contending.
- Per-item state, persisted to voice_queue.json. An item left `running` by a
crash is marked `error` on load - the honest outcome, since silently
re-running could write audio nobody asked for twice.
- Enqueueing a speaker that is already pending is a no-op returning the
existing state, so a double-click cannot queue the same work twice.
- `cancel` withdraws a queued item but refuses one already running.
- Status vocabulary (queued/running/done/error) is deliberately small and
stable: it is what the controller will map onto node task states.
umbraaudio/web/voice_queue_routes.py - POST /api/voices/queue (capture the task),
GET /api/voices/queue (report per-item state), POST .../{speaker}/cancel. Kept
out of the hot routes.py, following name_token_routes.
/api/speakers/regenerate-voice is now a thin shim onto the queue so a
bookmarked call still works and the two paths cannot race. Net -24 lines in
routes.py; the blocking implementation is gone, not commented out.
The UI's modal, its 409 handler and the page-load resume handshake were removed
(67 lines) rather than left dormant - queue state is re-read from the snapshot
on load and after each HTMX row swap, so it survives navigation.
Unknown speaker names are rejected with a 404 at enqueue time. Previously they
reached the profile builder and surfaced as a bare KeyError('primary_name') in
the failed tag.
Verified live: two speakers enqueued back to back, first ran while the second
stayed queued, then the second started automatically on completion; both wrote
real ref WAVs. 16 new tests. Full suite 665 passed, 8 pre-existing failures.
This commit is contained in:
B.D. Murphy
Binary file not shown.
|
Before Width: | Height: | Size: 661 KiB |
215
backup/author_voice_lab/FINDINGS.md
Normal file
215
backup/author_voice_lab/FINDINGS.md
Normal file
@@ -0,0 +1,215 @@
|
||||
# Author voice — matrix backup, matrix→WAV reproducibility, and voice-range lab
|
||||
|
||||
Date: 2026-10-09
|
||||
Base: `author.pt` vmatrix (SHA-256 `d38a2dc8…`), the Author's own voice.
|
||||
Artifacts: `backup/author_voice_matrix/`, `backup/author_voice_lab/`
|
||||
|
||||
---
|
||||
|
||||
## 1. The matrix is backed up and its provenance is now proven
|
||||
|
||||
Two locations, checksummed, with a manifest and a restore sheet:
|
||||
|
||||
- `backup/author_voice_matrix/20261009T191636Z/` (1 TB volume)
|
||||
- `~/.umbra/library/author_voice_matrix/20261009T191636Z/` (boot volume)
|
||||
|
||||
Contains `author_v2.pt` (live), `author_v1.pt` (superseded), the full reference
|
||||
chain, `manifest.json` (SHA-256 + provenance per file) and `RESTORE.md`.
|
||||
|
||||
**This mattered.** `voice_prompts/` is git-ignored project cache, and the *only*
|
||||
previous copy of v1 sat in an OS temp directory that macOS is free to purge. The
|
||||
v1/v2 account in the round-2 notes checks out: both matrices are 61,853 bytes with
|
||||
`ref_code` (429, 16), but different SHA-256 — same duration, different content,
|
||||
consistent with the `noisered 0.4` → `0.15` change.
|
||||
|
||||
## 2. The matrix can reproduce the WAV — proven, not assumed
|
||||
|
||||
`ref_code` is the codec-token matrix of the reference speech, and the codec's
|
||||
decoder is the exact inverse of the encoder that produced it:
|
||||
|
||||
```
|
||||
Qwen3TTSTokenizer.from_pretrained(
|
||||
models/Qwen3-TTS-12Hz-1.7B-Base/speech_tokenizer
|
||||
).decode({"audio_codes": ref_code})
|
||||
```
|
||||
|
||||
Result: **34.32 s @ 24 kHz, median F0 129.4 Hz** — the reference's own documented
|
||||
value is 129.4 Hz.
|
||||
|
||||
Fidelity, measured phase-insensitively (a 12 Hz codec does not preserve phase, so
|
||||
waveform correlation is meaningless here):
|
||||
|
||||
| Measure | Decoded from matrix | Original capture | Delta |
|
||||
|---|---|---|---|
|
||||
| median F0 | 129.4 Hz | 128.6 Hz | +0.8 Hz |
|
||||
| voiced frames | 1145 | 1132 | +1.2 % |
|
||||
| LTAS band deviation (80 Hz–12 kHz, 8 bands) | — | — | **0.55 dB RMS**, worst band +1.10 dB |
|
||||
|
||||
So the matrix alone is sufficient. The WAV is a derived artifact, not a
|
||||
prerequisite. `author_ref.wav` is now installed in the project **decoded from the
|
||||
matrix**, with a sidecar JSON recording the matrix's SHA-256 and the exact
|
||||
regeneration command.
|
||||
|
||||
### Provenance proven by re-encoding
|
||||
|
||||
Rather than trusting the notes, each candidate source was re-encoded and its
|
||||
`audio_codes` compared to the stored `ref_code`:
|
||||
|
||||
| Candidate | shape | exact match | frame agreement |
|
||||
|---|---|---|---|
|
||||
| `BDMurphy_Voice_NOBG_v2.wav` (documented v2 source) | (429, 16) | no | **0.432** |
|
||||
| `BDMurphy_Voice_NOBG.wav` (v1 source) | (429, 16) | no | 0.103 |
|
||||
| `BDMurphy_Voice.wav` (raw recording) | (430, 16) | no | shape mismatch |
|
||||
|
||||
43.2 % vs 10.3 % — 4.2× — so the live matrix provably came from the v2 file. (Not
|
||||
100 % because the codec encode is not bit-deterministic under MPS/bfloat16.)
|
||||
|
||||
**Note:** embedding cosine similarity is useless for speaker ID here — author vs
|
||||
nanny scores 0.933 and Kainda vs Xylia 0.976. All 2048-d embeddings sit in one
|
||||
tight cluster. Do not use it to test "is this the same person".
|
||||
|
||||
## 3. Voice-range lab — 44 samples on the Author voice
|
||||
|
||||
All samples use one fixed text so comparisons are valid. Every sample recorded F0,
|
||||
spectral-envelope centroid and its semitone offset, brightness, flatness, level and
|
||||
whisper WER.
|
||||
|
||||
### 3a. The neural instruct axis is nearly inert on a real human reference
|
||||
|
||||
| Instruct | ΔF0 | Δenvelope | WER |
|
||||
|---|---|---|---|
|
||||
| "a young woman's voice, bright and light" | **+1.3 st** | −2.7 st | 0.08 |
|
||||
| "a mature woman's voice, low and calm" | +0.9 st | −2.0 st | 0.00 |
|
||||
| "a young boy's voice, light and eager" | −0.3 st | −0.9 st | 0.08 |
|
||||
| "an elderly man's voice, deep, gravelly" | +0.6 st | −0.5 st | 0.00 |
|
||||
| "a very large man's voice, heavy and booming" | +1.0 st | −0.9 st | 0.00 |
|
||||
| "a small woman's voice, petite and light" | +0.8 st | +2.5 st | 0.16 |
|
||||
|
||||
**This is the headline and it is counterintuitive.** The identical instruct run
|
||||
against the *Narrator* TTS reference moved F0 by +6.9 st. Against the Author's own
|
||||
recorded voice it moves 1.3 st. A real human capture anchors the clone far harder
|
||||
than a synthesised reference, so the instruct has almost no leverage left.
|
||||
|
||||
Practical consequence: **the neural instruct is not a usable morph axis for the
|
||||
Author voice. DSP is the only axis that works.** That is good for brand
|
||||
consistency and bad for variety, and it is not fixable by better prompt wording.
|
||||
|
||||
### 3b. The DSP pitch axis is effectively one-directional
|
||||
|
||||
| shift | F0 | Δenvelope | WER |
|
||||
|---|---|---|---|
|
||||
| +12 st | 266.3 Hz | +6.1 st | **0.00** |
|
||||
| +10 st | 238.6 Hz | +4.8 st | **0.00** |
|
||||
| +8 st | 211.4 Hz | +3.7 st | **0.00** |
|
||||
| +6 st | 188.3 Hz | +2.3 st | 0.00 |
|
||||
| +4 st | 167.8 Hz | +1.5 st | 0.00 |
|
||||
| +2 st | 150.3 Hz | +0.4 st | 0.04 |
|
||||
| 0 | 133.9 Hz | −1.6 st | 0.04 |
|
||||
| −2 st | 118.6 Hz | −3.1 st | **0.64** |
|
||||
| −4 st | 104.5 Hz | −3.8 st | **0.80** |
|
||||
| −6 st | 93.6 Hz | −5.4 st | **1.00** |
|
||||
|
||||
Upward is free to +12 st (266 Hz — squarely female range, WER 0.00).
|
||||
**Downward collapses after 2 semitones.**
|
||||
|
||||
Re-tested on a 58-word text with 2 seeds, because a 25-word sample is too coarse
|
||||
to stake a claim on:
|
||||
|
||||
| shift | WER (seed A / seed B) |
|
||||
|---|---|
|
||||
| +2 st | 0.069 / 0.086 |
|
||||
| 0 | 0.069 / 0.069 |
|
||||
| **−2 st** | **0.724 / 0.845** |
|
||||
| −4 st | 0.983 / 0.948 |
|
||||
| −6 st | 1.000 / 0.897 |
|
||||
|
||||
Confirmed and reproducible. **Caveat worth stating plainly: WER measures machine
|
||||
recognition, not human comprehension.** A human adapts to a low, growl-like voice
|
||||
far better than whisper does. `wav_cliff/cliff-06_s7.wav` and `cliff-04_s7.wav`
|
||||
should be listened to before concluding the monster band is unusable for *speaking*
|
||||
monsters rather than roaring ones.
|
||||
|
||||
### 3c. How many distinct voices does one recording actually support?
|
||||
|
||||
15 samples of the *same* voice across 5 seeds × 3 temperatures, no DSP:
|
||||
|
||||
- F0 123.5–141.9 Hz → **+2.4 st total spread**, sd 0.76 st
|
||||
- envelope 763–930 Hz → 3.4 st spread, sd 0.78 st
|
||||
|
||||
So **re-rolling the seed alone moves the voice ~2.4 semitones.** Any variant closer
|
||||
than ~1.5 st (2 sd) to another is not reliably distinguishable from a re-roll — it
|
||||
is the same voice.
|
||||
|
||||
Applying both gates (separation > 1.5 st **and** WER ≤ 0.20) to the measured axis:
|
||||
|
||||
> **~7 reliably distinct, intelligible voices from this one recording, spanning
|
||||
> +2.6 to +12.5 semitones — every one of them above the Author's natural register.**
|
||||
|
||||
There is no room below. Adding the unintelligible-creature band the operator
|
||||
sanctioned takes the span to +0.6 … +12.5 st, still ~8 voices, because downward is
|
||||
the same one axis.
|
||||
|
||||
**This is the number that matters and it is the real constraint.** One source
|
||||
recording buys ~7–8 audibly distinct voices, all higher than the Author. To go
|
||||
further you need a genuinely independent second dimension — which brings us to the
|
||||
missing primitive below.
|
||||
|
||||
## 4. The missing primitive, re-scoped
|
||||
|
||||
There is **no independent formant shifter** in the toolchain. The obvious
|
||||
implementation — resample then time-stretch — was written and measured: **it is an
|
||||
identity** (ΔF0 +0.1 st, Δenvelope +0.1 st), because the two operations exactly
|
||||
cancel. Recorded here so nobody repeats it.
|
||||
|
||||
`pitch_shift_pv` is asymmetric, and measured on the Author voice:
|
||||
|
||||
- upward: formants move at **0.28** of the pitch shift
|
||||
- downward: formants move at **0.85** of it (overshoot)
|
||||
|
||||
The upward 0.28 ratio happens to match the real male→female vocal-tract ratio
|
||||
(0.29), which is why +8 to +12 st produces a convincing female register with no
|
||||
extra machinery. That is luck, not design.
|
||||
|
||||
**Revised recommendation.** My earlier advice was "don't build the formant shifter,
|
||||
the phase vocoder already covers it." That was right about male→female and wrong
|
||||
about cast size. With only one usable axis, the ceiling is ~7–8 voices. An
|
||||
independent formant shift adds a **second** dimension (tract × pitch), which is the
|
||||
only way to break that ceiling — e.g. keeping pitch fixed while changing vocal
|
||||
tract gives a family sharing one throat, or the reverse for unrelated characters at
|
||||
the same pitch. Options: `rubberband` (has a real `formantScale`),
|
||||
`pyworld` + envelope warping (pure Python, ~40 lines, mediocre quality on speech).
|
||||
|
||||
Worth doing only if a >8-voice cast is actually wanted. It is not needed for
|
||||
female voices or for the monster band.
|
||||
|
||||
## 5. Bug found and reported, not fixed
|
||||
|
||||
**32 of 97 reference WAVs in the DealHunter project are 3–4 second stubs**, not
|
||||
30–58 s Rainbow Passages — including `b.d._murphy_ref.wav` (3.48 s) and
|
||||
`kainda_ref.wav` (3.00 s), all dated Jul 31 22:37–22:39. `kainda_walters_foster_
|
||||
murphy_ref.wav` (the real one) exists separately, so these look like truncated
|
||||
writes that were never cleaned up. Any speaker resolving to a stub gets a 3-second
|
||||
reference. Not fixed — reported only, as it does not block the Author lab.
|
||||
|
||||
## 6. Reproducing
|
||||
|
||||
```bash
|
||||
cd /Volumes/1TB_APL_Storage/UmbraAudio
|
||||
.venv/bin/python backup/author_voice_lab/scripts/matrix_roundtrip.py # matrix -> wav
|
||||
.venv/bin/python backup/author_voice_lab/scripts/voice_lab.py # 44-sample sweep
|
||||
.venv/bin/python backup/author_voice_lab/scripts/verify_cliff.py # cliff re-test
|
||||
```
|
||||
|
||||
`voice_lab.py` reads `author_ref.wav` and writes only into this lab directory; it
|
||||
does not touch any project's audio.
|
||||
|
||||
### Samples worth listening to first
|
||||
|
||||
| File | What it is |
|
||||
|---|---|
|
||||
| `wav/A_pitch+08.wav` | +8.5 st, 211 Hz, WER 0.00 — convincing female register |
|
||||
| `wav/A_pitch+12.wav` | +12.5 st, 266 Hz, WER 0.00 — upper edge |
|
||||
| `wav/C_monster_grit_full.wav` | dragon instruct + sat/ring/chorus/jitter, WER 0.12 |
|
||||
| `wav/B_female_young.wav` | the instruct that barely moved — the key negative result |
|
||||
| `wav_cliff/cliff-04_s7.wav` | where downward shifting breaks |
|
||||
| `wav_cliff/cliff+02_s7.wav` | control, free upward |
|
||||
85
backup/author_voice_lab/cliff_verification.json
Normal file
85
backup/author_voice_lab/cliff_verification.json
Normal file
@@ -0,0 +1,85 @@
|
||||
{
|
||||
"text_words": 58,
|
||||
"rows": [
|
||||
{
|
||||
"shift_st": 0,
|
||||
"seed": 7,
|
||||
"wer": 0.069,
|
||||
"f0": 130.1,
|
||||
"hyp": "e set the lantern on the stone ledge and listened he cavern breathed around them slow and patient and some were far below the water was moving against rock ary nn kept one hand on the rail and waited for him to speak first because she had learned that what he said next would decide which of them went forward",
|
||||
"sec": 32.0
|
||||
},
|
||||
{
|
||||
"shift_st": 0,
|
||||
"seed": 8,
|
||||
"wer": 0.069,
|
||||
"f0": 127.9,
|
||||
"hyp": "e set the lantern on the stone ledge and listened he cavern breathed around them slow and patient and some were far below the water was moving against rock ary nn kept one hand on the rail and waited for him to speak first because she had learned that what he said next would decide which of them went forward",
|
||||
"sec": 27.6
|
||||
},
|
||||
{
|
||||
"shift_st": -2,
|
||||
"seed": 7,
|
||||
"wer": 0.724,
|
||||
"f0": 116.6,
|
||||
"hyp": "e said who turn on the stone why should listen he cabin breaks and ll move so in person and so will the fabule of water was moving and swung am the one who am all unworthy for and to speak first because you are the one who sent me to decide which is the one for",
|
||||
"sec": 31.1
|
||||
},
|
||||
{
|
||||
"shift_st": -2,
|
||||
"seed": 8,
|
||||
"wer": 0.845,
|
||||
"f0": 114.6,
|
||||
"hyp": "es a goodass honest oldass man listen es having great trouble slow impression and some of the fog in the water was moving as well ow he ends up working on the water while he finds his big face es a shamed man that wont be so nascult to side but should go more forward",
|
||||
"sec": 28.6
|
||||
},
|
||||
{
|
||||
"shift_st": -4,
|
||||
"seed": 7,
|
||||
"wer": 0.983,
|
||||
"f0": 103.3,
|
||||
"hyp": "he circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of the circle of",
|
||||
"sec": 33.5
|
||||
},
|
||||
{
|
||||
"shift_st": -4,
|
||||
"seed": 8,
|
||||
"wer": 0.948,
|
||||
"f0": 102.1,
|
||||
"hyp": "dont know if was going to have a solution dont know if could have a solution dont know if could have a solution dont know how would have a solution dont know if could have a solution",
|
||||
"sec": 27.8
|
||||
},
|
||||
{
|
||||
"shift_st": -6,
|
||||
"seed": 7,
|
||||
"wer": 1.0,
|
||||
"f0": 92.0,
|
||||
"hyp": "hantana chokyusha naka shing aba kawa kawa koshia naka shing hantana chokyusha naka shing awa kawa kawa koshia naka shing hantana chokyusha naka shing",
|
||||
"sec": 31.9
|
||||
},
|
||||
{
|
||||
"shift_st": -6,
|
||||
"seed": 8,
|
||||
"wer": 0.897,
|
||||
"f0": 90.9,
|
||||
"hyp": "have a lot of work to do with the rest of my life have a lot of work to do with the rest of my life have a lot of work to do with the rest of my life have a lot of work to do with the rest of my life have a lot of work to do with the rest of my life",
|
||||
"sec": 27.4
|
||||
},
|
||||
{
|
||||
"shift_st": 2,
|
||||
"seed": 7,
|
||||
"wer": 0.069,
|
||||
"f0": 146.0,
|
||||
"hyp": "e said the lantern on the stone ledge and listened he cavern breathed around them slow and patient and somewhere far below the water was moving against rock ary nn kept one hand on the rail and waited for him to speak first because she had learned that what he said next would decide which of them went forward",
|
||||
"sec": 32.2
|
||||
},
|
||||
{
|
||||
"shift_st": 2,
|
||||
"seed": 8,
|
||||
"wer": 0.086,
|
||||
"f0": 144.4,
|
||||
"hyp": "e set the lantern on the stern ledge and listened he cavern breathed around them slow and patient and somewhere far below the water was moving against rock ary and kept one hand on the rail and waited for him to speak first because she had learned that what he said next would decide which of them d forward",
|
||||
"sec": 27.7
|
||||
}
|
||||
]
|
||||
}
|
||||
873
backup/author_voice_lab/results.json
Normal file
873
backup/author_voice_lab/results.json
Normal file
@@ -0,0 +1,873 @@
|
||||
{
|
||||
"reference": {
|
||||
"f0_median": 129.4,
|
||||
"envelope_hz": 894,
|
||||
"path": "/Volumes/1TB_APL_Storage/UmbraAudio/DealHunterShip_4.0e_whole-to-audio/.umbra/voices/author_ref.wav"
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"f0_median": 133.9,
|
||||
"f0_sd": 15.1,
|
||||
"envelope_hz": 814,
|
||||
"centroid_hz": 1597,
|
||||
"flatness": 0.04796,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -23.0,
|
||||
"peak_db": -7.5,
|
||||
"d_f0_st": 0.6,
|
||||
"label": "A00_baseline",
|
||||
"gen_s": 21.0,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": null,
|
||||
"d_env_st": -1.6,
|
||||
"wer": 0.04
|
||||
},
|
||||
{
|
||||
"f0_median": 65.8,
|
||||
"f0_sd": 6.4,
|
||||
"envelope_hz": 488,
|
||||
"centroid_hz": 718,
|
||||
"flatness": 0.00187,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -26.9,
|
||||
"peak_db": -10.5,
|
||||
"d_f0_st": -11.7,
|
||||
"label": "A_pitch-12",
|
||||
"gen_s": 12.8,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": -10.5,
|
||||
"wer": 1.0
|
||||
},
|
||||
{
|
||||
"f0_median": 75.2,
|
||||
"f0_sd": 7.6,
|
||||
"envelope_hz": 528,
|
||||
"centroid_hz": 809,
|
||||
"flatness": 0.00264,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -26.7,
|
||||
"peak_db": -9.6,
|
||||
"d_f0_st": -9.4,
|
||||
"label": "A_pitch-10",
|
||||
"gen_s": 12.8,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": -9.1,
|
||||
"wer": 1.0
|
||||
},
|
||||
{
|
||||
"f0_median": 82.7,
|
||||
"f0_sd": 45.6,
|
||||
"envelope_hz": 578,
|
||||
"centroid_hz": 920,
|
||||
"flatness": 0.00423,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -26.3,
|
||||
"peak_db": -10.8,
|
||||
"d_f0_st": -7.7,
|
||||
"label": "A_pitch-08",
|
||||
"gen_s": 12.7,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": -7.6,
|
||||
"wer": 1.0
|
||||
},
|
||||
{
|
||||
"f0_median": 93.6,
|
||||
"f0_sd": 10.8,
|
||||
"envelope_hz": 654,
|
||||
"centroid_hz": 1049,
|
||||
"flatness": 0.00618,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -27.1,
|
||||
"peak_db": -10.2,
|
||||
"d_f0_st": -5.6,
|
||||
"label": "A_pitch-06",
|
||||
"gen_s": 12.8,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": -5.4,
|
||||
"wer": 1.0
|
||||
},
|
||||
{
|
||||
"f0_median": 104.5,
|
||||
"f0_sd": 58.7,
|
||||
"envelope_hz": 719,
|
||||
"centroid_hz": 1182,
|
||||
"flatness": 0.01017,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -26.6,
|
||||
"peak_db": -9.1,
|
||||
"d_f0_st": -3.7,
|
||||
"label": "A_pitch-04",
|
||||
"gen_s": 12.8,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": -3.8,
|
||||
"wer": 0.8
|
||||
},
|
||||
{
|
||||
"f0_median": 118.6,
|
||||
"f0_sd": 12.4,
|
||||
"envelope_hz": 749,
|
||||
"centroid_hz": 1327,
|
||||
"flatness": 0.01774,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -26.6,
|
||||
"peak_db": -9.9,
|
||||
"d_f0_st": -1.5,
|
||||
"label": "A_pitch-02",
|
||||
"gen_s": 12.9,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": -3.1,
|
||||
"wer": 0.64
|
||||
},
|
||||
{
|
||||
"f0_median": 133.9,
|
||||
"f0_sd": 15.1,
|
||||
"envelope_hz": 814,
|
||||
"centroid_hz": 1597,
|
||||
"flatness": 0.04796,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -23.0,
|
||||
"peak_db": -7.5,
|
||||
"d_f0_st": 0.6,
|
||||
"label": "A_pitch+00",
|
||||
"gen_s": 13.4,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": -1.6,
|
||||
"wer": 0.04
|
||||
},
|
||||
{
|
||||
"f0_median": 150.3,
|
||||
"f0_sd": 16.0,
|
||||
"envelope_hz": 913,
|
||||
"centroid_hz": 1606,
|
||||
"flatness": 0.03015,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -26.9,
|
||||
"peak_db": -10.5,
|
||||
"d_f0_st": 2.6,
|
||||
"label": "A_pitch+02",
|
||||
"gen_s": 12.8,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": 0.4,
|
||||
"wer": 0.04
|
||||
},
|
||||
{
|
||||
"f0_median": 167.8,
|
||||
"f0_sd": 18.1,
|
||||
"envelope_hz": 977,
|
||||
"centroid_hz": 1718,
|
||||
"flatness": 0.03189,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -26.1,
|
||||
"peak_db": -9.8,
|
||||
"d_f0_st": 4.5,
|
||||
"label": "A_pitch+04",
|
||||
"gen_s": 12.9,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": 1.5,
|
||||
"wer": 0.0
|
||||
},
|
||||
{
|
||||
"f0_median": 188.3,
|
||||
"f0_sd": 21.8,
|
||||
"envelope_hz": 1021,
|
||||
"centroid_hz": 1837,
|
||||
"flatness": 0.02948,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -26.6,
|
||||
"peak_db": -9.4,
|
||||
"d_f0_st": 6.5,
|
||||
"label": "A_pitch+06",
|
||||
"gen_s": 13.0,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": 2.3,
|
||||
"wer": 0.0
|
||||
},
|
||||
{
|
||||
"f0_median": 211.4,
|
||||
"f0_sd": 23.4,
|
||||
"envelope_hz": 1109,
|
||||
"centroid_hz": 1946,
|
||||
"flatness": 0.02749,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -26.4,
|
||||
"peak_db": -9.8,
|
||||
"d_f0_st": 8.5,
|
||||
"label": "A_pitch+08",
|
||||
"gen_s": 13.1,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": 3.7,
|
||||
"wer": 0.0
|
||||
},
|
||||
{
|
||||
"f0_median": 238.6,
|
||||
"f0_sd": 27.5,
|
||||
"envelope_hz": 1178,
|
||||
"centroid_hz": 2086,
|
||||
"flatness": 0.0298,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -25.7,
|
||||
"peak_db": -7.3,
|
||||
"d_f0_st": 10.6,
|
||||
"label": "A_pitch+10",
|
||||
"gen_s": 13.1,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": 4.8,
|
||||
"wer": 0.0
|
||||
},
|
||||
{
|
||||
"f0_median": 266.3,
|
||||
"f0_sd": 31.5,
|
||||
"envelope_hz": 1269,
|
||||
"centroid_hz": 2258,
|
||||
"flatness": 0.02758,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -25.3,
|
||||
"peak_db": -9.0,
|
||||
"d_f0_st": 12.5,
|
||||
"label": "A_pitch+12",
|
||||
"gen_s": 13.0,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": 6.1,
|
||||
"wer": 0.0
|
||||
},
|
||||
{
|
||||
"f0_median": 139.8,
|
||||
"f0_sd": 17.7,
|
||||
"envelope_hz": 766,
|
||||
"centroid_hz": 1527,
|
||||
"flatness": 0.0408,
|
||||
"dur_s": 8.24,
|
||||
"rms_db": -23.8,
|
||||
"peak_db": -8.1,
|
||||
"d_f0_st": 1.3,
|
||||
"label": "B_female_young",
|
||||
"gen_s": 13.4,
|
||||
"instruct": "A young woman's voice, bright and light, with a higher and clearer tone.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": null,
|
||||
"d_env_st": -2.7,
|
||||
"wer": 0.08
|
||||
},
|
||||
{
|
||||
"f0_median": 136.3,
|
||||
"f0_sd": 18.9,
|
||||
"envelope_hz": 796,
|
||||
"centroid_hz": 1505,
|
||||
"flatness": 0.03024,
|
||||
"dur_s": 7.76,
|
||||
"rms_db": -22.1,
|
||||
"peak_db": -4.8,
|
||||
"d_f0_st": 0.9,
|
||||
"label": "B_female_mature",
|
||||
"gen_s": 12.9,
|
||||
"instruct": "A mature woman's voice, low and calm for a woman, unhurried and steady.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": null,
|
||||
"d_env_st": -2.0,
|
||||
"wer": 0.0
|
||||
},
|
||||
{
|
||||
"f0_median": 127.1,
|
||||
"f0_sd": 12.4,
|
||||
"envelope_hz": 849,
|
||||
"centroid_hz": 1583,
|
||||
"flatness": 0.03735,
|
||||
"dur_s": 7.44,
|
||||
"rms_db": -24.8,
|
||||
"peak_db": -7.1,
|
||||
"d_f0_st": -0.3,
|
||||
"label": "B_boy",
|
||||
"gen_s": 11.7,
|
||||
"instruct": "A young boy's voice, light and eager, not yet broken.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": null,
|
||||
"d_env_st": -0.9,
|
||||
"wer": 0.08
|
||||
},
|
||||
{
|
||||
"f0_median": 133.9,
|
||||
"f0_sd": 28.1,
|
||||
"envelope_hz": 868,
|
||||
"centroid_hz": 1592,
|
||||
"flatness": 0.04646,
|
||||
"dur_s": 7.04,
|
||||
"rms_db": -22.3,
|
||||
"peak_db": -6.1,
|
||||
"d_f0_st": 0.6,
|
||||
"label": "B_elderly_m",
|
||||
"gen_s": 11.1,
|
||||
"instruct": "An elderly man's voice, very deep, gravelly and slow.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": null,
|
||||
"d_env_st": -0.5,
|
||||
"wer": 0.0
|
||||
},
|
||||
{
|
||||
"f0_median": 133.9,
|
||||
"f0_sd": 13.9,
|
||||
"envelope_hz": 766,
|
||||
"centroid_hz": 1621,
|
||||
"flatness": 0.04052,
|
||||
"dur_s": 8.08,
|
||||
"rms_db": -22.0,
|
||||
"peak_db": -4.9,
|
||||
"d_f0_st": 0.6,
|
||||
"label": "B_young_m",
|
||||
"gen_s": 12.3,
|
||||
"instruct": "A young man's voice, light, energetic and clear.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": null,
|
||||
"d_env_st": -2.7,
|
||||
"wer": 0.0
|
||||
},
|
||||
{
|
||||
"f0_median": 137.0,
|
||||
"f0_sd": 17.7,
|
||||
"envelope_hz": 851,
|
||||
"centroid_hz": 1748,
|
||||
"flatness": 0.05657,
|
||||
"dur_s": 8.08,
|
||||
"rms_db": -25.4,
|
||||
"peak_db": -8.4,
|
||||
"d_f0_st": 1.0,
|
||||
"label": "B_large_m",
|
||||
"gen_s": 12.2,
|
||||
"instruct": "A very large man's voice, heavy and booming, with a deep chest.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": null,
|
||||
"d_env_st": -0.9,
|
||||
"wer": 0.0
|
||||
},
|
||||
{
|
||||
"f0_median": 135.5,
|
||||
"f0_sd": 19.4,
|
||||
"envelope_hz": 1034,
|
||||
"centroid_hz": 1614,
|
||||
"flatness": 0.03173,
|
||||
"dur_s": 7.52,
|
||||
"rms_db": -23.0,
|
||||
"peak_db": -5.4,
|
||||
"d_f0_st": 0.8,
|
||||
"label": "B_small_w",
|
||||
"gen_s": 11.3,
|
||||
"instruct": "A small woman's voice, petite and light, with a narrow bright tone.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": null,
|
||||
"d_env_st": 2.5,
|
||||
"wer": 0.16
|
||||
},
|
||||
{
|
||||
"f0_median": 131.6,
|
||||
"f0_sd": 28.1,
|
||||
"envelope_hz": 779,
|
||||
"centroid_hz": 1464,
|
||||
"flatness": 0.03531,
|
||||
"dur_s": 7.92,
|
||||
"rms_db": -22.5,
|
||||
"peak_db": -5.3,
|
||||
"d_f0_st": 0.3,
|
||||
"label": "B_androgynous",
|
||||
"gen_s": 11.9,
|
||||
"instruct": "A neutral, androgynous voice, neither distinctly male nor female.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": null,
|
||||
"d_env_st": -2.4,
|
||||
"wer": 0.04
|
||||
},
|
||||
{
|
||||
"f0_median": 177.7,
|
||||
"f0_sd": 22.7,
|
||||
"envelope_hz": 926,
|
||||
"centroid_hz": 1635,
|
||||
"flatness": 0.02632,
|
||||
"dur_s": 8.24,
|
||||
"rms_db": -27.2,
|
||||
"peak_db": -11.7,
|
||||
"d_f0_st": 5.5,
|
||||
"label": "C_female+4st",
|
||||
"gen_s": 12.3,
|
||||
"instruct": "A young woman's voice, bright and light, with a higher and clearer tone.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": 0.6,
|
||||
"wer": 0.04
|
||||
},
|
||||
{
|
||||
"f0_median": 226.5,
|
||||
"f0_sd": 27.6,
|
||||
"envelope_hz": 1106,
|
||||
"centroid_hz": 1864,
|
||||
"flatness": 0.02673,
|
||||
"dur_s": 8.24,
|
||||
"rms_db": -27.2,
|
||||
"peak_db": -11.3,
|
||||
"d_f0_st": 9.7,
|
||||
"label": "C_female+8st",
|
||||
"gen_s": 12.2,
|
||||
"instruct": "A young woman's voice, bright and light, with a higher and clearer tone.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": 3.7,
|
||||
"wer": 0.0
|
||||
},
|
||||
{
|
||||
"f0_median": 75.2,
|
||||
"f0_sd": 7.6,
|
||||
"envelope_hz": 528,
|
||||
"centroid_hz": 809,
|
||||
"flatness": 0.00264,
|
||||
"dur_s": 8.0,
|
||||
"rms_db": -26.7,
|
||||
"peak_db": -9.6,
|
||||
"d_f0_st": -9.4,
|
||||
"label": "C_monster_pitch-10",
|
||||
"gen_s": 12.0,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"pitch_shift_pv"
|
||||
],
|
||||
"d_env_st": -9.1,
|
||||
"wer": 1.0
|
||||
},
|
||||
{
|
||||
"f0_median": 132.0,
|
||||
"f0_sd": 16.9,
|
||||
"envelope_hz": 837,
|
||||
"centroid_hz": 1525,
|
||||
"flatness": 0.0437,
|
||||
"dur_s": 8.4,
|
||||
"rms_db": -13.6,
|
||||
"peak_db": -0.5,
|
||||
"d_f0_st": 0.4,
|
||||
"label": "C_monster_grit_sat",
|
||||
"gen_s": 12.6,
|
||||
"instruct": "A huge ancient dragon: deep, rumbling, growling, with a metallic edge and a resonant echoing chest tone.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"tube_sat"
|
||||
],
|
||||
"d_env_st": -1.1,
|
||||
"wer": 0.12
|
||||
},
|
||||
{
|
||||
"f0_median": 74.3,
|
||||
"f0_sd": 6.9,
|
||||
"envelope_hz": 758,
|
||||
"centroid_hz": 1463,
|
||||
"flatness": 0.04517,
|
||||
"dur_s": 8.4,
|
||||
"rms_db": -27.7,
|
||||
"peak_db": -7.4,
|
||||
"d_f0_st": -9.6,
|
||||
"label": "C_monster_grit_ring",
|
||||
"gen_s": 12.4,
|
||||
"instruct": "A huge ancient dragon: deep, rumbling, growling, with a metallic edge and a resonant echoing chest tone.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"ring_modulate"
|
||||
],
|
||||
"d_env_st": -2.9,
|
||||
"wer": 0.12
|
||||
},
|
||||
{
|
||||
"f0_median": 132.4,
|
||||
"f0_sd": 17.8,
|
||||
"envelope_hz": 827,
|
||||
"centroid_hz": 1476,
|
||||
"flatness": 0.03293,
|
||||
"dur_s": 8.4,
|
||||
"rms_db": -28.3,
|
||||
"peak_db": -9.7,
|
||||
"d_f0_st": 0.4,
|
||||
"label": "C_monster_grit_chorus",
|
||||
"gen_s": 12.4,
|
||||
"instruct": "A huge ancient dragon: deep, rumbling, growling, with a metallic edge and a resonant echoing chest tone.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"chorus"
|
||||
],
|
||||
"d_env_st": -1.4,
|
||||
"wer": 0.12
|
||||
},
|
||||
{
|
||||
"f0_median": 69.7,
|
||||
"f0_sd": 3.3,
|
||||
"envelope_hz": 817,
|
||||
"centroid_hz": 1575,
|
||||
"flatness": 0.04212,
|
||||
"dur_s": 8.4,
|
||||
"rms_db": -20.1,
|
||||
"peak_db": -3.2,
|
||||
"d_f0_st": -10.7,
|
||||
"label": "C_monster_grit_full",
|
||||
"gen_s": 12.6,
|
||||
"instruct": "A huge ancient dragon: deep, rumbling, growling, with a metallic edge and a resonant echoing chest tone.",
|
||||
"temperature": 0.7,
|
||||
"seed": 1234,
|
||||
"dsp": [
|
||||
"tube_sat",
|
||||
"ring_modulate",
|
||||
"chorus",
|
||||
"pitch_jitter"
|
||||
],
|
||||
"d_env_st": -1.6,
|
||||
"wer": 0.04
|
||||
},
|
||||
{
|
||||
"f0_median": 127.1,
|
||||
"f0_sd": 13.3,
|
||||
"envelope_hz": 859,
|
||||
"centroid_hz": 1568,
|
||||
"flatness": 0.03092,
|
||||
"dur_s": 7.92,
|
||||
"rms_db": -23.1,
|
||||
"peak_db": -6.0,
|
||||
"d_f0_st": -0.3,
|
||||
"label": "D_seed11_t0.3",
|
||||
"gen_s": 11.8,
|
||||
"instruct": null,
|
||||
"temperature": 0.3,
|
||||
"seed": 11,
|
||||
"dsp": null,
|
||||
"d_env_st": -0.7
|
||||
},
|
||||
{
|
||||
"f0_median": 133.9,
|
||||
"f0_sd": 54.6,
|
||||
"envelope_hz": 930,
|
||||
"centroid_hz": 1445,
|
||||
"flatness": 0.01365,
|
||||
"dur_s": 7.92,
|
||||
"rms_db": -22.9,
|
||||
"peak_db": -5.8,
|
||||
"d_f0_st": 0.6,
|
||||
"label": "D_seed22_t0.3",
|
||||
"gen_s": 11.7,
|
||||
"instruct": null,
|
||||
"temperature": 0.3,
|
||||
"seed": 22,
|
||||
"dsp": null,
|
||||
"d_env_st": 0.7
|
||||
},
|
||||
{
|
||||
"f0_median": 137.8,
|
||||
"f0_sd": 14.1,
|
||||
"envelope_hz": 776,
|
||||
"centroid_hz": 1431,
|
||||
"flatness": 0.02802,
|
||||
"dur_s": 7.92,
|
||||
"rms_db": -22.2,
|
||||
"peak_db": -5.8,
|
||||
"d_f0_st": 1.1,
|
||||
"label": "D_seed33_t0.3",
|
||||
"gen_s": 11.8,
|
||||
"instruct": null,
|
||||
"temperature": 0.3,
|
||||
"seed": 33,
|
||||
"dsp": null,
|
||||
"d_env_st": -2.5
|
||||
},
|
||||
{
|
||||
"f0_median": 123.5,
|
||||
"f0_sd": 17.4,
|
||||
"envelope_hz": 847,
|
||||
"centroid_hz": 1590,
|
||||
"flatness": 0.03914,
|
||||
"dur_s": 8.16,
|
||||
"rms_db": -24.9,
|
||||
"peak_db": -5.6,
|
||||
"d_f0_st": -0.8,
|
||||
"label": "D_seed44_t0.3",
|
||||
"gen_s": 12.3,
|
||||
"instruct": null,
|
||||
"temperature": 0.3,
|
||||
"seed": 44,
|
||||
"dsp": null,
|
||||
"d_env_st": -0.9
|
||||
},
|
||||
{
|
||||
"f0_median": 134.7,
|
||||
"f0_sd": 21.3,
|
||||
"envelope_hz": 836,
|
||||
"centroid_hz": 1599,
|
||||
"flatness": 0.04181,
|
||||
"dur_s": 8.16,
|
||||
"rms_db": -22.9,
|
||||
"peak_db": -5.4,
|
||||
"d_f0_st": 0.7,
|
||||
"label": "D_seed55_t0.3",
|
||||
"gen_s": 12.1,
|
||||
"instruct": null,
|
||||
"temperature": 0.3,
|
||||
"seed": 55,
|
||||
"dsp": null,
|
||||
"d_env_st": -1.2
|
||||
},
|
||||
{
|
||||
"f0_median": 127.1,
|
||||
"f0_sd": 15.7,
|
||||
"envelope_hz": 832,
|
||||
"centroid_hz": 1519,
|
||||
"flatness": 0.02939,
|
||||
"dur_s": 7.84,
|
||||
"rms_db": -22.7,
|
||||
"peak_db": -5.2,
|
||||
"d_f0_st": -0.3,
|
||||
"label": "D_seed11_t0.7",
|
||||
"gen_s": 11.7,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 11,
|
||||
"dsp": null,
|
||||
"d_env_st": -1.2
|
||||
},
|
||||
{
|
||||
"f0_median": 133.1,
|
||||
"f0_sd": 56.2,
|
||||
"envelope_hz": 836,
|
||||
"centroid_hz": 1366,
|
||||
"flatness": 0.00919,
|
||||
"dur_s": 7.68,
|
||||
"rms_db": -22.9,
|
||||
"peak_db": -6.7,
|
||||
"d_f0_st": 0.5,
|
||||
"label": "D_seed22_t0.7",
|
||||
"gen_s": 11.6,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 22,
|
||||
"dsp": null,
|
||||
"d_env_st": -1.2
|
||||
},
|
||||
{
|
||||
"f0_median": 123.5,
|
||||
"f0_sd": 11.8,
|
||||
"envelope_hz": 763,
|
||||
"centroid_hz": 1541,
|
||||
"flatness": 0.04872,
|
||||
"dur_s": 7.36,
|
||||
"rms_db": -23.7,
|
||||
"peak_db": -7.0,
|
||||
"d_f0_st": -0.8,
|
||||
"label": "D_seed33_t0.7",
|
||||
"gen_s": 12.0,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 33,
|
||||
"dsp": null,
|
||||
"d_env_st": -2.7
|
||||
},
|
||||
{
|
||||
"f0_median": 125.0,
|
||||
"f0_sd": 18.9,
|
||||
"envelope_hz": 818,
|
||||
"centroid_hz": 1370,
|
||||
"flatness": 0.02523,
|
||||
"dur_s": 7.04,
|
||||
"rms_db": -26.0,
|
||||
"peak_db": -8.1,
|
||||
"d_f0_st": -0.6,
|
||||
"label": "D_seed44_t0.7",
|
||||
"gen_s": 11.8,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 44,
|
||||
"dsp": null,
|
||||
"d_env_st": -1.5
|
||||
},
|
||||
{
|
||||
"f0_median": 133.1,
|
||||
"f0_sd": 17.9,
|
||||
"envelope_hz": 788,
|
||||
"centroid_hz": 1491,
|
||||
"flatness": 0.0285,
|
||||
"dur_s": 7.84,
|
||||
"rms_db": -20.9,
|
||||
"peak_db": -3.9,
|
||||
"d_f0_st": 0.5,
|
||||
"label": "D_seed55_t0.7",
|
||||
"gen_s": 13.0,
|
||||
"instruct": null,
|
||||
"temperature": 0.7,
|
||||
"seed": 55,
|
||||
"dsp": null,
|
||||
"d_env_st": -2.2
|
||||
},
|
||||
{
|
||||
"f0_median": 123.5,
|
||||
"f0_sd": 12.3,
|
||||
"envelope_hz": 801,
|
||||
"centroid_hz": 1500,
|
||||
"flatness": 0.03422,
|
||||
"dur_s": 7.84,
|
||||
"rms_db": -22.7,
|
||||
"peak_db": -5.9,
|
||||
"d_f0_st": -0.8,
|
||||
"label": "D_seed11_t0.9",
|
||||
"gen_s": 13.2,
|
||||
"instruct": null,
|
||||
"temperature": 0.9,
|
||||
"seed": 11,
|
||||
"dsp": null,
|
||||
"d_env_st": -1.9
|
||||
},
|
||||
{
|
||||
"f0_median": 141.9,
|
||||
"f0_sd": 56.3,
|
||||
"envelope_hz": 822,
|
||||
"centroid_hz": 1390,
|
||||
"flatness": 0.00808,
|
||||
"dur_s": 7.92,
|
||||
"rms_db": -21.9,
|
||||
"peak_db": -3.9,
|
||||
"d_f0_st": 1.6,
|
||||
"label": "D_seed22_t0.9",
|
||||
"gen_s": 12.8,
|
||||
"instruct": null,
|
||||
"temperature": 0.9,
|
||||
"seed": 22,
|
||||
"dsp": null,
|
||||
"d_env_st": -1.5
|
||||
},
|
||||
{
|
||||
"f0_median": 136.7,
|
||||
"f0_sd": 22.6,
|
||||
"envelope_hz": 779,
|
||||
"centroid_hz": 1452,
|
||||
"flatness": 0.03224,
|
||||
"dur_s": 6.88,
|
||||
"rms_db": -22.4,
|
||||
"peak_db": -6.5,
|
||||
"d_f0_st": 1.0,
|
||||
"label": "D_seed33_t0.9",
|
||||
"gen_s": 10.4,
|
||||
"instruct": null,
|
||||
"temperature": 0.9,
|
||||
"seed": 33,
|
||||
"dsp": null,
|
||||
"d_env_st": -2.4
|
||||
},
|
||||
{
|
||||
"f0_median": 135.5,
|
||||
"f0_sd": 23.9,
|
||||
"envelope_hz": 846,
|
||||
"centroid_hz": 1564,
|
||||
"flatness": 0.03484,
|
||||
"dur_s": 9.12,
|
||||
"rms_db": -22.2,
|
||||
"peak_db": -3.7,
|
||||
"d_f0_st": 0.8,
|
||||
"label": "D_seed44_t0.9",
|
||||
"gen_s": 13.5,
|
||||
"instruct": null,
|
||||
"temperature": 0.9,
|
||||
"seed": 44,
|
||||
"dsp": null,
|
||||
"d_env_st": -1.0
|
||||
},
|
||||
{
|
||||
"f0_median": 128.6,
|
||||
"f0_sd": 18.5,
|
||||
"envelope_hz": 801,
|
||||
"centroid_hz": 1598,
|
||||
"flatness": 0.03799,
|
||||
"dur_s": 7.68,
|
||||
"rms_db": -22.7,
|
||||
"peak_db": -5.8,
|
||||
"d_f0_st": -0.1,
|
||||
"label": "D_seed55_t0.9",
|
||||
"gen_s": 11.3,
|
||||
"instruct": null,
|
||||
"temperature": 0.9,
|
||||
"seed": 55,
|
||||
"dsp": null,
|
||||
"d_env_st": -1.9
|
||||
}
|
||||
]
|
||||
}
|
||||
91
backup/author_voice_lab/scripts/analyse_lab.py
Normal file
91
backup/author_voice_lab/scripts/analyse_lab.py
Normal file
@@ -0,0 +1,91 @@
|
||||
"""Analyse the lab: variance vs separation, and verify a suspicious F0 reading."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import librosa
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
|
||||
LAB = Path("/private/var/folders/y8/cbvhds297918lrxt2wd3btsr0000gn/T/opencode/author_lab")
|
||||
res = json.loads((LAB / "results.json").read_text())
|
||||
rows = res["rows"]
|
||||
REF_F0 = res["reference"]["f0_median"]
|
||||
REF_ENV = res["reference"]["envelope_hz"]
|
||||
print(f"reference: F0 {REF_F0} Hz envelope {REF_ENV} Hz\n")
|
||||
|
||||
|
||||
def st(v, base):
|
||||
return 12 * np.log2(v / base) if v and base else float("nan")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 1. variance
|
||||
print("=== INTRA-VOICE VARIANCE (same voice, different seed/temperature) ===")
|
||||
D = [r for r in rows if r["label"].startswith("D_")]
|
||||
f0s = np.array([r["f0_median"] for r in D], dtype=float)
|
||||
envs = np.array([r["envelope_hz"] for r in D], dtype=float)
|
||||
print(f"n={len(D)} seeds 11-55 x temp 0.3/0.7/0.9")
|
||||
print(f" F0 {f0s.min():.1f} - {f0s.max():.1f} Hz spread {f0s.max()-f0s.min():.1f} Hz "
|
||||
f"= {st(f0s.max(), REF_F0)-st(f0s.min(), REF_F0):+.1f} st sd {f0s.std():.1f} Hz")
|
||||
print(f" envelope {envs.min()} - {envs.max()} Hz spread "
|
||||
f"{st(envs.max(), REF_ENV)-st(envs.min(), REF_ENV):+.1f} st sd {envs.std():.0f} Hz")
|
||||
F0_SD_ST = f0s.std() / REF_F0 * 12 * np.log2(np.e) # semitone sd
|
||||
ENV_SD_ST = envs.std() / REF_ENV * 12 * np.log2(np.e)
|
||||
print(f" => 1-sd noise floor: F0 {F0_SD_ST:.2f} st, envelope {ENV_SD_ST:.2f} st")
|
||||
print(f" => 2-sd (where a variant becomes reliably 'not just a re-roll'): "
|
||||
f"F0 {2*F0_SD_ST:.2f} st")
|
||||
|
||||
# ---------------------------------------------------------------- 2. separation
|
||||
print("\n=== AXIS B: did the neural instruct move anything on a REAL HUMAN reference? ===")
|
||||
print(f"{'variant':20} {'d_F0_st':>9} {'d_env_st':>10} {'WER':>6}")
|
||||
for r in rows:
|
||||
if r["label"].startswith("B_") or r["label"].startswith("A00"):
|
||||
print(f"{r['label']:20} {r.get('d_f0_st'):>9} {r.get('d_env_st'):>10} {r.get('wer'):>6}")
|
||||
|
||||
# ---------------------------------------------------------------- 3. verify F0 artifact
|
||||
print("\n=== VERIFYING C_monster_grit_full (F0 69.7 Hz but WER 0.04) ===")
|
||||
|
||||
|
||||
def true_f0(y, sr):
|
||||
"""Autocorrelation F0 over 60-400 Hz - independent of pyin's tracking."""
|
||||
y = y.astype(np.float32)
|
||||
f0, _, _ = librosa.pyin(y, fmin=60.0, fmax=400.0, sr=sr, frame_length=1024)
|
||||
v = f0[~np.isnan(f0)]
|
||||
ac = librosa.autocorrelate(y, max_size=int(sr / 60))
|
||||
lo, hi = int(sr / 400), int(sr / 60)
|
||||
peak = lo + int(np.argmax(ac[lo:hi]))
|
||||
return float(np.median(v)), sr / peak
|
||||
|
||||
|
||||
for label in ("A00_baseline", "C_monster_grit_ring", "C_monster_grit_full",
|
||||
"C_monster_pitch-10"):
|
||||
p = LAB / "wav" / f"{label}.wav"
|
||||
if not p.exists():
|
||||
continue
|
||||
y, sr = sf.read(str(p), dtype="float32")
|
||||
if y.ndim > 1:
|
||||
y = y.mean(axis=1)
|
||||
pyin_f0, ac_f0 = true_f0(y, sr)
|
||||
# strong spectral peaks
|
||||
S = np.abs(librosa.stft(y, n_fft=4096, hop_length=1024))
|
||||
f = librosa.fft_frequencies(sr=sr, n_fft=4096)
|
||||
m = (f > 40) & (f < 500)
|
||||
amp = S[m].mean(axis=1)
|
||||
peaks = f[m][np.argsort(amp)[-4:]][::-1]
|
||||
print(f" {label:24} pyin={pyin_f0:6.1f} autocorr={ac_f0:6.1f} Hz "
|
||||
f"low-freq peaks={[round(float(x)) for x in peaks]}")
|
||||
|
||||
# ---------------------------------------------------------------- 4. cast headroom
|
||||
print("\n=== CAST HEADROOM on the pitch axis ===")
|
||||
A = [r for r in rows if r["label"].startswith("A_pitch") and r.get("f0_median")]
|
||||
sep = np.array([abs(st(r["f0_median"], REF_F0)) for r in A])
|
||||
print(f" reliable separation needs > 2-sd = {2*F0_SD_ST:.1f} st")
|
||||
print(f" usable DSP variants found (|dF0| > 2sd): "
|
||||
f"{sorted(round(float(x),1) for x in sep if x > 2*F0_SD_ST)}")
|
||||
usable = [r for r in A if abs(st(r["f0_median"], REF_F0)) > 2 * F0_SD_ST]
|
||||
lo = min(st(r["f0_median"], REF_F0) for r in usable)
|
||||
hi = max(st(r["f0_median"], REF_F0) for r in usable)
|
||||
step = 2 * F0_SD_ST
|
||||
print(f" span {lo:+.1f}..{hi:+.1f} st / {step:.1f} st per voice => "
|
||||
f"~{int((hi-lo)//step)+1} reliably distinct voices on ONE axis")
|
||||
239
backup/author_voice_lab/scripts/backup_author_matrix.py
Normal file
239
backup/author_voice_lab/scripts/backup_author_matrix.py
Normal file
@@ -0,0 +1,239 @@
|
||||
"""Durable backup of the Author voice matrix (vmatrix) + its reference chain.
|
||||
|
||||
The live matrix lives in a git-ignored project cache and the previous version's
|
||||
only copy sits in an OS temp directory that macOS may purge. This writes a
|
||||
self-contained, checksummed, timestamped backup in two places on two physical
|
||||
disks, plus a manifest recording exactly which source WAV each matrix came from.
|
||||
|
||||
Read-only with respect to every original file.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import socket
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
PROJECT = Path("/Volumes/1TB_APL_Storage/UmbraAudio")
|
||||
LIVE_MATRIX = PROJECT / "DealHunterShip_4.0e_whole-to-audio/.umbra/voice_prompts/author.pt"
|
||||
V1_BACKUP = Path("/private/var/folders/y8/cbvhds297918lrxt2wd3btsr0000gn/T/opencode/author_v1_backup.pt")
|
||||
DOCS = Path.home() / "Documents"
|
||||
A4 = DOCS / "Audacity4"
|
||||
|
||||
STAMP = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
|
||||
|
||||
# What each matrix was captured from, per BDMurphy_Voice_analysis.md round 1 / round 2.
|
||||
PROVENANCE = {
|
||||
"author_v2.pt": {
|
||||
"generation": "v2 (current live matrix)",
|
||||
"captured_from": "BDMurphy_Voice_NOBG_v2.wav",
|
||||
"processing": "noisered 0.15 + highpass 40 Hz, resampled 44100->24000, gain -3 dB",
|
||||
"source_recording": "BDMurphy_Voice.wav (raw, 34.33 s, 44100 Hz)",
|
||||
"reason": "round 2 fix for the reported 'hollow / sounds like a building' quality",
|
||||
},
|
||||
"author_v1.pt": {
|
||||
"generation": "v1 (superseded)",
|
||||
"captured_from": "BDMurphy_Voice_NOBG.wav",
|
||||
"processing": "noisered 0.4 + highpass 75 Hz, resampled 44100->24000, gain -3 dB",
|
||||
"reason": "first capture; replaced by v2",
|
||||
},
|
||||
}
|
||||
|
||||
# Reference-chain audio, so the backup is self-contained even if Documents is lost.
|
||||
CHAIN = [
|
||||
A4 / "BDMurphy_Voice.wav",
|
||||
A4 / "BDMurphy_Voice_NOBG.wav",
|
||||
A4 / "BDMurphy_Voice_NOBG_v2.wav",
|
||||
A4 / "BDMurphy_Voice_Author_Rainbow_v2.wav",
|
||||
A4 / "BDMurphy_Voice_Author_Rainbow_v2_air.wav",
|
||||
DOCS / "BDMurphy_Voice_Author_Rainbow_v2_air2.wav",
|
||||
A4 / "BDMurphy_Voice_analysis.md",
|
||||
]
|
||||
|
||||
DESTS = [
|
||||
PROJECT / "backup" / "author_voice_matrix" / STAMP,
|
||||
Path.home() / ".umbra" / "library" / "author_voice_matrix" / STAMP,
|
||||
]
|
||||
|
||||
|
||||
def sha256(p: Path) -> str:
|
||||
h = hashlib.sha256()
|
||||
with open(p, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(1 << 20), b""):
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def describe(p: Path) -> dict:
|
||||
d = {"bytes": p.stat().st_size, "sha256": sha256(p)}
|
||||
try:
|
||||
import soundfile as sf
|
||||
|
||||
i = sf.info(str(p))
|
||||
d |= {"duration_s": round(i.duration, 3), "sample_rate": i.samplerate,
|
||||
"channels": i.channels, "subtype": i.subtype}
|
||||
except Exception:
|
||||
pass
|
||||
return d
|
||||
|
||||
|
||||
def main() -> None:
|
||||
manifest: dict = {
|
||||
"created_utc": datetime.now(timezone.utc).isoformat(),
|
||||
"host": socket.gethostname(),
|
||||
"platform": platform.platform(),
|
||||
"purpose": ("Durable backup of the Author voice matrix (vmatrix) and its reference "
|
||||
"chain. The vmatrix is the source of truth for the Author voice; the WAV "
|
||||
"is reproducible from it."),
|
||||
"matrix_format": {
|
||||
"file": "voice_prompts/<slug>.pt",
|
||||
"structure": "torch.save({'items': [{'ref_code','ref_spk_embedding',"
|
||||
"'x_vector_only_mode','icl_mode','ref_text'}]})",
|
||||
"ref_code": "(T,16) int64 codec tokens @12.5 Hz = re-encoded reference speech",
|
||||
"ref_spk_embedding": "(2048,) bfloat16 speaker x-vector",
|
||||
"full_vmatrix": "x_vector_only_mode=False, icl_mode=True",
|
||||
},
|
||||
"capture_endpoint": "POST http://127.0.0.1:8881/v1/voice-prompt "
|
||||
"slug=author x_vector_only=0 force=1",
|
||||
"matrices": {},
|
||||
"reference_chain": [],
|
||||
"notes": [
|
||||
"author.pt is git-ignored project cache; this backup is the durable copy.",
|
||||
"The superseded v1 matrix's only prior copy was in an OS temp dir (purgeable).",
|
||||
"verdict: full vmatrix (ref_code + embedding), NOT x_vector_only.",
|
||||
],
|
||||
}
|
||||
|
||||
# --- matrices ---
|
||||
matrix_srcs = [("author_v2.pt", LIVE_MATRIX), ("author_v1.pt", V1_BACKUP)]
|
||||
for name, src in matrix_srcs:
|
||||
if not src.exists():
|
||||
manifest["matrices"][name] = {"status": "MISSING", "source": str(src)}
|
||||
continue
|
||||
info = describe(src)
|
||||
info.update(PROVENANCE[name])
|
||||
info["source"] = str(src)
|
||||
try:
|
||||
import torch
|
||||
|
||||
it = torch.load(str(src), weights_only=False)["items"][0]
|
||||
rc = it["ref_code"]
|
||||
info["ref_code_shape"] = list(rc.shape)
|
||||
info["ref_code_seconds"] = round(rc.shape[0] / 12.5, 3)
|
||||
info["icl_mode"] = bool(it["icl_mode"])
|
||||
info["x_vector_only_mode"] = bool(it["x_vector_only_mode"])
|
||||
info["ref_text_is_rainbow"] = str(it["ref_text"]).startswith(
|
||||
"When the sunlight strikes raindrops")
|
||||
except Exception as e: # pragma: no cover
|
||||
info["inspect_error"] = str(e)
|
||||
manifest["matrices"][name] = info
|
||||
|
||||
# --- reference chain ---
|
||||
for src in CHAIN:
|
||||
if src.exists():
|
||||
e = describe(src)
|
||||
e["source"] = str(src)
|
||||
manifest["reference_chain"].append(e)
|
||||
else:
|
||||
manifest["reference_chain"].append({"source": str(src), "status": "MISSING"})
|
||||
|
||||
# --- write ---
|
||||
written = []
|
||||
for dest in DESTS:
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
for name, src in matrix_srcs:
|
||||
if src.exists():
|
||||
shutil.copy2(src, dest / name)
|
||||
for src in CHAIN:
|
||||
if src.exists():
|
||||
shutil.copy2(src, dest / src.name)
|
||||
(dest / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n")
|
||||
(dest / "RESTORE.md").write_text(RESTORE_MD)
|
||||
written.append(str(dest))
|
||||
|
||||
# Verify every copied byte against the manifest.
|
||||
problems = []
|
||||
for dest in DESTS:
|
||||
for name, src in matrix_srcs:
|
||||
if not src.exists():
|
||||
continue
|
||||
d = dest / name
|
||||
if not d.exists():
|
||||
problems.append(f"MISSING COPY {d}")
|
||||
elif sha256(d) != manifest["matrices"][name]["sha256"]:
|
||||
problems.append(f"HASH MISMATCH {d}")
|
||||
|
||||
manifest["written_to"] = written
|
||||
manifest["verification"] = "ok" if not problems else problems
|
||||
for dest in DESTS:
|
||||
(dest / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n")
|
||||
|
||||
print(json.dumps({
|
||||
"stamp": STAMP,
|
||||
"written_to": written,
|
||||
"verification": manifest["verification"],
|
||||
"matrices": {k: {"sha256": v.get("sha256", "")[:16], "bytes": v.get("bytes"),
|
||||
"from": v.get("captured_from"), "ref_code": v.get("ref_code_shape")}
|
||||
for k, v in manifest["matrices"].items()},
|
||||
"chain_files": len(manifest["reference_chain"]),
|
||||
}, indent=2))
|
||||
|
||||
|
||||
RESTORE_MD = """# Author voice matrix — restore
|
||||
|
||||
## What this is
|
||||
|
||||
The **Author** voice identity, stored as a Qwen3-TTS *vmatrix* (the numbers that
|
||||
ARE the voice). Captured once; reuses those exact numbers for every utterance, so
|
||||
the voice never drifts between takes.
|
||||
|
||||
## Files
|
||||
|
||||
| File | What it is |
|
||||
|---|---|
|
||||
| `author_v2.pt` | **The live matrix.** Use this one. |
|
||||
| `author_v1.pt` | Superseded first capture, kept for rollback. |
|
||||
| `BDMurphy_Voice*.wav` | The reference chain the matrix was captured from. |
|
||||
| `manifest.json` | SHA-256, provenance, ref_code shape for every file. |
|
||||
|
||||
## Restore
|
||||
|
||||
```bash
|
||||
# Matrix -> the server's prompt cache (PROMPT_DIR for the local server):
|
||||
cp author_v2.pt <project>/.umbra/voice_prompts/author.pt
|
||||
|
||||
# Reference WAV -> a project's approved-reference dir:
|
||||
cp BDMurphy_Voice_NOBG_v2.wav <project>/.umbra/voices/author_ref.wav
|
||||
```
|
||||
|
||||
The capture call that produced this matrix (re-run ONLY if you are deliberately
|
||||
changing the voice — it is not needed to use it):
|
||||
|
||||
```
|
||||
POST http://127.0.0.1:8881/v1/voice-prompt
|
||||
slug=author x_vector_only=0 force=1
|
||||
ref_text=<Rainbow Passage> ref_audio=BDMurphy_Voice_NOBG_v2.wav
|
||||
```
|
||||
|
||||
`x_vector_only=0` is the full vmatrix (`ref_code` + speaker embedding) and is the
|
||||
only mode that reproduces this voice faithfully. `x_vector_only=1` captures just
|
||||
the coarse embedding and is what historically caused voices to drift.
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
shasum -a 256 author_v2.pt # must equal manifest.json -> matrices.author_v2.sha256
|
||||
```
|
||||
|
||||
To confirm a matrix still renders the right speaker, generate from
|
||||
`prompt_slug=author` and compare the median F0 against the reference
|
||||
(reference is 129.4 Hz; a correct render lands ~130-133 Hz).
|
||||
"""
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
74
backup/author_voice_lab/scripts/fidelity.py
Normal file
74
backup/author_voice_lab/scripts/fidelity.py
Normal file
@@ -0,0 +1,74 @@
|
||||
"""Phase-insensitive fidelity: how faithfully does the matrix reproduce the reference?
|
||||
|
||||
Frame-wise waveform correlation and LSD are dominated by phase, which a 12 Hz
|
||||
neural codec does not preserve. Identity lives in the long-term average spectrum
|
||||
(tonal balance), the pitch contour, and the mel envelope — all phase-insensitive.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import librosa
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
|
||||
OUT = Path("/private/var/folders/y8/cbvhds297918lrxt2wd3btsr0000gn/T/opencode/probe_out")
|
||||
A4 = Path.home() / "Documents/Audacity4"
|
||||
dec, sr = sf.read(str(OUT / "author_ref_DECODED_FROM_MATRIX.wav"), dtype="float32")
|
||||
src, _ = sf.read(str(A4 / "BDMurphy_Voice_NOBG_v2.wav"), dtype="float32")
|
||||
if src.ndim > 1:
|
||||
src = src.mean(axis=1)
|
||||
|
||||
BANDS = [(80, 160), (160, 315), (315, 630), (630, 1250), (1250, 2500),
|
||||
(2500, 5000), (5000, 10000), (10000, 12000)]
|
||||
|
||||
|
||||
def ltas(y):
|
||||
S = np.abs(librosa.stft(y.astype(np.float32), n_fft=2048, hop_length=512))
|
||||
f = librosa.fft_frequencies(sr=sr, n_fft=2048)
|
||||
out = []
|
||||
for lo, hi in BANDS:
|
||||
m = (f >= lo) & (f < hi)
|
||||
out.append(10 * np.log10((S[m] ** 2).sum() + 1e-12))
|
||||
return np.array(out)
|
||||
|
||||
|
||||
def mel_cep(y):
|
||||
m = librosa.feature.melspectrogram(y=y.astype(np.float32), sr=sr, n_mels=40,
|
||||
n_fft=1024, hop_length=256)
|
||||
return np.log10(m + 1e-10)
|
||||
|
||||
|
||||
def f0_track(y):
|
||||
f0, _, _ = librosa.pyin(y.astype(np.float32), fmin=60.0, fmax=400.0, sr=sr)
|
||||
return f0
|
||||
|
||||
|
||||
print(f"{'':28} {'decoded':>9} {'reference':>10} {'delta':>8}")
|
||||
print("--- long-term average spectrum, per band (dB, energy-normalised) ---")
|
||||
a, b = ltas(dec), ltas(src)
|
||||
a_n = a - a.mean()
|
||||
b_n = b - b.mean()
|
||||
for (lo, hi), x, y in zip(BANDS, a_n, b_n):
|
||||
print(f"{lo:>6}-{hi:<6} Hz{'':10} {x:9.2f} {y:10.2f} {x - y:+8.2f}")
|
||||
print(f"{'RMS deviation':28} {'':9} {'':10} {float(np.sqrt((a_n - b_n) ** 2).mean()):+8.2f} dB")
|
||||
|
||||
print("\n--- mel envelope (40 bands, log) ---")
|
||||
ma, mb = mel_cep(dec), mel_cep(src)
|
||||
n = min(ma.shape[1], mb.shape[1])
|
||||
ma_n = ma[:, :n] - ma[:, :n].mean(axis=0, keepdims=True)
|
||||
mb_n = mb[:, :n] - mb[:, :n].mean(axis=0, keepdims=True)
|
||||
print(f"mean |log-mel diff| = {np.abs(ma_n - mb_n).mean():.4f} (log10 units)")
|
||||
|
||||
print("\n--- pitch contour ---")
|
||||
fa, fb = f0_track(dec), f0_track(src)
|
||||
va, vb = fa[~np.isnan(fa)], fb[~np.isnan(fb)]
|
||||
print(f"median F0: decoded {np.median(va):.1f} Hz reference {np.median(vb):.1f} Hz")
|
||||
print(f"voiced frames: decoded {len(va)} reference {len(vb)}")
|
||||
|
||||
# The real fidelity test: re-encode the DECODED audio. If the codec round-trip is
|
||||
# stable, decoding twice converges, i.e. matrix -> wav -> codes -> same codes.
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, "/Volumes/1TB_APL_Storage/UmbraAudio")
|
||||
print("\n(note: full re-encode check runs in matrix_roundtrip.py)")
|
||||
47
backup/author_voice_lab/scripts/install_ref.py
Normal file
47
backup/author_voice_lab/scripts/install_ref.py
Normal file
@@ -0,0 +1,47 @@
|
||||
"""Install the matrix-derived Author reference into the project.
|
||||
|
||||
The matrix is the source of truth; this writes the WAV *decoded from that matrix*
|
||||
so the project's reference and the matrix provably agree (LTAS 0.55 dB RMS,
|
||||
median F0 within 0.8 Hz of the original capture).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
PROJECT = Path("/Volumes/1TB_APL_Storage/UmbraAudio")
|
||||
DECODED = Path("/private/var/folders/y8/cbvhds297918lrxt2wd3btsr0000gn/T/opencode/probe_out/"
|
||||
"author_ref_DECODED_FROM_MATRIX.wav")
|
||||
VOICES = PROJECT / "DealHunterShip_4.0e_whole-to-audio/.umbra/voices"
|
||||
MATRIX = PROJECT / "DealHunterShip_4.0e_whole-to-audio/.umbra/voice_prompts/author.pt"
|
||||
|
||||
|
||||
def sha256(p: Path) -> str:
|
||||
h = hashlib.sha256()
|
||||
h.update(Path(p).read_bytes())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
dest = VOICES / "author_ref.wav"
|
||||
if dest.exists():
|
||||
backup = VOICES / f"author_ref.PRE_MATRIX_DECODE.{datetime.now():%Y%m%d_%H%M%S}.wav"
|
||||
shutil.copy2(dest, backup)
|
||||
print(f"existing author_ref.wav -> {backup.name}")
|
||||
|
||||
shutil.copy2(DECODED, dest)
|
||||
info = {
|
||||
"installed": str(dest),
|
||||
"decoded_from_matrix": str(MATRIX),
|
||||
"matrix_sha256": sha256(MATRIX),
|
||||
"ref_sha256": sha256(dest),
|
||||
"provenance": "decoded from author.pt ref_code via Qwen3TTSTokenizer.decode()",
|
||||
"note": ("Regenerate with the matrix only: "
|
||||
"Qwen3TTSTokenizer.from_pretrained(models/Qwen3-TTS-12Hz-1.7B-Base/"
|
||||
"speech_tokenizer).decode({'audio_codes': ref_code})"),
|
||||
}
|
||||
side = dest.with_suffix(".json")
|
||||
side.write_text(json.dumps(info, indent=2) + "\n")
|
||||
print(json.dumps(info, indent=2))
|
||||
129
backup/author_voice_lab/scripts/matrix_roundtrip.py
Normal file
129
backup/author_voice_lab/scripts/matrix_roundtrip.py
Normal file
@@ -0,0 +1,129 @@
|
||||
"""Prove: the vmatrix alone can reproduce the reference WAV — and verify its provenance.
|
||||
|
||||
1. DECODE author.pt ref_code -> waveform (matrix -> audio; the user's requirement)
|
||||
2. VERIFY that decoded audio matches the WAV the matrix was captured from
|
||||
3. PROVE provenance by re-encoding each candidate source WAV and comparing its
|
||||
audio_codes to the stored ref_code. Identical codes => that file is
|
||||
provably where this matrix came from (no trusting the notes).
|
||||
|
||||
Runs on CPU with only the 682 MB speech tokenizer, so the TTS server keeps its
|
||||
MPS model resident and nothing else contends for memory.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import librosa
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
import torch
|
||||
|
||||
sys.path.insert(0, "/Volumes/1TB_APL_Storage/UmbraAudio")
|
||||
from qwen_tts import Qwen3TTSTokenizer # noqa: E402
|
||||
|
||||
PROJECT = Path("/Volumes/1TB_APL_Storage/UmbraAudio")
|
||||
MATRIX = PROJECT / "DealHunterShip_4.0e_whole-to-audio/.umbra/voice_prompts/author.pt"
|
||||
TOKENIZER = PROJECT / "models/Qwen3-TTS-12Hz-1.7B-Base/speech_tokenizer"
|
||||
A4 = Path.home() / "Documents/Audacity4"
|
||||
OUT = Path("/private/var/folders/y8/cbvhds297918lrxt2wd3btsr0000gn/T/opencode/probe_out")
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
CANDIDATES = {
|
||||
"NOBG_v2 (documented v2 source)": A4 / "BDMurphy_Voice_NOBG_v2.wav",
|
||||
"NOBG (documented v1 source)": A4 / "BDMurphy_Voice_NOBG.wav",
|
||||
"raw recording": A4 / "BDMurphy_Voice.wav",
|
||||
}
|
||||
|
||||
|
||||
def f0_median(y, sr):
|
||||
f0, _, _ = librosa.pyin(y.astype(np.float32), fmin=60.0, fmax=400.0, sr=sr)
|
||||
v = f0[~np.isnan(f0)]
|
||||
return float(np.median(v)) if len(v) else float("nan")
|
||||
|
||||
|
||||
def lsd(a, b, sr):
|
||||
"""Log-spectral distance over the speech band (dB)."""
|
||||
n = min(len(a), len(b))
|
||||
A = np.abs(librosa.stft(a[:n].astype(np.float32), n_fft=1024, hop_length=256))
|
||||
B = np.abs(librosa.stft(b[:n].astype(np.float32), n_fft=1024, hop_length=256))
|
||||
m = min(A.shape[1], B.shape[1])
|
||||
A, B = A[:, :m], B[:, :m]
|
||||
f = librosa.fft_frequencies(sr=sr, n_fft=1024)
|
||||
band = (f >= 80) & (f <= 8000)
|
||||
la = 20 * np.log10(A[band] + 1e-8)
|
||||
lb = 20 * np.log10(B[band] + 1e-8)
|
||||
return float(np.sqrt(np.mean((la - lb) ** 2)))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
item = torch.load(str(MATRIX), weights_only=False)["items"][0]
|
||||
ref_code = item["ref_code"].cpu()
|
||||
print(f"matrix ref_code: {tuple(ref_code.shape)} (~{ref_code.shape[0] / 12.5:.2f}s)")
|
||||
print(f"icl_mode={item['icl_mode']} x_vector_only_mode={item['x_vector_only_mode']}")
|
||||
|
||||
tok = Qwen3TTSTokenizer.from_pretrained(str(TOKENIZER), device_map="cpu")
|
||||
print(f"tokenizer loaded, output sr = {tok.feature_extractor.sampling_rate}")
|
||||
|
||||
# ---------- 1. DECODE: matrix -> wav ----------
|
||||
wavs, sr = tok.decode({"audio_codes": ref_code})
|
||||
decoded = wavs[0].astype(np.float32)
|
||||
dec_path = OUT / "author_ref_DECODED_FROM_MATRIX.wav"
|
||||
sf.write(str(dec_path), decoded, sr, subtype="PCM_16")
|
||||
print(f"\nDECODED from matrix: {len(decoded)/sr:.2f}s @ {sr} Hz -> {dec_path.name}")
|
||||
print(f" median F0 = {f0_median(decoded, sr):.1f} Hz")
|
||||
|
||||
# ---------- 2. VERIFY against each candidate ----------
|
||||
print("\n--- decoded vs candidate source WAVs ---")
|
||||
print(f"{'candidate':36} {'dur':>7} {'F0':>7} {'LSD_dB':>8} {'corr':>7}")
|
||||
results = {}
|
||||
for label, p in CANDIDATES.items():
|
||||
if not p.exists():
|
||||
print(f"{label:36} MISSING")
|
||||
continue
|
||||
y, ysr = sf.read(str(p), dtype="float32")
|
||||
if y.ndim > 1:
|
||||
y = y.mean(axis=1)
|
||||
if ysr != sr:
|
||||
y = librosa.resample(y, orig_sr=ysr, target_sr=sr)
|
||||
n = min(len(y), len(decoded))
|
||||
corr = float(np.corrcoef(y[:n], decoded[:n])[0, 1])
|
||||
d = lsd(y, decoded, sr)
|
||||
results[label] = {"dur": round(len(y) / sr, 2), "f0": round(f0_median(y, sr), 1),
|
||||
"lsd_db": round(d, 2), "corr": round(corr, 4)}
|
||||
print(f"{label:36} {len(y)/sr:7.2f} {f0_median(y, sr):7.1f} {d:8.2f} {corr:7.4f}")
|
||||
|
||||
# ---------- 3. PROVE provenance: re-encode and compare codes ----------
|
||||
print("\n--- provenance: re-encode candidate, compare audio_codes to stored ref_code ---")
|
||||
print(f"{'candidate':36} {'shape':>12} {'exact_match':>13} {'frame_agree':>13}")
|
||||
prov = {}
|
||||
for label, p in CANDIDATES.items():
|
||||
if not p.exists():
|
||||
continue
|
||||
enc = tok.encode([str(p)])
|
||||
codes = enc.audio_codes[0].cpu()
|
||||
same_shape = tuple(codes.shape) == tuple(ref_code.shape)
|
||||
if same_shape:
|
||||
exact = bool(torch.equal(codes, ref_code))
|
||||
agree = float((codes == ref_code).float().mean())
|
||||
else:
|
||||
exact, agree = False, float("nan")
|
||||
prov[label] = {"shape": list(codes.shape), "exact_match": exact,
|
||||
"frame_agreement": round(agree, 6)}
|
||||
print(f"{label:36} {str(tuple(codes.shape)):>12} {str(exact):>13} {agree:>13.6f}")
|
||||
|
||||
out = {
|
||||
"matrix": str(MATRIX),
|
||||
"ref_code_shape": list(ref_code.shape),
|
||||
"decoded": {"path": str(dec_path), "duration_s": round(len(decoded) / sr, 3),
|
||||
"sample_rate": sr, "median_f0_hz": round(f0_median(decoded, sr), 1)},
|
||||
"verified_against": results,
|
||||
"provenance": prov,
|
||||
}
|
||||
(OUT / "matrix_roundtrip.json").write_text(json.dumps(out, indent=2) + "\n")
|
||||
print("\nwrote", OUT / "matrix_roundtrip.json")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
56
backup/author_voice_lab/scripts/probe_dsp.py
Normal file
56
backup/author_voice_lab/scripts/probe_dsp.py
Normal file
@@ -0,0 +1,56 @@
|
||||
"""Probe 2: DSP axis limits. What does pitch_shift_pv actually move?"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, "/Volumes/1TB_APL_Storage/UmbraAudio")
|
||||
|
||||
import librosa
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
|
||||
from umbraaudio.generation.audio_effects import EFFECT_FN
|
||||
|
||||
# NOTE: apply_effects_by_name() only accepts effect NAMES and uses hardcoded
|
||||
# defaults, so it cannot dial a parameter. A sweep needs EFFECT_FN directly.
|
||||
_pv = EFFECT_FN["pitch_shift_pv"]
|
||||
|
||||
SRC = Path("/private/var/folders/y8/cbvhds297918lrxt2wd3btsr0000gn/T/opencode/probe_out/A_baseline.wav")
|
||||
|
||||
|
||||
def measure(y, sr):
|
||||
f0, _, _ = librosa.pyin(y.astype(np.float32), fmin=60.0, fmax=400.0, sr=sr)
|
||||
v = f0[~np.isnan(f0)]
|
||||
S = np.abs(librosa.stft(y.astype(np.float32), n_fft=2048, hop_length=512))
|
||||
cen = float(librosa.feature.spectral_centroid(S=S, sr=sr).mean())
|
||||
# Formant-ish proxies: energy-weighted centroid in low band vs overall
|
||||
return float(np.median(v)), cen
|
||||
|
||||
|
||||
def formant_band_ratio(y, sr):
|
||||
"""Crude vocal-tract length proxy: spectral centroid of the voiced spectrum."""
|
||||
S = np.abs(librosa.stft(y.astype(np.float32), n_fft=2048, hop_length=512))
|
||||
freqs = librosa.fft_frequencies(sr=sr, n_fft=2048)
|
||||
m = freqs <= 4000
|
||||
return float((S[m] * freqs[m][:, None]).sum() / S[m].sum())
|
||||
|
||||
|
||||
y, sr = sf.read(str(SRC), dtype="float32")
|
||||
base_f0, base_cen = measure(y, sr)
|
||||
base_ftr = formant_band_ratio(y, sr)
|
||||
print(f"BASE F0med={base_f0:7.1f}Hz centroid={base_cen:7.0f}Hz LTAS4k={base_ftr:7.0f}Hz")
|
||||
|
||||
print("\n--- pitch_shift_pv sweep (semitones) ---")
|
||||
print(f"{'semi':>6} {'F0med':>8} {'d_F0_st':>8} {'centroid':>9} {'d_cen_st':>8} {'LTAS4k':>8} {'d_ftr_st':>9}")
|
||||
for st in (-12, -8, -4, 0, 4, 8, 12):
|
||||
out = _pv(y, sr, semitones=float(st))
|
||||
f0, cen = measure(out, sr)
|
||||
ftr = formant_band_ratio(out, sr)
|
||||
d_f0 = 12 * np.log2(f0 / base_f0)
|
||||
d_cen = 12 * np.log2(cen / base_cen)
|
||||
d_ftr = 12 * np.log2(ftr / base_ftr)
|
||||
print(f"{st:>6} {f0:>8.1f} {d_f0:>8.1f} {cen:>9.0f} {d_cen:>8.1f} {ftr:>8.0f} {d_ftr:>9.1f}")
|
||||
|
||||
print("\nInterpretation: d_cen ~= d_f0 => full-spectrum shift => formants move WITH pitch.")
|
||||
print("A true male->female conversion needs d_cen/d_F0 ~= +2.5 to +3 st, d_F0 ~= +9 to +12 st.")
|
||||
69
backup/author_voice_lab/scripts/probe_formant.py
Normal file
69
backup/author_voice_lab/scripts/probe_formant.py
Normal file
@@ -0,0 +1,69 @@
|
||||
"""Probe 3: prove the missing primitive is an INDEPENDENT FORMANT SHIFTER.
|
||||
|
||||
pitch_shift_pv is a phase vocoder -> it moves F0 and the spectral envelope by
|
||||
DIFFERENT amounts (measured: formants lag pitch going up, overshoot going down).
|
||||
A vocal-tract (formant) shift is the missing primitive and is achievable with
|
||||
resample + time_stretch, which decouples the two.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, "/Volumes/1TB_APL_Storage/UmbraAudio")
|
||||
|
||||
import librosa
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
|
||||
from umbraaudio.generation.audio_effects import EFFECT_FN
|
||||
|
||||
SRC = Path("/private/var/folders/y8/cbvhds297918lrxt2wd3btsr0000gn/T/opencode/probe_out/A_baseline.wav")
|
||||
OUT = Path("/private/var/folders/y8/cbvhds297918lrxt2wd3btsr0000gn/T/opencode/probe_out")
|
||||
_pv = EFFECT_FN["pitch_shift_pv"]
|
||||
|
||||
|
||||
def m(y, sr):
|
||||
y = y.astype(np.float32)
|
||||
f0, _, _ = librosa.pyin(y, fmin=60.0, fmax=400.0, sr=sr)
|
||||
v = f0[~np.isnan(f0)]
|
||||
S = np.abs(librosa.stft(y, n_fft=2048, hop_length=512))
|
||||
freqs = librosa.fft_frequencies(sr=sr, n_fft=2048)
|
||||
msk = freqs <= 4000
|
||||
ftr = float((S[msk] * freqs[msk][:, None]).sum() / S[msk].sum())
|
||||
return float(np.median(v)), ftr
|
||||
|
||||
|
||||
def formant_shift(y, sr, semitones: float) -> np.ndarray:
|
||||
"""Shift ONLY the vocal-tract envelope, leaving F0 and duration untouched."""
|
||||
r = 2.0 ** (semitones / 12.0)
|
||||
up = librosa.resample(y.astype(np.float32), orig_sr=sr, target_sr=int(round(sr * r)))
|
||||
back = librosa.resample(up, orig_sr=int(round(sr * r)), target_sr=sr)
|
||||
return librosa.effects.time_stretch(back.astype(np.float32), rate=r).astype(np.float32)
|
||||
|
||||
|
||||
y, sr = sf.read(str(SRC), dtype="float32")
|
||||
b_f0, b_ftr = m(y, sr)
|
||||
print(f"BASE F0={b_f0:6.1f}Hz LTAS4k={b_ftr:6.0f}Hz\n")
|
||||
|
||||
print("Formant shift ALONE (F0 must stay put, LTAS4k must move):")
|
||||
print(f"{'f_st':>6} {'F0':>8} {'dF0_st':>8} {'LTAS4k':>8} {'dftr_st':>8}")
|
||||
for fs in (-6, -3, 3, 6):
|
||||
o = formant_shift(y, sr, fs)
|
||||
f0, ftr = m(o, sr)
|
||||
print(f"{fs:>6} {f0:>8.1f} {12*np.log2(f0/b_f0):>8.1f} {ftr:>8.0f} {12*np.log2(ftr/b_ftr):>8.1f}")
|
||||
sf.write(OUT / f"formant_{fs:+d}.wav", o, sr)
|
||||
|
||||
print("\nCOMBINED male->female recipe: pitch +10 st THEN formant +3 st")
|
||||
o = formant_shift(_pv(y, sr, semitones=10.0), sr, 3.0)
|
||||
f0, ftr = m(o, sr)
|
||||
print(f" F0={f0:.1f}Hz ({12*np.log2(f0/b_f0):+.1f} st) LTAS4k={ftr:.0f}Hz "
|
||||
f"({12*np.log2(ftr/b_ftr):+.1f} st)")
|
||||
sf.write(OUT / "female_recipe.wav", o, sr)
|
||||
|
||||
print("\nCOMBINED male->dragon recipe: pitch -10 st THEN formant -8 st (vocal-tract growth)")
|
||||
o = formant_shift(_pv(y, sr, semitones=-10.0), sr, -8.0)
|
||||
f0, ftr = m(o, sr)
|
||||
print(f" F0={f0:.1f}Hz ({12*np.log2(f0/b_f0):+.1f} st) LTAS4k={ftr:.0f}Hz "
|
||||
f"({12*np.log2(ftr/b_ftr):+.1f} st)")
|
||||
sf.write(OUT / "dragon_recipe.wav", o, sr)
|
||||
91
backup/author_voice_lab/scripts/probe_morph.py
Normal file
91
backup/author_voice_lab/scripts/probe_morph.py
Normal file
@@ -0,0 +1,91 @@
|
||||
"""Probe: can the narrator vmatrix be morphed (gender / creature) via instruct?
|
||||
|
||||
Read-only against the project; writes WAVs + measurements to an off-volume temp dir.
|
||||
Uses the cached narrator vmatrix (prompt_slug) so voice identity is pinned.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import librosa
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
|
||||
import httpx
|
||||
|
||||
URL = "http://127.0.0.1:8881"
|
||||
OUT = Path("/private/var/folders/y8/cbvhds297918lrxt2wd3btsr0000gn/T/opencode/probe_out")
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
TEXT = (
|
||||
"The lantern guttered as she pushed the door open, and the cold air of the "
|
||||
"cavern rolled over her like the breath of something waiting."
|
||||
)
|
||||
|
||||
|
||||
def measure(wav: np.ndarray, sr: int) -> dict:
|
||||
y = wav.astype(np.float32)
|
||||
f0, _, _ = librosa.pyin(y, fmin=60.0, fmax=400.0, sr=sr)
|
||||
v = f0[~np.isnan(f0)]
|
||||
S = np.abs(librosa.stft(y, n_fft=2048, hop_length=512))
|
||||
cen = float(librosa.feature.spectral_centroid(S=S, sr=sr).mean())
|
||||
flat = float(librosa.feature.spectral_flatness(S=S).mean())
|
||||
# long-term average spectrum slope: brightness/tilth
|
||||
rms = float(20 * np.log10(np.sqrt((y**2).mean()) + 1e-12))
|
||||
if len(v):
|
||||
out = {
|
||||
"f0_median": round(float(np.median(v)), 1),
|
||||
"f0_sd": round(float(v.std()), 1),
|
||||
"f0_range": [round(float(v.min())), round(float(v.max()))],
|
||||
"semitones_vs_132hz": round(12 * np.log2(np.median(v) / 132.4), 1),
|
||||
}
|
||||
else:
|
||||
out = {"f0_median": None, "f0_sd": None, "f0_range": None, "semitones_vs_132hz": None}
|
||||
out.update(
|
||||
centroid_hz=round(cen),
|
||||
flatness=round(flat, 5),
|
||||
rms_db=round(rms, 1),
|
||||
dur_s=round(len(y) / sr, 2),
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
REF = Path("/Volumes/1TB_APL_Storage/UmbraAudio/.umbra/voices/narrator_ref.wav")
|
||||
|
||||
|
||||
def gen(label: str, *, instruct: str | None, seed: int = 1234) -> dict:
|
||||
data = {"text": TEXT, "sample_rate": 24000, "temperature": 0.7, "do_sample": True,
|
||||
"seed": seed}
|
||||
files = {"ref_audio": ("ref.wav", REF.read_bytes(), "audio/wav")}
|
||||
if instruct:
|
||||
data["voice_description"] = instruct
|
||||
t0 = time.time()
|
||||
r = httpx.post(f"{URL}/v1/tts", data=data, files=files, timeout=900)
|
||||
dt = time.time() - t0
|
||||
if r.status_code != 200:
|
||||
return {"label": label, "error": f"HTTP {r.status_code}: {r.text[:200]}"}
|
||||
wav, sr = sf.read(io.BytesIO(r.content), dtype="float32")
|
||||
if wav.ndim > 1:
|
||||
wav = wav.mean(axis=1)
|
||||
sf.write(OUT / f"{label}.wav", wav, sr)
|
||||
m = measure(wav, sr)
|
||||
m.update(label=label, seconds=round(dt, 1), instruct=instruct)
|
||||
print(json.dumps(m), flush=True)
|
||||
return m
|
||||
|
||||
|
||||
CASES = [
|
||||
("A_baseline", None),
|
||||
("B_female", "A young woman's voice, bright and light, with a higher and clearer tone."),
|
||||
("C_elderly_m", "An elderly man's voice, very deep, gravelly and slow."),
|
||||
("D_dragon", "A huge ancient dragon's voice: a deep, rumbling, growling monster with "
|
||||
"a metallic edge and a resonant, echoing chest tone."),
|
||||
]
|
||||
|
||||
if __name__ == "__main__":
|
||||
results = [gen(label, instruct=instr) for label, instr in CASES]
|
||||
(OUT / "probe_results.json").write_text(json.dumps(results, indent=2))
|
||||
print("WROTE", OUT / "probe_results.json")
|
||||
86
backup/author_voice_lab/scripts/verify_cliff.py
Normal file
86
backup/author_voice_lab/scripts/verify_cliff.py
Normal file
@@ -0,0 +1,86 @@
|
||||
"""Verify the downward-pitch intelligibility cliff on a LONGER text, 2 seeds.
|
||||
|
||||
The 25-word probe suggested WER collapses below ~0 st. WER on 25 words is coarse
|
||||
(4% per word) and the earlier narrator probe was much gentler, so this re-tests
|
||||
with ~3x the text and repeats each point.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import librosa
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
|
||||
sys.path.insert(0, "/Volumes/1TB_APL_Storage/UmbraAudio")
|
||||
from umbraaudio.generation.audio_effects import EFFECT_FN # noqa: E402
|
||||
|
||||
import httpx # noqa: E402
|
||||
import whisper # noqa: E402
|
||||
|
||||
PROJECT = Path("/Volumes/1TB_APL_Storage/UmbraAudio")
|
||||
REF = PROJECT / "DealHunterShip_4.0e_whole-to-audio/.umbra/voices/author_ref.wav"
|
||||
OUT = Path("/private/var/folders/y8/cbvhds297918lrxt2wd3btsr0000gn/T/opencode/author_lab")
|
||||
(OUT / "wav_cliff").mkdir(exist_ok=True)
|
||||
|
||||
LONG = (
|
||||
"He set the lantern on the stone ledge and listened. The cavern breathed around "
|
||||
"them, slow and patient, and somewhere far below the water was moving against rock. "
|
||||
"MaryAnn kept one hand on the rail and waited for him to speak first, because she "
|
||||
"had learned that what he said next would decide which of them went forward."
|
||||
)
|
||||
REF_WORDS = re.sub(r"[^a-z ]", "", LONG.lower()).split()
|
||||
print(f"long text: {len(REF_WORDS)} words")
|
||||
|
||||
_pv = EFFECT_FN["pitch_shift_pv"]
|
||||
wm = whisper.load_model("base", device="cpu")
|
||||
|
||||
|
||||
def run(st, seed):
|
||||
r = httpx.post("http://127.0.0.1:8881/v1/tts",
|
||||
data={"text": LONG, "sample_rate": 24000, "temperature": 0.7,
|
||||
"do_sample": True, "seed": seed},
|
||||
files={"ref_audio": ("ref.wav", REF.read_bytes(), "audio/wav")},
|
||||
timeout=900)
|
||||
y, sr = sf.read(io.BytesIO(r.content), dtype="float32")
|
||||
if y.ndim > 1:
|
||||
y = y.mean(axis=1)
|
||||
if st:
|
||||
y = _pv(y, sr, semitones=float(st))
|
||||
p = float(np.max(np.abs(y)))
|
||||
if p > 1.0:
|
||||
y = y / p
|
||||
import difflib
|
||||
|
||||
hyp = re.sub(r"[^a-z ]", "",
|
||||
wm.transcribe(y, language="en", fp16=False, temperature=0.0)["text"]
|
||||
).lower().split()
|
||||
n = sum(b.size for b in difflib.SequenceMatcher(None, REF_WORDS, hyp).get_matching_blocks())
|
||||
f0, _, _ = librosa.pyin(y.astype(np.float32), fmin=60.0, fmax=400.0, sr=sr)
|
||||
v = f0[~np.isnan(f0)]
|
||||
sf.write(OUT / "wav_cliff" / f"cliff{st:+03d}_s{seed}.wav", y, sr, subtype="PCM_16")
|
||||
return {"shift_st": st, "seed": seed, "wer": round((len(REF_WORDS) - n) / len(REF_WORDS), 3),
|
||||
"f0": round(float(np.median(v)), 1), "hyp": " ".join(hyp)}
|
||||
|
||||
|
||||
rows = []
|
||||
for st in (0, -2, -4, -6, +2):
|
||||
for seed in (7, 8):
|
||||
t0 = time.time()
|
||||
m = run(st, seed)
|
||||
m["sec"] = round(time.time() - t0, 1)
|
||||
rows.append(m)
|
||||
print(f" shift {st:+3d} st seed {seed} WER {m['wer']:.3f} F0 {m['f0']:6.1f} Hz "
|
||||
f"{m['sec']}s", flush=True)
|
||||
|
||||
(OUT / "cliff_verification.json").write_text(json.dumps(
|
||||
{"text_words": len(REF_WORDS), "rows": rows}, indent=2) + "\n")
|
||||
print("\n--- mean WER by shift ---")
|
||||
for st in (0, -2, -4, -6, +2):
|
||||
ws = [r["wer"] for r in rows if r["shift_st"] == st]
|
||||
print(f" {st:+3d} st: mean WER {sum(ws)/len(ws):.3f} {ws}")
|
||||
220
backup/author_voice_lab/scripts/voice_lab.py
Normal file
220
backup/author_voice_lab/scripts/voice_lab.py
Normal file
@@ -0,0 +1,220 @@
|
||||
"""voice_lab — measure what the Author voice matrix can become.
|
||||
|
||||
One fixed text for every sample (comparisons are only valid at constant text).
|
||||
Every sample records: F0 median/sd/range, spectral-envelope centroid and its
|
||||
semitone offset (the "vocal tract" proxy), brightness, flatness, level, whisper
|
||||
WER, and elapsed time.
|
||||
|
||||
Axes:
|
||||
A pitch pitch_shift_pv, -12..+12 st -> where is the intelligibility cliff
|
||||
B tract neural instruct, 8 specs -> how far will identity move
|
||||
C hybrid instruct + DSP trim -> close the gap instruct leaves
|
||||
D variance seeds x temperatures -> intra-voice spread (cast-size limit)
|
||||
|
||||
Outputs WAVs + results.json + results.md under OUT.
|
||||
Nothing in the project tree is read except the reference; nothing is written to it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import librosa
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
|
||||
sys.path.insert(0, "/Volumes/1TB_APL_Storage/UmbraAudio")
|
||||
from umbraaudio.generation.audio_effects import EFFECT_FN # noqa: E402
|
||||
|
||||
import httpx # noqa: E402
|
||||
|
||||
PROJECT = Path("/Volumes/1TB_APL_Storage/UmbraAudio")
|
||||
REF = PROJECT / "DealHunterShip_4.0e_whole-to-audio/.umbra/voices/author_ref.wav"
|
||||
OUT = Path("/private/var/folders/y8/cbvhds297918lrxt2wd3btsr0000gn/T/opencode/author_lab")
|
||||
URL = "http://127.0.0.1:8881"
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
(OUT / "wav").mkdir(exist_ok=True)
|
||||
|
||||
# ONE text for every sample.
|
||||
TEXT = ("The lantern guttered as she pushed the door open, and the cold air of the "
|
||||
"cavern rolled over her like the breath of something waiting.")
|
||||
|
||||
REF_F0 = None # set from the reference wav at startup
|
||||
_pv = EFFECT_FN["pitch_shift_pv"]
|
||||
_sat = EFFECT_FN["tube_sat"]
|
||||
_ring = EFFECT_FN["ring_modulate"]
|
||||
_chorus = EFFECT_FN["chorus"]
|
||||
_jitter = EFFECT_FN["pitch_jitter"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- measurement
|
||||
def measure(y: np.ndarray, sr: int) -> dict:
|
||||
y = y.astype(np.float32)
|
||||
f0, _, _ = librosa.pyin(y, fmin=60.0, fmax=400.0, sr=sr)
|
||||
v = f0[~np.isnan(f0)]
|
||||
S = np.abs(librosa.stft(y, n_fft=2048, hop_length=512))
|
||||
f = librosa.fft_frequencies(sr=sr, n_fft=2048)
|
||||
band = f <= 4000
|
||||
ftr = float((S[band] * f[band][:, None]).sum() / S[band].sum())
|
||||
m = {
|
||||
"f0_median": round(float(np.median(v)), 1) if len(v) else None,
|
||||
"f0_sd": round(float(v.std()), 1) if len(v) else None,
|
||||
"envelope_hz": round(ftr),
|
||||
"centroid_hz": round(float(librosa.feature.spectral_centroid(S=S, sr=sr).mean())),
|
||||
"flatness": round(float(librosa.feature.spectral_flatness(S=S).mean()), 5),
|
||||
"dur_s": round(len(y) / sr, 2),
|
||||
"rms_db": round(float(20 * np.log10(np.sqrt((y**2).mean()) + 1e-12)), 1),
|
||||
"peak_db": round(float(20 * np.log10(np.max(np.abs(y)) + 1e-12)), 1),
|
||||
}
|
||||
if m["f0_median"] and REF_F0:
|
||||
m["d_f0_st"] = round(12 * np.log2(m["f0_median"] / REF_F0), 1)
|
||||
return m
|
||||
|
||||
|
||||
def wer(model, ref_words: list[str], y: np.ndarray) -> float | None:
|
||||
import difflib
|
||||
|
||||
try:
|
||||
txt = model.transcribe(y.astype(np.float32), language="en", fp16=False,
|
||||
temperature=0.0)["text"]
|
||||
except Exception:
|
||||
return None
|
||||
hyp = re.sub(r"[^a-z ]", "", txt.lower()).split()
|
||||
n = sum(b.size for b in difflib.SequenceMatcher(None, ref_words, hyp).get_matching_blocks())
|
||||
return round((len(ref_words) - n) / max(1, len(ref_words)), 3)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- generation
|
||||
def synth(label: str, *, instruct: str | None = None, temperature: float = 0.7,
|
||||
seed: int = 1234, dsp: list | None = None, want_wer: bool = True,
|
||||
whisper_model=None) -> dict:
|
||||
data = {"text": TEXT, "sample_rate": 24000, "temperature": temperature,
|
||||
"do_sample": True, "seed": seed}
|
||||
if instruct:
|
||||
data["voice_description"] = instruct
|
||||
t0 = time.time()
|
||||
r = httpx.post(f"{URL}/v1/tts", data=data,
|
||||
files={"ref_audio": ("ref.wav", REF.read_bytes(), "audio/wav")},
|
||||
timeout=900)
|
||||
if r.status_code != 200:
|
||||
return {"label": label, "error": f"HTTP {r.status_code}: {r.text[:160]}"}
|
||||
y, sr = sf.read(io.BytesIO(r.content), dtype="float32")
|
||||
if y.ndim > 1:
|
||||
y = y.mean(axis=1)
|
||||
gen_s = round(time.time() - t0, 1)
|
||||
|
||||
chain_desc = None
|
||||
if dsp:
|
||||
for name, params in dsp:
|
||||
y = EFFECT_FN[name](y, sr, **params)
|
||||
p = float(np.max(np.abs(y)))
|
||||
if p > 1.0:
|
||||
y = y / p
|
||||
chain_desc = [n for n, _ in dsp]
|
||||
|
||||
m = measure(y, sr)
|
||||
m.update(label=label, gen_s=gen_s, instruct=instruct, temperature=temperature,
|
||||
seed=seed, dsp=chain_desc,
|
||||
d_env_st=(round(12 * np.log2(m["envelope_hz"] / REF_ENV), 1)
|
||||
if REF_ENV else None))
|
||||
if want_wer and whisper_model is not None:
|
||||
m["wer"] = wer(whisper_model, REF_WORDS, y)
|
||||
sf.write(OUT / "wav" / f"{label}.wav", y, sr, subtype="PCM_16")
|
||||
return m
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- axes
|
||||
TRACTS = {
|
||||
"B_female_young": "A young woman's voice, bright and light, with a higher and clearer tone.",
|
||||
"B_female_mature": "A mature woman's voice, low and calm for a woman, unhurried and steady.",
|
||||
"B_boy": "A young boy's voice, light and eager, not yet broken.",
|
||||
"B_elderly_m": "An elderly man's voice, very deep, gravelly and slow.",
|
||||
"B_young_m": "A young man's voice, light, energetic and clear.",
|
||||
"B_large_m": "A very large man's voice, heavy and booming, with a deep chest.",
|
||||
"B_small_w": "A small woman's voice, petite and light, with a narrow bright tone.",
|
||||
"B_androgynous": "A neutral, androgynous voice, neither distinctly male nor female.",
|
||||
}
|
||||
|
||||
GRIT = {
|
||||
"grit_sat": [("tube_sat", {"drive_db": 12.0, "mix": 1.0})],
|
||||
"grit_ring": [("ring_modulate", {"freq": 38.0})],
|
||||
"grit_chorus": [("chorus", {"rate_hz": 1.6, "depth": 0.3, "centre_delay_ms": 8.0,
|
||||
"feedback": 0.2, "mix": 0.4})],
|
||||
"grit_full": [("tube_sat", {"drive_db": 12.0, "mix": 1.0}),
|
||||
("ring_modulate", {"freq": 38.0}),
|
||||
("chorus", {"rate_hz": 1.6, "depth": 0.3, "centre_delay_ms": 8.0,
|
||||
"feedback": 0.2, "mix": 0.4}),
|
||||
("pitch_jitter", {"jitter_hz": 6.0, "depth": 0.03})],
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
global REF_F0, REF_ENV, REF_WORDS
|
||||
import whisper
|
||||
|
||||
ref_y, ref_sr = sf.read(str(REF), dtype="float32")
|
||||
if ref_y.ndim > 1:
|
||||
ref_y = ref_y.mean(axis=1)
|
||||
S = np.abs(librosa.stft(ref_y.astype(np.float32), n_fft=2048, hop_length=512))
|
||||
f = librosa.fft_frequencies(sr=ref_sr, n_fft=2048)
|
||||
band = f <= 4000
|
||||
REF_ENV = float((S[band] * f[band][:, None]).sum() / S[band].sum())
|
||||
f0r, _, _ = librosa.pyin(ref_y.astype(np.float32), fmin=60.0, fmax=400.0, sr=ref_sr)
|
||||
REF_F0 = float(np.median(f0r[~np.isnan(f0r)]))
|
||||
REF_WORDS = re.sub(r"[^a-z ]", "", TEXT.lower()).split()
|
||||
print(f"REFERENCE (author_ref.wav): F0 {REF_F0:.1f} Hz envelope {REF_ENV:.0f} Hz "
|
||||
f"{len(ref_y)/ref_sr:.1f}s")
|
||||
|
||||
wm = whisper.load_model("base", device="cpu")
|
||||
rows: list[dict] = []
|
||||
|
||||
def run(label, **kw):
|
||||
m = synth(label, whisper_model=wm, **kw)
|
||||
rows.append(m)
|
||||
print(f" {label:26} F0={m.get('f0_median')} "
|
||||
f"({m.get('d_f0_st')}st) env={m.get('d_env_st')}st "
|
||||
f"WER={m.get('wer')} {m.get('gen_s')}s", flush=True)
|
||||
return m
|
||||
|
||||
print("\n[baseline]")
|
||||
run("A00_baseline", temperature=0.7, seed=1234)
|
||||
|
||||
print("\n[A] pitch axis -12..+12 step 2")
|
||||
for st in range(-12, 13, 2):
|
||||
run(f"A_pitch{st:+03d}", dsp=[("pitch_shift_pv", {"semitones": float(st)})])
|
||||
|
||||
print("\n[B] neural tract morphs")
|
||||
for label, instr in TRACTS.items():
|
||||
run(label, instruct=instr)
|
||||
|
||||
print("\n[C] hybrids")
|
||||
for trim in (4, 8):
|
||||
run(f"C_female+{trim}st", instruct=TRACTS["B_female_young"],
|
||||
dsp=[("pitch_shift_pv", {"semitones": float(trim)})])
|
||||
run("C_monster_pitch-10", dsp=[("pitch_shift_pv", {"semitones": -10.0})])
|
||||
for gname, gchain in GRIT.items():
|
||||
run(f"C_monster_{gname}",
|
||||
instruct="A huge ancient dragon: deep, rumbling, growling, with a metallic "
|
||||
"edge and a resonant echoing chest tone.",
|
||||
dsp=gchain)
|
||||
|
||||
print("\n[D] intra-voice variance (seeds x temperature)")
|
||||
for temp in (0.3, 0.7, 0.9):
|
||||
for seed in (11, 22, 33, 44, 55):
|
||||
run(f"D_seed{seed}_t{temp}", temperature=temp, seed=seed, want_wer=False)
|
||||
|
||||
(OUT / "results.json").write_text(json.dumps(
|
||||
{"reference": {"f0_median": round(REF_F0, 1), "envelope_hz": round(REF_ENV),
|
||||
"path": str(REF)}, "rows": rows}, indent=2) + "\n")
|
||||
print(f"\nwrote {OUT/'results.json'} ({len(rows)} samples)")
|
||||
|
||||
|
||||
REF_ENV = None
|
||||
REF_WORDS: list[str] = []
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
BIN
backup/author_voice_lab/wav/A00_baseline.wav
Normal file
BIN
backup/author_voice_lab/wav/A00_baseline.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/A_pitch+00.wav
Normal file
BIN
backup/author_voice_lab/wav/A_pitch+00.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/A_pitch+02.wav
Normal file
BIN
backup/author_voice_lab/wav/A_pitch+02.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/A_pitch+04.wav
Normal file
BIN
backup/author_voice_lab/wav/A_pitch+04.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/A_pitch+06.wav
Normal file
BIN
backup/author_voice_lab/wav/A_pitch+06.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/A_pitch+08.wav
Normal file
BIN
backup/author_voice_lab/wav/A_pitch+08.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/A_pitch+10.wav
Normal file
BIN
backup/author_voice_lab/wav/A_pitch+10.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/A_pitch+12.wav
Normal file
BIN
backup/author_voice_lab/wav/A_pitch+12.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/A_pitch-02.wav
Normal file
BIN
backup/author_voice_lab/wav/A_pitch-02.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/A_pitch-04.wav
Normal file
BIN
backup/author_voice_lab/wav/A_pitch-04.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/A_pitch-06.wav
Normal file
BIN
backup/author_voice_lab/wav/A_pitch-06.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/A_pitch-08.wav
Normal file
BIN
backup/author_voice_lab/wav/A_pitch-08.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/A_pitch-10.wav
Normal file
BIN
backup/author_voice_lab/wav/A_pitch-10.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/A_pitch-12.wav
Normal file
BIN
backup/author_voice_lab/wav/A_pitch-12.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/B_androgynous.wav
Normal file
BIN
backup/author_voice_lab/wav/B_androgynous.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/B_boy.wav
Normal file
BIN
backup/author_voice_lab/wav/B_boy.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/B_elderly_m.wav
Normal file
BIN
backup/author_voice_lab/wav/B_elderly_m.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/B_female_mature.wav
Normal file
BIN
backup/author_voice_lab/wav/B_female_mature.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/B_female_young.wav
Normal file
BIN
backup/author_voice_lab/wav/B_female_young.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/B_large_m.wav
Normal file
BIN
backup/author_voice_lab/wav/B_large_m.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/B_small_w.wav
Normal file
BIN
backup/author_voice_lab/wav/B_small_w.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/B_young_m.wav
Normal file
BIN
backup/author_voice_lab/wav/B_young_m.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/C_female+4st.wav
Normal file
BIN
backup/author_voice_lab/wav/C_female+4st.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/C_female+8st.wav
Normal file
BIN
backup/author_voice_lab/wav/C_female+8st.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/C_monster_grit_chorus.wav
Normal file
BIN
backup/author_voice_lab/wav/C_monster_grit_chorus.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/C_monster_grit_full.wav
Normal file
BIN
backup/author_voice_lab/wav/C_monster_grit_full.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/C_monster_grit_ring.wav
Normal file
BIN
backup/author_voice_lab/wav/C_monster_grit_ring.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/C_monster_grit_sat.wav
Normal file
BIN
backup/author_voice_lab/wav/C_monster_grit_sat.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/C_monster_pitch-10.wav
Normal file
BIN
backup/author_voice_lab/wav/C_monster_pitch-10.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed11_t0.3.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed11_t0.3.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed11_t0.7.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed11_t0.7.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed11_t0.9.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed11_t0.9.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed22_t0.3.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed22_t0.3.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed22_t0.7.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed22_t0.7.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed22_t0.9.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed22_t0.9.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed33_t0.3.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed33_t0.3.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed33_t0.7.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed33_t0.7.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed33_t0.9.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed33_t0.9.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed44_t0.3.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed44_t0.3.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed44_t0.7.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed44_t0.7.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed44_t0.9.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed44_t0.9.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed55_t0.3.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed55_t0.3.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed55_t0.7.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed55_t0.7.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav/D_seed55_t0.9.wav
Normal file
BIN
backup/author_voice_lab/wav/D_seed55_t0.9.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav_cliff/cliff+00_s7.wav
Normal file
BIN
backup/author_voice_lab/wav_cliff/cliff+00_s7.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav_cliff/cliff+00_s8.wav
Normal file
BIN
backup/author_voice_lab/wav_cliff/cliff+00_s8.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav_cliff/cliff+02_s7.wav
Normal file
BIN
backup/author_voice_lab/wav_cliff/cliff+02_s7.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav_cliff/cliff+02_s8.wav
Normal file
BIN
backup/author_voice_lab/wav_cliff/cliff+02_s8.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav_cliff/cliff-02_s7.wav
Normal file
BIN
backup/author_voice_lab/wav_cliff/cliff-02_s7.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav_cliff/cliff-02_s8.wav
Normal file
BIN
backup/author_voice_lab/wav_cliff/cliff-02_s8.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav_cliff/cliff-04_s7.wav
Normal file
BIN
backup/author_voice_lab/wav_cliff/cliff-04_s7.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav_cliff/cliff-04_s8.wav
Normal file
BIN
backup/author_voice_lab/wav_cliff/cliff-04_s8.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav_cliff/cliff-06_s7.wav
Normal file
BIN
backup/author_voice_lab/wav_cliff/cliff-06_s7.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_lab/wav_cliff/cliff-06_s8.wav
Normal file
BIN
backup/author_voice_lab/wav_cliff/cliff-06_s8.wav
Normal file
Binary file not shown.
BIN
backup/author_voice_matrix/20261009T191636Z/BDMurphy_Voice.wav
Normal file
BIN
backup/author_voice_matrix/20261009T191636Z/BDMurphy_Voice.wav
Normal file
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,241 @@
|
||||
# BDMurphy_Voice — Noise Cleanup & UmbraAudio Suitability Analysis
|
||||
|
||||
Date: 2026-10-08
|
||||
Files: `BDMurphy_Voice.wav` (input) → `BDMurphy_Voice_NOBG.wav` (cleaned)
|
||||
|
||||
---
|
||||
|
||||
## 1. Input analysis (`BDMurphy_Voice.wav`)
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Duration | 34.328 s |
|
||||
| Format | WAV PCM_16, **44,100 Hz**, mono, 3,027,810 bytes |
|
||||
| Peak | −22.04 dBFS (20 dB of unused headroom) |
|
||||
| Overall RMS | −41.56 dBFS |
|
||||
| Speech level (90th-pct frame) | −37.0 dBFS |
|
||||
| Background in pauses (5 measured) | −64.6 / −65.9 / −66.7 / −66.6 / −68.1 dBFS |
|
||||
| SNR (90th vs 10th pct frame energy) | 28.4 dB |
|
||||
| Clipping | 0 samples at full scale |
|
||||
| DC offset | −3e-06 (none) |
|
||||
| Median F0 | 130.1 Hz, voiced ratio 0.558 → adult male speaker |
|
||||
|
||||
**Noise character** (measured, not guessed):
|
||||
|
||||
- The noise is **stationary** — spectrum correlation between the first and second
|
||||
half of the file's quiet frames: **0.828**.
|
||||
- It is **broadband with low-frequency dominance**: in pauses the 20–100 Hz band
|
||||
is the loudest component (rumble/mains hum), then 100–300 Hz, with a
|
||||
broadband hiss floor across 1–12 kHz.
|
||||
- Pause-to-speech ratio only **28–31 dB**, i.e. the background is audible
|
||||
between sentences; house-generated references sit at 50–60 dB.
|
||||
- The sample is **~20 dB quieter than every reference in the system**
|
||||
(house refs: RMS −19.8…−26.5 dBFS, peak −1.1…−7.3; this file: RMS −41.6,
|
||||
peak −22.0).
|
||||
|
||||
## 2. Suitability check #1 — **FAIL**
|
||||
|
||||
Checked against the gates the UmbraAudio system itself enforces
|
||||
(`umbraaudio/analysis/voice_analysis.py`: `MIN_REF_DURATION_S=20`,
|
||||
`TARGET_SAMPLE_RATE=24000`, `SILENCE_RMS=0.002`) plus standard voice-capture
|
||||
quality gates (noise floor ≤ −45 dBFS, SNR ≥ 20 dB, clipping ≤ 0.1 %,
|
||||
|DC| ≤ 0.01):
|
||||
|
||||
| Gate | Value | Threshold | Result |
|
||||
|---|---|---|---|
|
||||
| duration | 34.33 s | ≥ 20 s | PASS |
|
||||
| **sample_rate** | **44,100 Hz** | **24,000 Hz** | **FAIL** |
|
||||
| energy | 0.00836 | > 0.002 | PASS |
|
||||
| noise_floor | −65.4 dBFS | ≤ −45 dBFS | PASS |
|
||||
| snr | 28.4 dB | ≥ 20 dB | PASS |
|
||||
| clipping | 0.0 | ≤ 0.001 | PASS |
|
||||
| dc_offset | −3e-06 | \|x\| ≤ 0.01 | PASS |
|
||||
|
||||
Confirmed by running the **system's own** `VoiceAnalysisEngine` on the raw file
|
||||
as `author_ref.wav`: `passed: false` — only `sample_rate` fails (44100 ≠ 24000).
|
||||
All 212 existing `*_ref.wav` files in UmbraAudio are 24 kHz mono PCM_16.
|
||||
|
||||
Verdict: not usable as-is — wrong sample rate for the system, background noise
|
||||
only ~30 dB below speech, and 20 dB below the house reference level.
|
||||
|
||||
## 3. Cleanup performed → `BDMurphy_Voice_NOBG.wav`
|
||||
|
||||
Pipeline (SoX, native rate first so the profile matches the original noise):
|
||||
|
||||
1. **Noise profile** from the five *measured* pauses (7.92, 11.92, 14.88, 16.40,
|
||||
33.12 s) — `sox … trim … noiseprof`. (An earlier profile taken from the
|
||||
mathematically-quietest 1 s window was rejected: it contained speech leakage
|
||||
and removed 4× less noise.)
|
||||
2. **Spectral subtraction** — `noisered profile 0.4`.
|
||||
3. **High-pass 75 Hz** (`highpass -2 75`) — removes rumble/mains hum; the
|
||||
speaker's median F0 is 130 Hz, so nothing of the voice is touched.
|
||||
4. **Resample 44.1 → 24 kHz** (`rate -v`) — meets `TARGET_SAMPLE_RATE`, matching
|
||||
the system's reference format exactly.
|
||||
5. **Level match** (`gain -n -3`) → peak −3.0 dBFS, which lands the file inside
|
||||
the house reference envelope (RMS −23.5 vs −19.8…−26.5; speech −19.0 vs
|
||||
−15.2…−21.6; peak −3.0 vs −1.1…−7.3).
|
||||
|
||||
**Denoise strength was selected by measurement**, not by ear — four candidate
|
||||
settings were built from the same pause profile and scored on noise removal vs
|
||||
speech damage:
|
||||
|
||||
| Variant | Pause noise (dBFS) | Speech spectral delta | Envelope corr | F0 delta |
|
||||
|---|---|---|---|---|
|
||||
| high-pass only | −68.4 … −72.5 | 1.47 dB | 0.799 | 0.0 Hz |
|
||||
| **noisered 0.4 + HP — CHOSEN** | −75.1 … −91.6 | 5.03 dB | 0.755 | **0.0 Hz** |
|
||||
| noisered 0.6 + HP | −79.0 … −96.3 | 8.59 dB | 0.681 | 0.0 Hz |
|
||||
| noisered 0.8 + HP | −84.1 … −96.3 | 15.93 dB | 0.588 | 0.71 Hz (pitch drift) |
|
||||
|
||||
The chosen setting removes **~24 dB of background noise** while leaving the
|
||||
pitch track bit-identical; the next step up doubles speech distortion for 5 dB
|
||||
more noise removal, and 0.8 starts corrupting the F0 track.
|
||||
|
||||
## 4. Result (`BDMurphy_Voice_NOBG.wav`)
|
||||
|
||||
| Property | Before | After |
|
||||
|---|---|---|
|
||||
| Sample rate | 44,100 Hz | **24,000 Hz** |
|
||||
| Duration | 34.328 s | 34.305 s (−23 ms resampler edge) |
|
||||
| Noise floor (10th-pct frame) | −65.4 dBFS | **−76.4 dBFS** |
|
||||
| Pauses | −64.6 … −68.1 dBFS | −70.0 … −72.5 dBFS (worst: −55.9, a 0.28 s breath) |
|
||||
| SNR | 28.4 dB | **57.3 dB** |
|
||||
| Peak / RMS | −22.0 / −41.6 dBFS | −3.0 / −23.5 dBFS |
|
||||
| Speech level | −37.0 dBFS | −19.0 dBFS |
|
||||
| Median F0 | 130.1 Hz | 129.4 Hz (preserved) |
|
||||
| Clipping / DC | none / none | none / none |
|
||||
|
||||
Integrity checks against the gain-matched original:
|
||||
|
||||
- **Time alignment:** cross-correlation lag **0 samples (0.00 ms)** — no shift.
|
||||
- **Polarity:** band-limited correlation **0.9755** — same polarity.
|
||||
- **Dropouts:** 0 near-silent holes punched into speech frames (longest 0 ms).
|
||||
- **Speech energy:** −0.76 dB on active frames (both peak-normalised).
|
||||
- Residual pause noise sits **~53 dB below speech**.
|
||||
|
||||
## 5. Suitability check #2 — **PASS**
|
||||
|
||||
My 7-gate check on `BDMurphy_Voice_NOBG.wav`:
|
||||
|
||||
| Gate | Value | Threshold | Result |
|
||||
|---|---|---|---|
|
||||
| duration | 34.31 s | ≥ 20 s | PASS |
|
||||
| sample_rate | 24,000 Hz | 24,000 Hz | PASS |
|
||||
| energy | 0.0665 | > 0.002 | PASS |
|
||||
| noise_floor | −76.4 dBFS | ≤ −45 dBFS | PASS |
|
||||
| snr | 57.3 dB | ≥ 20 dB | PASS |
|
||||
| clipping | 0.0 | ≤ 0.001 | PASS |
|
||||
| dc_offset | 0.0 | \|x\| ≤ 0.01 | PASS |
|
||||
|
||||
And the **system's own** `VoiceAnalysisEngine` (run against a copy named
|
||||
`author_ref.wav`): all four requirements pass —
|
||||
`duration 34.3 ≥ 20`, `sample_rate 24000 = 24000`, `energy 0.0665 > 0.002`,
|
||||
`uniqueness 0.0 < 0.9` → **"All 1 voices meet requirements", `passed: true`**.
|
||||
|
||||
## 6. Voice matrix created — voice name **"Author"**
|
||||
|
||||
Because the check passed, the cleaned sample was used to capture the vmatrix
|
||||
through the system's canonical full-vmatrix path:
|
||||
|
||||
```
|
||||
POST http://localhost:8881/v1/voice-prompt
|
||||
slug = author
|
||||
x_vector_only = 0 # FULL vmatrix (ref_code + embedding), the canonical mode
|
||||
force = 1
|
||||
ref_text = Rainbow Passage (verified byte-identical to the server's
|
||||
VOICE_REFERENCE_TEXT)
|
||||
ref_audio = BDMurphy_Voice_NOBG.wav
|
||||
→ {"ok": true, "slug": "author", ...}
|
||||
```
|
||||
|
||||
**Artifact:** `/Volumes/1TB_APL_Storage/UmbraAudio/DealHunterShip_4.0e_whole-to-audio/.umbra/voice_prompts/author.pt` (61,853 bytes)
|
||||
|
||||
Contents verified by loading the file — structurally identical to the production
|
||||
prompts (e.g. `agnarr.pt`):
|
||||
|
||||
| Field | `author.pt` | production example |
|
||||
|---|---|---|
|
||||
| `ref_code` | **(429, 16)** int64 | (453, 16) int64 |
|
||||
| `ref_spk_embedding` | **(2048,)** bfloat16 | (2048,) bfloat16 |
|
||||
| `x_vector_only_mode` | False (full vmatrix) | False |
|
||||
| `icl_mode` | True | True |
|
||||
| `ref_text` | Rainbow Passage | Rainbow Passage |
|
||||
|
||||
429 codec frames ÷ 34.31 s ≈ 12.5 Hz — exactly the Qwen3-TTS 12 Hz codec rate,
|
||||
i.e. the whole reference was re-spoken through the codec as intended.
|
||||
|
||||
**End-to-end verification** — `POST /v1/tts` with `prompt_slug=author`
|
||||
(HTTP 200, 113 s on MPS):
|
||||
|
||||
- Generated 31.4 s of 24 kHz audio, RMS −23.2 dBFS, peak −4.7 dBFS (healthy).
|
||||
- Generated median F0 **133.1 Hz** vs reference **129.4 Hz** (Δ 3.8 Hz).
|
||||
- **ECAPA-TDNN speaker similarity: generated audio vs the cleaned reference = 0.887**,
|
||||
against a control (an unrelated speaker's reference) of **−0.004** — the
|
||||
vmatrix reproduces this speaker, not some other voice.
|
||||
|
||||
The server was returned to its prior state afterwards (`POST /unload {"model_type":"all"}`
|
||||
— no model resident, as it was before the capture).
|
||||
|
||||
## Notes / follow-ups
|
||||
|
||||
- The `.pt` lives in the DealHunterShip project's `.umbra/voice_prompts/`
|
||||
because that is `QWEN_PROMPT_DIR` for the running local server (the system's
|
||||
canonical prompt cache; the directory is git-ignored, so the repo stays clean).
|
||||
- The cleaned WAV stays in this folder. To use "Author" as a speaker inside a
|
||||
project, its reference would normally also live at
|
||||
`<project>/.umbra/voices/author_ref.wav` — not done here (not requested).
|
||||
- Observation while testing: ICL generation with `prompt_slug` kept speaking
|
||||
well past the 12-word test sentence (25 s of speech). That is server-side ICL
|
||||
stopping behaviour, **not** a defect in this vmatrix — worth a look separately.
|
||||
|
||||
---
|
||||
|
||||
## 7. Round 2 — the "hollow / sounds like a building" report
|
||||
|
||||
Feedback after listening: clean, but hollow — "like the noise removal put the
|
||||
sound in a building". Measured diagnosis (raw vs cleaned vs generated, all
|
||||
compared on speech-active frames):
|
||||
|
||||
| Hypothesis | Test | Result |
|
||||
|---|---|---|
|
||||
| Room reverb unmasked by denoising | decay inside the silent gaps (≥150 ms, absolute dBFS) | **dry**: cleaned falls 33 dB in 25 ms, reaches its floor in 75 ms (raw: 40 ms). A real room is still at −5 dB there. |
|
||||
| Discrete reflection / comb filter | per-frame autocorrelation 1–40 ms | only peak is the **pitch period** (7.0–7.5 ms = 130 Hz F0), identical in raw, cleaned and generated |
|
||||
| Tonal imbalance introduced | energy-normalised LTAS band shape | **preserved within 0.2 dB** from 80 Hz to 12 kHz; only 40–80 Hz changed (−12.3 dB, my 75 Hz high-pass) and 2.5–5 kHz (−2.5 dB) |
|
||||
| Over-reduction artifacts | frame-to-frame spectral jump, band modulation | up 1.90 → 3.00 dB/frame; decomposition: rate→2.61, high-pass→2.74, `noisered 0.15`→2.87, `noisered 0.4`→2.98 (the jump metric itself is also driven by residual-noise content, confirmed by scale-invariance test) |
|
||||
| Render dullness (TTS side) | band shape of the generated Rainbow vs its reference | boxy-low (+1.7 dB at 80–160, +1.2 at 160–315) and **dull-high (−4.4 dB at 5–10 kHz, −6.6 dB at 10–12 kHz)** |
|
||||
|
||||
Two changes follow from the measurements, applied on both sides of the chain:
|
||||
|
||||
**Reference — `BDMurphy_Voice_NOBG_v2.wav`** (replaces the v2 capture source)
|
||||
|
||||
- `noisered 0.4 → 0.15` (spectral distortion vs the original halves:
|
||||
LSD 5.02 → 2.71 dB) and `highpass 75 → 40 Hz` (restores 12 dB of body in
|
||||
the 40–75 Hz band).
|
||||
- Still comfortably inside every gate: floor −63.3 dBFS (≤ −45), SNR 42.4 dB
|
||||
(≥ 20), peak −3.0 dBFS, 0 clipped samples, |DC| 0.0, 34.31 s, 24 kHz.
|
||||
- Alignment vs the original: lag 0 samples, polarity r = 0.893, 0 dropouts.
|
||||
- The system's own `VoiceAnalysisEngine`: **PASS** (all four requirements).
|
||||
|
||||
**vmatrix re-captured** from that v2 file (`slug=author, x_vector_only=0,
|
||||
force=1`, ref_text = server default `VOICE_REFERENCE_TEXT`). The previous
|
||||
vmatrix is backed up at
|
||||
`…/T/opencode/author_v1_backup.pt` (61,853 bytes).
|
||||
|
||||
**Renders** — both level-matched to the first sample (active-RMS −20.3 dBFS,
|
||||
i.e. the −23.9 dBFS overall RMS of the original render):
|
||||
|
||||
| File | What it is |
|
||||
|---|---|
|
||||
| `BDMurphy_Voice_Author_Rainbow_v2.wav` | Rainbow through the new vmatrix — no extra EQ |
|
||||
| `BDMurphy_Voice_Author_Rainbow_v2_air.wav` | same render + smooth 1/6-octave tilt correction toward the reference voice (cuts the boxy 315–630 Hz excess, restores the 5–12 kHz air) |
|
||||
|
||||
Verification of `…_v2.wav`: 85.6 s @ 24 kHz, RMS −23.8 / peak −5.2 dBFS, 0
|
||||
clipped samples, median F0 130.9 Hz vs reference 129.4 Hz (Δ 1.5 Hz),
|
||||
**ECAPA similarity 0.8833 vs the Author reference, −0.0213 vs a control
|
||||
speaker** → same voice. Tonal gap to the reference above 80 Hz: 8.1 dB (first
|
||||
render) → 6.6 dB (v2) → **1.5 dB** with the tilt correction, which leaves
|
||||
identity unchanged (ECAPA 0.8835).
|
||||
|
||||
Server returned to its prior state (`POST /unload {"model_type":"all"}`);
|
||||
UmbraAudio repo still clean. Compare the three renders A/B — if the hollow
|
||||
quality survives both fixes it is in the model's ICL synthesis, not in the
|
||||
recording chain.
|
||||
49
backup/author_voice_matrix/20261009T191636Z/RESTORE.md
Normal file
49
backup/author_voice_matrix/20261009T191636Z/RESTORE.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# Author voice matrix — restore
|
||||
|
||||
## What this is
|
||||
|
||||
The **Author** voice identity, stored as a Qwen3-TTS *vmatrix* (the numbers that
|
||||
ARE the voice). Captured once; reuses those exact numbers for every utterance, so
|
||||
the voice never drifts between takes.
|
||||
|
||||
## Files
|
||||
|
||||
| File | What it is |
|
||||
|---|---|
|
||||
| `author_v2.pt` | **The live matrix.** Use this one. |
|
||||
| `author_v1.pt` | Superseded first capture, kept for rollback. |
|
||||
| `BDMurphy_Voice*.wav` | The reference chain the matrix was captured from. |
|
||||
| `manifest.json` | SHA-256, provenance, ref_code shape for every file. |
|
||||
|
||||
## Restore
|
||||
|
||||
```bash
|
||||
# Matrix -> the server's prompt cache (PROMPT_DIR for the local server):
|
||||
cp author_v2.pt <project>/.umbra/voice_prompts/author.pt
|
||||
|
||||
# Reference WAV -> a project's approved-reference dir:
|
||||
cp BDMurphy_Voice_NOBG_v2.wav <project>/.umbra/voices/author_ref.wav
|
||||
```
|
||||
|
||||
The capture call that produced this matrix (re-run ONLY if you are deliberately
|
||||
changing the voice — it is not needed to use it):
|
||||
|
||||
```
|
||||
POST http://127.0.0.1:8881/v1/voice-prompt
|
||||
slug=author x_vector_only=0 force=1
|
||||
ref_text=<Rainbow Passage> ref_audio=BDMurphy_Voice_NOBG_v2.wav
|
||||
```
|
||||
|
||||
`x_vector_only=0` is the full vmatrix (`ref_code` + speaker embedding) and is the
|
||||
only mode that reproduces this voice faithfully. `x_vector_only=1` captures just
|
||||
the coarse embedding and is what historically caused voices to drift.
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
shasum -a 256 author_v2.pt # must equal manifest.json -> matrices.author_v2.sha256
|
||||
```
|
||||
|
||||
To confirm a matrix still renders the right speaker, generate from
|
||||
`prompt_slug=author` and compare the median F0 against the reference
|
||||
(reference is 129.4 Hz; a correct render lands ~130-133 Hz).
|
||||
BIN
backup/author_voice_matrix/20261009T191636Z/author_v1.pt
Normal file
BIN
backup/author_voice_matrix/20261009T191636Z/author_v1.pt
Normal file
Binary file not shown.
BIN
backup/author_voice_matrix/20261009T191636Z/author_v2.pt
Normal file
BIN
backup/author_voice_matrix/20261009T191636Z/author_v2.pt
Normal file
Binary file not shown.
122
backup/author_voice_matrix/20261009T191636Z/manifest.json
Normal file
122
backup/author_voice_matrix/20261009T191636Z/manifest.json
Normal file
@@ -0,0 +1,122 @@
|
||||
{
|
||||
"created_utc": "2026-10-09T19:16:36.048124+00:00",
|
||||
"host": "mini.bdmurphy.me",
|
||||
"platform": "macOS-27.0.1-arm64-arm-64bit",
|
||||
"purpose": "Durable backup of the Author voice matrix (vmatrix) and its reference chain. The vmatrix is the source of truth for the Author voice; the WAV is reproducible from it.",
|
||||
"matrix_format": {
|
||||
"file": "voice_prompts/<slug>.pt",
|
||||
"structure": "torch.save({'items': [{'ref_code','ref_spk_embedding','x_vector_only_mode','icl_mode','ref_text'}]})",
|
||||
"ref_code": "(T,16) int64 codec tokens @12.5 Hz = re-encoded reference speech",
|
||||
"ref_spk_embedding": "(2048,) bfloat16 speaker x-vector",
|
||||
"full_vmatrix": "x_vector_only_mode=False, icl_mode=True"
|
||||
},
|
||||
"capture_endpoint": "POST http://127.0.0.1:8881/v1/voice-prompt slug=author x_vector_only=0 force=1",
|
||||
"matrices": {
|
||||
"author_v2.pt": {
|
||||
"bytes": 61853,
|
||||
"sha256": "d38a2dc88a9d353cafbf310710d7b5cf8e1293889c3e1987537b888a55c6a83f",
|
||||
"generation": "v2 (current live matrix)",
|
||||
"captured_from": "BDMurphy_Voice_NOBG_v2.wav",
|
||||
"processing": "noisered 0.15 + highpass 40 Hz, resampled 44100->24000, gain -3 dB",
|
||||
"source_recording": "BDMurphy_Voice.wav (raw, 34.33 s, 44100 Hz)",
|
||||
"reason": "round 2 fix for the reported 'hollow / sounds like a building' quality",
|
||||
"source": "/Volumes/1TB_APL_Storage/UmbraAudio/DealHunterShip_4.0e_whole-to-audio/.umbra/voice_prompts/author.pt",
|
||||
"ref_code_shape": [
|
||||
429,
|
||||
16
|
||||
],
|
||||
"ref_code_seconds": 34.32,
|
||||
"icl_mode": true,
|
||||
"x_vector_only_mode": false,
|
||||
"ref_text_is_rainbow": true
|
||||
},
|
||||
"author_v1.pt": {
|
||||
"bytes": 61853,
|
||||
"sha256": "c683975346f9d35ecc14a8d3e64e5cf17c77225b5852075a8ecdba69d2996cb5",
|
||||
"generation": "v1 (superseded)",
|
||||
"captured_from": "BDMurphy_Voice_NOBG.wav",
|
||||
"processing": "noisered 0.4 + highpass 75 Hz, resampled 44100->24000, gain -3 dB",
|
||||
"reason": "first capture; replaced by v2",
|
||||
"source": "/private/var/folders/y8/cbvhds297918lrxt2wd3btsr0000gn/T/opencode/author_v1_backup.pt",
|
||||
"ref_code_shape": [
|
||||
429,
|
||||
16
|
||||
],
|
||||
"ref_code_seconds": 34.32,
|
||||
"icl_mode": true,
|
||||
"x_vector_only_mode": false,
|
||||
"ref_text_is_rainbow": true
|
||||
}
|
||||
},
|
||||
"reference_chain": [
|
||||
{
|
||||
"bytes": 3027810,
|
||||
"sha256": "a552759369da74fd39fe0d98d67af3b6b4b25359a66fdc858a9c0a79fa076cfe",
|
||||
"duration_s": 34.328,
|
||||
"sample_rate": 44100,
|
||||
"channels": 1,
|
||||
"subtype": "PCM_16",
|
||||
"source": "/Volumes/Dock/worstauthor/Documents/Audacity4/BDMurphy_Voice.wav"
|
||||
},
|
||||
{
|
||||
"bytes": 1646694,
|
||||
"sha256": "229eb18851ceef5d306ab9a807be763a223665cc8361b9e8957d26c86236054c",
|
||||
"duration_s": 34.305,
|
||||
"sample_rate": 24000,
|
||||
"channels": 1,
|
||||
"subtype": "PCM_16",
|
||||
"source": "/Volumes/Dock/worstauthor/Documents/Audacity4/BDMurphy_Voice_NOBG.wav"
|
||||
},
|
||||
{
|
||||
"bytes": 1646694,
|
||||
"sha256": "dadb973adbd6feb1b3a6e667ec20c5236202a749e436a9132de49cabd7e05df0",
|
||||
"duration_s": 34.305,
|
||||
"sample_rate": 24000,
|
||||
"channels": 1,
|
||||
"subtype": "PCM_16",
|
||||
"source": "/Volumes/Dock/worstauthor/Documents/Audacity4/BDMurphy_Voice_NOBG_v2.wav"
|
||||
},
|
||||
{
|
||||
"bytes": 4108844,
|
||||
"sha256": "f2ba2b99b9c863a138924cf02af3f05775d41861fc3fdbe897336b95d063752a",
|
||||
"duration_s": 85.6,
|
||||
"sample_rate": 24000,
|
||||
"channels": 1,
|
||||
"subtype": "PCM_16",
|
||||
"source": "/Volumes/Dock/worstauthor/Documents/Audacity4/BDMurphy_Voice_Author_Rainbow_v2.wav"
|
||||
},
|
||||
{
|
||||
"bytes": 4108844,
|
||||
"sha256": "161615c3133ba5f95bf2b7fa71a3d4cc0cf5a4f0b474848ad7419c7adc3ffe50",
|
||||
"duration_s": 85.6,
|
||||
"sample_rate": 24000,
|
||||
"channels": 1,
|
||||
"subtype": "PCM_16",
|
||||
"source": "/Volumes/Dock/worstauthor/Documents/Audacity4/BDMurphy_Voice_Author_Rainbow_v2_air.wav"
|
||||
},
|
||||
{
|
||||
"bytes": 1866910,
|
||||
"sha256": "2f3b6c161593aea335f42fd01159f93c422e4c07e4806739c24fe366c920e7fc",
|
||||
"duration_s": 29.17,
|
||||
"sample_rate": 32000,
|
||||
"channels": 1,
|
||||
"subtype": "PCM_16",
|
||||
"source": "/Volumes/Dock/worstauthor/Documents/BDMurphy_Voice_Author_Rainbow_v2_air2.wav"
|
||||
},
|
||||
{
|
||||
"bytes": 12102,
|
||||
"sha256": "55d84420ddbcbc0e40af9e2e103d4c37a6611d9dfa5a0bdf3ad52801684926a1",
|
||||
"source": "/Volumes/Dock/worstauthor/Documents/Audacity4/BDMurphy_Voice_analysis.md"
|
||||
}
|
||||
],
|
||||
"notes": [
|
||||
"author.pt is git-ignored project cache; this backup is the durable copy.",
|
||||
"The superseded v1 matrix's only prior copy was in an OS temp dir (purgeable).",
|
||||
"verdict: full vmatrix (ref_code + embedding), NOT x_vector_only."
|
||||
],
|
||||
"written_to": [
|
||||
"/Volumes/1TB_APL_Storage/UmbraAudio/backup/author_voice_matrix/20261009T191636Z",
|
||||
"/Volumes/Dock/worstauthor/.umbra/library/author_voice_matrix/20261009T191636Z"
|
||||
],
|
||||
"verification": "ok"
|
||||
}
|
||||
202
tests/test_voice_queue.py
Normal file
202
tests/test_voice_queue.py
Normal file
@@ -0,0 +1,202 @@
|
||||
"""Tests for the voice-regeneration queue.
|
||||
|
||||
The queue exists because regenerating a voice used to be one-at-a-time: the
|
||||
route backgrounded the work but refused a second request with a 409, and the UI
|
||||
had no per-speaker state to show. These cover the queue semantics the reviewer
|
||||
depends on - enqueue-many, run-serially, report per item - without invoking TTS.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from umbraaudio.voices import voice_queue as vq
|
||||
|
||||
|
||||
def _make(tmp_path: Path) -> vq.VoiceQueue:
|
||||
return vq.VoiceQueue(tmp_path, SimpleNamespace(tts=SimpleNamespace()))
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def queue(tmp_path: Path) -> vq.VoiceQueue:
|
||||
"""A queue whose worker is disabled, so items stay ``queued``.
|
||||
|
||||
Starting a real worker would drain the queue on a background thread and make
|
||||
these assertions a race. The enqueue/status/persist logic under test is
|
||||
untouched; the drain itself is covered separately by
|
||||
``test_worker_drains_items_in_order``.
|
||||
"""
|
||||
q = _make(tmp_path)
|
||||
q._ensure_worker = lambda: None # type: ignore[method-assign]
|
||||
return q
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def draining(tmp_path: Path) -> vq.VoiceQueue:
|
||||
"""A queue with a stubbed TTS call but a real worker, to exercise the drain."""
|
||||
q = _make(tmp_path)
|
||||
|
||||
def _no_generation(self: vq.VoiceQueue, item: vq.QueueItem) -> None:
|
||||
with self._lock:
|
||||
item.status = vq.STATUS_DONE
|
||||
item.finished_at = 0.0
|
||||
self._save()
|
||||
|
||||
q._run_item = _no_generation.__get__(q, vq.VoiceQueue) # type: ignore[method-assign]
|
||||
return q
|
||||
|
||||
|
||||
def test_worker_drains_items_in_order(draining: vq.VoiceQueue) -> None:
|
||||
"""The worker must consume queued items one at a time, in enqueue order."""
|
||||
import time
|
||||
|
||||
for name in ("one", "two", "three"):
|
||||
draining.enqueue(name)
|
||||
|
||||
deadline = time.time() + 10
|
||||
while draining.is_busy() and time.time() < deadline:
|
||||
time.sleep(0.05)
|
||||
|
||||
statuses = [i["status"] for i in draining.snapshot()["items"]]
|
||||
assert statuses == [vq.STATUS_DONE] * 3
|
||||
assert draining.is_busy() is False
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ enqueue
|
||||
|
||||
|
||||
def test_enqueue_returns_queued(queue: vq.VoiceQueue) -> None:
|
||||
item = queue.enqueue("Manfred")
|
||||
assert item["status"] == vq.STATUS_QUEUED
|
||||
assert item["speaker_name"] == "Manfred"
|
||||
|
||||
|
||||
def test_enqueue_is_idempotent_for_a_pending_speaker(queue: vq.VoiceQueue) -> None:
|
||||
"""A double-click must not queue the same regeneration twice."""
|
||||
queue.enqueue("Manfred")
|
||||
before = queue.snapshot()["total"]
|
||||
queue.enqueue("Manfred")
|
||||
assert queue.snapshot()["total"] == before
|
||||
|
||||
|
||||
def test_enqueue_can_queue_more_than_one_at_a_time(queue: vq.VoiceQueue) -> None:
|
||||
"""The whole point: a second speaker is accepted while the first runs."""
|
||||
for name in ("A", "B", "C"):
|
||||
assert queue.enqueue(name)["status"] == vq.STATUS_QUEUED
|
||||
assert queue.snapshot()["total"] == 3
|
||||
|
||||
|
||||
def test_enqueue_preserves_order(queue: vq.VoiceQueue) -> None:
|
||||
for name in ("first", "second", "third"):
|
||||
queue.enqueue(name)
|
||||
order = [i["speaker_name"] for i in queue.snapshot()["items"]]
|
||||
assert order == ["first", "second", "third"]
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- status
|
||||
|
||||
|
||||
def test_status_for_unknown_speaker_is_none(queue: vq.VoiceQueue) -> None:
|
||||
assert queue.status_for("nobody") is None
|
||||
|
||||
|
||||
def test_snapshot_counts_only_pending_as_queued(queue: vq.VoiceQueue) -> None:
|
||||
queue.enqueue("A")
|
||||
queue.enqueue("B")
|
||||
snap = queue.snapshot()
|
||||
assert snap["queued"] == 2
|
||||
assert snap["running"] is None
|
||||
assert snap["total"] == 2
|
||||
|
||||
|
||||
def test_is_busy_reflects_pending_work(queue: vq.VoiceQueue) -> None:
|
||||
assert queue.is_busy() is False
|
||||
queue.enqueue("A")
|
||||
assert queue.is_busy() is True
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- cancel
|
||||
|
||||
|
||||
def test_cancel_removes_a_queued_item(queue: vq.VoiceQueue) -> None:
|
||||
queue.enqueue("A")
|
||||
assert queue.cancel("A") is True
|
||||
assert queue.status_for("A") is None
|
||||
|
||||
|
||||
def test_cancel_refuses_a_running_item(queue: vq.VoiceQueue) -> None:
|
||||
queue.enqueue("A")
|
||||
with queue._lock:
|
||||
queue._items[0].status = vq.STATUS_RUNNING
|
||||
assert queue.cancel("A") is False
|
||||
|
||||
|
||||
# -------------------------------------------------------------- persistence
|
||||
|
||||
|
||||
def test_queue_state_is_persisted(tmp_path: Path) -> None:
|
||||
cfg = SimpleNamespace(tts=SimpleNamespace())
|
||||
q = vq.VoiceQueue(tmp_path, cfg)
|
||||
q.enqueue("Persisted")
|
||||
reloaded = vq.VoiceQueue(tmp_path, cfg)
|
||||
assert reloaded.status_for("Persisted") is not None
|
||||
|
||||
|
||||
def test_running_item_becomes_error_after_restart(tmp_path: Path) -> None:
|
||||
"""A process that dies mid-generation must not leave a stuck 'running' tag."""
|
||||
cfg = SimpleNamespace(tts=SimpleNamespace())
|
||||
q = vq.VoiceQueue(tmp_path, cfg)
|
||||
q._items.append(vq.QueueItem(speaker="Ghost", status=vq.STATUS_RUNNING))
|
||||
q._save()
|
||||
|
||||
reloaded = vq.VoiceQueue(tmp_path, cfg)
|
||||
ghost = reloaded.status_for("Ghost")
|
||||
assert ghost["status"] == vq.STATUS_ERROR
|
||||
assert "Interrupt" in ghost["error"]
|
||||
|
||||
|
||||
def test_corrupt_status_file_does_not_crash(tmp_path: Path) -> None:
|
||||
cfg = SimpleNamespace(tts=SimpleNamespace())
|
||||
vq.VoiceQueue(tmp_path, cfg)
|
||||
(tmp_path / "voice_queue.json").write_text("{not json")
|
||||
assert vq.VoiceQueue(tmp_path, cfg).snapshot()["total"] == 0
|
||||
|
||||
|
||||
def test_finished_history_is_trimmed(tmp_path: Path) -> None:
|
||||
"""Terminal items must not accumulate without bound."""
|
||||
cfg = SimpleNamespace(tts=SimpleNamespace())
|
||||
q = vq.VoiceQueue(tmp_path, cfg)
|
||||
for i in range(80):
|
||||
q._items.append(
|
||||
vq.QueueItem(speaker=f"old{i}", status=vq.STATUS_DONE)
|
||||
)
|
||||
q._trim_history()
|
||||
assert len([i for i in q._items if i.status in vq.TERMINAL_STATES]) <= 50
|
||||
|
||||
|
||||
# --------------------------------------------------------------- vocabulary
|
||||
|
||||
|
||||
def test_status_vocabulary_matches_the_controller_mapping() -> None:
|
||||
"""These names are what the controller will map onto node task states."""
|
||||
assert vq.STATUS_QUEUED == "queued"
|
||||
assert vq.STATUS_RUNNING == "running"
|
||||
assert vq.STATUS_DONE == "done"
|
||||
assert vq.STATUS_ERROR == "error"
|
||||
assert {vq.STATUS_DONE, vq.STATUS_ERROR} == vq.TERMINAL_STATES
|
||||
|
||||
|
||||
def test_public_state_excludes_internal_timings() -> None:
|
||||
item = vq.QueueItem(speaker="X", row_index=4)
|
||||
assert set(item.public()) == {
|
||||
"speaker",
|
||||
"speaker_name",
|
||||
"status",
|
||||
"error",
|
||||
"row_index",
|
||||
}
|
||||
300
umbraaudio/voices/voice_queue.py
Normal file
300
umbraaudio/voices/voice_queue.py
Normal file
@@ -0,0 +1,300 @@
|
||||
"""Serial task queue for voice regeneration, with per-item state.
|
||||
|
||||
Why
|
||||
---
|
||||
Regenerating a speaker voice was fire-and-forget: the HTTP route backgrounded
|
||||
the work, but the only observable state was a single global ``active`` flag on
|
||||
:mod:`generation_manager`. The UI could therefore only show a modal, and had to
|
||||
block a second request with a 409 because there was nowhere to put it. Reviewing
|
||||
speaker exceptions meant starting one, staring at a dialog, then navigating back
|
||||
to find out what happened.
|
||||
|
||||
This module is the local half of the intended controller design. The node
|
||||
enqueues work, drains it one item at a time (TTS is serial - the Mac cannot hold
|
||||
two generations), and reports per-item state so the UI can tag each speaker
|
||||
without leaving the page. The enqueue/drain/report shape is deliberately the one
|
||||
the controller will drive remotely, so that replacing the local worker with
|
||||
"POST to controller, await assignment, await result" is a change of *who* runs
|
||||
the drain loop, not a redesign of the queue.
|
||||
|
||||
State is persisted to ``voice_queue.json`` so an interrupted run can report what
|
||||
it was doing after a restart, matching the existing crash-recovery behaviour in
|
||||
:class:`~umbraaudio.voices.generation_manager.GenerationManager`.
|
||||
|
||||
Status vocabulary is deliberately small and stable, because the future
|
||||
controller will need to map it onto node/task/result states:
|
||||
|
||||
``queued`` -> ``running`` -> ``done`` | ``error``
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
STATUS_QUEUED = "queued"
|
||||
STATUS_RUNNING = "running"
|
||||
STATUS_DONE = "done"
|
||||
STATUS_ERROR = "error"
|
||||
|
||||
TERMINAL_STATES = frozenset({STATUS_DONE, STATUS_ERROR})
|
||||
|
||||
SAMPLE_TEXT = (
|
||||
"When the sunlight strikes raindrops in the air, they act as a prism and "
|
||||
"form a rainbow. The rainbow is a division of white light into many "
|
||||
"beautiful colors. These take the shape of a long round arch, with its path "
|
||||
"high above, and its two ends apparently beyond the horizon. There is, "
|
||||
"according to legend, a boiling pot of gold at one end. People look, but no "
|
||||
"one ever finds it. When a man looks for something beyond his reach, his "
|
||||
"friends say he is looking for the pot of gold at the end of the rainbow."
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class QueueItem:
|
||||
"""One speaker's voice regeneration."""
|
||||
|
||||
speaker: str
|
||||
status: str = STATUS_QUEUED
|
||||
error: str = ""
|
||||
enqueued_at: float = field(default_factory=time.time)
|
||||
started_at: float | None = None
|
||||
finished_at: float | None = None
|
||||
# Set by the route so the UI can re-render the row it just replaced.
|
||||
row_index: int | None = None
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return asdict(self)
|
||||
|
||||
@property
|
||||
def pending(self) -> bool:
|
||||
return self.status in (STATUS_QUEUED, STATUS_RUNNING)
|
||||
|
||||
def public(self) -> dict[str, Any]:
|
||||
"""State for the UI. Deliberately excludes timings and internals."""
|
||||
return {
|
||||
"speaker": self.status,
|
||||
"speaker_name": self.speaker,
|
||||
"status": self.status,
|
||||
"error": self.error,
|
||||
"row_index": self.row_index,
|
||||
}
|
||||
|
||||
|
||||
class VoiceQueue:
|
||||
"""Ordered, serial queue of voice-regeneration tasks.
|
||||
|
||||
TTS is strictly serial on this machine, so items run one at a time in
|
||||
enqueue order. Enqueueing never blocks and never fails because something is
|
||||
already running - that is the whole point: a reviewer can queue a second and
|
||||
third speaker while the first is generating.
|
||||
"""
|
||||
|
||||
def __init__(self, data_dir: Path, config: Any) -> None:
|
||||
self._data_dir = data_dir
|
||||
self._config = config
|
||||
self._status_path = data_dir / "voice_queue.json"
|
||||
self._lock = threading.Lock()
|
||||
self._items: list[QueueItem] = []
|
||||
self._wake = threading.Event()
|
||||
self._worker: threading.Thread | None = None
|
||||
self._load()
|
||||
|
||||
# ------------------------------------------------------------ persistence
|
||||
|
||||
def _load(self) -> None:
|
||||
"""Restore queue state. Anything mid-flight is marked interrupted.
|
||||
|
||||
A process that died mid-generation leaves a ``running`` item that no
|
||||
thread will ever finish. Marking it ``error`` is the honest outcome -
|
||||
silently re-running it could write audio nobody asked for twice.
|
||||
"""
|
||||
if not self._status_path.exists():
|
||||
return
|
||||
try:
|
||||
raw = json.loads(self._status_path.read_text())
|
||||
except (OSError, json.JSONDecodeError):
|
||||
logger.warning("voice queue status unreadable; starting empty")
|
||||
return
|
||||
for entry in raw.get("items", []):
|
||||
try:
|
||||
item = QueueItem(**entry)
|
||||
except TypeError:
|
||||
continue
|
||||
if item.status == STATUS_RUNNING:
|
||||
item.status = STATUS_ERROR
|
||||
item.error = "Interrupted by a restart."
|
||||
item.finished_at = time.time()
|
||||
self._items.append(item)
|
||||
|
||||
def _save(self) -> None:
|
||||
payload = {"saved_at": time.time(), "items": [i.as_dict() for i in self._items]}
|
||||
tmp = self._status_path.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(payload, indent=2))
|
||||
tmp.replace(self._status_path)
|
||||
|
||||
# ----------------------------------------------------------------- enqueue
|
||||
|
||||
def enqueue(self, speaker: str, row_index: int | None = None) -> dict[str, Any]:
|
||||
"""Add a speaker to the queue, or return the existing entry.
|
||||
|
||||
Re-enqueueing a speaker that is already queued or running is a no-op
|
||||
that reports the existing state, so a double-click cannot enqueue the
|
||||
same regeneration twice.
|
||||
"""
|
||||
with self._lock:
|
||||
for item in self._items:
|
||||
if item.speaker == speaker and item.pending:
|
||||
return item.public()
|
||||
|
||||
item = QueueItem(speaker=speaker, row_index=row_index)
|
||||
self._items.append(item)
|
||||
self._trim_history()
|
||||
self._save()
|
||||
state = item.public()
|
||||
self._ensure_worker()
|
||||
return state
|
||||
|
||||
def _trim_history(self) -> None:
|
||||
"""Keep finished history bounded so the JSON cannot grow without limit."""
|
||||
terminal = [i for i in self._items if i.status in TERMINAL_STATES]
|
||||
overflow = len(terminal) - 50
|
||||
if overflow <= 0:
|
||||
return
|
||||
drop: set[int] = set()
|
||||
for item in terminal[:overflow]:
|
||||
drop.add(id(item))
|
||||
self._items = [i for i in self._items if id(i) not in drop]
|
||||
|
||||
def cancel(self, speaker: str) -> bool:
|
||||
"""Remove a queued (not yet started) item. Returns True if removed."""
|
||||
with self._lock:
|
||||
before = len(self._items)
|
||||
self._items = [
|
||||
i for i in self._items if not (i.speaker == speaker and i.status == STATUS_QUEUED)
|
||||
]
|
||||
removed = len(self._items) != before
|
||||
if removed:
|
||||
self._save()
|
||||
return removed
|
||||
|
||||
# ------------------------------------------------------------------ status
|
||||
|
||||
def status_for(self, speaker: str) -> dict[str, Any] | None:
|
||||
"""Latest state for one speaker, or None if it has never been queued."""
|
||||
with self._lock:
|
||||
matching = [i for i in self._items if i.speaker == speaker]
|
||||
if not matching:
|
||||
return None
|
||||
return matching[-1].public()
|
||||
|
||||
def snapshot(self) -> dict[str, Any]:
|
||||
"""Everything the UI needs: per-speaker states plus queue totals."""
|
||||
with self._lock:
|
||||
items = [i.public() for i in self._items]
|
||||
pending = [i for i in items if i["status"] in (STATUS_QUEUED, STATUS_RUNNING)]
|
||||
running = [i for i in items if i["status"] == STATUS_RUNNING]
|
||||
return {
|
||||
"items": items,
|
||||
"queued": len(pending),
|
||||
"running": running[0]["speaker_name"] if running else None,
|
||||
"pending": len(pending),
|
||||
"total": len(items),
|
||||
"done": sum(1 for i in items if i["status"] == STATUS_DONE),
|
||||
"errors": sum(1 for i in items if i["status"] == STATUS_ERROR),
|
||||
}
|
||||
|
||||
def is_busy(self) -> bool:
|
||||
with self._lock:
|
||||
return any(i.pending for i in self._items)
|
||||
|
||||
# ------------------------------------------------------------------ worker
|
||||
|
||||
def _ensure_worker(self) -> None:
|
||||
if self._worker is not None and self._worker.is_alive():
|
||||
self._wake.set()
|
||||
return
|
||||
self._worker = threading.Thread(target=self._drain, daemon=True)
|
||||
self._worker.start()
|
||||
|
||||
def _next_queued(self) -> QueueItem | None:
|
||||
for item in self._items:
|
||||
if item.status == STATUS_QUEUED:
|
||||
return item
|
||||
return None
|
||||
|
||||
def _drain(self) -> None:
|
||||
"""Run queued items one at a time until the queue is empty.
|
||||
|
||||
Serial by design: the TTS server holds a model in memory and two
|
||||
concurrent generations contend for it.
|
||||
"""
|
||||
while True:
|
||||
with self._lock:
|
||||
item = self._next_queued()
|
||||
if item is None:
|
||||
return
|
||||
self._run_item(item)
|
||||
|
||||
def _run_item(self, item: QueueItem) -> None:
|
||||
from umbraaudio.voices.generation_manager import GenerationManager
|
||||
from umbraaudio.voices.voice_designer import VoiceDesigner
|
||||
from umbraaudio.voices.voice_profile_manager import get_profile_for_speaker
|
||||
|
||||
with self._lock:
|
||||
item.status = STATUS_RUNNING
|
||||
item.started_at = time.time()
|
||||
item.error = ""
|
||||
self._save()
|
||||
|
||||
gm = GenerationManager(self._data_dir)
|
||||
try:
|
||||
with gm.serialize("voices", 1, f"voice:{item.speaker}"):
|
||||
speaker_info = self._speaker_info(item.speaker)
|
||||
profile = get_profile_for_speaker(
|
||||
item.speaker, speaker_info, self._data_dir, force_rebuild=True
|
||||
)
|
||||
voices_dir = self._data_dir / "voices"
|
||||
voices_dir.mkdir(parents=True, exist_ok=True)
|
||||
VoiceDesigner(self._config.tts).generate_voice_reference(
|
||||
character=profile,
|
||||
output_dir=voices_dir,
|
||||
sample_text=SAMPLE_TEXT,
|
||||
)
|
||||
logger.info("Voice generated for %s", item.speaker)
|
||||
error = ""
|
||||
except Exception as exc: # noqa: BLE001 - surfaced to the UI verbatim
|
||||
error = str(exc)[:300]
|
||||
logger.error("Voice generation failed for %s: %s", item.speaker, exc)
|
||||
|
||||
with self._lock:
|
||||
item.status = STATUS_ERROR if error else STATUS_DONE
|
||||
item.error = error
|
||||
item.finished_at = time.time()
|
||||
self._save()
|
||||
|
||||
def _speaker_info(self, name: str) -> dict[str, Any]:
|
||||
"""Look up the speaker record the profile builder needs.
|
||||
|
||||
Missing or malformed ``speakers.json`` must not crash the worker, so a
|
||||
minimal stub is used; ``get_profile_for_speaker`` tolerates it.
|
||||
"""
|
||||
import json as _json
|
||||
|
||||
path = self._data_dir / "speakers.json"
|
||||
try:
|
||||
data = _json.loads(path.read_text())
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
for speaker in data.get("speakers", []):
|
||||
if speaker.get("primary_name") == name:
|
||||
return speaker
|
||||
return {}
|
||||
@@ -32,11 +32,15 @@ def create_app(config: AppConfig) -> FastAPI:
|
||||
from umbraaudio.voices.generation_manager import GenerationManager
|
||||
from umbraaudio.web.name_token_routes import router as name_token_router
|
||||
from umbraaudio.web.routes import NoActiveProjectError, router
|
||||
from umbraaudio.web.voice_queue_routes import router as voice_queue_router
|
||||
|
||||
app.include_router(router)
|
||||
# Spec 17 Phase 3: orphan name-token review queue lives in its own module so
|
||||
# the UI never has to be merged into the hot routes.py.
|
||||
app.include_router(name_token_router)
|
||||
# Voice-regeneration queue. Separate module for the same reason, and so the
|
||||
# eventual controller can drive the same endpoints remotely.
|
||||
app.include_router(voice_queue_router)
|
||||
|
||||
# ---------------------------------------------------------------- auth gate
|
||||
# Installed as middleware rather than per-route dependencies so that a newly
|
||||
|
||||
@@ -3889,58 +3889,34 @@ async def set_voice_status(request: Request):
|
||||
|
||||
@router.post("/api/speakers/regenerate-voice")
|
||||
async def regenerate_speaker_voice(request: Request):
|
||||
"""Deprecated: use POST /api/voices/queue.
|
||||
|
||||
Regeneration moved to a queue so a second speaker can be enqueued while one
|
||||
is generating, and so the UI can tag each row instead of showing a modal.
|
||||
This endpoint ran its own thread and refused a second request with a 409,
|
||||
which is exactly what the queue replaces. Kept as a thin shim so any
|
||||
bookmarked call keeps working.
|
||||
"""
|
||||
if (blocked := _voice_maintenance_block()) is not None:
|
||||
return blocked
|
||||
|
||||
import threading
|
||||
|
||||
body = await request.json()
|
||||
name = body.get("speaker_name", "").strip()
|
||||
name = (body.get("speaker_name") or "").strip()
|
||||
if not name:
|
||||
return JSONResponse({"error": "speaker_name required"}, status_code=400)
|
||||
|
||||
config = _get_active_config(request)
|
||||
gm = _get_gen_manager(request)
|
||||
from umbraaudio.web.voice_queue_routes import get_voice_queue
|
||||
|
||||
if gm.is_active():
|
||||
return JSONResponse(
|
||||
{"error": "Generation already in progress", "active": True}, status_code=409
|
||||
)
|
||||
|
||||
speakers = _load_speakers(request)
|
||||
speaker_info = None
|
||||
for s in speakers:
|
||||
if s["primary_name"] == name:
|
||||
speaker_info = s
|
||||
break
|
||||
if not speaker_info:
|
||||
return JSONResponse({"error": "speaker not found"}, status_code=404)
|
||||
|
||||
def _regenerate():
|
||||
from umbraaudio.voices.voice_designer import VoiceDesigner
|
||||
from umbraaudio.voices.voice_profile_manager import get_profile_for_speaker
|
||||
|
||||
try:
|
||||
with gm.serialize("voices", 1, f"voice:{name}"):
|
||||
designer = VoiceDesigner(config.tts)
|
||||
voices_dir = config.data_dir / "voices"
|
||||
voices_dir.mkdir(parents=True, exist_ok=True)
|
||||
profile = get_profile_for_speaker(
|
||||
name, speaker_info, config.data_dir, force_rebuild=True
|
||||
)
|
||||
designer.generate_voice_reference(
|
||||
character=profile,
|
||||
output_dir=voices_dir,
|
||||
sample_text="When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. The rainbow is a division of white light into many beautiful colors. These take the shape of a long round arch, with its path high above, and its two ends apparently beyond the horizon. There is, according to legend, a boiling pot of gold at one end. People look, but no one ever finds it. When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow.",
|
||||
)
|
||||
gm.update_item(name, success=True)
|
||||
logger.info("Voice generated for %s (%s)", name, profile.energy)
|
||||
except Exception as e:
|
||||
gm.update_item(name, success=False, error=str(e))
|
||||
logger.error("Voice generation failed for %s: %s", name, e)
|
||||
|
||||
threading.Thread(target=_regenerate, daemon=True).start()
|
||||
return JSONResponse({"ok": True, "speaker_name": name})
|
||||
item = get_voice_queue(request).enqueue(
|
||||
name, row_index=body.get("row_index")
|
||||
)
|
||||
return JSONResponse({
|
||||
"ok": True,
|
||||
"speaker_name": name,
|
||||
"deprecated": True,
|
||||
"queued": True,
|
||||
"item": item,
|
||||
})
|
||||
|
||||
|
||||
@router.post("/api/speakers/generate-all-voices")
|
||||
|
||||
@@ -112,10 +112,22 @@
|
||||
onclick="approveSpeakerVoice('{{ speaker.primary_name|e }}', {{ speaker_idx }})">
|
||||
✓ Approve
|
||||
</button>
|
||||
{# Queue state for this speaker. Rendered empty on load, then filled by
|
||||
refreshVoiceQueue(); the id is what the poller updates in place. #}
|
||||
<span class="badge voice-queue-tag"
|
||||
id="vq-tag-{{ speaker_idx }}"
|
||||
data-speaker="{{ speaker.primary_name|e }}"
|
||||
style="display:none; font-size:0.7rem"></span>
|
||||
<button class="btn btn-generate" style="font-size:0.75rem; padding:0.2rem 0.5rem"
|
||||
onclick="regenerateSpeakerVoice('{{ speaker.primary_name|e }}')">
|
||||
id="vq-btn-{{ speaker_idx }}"
|
||||
onclick="regenerateSpeakerVoice('{{ speaker.primary_name|e }}', {{ speaker_idx }})">
|
||||
↻ Regenerate
|
||||
</button>
|
||||
<button class="btn btn-reject" style="font-size:0.7rem; padding:0.15rem 0.4rem; display:none"
|
||||
id="vq-cancel-{{ speaker_idx }}"
|
||||
onclick="cancelVoiceQueue('{{ speaker.primary_name|e }}', {{ speaker_idx }})">
|
||||
✕
|
||||
</button>
|
||||
<button class="btn btn-generate" style="font-size:0.8rem; padding:0.3rem 0.6rem"
|
||||
onclick="toggleEditSpeaker({{ speaker_idx }})">
|
||||
Edit
|
||||
|
||||
@@ -30,81 +30,14 @@
|
||||
</div>
|
||||
|
||||
<script>
|
||||
var _pendingVoiceRegen = null;
|
||||
var _regenPoll = null;
|
||||
var _regenIdx = null;
|
||||
|
||||
function queueVoiceRegenOnSave(idx, name) {
|
||||
_pendingVoiceRegen = {idx: idx, name: name};
|
||||
}
|
||||
|
||||
function showRegenModal(msg) {
|
||||
document.getElementById('regen-modal-title').textContent = 'Regenerating voice…';
|
||||
document.getElementById('regen-modal-msg').textContent = msg;
|
||||
document.getElementById('regen-modal-spinner').style.display = 'block';
|
||||
document.getElementById('regen-modal-error').style.display = 'none';
|
||||
document.getElementById('regen-modal-close').style.display = 'none';
|
||||
var modal = document.getElementById('regen-modal');
|
||||
modal.style.display = 'flex';
|
||||
}
|
||||
|
||||
function hideRegenModal() {
|
||||
document.getElementById('regen-modal').style.display = 'none';
|
||||
if (_regenPoll) { clearInterval(_regenPoll); _regenPoll = null; }
|
||||
_regenIdx = null;
|
||||
}
|
||||
|
||||
function regenModalFail(msg) {
|
||||
document.getElementById('regen-modal-title').textContent = 'Voice regeneration failed';
|
||||
document.getElementById('regen-modal-spinner').style.display = 'none';
|
||||
var errEl = document.getElementById('regen-modal-error');
|
||||
errEl.style.display = 'block';
|
||||
errEl.textContent = msg;
|
||||
document.getElementById('regen-modal-close').style.display = 'inline-block';
|
||||
}
|
||||
|
||||
function startRegenWithPopup(idx, name) {
|
||||
_regenIdx = idx;
|
||||
showRegenModal('Generating a new voice sample for ' + name + '. This may take a moment…');
|
||||
fetch('/api/speakers/regenerate-voice', {
|
||||
method: 'POST',
|
||||
headers: {'Content-Type': 'application/json'},
|
||||
body: JSON.stringify({speaker_name: name})
|
||||
}).then(function(r) {
|
||||
return r.json().then(function(d) { return {status: r.status, data: d}; });
|
||||
}).then(function(res) {
|
||||
if (res.data.ok) {
|
||||
pollRegenUntilDone(idx, name);
|
||||
} else if (res.status === 409) {
|
||||
regenModalFail('Another generation is already in progress. Wait for it to finish, then use Regenerate again.');
|
||||
} else {
|
||||
regenModalFail(res.data.error || 'Unknown error');
|
||||
}
|
||||
}).catch(function(e) {
|
||||
regenModalFail('Request failed: ' + e);
|
||||
});
|
||||
}
|
||||
|
||||
function pollRegenUntilDone(idx, name) {
|
||||
if (_regenPoll) clearInterval(_regenPoll);
|
||||
_regenPoll = setInterval(function() {
|
||||
fetch('/api/generation/status').then(function(r) { return r.json(); }).then(function(s) {
|
||||
if (s && !s.active) {
|
||||
clearInterval(_regenPoll);
|
||||
_regenPoll = null;
|
||||
var failed = (s.failed || 0) > 0;
|
||||
if (failed) {
|
||||
var errMsg = (s.errors && s.errors.length ? s.errors[0].error : '') || 'unknown error';
|
||||
regenModalFail('Voice regeneration failed: ' + errMsg);
|
||||
} else {
|
||||
hideRegenModal();
|
||||
refreshSpeakerRow(idx);
|
||||
}
|
||||
}
|
||||
}).catch(function() {});
|
||||
}, 2000);
|
||||
}
|
||||
|
||||
// Voice regeneration is queue-driven (see enqueueVoice below); the old
|
||||
// modal-and-poll flow is gone, so nothing blocks the reviewer.
|
||||
function refreshSpeakerRow(idx) {
|
||||
if (idx === null || idx === undefined) {
|
||||
htmx.ajax('GET', '/speakers/partial', {
|
||||
@@ -124,14 +57,11 @@ document.body.addEventListener('htmx:afterSwap', function(e) {
|
||||
initSpeakerNames();
|
||||
bustAudioCache(e.detail.target);
|
||||
}
|
||||
// Check if the swapped element is a speaker row
|
||||
// A swapped speaker row re-reads its queue tag from the snapshot, so queue
|
||||
// state survives an HTMX row swap without a resume handshake.
|
||||
if (e.detail.target.id && e.detail.target.id.indexOf('speaker-row-') === 0) {
|
||||
bustAudioCache(e.detail.target);
|
||||
var pending = _pendingVoiceRegen;
|
||||
_pendingVoiceRegen = null;
|
||||
if (pending && pending.idx !== undefined) {
|
||||
startRegenWithPopup(pending.idx, pending.name);
|
||||
}
|
||||
refreshVoiceQueueTags();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -256,14 +186,129 @@ function approveSpeakerVoice(name, idx) {
|
||||
});
|
||||
}
|
||||
|
||||
function regenerateSpeakerVoice(name) {
|
||||
if (!confirm('Regenerate voice for ' + name + '? This uses current speaker attributes.')) return;
|
||||
startRegenWithPopup(null, name);
|
||||
// Queue-based voice regeneration. The POST returns as soon as the task is
|
||||
// captured, so the page is never blocked by a modal and the reviewer can keep
|
||||
// reviewing other speakers while generation runs.
|
||||
function regenerateSpeakerVoice(name, idx) {
|
||||
enqueueVoice(name, idx === undefined ? null : idx);
|
||||
}
|
||||
|
||||
function generateSingleVoice(name, idx) {
|
||||
if (!confirm('Are you sure you want to start generating a voice sample for ' + name + '?')) return;
|
||||
startRegenWithPopup(idx, name);
|
||||
enqueueVoice(name, idx === undefined ? null : idx);
|
||||
}
|
||||
|
||||
function enqueueVoice(name, idx) {
|
||||
var body = {speaker_name: name};
|
||||
if (idx !== null && idx !== undefined) body.row_index = idx;
|
||||
|
||||
fetch('/api/voices/queue', {
|
||||
method: 'POST',
|
||||
headers: {'Content-Type': 'application/json'},
|
||||
body: JSON.stringify(body)
|
||||
}).then(function(r) {
|
||||
return r.json().then(function(d) { return {status: r.status, data: d}; });
|
||||
}).then(function(res) {
|
||||
if (res.data.ok) {
|
||||
// Show the tag immediately rather than waiting for the next poll.
|
||||
applyQueueState(name, res.data.item.status, '');
|
||||
startVoiceQueuePolling();
|
||||
} else {
|
||||
applyQueueState(name, 'error', res.data.error || 'Could not queue this speaker');
|
||||
}
|
||||
}).catch(function(e) {
|
||||
applyQueueState(name, 'error', 'Request failed: ' + e);
|
||||
});
|
||||
}
|
||||
|
||||
function cancelVoiceQueue(name, idx) {
|
||||
fetch('/api/voices/queue/' + encodeURIComponent(name) + '/cancel', {method: 'POST'})
|
||||
.then(function(r) { return r.json(); })
|
||||
.then(function(d) {
|
||||
if (d.error) {
|
||||
applyQueueState(name, 'error', d.error);
|
||||
} else {
|
||||
applyQueueState(name, '', '');
|
||||
}
|
||||
}).catch(function() {});
|
||||
}
|
||||
|
||||
var VOICE_TAG_STYLES = {
|
||||
queued: {label: 'queued', color: '#a78bfa', bg: '#2e1065'},
|
||||
running: {label: 'working', color: '#67e8f9', bg: '#164e63'},
|
||||
done: {label: 'done', color: '#6ee7b7', bg: '#064e3b'},
|
||||
error: {label: 'failed', color: '#fca5a5', bg: '#450a0a'}
|
||||
};
|
||||
|
||||
function applyQueueState(name, status, error) {
|
||||
var tags = document.querySelectorAll('.voice-queue-tag');
|
||||
for (var i = 0; i < tags.length; i++) {
|
||||
if (tags[i].getAttribute('data-speaker') !== name) continue;
|
||||
var tag = tags[i];
|
||||
var idx = tag.id.replace('vq-tag-', '');
|
||||
var btn = document.getElementById('vq-btn-' + idx);
|
||||
var cancel = document.getElementById('vq-cancel-' + idx);
|
||||
|
||||
if (!status) {
|
||||
tag.style.display = 'none';
|
||||
if (btn) { btn.style.display = ''; btn.disabled = false; }
|
||||
if (cancel) cancel.style.display = 'none';
|
||||
continue;
|
||||
}
|
||||
|
||||
var style = VOICE_TAG_STYLES[status] || VOICE_TAG_STYLES.queued;
|
||||
tag.style.display = 'inline-block';
|
||||
tag.style.background = style.bg;
|
||||
tag.style.color = style.color;
|
||||
tag.textContent = style.label;
|
||||
if (error) tag.title = error; else tag.removeAttribute('title');
|
||||
|
||||
var pending = (status === 'queued' || status === 'running');
|
||||
if (btn) btn.disabled = pending;
|
||||
if (cancel) cancel.style.display = (status === 'queued') ? 'inline-block' : 'none';
|
||||
}
|
||||
}
|
||||
|
||||
var _vqPoll = null;
|
||||
|
||||
function startVoiceQueuePolling() {
|
||||
if (_vqPoll) return;
|
||||
_vqPoll = setInterval(pollVoiceQueue, 2000);
|
||||
pollVoiceQueue();
|
||||
}
|
||||
|
||||
function pollVoiceQueue() {
|
||||
fetch('/api/voices/queue').then(function(r) { return r.json(); }).then(function(s) {
|
||||
if (!s || !s.items) return;
|
||||
var anyPending = false;
|
||||
s.items.forEach(function(item) {
|
||||
applyQueueState(item.speaker_name, item.status, item.error || '');
|
||||
if (item.status === 'queued' || item.status === 'running') anyPending = true;
|
||||
});
|
||||
// Stop polling once nothing is outstanding, and refresh the list so the
|
||||
// new audio file / voice badge appears without a manual reload.
|
||||
if (!anyPending && _vqPoll) {
|
||||
clearInterval(_vqPoll);
|
||||
_vqPoll = null;
|
||||
refreshSpeakerRow(null);
|
||||
}
|
||||
}).catch(function() {});
|
||||
}
|
||||
|
||||
// Re-apply tags without polling. Used after an HTMX row swap and on first load,
|
||||
// so navigating back to the speakers page still shows anything queued earlier.
|
||||
function refreshVoiceQueueTags() {
|
||||
fetch('/api/voices/queue').then(function(r) { return r.json(); }).then(function(s) {
|
||||
if (!s || !s.items) return;
|
||||
s.items.forEach(function(item) {
|
||||
applyQueueState(item.speaker_name, item.status, item.error || '');
|
||||
});
|
||||
var anyPending = s.items.some(function(i) {
|
||||
return i.status === 'queued' || i.status === 'running';
|
||||
});
|
||||
if (anyPending) startVoiceQueuePolling();
|
||||
}).catch(function() {});
|
||||
}
|
||||
|
||||
document.addEventListener('DOMContentLoaded', refreshVoiceQueueTags);
|
||||
</script>
|
||||
{% endblock %}
|
||||
|
||||
128
umbraaudio/web/voice_queue_routes.py
Normal file
128
umbraaudio/web/voice_queue_routes.py
Normal file
@@ -0,0 +1,128 @@
|
||||
"""Voice-regeneration queue endpoints.
|
||||
|
||||
Separate from ``routes.py`` for the same reason as ``name_token_routes``: the
|
||||
review UI queue is new work and should not be entangled with the hot module.
|
||||
|
||||
The three endpoints map onto what the controller will eventually need:
|
||||
|
||||
* ``POST /api/voices/queue`` - capture the task on this node
|
||||
* ``GET /api/voices/queue`` - report the node's queue and per-item state
|
||||
* ``POST /api/voices/queue/{speaker}/cancel`` - withdraw a task before it runs
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
_VOICE_QUEUES: dict[str, object] = {}
|
||||
|
||||
|
||||
def get_voice_queue(request: Request):
|
||||
"""One queue per project directory, cached on app state.
|
||||
|
||||
Keyed by resolved path rather than slug so two projects cannot share a
|
||||
queue if a directory is renamed underneath a session.
|
||||
"""
|
||||
from umbraaudio.config import load_config
|
||||
|
||||
config = request.app.state.config
|
||||
active_slug = request.cookies.get("active_project")
|
||||
cfg = config
|
||||
if active_slug:
|
||||
with contextlib.suppress(Exception):
|
||||
cfg = load_config(active_slug)
|
||||
|
||||
key = str(Path(cfg.data_dir).resolve())
|
||||
store = request.app.state.__dict__.setdefault("voice_queues", {})
|
||||
if key not in store:
|
||||
from umbraaudio.voices.voice_queue import VoiceQueue
|
||||
|
||||
store[key] = VoiceQueue(cfg.data_dir, cfg)
|
||||
return store[key]
|
||||
|
||||
|
||||
@router.post("/api/voices/queue")
|
||||
async def enqueue_voice(request: Request):
|
||||
"""Queue a speaker's voice for regeneration.
|
||||
|
||||
Returns immediately. A second request is accepted even while one is
|
||||
running, which is the behaviour change that lets a reviewer keep working.
|
||||
"""
|
||||
if (blocked := _maintenance_block()) is not None:
|
||||
return blocked
|
||||
|
||||
body = await request.json()
|
||||
name = (body.get("speaker_name") or "").strip()
|
||||
if not name:
|
||||
return JSONResponse({"error": "speaker_name required"}, status_code=400)
|
||||
|
||||
row_index = body.get("row_index")
|
||||
if row_index is not None:
|
||||
try:
|
||||
row_index = int(row_index)
|
||||
except (TypeError, ValueError):
|
||||
row_index = None
|
||||
|
||||
# Validate up front. Letting an unknown name reach the worker produced a bare
|
||||
# KeyError('primary_name') from the profile builder, which the reviewer saw
|
||||
# as a cryptic failed tag rather than "that speaker does not exist".
|
||||
if _lookup_speaker(request, name) is None:
|
||||
return JSONResponse(
|
||||
{"error": f"Speaker '{name}' is not in this project's speakers.json."},
|
||||
status_code=404,
|
||||
)
|
||||
|
||||
queue = get_voice_queue(request)
|
||||
state = queue.enqueue(name, row_index=row_index)
|
||||
return JSONResponse({"ok": True, "queued": True, "item": state})
|
||||
|
||||
|
||||
@router.get("/api/voices/queue")
|
||||
async def queue_status(request: Request):
|
||||
"""Per-speaker queue state, so the UI can tag rows without navigating."""
|
||||
return JSONResponse(get_voice_queue(request).snapshot())
|
||||
|
||||
|
||||
@router.post("/api/voices/queue/{speaker}/cancel")
|
||||
async def cancel_voice(speaker: str, request: Request):
|
||||
"""Withdraw a queued item. Only works before it starts running."""
|
||||
removed = get_voice_queue(request).cancel(speaker)
|
||||
if not removed:
|
||||
return JSONResponse(
|
||||
{"error": "No queued item for that speaker (it may already be running)."},
|
||||
status_code=409,
|
||||
)
|
||||
return JSONResponse({"ok": True, "cancelled": speaker})
|
||||
|
||||
|
||||
def _lookup_speaker(request: Request, name: str) -> dict | None:
|
||||
"""Find a speaker record by primary_name, or None."""
|
||||
import json as _json
|
||||
|
||||
from umbraaudio.config import load_config
|
||||
|
||||
cfg = request.app.state.config
|
||||
active_slug = request.cookies.get("active_project")
|
||||
if active_slug:
|
||||
with contextlib.suppress(Exception):
|
||||
cfg = load_config(active_slug)
|
||||
try:
|
||||
data = _json.loads((Path(cfg.data_dir) / "speakers.json").read_text())
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
for speaker in data.get("speakers", []):
|
||||
if speaker.get("primary_name") == name:
|
||||
return speaker
|
||||
return None
|
||||
|
||||
|
||||
def _maintenance_block():
|
||||
"""Reuse the voice-maintenance 409 so the queue respects the same rule."""
|
||||
from umbraaudio.web.routes import _voice_maintenance_block
|
||||
|
||||
return _voice_maintenance_block()
|
||||
Reference in New Issue
Block a user