UmbraAudio/specs/12-quality.md

Spec: Quality Checking

Overview

Quality checking validates generated audio against configurable thresholds for MOS (Mean Opinion Score), WER (Word Error Rate), speaker similarity, and ACX compliance. Runs both during generation (per-file) and as a post-generation audit.

Components

1. Quality Checker (quality/__init__.py)

class QualityChecker:
    def __init__(self, config: Any = None) -> None

    def check_audio_quality(self, audio_path: Path) -> dict[str, float | None]
    def measure_mos(self, audio_path: Path) -> float | None
    def measure_wer(self, audio_path: Path, reference_text: str) -> float | None

MOS Measurement

def measure_mos(self, audio_path: Path) -> float | None

• Uses speechmos.jnt_fer_2023 (UTMOS model)
• Reads audio via soundfile
• Returns float score (1.0–5.0) or None on failure
• Model loaded lazily on first call

WER Measurement

def measure_wer(self, audio_path: Path, reference_text: str) -> float | None

• Uses OpenAI Whisper ("base" model) for transcription
• Computes WER via Levenshtein distance
• Returns min(wer, 1.0) (capped at 1.0)

Levenshtein Distance:

@staticmethod
def _levenshtein(s1: str, s2: str) -> int

Standard dynamic programming implementation.

Full Quality Check

def check_audio_quality(self, audio_path: Path) -> dict[str, float | None]

Returns:

{
    "mos_score": float | None,
    "wer_score": None,            # Not implemented in current pipeline
    "speaker_similarity": None,   # Not implemented in current pipeline
    "artifact_score": None,       # Not implemented in current pipeline
}

2. Quality Report (quality/report.py)

class QualityReport:
    def __init__(self) -> None
    def add_metric(self, name: str, value: float, threshold: float, passed: bool) -> None
    def evaluate(self, config: QualityConfig) -> bool
    def to_dict(self) -> dict
    def save(self, output_path: Path) -> None
    @classmethod
    def load(cls, path: Path) -> QualityReport

Evaluation logic:

def evaluate(self, config: QualityConfig) -> bool:
    return (
        self.metrics.get("mos_score", 0) >= config.min_mos
        and self.metrics.get("wer_score", 1) <= config.max_wer
        and self.metrics.get("speaker_similarity", 0) >= config.min_speaker_similarity
    )

3. Consistency Checker (quality/consistency.py)

class ConsistencyChecker:
    def check_cross_chapter(
        self,
        chapter_audio_files: list[Path],
        character_name: str,
        threshold: float = 0.85,
    ) -> dict

Algorithm:

1. Extract speaker embedding from each file using SpeechBrain ECAPA-TDNN
2. Compare all embeddings to the first one (reference)
3. Compute cosine similarity for each pair

Returns:

{
    "consistent": bool,        # All similarities >= threshold
    "min_score": float,        # Minimum similarity score
    "threshold": float,        # Comparison threshold
    "scores": list[float],     # All similarity scores
    "character": str,          # Character name
}

4. ACX Compliance (in AudioNormalizer)

# From assembly/normalize.py
def check_acx_compliance(self, audio_path: Path) -> dict

ACX Requirements:

Metric	Range	Check
RMS level	-23.0 to -18.0 dB	-23.0 <= rms_db <= -18.0
Peak level	Below -3.0 dB	peak_db < -3.0

Returns:

{
    "rms_db": float,
    "peak_db": float,
    "rms_in_range": bool,
    "peak_below_limit": bool,
    "compliant": bool,
}

Quality Thresholds (from QualityConfig)

Metric	Threshold	Description
min_mos	3.8	Minimum Mean Opinion Score
min_test_mos	4.0	Minimum MOS for test pass
max_wer	0.03	Maximum Word Error Rate (3%)
min_speaker_similarity	0.80	Minimum speaker embedding similarity
min_test_speaker_similarity	0.85	Minimum for test pass
min_artifact_score	3.5	Minimum artifact detection score

Defect Tracking

Quality issues are logged to the database:

db.log_defect(
    project_id=1,
    stage="quality",
    severity="WARNING",    # or "ERROR", "CRITICAL"
    message="ACX non-compliance: RMS=-24.5dB, Peak=-2.1dB",
    context=None,
    chapter_id=5,
)

Severity levels:

• WARNING — Below threshold but non-blocking (e.g., ACX RMS slightly out of range)
• ERROR — Significant quality issue (e.g., generation failure)
• CRITICAL — Pipeline-level failure (e.g., model loading error)

Defect Output

Defects are written to .umbra/defects.json:

{
    "defects": [
        {
            "stage": "quality",
            "severity": "WARNING",
            "message": "ACX non-compliance: RMS=-24.5dB",
            "chapter_id": 5
        }
    ],
    "summary": {
        "total_defects": 6,
        "warnings": 6,
        "errors": 0,
        "critical": 0
    }
}

Integration Points

Stage	Quality Check	When
Pass 5 (Generate)	File existence check	Per utterance
Post-generation	ACX compliance	Per file
Pass 4 (Test)	Full quality suite	Single chapter
Web UI	Defect display	On demand
Export	Pre-export validation	Before stitching

SpeechBrain ECAPA-TDNN

# From voices/embeddings.py
class EmbeddingManager:
    def _ensure_loaded(self) -> None:
        self._encoder = EncoderClassifier.from_hparams(
            source="speechbrain/spkrec-ecapa-voxceleb"
        )

    def extract(self, audio_path: Path) -> bytes
    def compare(self, emb1: bytes, emb2: bytes) -> float
    def verify_consistency(self, reference, test_audio, threshold=0.85) -> tuple[bool, float]

• Cosine similarity: dot(a,b) / (norm(a) * norm(b))
• Range: [-1, 1] (1 = identical, 0 = unrelated, -1 = inverse)
• Typical threshold: 0.85 for same-speaker verification