UmbraTrans/docs/prompt-identity.md

Prompt identity, and knowing what produced a result

The problem, observed rather than predicted

UmbraAudio treats its LLM prompts as markdown files read at runtime:

PROMPT_PATH = Path(__file__).parent / "CHARACTER_SEGMENTATION_PROMPT.md"

def _load_prompt(self) -> str:
    return PROMPT_PATH.read_text(encoding="utf-8")

prompt = prompt_template.replace("{chapter_text}", text)
prompt = prompt.replace("{character_profiles}", profiles)
client.chat.completions.create(
    messages=[{"role": "user", "content": prompt}], temperature=0.2)

That is a good design for using a prompt. Editing the file and re-running needs no code change, and there is nothing to redeploy. Three things are missing, and all three bite the moment a prompt is iterated on — which is the normal way to improve one.

1. No prompt identity on the output. utterances records text, speaker_name, emotion, pace. It records nothing about which prompt produced them. After an edit and a re-run there is no way to answer "which paragraphs were scanned under which version of the prompt?" — not difficult, unanswerable.

2. No caching, so no invalidation to reason about. llm_segmenter contains no cache references at all. Re-running analyze re-sends every chapter. Editing a prompt therefore costs a full book scan every time, which is the cost that makes people avoid iterating on prompts at all.

3. Re-analysis appends rather than replaces. insert_utterance is an unconditional INSERT with no upsert key, so a second analyze adds duplicate utterances rather than superseding the old ones.

None of this is a wrong decision. It is a reasonable set of choices for a pipeline that has run a book a fixed number of times. It stops being reasonable the moment the prompt becomes something you tune — which, for the intent scan, it certainly is.

The precedent already exists, one layer too late

umbraaudio/generation/staleness.py solves exactly this for audio, and its docstring explains why:

"Indexes stay stable across re-segmentation while the underlying text changes, so a 'file exists' check silently reuses audio that no longer matches the utterance text."

So each rendered wav gets a sidecar recording the exact spoken text, the speaker, a voice_fp derived from the voice reference, and the full sha256. A file counts as current only when all of them match. The reasoning is correct and the implementation is careful.

It applies to audio only. Analysis output has no equivalent, and analysis is the earlier and cheaper thing to redo — which is the wrong way round: the expensive artefact is protected and the cheap one is not.

This design extends the existing mechanism one layer earlier rather than inventing one.

What to record

Three fields, on every row that any LLM call produced.

Field	Type	Why
producer	text	Which analysis produced this row: segment, intent_scan, translate
prompt_id	text	The logical name and version: segment/v2, intent_scan/v1
prompt_sha	char(64)	SHA-256 of the exact prompt text sent

prompt_id is human-facing and is what you reason with. prompt_sha is the mechanical guarantee: two runs claiming intent_scan/v1 with different hashes are a mistake someone made, and it should be detectable rather than assumed absent.

The hash is of the fully rendered prompt — after placeholder substitution — because that is what actually reached the model. Hashing the template alone would claim two runs over different chapters were the same call, which they were not.

Layout

prompts/
  segment/
    v1.md
    v2.md
  intent_scan/
    v1.md
  translate/
    v1.md
  registry.yaml        # current version per producer

Keep them as files. That property is worth preserving: a prompt is prose, it is read and edited by people, and it belongs in version control beside the code that sends it. The change is that each gets a name and a history, rather than being a single mutable file whose past is unrecoverable.

registry.yaml says which version is current:

segment:      { current: v2 }
intent_scan:  { current: v1 }
translate:    { current: v1 }

Resolved once at the start of a run, and the resolved id written onto every row it produces. A run in progress therefore cannot be affected by someone editing a file mid-flight — which, with a single mutable prompt file, it can.

The behaviour this buys

Re-scan is scoped. "Re-run the intent scan under v2 for chapters 1–4" is a query, not a full-book bill.

Answers become possible. Which paragraphs were scanned under v1? Which ones changed? Was a given output produced by a prompt that has since been edited? All of it is a WHERE.

A prompt edit becomes reviewable. The diff between v1.md and v2.md is a normal code review, and the diff between the outputs of v1 and v2 is a measurable change rather than an anecdote. That second one is the actual point: when a prompt change improves the translations, you want to know it did rather than assume it.

What this does not do

It does not make prompts better. It makes iterating on them cheap and measurable. The intent scan still needs a prompt that captures narrative intent, and that is a design question this does not answer.

It does not cache LLM responses. That is a separate decision with a separate cost: caching means deciding when a cached answer is wrong, and for segmentation the answer depends on the chapter and the character profiles. Worth doing later, once prompts are being versioned and the invalidation question has a shape. Recording identity is a prerequisite for it, which is why it comes first.

It does not touch the audio side. UmbraAudio's own two prompts should record identity the same way for the same reasons. That is a change to their repository, not this one.

Applying it to the trans system

The intent scan is the first consumer, and it is where the cost of getting this wrong is highest — it runs over the whole book and its output drives translation.

-- Paragraph-level intent, from an LLM scan. Producer-tracked from the first row,
-- because retrofitting identity onto output you cannot attribute is not possible.
CREATE TABLE paragraph_intent (
    id                uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    project_id        uuid NOT NULL,
    chapter_id        uuid NOT NULL,
    scene_id          text,
    ordinal           integer NOT NULL,        -- position within the chapter
    source_span       text NOT NULL,           -- the text this intent is about
    intent            text NOT NULL,           -- what the passage is DOING
    stakes            text,                   -- what turns on it
    reader_effect     text,                   -- what it should make the reader feel
    register          text,                   -- formal / conversational / intimate
    confidence        real NOT NULL DEFAULT 0,
    producer          text NOT NULL,          -- 'intent_scan'
    prompt_id         text NOT NULL,          -- 'intent_scan/v1'
    prompt_sha        char(64) NOT NULL,
    created_at        timestamptz NOT NULL DEFAULT now(),
    UNIQUE (chapter_id, ordinal)
);

-- Re-scannable without a full-book bill: the rows a given prompt version produced.
CREATE INDEX paragraph_intent_by_prompt ON paragraph_intent (project_id, prompt_id);

scene_id is text rather than an integer because UmbraAudio's scene identity is a literal <<<>>> marker in the text stream, and inventing a numeric scheme the audio system cannot produce would mean the two never join.

UNIQUE (chapter_id, ordinal) matters more than it looks: it makes a re-scan an upsert, which fixes defect 3 above. Without it a second scan duplicates every row, and the trans system inherits the problem UmbraAudio has today.

Testable claims

Each of these is a behaviour, not a property of the schema:

1. Two runs of the same prompt version over the same input produce the same (producer, prompt_id, prompt_sha) triple. A mismatch means the file changed without being bumped, and is an error rather than a silent difference.
2. Re-running the intent scan under a new prompt_id leaves every paragraph_intent row from the old version intact and queryable. If it overwrites them, the point is lost.
3. A scan re-run with the same prompt_id but a different prompt_sha is refused. Otherwise "v1" would mean two different things and every query against it would lie.
4. Re-running the scan does not increase the row count. This is the upsert, and it is the defect that exists today.
5. Every paragraph_intent row names a prompt_id that exists in registry.yaml.

Claim 3 is the one that earns the mechanism. Without it prompt_id is a label that can silently become untrue, and a label that can silently become untrue is worse than no label — it looks like provenance.