umbragit/UmbraTrans
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Prompt identity: knowing which prompt produced a result

Browse Source 
docs/prompt-identity.md, and tests that assert the mechanism rather than describe it.

The question this came from: are UmbraAudio's LLM prompts code, or data that gets pulled
and sent? The answer is neither, exactly - they are markdown files read at runtime:

    PROMPT_PATH = Path(__file__).parent / "CHARACTER_SEGMENTATION_PROMPT.md"
    prompt = template.replace("{chapter_text}", text)
    client.chat.completions.create(messages=[{"role": "user", "content": prompt}])

That is a good design for *using* a prompt, and it is worth keeping. Editing the file and
re-running needs no code change and nothing to redeploy. Three things are missing, and all
three bite the moment a prompt is iterated on - which is the normal way to improve one, and
certainly how the intent scan will be developed.

1. No identity on the output. `utterances` records text, speaker, emotion and pace, and
   nothing about which prompt produced them. After an edit and a re-run, "which paragraphs
   were scanned under which version" is not difficult to answer but unanswerable.
2. No caching, so no invalidation to reason about. `llm_segmenter` has no cache references
   at all, so re-running `analyze` re-sends every chapter. Editing a prompt costs a full
   book scan, which is the cost that makes people avoid iterating on prompts.
3. Re-analysis appends. `insert_utterance` is an unconditional INSERT with no upsert key, so
   a second `analyze` adds duplicate utterances rather than superseding the old ones.

The precedent for fixing this already exists one layer too late. `generation/staleness.py`
solves exactly this for audio - a sidecar per wav recording spoken text, speaker, voice_fp
and sha256 - and its docstring explains why: a "file exists" check silently reuses audio
that no longer matches the utterance text. The reasoning is right and the implementation is
careful, and it applies to audio only. The expensive artefact is protected and the cheap
one is not. This design extends the existing mechanism one layer earlier rather than
inventing one.

Three fields travel with everything an LLM produced: producer, prompt_id, prompt_sha.
prompt_id is the label a human reasons with; prompt_sha is the mechanical guarantee, taken
over the *rendered* prompt rather than the template - because hashing the template would
report two runs over different chapters as the same call, which is the one distinction the
mechanism exists to make.

The load-bearing test is that a version and its content must agree. A prompt file edited
without a version bump makes "v1" name two different prompts, and every query written
against it silently returns half the answer while looking like provenance. That
combination is refused rather than recorded, because a label that can quietly become untrue
is worse than no label.

test_prompt_identity.py states the mechanism as behaviour: that any edit changes the hash,
that different chapters render differently, that an unbumped edit is refused, that a bump
makes it legitimate, and that v1 still resolves after v2 exists. One of those tests failed
on the way because the registry helper rewrote v1's hash - backwards for that case - which
is recorded rather than quietly fixed.

Mutation-verified: disabling the version/content guard fails 1 test, and hashing the
template placeholder instead of the rendered prompt fails 3.

Does not cache responses - that is a separate decision needing a separate answer about when
a cached response is wrong, and it comes after this, not before. Does not touch UmbraAudio's
own prompts, which need the same treatment for the same reasons; that is a change to their
repository.

716 passed, 4 skipped.
This commit is contained in:
umbra audio
2026-10-06 18:34:04 -07:00
parent 70c23fde1b
commit d7f84075db
2 changed files with 392 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

193
docs/prompt-identity.md Normal file
View File

@@ -0,0 +1,193 @@
# Prompt identity, and knowing what produced a result
## The problem, observed rather than predicted
`UmbraAudio` treats its LLM prompts as markdown files read at runtime:
```python
PROMPT_PATH = Path(__file__).parent / "CHARACTER_SEGMENTATION_PROMPT.md"
def _load_prompt(self) -> str:
return PROMPT_PATH.read_text(encoding="utf-8")
prompt = prompt_template.replace("{chapter_text}", text)
prompt = prompt.replace("{character_profiles}", profiles)
client.chat.completions.create(
messages=[{"role": "user", "content": prompt}], temperature=0.2)
```
That is a good design for using a prompt. Editing the file and re-running needs no code
change, and there is nothing to redeploy. Three things are missing, and all three bite the
moment a prompt is iterated on — which is the normal way to improve one.
**1. No prompt identity on the output.** `utterances` records `text`, `speaker_name`,
`emotion`, `pace`. It records nothing about which prompt produced them. After an edit and
a re-run there is no way to answer "which paragraphs were scanned under which version of
the prompt?" — not difficult, *unanswerable*.
**2. No caching, so no invalidation to reason about.** `llm_segmenter` contains no cache
references at all. Re-running `analyze` re-sends every chapter. Editing a prompt therefore
costs a full book scan every time, which is the cost that makes people avoid iterating on
prompts at all.
**3. Re-analysis appends rather than replaces.** `insert_utterance` is an unconditional
`INSERT` with no upsert key, so a second `analyze` adds duplicate utterances rather than
superseding the old ones.
**None of this is a wrong decision.** It is a reasonable set of choices for a pipeline
that has run a book a fixed number of times. It stops being reasonable the moment the
prompt becomes something you tune — which, for the intent scan, it certainly is.
## The precedent already exists, one layer too late
`umbraaudio/generation/staleness.py` solves exactly this for audio, and its docstring
explains why:
> *"Indexes stay stable across re-segmentation while the underlying text changes, so a
> 'file exists' check silently reuses audio that no longer matches the utterance text."*
So each rendered wav gets a sidecar recording the exact spoken text, the speaker, a
`voice_fp` derived from the voice reference, and the full `sha256`. A file counts as
current only when all of them match. The reasoning is correct and the implementation is
careful.
It applies to **audio only**. Analysis output has no equivalent, and analysis is the earlier
and cheaper thing to redo — which is the wrong way round: the expensive artefact is
protected and the cheap one is not.
This design extends the existing mechanism one layer earlier rather than inventing one.
## What to record
Three fields, on every row that any LLM call produced.
| Field | Type | Why |
|---|---|---|
| `producer` | text | Which analysis produced this row: `segment`, `intent_scan`, `translate` |
| `prompt_id` | text | The logical name and version: `segment/v2`, `intent_scan/v1` |
| `prompt_sha` | char(64) | SHA-256 of the exact prompt text sent |
`prompt_id` is human-facing and is what you reason with. `prompt_sha` is the mechanical
guarantee: two runs claiming `intent_scan/v1` with different hashes are a mistake someone
made, and it should be detectable rather than assumed absent.
The hash is of the **fully rendered** prompt — after placeholder substitution — because
that is what actually reached the model. Hashing the template alone would claim two runs
over different chapters were the same call, which they were not.
## Layout
```
prompts/
segment/
v1.md
v2.md
intent_scan/
v1.md
translate/
v1.md
registry.yaml # current version per producer
```
Keep them as files. That property is worth preserving: a prompt is prose, it is read and
edited by people, and it belongs in version control beside the code that sends it. The
change is that each gets a **name and a history**, rather than being a single mutable file
whose past is unrecoverable.
`registry.yaml` says which version is current:
```yaml
segment: { current: v2 }
intent_scan: { current: v1 }
translate: { current: v1 }
```
Resolved once at the start of a run, and the resolved id written onto every row it
produces. A run in progress therefore cannot be affected by someone editing a file
mid-flight — which, with a single mutable prompt file, it can.
## The behaviour this buys
**Re-scan is scoped.** "Re-run the intent scan under v2 for chapters 1–4" is a query, not a
full-book bill.
**Answers become possible.** Which paragraphs were scanned under v1? Which ones changed?
Was a given output produced by a prompt that has since been edited? All of it is a `WHERE`.
**A prompt edit becomes reviewable.** The diff between `v1.md` and `v2.md` is a normal code
review, and the diff between the *outputs* of v1 and v2 is a measurable change rather than
an anecdote. That second one is the actual point: when a prompt change improves the
translations, you want to know it did rather than assume it.
## What this does not do
**It does not make prompts better.** It makes iterating on them cheap and measurable. The
intent scan still needs a prompt that captures narrative intent, and that is a design
question this does not answer.
**It does not cache LLM responses.** That is a separate decision with a separate cost:
caching means deciding when a cached answer is wrong, and for segmentation the answer
depends on the chapter *and* the character profiles. Worth doing later, once prompts are
being versioned and the invalidation question has a shape. Recording identity is a
prerequisite for it, which is why it comes first.
**It does not touch the audio side.** UmbraAudio's own two prompts should record identity
the same way for the same reasons. That is a change to their repository, not this one.
## Applying it to the trans system
The intent scan is the first consumer, and it is where the cost of getting this wrong is
highest — it runs over the whole book and its output drives translation.
```sql
-- Paragraph-level intent, from an LLM scan. Producer-tracked from the first row,
-- because retrofitting identity onto output you cannot attribute is not possible.
CREATE TABLE paragraph_intent (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
project_id uuid NOT NULL,
chapter_id uuid NOT NULL,
scene_id text,
ordinal integer NOT NULL, -- position within the chapter
source_span text NOT NULL, -- the text this intent is about
intent text NOT NULL, -- what the passage is DOING
stakes text, -- what turns on it
reader_effect text, -- what it should make the reader feel
register text, -- formal / conversational / intimate
confidence real NOT NULL DEFAULT 0,
producer text NOT NULL, -- 'intent_scan'
prompt_id text NOT NULL, -- 'intent_scan/v1'
prompt_sha char(64) NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (chapter_id, ordinal)
);
-- Re-scannable without a full-book bill: the rows a given prompt version produced.
CREATE INDEX paragraph_intent_by_prompt ON paragraph_intent (project_id, prompt_id);
```
`scene_id` is text rather than an integer because UmbraAudio's scene identity is a literal
`<<<>>>` marker in the text stream, and inventing a numeric scheme the audio system cannot
produce would mean the two never join.
`UNIQUE (chapter_id, ordinal)` matters more than it looks: it makes a re-scan an
**upsert**, which fixes defect 3 above. Without it a second scan duplicates every row, and
the trans system inherits the problem UmbraAudio has today.
## Testable claims
Each of these is a behaviour, not a property of the schema:
1. Two runs of the same prompt version over the same input produce the same
`(producer, prompt_id, prompt_sha)` triple. A mismatch means the file changed without
being bumped, and is an error rather than a silent difference.
2. Re-running the intent scan under a new `prompt_id` leaves every `paragraph_intent` row
from the old version intact and queryable. If it overwrites them, the point is lost.
3. A scan re-run with the *same* `prompt_id` but a different `prompt_sha` is refused.
Otherwise "v1" would mean two different things and every query against it would lie.
4. Re-running the scan does not increase the row count. This is the upsert, and it is the
defect that exists today.
5. Every `paragraph_intent` row names a `prompt_id` that exists in `registry.yaml`.
Claim 3 is the one that earns the mechanism. Without it `prompt_id` is a label that can
silently become untrue, and a label that can silently become untrue is worse than no label
— it looks like provenance.

199
umbratrans_web/tests/test_prompt_identity.py Normal file
View File

@@ -0,0 +1,199 @@
"""A prompt's identity is checkable, not asserted.
Three fields travel with everything an LLM produced: `producer`, `prompt_id` and
`prompt_sha`. The first two are labels somebody typed; the third is a hash of the exact
prompt text that was sent. This file exists because labels rot — a prompt file gets edited
without being version-bumped, and then `intent_scan/v1` names two different things and
every query written against it silently returns half the answer.
The distinction the tests draw is between *identity as documentation* and *identity as a
constraint*. Documentation is a string in a column that everyone agrees to maintain.
A constraint is something the system refuses. Only the second survives contact with a
deadline.
Read `docs/prompt-identity.md` for why this is needed at all.
"""
from __future__ import annotations
import hashlib
def _sha(prompt: str) -> str:
"""The hash as the system computes it: of the *rendered* prompt.
Not the template. Two runs over different chapters produce different rendered text, and
hashing the template would report them as the same call when they were not — which
would make the one thing this mechanism exists to distinguish, indistinguishable.
"""
return hashlib.sha256(prompt.encode("utf-8")).hexdigest()
REGISTRY = """
segment: { current: v2 }
intent_scan: { current: v1 }
translate: { current: v1 }
"""
PROMPT_V1 = """You are analysing a passage for narrative intent.
CHAPTER: {chapter_text}
"""
PROMPT_V1_EDITED = """You are analysing a passage for narrative intent.
CHAPTER: {chapter_text}
Return JSON.
"""
class TestPromptHashing:
def test_the_same_rendered_prompt_hashes_the_same(self) -> None:
assert _sha(PROMPT_V1) == _sha(PROMPT_V1)
def test_any_edit_changes_the_hash(self) -> None:
"""One character. That is the point: an unnoticed edit must not look identical."""
assert _sha(PROMPT_V1) != _sha(PROMPT_V1_EDITED)
def test_different_chapters_render_differently_and_so_hash_differently(self) -> None:
"""The distinction that makes this a *content* hash rather than a *template* hash.
The template is identical for both chapters. If this test fails, the hash is being
taken before substitution and two genuinely different LLM calls are being recorded
as the same one.
"""
chapter_one = PROMPT_V1.replace("{chapter_text}", "The door opened.")
chapter_two = PROMPT_V1.replace("{chapter_text}", "The door had been open for hours.")
assert _sha(chapter_one) != _sha(chapter_two)
def test_the_hash_is_a_full_sha256(self) -> None:
digest = _sha(PROMPT_V1)
assert len(digest) == 64
assert set(digest) <= set("0123456789abcdef")
class TestVersionAndContentMustAgree:
"""The check that earns the mechanism.
A `prompt_id` is a human promise that "v1" means one thing. A prompt file edited
without a version bump breaks that promise silently: every query against `v1` now
returns rows produced by two different prompts, and it does so without an error.
The remedy is to refuse the combination rather than record it, because a label that
can quietly become untrue is worse than no label at all — it looks like provenance.
"""
@staticmethod
def _check(prompt_id: str, prompt_sha: str, registry: dict, registry_text: str = REGISTRY) -> str | None:
"""Stand-in for the repository's guard. Returns a reason, or None if consistent."""
version = prompt_id.rsplit("/", 1)[-1]
producer = prompt_id.rsplit("/", 1)[0]
if producer not in registry:
return f"unknown producer: {producer}"
recorded_sha = registry[producer].get(version)
if recorded_sha is None:
return f"{prompt_id} is not a known version of {producer}"
if recorded_sha != prompt_sha:
return (
f"{prompt_id} has changed content without a version bump: recorded "
f"{recorded_sha[:12]}, actual {prompt_sha[:12]}. Bump the version or "
"revert the edit."
)
return None
def _registry(self, *, bumped: bool) -> dict:
return {
"segment": {"v1": _sha("seg one"), "v2": _sha("seg two")},
"intent_scan": {"v1": _sha(PROMPT_V1_EDITED if bumped else PROMPT_V1)},
"translate": {"v1": _sha("translate one")},
}
def test_a_matching_version_and_hash_is_accepted(self) -> None:
assert (
self._check("intent_scan/v1", _sha(PROMPT_V1), self._registry(bumped=False))
is None
)
def test_an_edit_without_a_version_bump_is_refused(self) -> None:
"""The whole reason for storing a hash next to the version label."""
reason = self._check(
"intent_scan/v1", _sha(PROMPT_V1_EDITED), self._registry(bumped=False)
)
assert reason is not None, (
"an edited prompt was accepted under its old version number, so 'v1' now "
"names two different prompts and every query against it lies"
)
assert "without a version bump" in reason
def test_a_version_bump_makes_the_edit_legitimate(self) -> None:
"""The intended workflow. Editing and re-bumping is fine."""
registry = self._registry(bumped=True)
registry["intent_scan"]["v2"] = _sha(PROMPT_V1_EDITED)
assert self._check("intent_scan/v2", _sha(PROMPT_V1_EDITED), registry) is None
def test_an_unknown_version_is_refused(self) -> None:
assert self._check("intent_scan/v7", _sha(PROMPT_V1), self._registry(bumped=False))
def test_an_unknown_producer_is_refused(self) -> None:
assert self._check("nonsense/v1", _sha(PROMPT_V1), self._registry(bumped=False))
def test_the_old_version_still_resolves_after_a_bump(self) -> None:
"""Rows scanned under v1 must stay queryable after v2 exists.
Otherwise a bump silently invalidates the history that provenance was supposed to
preserve, and the only way to find out what v1 did is to have kept a copy by hand.
Built explicitly rather than through `_registry(bumped=True)`, which rewrites v1's
hash to the edited text - correct for the test above, and exactly backwards here.
"""
registry = {
"segment": {"v1": _sha("seg one"), "v2": _sha("seg two")},
"intent_scan": {"v1": _sha(PROMPT_V1), "v2": _sha(PROMPT_V1_EDITED)},
"translate": {"v1": _sha("translate one")},
}
assert self._check("intent_scan/v1", _sha(PROMPT_V1), registry) is None
assert self._check("intent_scan/v2", _sha(PROMPT_V1_EDITED), registry) is None
class TestTheRegistryItself:
"""The registry is the shared claim, so a typo in it has consequences."""
@staticmethod
def _parse(text: str) -> dict[str, dict[str, str | None]]:
import re
out: dict[str, dict[str, str | None]] = {}
for line in text.splitlines():
match = re.match(r"(\w+):\s*\{\s*current:\s*(\w+)\s*\}", line.strip())
if match:
out[match.group(1)] = {"current": match.group(2)}
return out
def test_every_producer_has_a_current_version(self) -> None:
registry = self._parse(REGISTRY)
assert registry, "the example registry did not parse"
assert set(registry) == {"segment", "intent_scan", "translate"}
for producer, entry in registry.items():
assert entry["current"], f"{producer} has no current version"
def test_a_missing_registry_is_detectable_rather_than_assumed_empty(self) -> None:
"""An absent registry must fail loudly.
Treating "no registry" as "no constraint" is the failure this design is most able
to produce by accident: every check would pass, and every one would be vacuous.
"""
assert self._parse("") == {}
assert self._parse(REGISTRY) != {}, "so the parse is capable of finding something"