Judge a re-flowed passage as a window, not one paragraph at a time
The first version of the drift check asked two yes/no questions per paragraph: is anything missing, is anything extra. Measured on the real ch-15 window it was wrong in both directions: * It flagged five paragraphs that were fine. The model reported "the adverb 'then' is missing" and "the adverb 'Enthusiastically' is missing" on translations that contain both, and it called a singular/plural difference a missing sentence. A stage whose action is to delete a line cannot be allowed to over-report at that rate: five of fifteen flags were wrong. * It missed two paragraphs that are drifted, because those paragraphs are short and their Chinese looks like a plausible rendering of their own English when seen alone. So the check now asks a different question - which paragraph does this translation actually render - and shows the model the neighbouring paragraphs, because that question cannot be answered without them. The verdict is one of aligned / incomplete / drift, and only a translation that belongs to another paragraph is dropped. The paragraphs are then judged as a window. A re-flowed passage cannot be judged one paragraph at a time: the model calls the two paragraphs before a real drift "aligned" when their Chinese is demonstrably the neighbouring paragraph's, and the start of a drift is not knowable from any single paragraph. Inside a window that contains a confirmed drift, a fragment the model called "aligned" is dropped as well; one it called "incomplete" keeps its text and is flagged, because that verdict says the translation really does render its own leading content. On the real ch-15 window this now finds exactly the drift: ten paragraphs (p0095-p0104) dropped, and p0094 kept and flagged because its Chinese is a correct translation of its first two sentences. The earlier version dropped p0094 too, losing two good lines, and kept the two drifted paragraphs the model wrongly called aligned. The cost of the choice is explicit: a fragment inside a broken window can be dropped on an unreliable verdict, so the audiobook has a gap there. The alternative is a line in the audiobook that belongs to a different sentence. 58 tests, ruff clean.
This commit is contained in:
Umbra Tools
@@ -558,6 +558,17 @@ SYSTEM_VERIFIER = """你是一位双语校对，负责检查一段译文是否
|
||||
译文是按段落 id 存储的；如果写译文时把内容写到了相邻段落，id 和内容就会错位。
|
||||
只输出 JSON，不要解释，不要 Markdown 代码围栏。"""
|
||||
|
||||
|
||||
# The three verdicts. The distinction matters: only a paragraph whose
|
||||
# translation belongs to a *different* paragraph may be dropped from the
|
||||
# delivery. A paragraph that merely leaves something out is still the right
|
||||
# translation in the right place, and dropping it would delete a good line and
|
||||
# leave a hole where the audio should have spoken.
|
||||
VERDICT_ALIGNED = "aligned"
|
||||
VERDICT_INCOMPLETE = "incomplete"
|
||||
VERDICT_DRIFT = "drift"
|
||||
VERDICTS = (VERDICT_ALIGNED, VERDICT_INCOMPLETE, VERDICT_DRIFT)
|
||||
|
||||
# A paragraph whose English does not end on a sentence boundary is a fragment:
|
||||
# Pass 0 split the original passage mid-sentence. A fragment is where the
|
||||
# drafter is most likely to re-flow the passage into natural target-language
|
||||
@@ -578,19 +589,50 @@ def is_fragment(text: str) -> bool:
|
||||
return stripped[-1] not in _SENTENCE_END
|
||||
|
||||
|
||||
def verification_prompt(source_text: str, translation: str) -> str:
|
||||
"""Ask whether *translation* renders *source_text* and nothing else."""
|
||||
return f"""英文原文：
|
||||
{source_text}
|
||||
def verification_prompt(
|
||||
source_text: str,
|
||||
translation: str,
|
||||
previous_text: str = "",
|
||||
next_text: str = "",
|
||||
) -> str:
|
||||
"""Ask which paragraph *translation* actually renders.
|
||||
|
||||
中文译文：
|
||||
The neighbouring paragraphs are given because the question "does this
|
||||
translation belong to this paragraph?" cannot be answered without them:
|
||||
a drifted paragraph's translation is a perfectly good translation - of
|
||||
the paragraph next door.
|
||||
"""
|
||||
previous_block = f"\n【前一段原文】\n{previous_text}\n" if previous_text else ""
|
||||
next_block = f"\n【后一段原文】\n{next_text}\n" if next_text else ""
|
||||
return f"""【本段原文】
|
||||
{source_text}
|
||||
{previous_block}{next_block}
|
||||
【本段中文译文】
|
||||
{translation}
|
||||
|
||||
请检查这段译文与原文的对应关系：
|
||||
1. 原文里有没有意思在译文里找不到的内容？（missing）
|
||||
2. 译文里有没有原文没有讲的内容？例如译文里混进了相邻段落的话。（extra）
|
||||
请判断这段中文译文到底对应哪一段原文，三选一：
|
||||
|
||||
只输出 JSON：{{"missing": true 或 false, "extra": true 或 false, "reason": "一句话理由"}}"""
|
||||
- "aligned"：译文对应本段原文，哪怕不完美。措辞不同、近义词、漏掉一个副词、
|
||||
数字或单复数的小差异，都算 aligned。
|
||||
- "incomplete"：译文对应本段原文，但明显少了一部分内容，而且没有混入别段内容。
|
||||
- "drift"：译文对应的是前一段或后一段原文的内容（内容错位）。
|
||||
|
||||
举例（这些都算 aligned，措辞或数量上的小差异不算错位）：
|
||||
* 本段 "The hunter robot holds out a hand and says to her"
|
||||
译文 "机器人伸出双手对她说" —— 单复数不一致，still aligned。
|
||||
* 本段 "Her brow furrowed, then says" 译文 "她眉头紧锁，说道" —— 省略 then，still aligned。
|
||||
* 本段 "...You gave me the helmet..." 译文 "...你给我这顶帽子..." —— 头盔写成帽子，still aligned。
|
||||
|
||||
举例（这些算 drift，因为译文是相邻那一段的内容）：
|
||||
* 本段 "“Yes, Captain.”" 译文 "“上一次有人登上这艘舰桥还是三十四年前，那时是西顿船长在任？”"
|
||||
—— 这是上一段的译文。
|
||||
* 本段 "“The last time someone was on this bridge...?" 译文 "“Captain”字样。她环顾四周……"
|
||||
—— 这是上一段的译文。
|
||||
|
||||
举例（这种算 incomplete：译文是自己这段的内容，但少了后面几句）：
|
||||
* 本段有三句话，译文只译了前两句的意思，译文内容没有混入别段。
|
||||
|
||||
只输出 JSON：{{"kind": "aligned|incomplete|drift", "reason": "一句话理由"}}"""
|
||||
|
||||
|
||||
def verify_translation(
|
||||
@@ -598,39 +640,36 @@ def verify_translation(
|
||||
translation: str,
|
||||
llm: LLM,
|
||||
model: str = DEFAULT_ALIGN_MODEL,
|
||||
) -> tuple[bool, str]:
|
||||
"""Does *translation* render *source_text* exactly?
|
||||
previous_text: str = "",
|
||||
next_text: str = "",
|
||||
) -> tuple[str, str]:
|
||||
"""Which paragraph does *translation* actually render?
|
||||
|
||||
Returns ``(ok, reason)``. ``ok`` is False when the translation is missing
|
||||
content the source has, or carries content the source does not - which is
|
||||
what a paragraph-boundary drift looks like from the inside.
|
||||
Returns ``(verdict, reason)`` where *verdict* is one of :data:`VERDICTS`.
|
||||
A model failure is reported as ``drift``: the delivery must not carry a
|
||||
line nobody checked.
|
||||
"""
|
||||
prompt = verification_prompt(source_text, translation, previous_text, next_text)
|
||||
try:
|
||||
data, _usage = llm.chat_json(
|
||||
SYSTEM_VERIFIER,
|
||||
verification_prompt(source_text, translation),
|
||||
prompt,
|
||||
model=model,
|
||||
temperature=0.0,
|
||||
cache_key=f"verify-{hashlib.sha256((source_text + translation).encode('utf-8')).hexdigest()[:16]}",
|
||||
cache_key=f"verify-{hashlib.sha256(prompt.encode('utf-8')).hexdigest()[:16]}",
|
||||
)
|
||||
except LLMError as exc:
|
||||
return False, f"verification model failed: {exc}"
|
||||
return VERDICT_DRIFT, f"verification model failed: {exc}"
|
||||
|
||||
if not isinstance(data, dict):
|
||||
return False, "verification answer was not an object"
|
||||
missing = bool(data.get("missing"))
|
||||
extra = bool(data.get("extra"))
|
||||
return VERDICT_DRIFT, "verification answer was not an object"
|
||||
kind = str(data.get("kind") or "").strip().lower()
|
||||
reason = str(data.get("reason") or "").strip()
|
||||
if missing or extra:
|
||||
parts = []
|
||||
if missing:
|
||||
parts.append("the translation is missing content the source has")
|
||||
if extra:
|
||||
parts.append("the translation carries content the source does not have")
|
||||
if reason:
|
||||
parts.append(reason)
|
||||
return False, "; ".join(parts)
|
||||
return True, reason
|
||||
if kind not in VERDICTS:
|
||||
# An unrecognised verdict is not a pass. A model that cannot answer
|
||||
# the question has not verified anything.
|
||||
return VERDICT_DRIFT, f"verification answer was not understood: {kind or '<empty>'}"
|
||||
return kind, reason
|
||||
|
||||
|
||||
def _apply_drift_verification(
|
||||
@@ -641,18 +680,72 @@ def _apply_drift_verification(
|
||||
llm: LLM,
|
||||
model: str,
|
||||
) -> None:
|
||||
"""Flag the paragraphs whose translation does not render their own source.
|
||||
"""Check the paragraphs whose translation may have drifted across ids.
|
||||
|
||||
The check starts at a sentence fragment and follows the drift until the
|
||||
passage re-synchronises. A drifted paragraph is marked ``mismatch`` and
|
||||
its segments are dropped: the delivery must not carry text that renders
|
||||
the wrong source utterance, and a visible gap is better than a confident
|
||||
wrong line in the audiobook.
|
||||
The check starts at a sentence fragment - Pass 0 split the original
|
||||
passage mid-sentence there, and a fragment is exactly where an
|
||||
intent-based drafter is told to write natural target-language prose and
|
||||
lose the id-to-text correspondence.
|
||||
|
||||
The paragraphs are then judged as a *window* rather than one at a time,
|
||||
because a re-flowed passage cannot be judged one paragraph at a time:
|
||||
the same paragraph can be called "aligned" and be wrong (the model does
|
||||
exactly that on the two paragraphs before a real drift), and the start of
|
||||
a drift is not knowable from any single paragraph.
|
||||
|
||||
A paragraph the model says belongs to another paragraph is marked
|
||||
``mismatch`` and its segments are dropped: the delivery must not carry
|
||||
text that renders the wrong source utterance, and a visible gap is better
|
||||
than a confident wrong line in the audiobook. A paragraph that is merely
|
||||
incomplete keeps its segments and is flagged for review, unless the
|
||||
window it sits in is already broken.
|
||||
|
||||
The cost of this choice is explicit: a fragment inside a broken window is
|
||||
dropped even when the model called it aligned, so the audiobook has a
|
||||
gap there. The alternative - keeping it - puts a line in the audiobook
|
||||
that belongs to a different sentence, which is worse.
|
||||
"""
|
||||
by_id = {alignment.paragraph_id: alignment for alignment in alignments}
|
||||
drift_active = False
|
||||
paragraphs = list(chapter.get("paragraphs") or [])
|
||||
index_of = {str(p.get("id") or ""): i for i, p in enumerate(paragraphs)}
|
||||
|
||||
for paragraph in chapter.get("paragraphs") or []:
|
||||
# A re-flowed passage is judged as a *window*, not as a series of
|
||||
# independent paragraphs. Two measurements forced this shape:
|
||||
#
|
||||
# * The same paragraph can be judged "aligned" and be wrong. In the real
|
||||
# ch-15 window, the model called the two paragraphs before the drift
|
||||
# "aligned" when their Chinese was demonstrably the neighbouring
|
||||
# paragraph's.
|
||||
# * The start of a drift is not knowable from any single paragraph. The
|
||||
# drafter re-flowed the passage, so the content of paragraph N may sit
|
||||
# under paragraph N+1, and the first symptom can be a long way from the
|
||||
# cause.
|
||||
#
|
||||
# So: judge every candidate, then resolve each window as a whole. A
|
||||
# window that contains a confirmed drift is broken, and inside a broken
|
||||
# window a per-paragraph "aligned" verdict is not trustworthy.
|
||||
window: list[tuple[str, ParagraphAlignment, str, bool, str]] = []
|
||||
window_has_drift = False
|
||||
|
||||
def resolve_window() -> None:
|
||||
"""Apply the window's verdict to the paragraphs inside it."""
|
||||
nonlocal window_has_drift
|
||||
for _paragraph_id, alignment, reason, was_fragment, verdict in window:
|
||||
if alignment.method == "mismatch":
|
||||
continue
|
||||
if window_has_drift and was_fragment and verdict == VERDICT_ALIGNED:
|
||||
# A fragment inside a confirmed drift is exactly where the
|
||||
# re-flow happened, and the model's "aligned" verdict there is
|
||||
# the one that is known to be unreliable. A paragraph the
|
||||
# model called "incomplete" is not dropped on this evidence:
|
||||
# it demonstrably renders its own leading content.
|
||||
alignment.method = "mismatch"
|
||||
alignment.problems = [f"translation is part of a drifted passage: {reason}"]
|
||||
alignment.segments = []
|
||||
window.clear()
|
||||
window_has_drift = False
|
||||
|
||||
for paragraph in paragraphs:
|
||||
paragraph_id = str(paragraph.get("id") or "")
|
||||
alignment = by_id.get(paragraph_id)
|
||||
if alignment is None:
|
||||
@@ -662,18 +755,61 @@ def _apply_drift_verification(
|
||||
if not source_text or not translation:
|
||||
continue
|
||||
|
||||
if not (drift_active or is_fragment(source_text)):
|
||||
fragment = is_fragment(source_text)
|
||||
if not window and not fragment:
|
||||
continue
|
||||
|
||||
ok, reason = verify_translation(source_text, translation, llm, model)
|
||||
if ok:
|
||||
drift_active = False
|
||||
position = index_of.get(paragraph_id, -1)
|
||||
previous_text = ""
|
||||
next_text = ""
|
||||
if 0 <= position - 1 < len(paragraphs):
|
||||
previous_text = str(paragraphs[position - 1].get("text") or "").strip()
|
||||
if 0 <= position + 1 < len(paragraphs):
|
||||
next_text = str(paragraphs[position + 1].get("text") or "").strip()
|
||||
|
||||
verdict, reason = verify_translation(
|
||||
source_text,
|
||||
translation,
|
||||
llm,
|
||||
model,
|
||||
previous_text=previous_text,
|
||||
next_text=next_text,
|
||||
)
|
||||
|
||||
if verdict == VERDICT_ALIGNED:
|
||||
if window and not fragment:
|
||||
# The passage has re-synchronised: this paragraph is a
|
||||
# complete sentence and the model says it is where it
|
||||
# belongs. The window ends here.
|
||||
resolve_window()
|
||||
else:
|
||||
window.append((paragraph_id, alignment, reason, fragment, verdict))
|
||||
continue
|
||||
|
||||
drift_active = True
|
||||
alignment.method = "mismatch"
|
||||
alignment.problems = [f"translation does not render its own source: {reason}"]
|
||||
alignment.segments = []
|
||||
if verdict == VERDICT_DRIFT:
|
||||
window_has_drift = True
|
||||
alignment.method = "mismatch"
|
||||
alignment.problems = [f"translation belongs to another paragraph: {reason}"]
|
||||
alignment.segments = []
|
||||
elif window_has_drift:
|
||||
# Inside a confirmed drift, "my own text, truncated" is part of
|
||||
# the same defect: the drafter re-flowed the passage and this
|
||||
# paragraph is one of the pieces it was cut into. The English
|
||||
# around a drift is often itself a sentence split across
|
||||
# paragraph ids, so a truncated lead-in here is the tail of the
|
||||
# previous paragraph rather than a translation of this one.
|
||||
alignment.method = "mismatch"
|
||||
alignment.problems = [f"translation is part of a drifted passage: {reason}"]
|
||||
alignment.segments = []
|
||||
else:
|
||||
# Nothing has been proven wrong yet, so the text is kept and
|
||||
# flagged: it is still this paragraph's own translation, just
|
||||
# short of the whole of it.
|
||||
alignment.problems = [f"translation is incomplete: {reason}"]
|
||||
|
||||
window.append((paragraph_id, alignment, reason, fragment, verdict))
|
||||
|
||||
resolve_window()
|
||||
|
||||
|
||||
def align_chapter(
|
||||
|
||||
@@ -418,11 +418,11 @@ class DriftVerificationTests(unittest.TestCase):
|
||||
|
||||
def chat_json(self, system: str, user: str, **kwargs: object) -> tuple[dict, object]:
|
||||
self.calls += 1
|
||||
return ({"missing": False, "extra": False, "reason": ""}, None)
|
||||
return ({"kind": "aligned", "reason": ""}, None)
|
||||
|
||||
llm = StubLLM()
|
||||
ok, _reason = verify_translation("The chair in the middle has", "中间的椅子旁", llm)
|
||||
self.assertTrue(ok)
|
||||
verdict, _reason = verify_translation("The chair in the middle has", "中间的椅子旁", llm)
|
||||
self.assertEqual(verdict, "aligned")
|
||||
self.assertEqual(llm.calls, 1)
|
||||
|
||||
def test_a_drifted_translation_is_caught(self) -> None:
|
||||
@@ -435,25 +435,89 @@ class DriftVerificationTests(unittest.TestCase):
|
||||
def chat_json(self, system: str, user: str, **kwargs: object) -> tuple[dict, object]:
|
||||
self.calls += 1
|
||||
return (
|
||||
{"missing": False, "extra": True, "reason": "the text is about another paragraph"},
|
||||
{"kind": "drift", "reason": "the text is about another paragraph"},
|
||||
None,
|
||||
)
|
||||
|
||||
llm = StubLLM()
|
||||
ok, reason = verify_translation(
|
||||
verdict, reason = verify_translation(
|
||||
'"Yes, Captain."',
|
||||
"“上一次有人登上这艘舰桥还是三十四年前，那时是西顿船长在任？”",
|
||||
llm,
|
||||
)
|
||||
self.assertFalse(ok)
|
||||
self.assertEqual(verdict, "drift")
|
||||
self.assertIn("another paragraph", reason)
|
||||
|
||||
def test_an_unanswerable_verification_is_not_a_pass(self) -> None:
|
||||
"""A model that cannot answer the question has verified nothing."""
|
||||
|
||||
class StubLLM:
|
||||
def chat_json(self, system: str, user: str, **kwargs: object) -> tuple[dict, object]:
|
||||
return ({"reason": "I am not sure"}, None)
|
||||
|
||||
verdict, reason = verify_translation("The ship was large.", "这艘船很大。", StubLLM())
|
||||
self.assertEqual(verdict, "drift")
|
||||
self.assertIn("not understood", reason)
|
||||
|
||||
def test_the_neighbouring_paragraphs_are_shown_to_the_model(self) -> None:
|
||||
"""A paragraph's translation cannot be judged without its neighbours."""
|
||||
|
||||
seen: dict[str, str] = {}
|
||||
|
||||
class StubLLM:
|
||||
def chat_json(self, system: str, user: str, **kwargs: object) -> tuple[dict, object]:
|
||||
seen["user"] = user
|
||||
return ({"kind": "aligned", "reason": ""}, None)
|
||||
|
||||
verify_translation(
|
||||
"The chair in the middle has",
|
||||
"中间的椅子旁",
|
||||
StubLLM(),
|
||||
previous_text="On the right, the console says",
|
||||
next_text='"Captain"',
|
||||
)
|
||||
self.assertIn("On the right, the console says", seen["user"])
|
||||
self.assertIn("Captain", seen["user"])
|
||||
|
||||
def test_an_incomplete_paragraph_keeps_its_text_and_is_flagged(self) -> None:
|
||||
"""It is still the right translation in the right place."""
|
||||
|
||||
class StubLLM:
|
||||
def chat_json(self, system: str, user: str, **kwargs: object) -> tuple[dict, object]:
|
||||
return ({"kind": "incomplete", "reason": "漏译了最后一句"}, None)
|
||||
|
||||
chapter = {
|
||||
"id": "ch-1",
|
||||
"paragraphs": [
|
||||
{
|
||||
"id": "ch-1-p0000",
|
||||
"speaker": "narrator",
|
||||
"type": "narration",
|
||||
"text": "The chair in the middle has",
|
||||
"source_indices": [0],
|
||||
"source_ids": ["utt-100"],
|
||||
}
|
||||
],
|
||||
}
|
||||
alignments, report = align_chapter(
|
||||
chapter,
|
||||
{"ch-1-p0000": "中间的椅子旁"},
|
||||
bundle_fixture(),
|
||||
llm=StubLLM(),
|
||||
verify=True,
|
||||
)
|
||||
self.assertNotEqual(alignments[0].method, "mismatch")
|
||||
self.assertTrue(alignments[0].segments)
|
||||
self.assertTrue(alignments[0].needs_review)
|
||||
self.assertIn("incomplete", alignments[0].problems[0])
|
||||
self.assertEqual(report["needs_review"], 1)
|
||||
|
||||
def test_a_drifted_paragraph_is_dropped_from_the_delivery(self) -> None:
|
||||
"""A visible gap is better than a confident wrong line in the audio."""
|
||||
|
||||
class StubLLM:
|
||||
def chat_json(self, system: str, user: str, **kwargs: object) -> tuple[dict, object]:
|
||||
return ({"missing": False, "extra": True, "reason": "belongs to another paragraph"}, None)
|
||||
return ({"kind": "drift", "reason": "belongs to another paragraph"}, None)
|
||||
|
||||
chapter = {
|
||||
"id": "ch-1",
|
||||
@@ -490,7 +554,7 @@ class DriftVerificationTests(unittest.TestCase):
|
||||
|
||||
def chat_json(self, system: str, user: str, **kwargs: object) -> tuple[dict, object]:
|
||||
self.calls += 1
|
||||
return ({"missing": False, "extra": False}, None)
|
||||
return ({"kind": "aligned", "reason": ""}, None)
|
||||
|
||||
chapter = {
|
||||
"id": "ch-1",
|
||||
@@ -519,8 +583,8 @@ class DriftVerificationTests(unittest.TestCase):
|
||||
def chat_json(self, system: str, user: str, **kwargs: object) -> tuple[dict, object]:
|
||||
self.calls += 1
|
||||
if self.calls <= 3:
|
||||
return ({"missing": False, "extra": True, "reason": "drifted"}, None)
|
||||
return ({"missing": False, "extra": False}, None)
|
||||
return ({"kind": "drift", "reason": "drifted"}, None)
|
||||
return ({"kind": "aligned", "reason": ""}, None)
|
||||
|
||||
def para(pid: str, text: str, sid: str, index: int) -> dict:
|
||||
return {
|
||||
@@ -558,6 +622,201 @@ class DriftVerificationTests(unittest.TestCase):
|
||||
# it was checked because the passage had not re-synchronised yet.
|
||||
self.assertEqual(llm.calls, 4)
|
||||
|
||||
def test_an_incomplete_paragraph_inside_a_drift_run_is_part_of_the_drift(self) -> None:
|
||||
"""The drafter re-flowed the passage; this paragraph is one of the pieces.
|
||||
|
||||
The English around a drift is often itself a sentence split across
|
||||
paragraph ids, so a truncated lead-in inside a confirmed drift is the
|
||||
tail of the previous paragraph, not a translation of this one.
|
||||
"""
|
||||
|
||||
class DriftingLLM:
|
||||
def __init__(self) -> None:
|
||||
self.calls = 0
|
||||
|
||||
def chat_json(self, system: str, user: str, **kwargs: object) -> tuple[dict, object]:
|
||||
self.calls += 1
|
||||
if self.calls == 1:
|
||||
return ({"kind": "drift", "reason": "belongs to another paragraph"}, None)
|
||||
if self.calls == 2:
|
||||
return ({"kind": "incomplete", "reason": "only the first phrase"}, None)
|
||||
return ({"kind": "aligned", "reason": ""}, None)
|
||||
|
||||
def para(pid: str, text: str, sid: str, index: int) -> dict:
|
||||
return {
|
||||
"id": pid,
|
||||
"speaker": "narrator",
|
||||
"type": "narration",
|
||||
"text": text,
|
||||
"source_indices": [index],
|
||||
"source_ids": [sid],
|
||||
}
|
||||
|
||||
chapter = {
|
||||
"id": "ch-1",
|
||||
"paragraphs": [
|
||||
para("ch-1-p0000", "The chair in the middle has", "utt-100", 0),
|
||||
para("ch-1-p0001", "on the console.", "utt-101", 1),
|
||||
para("ch-1-p0002", "She looked around and saw dust everywhere.", "utt-102", 2),
|
||||
],
|
||||
}
|
||||
alignments, _report = align_chapter(
|
||||
chapter,
|
||||
{p["id"]: "译文" for p in chapter["paragraphs"]},
|
||||
bundle_fixture(),
|
||||
llm=DriftingLLM(),
|
||||
verify=True,
|
||||
)
|
||||
methods = [a.method for a in alignments]
|
||||
self.assertEqual(methods[0], "mismatch")
|
||||
self.assertEqual(methods[1], "mismatch")
|
||||
self.assertEqual(methods[2], "one_to_one")
|
||||
|
||||
def test_an_incomplete_paragraph_before_any_drift_keeps_its_text(self) -> None:
|
||||
"""Nothing has been proven wrong yet, so the line is kept and flagged."""
|
||||
|
||||
class IncompleteLLM:
|
||||
def __init__(self) -> None:
|
||||
self.calls = 0
|
||||
|
||||
def chat_json(self, system: str, user: str, **kwargs: object) -> tuple[dict, object]:
|
||||
self.calls += 1
|
||||
return ({"kind": "incomplete", "reason": "漏译了最后一句"}, None)
|
||||
|
||||
chapter = {
|
||||
"id": "ch-1",
|
||||
"paragraphs": [
|
||||
{
|
||||
"id": "ch-1-p0000",
|
||||
"speaker": "narrator",
|
||||
"type": "narration",
|
||||
"text": "The chair in the middle has",
|
||||
"source_indices": [0],
|
||||
"source_ids": ["utt-100"],
|
||||
}
|
||||
],
|
||||
}
|
||||
alignments, _report = align_chapter(
|
||||
chapter,
|
||||
{"ch-1-p0000": "中间的椅子旁"},
|
||||
bundle_fixture(),
|
||||
llm=IncompleteLLM(),
|
||||
verify=True,
|
||||
)
|
||||
self.assertNotEqual(alignments[0].method, "mismatch")
|
||||
self.assertTrue(alignments[0].segments)
|
||||
|
||||
def test_a_fragment_called_aligned_inside_a_broken_window_is_dropped(self) -> None:
|
||||
"""The model calls these two "aligned" and they are wrong.
|
||||
|
||||
Measured on the real ch-15 window: the Chinese under the two
|
||||
paragraphs before the drift belongs to the paragraph before them.
|
||||
The per-paragraph verdict is not trustworthy inside a re-flowed
|
||||
passage, so a fragment in a broken window is dropped even when the
|
||||
model says it is where it belongs.
|
||||
"""
|
||||
|
||||
class StubLLM:
|
||||
def __init__(self) -> None:
|
||||
self.calls = 0
|
||||
|
||||
def chat_json(self, system: str, user: str, **kwargs: object) -> tuple[dict, object]:
|
||||
self.calls += 1
|
||||
if self.calls == 1:
|
||||
return ({"kind": "aligned", "reason": ""}, None)
|
||||
if self.calls == 2:
|
||||
return ({"kind": "drift", "reason": "belongs to another paragraph"}, None)
|
||||
return ({"kind": "aligned", "reason": ""}, None)
|
||||
|
||||
def para(pid: str, text: str, sid: str, index: int) -> dict:
|
||||
return {
|
||||
"id": pid,
|
||||
"speaker": "narrator",
|
||||
"type": "narration",
|
||||
"text": text,
|
||||
"source_indices": [index],
|
||||
"source_ids": [sid],
|
||||
}
|
||||
|
||||
chapter = {
|
||||
"id": "ch-1",
|
||||
"paragraphs": [
|
||||
para("ch-1-p0000", '"operations"', "utt-100", 0),
|
||||
para("ch-1-p0001", "On the right, the console says", "utt-101", 1),
|
||||
para("ch-1-p0002", "The chair in the middle has", "utt-102", 2),
|
||||
para("ch-1-p0003", "She grabbed the dusty chair.", "utt-103", 3),
|
||||
],
|
||||
}
|
||||
alignments, report = align_chapter(
|
||||
chapter,
|
||||
{p["id"]: "译文" for p in chapter["paragraphs"]},
|
||||
bundle_fixture(),
|
||||
llm=StubLLM(),
|
||||
verify=True,
|
||||
)
|
||||
methods = [a.method for a in alignments]
|
||||
# The first fragment was called "aligned" but sits in a broken window.
|
||||
self.assertEqual(methods[0], "mismatch")
|
||||
self.assertEqual(methods[1], "mismatch")
|
||||
self.assertEqual(methods[2], "mismatch")
|
||||
# The last paragraph is a complete sentence and re-synchronises the
|
||||
# passage, so it is kept.
|
||||
self.assertEqual(methods[3], "one_to_one")
|
||||
self.assertEqual(report["needs_review"], 3)
|
||||
|
||||
def test_a_fragment_called_incomplete_at_the_head_of_a_window_keeps_its_text(self) -> None:
|
||||
"""It demonstrably renders its own leading content.
|
||||
|
||||
Measured on the real ch-15 window: the first paragraph's Chinese is a
|
||||
correct translation of its first two sentences, and only its third
|
||||
sentence was re-flowed away. Dropping it would lose two good lines to
|
||||
save nothing.
|
||||
"""
|
||||
|
||||
class StubLLM:
|
||||
def __init__(self) -> None:
|
||||
self.calls = 0
|
||||
|
||||
def chat_json(self, system: str, user: str, **kwargs: object) -> tuple[dict, object]:
|
||||
self.calls += 1
|
||||
if self.calls == 1:
|
||||
return ({"kind": "incomplete", "reason": "漏译了第三句"}, None)
|
||||
if self.calls == 2:
|
||||
return ({"kind": "drift", "reason": "belongs to another paragraph"}, None)
|
||||
return ({"kind": "aligned", "reason": ""}, None)
|
||||
|
||||
def para(pid: str, text: str, sid: str, index: int) -> dict:
|
||||
return {
|
||||
"id": pid,
|
||||
"speaker": "narrator",
|
||||
"type": "narration",
|
||||
"text": text,
|
||||
"source_indices": [index],
|
||||
"source_ids": [sid],
|
||||
}
|
||||
|
||||
chapter = {
|
||||
"id": "ch-1",
|
||||
"paragraphs": [
|
||||
para("ch-1-p0000", "Three chairs face forward, each equipped with a screen", "utt-100", 0),
|
||||
para("ch-1-p0001", "The chair in the middle has", "utt-101", 1),
|
||||
para("ch-1-p0002", "She grabbed the dusty chair.", "utt-102", 2),
|
||||
],
|
||||
}
|
||||
alignments, report = align_chapter(
|
||||
chapter,
|
||||
{p["id"]: "译文" for p in chapter["paragraphs"]},
|
||||
bundle_fixture(),
|
||||
llm=StubLLM(),
|
||||
verify=True,
|
||||
)
|
||||
self.assertNotEqual(alignments[0].method, "mismatch")
|
||||
self.assertTrue(alignments[0].segments)
|
||||
self.assertTrue(alignments[0].needs_review)
|
||||
self.assertEqual(alignments[1].method, "mismatch")
|
||||
self.assertEqual(alignments[2].method, "one_to_one")
|
||||
self.assertEqual(report["needs_review"], 2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Reference in New Issue
Block a user