master
30 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
umbra audio
|
3e0568b9d1 |
Generate backend endpoints for language-project workflow
Add REST endpoints and frontend wiring for per-language project actions:
- POST /api/workspace/{project}/translation/{lang}/queue — queue translation
- POST /api/workspace/{project}/audio/{lang}/queue — queue audio generation
- GET/POST /api/workspace/{project}/format/{lang} — format metadata (EPUB/PDF)
- GET /api/workspace/{project}/review/{lang}/chapters/{chapter} — utterance review
- GET/POST /api/workspace/{project}/front-matter/{lang} — front matter editor
- GET /api/workspace/{project}/chapters/{lang} — chapter list
- GET /api/workspace/{project}/chapter-content/{lang}/{chapter} — chapter text
Frontend fixes:
- Review page shows per-language text (English source, Chinese translation, etc.)
- Non-translated languages show 'Queue translation' button only
- Speaker data from English source matched by paragraph ID
- UmbraAudio column layout: Speaker, Variation, Text, Actions
- Chapter tree in left sidebar: Front matter, Back matter, Body
- Audio Format label at top of sidebar
- Pandoc formatting toolbar for front matter text editing
|
||
|
umbra audio
|
dda079f605 |
commit audio ingest: apply 0008+0009 migrations, wire Postgres adapter to commit path
- Apply migration 0008_audio_source_import (additive schema for source import) - Apply migration 0009_translation_import_runs (ledger table for import runs) - Wire PostgresAudioImportStore into HTTP commit path with org from node credential - Commit path uses PG store with org from node, no payload org - Bundle hash 21ce5687… remains frozen until v2; no authority transferred |
||
|
umbra audio
|
e35d7a144f | Add Apache-2.0 LICENSE, NOTICE, and SPDX headers | ||
|
umbra audio
|
d7f84075db |
Prompt identity: knowing which prompt produced a result
docs/prompt-identity.md, and tests that assert the mechanism rather than describe it.
The question this came from: are UmbraAudio's LLM prompts code, or data that gets pulled
and sent? The answer is neither, exactly - they are markdown files read at runtime:
PROMPT_PATH = Path(__file__).parent / "CHARACTER_SEGMENTATION_PROMPT.md"
prompt = template.replace("{chapter_text}", text)
client.chat.completions.create(messages=[{"role": "user", "content": prompt}])
That is a good design for *using* a prompt, and it is worth keeping. Editing the file and
re-running needs no code change and nothing to redeploy. Three things are missing, and all
three bite the moment a prompt is iterated on - which is the normal way to improve one, and
certainly how the intent scan will be developed.
1. No identity on the output. `utterances` records text, speaker, emotion and pace, and
nothing about which prompt produced them. After an edit and a re-run, "which paragraphs
were scanned under which version" is not difficult to answer but unanswerable.
2. No caching, so no invalidation to reason about. `llm_segmenter` has no cache references
at all, so re-running `analyze` re-sends every chapter. Editing a prompt costs a full
book scan, which is the cost that makes people avoid iterating on prompts.
3. Re-analysis appends. `insert_utterance` is an unconditional INSERT with no upsert key, so
a second `analyze` adds duplicate utterances rather than superseding the old ones.
The precedent for fixing this already exists one layer too late. `generation/staleness.py`
solves exactly this for audio - a sidecar per wav recording spoken text, speaker, voice_fp
and sha256 - and its docstring explains why: a "file exists" check silently reuses audio
that no longer matches the utterance text. The reasoning is right and the implementation is
careful, and it applies to audio only. The expensive artefact is protected and the cheap
one is not. This design extends the existing mechanism one layer earlier rather than
inventing one.
Three fields travel with everything an LLM produced: producer, prompt_id, prompt_sha.
prompt_id is the label a human reasons with; prompt_sha is the mechanical guarantee, taken
over the *rendered* prompt rather than the template - because hashing the template would
report two runs over different chapters as the same call, which is the one distinction the
mechanism exists to make.
The load-bearing test is that a version and its content must agree. A prompt file edited
without a version bump makes "v1" name two different prompts, and every query written
against it silently returns half the answer while looking like provenance. That
combination is refused rather than recorded, because a label that can quietly become untrue
is worse than no label.
test_prompt_identity.py states the mechanism as behaviour: that any edit changes the hash,
that different chapters render differently, that an unbumped edit is refused, that a bump
makes it legitimate, and that v1 still resolves after v2 exists. One of those tests failed
on the way because the registry helper rewrote v1's hash - backwards for that case - which
is recorded rather than quietly fixed.
Mutation-verified: disabling the version/content guard fails 1 test, and hashing the
template placeholder instead of the rendered prompt fails 3.
Does not cache responses - that is a separate decision needing a separate answer about when
a cached response is wrong, and it comes after this, not before. Does not touch UmbraAudio's
own prompts, which need the same treatment for the same reasons; that is a change to their
repository.
716 passed, 4 skipped.
|
||
|
umbra audio
|
70c23fde1b |
A precise request for the audio system's data, plus a validator to check it against
Asks for what is needed to write a correct mapping, in a form that can be verified rather than read. Two documents: docs/audio-system-data-request.md - what to send, in what format, and why that format. docs/audio-system-bundle-validator.py - ships *with* the request rather than after it, so the sender can check their own work before it travels. Standard library only: whoever runs it runs it on the audio system, and adding a dependency to someone's working machine to validate a file is not a reasonable thing to ask. That means the schema checking is the subset possible without a library - a smoke alarm, not an inspector - and the file says so rather than implying more. The reasoning behind the format is the substance. Prose cannot be validated: every mismatch is something a human has to notice, and the ones that get missed are the interesting ones - derived chapter order, two fields that mean slightly different things, a path relative to something unspecified. A validator that names the record and the field turns a round trip into a checklist, and reports every problem rather than stopping at the first, because a validator that stops at the first failure makes a three-round-trip conversation. Also asks for the things that are cheapest to supply and most expensive to guess: the vocabulary table (guessing terminology is how a mapping looks correct and reads the wrong field), which fields are derived (a derived chapter order treated as authoritative will disagree the first time somebody reorders a chapter by hand), and what the TTS engine actually is - fork or patch, what changed, whether it loads a model and how big. That last one decides whether a 48 GB CPU-only node can run it at all, so the speak work function cannot be written until it is answered. Explicitly not asked for: media files, source code, screenshots, or anything the sender would have to reconstruct. "We do not record this" is a complete answer and far better than an estimate presented as fact. The validator was itself tested against a deliberately broken bundle, which is the only way to know a check can fail. It caught a dangling reference, a unitless duration, two unanchored paths and a planted credential, and passed a correct bundle. Two flaws in it were found that way and fixed: it reported the same path problem twice, and every credential twice, because a tool whose only job is to be trusted about specifics loses credibility the first time it says the same thing twice. 704 passed, 4 skipped - no product code touched. |
||
|
umbra audio
|
359e566a1f |
Record the TTS engine decision and what it still depends on
Qwen3-TTS as modified on the audio node, kept. Quality over time is the principle that settles it, and the engine already in use is the one worth keeping. The modifications are on the audio node and unknown here, so the speak work function cannot be written blind. Listed what is needed: what changed and why, whether it is a fork or a carryable patch, the input contract, and whether it loads a model and how large - which decides whether node 238 can run it at all. Parallel synthesis is a requirement, not an optimisation. The mechanism exists already (submit_children and the decompose route), which means the work-function protocol has to be designed for decomposition rather than have parallelism retrofitted into it. |
||
|
umbra audio
|
4ce121470e |
A node may do work for any tenant, and the pipeline plan
`claim_job` took an `organization_id` and matched it against the job's. Read plainly that says: a machine may only do work for the tenant that owns it. Both paired nodes belong to BDMurphy and The Last Dwarf belongs to WorstAuthor, so a node could never do a word of the audio node's work however idle it was. The only way to have found that out was to queue the job and watch it sit there. `place()` had the same fault - it listed candidate nodes by organization - so the scheduler could not use the fleet across tenants either. Both are fixed. A node is a machine. It belongs to whoever administers it, not to a tenant, and a tenant's data reaches it inside one job's payload for the duration of that job, never by widening what the node itself can see. `kind` and the placement requirements decide what a node may do; ownership decides who may switch it off. What did not change: a node still returns one job's payload and nothing else, so it still cannot enumerate another tenant's work. test_node_tenancy.py asserts the behaviour, and was confirmed to detect the old one: five of its seven tests fail when the tenant filter is restored. Also adds docs/plan-translation-pipeline.md, which separates the critical path - the work-function protocol, TTS, assembly, lineage, and one finished episode of The Last Dwarf - from the work that can proceed beside it. Writing those tests cost more time than it should have and the reason is worth recording: four of them failed for reasons that had nothing to do with tenancy. A job left behind by one test is claimed by the next, which reads as the rule breaking. `TRUNCATE ... CASCADE` on a table shared with other fixtures destroyed `project_publishers` and surfaced two files away as "relation does not exist". The queue is now left alone and the claims filter on a job kind only this file creates, so a neighbour's leftovers cannot be returned and the assertions can be exact. 547 passed, 29 skipped. |
||
|
umbra audio
|
2cdebf5ac3 |
Schema per organization: the migration, and why these four tables
Umbratools_2.md 5.0 replaces shared-tables-plus-RLS with one schema per
organization, because RLS was measured to be unenforceable twice: the
application connects as the table owner so the policies did not apply to it
(gate G2), and one app function was PUBLIC-executable until it was fixed by
hand against one database (defect B4). A model whose correctness depends on a
grant somebody remembers to make is not a boundary.
The classification comes before the SQL because the SQL's blast radius depends
on it, and the obvious reading of "move the tenant tables" is wrong for eight of
the tables carrying an organization_id. Four move: projects, project_authors,
project_publishers, invitations. The other twenty-six stay, each for a reason
read out of the actual columns and foreign keys rather than the table names.
Two findings the plan did not price, both recorded in the classification:
- user_roles cannot move. It is the index that resolves a session to an
organization, so putting it in a tenant schema means finding the tenant by
searching every tenant schema, on the login path. It also holds
platform-scope rows, which have no organization to belong to. So one table
stays shared, and 5.0's "no table is shared between organizations" is not
fully achievable while the session-to-tenant resolution is a table.
- moving projects forces dropping three foreign keys that point at it from
artifacts, jobs and user_roles, because a foreign key cannot name a table
whose schema depends on the row. That is the one place the decision costs a
database-enforced guarantee, and it is written down rather than papered over
with a trigger that would have to rediscover which tenant is meant.
The migration names each tenant schema from the organization id rather than
storing it, so a name cannot disagree with the rows and the rollback can rebuild
the organization_id column from the schema. It is idempotent - verified by
applying it twice - and the rollback is a file that runs rather than a comment
block, because a rollback for a data migration that only exists as a comment is a
rollback nobody runs. Both were applied to the scratch database, rolled back, and
applied again.
|
||
|
umbra audio
|
11f27f5ef9 |
Capability profile for node 238, measured over SSH
Reached 238 via public-key enrolment - no password anywhere. The first attempt failed because the key had never landed in authorized_keys; the second worked once the fingerprint matched. Findings worth keeping beyond this node: The VM reports 'AMD Ryzen 9 3900X' and then lists 65 CPUID flags that stop at SSE4.2. No avx, no avx2, no fma. That silicon has AVX2 and FMA, so VirtualBox is masking them and /proc/cpuinfo is a declaration, not a measurement. Recorded as *unverified* rather than absent, because confirming it needs an actual AVX instruction executed and there is no gcc on the guest - a flag list cannot settle it. This is the defect the compute design already has, one layer down. The agent sets throughput_factor by string-matching a GPU model (40.0 if '4070' in name else 20.0), which is a declared factor wearing a measured one's name. Here the hypervisor makes the same kind of lie. Capability discovery therefore has to probe by executing instructions and treat an unverifiable feature as unavailable. Also: cpu.max is 'max 100000' so the guest is not CPU-capped, memory is 47.0 GiB unballooned, no GPU and no nested virt flags, and umbra has no sudo. Recorded in docs/ node-238-capability-profile.md with how each value was obtained. |
||
|
umbra audio
|
0c35eb2d03 |
Allow the address the service binds to as a Host value
The deployment listens on 192.168.123.14 and TrustedHostMiddleware rejected that same address, so every request by IP returned a bare 400 Invalid host header - including /api/health, which is indistinguishable from the service being down. It has cost this project twice. Found when the first pairing attempt from 192.168.123.238 came back 400 with no body and no logged detail. The pairing code was fine and went unspent; the request never reached the route. Both umbratrans.bdmurphy.me and 192.168.123.14 are now allowed. The 0600 on the env file is unchanged, and the correction is recorded in the alignment doc's transport note so the audio node is not sent looking for a service outage. |
||
|
umbra audio
|
2ff322dcd2 |
Close all thirteen app functions in the migration, not by hand
B4 was recorded as open against the deployment. It was not open there - the ten original functions had been closed by hand - but it was open everywhere else. A database restored from a backup of the deployment comes back with 10 of 13 PUBLIC-executable, because a hand-applied GRANT is a property of one database and does not survive a restore. Observed on scratch, built from the pre-0003 dump. So 0003 now revokes from PUBLIC on all thirteen, by full signature taken from pg_proc.oid::regprocedure: a bare () does not resolve for a function taking one argument, and REVOKE on a nonexistent name aborts the whole migration. Three findings while doing it, none of which would have shown up without running the tests: - My own three guard functions went in PUBLIC-executable, reproducing the exact defect B4 is about. - Revoking PUBLIC without regranting broke five tenancy tests with a permission denied error on the platform-admin helper. Every RLS policy calls it, so a missing grant here looks like a broken policy rather than a missing grant. Regranted to the roles that actually read and write, guarded by EXISTS so a role added later is covered by the next migration rather than by luck. - The revoke block originally sat before the functions were created, so it failed on a fresh restore and passed on the already-migrated database - passing by accident, the failure mode this project keeps hitting. Both revoke blocks now sit after the triggers, and the fresh-restore path is the one verified. Verified by restoring the pre-0003 backup into scratch, applying 0003, and running the suite: 0 of 13 PUBLIC-executable, 4 guards, 474 passed, 27 skipped. Re-applying is a no-op. Deployment data unchanged (users=2 orgs=3 audit=43 perms=20); health 200, heartbeat 401 without a credential and 200 with one, so the revoke broke nothing. G2 is still open: 26 tables have RLS enabled but not forced, and the application connects as the owner. |
||
|
umbra audio
|
c0a1971a52 |
Stage 1 credential issued; record what it was verified not to do
The doc previously said 'ask and we will issue'. It now carries the node id, the key prefix, the four observed checks, and the heartbeat payload shape. The key itself is not in the document. It is in a 0600 file on oldbox, because a credential in a git-tracked design document is a credential in every clone and in every paste of it. The two rows worth reading are the third and fourth: a heartbeat-only credential claiming a job is refused 403, and the job is still queued with no lease owner. The refusal happens at the scope check, before the job is touched - a refusal that had already half-claimed would leave the queue believing a node is working when it is not. |
||
|
umbra
|
898906ab04 |
Forward-only link agreed, and the connection sequence the audio node asked for
D16: a book_local character revealed to recur keeps its released entity and gains a recurs_as edge, rather than being merged. Its cost is written down rather than discovered later - a retelling from the earlier book's point of view treats the two as different people. Chosen because the freeze is what protects reproducibility and a merge puts it at risk for a gain that is real but retrospective. Asked whether the document was ready to feed back and how the audio node connects. Reading it against that question found four gaps, all of which were mine: - One report skeleton was given for nine phases, which specifies nothing. evidence now has a tabulated shape per phase, so the reports are mechanically comparable rather than approximately so. - The identity probe said "we send you a fixed set" without saying how the set was fixed, which would have made phase 3 either unreproducible or impossible. Now: you pick the project, sort by source_unit_key, every 17th, cap 200, and report the stride. - Section 9 said "contact the controller on 192.168.123.14:8442" and stopped, which is how this session's own 400s happened twice. - There was no connection sequence at all. Section 10 adds it, in four stages. Stages 0 and 1 are possible today and carry no project data: Stage 0 is reachability, where a 401 from the node surface is the correct answer and the evidence; Stage 1 is the node report they already wrote, sent as a heartbeat. Stage 1 needs no schema change and no new endpoint, because node_heartbeats.raw is an open jsonb. Stages 2 and 3 are ours to build and are listed as such. Two transport facts stated plainly, both learned the hard way here. The deployment accepts exactly one Host header and returns 400 on everything otherwise, including health. And there is no TLS, despite the service description claiming "TLS terminated upstream by Caddy" - Caddy is inactive and its configuration is empty. Tolerable for a read-only node report on a LAN, not tolerable for the ingest path, so putting Caddy in front of the port is a Stage 3 prerequisite rather than an extra. It starts read-only for the same reason the compute substrate does: a node is online because it heartbeated, not because it was created. A mirror that begins with data cannot distinguish "the node is broken" from "the data is bad", and their speaker counts already disagree. One stale block survived a previous edit - a replace that matched nothing and reported success. It now asserts before replacing. |
||
|
umbra
|
717e1d56b9 |
Editions, character persistence, and a freeze anchored per character
Five more decisions, and the second one closes the last three questions plus reveals two things the model did not have. Not every character recurs. A subset persist across books and the rest are local to one, and persistence is *declared* rather than inferred - which turns out to bear directly on a data quality problem we already knew about. The three-way speaker-count disagreements may be partly different characters who share a name: "The Big One" in book one is not necessarily "The Big One" in book two. Declaring persistence gives a way to tell those cases apart instead of reconciling them into one row. I-A holds for every character; I-B is only meaningful for a persistent one, and vacuously true otherwise, which is correct rather than unmodelled. The freeze is anchored once, per character, at the release of the first book they appear in - not per book and not per series. New invariant I-G, which is stricter than I-F for a recurring character and identical for a book-local one. Capture is selective, which has one sharp edge. If not every character needs every aspect, then a missing personality artifact means one of three quite different things: not attempted, attempted and not applicable, or attempted and failed. Only the third is a defect, and a nullable model cannot tell them apart - so it reports a healthy system as incomplete or an incomplete one as healthy. Absence is therefore typed, not null, and the validator treats in_progress as never satisfied. Published editions change, and a correction, a new cover or a new foreword is a new release rather than an edit. So editions sit between the work and the artifacts, and projects.status = released moves onto the edition: the work is not released, it has releases. Content- addressing makes this nearly free - an edition that fixes a paragraph mostly reuses the same artifacts by hash, and only genuinely changed content becomes new. That resolves the obvious tension with the freeze: a correction in edition 2 that contradicts edition 1 is handled by a new attribute state only later editions reference, which is I-H. The freeze unit is a single work, so book 1 can be released while book 2 is still being written. That leaves one question the answers created rather than closed: a book-local character revealed to recur is two entities for one person, and merging them would alter attributes I-G has frozen. Forward-only link, or merge with a snapshot? Recommended the forward-only link, because the freeze is what protects reproducibility and the gain from a merge is retrospective. It is a question about the fiction. Two new validator phases: persistence and partial-capture, where a mismatch in declared persistence is a blocker and disagreement in counts is the expected finding; and an edition probe, where unchanged content must appear with the same hash across editions. Two lines in 1.5.2 still described the older per-work freeze and were corrected rather than left to contradict the new anchoring. 670 lines, 15 decisions, 9 validator phases, 8 invariants each checked to have a home, 2 questions open. |
||
|
umbra
|
21d0949b84 |
Character attributes change over the story: the largest model change yet
Four more decisions, and the last one reshapes the model. Personality is one thing per character: global across books and language-neutral, because the disposition the reader meets should not change with the language or the situation. Scope is the character alone, so `lang` is not part of a character_profile's identity. A translation that cannot carry the disposition is a finding about the performance, not a second personality - and modelling it as a second personality would make exactly that inconsistency undetectable. A voice is a matrix of parameter values, and a variant is a small documented change to that matrix. That answers the scope question better than anything the document proposed: same identity, different values, difference recorded as data. New invariant I-E, because a variant that changes an undeclared value is the failure mode. The large one: a work is mutable until publication and frozen after it, like software before deployment. Character attributes are defined in the first work and change only explicitly - John goes through trauma in book one, and in book two the situation costs him a leg. So attributes are time-scoped and changes are events, never silent edits. The trap here is one `characters.attributes` column, and the operator's own two answers already separate what it would have merged. Disposition is global and stable; circumstance is scoped to the timeline. The amputation is circumstance, and whether it alters disposition is an authorial judgement recorded as an explicit change, never a consequence derived automatically. If the two classes shared a table, those rules would collide in the schema and someone would resolve it by re-deriving personality from circumstance - destroying precisely the consistency that was designed. Three classes: identity, disposition, circumstance. The freeze needed no migration: `projects.status` already carries `released`, which is exactly the release boundary described. Added `series_works` for the ordering, `character_attribute_defs` for the classes, `character_attribute_states` append-only with an effective position, and `voice_matrix_values` plus `voice_variant_deltas`. Six invariants now, each checked to have a home rather than being an aspiration with no column behind it. Two new validator phases: a timeline probe where a circumstance change altering an accepted personality is a blocker, and a voice matrix probe for undeclared variant changes. One question left, because D9 says "a set of items must remain the same" without naming the set. We assumed the minimum that makes the rest work and flagged the assumption as load-bearing: do released character attributes freeze, or only the released rendering? Does a published edition ever change - is there an edition concept? And is the freeze unit a single work or the whole series? Corrected two stale rows now that D1 is answered: one still said DECISION-1 was unratified, and one still offered it to them. |
||
|
umbra
|
838733d51d |
Data model alignment: six decisions recorded, character model rewritten
The alignment document for the audio system now carries what the operator decided, and the character model is materially different from the one I first wrote. The correction that mattered: I had framed speaker and author as two names for one thing that might get conflated. They are not adjacent, they are a chain, and the expensive trap is a third thing. A character may be an author *inside the fiction* - a character who writes a book within the story. That is the character's occupation and it does not reclassify them; the book author can make a character an author and the character remains a character. So "author" appears in two unrelated places - the byline on a work, and a job inside a story - and merging them puts every in-story author on the credit pages. Decisions recorded: the controller is authoritative for data but does not specify algorithms; compute and capability design is deferred to its own phase and does not block this one; a character may belong to several series; a character has an LLM-derived personality; a character's voice is consistent across every utterance and every book; and a character may have additional voices, each designed, captured and stored with a reason. Three things fell out of that which the model did not have: - Personality is LLM output, so it is an artifact with lineage, not a field. Re-analysis creates a version; the accepted one is a pointer. - Speaker is derived - a character with lines in a work - so it is a flag on an appearance, not an entity. Modelling it as an entity is what forces every language to have its own cast. - Voice variants are passage-shaped, not character-shaped. All three of the reasons given are scoped to a run of utterances, so a variant needs a range or it silently becomes the character's voice everywhere. Four invariants now have a home, checked rather than asserted: one primary voice per (character, language); that primary voice keyed so no book can change it; no variant without a reason; personality never edited in place. Corrected a stale row in section 4 that still named the old table scheme, and re-checked that no invariant is an aspiration without a column behind it. Nine new tables are required and none is claimed as existing. Six of the open questions are answered and four remain, three of them sharper now: whether personality is per character or per language, whether it is global or genuinely differs per book, what a variant's scope is expressed in, and whether a source revision belongs to a project or to a series. |
||
|
umbra
|
0f5c1e3408 |
Run the agent as the service account it claims to be
The compute proof of concept had never been run with the agent as umbra-poc. Running it that way found three defects, all the same shape: a check that passed because its mechanism did nothing. The demonstration never reached the agent at all. It held the pid it spawned, which under `sudo -n -u` is the sudo wrapper - sudo forks the agent before exec'ing. Signalling that killed the wrapper and left the worker running, renewing its lease and holding the job indefinitely, under a printed note saying "exactly like a power cut". An unprivileged process cannot signal another account's process at all, and kill(2) on a process group succeeds as soon as one member is signalled, so the wrapper's death read as the agent's. Every run leaked an agent. Lease recovery was a diagnosis rather than a mechanism. sweep_expired_leases documented itself as "called by the controller's own timer" and there was no timer; a killed node's job came back only when somebody noticed a stalled job and posted to an administrative route by hand. The proof of concept swept by hand and recorded the result as automatic. app.py now runs the loop for the life of the application, building the compute repository at startup so a restart mid-job still recovers it. The power cut is now the agent's own: os._exit(137) the instant it claims a job, before running it, because a process that does not own the agent cannot signal it. A clean stop goes through a stop_file the agent polls, for the same reason. Liveness is measured by finding the worker's own pid in /proc matched on its config path, sampled repeatedly and accumulated - the wrapper and the worker are not simultaneous, so a single sample finds only the wrapper. The node key travels by `sudo --preserve-env` rather than as VAR=value in argv, because argv is world-readable through /proc/<pid>/cmdline and the environment is not. The demonstration checks whether the key will survive before starting and falls back to a pipe, which node_agent.py --node-key-stdin accepts. Neither a file nor argv ever holds it. Eight tests, each verified to fail against a mutation of the code it covers: three for the sweeper (it must return an expired lease, must survive a failing round, must not invent a repository it has no database for) and five driving a real agent process against a stub controller (stops on the stop file, exits 137 holding an uncompleted job, takes the key from both routes and neither, reports an empty pipe as an empty pipe). 448 pass with a database, 421 without. The deployment database is untouched: users=2 orgs=3 projects=0 audit=43, no compute tables. |
||
|
umbra audio
|
bfde763f00 |
Fix a 500 on /api/health from an uninterpolated SQL placeholder
The first deploy of the previous commit took the health endpoint down. The
tenant-row predicate had been added to `count_users_with_role` as a literal
`{TENANT_ROW_PREDICATE}` inside a plain triple-quoted string rather than an
f-string, so the braces were sent to PostgreSQL as text and the statement failed
to parse. That query backs `setup_is_required`, which every unauthenticated
health probe calls, so `/api/health` returned 500 and the deployment reported
itself broken.
Nothing caught it before production, and the reasons are worth keeping:
- the unit tests use the in-memory repository, which has no SQL;
- the integration tests did not call `count_users_with_role`;
- and the source-level test I wrote to pin the predicate's presence was
satisfied just as well by the broken version, because the broken version does
contain the text. A test that checks a predicate was *written down* cannot tell
a working predicate from an inert one.
Two tests now, each catching the regression on its own:
- `test_no_sql_string_carries_an_uninterpolated_placeholder` walks the parse tree
rather than the text, so a `{...}` in a non-f-string fails whether or not the
surrounding code mentions the constant. It reports the offending query, not a
line number: `inspect.getsource` returns a class body lifted out of the file, so
its line numbers do not point anywhere a reader could open.
- `test_the_health_probe_queries_run_against_the_real_schema` runs the probe's
own queries against the real schema, and
`test_setup_is_required_ignores_project_assignments` holds the predicate's
purpose: a project row is never mistaken for a platform administrator.
413 tests pass with the integration suite; 388 without a database.
|
||
|
umbra audio
|
ed72f61975 |
Separate organizations, and give members their organization's projects
An organization's work is separate from every other organization's, and the separation is now one module rather than a property each route remembers. `tenancy.py` resolves the caller's scope in a single place. A member is pinned to the organization on their own assignment, expressed as a concrete identifier rather than a flag, so "no filter" is unreachable except for a platform administrator. Every read and write resolves through it, so there is no per-route variation to drift. Within an organization, every account sees every project. The brief is explicit that two people working on different books both see both, and assignment says who is *working* on a project rather than who may *know* it exists. This diverges from `app.can_access_project`, which restricts visibility to assigned members; the divergence, and what it means if row-level security is ever switched on, are recorded in migrations/README.md because that would otherwise be a silent tenant-internal regression no test here would catch. Two rules, both learned by getting them wrong first: - A read naming another organization is ignored, because refusing it would confirm the identifier exists. A write is refused rather than retargeted, because a project landing in an organization the caller did not ask for is somewhere they will not look. - A missing project and a foreign one get the same error, so an identifier cannot be probed across a boundary. Projects: the `projects` table, its statuses, and the project-scoped arm of `user_roles` were already in the schema and unused. No migration. The dashboard is one server-scoped call, `/api/workspace`, rather than several assembled in the browser — pieces from different tenants that nobody notices are not a failure mode worth shipping. Filters narrow the list already in the page, so switching cannot mix two fetches, and an empty list is distinguishable from a failed one. **A sign-in lockout, found by the integration suite.** An assignment is a second active `user_roles` row, and `_resolve_assignment` refused an account with more than one. So assigning anybody to anything made them unable to authenticate — locked out of the page the assignment was meant to show them. `_sync_assignment` had the same defect. Six queries were scope-blind; three were harmless only because the assignment role is named `viewer` and never matched a tenant role name, which is luck rather than a property, so a structural test now pins the predicate on every one. The guard against two *tenant* rows survives, narrowed to the ambiguity that actually matters. Also fixed: `get_project_by_slug` cast `slug::text`, making citext's case-insensitivity a case-*sensitive* comparison that appeared to work everywhere else. An assignment grants the `viewer` project role, since no `member` role exists at project scope; it is a named constant, not a query literal, because the project workflow is not defined yet. 410 tests pass with the integration suite against a real database; 386 without. |
||
|
umbra audio
|
e034061ec1 |
Add password reset and self-service account settings
An administrator could not reset a password for bdmurph73@gmail.com, and there was no way for a signed-in person to change their own. Both are now in place. Migration 0002 gives single-use tokens a purpose. 0001 reused the control plane's unused bootstrap_tokens for email verification, and the table has no notion of what a token is for, which was invisible while there was only one kind. With a second kind it stops being invisible: a reset link clicked by anyone holding it would confirm an address, and a verification link would set a password. The column defaults to the existing purpose, so no row is rewritten and verification is unchanged. Every token lookup now filters on purpose, and the adapter reads the purpose from the row rather than hardcoding it. An administrator may start a reset but never finish one. The link goes to the address on the account, only its holder can complete it, and the endpoint returns a bool rather than a password. An administrator who can set someone's password can also sign in as them indefinitely. A reset revokes every session, because a reset is what someone reaches for when they think their password is compromised, and revoking only the session that completed it would leave a stolen cookie working. Reset links live one hour against the two days a verification link gets. A verification link confirms an address the person already controls; a reset link is a live credential for as long as it is valid. Three bugs found while building it, all mine: - request_password_reset did not call _authorize_target, so an organization administrator could start a reset on an account in another tenant. Caught by a test written for the scoping rule that revoke_sessions already followed. - The in-memory repository never checked token expiry, only PostgreSQL did. So an expired link was honoured in every test and refused in production - the wrong direction for a divergence between two stores of the same rule. - The PostgreSQL adapter's get_live_verification_token took a purpose but not the now its in-memory counterpart needed, so the first call against a real database raised TypeError and returned a 500. The conformance test only checked that a method existed. It now compares signatures, which immediately caught that the protocol itself was missing the parameter. The reported failure was a deadlock rather than a missing feature. The account was invited: it could not sign in because the address was unverified, and I refused to reset anything not yet active. It is now allowed, and completing the reset also confirms the address, since holding the link proves control of it. Without that the flow would not converge - reset the password, sign in, refused again, using a verification link that was never sent. Suspended and deactivated accounts are still refused, because a reset restores sign-in and would otherwise undo a suspension without going through the status controls. Verified live: the account that could not be reset is now active, verified and able to sign in, with password.reset_requested, password.reset_completed and email.verified all in the audit trail. A reset token is refused by the verification endpoint, an invented token gives the same answer as a used one, and a token in a reset URL is redacted from the access log. 327 tests pass, 337 with the integration suite. |
||
|
umbra audio
|
b27915bc68 |
Fix the confirmation link, and stop the test suite eating a database
Two separate problems, one of them mine and serious. The confirmation link did nothing. `request` in api.js refuses a state-changing call with no CSRF token *in the browser*, before fetch is reached, so no request was sent and nothing appeared in the service log. The confirmation page is the only page that arrives without having made a safe request first, because it is opened straight from a link in an email, so it was the only one with no token. Added `ensureCsrfToken`, which fetches one if there is none, and called it before the POST. The alternative was exempting /api/auth/verify-email from CSRF the way login and first-run setup are. Those are exempt because nothing about them relies on a session; this one is state-changing and a session can exist, so the exemption list stays short. Verified against a live token: page loads, no cookie, priming GET, then POST returns 200 and the account becomes active. Replaying the same link then returns 400, so single-use still holds. The second problem is worse. This commit's earlier state truncated users, organizations and audit_events after every test, and I ran it against the deployment database. It destroyed the two accounts and the organization that were in it. My own "leave the database clean" change from two commits ago caused it: TRUNCATE removes the suite's rows and every other row equally, and the check that had been passing only asserted the suite's rows were gone, which is also true of a full database. It passed because I had just emptied that database by hand, so it never tested the case that mattered. Tried fixing it by deleting selectively instead. That cannot work either: audit_events has no ON DELETE CASCADE, and app.prevent_audit_mutation raises on delete, so the rows cannot be removed one at a time at all. So the safety is not in cleaning up carefully, it is in refusing to start. Two guards: the DSN must be accompanied by UMBRATRANS_TEST_DATABASE_IS_DISPOSABLE=1, because a database name is not evidence and an empty database is indistinguishable from an unconfigured deployment; and the suite then refuses to run if the target holds any rows, naming them. Verified both ways, with a planted row that survives. The deployment database is cleared and first-run setup is open again. |
||
|
umbra audio
|
2b3a093e96 |
Configure the domain mail transport and run in production
bdmurphy.me's MX records point at privateemail.com. Its submission host is smtp.privateemail.com:587 with STARTTLS, offering PLAIN and LOGIN over TLS 1.3. Not the MX host: an MX serves inbound mail and its submission ports are closed, so the obvious place to look has nothing listening. A test message to umbra@bdmurphy.me was delivered, and so was a real confirmation message through the provisioning path, leaving the account in `invited` awaiting confirmation. Flipping to production surfaced two guards that had become limitations rather than protections, and both are fixed rather than routed around: - `serve` refused any production configuration. It is now the deployment command as well as the development one, so it takes --allow-production by name. That flag is an acknowledgement, not the control: the configuration gate independently refuses a deployment without a PostgreSQL DSN, Secure cookies, CSRF, an absolute verification base URL, and a mail transport. - `create_app` refused to build a production app unless a repository was injected, with a message saying the PostgreSQL adapter "is not part of this slice". It is. The check is now about which adapter is in use rather than where it came from, stated twice: once against the configured kind before anything is opened, and once against the built object, because an injected repository bypasses the first. Two tests asserted the old behaviour and were rewritten rather than deleted. One claimed a production app must be refused for want of an injected repository, which would have made a production deployment impossible to start; it now covers both routes to a development store. The other built a production app on an in-memory repository to assert a Set-Cookie header, which the corrected rule refuses; it now asserts the cookie flags on settings that ask for them, with the production half covered by the configuration gate that refuses insecure cookies. /api/health now reports environment=production, repository=postgres, mail_deliverable=true, setup_required=false, and the service survives a restart with its data intact. |
||
|
umbra audio
|
c8b8cdb118 |
Deploy the service: dedicated database, unit, and the public hostname
Brings up a working deployment and, in the process, fixes a unit that could not start and a test suite that had been writing into it. - Created the dedicated `umbratrans` database owned by the non-superuser role `umbra`, applied the control-plane schema and the phase-1 migration. The migration is idempotent, verified by applying it twice. The existing `umbracontrol` control plane is untouched, which is the point of a separate database. - The unit could not start: `ProtectHome=read-only` forbids the write that `uv run` makes to ~/.cache, so it exited 2 and restart-looped until systemd's start limit stopped it. Switched to the venv interpreter directly, which is the right shape for a service anyway: a fixed interpreter, nothing to resolve at boot, and no developer tool's cache able to change what runs. - The integration suite leaves the database as it found it. It creates real accounts, sessions, credentials and audit rows and used to leave every one behind, which was only visible because the scratch database was disposable. Running it against the deployment database filled it with ten test accounts, and first-run setup then reported that an administrator already existed — a confusing failure with no visible cause. Teardown truncates the rows the suite's code creates and leaves the schema's reference data alone. - Bound to 192.168.123.14:8442, the private LAN address the reverse proxy reaches, replacing a static file server that had no API behind it. The public hostname now serves the real application. First-run setup is open, and the deployment runs staging with the mail requirement explicitly waived: /api/health reports mail_deliverable=false, the control panel shows a standing banner, and account creation is refused there. Sign-in and setup work; nothing needing a delivered email does, until SMTP credentials arrive. 288 tests pass against this database, and leave it clean. |
||
|
umbra audio
|
ef6bf1e6a3 |
Add first-run setup so a deployment can be finished in the browser
The gap this closes: the operator command that creates the first administrator refuses to run in production, and production refuses to start without SMTP credentials that only the operator has. So a real deployment could not be brought up at all. Everything else already worked. Setup is offered only when both of these hold, checked against the store on every request rather than cached at boot: * no platform administrator exists, and * the token generated at that boot is presented. The token is what keeps this from being first-requester-wins: anyone who reached the URL before the operator would otherwise become the platform administrator. It is written to the journal, so control of the service is still what authorises the first administrator, and it is regenerated on restart. It is spent when it authorises rather than when the account is created, so a password the policy rejects cannot leave a shown token usable. Five failed attempts burn it. Comparison is constant-time. The status endpoint never returns it, and there is no endpoint that would. One deliberate exception to the log filter: the announcement uses no key=value shape, so redact_text leaves it alone, and a test pins both halves — the setup code still gets through, and `setup_token=` still does not. The filter is not weakened, disabled, or special-cased. Also: - a `staging` environment, so a deployment can be brought up before SMTP exists. It holds every production rule except that one, and the waiver that permits it is refused in production outright, reported by /api/health as mail_deliverable, and shown as a standing banner in the control panel, which then refuses to create accounts. A staging deployment cannot become a convenient way to run something production would not allow. - `serve` accepts `--allow-lan-bind` for a named private address, because the reverse proxy runs on another host. It refuses a wildcard and a public address, so the flag cannot open the port to the internet. AF_UNIX was added to the unit's address-family set, without which the PostgreSQL socket connection cannot be made. - `count_users_with_role` on the repository, counting active assignments rather than profile rows, because a suspended administrator is not one. - setup.html and the sign-in link that appears only while setup is outstanding, plus the deployment runbook and a rootless unit. A test of mine was asserting over every credential row in the shared scratch database rather than the one it created, so it passed only while no account had replaced its one-time password. Scoped it, and added the other half of the property. 279 pass without a database, 288 against one. |
||
|
umbra audio
|
d1b19fbdb3 |
Update the site icon and wire the UI to email login
Icon: the supplied artwork is now UmbraTrans_Icon2_899x920.jpg, gold and
cyan on white, with the wordmark corrected to "UmbraTrans". It is copied
verbatim to web/assets/umbratrans-icon.jpg as the source, and
web/assets/umbratrans-emblem.png is derived from it and is what the pages
load: the book cropped to its ink bounds, the white page made transparent
by flood-filling inward from the corners so the white lettering inside
the book survives, letterboxed square rather than side-trimmed so no part
of the emblem is cut, and emitted at 256px.
A square crop of a 827x695 emblem has to choose between trimming the
sides and letterboxing, and trimming was cutting the book's frame. The
glow also moved from box-shadow to filter, because a box-shadow follows
the border box rather than the artwork's alpha and would have drawn a
glowing rectangle behind a book-shaped emblem.
The UI was still written against username login, so the sign-in form
could not have worked against the API: it posted {username, password} and
LoginRequest requires {email, password}. Fixed, along with the rest of
the email-verification flow that had no UI at all:
- the sign-in form collects an email address, and its copy says so;
- a 403 email_not_verified, which is only ever reached once the password
is already proven correct, reveals a panel offering a new link. The
resend wording is the server's and is identical for real and unknown
addresses;
- a confirmation view accepts a pasted link or token, and one is opened
automatically when the URL carries a token, which is then stripped from
the address bar so it is not left in history;
- the password-policy hint read the sign-in field to tell the user their
password must avoid their "account name", but the rule checks the
account's username. It now uses the signed-in account's handle, and
says nothing rather than the wrong thing when there is no handle;
- the provision form called email optional while the API requires it,
and let a member be created with no organization, which the store
rejects. Both are caught client-side now with wording that names the
field to fix;
- `serve` accepts --repository postgres, so the app can be exercised
against the authoritative store.
Verified against a live server on PostgreSQL: sign-in with the email
address, forced first-password change, provisioning an invited member,
401 for a wrong password and 403 email_not_verified for the right one,
and byte-identical resend answers for a real and an unknown address.
|
||
|
umbra audio
|
0461dff9d2 |
Use the email address as the login identifier and require confirmation
Sign-in now takes the email address rather than a username. This needed no migration: users.email is already citext NOT NULL UNIQUE in the control plane, so it is a case-insensitively unique login key on its own. username survives as an optional display handle for audit lines and is never a login path, which removes the redundancy of asking an administrator for two identifiers. An account created by an administrator now starts in the control plane's invited state and cannot sign in until a single-use link is consumed. The token maps onto bootstrap_tokens, which was already shaped for exactly this and had been unused, so that needed no migration either. Modelling invited as a real state also retires the adapter's earlier fail-closed refusal of that value, which existed because invited was previously unrepresentable. Three decisions worth stating plainly. The "email not verified" message is sent only after the password has been verified. Reporting it earlier would turn the login form into an oracle for which addresses have accounts, which is the one thing the uniform authentication error exists to prevent. A test asserts that a wrong password and an unknown account are byte-identical, and it caught a real bug: the check was first written before the password comparison, and the generic "not active" branch swallowed invited accounts. Both are now ordered deliberately. The operator bootstrap path does not require verification. The operator is attesting to the address themselves, which is the standing an administrator already has, and requiring a delivered email there would deadlock first-time setup on SMTP configuration that does not exist yet. The logging mail transport is refused in production. A deployment that logged links instead of sending them would appear to work while every new account sat permanently unverified. The development transport writes the link with the token redacted, because a log is where live credentials end up. That means a human cannot complete the flow by reading the log; the test harness captures the message in memory instead. This is a deliberate limitation rather than an oversight, and it is documented. Also: production now requires a mail transport and an absolute verification base URL alongside the existing DSN gate, because an account that cannot be verified cannot be used at all. 269 tests pass, 3 skipped. Verified end to end against a real uvicorn server: bootstrap login, organization creation, an invited account with its one-time password, and a handle rejected as a login path. |
||
|
umbra audio
|
31fd8720ec |
Add the PostgreSQL adapter for the authoritative store
Implements AccountRepository against the control-plane schema plus the phase-1 identity migration, and wires it in so production can select it. The adapter is not a thin SQL wrapper, because the control plane does not store the two things the application needs on the users row. Role and organization come from user_roles. There is no role column and no organization column on users; a role is assigned through user_roles with a scope, and app.validate_user_role_scope() enforces that the role's scope matches the assignment. Resolving one role per user is therefore a join, and a user holding more than one active assignment is refused rather than resolved to an arbitrary row. The organization on that assignment becomes the scope of every administrative decision, so guessing would hand out the wrong tenant's authority. Reconciling this with project-scoped membership stays an open review question, recorded as such. Email became a required field on account creation. users.email is NOT NULL UNIQUE, so accepting a missing address would let account creation succeed against the development repositories and then fail on the authoritative store. It is now required in the service, the API schema, and the bootstrap command, so it fails identically everywhere. Audit identifiers now belong to the store. audit_events.id is a bigint identity, so the adapter reports that value; the application correlation UUID is preserved in the detail and never displaces the row key. AuditEvent.id is a string for that reason, which removes a type the two stores disagreed about. Three further fail-closed behaviours: a users.status of invited or locked is refused rather than coerced onto active; a row with mfa_required is refused, because this phase cannot satisfy a second factor and authenticating it anyway would grant access to someone who was supposed to present one; and a missing organization blocks access rather than granting it. save_user deliberately does not write username or email, since both are identity keys with no reviewed change path and no audit event behind them. Every operation runs in one transaction on one pooled connection and publishes the caller with SET LOCAL app.current_user_id, so a returned connection cannot carry one request's identity into the next. That is inert until the reviewed RLS policies exist; the explicit WHERE clauses enforce scope today. A statement timeout bounds how long a pooled connection can be held. Production fails closed at two independent gates: the settings layer refuses without a DSN, and the repository boundary refuses to build a PostgreSQL store without one. Verified by running both, including that an unreachable host refuses to start rather than degrading. The migration gains an index on audit_events (target_type, target_id), which the administrator audit filter needs and nothing covered, and the review questions now include the three things that must be checked before provisioning: the roles rows and their scopes, any row with mfa_required set, and any user holding more than one active assignment. Verification, and its limit. 236 tests pass with 3 skipped. The adapter's row mapping, role resolution, conflict mapping, and configuration are unit tested and run everywhere. Its SQL, transactions, and pooled reuse are NOT executed: this environment has no database credentials, because sudo to the postgres user needs a password, there is no umbra database role, and there is no .pgpass. The integration suite is written and selected by UMBRATRANS_TEST_DSN, and its fixture refuses any DSN whose database name contains umbracontrol. A green run here says nothing about whether the SQL is correct. |
||
|
umbra audio
|
0b9c4d4220 |
Add organization tenancy and same-origin frontend mounting
The administrator interface needs organizations to manage, and the slice had none: users.organization_id was a bare UUID pointing at nothing, so there was no tenant to create, list, or validate an account against. Tenancy: - Organization record with name, slug, and status active|suspended|archived, matching the control-plane vocabulary rather than reusing the account one. - Reads are open to any administrator; every write is platform-admin only, so the tenant boundary stays under a single authority. An organization admin is pinned to the organization on their own principal. - A derived slug is normalized from the display name, but a slug the operator typed must already be canonical, so nothing is silently rewritten behind their back. archived is terminal. - GET /api/uadmin/summary for the dashboard, recomputed per request from the same scoped queries the list endpoints use, so a summary cannot disclose a number the caller could not reach by listing. Two integrity gaps closed: 1. Account creation accepted any UUID as an organization. It now requires the organization to exist and to be active. 2. Suspending an organization revoked member sessions but did not stop new logins, because each member's own account status stays active. Both authenticate and resolve_session now refuse a member of a non-active organization, and a missing organization record blocks access rather than granting it. A test asserts the member is still active yet locked out. Migration, still unapplied: section 4 creates organizations with CREATE TABLE IF NOT EXISTS so a control-plane table is skipped, never redefined, and the rollback drops it only when this migration actually created it. The membership divergence is recorded as the top open review question rather than decided: user_roles is the control plane's membership source, this phase uses a column, and the migration adds no membership column so the choice stays open. Serving: UMBRATRANS_STATIC_ROOT mounts the browser client at / after the API routes, so the session cookie and CSRF header work without CORS. HTML responses get their own CSP allowing only same-origin script and style, which is why the frontend carries no inline script or style. Verification: 204 tests pass (21 new organization tests plus the existing suite), ruff check and format clean. Also exercised against a real uvicorn server rather than only the test client: bootstrap, forced password change, organization creation, one-time password issue, member lockout on organization suspension, and rejection of an account assigned to a suspended tenant. |
||
|
umbra audio
|
b474995414 |
Add phase-1 account management backend slice
Implements the account and user-management system behind the new web UI against development repositories only. Nothing here connects to, reads, or modifies the production umbracontrol database, and no account is created in any approved environment. Account service and API: - Local temporary-password and OIDC enrollment, separated by type. - Argon2id credentials, must-change enforcement, server-side sessions with hashed tokens, idle and absolute deadlines, rotation and revocation. - Administrator account management: list, get, create, suspend, reactivate, deactivate, revoke sessions, org-scoped audit view. - Hidden-prompt operator command for the first administrator, refusing production mode and refusing to run without a TTY. Temporary passwords are never accepted from a client. The request schema has no password field and refuses unknown fields; the administrator path generates the one-time password in process, hashes it immediately, and returns it once in the creation response. The account can reach no route, including /api/auth/me, until it replaces that password, after which the one-time value no longer authenticates. Security posture: fail-closed configuration, Secure cookies in production, double-submit CSRF, constant-time token comparison, generic authentication errors, audit detail allow-listing, and logger-tree secret redaction. Reviewed but unapplied: migrations/0001_phase1_identity.sql adds users.username, a separate user_credentials table, and additive session columns, with a rollback and open review questions. The PostgreSQL adapter is deliberately absent, so the application refuses to start in production rather than run against an unreviewed store. Verification: 169 tests pass (148 new, 21 pre-existing); ruff check, ruff format, compileall, and make web-check are clean. |
||
|
umbra audio
|
37b2056808 | Add UmbraTrans UI prototype and integration design |