UmbraTrans/umbratrans_web/migrations/README.md

Phase-1 identity migrations

Status: proposed, never applied, awaiting review.

The SQL in this directory has not been executed against any database, including any development database. It exists so a reviewer can assess the phase-1 identity change before an operator runs it under a change ticket.

File	Purpose
0001_phase1_identity.sql	Forward migration: users.username, user_credentials, organizations, additive sessions columns
0001_phase1_identity_rollback.sql	Destructive rollback with prerequisites and verification queries

What the forward migration changes

Exactly four things, matching docs/umbratrans-backend-design.md:

1. users.username — nullable, unique, citext (case-insensitive). Existing rows stay NULL until an operator supplies an explicit mapping.
2. user_credentials — one credential row per user, with credential_type in (local_password, oidc), an Argon2id password_hash that is required for local_password and forbidden for oidc, must_change_password, failed_login_count, locked_until, last_login_at, password_changed_at, and timestamps.
3. organizations — created with CREATE TABLE IF NOT EXISTS and the slug constraint attached, so it is a no-op when the control plane already owns the table. The application's phase-1 service validates users.organization_id against it, so without it that column referenced nothing.
4. sessions — additive columns mfa_verified_at, idle_deadline_at, last_seen_at. Nothing is dropped, renamed, or retyped.

It creates no credential rows, backfills no username, inserts no organization, and changes no project, role, permission, invitation, or audit table.

Review questions, and which are now answered

The questions below were BLOCKER-level unknowns while this workspace had no access to a real schema. They have since been checked by applying this file, and the PostgresAccountRepository adapter, to a PostgreSQL 16 instance carrying an exact copy of the control-plane schema. Answered items below are verified facts now. Open items still need a human with production access.

Answered by running against the real schema

• users has a uuid primary key named id and no username column. Confirmed. The column and its unique constraint are created by this file.
• organizations already exists in the control plane, with UNIQUE (slug) already attached as organizations_slug_key, and a status check constraint allowing active, suspended, and archived. This is why sections 4 uses CREATE TABLE IF NOT EXISTS and guards the constraint attachment: an unguarded ADD CONSTRAINT ... UNIQUE USING INDEX fails on an index that is already attached, and because the file runs in one transaction that failure rolls back every other change. The same guard is now on users_username_key so the file can be applied twice.
• The "current user" RLS helper is app.current_user_id(), and it reads the GUC app.user_id — not app.current_user_id. Every policy helper resolves the caller through it (app.is_platform_admin, app.can_view_user, app.is_organization_member, app.is_organization_admin, app.is_organization_admin_for_project), so this single setting is the whole identity contract. The adapter sets it with set_config('app.user_id', $1, true), and PostgresAccountRepository.verify_rls_contract() runs at startup to prove the two agree. It is not a formality: the adapter originally set a similarly named GUC, which left every helper resolving to NULL, every policy denying, and every query returning zero rows — a silent total lockout that presents as an empty database. SET itself takes no bind parameters at all, so set_config is also what makes the value parameterisable.
• RLS is enabled but not forced on audit_events, bootstrap_tokens, invitations, mfa_methods, organizations, permissions, projects, role_permissions, roles, sessions, user_roles, and users (relrowsecurity = true, relforcerowsecurity = false). A table owner therefore bypasses the policies, and the adapter's own WHERE clauses are what currently enforce tenant scope. The policies are defence in depth for a non-owner application role, not the primary control. That is a decision to confirm: if the application is ever to run as a non-owner role, the policies become load-bearing and the isolation tests below must pass first.
• A generic set_updated_at() trigger helper exists (app.set_updated_at), so updated_at is maintained by the database on the tables that use it.
• The roles rows the application needs are present with the right scopes: platform_admin at platform, and organization_admin and member at organization. This matters because app.validate_user_role_scope() runs BEFORE INSERT OR UPDATE on user_roles and enforces the pairing, so a missing or wrongly scoped role fails account creation. Verified, but re-check before provisioning rather than trusting the copy:

  SELECT name, scope FROM public.roles
   WHERE name IN ('platform_admin', 'organization_admin', 'member')
   ORDER BY name;
  

• Every non-platform role is organization-scoped, and user_roles_check enforces it: scope = 'organization' requires organization_id IS NOT NULL. So a member with no organization is not representable in the authoritative schema. The service now refuses that combination itself, rather than letting it succeed against the development JSON store and fail in production.
• bootstrap_tokens is shaped for verification tokens — (id, user_id, token_hash, expires_at, consumed_at, created_at) with a unique token_hash — and is unused, which is why email validation needs no migration. It has no purpose column, so the adapter reports every row as an email-verification token. Phase 1 issues no other kind; a second purpose would need a column added with it.
• citext is available, so users.email is case-insensitive in the database and the adapter does not need to normalise the login key itself.

Still open

• Membership model divergence (the important one). The control plane derives organization membership from user_roles with scope = 'organization', and supports a project-scoped membership the application slice does not model. The application stores membership as a column on the user record. One of these has to change:

  • keep the column and accept that a user has at most one organization and no project membership, or
  • move the service onto user_roles, which means reading and writing role assignments and reconciling RoleName with the roles table.

  The adapter resolves the role and organization from user_roles at read time and writes the assignment on create, so it works against the real schema today, but the divergence is unresolved and it is why the adapter refuses a user holding more than one active assignment rather than picking one. The organization on that assignment is the scope of every administrative decision, so guessing would hand out the wrong tenant's authority. Project membership remains unmodelled.

• Can a client influence app.user_id? The policies read it as an ordinary session GUC. Whether an unprivileged client can SET it directly depends on the role the application connects as, which is still unknown. If it can, the policies are not a security boundary at all and the adapter's own scoping is the only control. This needs an escalation test against the real application role.

• Does any row carry users.mfa_required = true? The adapter refuses to resolve an account that requires a second factor, because this phase cannot satisfy one. Until MFA is implemented, every row must be false or those accounts are unusable:

  SELECT count(*) FROM public.users WHERE mfa_required;
  

• Does every user hold at most one active user_roles row? The adapter refuses to resolve a user with more than one active assignment rather than pick one, because the organization on the assignment becomes the scope of every administrative decision:

  SELECT user_id, count(*) FROM public.user_roles
   WHERE status = 'active' GROUP BY 1 HAVING count(*) > 1;
  

• What are the migration role and application role names? The verification run used a role owning the scratch database, which bypasses RLS. A production application role should be a non-owner so the policies apply.

• Is sessions large enough that CREATE INDEX must be replaced with CREATE INDEX CONCURRENTLY outside the transaction?

• Which existing sessions column should backfill last_seen_at, if any?

• Does the production organizations table match the copy used for verification, in particular the status check constraint the rollback keys on?

Project visibility: the application and the database disagree

This is the most consequential open item, and it is a decision, not a bug.

The brief that prompted tenant separation is explicit: two people in one organization work on different books and both books are visible to both of them. The control plane's own policy helper says the opposite.

app.can_access_project(project_id) returns true only for a platform administrator, the organization's administrator, or someone holding a project-scoped role on that specific project:

OR EXISTS (
    SELECT 1 FROM user_roles ur
     WHERE ur.user_id = app.current_user_id()
       AND ur.project_id = target_project_id
       AND ur.scope = 'project'
       AND ur.status = 'active'
)

The application implements the brief, not the helper. Within an organization every project is visible to every account, and a project-scoped assignment says who is working on a project rather than who may know it exists. The rule lives in umbratrans_web/tenancy.py and is held by test_tenancy.py, test_api_tenancy.py, and test_the_organization_wide_listing_reaches_a_members_projects.

The two rules agree on the question that matters most — no caller sees across an organization boundary — and differ only on project-level visibility.

Why the application wins today: row-level security is not enforced, because the application owns its tables and no policy is forced. The adapter's own WHERE clauses are therefore the only control, and they implement the brief.

What happens if RLS is switched on: projects_select becomes load-bearing and app.can_access_project will restrict projects to assigned members. Under the brief that is a behaviour change, not a security improvement: a member assigned to one book would stop seeing their colleague's. The policy must be amended in the same change that forces RLS, to something like

USING (app.is_platform_admin() OR app.is_organization_member(organization_id))

Doing both in one reviewed migration is the requirement. Forcing RLS without amending the policy would ship a silent, tenant-internal regression that no test in this repository would catch, because the tests exercise the adapter and the adapter does not ask the database for permission.

No migration was needed for any of this. The projects table, its (id, organization_id) unique constraint, the on_hold/released/archived statuses, and the project-scoped arm of user_roles were all already in the schema and unused. The application now uses them; it does not alter them.

One consequence worth naming

An assignment writes a project-scoped user_roles row carrying the viewer role, because there is no member role at project scope - the seeded project roles are translator, proofreader, project_manager, project_owner, release_manager, audio_director, and viewer, and all of them name a job rather than participation. viewer is the least privileged and the only one claiming nothing about what the assignee does there.

This is provisional because the project workflow is not defined yet. When it is, PostgresAccountRepository.PROJECT_ASSIGNMENT_ROLE is where the role changes. It is a constant rather than a query literal so that the change does not require a schema migration.

An assignment is a second user_roles row, and that broke sign-in

An assignment writes a second active user_roles row for the account, alongside the organization-scoped one: a tenant row saying which organization the account belongs to, and a project row saying who is working on what. A project row also carries an organization, so nothing but scope tells them apart.

This shipped as a lockout and was caught only by running the integration suite. _resolve_assignment refused an account holding more than one active row. So the moment anybody was assigned to a project they held two rows, became unresolvable, and could not authenticate — locked out of the very page the assignment was meant to show them. _sync_assignment had the same defect, so a role change on an assigned member failed too.

Every query that asks "which organization is this account in", or "what role does it hold there", now applies TENANT_ROW_PREDICATE (ur.scope <> 'project') in postgres_repository.py. Two exemptions, both narrow and both reasoned in place: a row a statement creates (the scope was just computed and is in the column list), and the one write keyed on the row's own primary key in _sync_assignment, which names exactly one row already chosen by a preceding SELECT.

The guard itself survives, narrowed to the ambiguity that actually matters. Two tenant rows is still refused, because the organization returned scopes every administrative decision and resolving to whichever sorted first would hand out the wrong tenant's authority. test_two_tenants_for_one_account_is_still_refused holds that down, writing the impossible state directly because no service path produces it.

Two tests hold this, and each catches the regression on its own:

• test_an_assigned_member_can_still_sign_in — behavioural, and deliberately asserts the sign-in path rather than the resolver, because the resolver being correct is not the same as the account being usable.
• test_every_user_roles_query_distinguishes_tenancy_from_project_membership — structural, over the adapter's SQL. Needed because three of the six scope-blind queries were harmless only because the assignment role is named viewer and never matched a tenant role name. That is luck, and a behavioural test cannot tell luck from a rule.

The review gate's own query must be amended

The existing gate asks:

SELECT user_id, count(*) FROM public.user_roles
 WHERE status = 'active' GROUP BY 1 HAVING count(*) > 1;

As written it is now wrong. It fires on every assigned member, so its meaning silently changes from "this account has two tenants" to "this account is working on something" — which is not a tenancy error. It must be narrowed:

SELECT user_id, count(*) FROM public.user_roles
 WHERE status = 'active' AND project_id IS NULL
 GROUP BY 1 HAVING count(*) > 1;

Verified against a real database, in a rolled-back transaction, with an organization, a project, a member, and both assignment rows in place. Without the predicate the query flagged 1 row; with it, 0. The gate above is the one place in this document that a reviewer must edit before approving.

Section 5 of 0003: the datastore

0003 now carries publishers, project_publishers, authors, project_authors, artifacts and job_artifacts, plus eight permission keys. It is folded in here rather than deferred to 0004 because 0003 has still never been applied to the deployment, and an unapplied migration can still be edited. Once it has run, this is 0004 and a discussion rather than a change.

The policies have now been observed refusing things. They had not been, and until this was done every statement in this project about tenant isolation was a statement about the adapter's own WHERE clauses.

The reason was structural: RLS does not apply to a table's owner until FORCE ROW LEVEL SECURITY, and every connection in this project is the owner. A probe run as the owner showed organization B's administrator successfully UPDATE-ing organization A's imprint — exactly what WITH CHECK was written to prevent — and a policy that cannot refuse is indistinguishable from a policy that is absent.

test_rls_tenancy.py runs as rls_probe: a role with plain table privileges and no ownership. Eight tests, each checked against a mutation:

what	observed
organization A's artifacts	1 visible, B's 0
organization B's artifacts	1 visible, A's 0
no app.user_id set	0 rows on all four datastore tables
B reading A's imprint	allowed — publishers are shared reference data
B updating A's imprint	0 rows changed, no error
A updating its own imprint	1 row changed
B inserting an artifact into A's project	refused
A claiming B as owner_organization_id	refused
A creating a publisher with no owner	refused

Creating the role needs superuser:

sudo -u postgres psql -d <scratch> -c \
  "CREATE ROLE rls_probe NOLOGIN; GRANT rls_probe TO umbra;"

Without it the module skips, with the reason stated.

What this does not close. The application still connects as the owner, so in the running system these policies still enforce nothing. What is now established is that the policies are correct — so FORCE ROW LEVEL SECURITY plus a non-owner application role (gate G2) is a configuration change with a known-good target, rather than an untested hope. The gate is not closed.

Two behaviours that application code has to know, asserted so a change is noticed:

• A filtered UPDATE or DELETE raises nothing. The statement succeeds and affects zero rows. Code checking "did it raise?" instead of "did it change a row?" will believe it succeeded.
• An INSERT violating WITH CHECK raises InsufficientPrivilege.

And one trap: a second permissive policy cannot narrow the first. PostgreSQL combines permissive policies with OR, so adding a "platform administrators may also" policy to carve out an exception widens access for everybody. The WITH CHECK on the single FOR ALL policy is doing that job alone.

test_rls_coverage.py covers the schema side: relrowsecurity is true on every table carrying a policy, each has one, the shared reference tables keep a SELECT-specific policy admitting any authenticated principal, publishers and authors carry no credential column, and artifacts' unique index is keyed on (project_id, content_hash, …). That last one exists because RLS was initially not enabled on four of the six new tables — a policy on a table where it cannot apply is listed in pg_policies and enforces nothing, and a check for "does the policy exist" would have passed throughout.

The GPU device permissions, after the 2026-10-01 reboot

The one part of this work whose boot behaviour could not be checked until the machine rebooted. It was checked, and it was wrong.

What happened. /dev/nvidiactl came back 0666 root:root, so the controller could open the card again — the exact thing the udev rule was applied to prevent.

Why, in three layers.

1. umbra-nvidia-devices.service ran at 05:38:24, ordered After=systemd-udev-settle, and the driver had not created a single node yet. The helper globbed /dev, found nothing, and exited 0. udev-settle means "udev has finished its queue", not "the proprietary driver has finished initialising". The helper ended in an unconditional exit 0 — written deliberately, so that an early boot would not fail a unit — and that is what made "fixed the permissions" indistinguishable from "there was nothing to fix".
2. Ordering could not have been enough anyway. nvidia-persistenced chmods /dev/nvidiactl back to 0666 on every initialisation, through nvidia-modprobe. That happens after any boot-time unit and is invisible to udev, which does not manage these nodes. So the invariant is eventual, not ordered: the unit runs After=nvidia-persistenced.service, and umbra-nvidia-devices.timer re-asserts every two minutes.
3. The timer did nothing at first. RemainAfterExit=yes on the oneshot kept it "active (exited)", so systemd treated every activation as already satisfied — the timer fired, the service reported Result=success and ExecMainStatus=0, and the permission was still wrong two minutes later. The original bug, repeated one level out, inside the mechanism built to repair it.

A regression this caused. Restricting the nodes to 0660 root:video stopped nvidia-persistenced working: it drops to uid 122 with setuid and without initgroups(), so it never receives the video supplementary group and failed with "Insufficient Permissions". It had been running fine before. A systemd drop-in runs it as root — upstream's own default — which works regardless of the device group and does not widen the permissions.

Verified after the fact, by breaking it deliberately and watching:

chmod 0666 /dev/nvidiactl          -> 666
(140s later, after the timer)      -> 660 root:video

with the journal showing the helper running and reporting restricted 5 node(s). poc/nvidia-device-access.sh show is the one command that answers whether it is currently holding, and it now distinguishes "no" from "could not check" — the latter being a fact about the tool, not about the machine.

Verified across four consecutive reboots, and the fourth was measured rather than inferred:

account	role	/dev/nvidia*
ollama	the account that opens the card	yes — nvidia-smi -L
umbra-poc	the agent, for its measurement	yes — nvidia-smi -L
umbra	runs the controller	no — refused

The first two boots were 0660 root:video with the timer running and the unit's journal showing the helper working, which established that the mechanism holds. The fourth added the direct measurement of all three accounts, which is what was still inferior: the device bits and the video memberships were identical to boots where nvidia-smi had been run under those accounts, but identical configuration is not the same as an observed result.

poc/nvidia-device-access.sh show answers whether it is holding, and — since it distinguishes "no" from "could not check" — it says so when it cannot ask. After a reboot there is no cached sudo credential, so the two sudo -n probes report could not check. That is the honest answer and it is why the direct run was worth doing: the tool had been telling the truth about its own blindness, and the boundary had been claimed from configuration alone.

A second finding from the same reboot, unrelated to the GPU. The deployment did not come up. Something else in a development session held port 8442 — a python3 -m http.server serving web/, which the backend mounts itself and does not need — and the service failed five times in ten seconds. With StartLimitIntervalSec=120 and StartLimitBurst=5 that exhausted the budget, and systemd then never tried again: the deployment stayed down after the port was free, and needed a person to notice. A transient condition had become a permanent outage, twice.

Widening that to 500 seconds and a burst of 20 did not fix it. It bought more time and nothing else: on the next boot the squatter outlasted twenty restarts, the unit returned to the same permanent failed, and the deployment stayed down with the port free.

That fix's verification missed it, and the reason is worth recording. The port was freed after three restarts — inside the window — so the test exercised the easy case and the commit message said fixed. The real case is a fault that lasts longer than the budget, which is exactly the case the budget exists to stop handling.

So the rate limit is the wrong mechanism, not merely the wrong value: any finite budget can be exhausted by a fault that outlasts it, and the resulting failure is silent and permanent. StartLimitIntervalSec=0 disables rate limiting; the unit retries for as long as the fault lasts and binds the moment it clears.

Verified against the case that broke it twice: port held for 200 seconds, which drove 31 restarts — past the old limit of 20 — with the unit never reaching failed. Freeing the port produced 200 from /api/health within eight seconds, with nothing having touched the service. A genuine crash loop now logs every 5 seconds instead of stopping, which is noisier and vastly better than a deployment that is quietly down. There is systemctl --user stop for when you mean it.

The data-model alignment document

docs/data-model-alignment-v1.md (814 lines) is the contract proposed to the audio node on Mini-M4. Sixteen decisions recorded, nine validator phases, eight invariants, two questions open. It is the counterpart to this README rather than part of it: it describes the content model, this describes the security and operations posture of the machine holding it.

Two things from it belong here because they are security posture, not vocabulary:

• A node may not write to the datastore. Node credentials are compute:read, compute:claim, compute:complete, compute:heartbeat — deliberately no data access. The audio node becomes the writer when the mirror opens, and that scope has to be granted as a decision rather than arrived at by widening an existing one.
• The connection sequence (§10) starts read-only, and Stage 0 expects a 401 from the node surface. A 200 without a credential would be a finding to report, not a stage to pass.

Also worth knowing: the deployment accepts exactly one Host header (ALLOWED_HOSTS=umbratrans.bdmurphy.me) and returns 400 on every request otherwise, including /api/health. This has the appearance of the service being down.

Gates that must pass before approval

• Logical backup taken, checksummed, and restore-tested on a scratch database.
• Migration role and application role privileges verified.
• RLS tests covering two organizations and two projects, including FORCE ROW LEVEL SECURITY and a client-controlled-GUC escalation attempt. Half done. test_rls_tenancy.py now covers two organizations and two projects against a non-owner role, so the policies are verified correct — which did not previously hold. Still missing: FORCE ROW LEVEL SECURITY and the GUC-escalation attempt, and a non-owner role for the application itself. Until the application stops connecting as the owner, the running system is still protected only by the adapter's WHERE clauses. This remains the most important gap, and it is now a configuration change with a known-good target.
• Account-management tests passing against an isolated PostgreSQL database with the PostgreSQL adapter implemented. Done: the file applies cleanly to an exact copy of the control-plane schema and is idempotent on re-apply, and the suite passes against it (263 tests) via UMBRATRANS_TEST_DSN.
• Rollback rehearsed on the scratch database.

Until then, production PostgreSQL stays untouched and the application runs against a development repository.

Data classification

• user_credentials.password_hash — authentication secret (hash only). Never selected into a report, an endpoint response, or a log.
• sessions.token_hash — authentication secret (hash only), unchanged by this migration.
• users.username and users.email — personal data. Included in data-export and retention reviews for the control plane.