UmbraAudio/specs/15-multi-project.md

# Spec: Multi-Project Support

## Overview

UmbraAudio currently operates on a single project stored in `.umbra/`. This spec adds first-class support for **multiple independent book projects**, each isolated with its own data, configuration, and outputs. Projects are identified by a slug (e.g., `deal-hunter`, `the-great-adventure`) and can be created, listed, switched, and removed via CLI and Web UI.

## Architecture

### Project Root Layout

```
UmbraAudio/                          # Repository root
├── projects/                        # Project registry (new)
│   ├── index.json                   # Master index of all projects
│   ├── deal-hunter/                 # Project slug directory
│   │   ├── .umbra/                  # Project data (was .umbra/)
│   │   ├── config.yaml              # Project-local overrides
│   │   └── manifest.json            # Project metadata
│   ├── the-great-adventure/
│   │   ├── .umbra/
│   │   ├── config.yaml
│   │   └── manifest.json
│   └── ...
├── .umbra/                          # DEPRECATED: legacy single-project location
└── umbraaudio/                      # Package code (unchanged)
```

### Project Identity

| Field | Type | Description |
|-------|------|-------------|
| `slug` | string | URL-safe identifier (kebab-case), unique, immutable |
| `title` | string | Human-readable book title |
| `author` | string | Author name |
| `created_at` | ISO8601 | Creation timestamp |
| `updated_at` | ISO8601 | Last activity timestamp |
| `status` | enum | `active`, `archived`, `error` |
| `source_format` | string | `docx`, `epub`, `pdf`, `txt`, `md` |
| `source_path` | string | Path to original manuscript (relative to project dir or absolute) |
| `chapters` | int | Number of parsed chapters |
| `speakers` | int | Number of discovered speakers |
| `pipeline_step` | string | Current pipeline stage (`parsed`, `analyzed`, `designed`, `mapped`, `tested`, `generated`, `exported`) |

### Project Index (`projects/index.json`)

```json
{
  "version": 1,
  "default_project": "deal-hunter",
  "projects": [
    {
      "slug": "deal-hunter",
      "title": "Deal Hunter",
      "author": "B.D. Murphy",
      "created_at": "2026-08-01T04:22:45",
      "updated_at": "2026-08-06T10:30:00",
      "status": "active",
      "source_format": "docx",
      "source_path": "uploads/DealHunterShip_4.0e_whole-to-audio.docx",
      "chapters": 35,
      "speakers": 61,
      "pipeline_step": "generated"
    },
    {
      "slug": "the-great-adventure",
      "title": "The Great Adventure",
      "author": "John Smith",
      "created_at": "2026-08-06T14:00:00",
      "updated_at": "2026-08-06T14:00:00",
      "status": "active",
      "source_format": "epub",
      "source_path": "uploads/the-great-adventure.epub",
      "chapters": 12,
      "speakers": 8,
      "pipeline_step": "parsed"
    }
  ]
}
```

## Data Isolation

Each project is **completely independent**:

| Resource | Location | Shared? |
|----------|----------|---------|
| SQLite database (`project.db`) | `projects/{slug}/.umbra/project.db` | No |
| Speakers JSON | `projects/{slug}/.umbra/speakers.json` | No |
| Voice assignments | `projects/{slug}/.umbra/voice_assignments.json` | No |
| Voice references | `projects/{slug}/.umbra/voices/` | No |
| Chapter audio | `projects/{slug}/.umbra/chapter_audio/` | No |
| Voice tests | `projects/{slug}/.umbra/voice_tests/` | No |
| Pronunciation guide | `projects/{slug}/.umbra/pronunciation_guide/` | No |
| Exports | `projects/{slug}/export/` | No |
| Logs | `projects/{slug}/.umbra/logs/` | No |
| **Voice Registry DB** | `.umbra/voice_registry.db` | **Yes** (global) |
| **Phonetic names** | `.umbra/phonetic_names.json` | **Yes** (global) |
| **Qwen TTS models** | `models/` | **Yes** (global) |
| **LM Studio** | external | **Yes** (global) |

**Shared resources** are global infrastructure; **project resources** are fully isolated.

## Configuration

### Global Config (`config.yaml` at repo root)

```yaml
# Global settings (apply to all projects unless overridden)
tts:
  engine: qwen
  device: cuda
  sample_rate: 24000
  qwen_tts_url: "http://localhost:8000"
  voicedesign_model: "/mnt/models/qwen3-tts/Qwen3-TTS-12Hz-1.7B-VoiceDesign"
  base_model: "/mnt/models/qwen3-tts/Qwen3-TTS-12Hz-1.7B-Base"

llm:
  provider: auto
  lmstudio_base_url: "http://localhost:1234/v1"
  lmstudio_model: "google/gemma-4-e4b"
  temperature: 0.3
  max_tokens: 4096

web:
  host: "0.0.0.0"
  port: 8420
  auto_open: false

quality:
  min_mos: 3.8
  min_test_mos: 4.0
  max_wer: 0.03
  min_speaker_similarity: 0.80

audio:
  target_rms_db: -20.0
  peak_limit_db: -3.0
```

### Project-Local Config (`projects/{slug}/config.yaml`)

```yaml
# Optional overrides for this project only
project:
  slug: "deal-hunter"
  title: "Deal Hunter"
  author: "B.D. Murphy"

tts:
  engine: qwen
  # device inherits from global
  # qwen_tts_url inherits from global

llm:
  # lmstudio_model can override per project
  # temperature can override per project

quality:
  min_mos: 4.0  # stricter for this project

audio:
  target_rms_db: -18.0  # louder master
```

**Merge precedence**: project config → global config → defaults.

## CLI Interface

### Project Management Commands

```bash
# List all projects
python -m umbraaudio.cli project list

# Show current project
python -m umbraaudio.cli project current

# Create new project from manuscript
python -m umbraaudio.cli project create \
    --title "The Great Adventure" \
    --author "John Smith" \
    --source /path/to/book.epub \
    --slug the-great-adventure

# Switch active project
python -m umbraaudio.cli project switch the-great-adventure

# Show project details
python -m umbraaudio.cli project info the-great-adventure

# Archive project (keeps data, marks inactive)
python -m umbraaudio.cli project archive the-great-adventure

# Restore archived project
python -m umbraaudio.cli project restore the-great-adventure

# Delete project (removes all data - requires confirmation)
python -m umbraaudio.cli project delete the-great-adventure --confirm

# Run pipeline commands on specific project
python -m umbraaudio.cli analyze --project the-great-adventure
python -m umbraaudio.cli generate --project the-great-adventure
python -m umbraaudio.cli pipeline --project the-great-adventure -c all
```

### Default Project Behavior

- If `--project` not specified, uses `default_project` from `projects/index.json`
- If no default set, uses first project in index
- Legacy single-project mode: if `projects/` doesn't exist but `.umbra/` does, auto-migrate on first run

## Web UI

### Project Selector

- Top-left dropdown in navbar showing current project
- Clicking opens modal with project cards (title, author, status, last updated)
- "New Project" button in modal → upload form
- "Switch" action reloads page with new project context
- "Create New" navigates to `/projects/new`

### Project-Aware Endpoints

All existing endpoints become project-scoped:

| Old Path | New Path | Notes |
|----------|----------|-------|
| `/` | `/projects/{slug}/` | Dashboard |
| `/voices` | `/projects/{slug}/voices` | Voice review |
| `/speakers` | `/projects/{slug}/speakers` | Speaker review |
| `/chapters-audio` | `/projects/{slug}/chapters-audio` | Audio review |
| `/proper-names` | `/projects/{slug}/proper-names` | Pronunciation |
| `/api/upload-manuscript` | `/projects/{slug}/api/upload-manuscript` | Upload |
| `/api/tts/health` | `/api/tts/health` | Global (unchanged) |
| `/api/models/status` | `/api/models/status` | Global (unchanged) |

### Project Creation Flow (Web)

1. User clicks "New Project" → `/projects/new`
2. Form: title, author, file upload, optional slug (auto-generated from title)
3. POST `/projects` → creates project directory, saves manuscript, returns project slug
4. Redirect to `/projects/{slug}/` (dashboard)
5. Background parsing starts automatically

### API Endpoints

```python
# Project CRUD
GET    /api/projects                    # List all projects
POST   /api/projects                    # Create new project
GET    /api/projects/{slug}             # Get project info
PUT    /api/projects/{slug}             # Update project metadata
DELETE /api/projects/{slug}             # Delete project (with ?confirm=true)
POST   /api/projects/{slug}/archive     # Archive project
POST   /api/projects/{slug}/restore     # Restore project
POST   /api/projects/{slug}/set-default # Set as default

# Project-scoped existing endpoints (all prefixed with /projects/{slug})
GET    /projects/{slug}/api/voices
POST   /projects/{slug}/api/voices/regenerate
GET    /projects/{slug}/api/chapters
# ... etc
```

## Database Changes

### Project Registry Table (Global)

```sql
-- In a new global SQLite DB: .umbra/projects.db
CREATE TABLE IF NOT EXISTS projects (
    slug TEXT PRIMARY KEY,
    title TEXT NOT NULL,
    author TEXT,
    created_at TEXT NOT NULL DEFAULT (datetime('now')),
    updated_at TEXT NOT NULL DEFAULT (datetime('now')),
    status TEXT NOT NULL DEFAULT 'active',  -- active, archived, error
    source_format TEXT,
    source_path TEXT,
    chapters INTEGER DEFAULT 0,
    speakers INTEGER DEFAULT 0,
    pipeline_step TEXT DEFAULT 'initialized',
    config_json TEXT,  -- project-local config overrides
    manifest_json TEXT -- extended metadata
);

CREATE INDEX IF NOT EXISTS idx_projects_status ON projects(status);
CREATE INDEX IF NOT EXISTS idx_projects_updated ON projects(updated_at);
```

### Per-Project Database (Unchanged)

Each project's `project.db` retains the exact same schema as today:
- `project` table (single row per project)
- `chapters`, `characters`, `utterances`, `generated_audio`, etc.
- No cross-project foreign keys

### Migration from Legacy Single-Project

```python
# On first run if .umbra/ exists but projects/ doesn't:
1. Create projects/ directory
2. Generate slug from project name in .umbra/project.db
3. Move .umbra/ → projects/{slug}/.umbra/
4. Create projects/{slug}/config.yaml (empty/inherit all)
5. Create projects/{slug}/manifest.json
6. Create projects/index.json with the migrated project as default
7. Update .env UMBRA_DATA_DIR to point to projects/{slug}/.umbra (or use runtime resolution)
```

## Config Resolution

```python
def load_config(project_slug: str | None = None) -> AppConfig:
    # 1. Load global config.yaml
    global_cfg = load_yaml("config.yaml")
    
    # 2. Resolve project slug
    if project_slug is None:
        project_slug = get_default_project_slug()  # from index.json
    
    # 3. Load project config
    project_cfg = load_yaml(f"projects/{project_slug}/config.yaml") if exists else {}
    
    # 4. Merge: project overrides global
    merged = deep_merge(global_cfg, project_cfg)
    merged["project_slug"] = project_slug
    merged["data_dir"] = Path(f"projects/{project_slug}/.umbra")
    merged["export_dir"] = Path(f"projects/{project_slug}/export")
    
    return AppConfig(**merged)
```

## Pipeline Execution

### Per-Project Pipeline State

Each project tracks its own pipeline status in `projects/{slug}/.umbra/pipeline_status.json`:

```json
{
  "step": "generation",
  "status": "running",
  "detail": "Chapter 12 of 35",
  "progress": 34,
  "started_at": "2026-08-06T10:00:00",
  "updated_at": "2026-08-06T10:15:00"
}
```

### Concurrent Execution

- Multiple projects can run pipeline steps **simultaneously** if resources allow
- Qwen TTS server is shared; requests from different projects queued
- LM Studio is shared; requests from different projects queued
- Web UI shows status for active project only; other projects' status visible in project list

## Voice Registry Integration

The global voice registry (`voice_registry.db`) tracks voices **across all projects**:

```sql
-- Extended schema
ALTER TABLE voice_registry ADD COLUMN source_projects TEXT; -- JSON array of project slugs
ALTER TABLE voice_registry ADD COLUMN last_used_project TEXT;
ALTER TABLE voice_registry ADD COLUMN last_used_at TEXT;
```

- A voice created in Project A can be reused in Project B
- Web UI "Voice Registry" page shows all voices with project badges
- When generating voice reference, if voice exists in registry and matches description, reuse it

## Testing Strategy

### Unit Tests

```python
# tests/test_multi_project.py
def test_create_project_creates_directory_structure()
def test_project_index_persists_and_loads()
def test_config_merge_precedence()
def test_load_config_resolves_data_dir_per_project()
def test_migrate_legacy_single_project()
def test_project_slug_uniqueness()
def test_archive_restore_cycle()
def test_delete_project_removes_all_data()
```

### Integration Tests

```python
# tests/test_integration_multi_project.py
def test_two_projects_independent_pipeline()
def test_shared_voice_registry_reuse()
def test_web_ui_project_switching()
def test_concurrent_generation_different_projects()
```

## Migration Checklist

- [ ] Create `projects/` directory structure
- [ ] Add `ProjectManager` class (`umbraaudio/project_manager.py`)
- [ ] Update `load_config()` to accept project_slug
- [ ] Add `project` command group to CLI
- [ ] Update all CLI commands to accept `--project` flag
- [ ] Update web routes to be project-scoped
- [ ] Add project selector to web UI navbar
- [ ] Create `/projects/new` page
- [ ] Create `/api/projects` endpoints
- [ ] Add global `projects.db` for registry
- [ ] Implement legacy migration on startup
- [ ] Update `PipelineManager` to use project-scoped data_dir
- [ ] Update `GenerationManager` to use project-scoped data_dir
- [ ] Update all scripts to accept `--project` argument
- [ ] Add tests for multi-project isolation
- [ ] Update documentation

## Backward Compatibility

| Scenario | Behavior |
|----------|----------|
| Legacy `.umbra/` exists, no `projects/` | Auto-migrate on first run |
| `UMBRA_DATA_DIR` env var set | Used as-is (legacy mode) |
| `--project` not specified | Uses default from index.json |
| Scripts without `--project` | Use default project |
| Web UI accessed at `/` | Redirects to `/projects/{default}/` |

## Future Considerations

1. **Project Templates**: Pre-defined voice configs, quality settings per genre
2. **Project Cloning**: Duplicate project structure for series books
3. **Shared Character Library**: Export/import characters across projects
4. **Multi-Book Series**: Parent project with child projects (volumes)
5. **Remote Project Sync**: Push/pull project data to cloud storage
6. **Project Permissions**: Read-only, collaborator, admin roles for web UI