Umbraformat/docs/OLD_Book_formatting_spec.md.old

# UmbraFormat — Book Formatting Specification

## 1. Overview

UmbraFormat is a manuscript formatting tool that converts tagged `.docx` manuscripts into print-ready trade paperback documents. It implements the **B.D. Murphy General Book Formatting Specification**, enforcing consistent page layout, styles, headers, footers, and page numbering across all output.

**Version:** 1.1.0
**Tech Stack:** Python 3.12, Flask, python-docx, LibreHeadless (PDF conversion)

---

## 2. System Architecture

```
┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Web UI      │────▶│  Analyzer    │────▶│  Formatter   │
│  (Flask)     │     │  (analyzer)  │     │  (formatter) │
└──────────────┘     └──────────────┘     └──────┬───────┘
                                                  │
                                           ┌──────▼───────┐
                                           │  LibreOffice │
                                           │  (PDF conv.) │
                                           └──────────────┘
```

### Modules

| Module | File | Lines | Responsibility |
|--------|------|-------|----------------|
| `app.py` | `umbraformat/app.py` | 187 | Flask routes, file upload, API endpoints |
| `analyzer.py` | `umbraformat/analyzer.py` | 332 | DOCX analysis against spec |
| `formatter.py` | `umbraformat/formatter.py` | 1062 | Two-pass document reconstruction engine |

---

## 3. Input Specification

### 3.1 File Format

- **Type:** `.docx` (Office Open XML)
- **Max size:** 50 MB
- **Encoding:** UTF-8 (implicit in OOXML)

### 3.2 Structural Tags

All tags must appear on their own line, with no surrounding content.

| Tag | Purpose |
|-----|---------|
| `[book start]` | Beginning of the manuscript (title page) |
| `[chapter]` | Marks the start of each new chapter |
| `[font]` / `[/font]` | Passthrough block — original formatting preserved verbatim |
| `[copyright]` | Copyright page content follows |
| `[TOC]` | Table of contents placeholder |
| `[back matter]` | Back matter section (verbatim copy) |
| `[Req review]` | Review request page (verbatim copy) |
| `[about author]` | About the author page (verbatim copy) |
| `<<<>>>` | Scene break marker within chapters |

### 3.3 Manuscript Structure

```
[book start]
Title: The Book Title
Subtitle: A Subtitle
Author: Author Name
Imprint: Publisher Name
[optional images]
[chapter]
Chapter 1: Chapter Title
Paragraph text...
Paragraph text...
<<<>>>
Scene break...
[font]
[verbatim block]
[/font]
[chapter]
Chapter 2: Chapter Title
...
[copyright]
Copyright page text...
[TOC]
Table of contents...
[back matter]
Back matter content...
[Req review]
Review request text...
[about author]
Author bio text...
```

---

## 4. Output Specification

### 4.1 Page Layout

| Property | Value |
|----------|-------|
| Page size | 5.5 × 8.5 inches (trade paperback) |
| Orientation | Portrait |
| Mirror margins | Yes |
| Different first page | Yes |

### 4.2 Margins

| Margin | Value |
|--------|-------|
| Top | 0.35 inches |
| Bottom | 0.25 inches |
| Inside (configurable) | 0.375 – 0.875 inches (default: 0.55) |
| Outside | 0.40 inches |

**Inner margin reference by page count:**

| Page Count | Inner Margin |
|------------|--------------|
| < 100 | 0.375 inches |
| 100–200 | 0.50 inches |
| 200–400 | 0.625 inches |
| 400–600 | 0.75 inches |
| > 600 | 0.875 inches |

### 4.3 Headers

| Section Type | Header Behavior |
|--------------|-----------------|
| Title page (Section 0) | No header |
| Front matter | No header |
| Chapter sections | Alternating: odd = book title, even = author name |
| Back matter / verbatim | No header |

- Header distance: 0.50 inches from page edge
- Enabled via `evenAndOddHeaders` in document settings

### 4.4 Footers

| Section Type | Footer Behavior |
|--------------|-----------------|
| Title page | No footer |
| Front matter | PAGE field (Roman numerals, centered) |
| Chapter sections | PAGE field (Arabic numerals, centered) |
| Back matter / verbatim | PAGE field (Arabic, centered) |

- Footer distance: 0.50 inches from page edge

### 4.5 Page Numbering

| Section | Numbering |
|---------|-----------|
| Front matter (copyright, TOC) | Lower Roman (i, ii, iii...) |
| Chapters onward | Decimal (1, 2, 3...), starting at 1 |

---

## 5. Typography & Styles

### 5.1 Named Styles

#### `WA Chapter` — Chapter Title

| Property | Value |
|----------|-------|
| Font | Caladea |
| Size | 18 pt |
| Alignment | Center |
| Spacing before | 24 pt |
| Spacing after | 0 pt |
| Line spacing | Single |

#### `WA Nrml Pgr` — Normal Paragraph

| Property | Value |
|----------|-------|
| Font | Caladea |
| Size | 12 pt |
| First-line indent | 0.30 inches |
| Line spacing | 15 pt (exactly) |
| Spacing after | 0 pt |

#### `WA NChp Pgr` — First Paragraph of Chapter (No Indent)

| Property | Value |
|----------|-------|
| Font | Caladea |
| Size | 12 pt |
| First-line indent | 0 inches |
| Line spacing | 14 pt (exactly) |
| Spacing after | 0 pt |
| First letter handling | Uppercase, drop-cap ready |

#### `WA Section` — Section Heading

| Property | Value |
|----------|-------|
| Font | Caladea |
| Size | 16 pt |
| Alignment | Left |
| Spacing before | 18 pt |
| Spacing after | 6 pt |

#### `WA Scene Break` — Scene Break

| Property | Value |
|----------|-------|
| Font | Caladea |
| Size | 14 pt |
| Alignment | Center |
| Content | `<<<>>>` rendered as decorative break |

#### `WA Thkng` — Internal Thought (Character Style)

| Property | Value |
|----------|-------|
| Font | Caladea |
| Size | 12 pt |
| Italic | Yes |

### 5.2 Title Page Typography

| Element | Font | Size | Alignment |
|---------|------|------|-----------|
| Book title | Caladea | 48 pt | Center |
| Subtitle | Caladea | 18 pt | Center |
| Author name | Caladea | 36 pt | Center |
| Imprint / Publisher | Caladea | 18 pt | Center |

Title page includes 24 empty spacer paragraphs to push content into vertical center.

### 5.3 Images

- Title page images are placed side-by-side using tab stops
- Image relationships (`r:embed`) are remapped during document reconstruction
- Deep-copies drawing XML to preserve original image data

---

## 6. Formatting Pipeline

### 6.1 Pass 1 — Content Assembly

1. **Tag scanning:** `_find_tags()` locates all structural markers
2. **Title extraction:** Parses title, subtitle, author, imprint, images from `[book start]` block
3. **Chapter extraction:** `_extract_chapters()` groups paragraphs by `[chapter]` tags, tracks `[font]`/`[/font]` passthrough blocks, detects `<<<>>>` scene breaks
4. **Verbatim extraction:** Back matter, review request, about author sections copied with raw XML deep-copy (preserves hyperlinks, images, formatting)
5. **Document creation:** New `.docx` built from scratch with:
   - Section 0: Title page (spacer layout)
   - Section 1: Copyright + TOC (with PAGEREF fields)
   - Sections 2+: Chapters (each in its own section)
   - Final sections: Verbatim content

### 6.2 Pass 2 — Section Properties

1. **Settings update:** Enable `evenAndOddHeaders`
2. **Per-section configuration:**
   - Page size (5.5 × 8.5")
   - Margins (mirror, configurable inner)
   - Header/footer content (PAGE fields, alternating headers)
   - Page numbering format (Roman vs. Arabic)
3. **OOXML schema reordering:** `w:sectPr` children reordered for Word compatibility

### 6.3 PDF Conversion

- Uses LibreOffice headless: `libreoffice --headless --convert-to pdf`
- Timeout: 120 seconds
- Output: same filename with `.pdf` extension

---

## 7. API Specification

### 7.1 Endpoints

| Method | Route | Description |
|--------|-------|-------------|
| `GET` | `/` | Home page |
| `GET` | `/format` | Upload/format page |
| `POST` | `/api/analyze` | Upload and analyze a `.docx` |
| `POST` | `/api/format` | Format an analyzed document |
| `GET` | `/api/download/<filename>` | Download formatted `.docx` |
| `GET` | `/api/download/<filename>/pdf` | Download formatted `.pdf` |
| `GET` | `/api/diag/<name>` | Download diagnostic output |
| `GET` | `/api/version` | Return version JSON |

### 7.2 Analysis Response (`/api/analyze`)

```json
{
  "page_setup": {
    "width": 5.5,
    "height": 8.5,
    "orientation": "PORTRAIT"
  },
  "margins": {
    "top": 0.35,
    "bottom": 0.25,
    "inside": 0.55,
    "outside": 0.40,
    "mirror_margins": true
  },
  "sections": [],
  "styles_found": {
    "WA Nrml Pgr": {
      "properties": {},
      "issues": [],
      "count": 150
    }
  },
  "styles_missing": [],
  "paragraph_count": 300,
  "paragraphs_by_style": {},
  "issues": [],
  "warnings": [],
  "chapter_word_count": 45000,
  "upload_id": "uuid-string",
  "original_filename": "manuscript.docx"
}
```

### 7.3 Format Request (`/api/format`)

```json
{
  "upload_id": "uuid-string",
  "font": "Caladea",
  "fontSize": 12,
  "innerMargin": 0.55
}
```

### 7.4 File Storage

- **Uploads:** `umbraformat/uploads/<uuid>.docx` + `<uuid>_analysis.json`
- **Formatted output:** `umbraformat/formatted/<uuid>_formatted.docx` + `<uuid>_formatted.pdf`
- **Name mapping:** `umbraformat/formatted/<uuid>_formatted.name` (maps UUID to original filename)

---

## 8. Configuration

### 8.1 Application Limits

| Setting | Value |
|---------|-------|
| Max upload size | 50 MB |
| Allowed extensions | `.docx` only |
| Server host | `0.0.0.0` |
| Server port | `5001` |

### 8.2 Docker

- Base image: `python:3.12-slim`
- LibreOffice installed for PDF conversion
- Volumes: `uploads/`, `formatted/`

---

## 9. Testing

### 9.1 Current State

No automated test suite exists. Manual testing performed through the web UI with output artifacts in `formatted/`:

| Artifact | Purpose |
|----------|---------|
| `test_stage_a.docx` – `test_stage_e.docx` | Incremental feature tests |
| `test_feat_full.docx` | Full feature integration |
| `test_feat_toc.docx` | Table of contents |
| `test_feat_pageref.docx` | Page reference fields |
| `test_feat_bookmarks.docx` | Bookmarks |
| `diag_minimal.docx` | Diagnostic output |

### 9.2 Recommended Test Coverage

- **Unit tests:** Tag parsing, style generation, margin calculations
- **Integration tests:** Full pipeline with sample manuscripts
- **Regression tests:** Document comparison against known-good outputs
- **Edge cases:** Empty chapters, missing tags, oversized images, 34+ chapters

---

## 10. Known Constraints & Limitations

1. **No undo/history:** Formatting is destructive — original file is not modified, but formatted output replaces previous version
2. **Single-user:** No concurrent session management; stateless file-based storage
3. **Tag-dependent:** Manuscript must use exact tag syntax; malformed tags cause silent failures
4. **Fixed page size:** 5.5 × 8.5 inches hardcoded; not configurable for other trim sizes
5. **Font-limited:** Only Caladea (body) and Georgia (headings) defined in spec
6. **No database:** All state stored on filesystem; no persistence across container restarts unless volumes mounted
7. **LibreOffice dependency:** PDF conversion requires LibreOffice headless, adding ~300 MB to Docker image

---

## 11. Future Considerations

- Automated test suite with pytest
- Support for additional trim sizes (6×9, 5×8)
- Custom font configuration
- Batch processing (multiple manuscripts)
- Version history / rollback
- Database-backed project tracking
- Webhook notifications on format completion
- Pre-flight checks for common manuscript issues (missing tags, inconsistent styles)