docs: add library-scan design (ingest on-disk albums as have+monitored)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 16:45:58 +02:00
parent f59f972c1b
commit a4ae60a99e
1 changed files with 133 additions and 0 deletions
@@ -0,0 +1,133 @@
+# Lyra — Library Scan (Design)
+
+**Date:** 2026-07-11
+**Status:** Approved design, ready for implementation planning
+**Scope:** Ingest an existing on-disk library into Lyra's have + monitor state.
+
+## Problem / Goal
+
+Lyra only "knows about" albums it downloaded (via `LibraryItem`/`MonitoredRelease` rows).
+Albums already on disk — from prior downloads whose DB rows were lost, or a pre-existing
+collection — are invisible: not shown as owned, not monitored, and re-downloadable. A
+**library scan** walks `/music`, matches each album to MusicBrainz, and records it as
+**have** (`LibraryItem`) and **on monitor** (`MonitoredRelease`, fulfilled), with the artist
+**followed** (`WatchedArtist`), so the library manager reflects the real collection.
+
+## Decisions (from brainstorming)
+
+- **Identification:** folder names first (`Artist/Album (Year)/`), falling back to embedded
+  tags (mutagen) when the folder doesn't yield an artist+album.
+- **MusicBrainz matching:** resolve every album; **skip (log) anything MB can't match** — only
+  matched albums are recorded (nothing half-recorded).
+- **Trigger:** a manual **"Scan library"** button (no auto-scan on startup — avoids re-hitting
+  MB for the whole library on every boot).
+- **Artists:** scanned artists become **followed** (`WatchedArtist`, `autoMonitorFuture=false`)
+  so they appear under `/artists`.
+
+## Non-goals
+
+- No downloading, tagging, or file moves — scan is read-only on `/music`.
+- No auto-scan on startup; no filesystem watching.
+- No multi-disc/compilation special handling; each album folder is one album.
+- No re-tagging or re-organizing existing files.
+
+## Architecture
+
+The scan runs in the **worker** (it owns the `/music` mount, the MusicBrainz resolver, and
+mutagen). The worker has no HTTP server, so the UI triggers it through the database:
+
+```
+[Settings "Scan library" button] --POST /api/scan--> Config: scan.requested = "true"
+                                                          │
+[worker loop] --sees scan.requested--> scan_library() --> upserts LibraryItem/MonitoredRelease/WatchedArtist
+                                                          └─> Config: scan.result = "imported X, skipped Y", clears scan.requested
+[Settings] --GET /api/scan--> reads Config: scan.result   (status line)
+```
+
+This mirrors the existing monitor-tick pattern (the worker loop already reads config and does
+periodic work). No new transport.
+
+## The scan (`worker/lyra_worker/scan.py`)
+
+`scan_library(conn, resolver, dest_root="/music") -> ScanResult(imported, skipped)`. For each
+`Artist/Album[ (Year)]/` directory under `dest_root` (skipping `.staging` and dot-dirs):
+
+1. **Identify** artist + album + year:
+   - Parse the folder names: artist = parent dir; `album, year = parse "Name (YYYY)"` (year
+     optional).
+   - If the folder parse yields a blank artist or album, read the first audio file's embedded
+     tags (mutagen: `artist`, `album`) as a fallback.
+   - Skip folders with no audio files.
+2. **Quality:** read the first audio file via mutagen (`bits_per_sample`, `sample_rate`,
+   lossless-by-extension) → build a `Quality` → `quality_class` (3/2/1). This is the recorded
+   `currentQualityClass`/`LibraryItem.qualityClass`.
+3. **Resolve** `(artist, album)` via the MusicBrainz resolver → canonical artist/album/year +
+   **`rgMbid`** + **`artistMbid`**. **No match → skip** (increment `skipped`, log).
+4. **Record (idempotent upserts):**
+   - **`WatchedArtist`** — `INSERT ... ON CONFLICT (mbid) DO NOTHING` when `artistMbid` is
+     present (`autoMonitorFuture=false`, `monitorFrom=now`). If `artistMbid` is blank, the
+     release is monitored standalone (`watchedArtistId=null`).
+   - **`MonitoredRelease`** — `INSERT ... ON CONFLICT (rgMbid) DO UPDATE SET monitored=true,
+     state='fulfilled', currentQualityClass=<q>, firstGrabbedAt=COALESCE(existing, now),
+     watchedArtistId=<artist>`. ("on monitor")
+   - **`LibraryItem`** — `INSERT ... ON CONFLICT (artist, album) DO NOTHING` (path=`album_dir`,
+     source `"scan"`, format, qualityClass). Never overwrites a real download's provenance.
+     ("have")
+   - Count as `imported` when a new `LibraryItem` (or monitored record) is created.
+
+Re-scanning is safe: existing rows are updated-or-left, never duplicated.
+
+## Data-model touch-points
+
+- **`MBTarget`** gains `rg_mbid: str = ""` and `artist_mbid: str = ""`; the `MusicBrainzResolver`
+  already computes the release-group and artist-credit — it fills these (they are currently
+  discarded). This also lets pipeline jobs link to release-groups later.
+- **`LibraryItem.requestId` becomes nullable** (`String?`). A scanned "have" record has no
+  originating `Request`; making the FK optional lets scanned items exist without a synthetic
+  request (which would otherwise flood the queue view). Downloads still set `requestId`. Additive,
+  backward-compatible migration (`DROP NOT NULL`).
+- **Config keys:** `scan.requested` (web sets `"true"`, worker clears) and `scan.result` (worker
+  writes a human summary + timestamp; Settings displays it). Plain (non-secret) `Config` rows.
+
+Everything else reuses existing tables (`WatchedArtist`, `MonitoredRelease`, `LibraryItem`).
+
+## Web
+
+- `POST /api/scan` → upsert `Config` `scan.requested="true"` → 202.
+- `GET /api/scan` → return `Config` `scan.result` (or "never run").
+- **Settings** gains a "Scan library" button + a status line (last result), following the
+  existing settings-form style.
+
+## Worker wiring
+
+`run_forever()`'s loop, alongside the monitor tick: if `config["scan.requested"]` is truthy, run
+`scan_library(conn, resolver, DEST_ROOT)`, write `scan.result`, and clear `scan.requested`.
+Wrapped in try/except (log, rollback) so a scan error can't kill the worker — consistent with the
+monitor/pipeline guards.
+
+## Testing
+
+Offline, zero real MB/downloads:
+
+- **`scan_library`** unit tests with a temp `/music` tree, a **fake resolver** (canned MBTarget
+  incl. `rg_mbid`/`artist_mbid`, or `None` to force a skip), and a seeded DB:
+  - folder-name identification; tag fallback when the folder is blank; skip folders with no audio.
+  - quality read (a real tiny FLAC fixture, or a fake quality reader seam) → correct
+    `quality_class`.
+  - a matched album creates `LibraryItem` (have) + `MonitoredRelease` (monitored, fulfilled) +
+    `WatchedArtist` (followed); an unmatched album is skipped and records nothing.
+  - **idempotent re-scan** — running twice doesn't duplicate rows or flip a real download's
+    `source` to `scan`.
+- **Resolver extension** — the opt-in live resolver test asserts `rg_mbid`/`artist_mbid` are
+  populated; offline construction stays lazy.
+- **Web** — `POST`/`GET /api/scan` route tests against seeded `lyra_test`.
+- **Backward-compat** — the nullable-`requestId` migration doesn't break existing pipeline/import
+  tests; the full worker + web suites stay green.
+
+## Notes
+
+Reading audio quality via mutagen is the one place the scan touches file internals; the mutagen
+import stays lazy (as in the tagger). After this lands: click **Scan library**, and the existing
+`/music` albums (e.g. *Continuum*, *Sob Rock*) show up as have + monitored with their real
+quality, and their artists appear under `/artists`. Deferred: auto-scan/watching, multi-disc
+handling, cover/tag repair.