From a4ae60a99e6eefa35caa92bbab7455061989fc50 Mon Sep 17 00:00:00 2001 From: Jonathan Date: Sat, 11 Jul 2026 16:45:58 +0200 Subject: [PATCH] docs: add library-scan design (ingest on-disk albums as have+monitored) Co-Authored-By: Claude Opus 4.8 (1M context) --- .../2026-07-11-lyra-library-scan-design.md | 133 ++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-11-lyra-library-scan-design.md diff --git a/docs/superpowers/specs/2026-07-11-lyra-library-scan-design.md b/docs/superpowers/specs/2026-07-11-lyra-library-scan-design.md new file mode 100644 index 0000000..00951a6 --- /dev/null +++ b/docs/superpowers/specs/2026-07-11-lyra-library-scan-design.md @@ -0,0 +1,133 @@ +# Lyra — Library Scan (Design) + +**Date:** 2026-07-11 +**Status:** Approved design, ready for implementation planning +**Scope:** Ingest an existing on-disk library into Lyra's have + monitor state. + +## Problem / Goal + +Lyra only "knows about" albums it downloaded (via `LibraryItem`/`MonitoredRelease` rows). +Albums already on disk — from prior downloads whose DB rows were lost, or a pre-existing +collection — are invisible: not shown as owned, not monitored, and re-downloadable. A +**library scan** walks `/music`, matches each album to MusicBrainz, and records it as +**have** (`LibraryItem`) and **on monitor** (`MonitoredRelease`, fulfilled), with the artist +**followed** (`WatchedArtist`), so the library manager reflects the real collection. + +## Decisions (from brainstorming) + +- **Identification:** folder names first (`Artist/Album (Year)/`), falling back to embedded + tags (mutagen) when the folder doesn't yield an artist+album. +- **MusicBrainz matching:** resolve every album; **skip (log) anything MB can't match** — only + matched albums are recorded (nothing half-recorded). +- **Trigger:** a manual **"Scan library"** button (no auto-scan on startup — avoids re-hitting + MB for the whole library on every boot). +- **Artists:** scanned artists become **followed** (`WatchedArtist`, `autoMonitorFuture=false`) + so they appear under `/artists`. + +## Non-goals + +- No downloading, tagging, or file moves — scan is read-only on `/music`. +- No auto-scan on startup; no filesystem watching. +- No multi-disc/compilation special handling; each album folder is one album. +- No re-tagging or re-organizing existing files. + +## Architecture + +The scan runs in the **worker** (it owns the `/music` mount, the MusicBrainz resolver, and +mutagen). The worker has no HTTP server, so the UI triggers it through the database: + +``` +[Settings "Scan library" button] --POST /api/scan--> Config: scan.requested = "true" + │ +[worker loop] --sees scan.requested--> scan_library() --> upserts LibraryItem/MonitoredRelease/WatchedArtist + └─> Config: scan.result = "imported X, skipped Y", clears scan.requested +[Settings] --GET /api/scan--> reads Config: scan.result (status line) +``` + +This mirrors the existing monitor-tick pattern (the worker loop already reads config and does +periodic work). No new transport. + +## The scan (`worker/lyra_worker/scan.py`) + +`scan_library(conn, resolver, dest_root="/music") -> ScanResult(imported, skipped)`. For each +`Artist/Album[ (Year)]/` directory under `dest_root` (skipping `.staging` and dot-dirs): + +1. **Identify** artist + album + year: + - Parse the folder names: artist = parent dir; `album, year = parse "Name (YYYY)"` (year + optional). + - If the folder parse yields a blank artist or album, read the first audio file's embedded + tags (mutagen: `artist`, `album`) as a fallback. + - Skip folders with no audio files. +2. **Quality:** read the first audio file via mutagen (`bits_per_sample`, `sample_rate`, + lossless-by-extension) → build a `Quality` → `quality_class` (3/2/1). This is the recorded + `currentQualityClass`/`LibraryItem.qualityClass`. +3. **Resolve** `(artist, album)` via the MusicBrainz resolver → canonical artist/album/year + + **`rgMbid`** + **`artistMbid`**. **No match → skip** (increment `skipped`, log). +4. **Record (idempotent upserts):** + - **`WatchedArtist`** — `INSERT ... ON CONFLICT (mbid) DO NOTHING` when `artistMbid` is + present (`autoMonitorFuture=false`, `monitorFrom=now`). If `artistMbid` is blank, the + release is monitored standalone (`watchedArtistId=null`). + - **`MonitoredRelease`** — `INSERT ... ON CONFLICT (rgMbid) DO UPDATE SET monitored=true, + state='fulfilled', currentQualityClass=, firstGrabbedAt=COALESCE(existing, now), + watchedArtistId=`. ("on monitor") + - **`LibraryItem`** — `INSERT ... ON CONFLICT (artist, album) DO NOTHING` (path=`album_dir`, + source `"scan"`, format, qualityClass). Never overwrites a real download's provenance. + ("have") + - Count as `imported` when a new `LibraryItem` (or monitored record) is created. + +Re-scanning is safe: existing rows are updated-or-left, never duplicated. + +## Data-model touch-points + +- **`MBTarget`** gains `rg_mbid: str = ""` and `artist_mbid: str = ""`; the `MusicBrainzResolver` + already computes the release-group and artist-credit — it fills these (they are currently + discarded). This also lets pipeline jobs link to release-groups later. +- **`LibraryItem.requestId` becomes nullable** (`String?`). A scanned "have" record has no + originating `Request`; making the FK optional lets scanned items exist without a synthetic + request (which would otherwise flood the queue view). Downloads still set `requestId`. Additive, + backward-compatible migration (`DROP NOT NULL`). +- **Config keys:** `scan.requested` (web sets `"true"`, worker clears) and `scan.result` (worker + writes a human summary + timestamp; Settings displays it). Plain (non-secret) `Config` rows. + +Everything else reuses existing tables (`WatchedArtist`, `MonitoredRelease`, `LibraryItem`). + +## Web + +- `POST /api/scan` → upsert `Config` `scan.requested="true"` → 202. +- `GET /api/scan` → return `Config` `scan.result` (or "never run"). +- **Settings** gains a "Scan library" button + a status line (last result), following the + existing settings-form style. + +## Worker wiring + +`run_forever()`'s loop, alongside the monitor tick: if `config["scan.requested"]` is truthy, run +`scan_library(conn, resolver, DEST_ROOT)`, write `scan.result`, and clear `scan.requested`. +Wrapped in try/except (log, rollback) so a scan error can't kill the worker — consistent with the +monitor/pipeline guards. + +## Testing + +Offline, zero real MB/downloads: + +- **`scan_library`** unit tests with a temp `/music` tree, a **fake resolver** (canned MBTarget + incl. `rg_mbid`/`artist_mbid`, or `None` to force a skip), and a seeded DB: + - folder-name identification; tag fallback when the folder is blank; skip folders with no audio. + - quality read (a real tiny FLAC fixture, or a fake quality reader seam) → correct + `quality_class`. + - a matched album creates `LibraryItem` (have) + `MonitoredRelease` (monitored, fulfilled) + + `WatchedArtist` (followed); an unmatched album is skipped and records nothing. + - **idempotent re-scan** — running twice doesn't duplicate rows or flip a real download's + `source` to `scan`. +- **Resolver extension** — the opt-in live resolver test asserts `rg_mbid`/`artist_mbid` are + populated; offline construction stays lazy. +- **Web** — `POST`/`GET /api/scan` route tests against seeded `lyra_test`. +- **Backward-compat** — the nullable-`requestId` migration doesn't break existing pipeline/import + tests; the full worker + web suites stay green. + +## Notes + +Reading audio quality via mutagen is the one place the scan touches file internals; the mutagen +import stays lazy (as in the tagger). After this lands: click **Scan library**, and the existing +`/music` albums (e.g. *Continuum*, *Sob Rock*) show up as have + monitored with their real +quality, and their artists appear under `/artists`. Deferred: auto-scan/watching, multi-disc +handling, cover/tag repair.