diff --git a/docs/superpowers/specs/2026-07-11-lyra-staged-download-import-design.md b/docs/superpowers/specs/2026-07-11-lyra-staged-download-import-design.md new file mode 100644 index 0000000..1219fb3 --- /dev/null +++ b/docs/superpowers/specs/2026-07-11-lyra-staged-download-import-design.md @@ -0,0 +1,139 @@ +# Lyra — Staged Download + Atomic Import (Design) + +**Date:** 2026-07-11 +**Status:** Approved design, ready for implementation planning +**Scope:** Acquisition-engine pipeline fix (slice 1 hardening) + +## Problem + +Downloads write **directly into the final library folder** `/music/{Artist}/{Album (Year)}/` +(`pipeline.py`: `dest = album_dir(dest_root, target)`). When a download partially fails +and the album is retried (or re-grabbed) — which the slice-2 monitor now does routinely — +the new attempt's files land in the **same folder** beside the failed attempt's orphans. +The completeness check (`result.track_count < expected`) only detects *too few* files, never +*too many*, so the mixed folder passes; the tagger's `plan_track_names` then renumbers **every** +audio file in the folder by sorted order, absorbing the orphans as bogus extra tracks. + +**Observed:** Radiohead *In Rainbows* imported as a 12-file album — the correct 10 tracks +plus two orphaned duplicates (`11 08. Radiohead - House Of Cards.flac`, +`12 10. Radiohead - Videotape.flac`) from a failed first attempt, plus a leftover streamrip +subfolder. A milder form (leftover `[FLAC][24B-96kHz]/__artwork` subdir) appears even on clean +imports because `_flatten_audio` leaves non-audio subfolders behind. + +This directly contradicts the original acquisition-engine spec ("Download writes to a temp dir; +only Import commits to `/music`. A crash never leaves a half-file in the library.") — that +isolation was never implemented. + +## Goals + +- A partial or failed download **never** touches the library. +- A retry or re-grab starts from a **clean slate** — no cross-attempt or cross-job file mixing. +- An imported album folder contains **exactly** the album's tagged tracks (plus one cover image) + and nothing else — no streamrip nesting, no `__artwork` leftovers. +- Upgrades cleanly **replace** the previous lower-quality files. +- Fully offline-testable (fake adapters + temp dirs); no real downloads in the suite. + +## Non-goals + +- Changing adapters or the tagger (they keep operating on whatever directory they're handed). +- Changing match/rank/confidence/intake logic. +- Multi-disc or multi-source album stitching. + +## Design + +### Staging directory + +A new helper `staging_dir(dest_root, job_id) -> "{dest_root}/.staging/{job_id}"`. It lives on +the **same filesystem** as the library, so the final placement is a cheap same-device move, and +`.staging` (a dot-dir at the library root) sits outside the artist tree so library scans ignore it. + +### Pipeline flow (changed stages only) + +``` +④ DOWNLOAD → into a fresh staging dir (cleared before EACH candidate attempt) +⑤ TAG → completeness check + tagging run against the staging copy +⑥ IMPORT → move ONLY the tagged audio (+ one cover.jpg) into the final album_dir, + replacing any existing folder there; then remove the staging dir. +``` + +- **Download:** `staging = staging_dir(dest_root, job_id)`. For each ranked candidate, the staging + dir is cleared to empty *before* the adapter downloads into it, so a failed candidate's partial + files never mix into the next candidate's attempt. The winning candidate leaves its files alone + in staging. (Adapters are unchanged — they still `download(candidate, dest, on_progress)` into + whatever `dest` they're given; `dest` is now the staging dir.) +- **Tag:** unchanged logic, but the tag target is the staging dir (`result.path or staging`). The + completeness check (`result.track_count < expected`) is unchanged and now measures the isolated + staging copy. +- **Import (option A — clean move):** a new `import_album(staging, final) -> final`: + 1. If `final` already exists (an upgrade), remove it. + 2. Create a fresh `final`. + 3. Move every **audio file** (by `_AUDIO_EXT`) from the staging root into `final`. These are the + tracks the tagger already renamed to `NN Title.ext`. + 4. If a cover image exists anywhere in staging (`cover.*` / `folder.*`), move one to + `final/cover.jpg`. Everything else in staging (nested subfolders, streamrip artifacts) is + discarded. + `LibraryItem.path` remains the final `album_dir`. +- **Cleanup:** the job's staging dir is removed at the end of the download→import section whether + the job succeeded or failed (a `try/finally` around those stages). A partial/failed download + leaves the library untouched and no staging residue. + +### Crash safety + +On worker startup, `clear_staging_root(dest_root)` removes `{dest_root}/.staging` wholesale, so +staging dirs orphaned by a crash mid-pipeline are swept before the loop begins. (Staging holds no +committed state — anything there is by definition incomplete.) + +### Interaction with existing behavior + +- **Intake dedup** (unchanged) still short-circuits before download when the library already has a + copy at/above cutoff (monitored jobs) or any copy (manual jobs), so import is reached only when a + write is intended. +- **Upgrade replace:** `_import`'s DB upsert already updates the `LibraryItem` row only when the new + `qualityClass` is strictly higher; `import_album` replaces the on-disk folder when import is + reached. Because intake only lets a monitored upgrade through when the current copy is below + cutoff and the ranker picks the best available candidate, the replaced files are the intended + better copy. (Edge case where "best available" equals the existing quality replaces same-quality + files — wasteful but not corrupting; noted as a minor follow-up, not addressed here.) + +## Shared interfaces + +```python +# worker/lyra_worker/library.py +def staging_dir(dest_root: str, job_id: str) -> str: ... # "{dest_root}/.staging/{job_id}" +def clear_staging_root(dest_root: str) -> None: ... # rm -rf "{dest_root}/.staging" +def import_album(staging: str, final: str) -> str: ... # clean audio(+cover) move → final + +# worker/lyra_worker/pipeline.py +# download stage: dest = staging_dir(dest_root, job_id); clear before each candidate +# tag stage: operates on the staging path +# import stage: final = album_dir(dest_root, target); import_album(staging, final) +# try/finally around download→import removes the job's staging dir + +# worker/lyra_worker/main.py +# run_forever(): clear_staging_root(dest_root) once before the claim loop +``` + +## Testing strategy + +Offline, zero real downloads: + +- **`staging_dir` / `clear_staging_root`** — pure path + fs tests. +- **`import_album`** (temp dirs) — moves only audio into a fresh `final`; carries one `cover.jpg`; + discards nested subfolders/artifacts; replaces an existing `final` (upgrade); leaves no staging + residue. +- **Pipeline, fake adapter that writes files into the staging `dest`:** + - happy path → the album lands in `final` with exactly the expected tracks; staging is gone. + - **partial download → needs_attention AND the library is untouched** (no `final` dir created). + - **regression (the In Rainbows bug):** a first job whose download partially fails, then a second + job that fully downloads → `final` contains exactly the expected tracks with **no orphans**. + This test fails against the pre-fix code (direct-to-library writes) and passes after. +- **Backward-compat:** existing pipeline/tag tests are updated for the new staging dest and the + tag-target/import-path split (the tagger is now called on the staging dir; the recorded library + path is the final `album_dir`). Every worker test stays green. + +## Notes + +This is a self-contained slice-1 pipeline hardening. After it lands: re-enable the monitor +(`monitor.enabled=true`), re-grab *In Rainbows* (already reset to `wanted`), and confirm the folder +imports clean. The other deferred-hardening items (crash-resume mid-pipeline, per-track duration +matching, cover-art embedding) remain out of scope.