docs: add staged-download + atomic-import design (pipeline fix)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/superpowers/specs/2026-07-11-lyra-staged-download-import-design.md

@@ -0,0 +1,139 @@
+# Lyra — Staged Download + Atomic Import (Design)
+
+**Date:** 2026-07-11
+**Status:** Approved design, ready for implementation planning
+**Scope:** Acquisition-engine pipeline fix (slice 1 hardening)
+
+## Problem
+
+Downloads write **directly into the final library folder** `/music/{Artist}/{Album (Year)}/`
+(`pipeline.py`: `dest = album_dir(dest_root, target)`). When a download partially fails
+and the album is retried (or re-grabbed) — which the slice-2 monitor now does routinely —
+the new attempt's files land in the **same folder** beside the failed attempt's orphans.
+The completeness check (`result.track_count < expected`) only detects *too few* files, never
+*too many*, so the mixed folder passes; the tagger's `plan_track_names` then renumbers **every**
+audio file in the folder by sorted order, absorbing the orphans as bogus extra tracks.
+
+**Observed:** Radiohead *In Rainbows* imported as a 12-file album — the correct 10 tracks
+plus two orphaned duplicates (`11 08. Radiohead - House Of Cards.flac`,
+`12 10. Radiohead - Videotape.flac`) from a failed first attempt, plus a leftover streamrip
+subfolder. A milder form (leftover `[FLAC][24B-96kHz]/__artwork` subdir) appears even on clean
+imports because `_flatten_audio` leaves non-audio subfolders behind.
+
+This directly contradicts the original acquisition-engine spec ("Download writes to a temp dir;
+only Import commits to `/music`. A crash never leaves a half-file in the library.") — that
+isolation was never implemented.
+
+## Goals
+
+- A partial or failed download **never** touches the library.
+- A retry or re-grab starts from a **clean slate** — no cross-attempt or cross-job file mixing.
+- An imported album folder contains **exactly** the album's tagged tracks (plus one cover image)
+  and nothing else — no streamrip nesting, no `__artwork` leftovers.
+- Upgrades cleanly **replace** the previous lower-quality files.
+- Fully offline-testable (fake adapters + temp dirs); no real downloads in the suite.
+
+## Non-goals
+
+- Changing adapters or the tagger (they keep operating on whatever directory they're handed).
+- Changing match/rank/confidence/intake logic.
+- Multi-disc or multi-source album stitching.
+
+## Design
+
+### Staging directory
+
+A new helper `staging_dir(dest_root, job_id) -> "{dest_root}/.staging/{job_id}"`. It lives on
+the **same filesystem** as the library, so the final placement is a cheap same-device move, and
+`.staging` (a dot-dir at the library root) sits outside the artist tree so library scans ignore it.
+
+### Pipeline flow (changed stages only)
+
+```
+④ DOWNLOAD → into a fresh staging dir (cleared before EACH candidate attempt)
+⑤ TAG      → completeness check + tagging run against the staging copy
+⑥ IMPORT   → move ONLY the tagged audio (+ one cover.jpg) into the final album_dir,
+             replacing any existing folder there; then remove the staging dir.
+```
+
+- **Download:** `staging = staging_dir(dest_root, job_id)`. For each ranked candidate, the staging
+  dir is cleared to empty *before* the adapter downloads into it, so a failed candidate's partial
+  files never mix into the next candidate's attempt. The winning candidate leaves its files alone
+  in staging. (Adapters are unchanged — they still `download(candidate, dest, on_progress)` into
+  whatever `dest` they're given; `dest` is now the staging dir.)
+- **Tag:** unchanged logic, but the tag target is the staging dir (`result.path or staging`). The
+  completeness check (`result.track_count < expected`) is unchanged and now measures the isolated
+  staging copy.
+- **Import (option A — clean move):** a new `import_album(staging, final) -> final`:
+  1. If `final` already exists (an upgrade), remove it.
+  2. Create a fresh `final`.
+  3. Move every **audio file** (by `_AUDIO_EXT`) from the staging root into `final`. These are the
+     tracks the tagger already renamed to `NN Title.ext`.
+  4. If a cover image exists anywhere in staging (`cover.*` / `folder.*`), move one to
+     `final/cover.jpg`. Everything else in staging (nested subfolders, streamrip artifacts) is
+     discarded.
+  `LibraryItem.path` remains the final `album_dir`.
+- **Cleanup:** the job's staging dir is removed at the end of the download→import section whether
+  the job succeeded or failed (a `try/finally` around those stages). A partial/failed download
+  leaves the library untouched and no staging residue.
+
+### Crash safety
+
+On worker startup, `clear_staging_root(dest_root)` removes `{dest_root}/.staging` wholesale, so
+staging dirs orphaned by a crash mid-pipeline are swept before the loop begins. (Staging holds no
+committed state — anything there is by definition incomplete.)
+
+### Interaction with existing behavior
+
+- **Intake dedup** (unchanged) still short-circuits before download when the library already has a
+  copy at/above cutoff (monitored jobs) or any copy (manual jobs), so import is reached only when a
+  write is intended.
+- **Upgrade replace:** `_import`'s DB upsert already updates the `LibraryItem` row only when the new
+  `qualityClass` is strictly higher; `import_album` replaces the on-disk folder when import is
+  reached. Because intake only lets a monitored upgrade through when the current copy is below
+  cutoff and the ranker picks the best available candidate, the replaced files are the intended
+  better copy. (Edge case where "best available" equals the existing quality replaces same-quality
+  files — wasteful but not corrupting; noted as a minor follow-up, not addressed here.)
+
+## Shared interfaces
+
+```python
+# worker/lyra_worker/library.py
+def staging_dir(dest_root: str, job_id: str) -> str: ...        # "{dest_root}/.staging/{job_id}"
+def clear_staging_root(dest_root: str) -> None: ...             # rm -rf "{dest_root}/.staging"
+def import_album(staging: str, final: str) -> str: ...          # clean audio(+cover) move → final
+
+# worker/lyra_worker/pipeline.py
+#   download stage: dest = staging_dir(dest_root, job_id); clear before each candidate
+#   tag stage:      operates on the staging path
+#   import stage:   final = album_dir(dest_root, target); import_album(staging, final)
+#   try/finally around download→import removes the job's staging dir
+
+# worker/lyra_worker/main.py
+#   run_forever(): clear_staging_root(dest_root) once before the claim loop
+```
+
+## Testing strategy
+
+Offline, zero real downloads:
+
+- **`staging_dir` / `clear_staging_root`** — pure path + fs tests.
+- **`import_album`** (temp dirs) — moves only audio into a fresh `final`; carries one `cover.jpg`;
+  discards nested subfolders/artifacts; replaces an existing `final` (upgrade); leaves no staging
+  residue.
+- **Pipeline, fake adapter that writes files into the staging `dest`:**
+  - happy path → the album lands in `final` with exactly the expected tracks; staging is gone.
+  - **partial download → needs_attention AND the library is untouched** (no `final` dir created).
+  - **regression (the In Rainbows bug):** a first job whose download partially fails, then a second
+    job that fully downloads → `final` contains exactly the expected tracks with **no orphans**.
+    This test fails against the pre-fix code (direct-to-library writes) and passes after.
+- **Backward-compat:** existing pipeline/tag tests are updated for the new staging dest and the
+  tag-target/import-path split (the tagger is now called on the staging dir; the recorded library
+  path is the final `album_dir`). Every worker test stays green.
+
+## Notes
+
+This is a self-contained slice-1 pipeline hardening. After it lands: re-enable the monitor
+(`monitor.enabled=true`), re-grab *In Rainbows* (already reset to `wanted`), and confirm the folder
+imports clean. The other deferred-hardening items (crash-resume mid-pipeline, per-track duration
+matching, cover-art embedding) remain out of scope.