# Lyra — Acquisition Engine (Design) **Date:** 2026-07-10 **Status:** Approved design, ready for implementation planning **Slice:** 1 of 3 (Acquisition engine) ## Overview Lyra is a self-hosted, multi-source music acquisition and library tool intended to replace Lidarr. It discovers, ranks, downloads, tags, and organizes music from several sources of differing quality and legality: **Qobuz** (lossless/hi-res), **Soulseek** (P2P), **YouTube** (universal lossy fallback), and **Spotify** (used later for discovery/metadata only, not downloading). The full product is built in three slices, in order: 1. **Acquisition engine** — *this spec*. Given an album/track, find the best available source, download it, tag it against MusicBrainz, and file it into a clean library. 2. **Library manager** — Lidarr-style monitoring: wanted lists, watched artists, auto-grab of new releases. Builds on the acquisition engine. 3. **Discovery** — Spotify/recommendation-driven finding of new music, feeding into acquisition. This document designs slice 1 only. Slices 2 and 3 get their own spec → plan → implementation cycles. ### Goals - Given a requested album or track, reliably obtain the best-quality complete copy available across sources, fully automatically. - Quarantine all source-specific complexity behind a uniform adapter interface, so the rest of the system is source-agnostic and adding a source is additive. - Never silently fail: any request that cannot be satisfied lands in an explicit `needs-attention` state. - Be fully developable and testable with **zero real credentials** via fixtures and fake adapters. ### Non-goals (this slice) - Artist monitoring / auto-grab of new releases (slice 2). - Discovery / recommendations (slice 3). - Spotify as a download source (used for metadata/discovery only, later). - Transcoding — originals are kept as-is. - Stitching a single album from multiple Soulseek peers (flagged for later). ## Architecture Three Docker containers on the user's home server, alongside the existing Lidarr/NZBget stack. ``` ┌─────────────────┐ jobs ┌──────────────────┐ │ Next.js app │──────────────▶│ Python worker │ │ (UI + API) │ Postgres │ (acquisition) │ │ │◀──────────────│ │ │ search, request │ status │ streamrip (Qobuz)│ │ queue, progress │ │ yt-dlp (YouTube) │ └─────────────────┘ │ slskd client │ │ │ beets/mutagen tag│ ▼ └────────┬─────────┘ Postgres │ (requests, jobs, ▼ candidates, config, /music (library) library_items) + slskd container ``` **Components:** - **Next.js container (UI + API)** — the user-facing app: search, request an album/track, watch the download queue, browse results, manage settings/creds. Thin API over Postgres. The user's preferred stack. **Never touches source tools directly.** - **Python worker container (acquisition)** — owns all source integration. Polls the job queue, runs the six-stage pipeline, reports progress back through Postgres. Contains streamrip, yt-dlp, the slskd client, and beets/mutagen. - **slskd container** — the off-the-shelf headless Soulseek daemon. Lyra talks to its REST API; we do not reimplement Soulseek. - **Postgres** — single source of truth shared by the Next.js app and the worker. - **Shared `/music` volume** — the destination library. **Key principle:** the Next.js side reasons about *requests and jobs*; the worker reasons about *candidates and files*. Neither reaches into the other's domain. All source complexity is isolated in the worker behind the adapter interface. ### Worker communication Postgres is the single source of truth. The worker **polls a `jobs` table** and claims work — no Redis/Celery for a single-user home server. If push/scale is ever needed, Redis+RQ can be introduced behind the same table without changing the data model. This keeps v1 to three containers (plus Postgres). ## The acquisition pipeline A request flows through six stages, each with one responsibility and a clear hand-off, so each is independently testable. ``` ① INTAKE ──▶ ② MATCH ──▶ ③ RANK ──▶ ④ DOWNLOAD ──▶ ⑤ TAG ──▶ ⑥ IMPORT ``` **① Intake** — resolve the request to a **canonical MusicBrainz target**: release-group + release MBID, artist, ordered tracklist, expected track count, and per-track durations. Every later stage measures against this anchor. **② Match** — query each source adapter **in parallel**. Each returns zero or more **candidates**, self-describing `{source, format, quality, trackCount, confidence}`. Adapters do not download here; they only report what they could deliver. **③ Rank** — score candidates by the quality-first policy, gated by a match-confidence threshold. A high-quality but wrong/incomplete release loses to a correct lower-quality one. The best qualifying candidate wins, automatically (no user confirmation). Quality ranking (best → worst): `Qobuz hi-res → Qobuz lossless → Soulseek FLAC → Soulseek MP3 → YouTube` **④ Download** — the winning adapter fetches and streams progress back. On failure, **fall through** to the next-best candidate, then the next source. YouTube is the floor. **⑤ Tag** — verify integrity (complete, not truncated/silent), tag against MusicBrainz via beets/mutagen, embed cover art. Originals kept; no transcoding. **⑥ Import** — dedupe against the existing library (do not re-grab), move into the fresh layout `Artist/Album (Year)/## Title.flac`, record the winning source+quality, mark the request done. ### Job state machine ``` requested → matching → matched → downloading → tagging → imported ↓ ↓ ↓ ↓ (no match / all sources failed / bad file) → needs-attention ``` Nothing silently disappears — an unsatisfiable request lands in `needs-attention`, not lost. ## Source adapters Every source implements the **same contract**; the pipeline is source-agnostic. Adding Tidal/Deezer/Bandcamp later is a new adapter and nothing else. ``` SourceAdapter: name → "qobuz" | "soulseek" | "youtube" tier → base quality rank (Qobuz=0, Soulseek=1, YouTube=2) health() → configured & reachable? (creds valid, slskd up) search(target: MBTarget) → Candidate[] # metadata only download(candidate, dest, onProgress) → DownloadResult ``` A **Candidate** is self-describing so ranking needs no source-specific logic: ``` Candidate { source, sourceRef # opaque handle back to the adapter format # FLAC / MP3 / OPUS / AAC quality # bitDepth+sampleRate (24/96) or bitrate (320k) trackCount # for completeness vs. MB tracklist confidence # 0–1: how sure this is the right release } ``` **Per-adapter specifics (isolated in the worker):** | Adapter | Built on | Auth | Notes | |------------|-----------------|-------------------------------|-------| | Qobuz | `streamrip` | app id/secret + account token | True lossless/hi-res; cleanest matches. Primary source. | | Soulseek | `slskd` REST API| slskd handles SoulSeek login | Peer files grouped into album candidates; quality inferred from extension/bitrate; dead/slow peers → next candidate. | | YouTube | `yt-dlp` | none (cookies optional) | Universal lossy floor; match artist+album → YT Music album/playlist. | **Confidence scoring** is shared and source-agnostic: compares each candidate to the MB target on title/artist fuzzy match, track count, and total duration. Below threshold → discarded (prevents grabbing a live/bootleg when the studio album was requested). ## Data model Postgres, shared by both containers. ``` artists mbid, name, ... # cached MB metadata releases mbid, artist_id, title, year, # the album (release-group/release) track_count, tracklist(json) requests id, release_id | track, requested_at, # user intent status jobs id, request_id, state, attempts, # pipeline execution; current_stage, error, updated_at, # state machine + claim claimed_at candidates id, job_id, source, format, quality, # what Match found track_count, confidence, source_ref, chosen(bool) library_items release_id, path, source, quality, # what is on disk → dedupe imported_at config key, value (+ encrypted credentials) # Qobuz token, slskd url, paths ``` **Flow across the split:** 1. Next.js writes a `request` + an initial `job (state=requested)`. 2. Worker polls `jobs`, claims one (sets `claimed_at`), advances it stage by stage, writing `candidates` and updating `job.state`/`current_stage`. 3. Next.js reads `job.state`/`current_stage` to render live progress (poll or SSE). 4. On success, worker writes a `library_items` row → future requests dedupe against it. **Credentials** (Qobuz token, slskd URL/key) are stored encrypted in `config`, written via the settings UI, read only by the worker. ## Error handling & edge cases The design assumes sources will fail routinely; graceful fall-through is the norm. **Fall-through ladder:** within a source, try the next-best candidate; across sources, drop to the next tier; YouTube is the floor; if even that fails → `needs-attention`, never silently dropped. | Situation | Handling | |-----------|----------| | No match anywhere | Job → `needs-attention` ("no candidates above threshold"). User may lower threshold or pick manually. | | Soulseek dead/slow peer | Transfer timeout → abandon candidate, try next peer/candidate. Bounded, configurable retries. | | Soulseek partial album | Completeness check fails at Match → candidate deprioritized. v1 does not stitch multi-peer albums. | | Wrong/mislabeled release | Confidence gate on duration + track count catches most; borderline → `needs-attention`. | | Corrupt/truncated download | Integrity check at Tag (track count + non-zero/non-silent) → discard, fall through. | | Already in library | Dedupe at Import against `library_items`, also checked early. Re-grab only if new copy is higher quality (optional upgrade flag). | | Qobuz token expired | `health()` fails fast → job pauses, UI surfaces "re-auth Qobuz". | | Worker crash mid-job | Jobs claimed with `claimed_at`; stale in-progress jobs reclaimed and resumed from `current_stage` (stages idempotent). | | Rate limiting (Qobuz/YT) | Per-adapter backoff + capped retry; exceeding cap → `needs-attention`. | **Idempotency principle:** every stage can be safely re-run. Download writes to a temp dir; only Import commits to `/music`. A crash never leaves a half-file in the library. ## Testing strategy Because source complexity is quarantined behind the adapter interface, most of the system is testable without hitting any real API. - **Adapter contract tests** — one shared suite every adapter must pass (well-formed `Candidate`s; `download()` reports progress and yields a valid result; `health()` behaves). Run against **recorded fixtures** (cassettes of real Qobuz/slskd/yt-dlp responses): fast, offline, deterministic. - **Ranker/confidence tests** — pure functions, no I/O. Synthetic candidate sets + MB targets; assert the right winner and that the confidence gate rejects wrong releases. Highest-value unit surface. - **Pipeline/state-machine tests** — drive a job through all six stages with **fake adapters** (always-succeeds, always-fails, partial) to prove fall-through, `needs-attention`, and crash-resume. - **Integration smoke test** — small opt-in suite hitting real sources for one known album, gated behind an env flag; not run in normal CI, not needed for development. - **Next.js side** — API route tests against a seeded Postgres; a couple of end-to-end UI checks (submit request → watch it move through states). **Goal:** build and test the entire acquisition engine on a laptop with zero real credentials, using fixtures and fakes; real sources enter only for the opt-in smoke test.