Lyra/docs/superpowers/specs/2026-07-10-lyra-acquisition-engine-design.md

Lyra — Acquisition Engine (Design)

Date: 2026-07-10 Status: Approved design, ready for implementation planning Slice: 1 of 3 (Acquisition engine)

Overview

Lyra is a self-hosted, multi-source music acquisition and library tool intended to replace Lidarr. It discovers, ranks, downloads, tags, and organizes music from several sources of differing quality and legality: Qobuz (lossless/hi-res), Soulseek (P2P), YouTube (universal lossy fallback), and Spotify (used later for discovery/metadata only, not downloading).

The full product is built in three slices, in order:

1. Acquisition engine — this spec. Given an album/track, find the best available source, download it, tag it against MusicBrainz, and file it into a clean library.
2. Library manager — Lidarr-style monitoring: wanted lists, watched artists, auto-grab of new releases. Builds on the acquisition engine.
3. Discovery — Spotify/recommendation-driven finding of new music, feeding into acquisition.

This document designs slice 1 only. Slices 2 and 3 get their own spec → plan → implementation cycles.

Goals

• Given a requested album or track, reliably obtain the best-quality complete copy available across sources, fully automatically.
• Quarantine all source-specific complexity behind a uniform adapter interface, so the rest of the system is source-agnostic and adding a source is additive.
• Never silently fail: any request that cannot be satisfied lands in an explicit needs-attention state.
• Be fully developable and testable with zero real credentials via fixtures and fake adapters.

Non-goals (this slice)

• Artist monitoring / auto-grab of new releases (slice 2).
• Discovery / recommendations (slice 3).
• Spotify as a download source (used for metadata/discovery only, later).
• Transcoding — originals are kept as-is.
• Stitching a single album from multiple Soulseek peers (flagged for later).

Architecture

Three Docker containers on the user's home server, alongside the existing Lidarr/NZBget stack.

┌─────────────────┐     jobs      ┌──────────────────┐
│  Next.js app    │──────────────▶│  Python worker   │
│  (UI + API)     │   Postgres    │  (acquisition)   │
│                 │◀──────────────│                  │
│ search, request │   status      │ streamrip (Qobuz)│
│ queue, progress │               │ yt-dlp (YouTube) │
└─────────────────┘               │ slskd client     │
        │                         │ beets/mutagen tag│
        ▼                         └────────┬─────────┘
   Postgres                                │
   (requests, jobs,                        ▼
    candidates, config,               /music (library)
    library_items)                    + slskd container

Components:

• Next.js container (UI + API) — the user-facing app: search, request an album/track, watch the download queue, browse results, manage settings/creds. Thin API over Postgres. The user's preferred stack. Never touches source tools directly.
• Python worker container (acquisition) — owns all source integration. Polls the job queue, runs the six-stage pipeline, reports progress back through Postgres. Contains streamrip, yt-dlp, the slskd client, and beets/mutagen.
• slskd container — the off-the-shelf headless Soulseek daemon. Lyra talks to its REST API; we do not reimplement Soulseek.
• Postgres — single source of truth shared by the Next.js app and the worker.
• Shared /music volume — the destination library.

Key principle: the Next.js side reasons about requests and jobs; the worker reasons about candidates and files. Neither reaches into the other's domain. All source complexity is isolated in the worker behind the adapter interface.

Worker communication

Postgres is the single source of truth. The worker polls a jobs table and claims work — no Redis/Celery for a single-user home server. If push/scale is ever needed, Redis+RQ can be introduced behind the same table without changing the data model. This keeps v1 to three containers (plus Postgres).

The acquisition pipeline

A request flows through six stages, each with one responsibility and a clear hand-off, so each is independently testable.

① INTAKE ──▶ ② MATCH ──▶ ③ RANK ──▶ ④ DOWNLOAD ──▶ ⑤ TAG ──▶ ⑥ IMPORT

① Intake — resolve the request to a canonical MusicBrainz target: release-group + release MBID, artist, ordered tracklist, expected track count, and per-track durations. Every later stage measures against this anchor.

② Match — query each source adapter in parallel. Each returns zero or more candidates, self-describing {source, format, quality, trackCount, confidence}. Adapters do not download here; they only report what they could deliver.

③ Rank — score candidates by the quality-first policy, gated by a match-confidence threshold. A high-quality but wrong/incomplete release loses to a correct lower-quality one. The best qualifying candidate wins, automatically (no user confirmation).

Quality ranking (best → worst): Qobuz hi-res → Qobuz lossless → Soulseek FLAC → Soulseek MP3 → YouTube

④ Download — the winning adapter fetches and streams progress back. On failure, fall through to the next-best candidate, then the next source. YouTube is the floor.

⑤ Tag — verify integrity (complete, not truncated/silent), tag against MusicBrainz via beets/mutagen, embed cover art. Originals kept; no transcoding.

⑥ Import — dedupe against the existing library (do not re-grab), move into the fresh layout Artist/Album (Year)/## Title.flac, record the winning source+quality, mark the request done.

Job state machine

requested → matching → matched → downloading → tagging → imported
                 ↓         ↓          ↓            ↓
              (no match / all sources failed / bad file) → needs-attention

Nothing silently disappears — an unsatisfiable request lands in needs-attention, not lost.

Source adapters

Every source implements the same contract; the pipeline is source-agnostic. Adding Tidal/Deezer/Bandcamp later is a new adapter and nothing else.

SourceAdapter:
  name          → "qobuz" | "soulseek" | "youtube"
  tier          → base quality rank (Qobuz=0, Soulseek=1, YouTube=2)
  health()      → configured & reachable? (creds valid, slskd up)
  search(target: MBTarget)              → Candidate[]   # metadata only
  download(candidate, dest, onProgress) → DownloadResult

A Candidate is self-describing so ranking needs no source-specific logic:

Candidate {
  source, sourceRef     # opaque handle back to the adapter
  format                # FLAC / MP3 / OPUS / AAC
  quality               # bitDepth+sampleRate (24/96) or bitrate (320k)
  trackCount            # for completeness vs. MB tracklist
  confidence            # 0–1: how sure this is the right release
}

Per-adapter specifics (isolated in the worker):

Adapter	Built on	Auth	Notes
Qobuz	streamrip	app id/secret + account token	True lossless/hi-res; cleanest matches. Primary source.
Soulseek	slskd REST API	slskd handles SoulSeek login	Peer files grouped into album candidates; quality inferred from extension/bitrate; dead/slow peers → next candidate.
YouTube	yt-dlp	none (cookies optional)	Universal lossy floor; match artist+album → YT Music album/playlist.

Confidence scoring is shared and source-agnostic: compares each candidate to the MB target on title/artist fuzzy match, track count, and total duration. Below threshold → discarded (prevents grabbing a live/bootleg when the studio album was requested).

Data model

Postgres, shared by both containers.

artists        mbid, name, ...                          # cached MB metadata
releases       mbid, artist_id, title, year,            # the album (release-group/release)
               track_count, tracklist(json)
requests       id, release_id | track, requested_at,    # user intent
               status
jobs           id, request_id, state, attempts,         # pipeline execution;
               current_stage, error, updated_at,        #   state machine + claim
               claimed_at
candidates     id, job_id, source, format, quality,     # what Match found
               track_count, confidence, source_ref,
               chosen(bool)
library_items  release_id, path, source, quality,       # what is on disk → dedupe
               imported_at
config         key, value  (+ encrypted credentials)    # Qobuz token, slskd url, paths

Flow across the split:

1. Next.js writes a request + an initial job (state=requested).
2. Worker polls jobs, claims one (sets claimed_at), advances it stage by stage, writing candidates and updating job.state/current_stage.
3. Next.js reads job.state/current_stage to render live progress (poll or SSE).
4. On success, worker writes a library_items row → future requests dedupe against it.

Credentials (Qobuz token, slskd URL/key) are stored encrypted in config, written via the settings UI, read only by the worker.

Error handling & edge cases

The design assumes sources will fail routinely; graceful fall-through is the norm.

Fall-through ladder: within a source, try the next-best candidate; across sources, drop to the next tier; YouTube is the floor; if even that fails → needs-attention, never silently dropped.

Situation	Handling
No match anywhere	Job → needs-attention ("no candidates above threshold"). User may lower threshold or pick manually.
Soulseek dead/slow peer	Transfer timeout → abandon candidate, try next peer/candidate. Bounded, configurable retries.
Soulseek partial album	Completeness check fails at Match → candidate deprioritized. v1 does not stitch multi-peer albums.
Wrong/mislabeled release	Confidence gate on duration + track count catches most; borderline → needs-attention.
Corrupt/truncated download	Integrity check at Tag (track count + non-zero/non-silent) → discard, fall through.
Already in library	Dedupe at Import against library_items, also checked early. Re-grab only if new copy is higher quality (optional upgrade flag).
Qobuz token expired	health() fails fast → job pauses, UI surfaces "re-auth Qobuz".
Worker crash mid-job	Jobs claimed with claimed_at; stale in-progress jobs reclaimed and resumed from current_stage (stages idempotent).
Rate limiting (Qobuz/YT)	Per-adapter backoff + capped retry; exceeding cap → needs-attention.

Idempotency principle: every stage can be safely re-run. Download writes to a temp dir; only Import commits to /music. A crash never leaves a half-file in the library.

Testing strategy

Because source complexity is quarantined behind the adapter interface, most of the system is testable without hitting any real API.

• Adapter contract tests — one shared suite every adapter must pass (well-formed Candidates; download() reports progress and yields a valid result; health() behaves). Run against recorded fixtures (cassettes of real Qobuz/slskd/yt-dlp responses): fast, offline, deterministic.
• Ranker/confidence tests — pure functions, no I/O. Synthetic candidate sets + MB targets; assert the right winner and that the confidence gate rejects wrong releases. Highest-value unit surface.
• Pipeline/state-machine tests — drive a job through all six stages with fake adapters (always-succeeds, always-fails, partial) to prove fall-through, needs-attention, and crash-resume.
• Integration smoke test — small opt-in suite hitting real sources for one known album, gated behind an env flag; not run in normal CI, not needed for development.
• Next.js side — API route tests against a seeded Postgres; a couple of end-to-end UI checks (submit request → watch it move through states).

Goal: build and test the entire acquisition engine on a laptop with zero real credentials, using fixtures and fakes; real sources enter only for the opt-in smoke test.