Lyra/docs/superpowers/specs/2026-07-10-lyra-acquisition-engine-design.md

# Lyra — Acquisition Engine (Design)

**Date:** 2026-07-10
**Status:** Approved design, ready for implementation planning
**Slice:** 1 of 3 (Acquisition engine)

## Overview

Lyra is a self-hosted, multi-source music acquisition and library tool intended to
replace Lidarr. It discovers, ranks, downloads, tags, and organizes music from
several sources of differing quality and legality: **Qobuz** (lossless/hi-res),
**Soulseek** (P2P), **YouTube** (universal lossy fallback), and **Spotify** (used
later for discovery/metadata only, not downloading).

The full product is built in three slices, in order:

1. **Acquisition engine** — *this spec*. Given an album/track, find the best
   available source, download it, tag it against MusicBrainz, and file it into a
   clean library.
2. **Library manager** — Lidarr-style monitoring: wanted lists, watched artists,
   auto-grab of new releases. Builds on the acquisition engine.
3. **Discovery** — Spotify/recommendation-driven finding of new music, feeding
   into acquisition.

This document designs slice 1 only. Slices 2 and 3 get their own spec → plan →
implementation cycles.

### Goals

- Given a requested album or track, reliably obtain the best-quality complete copy
  available across sources, fully automatically.
- Quarantine all source-specific complexity behind a uniform adapter interface, so
  the rest of the system is source-agnostic and adding a source is additive.
- Never silently fail: any request that cannot be satisfied lands in an explicit
  `needs-attention` state.
- Be fully developable and testable with **zero real credentials** via fixtures and
  fake adapters.

### Non-goals (this slice)

- Artist monitoring / auto-grab of new releases (slice 2).
- Discovery / recommendations (slice 3).
- Spotify as a download source (used for metadata/discovery only, later).
- Transcoding — originals are kept as-is.
- Stitching a single album from multiple Soulseek peers (flagged for later).

## Architecture

Three Docker containers on the user's home server, alongside the existing
Lidarr/NZBget stack.

```
┌─────────────────┐     jobs      ┌──────────────────┐
│  Next.js app    │──────────────▶│  Python worker   │
│  (UI + API)     │   Postgres    │  (acquisition)   │
│                 │◀──────────────│                  │
│ search, request │   status      │ streamrip (Qobuz)│
│ queue, progress │               │ yt-dlp (YouTube) │
└─────────────────┘               │ slskd client     │
        │                         │ beets/mutagen tag│
        ▼                         └────────┬─────────┘
   Postgres                                │
   (requests, jobs,                        ▼
    candidates, config,               /music (library)
    library_items)                    + slskd container
```

**Components:**

- **Next.js container (UI + API)** — the user-facing app: search, request an
  album/track, watch the download queue, browse results, manage settings/creds.
  Thin API over Postgres. The user's preferred stack. **Never touches source tools
  directly.**
- **Python worker container (acquisition)** — owns all source integration. Polls
  the job queue, runs the six-stage pipeline, reports progress back through
  Postgres. Contains streamrip, yt-dlp, the slskd client, and beets/mutagen.
- **slskd container** — the off-the-shelf headless Soulseek daemon. Lyra talks to
  its REST API; we do not reimplement Soulseek.
- **Postgres** — single source of truth shared by the Next.js app and the worker.
- **Shared `/music` volume** — the destination library.

**Key principle:** the Next.js side reasons about *requests and jobs*; the worker
reasons about *candidates and files*. Neither reaches into the other's domain. All
source complexity is isolated in the worker behind the adapter interface.

### Worker communication

Postgres is the single source of truth. The worker **polls a `jobs` table** and
claims work — no Redis/Celery for a single-user home server. If push/scale is ever
needed, Redis+RQ can be introduced behind the same table without changing the data
model. This keeps v1 to three containers (plus Postgres).

## The acquisition pipeline

A request flows through six stages, each with one responsibility and a clear
hand-off, so each is independently testable.

```
① INTAKE ──▶ ② MATCH ──▶ ③ RANK ──▶ ④ DOWNLOAD ──▶ ⑤ TAG ──▶ ⑥ IMPORT
```

**① Intake** — resolve the request to a **canonical MusicBrainz target**:
release-group + release MBID, artist, ordered tracklist, expected track count, and
per-track durations. Every later stage measures against this anchor.

**② Match** — query each source adapter **in parallel**. Each returns zero or more
**candidates**, self-describing `{source, format, quality, trackCount, confidence}`.
Adapters do not download here; they only report what they could deliver.

**③ Rank** — score candidates by the quality-first policy, gated by a
match-confidence threshold. A high-quality but wrong/incomplete release loses to a
correct lower-quality one. The best qualifying candidate wins, automatically (no
user confirmation).

Quality ranking (best → worst):
`Qobuz hi-res → Qobuz lossless → Soulseek FLAC → Soulseek MP3 → YouTube`

**④ Download** — the winning adapter fetches and streams progress back. On failure,
**fall through** to the next-best candidate, then the next source. YouTube is the
floor.

**⑤ Tag** — verify integrity (complete, not truncated/silent), tag against
MusicBrainz via beets/mutagen, embed cover art. Originals kept; no transcoding.

**⑥ Import** — dedupe against the existing library (do not re-grab), move into the
fresh layout `Artist/Album (Year)/## Title.flac`, record the winning source+quality,
mark the request done.

### Job state machine

```
requested → matching → matched → downloading → tagging → imported
                 ↓         ↓          ↓            ↓
              (no match / all sources failed / bad file) → needs-attention
```

Nothing silently disappears — an unsatisfiable request lands in `needs-attention`,
not lost.

## Source adapters

Every source implements the **same contract**; the pipeline is source-agnostic.
Adding Tidal/Deezer/Bandcamp later is a new adapter and nothing else.

```
SourceAdapter:
  name          → "qobuz" | "soulseek" | "youtube"
  tier          → base quality rank (Qobuz=0, Soulseek=1, YouTube=2)
  health()      → configured & reachable? (creds valid, slskd up)
  search(target: MBTarget)              → Candidate[]   # metadata only
  download(candidate, dest, onProgress) → DownloadResult
```

A **Candidate** is self-describing so ranking needs no source-specific logic:

```
Candidate {
  source, sourceRef     # opaque handle back to the adapter
  format                # FLAC / MP3 / OPUS / AAC
  quality               # bitDepth+sampleRate (24/96) or bitrate (320k)
  trackCount            # for completeness vs. MB tracklist
  confidence            # 0–1: how sure this is the right release
}
```

**Per-adapter specifics (isolated in the worker):**

| Adapter    | Built on        | Auth                          | Notes |
|------------|-----------------|-------------------------------|-------|
| Qobuz      | `streamrip`     | app id/secret + account token | True lossless/hi-res; cleanest matches. Primary source. |
| Soulseek   | `slskd` REST API| slskd handles SoulSeek login  | Peer files grouped into album candidates; quality inferred from extension/bitrate; dead/slow peers → next candidate. |
| YouTube    | `yt-dlp`        | none (cookies optional)       | Universal lossy floor; match artist+album → YT Music album/playlist. |

**Confidence scoring** is shared and source-agnostic: compares each candidate to the
MB target on title/artist fuzzy match, track count, and total duration. Below
threshold → discarded (prevents grabbing a live/bootleg when the studio album was
requested).

## Data model

Postgres, shared by both containers.

```
artists        mbid, name, ...                          # cached MB metadata
releases       mbid, artist_id, title, year,            # the album (release-group/release)
               track_count, tracklist(json)
requests       id, release_id | track, requested_at,    # user intent
               status
jobs           id, request_id, state, attempts,         # pipeline execution;
               current_stage, error, updated_at,        #   state machine + claim
               claimed_at
candidates     id, job_id, source, format, quality,     # what Match found
               track_count, confidence, source_ref,
               chosen(bool)
library_items  release_id, path, source, quality,       # what is on disk → dedupe
               imported_at
config         key, value  (+ encrypted credentials)    # Qobuz token, slskd url, paths
```

**Flow across the split:**

1. Next.js writes a `request` + an initial `job (state=requested)`.
2. Worker polls `jobs`, claims one (sets `claimed_at`), advances it stage by stage,
   writing `candidates` and updating `job.state`/`current_stage`.
3. Next.js reads `job.state`/`current_stage` to render live progress (poll or SSE).
4. On success, worker writes a `library_items` row → future requests dedupe against
   it.

**Credentials** (Qobuz token, slskd URL/key) are stored encrypted in `config`,
written via the settings UI, read only by the worker.

## Error handling & edge cases

The design assumes sources will fail routinely; graceful fall-through is the norm.

**Fall-through ladder:** within a source, try the next-best candidate; across
sources, drop to the next tier; YouTube is the floor; if even that fails →
`needs-attention`, never silently dropped.

| Situation | Handling |
|-----------|----------|
| No match anywhere | Job → `needs-attention` ("no candidates above threshold"). User may lower threshold or pick manually. |
| Soulseek dead/slow peer | Transfer timeout → abandon candidate, try next peer/candidate. Bounded, configurable retries. |
| Soulseek partial album | Completeness check fails at Match → candidate deprioritized. v1 does not stitch multi-peer albums. |
| Wrong/mislabeled release | Confidence gate on duration + track count catches most; borderline → `needs-attention`. |
| Corrupt/truncated download | Integrity check at Tag (track count + non-zero/non-silent) → discard, fall through. |
| Already in library | Dedupe at Import against `library_items`, also checked early. Re-grab only if new copy is higher quality (optional upgrade flag). |
| Qobuz token expired | `health()` fails fast → job pauses, UI surfaces "re-auth Qobuz". |
| Worker crash mid-job | Jobs claimed with `claimed_at`; stale in-progress jobs reclaimed and resumed from `current_stage` (stages idempotent). |
| Rate limiting (Qobuz/YT) | Per-adapter backoff + capped retry; exceeding cap → `needs-attention`. |

**Idempotency principle:** every stage can be safely re-run. Download writes to a
temp dir; only Import commits to `/music`. A crash never leaves a half-file in the
library.

## Testing strategy

Because source complexity is quarantined behind the adapter interface, most of the
system is testable without hitting any real API.

- **Adapter contract tests** — one shared suite every adapter must pass (well-formed
  `Candidate`s; `download()` reports progress and yields a valid result; `health()`
  behaves). Run against **recorded fixtures** (cassettes of real Qobuz/slskd/yt-dlp
  responses): fast, offline, deterministic.
- **Ranker/confidence tests** — pure functions, no I/O. Synthetic candidate sets +
  MB targets; assert the right winner and that the confidence gate rejects wrong
  releases. Highest-value unit surface.
- **Pipeline/state-machine tests** — drive a job through all six stages with **fake
  adapters** (always-succeeds, always-fails, partial) to prove fall-through,
  `needs-attention`, and crash-resume.
- **Integration smoke test** — small opt-in suite hitting real sources for one known
  album, gated behind an env flag; not run in normal CI, not needed for development.
- **Next.js side** — API route tests against a seeded Postgres; a couple of
  end-to-end UI checks (submit request → watch it move through states).

**Goal:** build and test the entire acquisition engine on a laptop with zero real
credentials, using fixtures and fakes; real sources enter only for the opt-in smoke
test.