From 3adefe929b6adb86af9359aa55de1aa8ddc4ddd5 Mon Sep 17 00:00:00 2001 From: Jonathan Date: Fri, 10 Jul 2026 16:14:55 +0200 Subject: [PATCH] Add Lyra acquisition-engine design spec, README, gitignore Initial design for slice 1 (acquisition engine): three-container architecture (Next.js UI, Python worker, slskd), the six-stage pipeline, source-adapter contract, data model, error handling, and testing strategy. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 39 +++ README.md | 30 ++ ...26-07-10-lyra-acquisition-engine-design.md | 258 ++++++++++++++++++ 3 files changed, 327 insertions(+) create mode 100644 .gitignore create mode 100644 README.md create mode 100644 docs/superpowers/specs/2026-07-10-lyra-acquisition-engine-design.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..6406360 --- /dev/null +++ b/.gitignore @@ -0,0 +1,39 @@ +# Dependencies +node_modules/ +.pnp +.pnp.js +__pycache__/ +*.py[cod] +.venv/ +venv/ +env/ + +# Next.js +.next/ +out/ +build/ +dist/ + +# Env & secrets +.env +.env*.local +*.local + +# Docker / runtime data +music/ +data/ +postgres-data/ +slskd-data/ + +# Logs +*.log +npm-debug.log* +yarn-debug.log* +yarn-error.log* + +# OS / editor +.DS_Store +Thumbs.db +.idea/ +.vscode/ +*.swp diff --git a/README.md b/README.md new file mode 100644 index 0000000..e242e6f --- /dev/null +++ b/README.md @@ -0,0 +1,30 @@ +# Lyra + +Self-hosted, multi-source music acquisition and library tool — a replacement for +Lidarr + NZBget. Lyra discovers, ranks, downloads, tags, and organizes music across +sources of differing quality: **Qobuz** (lossless/hi-res), **Soulseek** (P2P), +**YouTube** (universal lossy fallback), and **Spotify** (discovery/metadata only). + +## Status + +Early design. Built in three slices: + +1. **Acquisition engine** — *in progress*. Given an album/track, find the best + available source, download it, tag it against MusicBrainz, and file it into a + clean library. +2. **Library manager** — Lidarr-style monitoring, wanted lists, auto-grab of new + releases. +3. **Discovery** — Spotify/recommendation-driven finding of new music. + +## Architecture + +Three Docker containers on a home server: + +- **Next.js app** — UI + API (search, request, queue, progress, settings). +- **Python worker** — all source integration (streamrip, yt-dlp, slskd client, + beets/mutagen), running the six-stage acquisition pipeline. +- **slskd** — off-the-shelf headless Soulseek daemon. + +**Postgres** is the shared source of truth; a shared `/music` volume is the library. + +See [`docs/superpowers/specs/`](docs/superpowers/specs/) for the full design. diff --git a/docs/superpowers/specs/2026-07-10-lyra-acquisition-engine-design.md b/docs/superpowers/specs/2026-07-10-lyra-acquisition-engine-design.md new file mode 100644 index 0000000..728138f --- /dev/null +++ b/docs/superpowers/specs/2026-07-10-lyra-acquisition-engine-design.md @@ -0,0 +1,258 @@ +# Lyra — Acquisition Engine (Design) + +**Date:** 2026-07-10 +**Status:** Approved design, ready for implementation planning +**Slice:** 1 of 3 (Acquisition engine) + +## Overview + +Lyra is a self-hosted, multi-source music acquisition and library tool intended to +replace Lidarr. It discovers, ranks, downloads, tags, and organizes music from +several sources of differing quality and legality: **Qobuz** (lossless/hi-res), +**Soulseek** (P2P), **YouTube** (universal lossy fallback), and **Spotify** (used +later for discovery/metadata only, not downloading). + +The full product is built in three slices, in order: + +1. **Acquisition engine** — *this spec*. Given an album/track, find the best + available source, download it, tag it against MusicBrainz, and file it into a + clean library. +2. **Library manager** — Lidarr-style monitoring: wanted lists, watched artists, + auto-grab of new releases. Builds on the acquisition engine. +3. **Discovery** — Spotify/recommendation-driven finding of new music, feeding + into acquisition. + +This document designs slice 1 only. Slices 2 and 3 get their own spec → plan → +implementation cycles. + +### Goals + +- Given a requested album or track, reliably obtain the best-quality complete copy + available across sources, fully automatically. +- Quarantine all source-specific complexity behind a uniform adapter interface, so + the rest of the system is source-agnostic and adding a source is additive. +- Never silently fail: any request that cannot be satisfied lands in an explicit + `needs-attention` state. +- Be fully developable and testable with **zero real credentials** via fixtures and + fake adapters. + +### Non-goals (this slice) + +- Artist monitoring / auto-grab of new releases (slice 2). +- Discovery / recommendations (slice 3). +- Spotify as a download source (used for metadata/discovery only, later). +- Transcoding — originals are kept as-is. +- Stitching a single album from multiple Soulseek peers (flagged for later). + +## Architecture + +Three Docker containers on the user's home server, alongside the existing +Lidarr/NZBget stack. + +``` +┌─────────────────┐ jobs ┌──────────────────┐ +│ Next.js app │──────────────▶│ Python worker │ +│ (UI + API) │ Postgres │ (acquisition) │ +│ │◀──────────────│ │ +│ search, request │ status │ streamrip (Qobuz)│ +│ queue, progress │ │ yt-dlp (YouTube) │ +└─────────────────┘ │ slskd client │ + │ │ beets/mutagen tag│ + ▼ └────────┬─────────┘ + Postgres │ + (requests, jobs, ▼ + candidates, config, /music (library) + library_items) + slskd container +``` + +**Components:** + +- **Next.js container (UI + API)** — the user-facing app: search, request an + album/track, watch the download queue, browse results, manage settings/creds. + Thin API over Postgres. The user's preferred stack. **Never touches source tools + directly.** +- **Python worker container (acquisition)** — owns all source integration. Polls + the job queue, runs the six-stage pipeline, reports progress back through + Postgres. Contains streamrip, yt-dlp, the slskd client, and beets/mutagen. +- **slskd container** — the off-the-shelf headless Soulseek daemon. Lyra talks to + its REST API; we do not reimplement Soulseek. +- **Postgres** — single source of truth shared by the Next.js app and the worker. +- **Shared `/music` volume** — the destination library. + +**Key principle:** the Next.js side reasons about *requests and jobs*; the worker +reasons about *candidates and files*. Neither reaches into the other's domain. All +source complexity is isolated in the worker behind the adapter interface. + +### Worker communication + +Postgres is the single source of truth. The worker **polls a `jobs` table** and +claims work — no Redis/Celery for a single-user home server. If push/scale is ever +needed, Redis+RQ can be introduced behind the same table without changing the data +model. This keeps v1 to three containers (plus Postgres). + +## The acquisition pipeline + +A request flows through six stages, each with one responsibility and a clear +hand-off, so each is independently testable. + +``` +① INTAKE ──▶ ② MATCH ──▶ ③ RANK ──▶ ④ DOWNLOAD ──▶ ⑤ TAG ──▶ ⑥ IMPORT +``` + +**① Intake** — resolve the request to a **canonical MusicBrainz target**: +release-group + release MBID, artist, ordered tracklist, expected track count, and +per-track durations. Every later stage measures against this anchor. + +**② Match** — query each source adapter **in parallel**. Each returns zero or more +**candidates**, self-describing `{source, format, quality, trackCount, confidence}`. +Adapters do not download here; they only report what they could deliver. + +**③ Rank** — score candidates by the quality-first policy, gated by a +match-confidence threshold. A high-quality but wrong/incomplete release loses to a +correct lower-quality one. The best qualifying candidate wins, automatically (no +user confirmation). + +Quality ranking (best → worst): +`Qobuz hi-res → Qobuz lossless → Soulseek FLAC → Soulseek MP3 → YouTube` + +**④ Download** — the winning adapter fetches and streams progress back. On failure, +**fall through** to the next-best candidate, then the next source. YouTube is the +floor. + +**⑤ Tag** — verify integrity (complete, not truncated/silent), tag against +MusicBrainz via beets/mutagen, embed cover art. Originals kept; no transcoding. + +**⑥ Import** — dedupe against the existing library (do not re-grab), move into the +fresh layout `Artist/Album (Year)/## Title.flac`, record the winning source+quality, +mark the request done. + +### Job state machine + +``` +requested → matching → matched → downloading → tagging → imported + ↓ ↓ ↓ ↓ + (no match / all sources failed / bad file) → needs-attention +``` + +Nothing silently disappears — an unsatisfiable request lands in `needs-attention`, +not lost. + +## Source adapters + +Every source implements the **same contract**; the pipeline is source-agnostic. +Adding Tidal/Deezer/Bandcamp later is a new adapter and nothing else. + +``` +SourceAdapter: + name → "qobuz" | "soulseek" | "youtube" + tier → base quality rank (Qobuz=0, Soulseek=1, YouTube=2) + health() → configured & reachable? (creds valid, slskd up) + search(target: MBTarget) → Candidate[] # metadata only + download(candidate, dest, onProgress) → DownloadResult +``` + +A **Candidate** is self-describing so ranking needs no source-specific logic: + +``` +Candidate { + source, sourceRef # opaque handle back to the adapter + format # FLAC / MP3 / OPUS / AAC + quality # bitDepth+sampleRate (24/96) or bitrate (320k) + trackCount # for completeness vs. MB tracklist + confidence # 0–1: how sure this is the right release +} +``` + +**Per-adapter specifics (isolated in the worker):** + +| Adapter | Built on | Auth | Notes | +|------------|-----------------|-------------------------------|-------| +| Qobuz | `streamrip` | app id/secret + account token | True lossless/hi-res; cleanest matches. Primary source. | +| Soulseek | `slskd` REST API| slskd handles SoulSeek login | Peer files grouped into album candidates; quality inferred from extension/bitrate; dead/slow peers → next candidate. | +| YouTube | `yt-dlp` | none (cookies optional) | Universal lossy floor; match artist+album → YT Music album/playlist. | + +**Confidence scoring** is shared and source-agnostic: compares each candidate to the +MB target on title/artist fuzzy match, track count, and total duration. Below +threshold → discarded (prevents grabbing a live/bootleg when the studio album was +requested). + +## Data model + +Postgres, shared by both containers. + +``` +artists mbid, name, ... # cached MB metadata +releases mbid, artist_id, title, year, # the album (release-group/release) + track_count, tracklist(json) +requests id, release_id | track, requested_at, # user intent + status +jobs id, request_id, state, attempts, # pipeline execution; + current_stage, error, updated_at, # state machine + claim + claimed_at +candidates id, job_id, source, format, quality, # what Match found + track_count, confidence, source_ref, + chosen(bool) +library_items release_id, path, source, quality, # what is on disk → dedupe + imported_at +config key, value (+ encrypted credentials) # Qobuz token, slskd url, paths +``` + +**Flow across the split:** + +1. Next.js writes a `request` + an initial `job (state=requested)`. +2. Worker polls `jobs`, claims one (sets `claimed_at`), advances it stage by stage, + writing `candidates` and updating `job.state`/`current_stage`. +3. Next.js reads `job.state`/`current_stage` to render live progress (poll or SSE). +4. On success, worker writes a `library_items` row → future requests dedupe against + it. + +**Credentials** (Qobuz token, slskd URL/key) are stored encrypted in `config`, +written via the settings UI, read only by the worker. + +## Error handling & edge cases + +The design assumes sources will fail routinely; graceful fall-through is the norm. + +**Fall-through ladder:** within a source, try the next-best candidate; across +sources, drop to the next tier; YouTube is the floor; if even that fails → +`needs-attention`, never silently dropped. + +| Situation | Handling | +|-----------|----------| +| No match anywhere | Job → `needs-attention` ("no candidates above threshold"). User may lower threshold or pick manually. | +| Soulseek dead/slow peer | Transfer timeout → abandon candidate, try next peer/candidate. Bounded, configurable retries. | +| Soulseek partial album | Completeness check fails at Match → candidate deprioritized. v1 does not stitch multi-peer albums. | +| Wrong/mislabeled release | Confidence gate on duration + track count catches most; borderline → `needs-attention`. | +| Corrupt/truncated download | Integrity check at Tag (track count + non-zero/non-silent) → discard, fall through. | +| Already in library | Dedupe at Import against `library_items`, also checked early. Re-grab only if new copy is higher quality (optional upgrade flag). | +| Qobuz token expired | `health()` fails fast → job pauses, UI surfaces "re-auth Qobuz". | +| Worker crash mid-job | Jobs claimed with `claimed_at`; stale in-progress jobs reclaimed and resumed from `current_stage` (stages idempotent). | +| Rate limiting (Qobuz/YT) | Per-adapter backoff + capped retry; exceeding cap → `needs-attention`. | + +**Idempotency principle:** every stage can be safely re-run. Download writes to a +temp dir; only Import commits to `/music`. A crash never leaves a half-file in the +library. + +## Testing strategy + +Because source complexity is quarantined behind the adapter interface, most of the +system is testable without hitting any real API. + +- **Adapter contract tests** — one shared suite every adapter must pass (well-formed + `Candidate`s; `download()` reports progress and yields a valid result; `health()` + behaves). Run against **recorded fixtures** (cassettes of real Qobuz/slskd/yt-dlp + responses): fast, offline, deterministic. +- **Ranker/confidence tests** — pure functions, no I/O. Synthetic candidate sets + + MB targets; assert the right winner and that the confidence gate rejects wrong + releases. Highest-value unit surface. +- **Pipeline/state-machine tests** — drive a job through all six stages with **fake + adapters** (always-succeeds, always-fails, partial) to prove fall-through, + `needs-attention`, and crash-resume. +- **Integration smoke test** — small opt-in suite hitting real sources for one known + album, gated behind an env flag; not run in normal CI, not needed for development. +- **Next.js side** — API route tests against a seeded Postgres; a couple of + end-to-end UI checks (submit request → watch it move through states). + +**Goal:** build and test the entire acquisition engine on a laptop with zero real +credentials, using fixtures and fakes; real sources enter only for the opt-in smoke +test.