Add Lyra acquisition-engine design spec, README, gitignore
Initial design for slice 1 (acquisition engine): three-container architecture (Next.js UI, Python worker, slskd), the six-stage pipeline, source-adapter contract, data model, error handling, and testing strategy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jonathan
@@ -0,0 +1,258 @@
|
||||
# Lyra — Acquisition Engine (Design)
|
||||
|
||||
**Date:** 2026-07-10
|
||||
**Status:** Approved design, ready for implementation planning
|
||||
**Slice:** 1 of 3 (Acquisition engine)
|
||||
|
||||
## Overview
|
||||
|
||||
Lyra is a self-hosted, multi-source music acquisition and library tool intended to
|
||||
replace Lidarr. It discovers, ranks, downloads, tags, and organizes music from
|
||||
several sources of differing quality and legality: **Qobuz** (lossless/hi-res),
|
||||
**Soulseek** (P2P), **YouTube** (universal lossy fallback), and **Spotify** (used
|
||||
later for discovery/metadata only, not downloading).
|
||||
|
||||
The full product is built in three slices, in order:
|
||||
|
||||
1. **Acquisition engine** — *this spec*. Given an album/track, find the best
|
||||
available source, download it, tag it against MusicBrainz, and file it into a
|
||||
clean library.
|
||||
2. **Library manager** — Lidarr-style monitoring: wanted lists, watched artists,
|
||||
auto-grab of new releases. Builds on the acquisition engine.
|
||||
3. **Discovery** — Spotify/recommendation-driven finding of new music, feeding
|
||||
into acquisition.
|
||||
|
||||
This document designs slice 1 only. Slices 2 and 3 get their own spec → plan →
|
||||
implementation cycles.
|
||||
|
||||
### Goals
|
||||
|
||||
- Given a requested album or track, reliably obtain the best-quality complete copy
|
||||
available across sources, fully automatically.
|
||||
- Quarantine all source-specific complexity behind a uniform adapter interface, so
|
||||
the rest of the system is source-agnostic and adding a source is additive.
|
||||
- Never silently fail: any request that cannot be satisfied lands in an explicit
|
||||
`needs-attention` state.
|
||||
- Be fully developable and testable with **zero real credentials** via fixtures and
|
||||
fake adapters.
|
||||
|
||||
### Non-goals (this slice)
|
||||
|
||||
- Artist monitoring / auto-grab of new releases (slice 2).
|
||||
- Discovery / recommendations (slice 3).
|
||||
- Spotify as a download source (used for metadata/discovery only, later).
|
||||
- Transcoding — originals are kept as-is.
|
||||
- Stitching a single album from multiple Soulseek peers (flagged for later).
|
||||
|
||||
## Architecture
|
||||
|
||||
Three Docker containers on the user's home server, alongside the existing
|
||||
Lidarr/NZBget stack.
|
||||
|
||||
```
|
||||
┌─────────────────┐ jobs ┌──────────────────┐
|
||||
│ Next.js app │──────────────▶│ Python worker │
|
||||
│ (UI + API) │ Postgres │ (acquisition) │
|
||||
│ │◀──────────────│ │
|
||||
│ search, request │ status │ streamrip (Qobuz)│
|
||||
│ queue, progress │ │ yt-dlp (YouTube) │
|
||||
└─────────────────┘ │ slskd client │
|
||||
│ │ beets/mutagen tag│
|
||||
▼ └────────┬─────────┘
|
||||
Postgres │
|
||||
(requests, jobs, ▼
|
||||
candidates, config, /music (library)
|
||||
library_items) + slskd container
|
||||
```
|
||||
|
||||
**Components:**
|
||||
|
||||
- **Next.js container (UI + API)** — the user-facing app: search, request an
|
||||
album/track, watch the download queue, browse results, manage settings/creds.
|
||||
Thin API over Postgres. The user's preferred stack. **Never touches source tools
|
||||
directly.**
|
||||
- **Python worker container (acquisition)** — owns all source integration. Polls
|
||||
the job queue, runs the six-stage pipeline, reports progress back through
|
||||
Postgres. Contains streamrip, yt-dlp, the slskd client, and beets/mutagen.
|
||||
- **slskd container** — the off-the-shelf headless Soulseek daemon. Lyra talks to
|
||||
its REST API; we do not reimplement Soulseek.
|
||||
- **Postgres** — single source of truth shared by the Next.js app and the worker.
|
||||
- **Shared `/music` volume** — the destination library.
|
||||
|
||||
**Key principle:** the Next.js side reasons about *requests and jobs*; the worker
|
||||
reasons about *candidates and files*. Neither reaches into the other's domain. All
|
||||
source complexity is isolated in the worker behind the adapter interface.
|
||||
|
||||
### Worker communication
|
||||
|
||||
Postgres is the single source of truth. The worker **polls a `jobs` table** and
|
||||
claims work — no Redis/Celery for a single-user home server. If push/scale is ever
|
||||
needed, Redis+RQ can be introduced behind the same table without changing the data
|
||||
model. This keeps v1 to three containers (plus Postgres).
|
||||
|
||||
## The acquisition pipeline
|
||||
|
||||
A request flows through six stages, each with one responsibility and a clear
|
||||
hand-off, so each is independently testable.
|
||||
|
||||
```
|
||||
① INTAKE ──▶ ② MATCH ──▶ ③ RANK ──▶ ④ DOWNLOAD ──▶ ⑤ TAG ──▶ ⑥ IMPORT
|
||||
```
|
||||
|
||||
**① Intake** — resolve the request to a **canonical MusicBrainz target**:
|
||||
release-group + release MBID, artist, ordered tracklist, expected track count, and
|
||||
per-track durations. Every later stage measures against this anchor.
|
||||
|
||||
**② Match** — query each source adapter **in parallel**. Each returns zero or more
|
||||
**candidates**, self-describing `{source, format, quality, trackCount, confidence}`.
|
||||
Adapters do not download here; they only report what they could deliver.
|
||||
|
||||
**③ Rank** — score candidates by the quality-first policy, gated by a
|
||||
match-confidence threshold. A high-quality but wrong/incomplete release loses to a
|
||||
correct lower-quality one. The best qualifying candidate wins, automatically (no
|
||||
user confirmation).
|
||||
|
||||
Quality ranking (best → worst):
|
||||
`Qobuz hi-res → Qobuz lossless → Soulseek FLAC → Soulseek MP3 → YouTube`
|
||||
|
||||
**④ Download** — the winning adapter fetches and streams progress back. On failure,
|
||||
**fall through** to the next-best candidate, then the next source. YouTube is the
|
||||
floor.
|
||||
|
||||
**⑤ Tag** — verify integrity (complete, not truncated/silent), tag against
|
||||
MusicBrainz via beets/mutagen, embed cover art. Originals kept; no transcoding.
|
||||
|
||||
**⑥ Import** — dedupe against the existing library (do not re-grab), move into the
|
||||
fresh layout `Artist/Album (Year)/## Title.flac`, record the winning source+quality,
|
||||
mark the request done.
|
||||
|
||||
### Job state machine
|
||||
|
||||
```
|
||||
requested → matching → matched → downloading → tagging → imported
|
||||
↓ ↓ ↓ ↓
|
||||
(no match / all sources failed / bad file) → needs-attention
|
||||
```
|
||||
|
||||
Nothing silently disappears — an unsatisfiable request lands in `needs-attention`,
|
||||
not lost.
|
||||
|
||||
## Source adapters
|
||||
|
||||
Every source implements the **same contract**; the pipeline is source-agnostic.
|
||||
Adding Tidal/Deezer/Bandcamp later is a new adapter and nothing else.
|
||||
|
||||
```
|
||||
SourceAdapter:
|
||||
name → "qobuz" | "soulseek" | "youtube"
|
||||
tier → base quality rank (Qobuz=0, Soulseek=1, YouTube=2)
|
||||
health() → configured & reachable? (creds valid, slskd up)
|
||||
search(target: MBTarget) → Candidate[] # metadata only
|
||||
download(candidate, dest, onProgress) → DownloadResult
|
||||
```
|
||||
|
||||
A **Candidate** is self-describing so ranking needs no source-specific logic:
|
||||
|
||||
```
|
||||
Candidate {
|
||||
source, sourceRef # opaque handle back to the adapter
|
||||
format # FLAC / MP3 / OPUS / AAC
|
||||
quality # bitDepth+sampleRate (24/96) or bitrate (320k)
|
||||
trackCount # for completeness vs. MB tracklist
|
||||
confidence # 0–1: how sure this is the right release
|
||||
}
|
||||
```
|
||||
|
||||
**Per-adapter specifics (isolated in the worker):**
|
||||
|
||||
| Adapter | Built on | Auth | Notes |
|
||||
|------------|-----------------|-------------------------------|-------|
|
||||
| Qobuz | `streamrip` | app id/secret + account token | True lossless/hi-res; cleanest matches. Primary source. |
|
||||
| Soulseek | `slskd` REST API| slskd handles SoulSeek login | Peer files grouped into album candidates; quality inferred from extension/bitrate; dead/slow peers → next candidate. |
|
||||
| YouTube | `yt-dlp` | none (cookies optional) | Universal lossy floor; match artist+album → YT Music album/playlist. |
|
||||
|
||||
**Confidence scoring** is shared and source-agnostic: compares each candidate to the
|
||||
MB target on title/artist fuzzy match, track count, and total duration. Below
|
||||
threshold → discarded (prevents grabbing a live/bootleg when the studio album was
|
||||
requested).
|
||||
|
||||
## Data model
|
||||
|
||||
Postgres, shared by both containers.
|
||||
|
||||
```
|
||||
artists mbid, name, ... # cached MB metadata
|
||||
releases mbid, artist_id, title, year, # the album (release-group/release)
|
||||
track_count, tracklist(json)
|
||||
requests id, release_id | track, requested_at, # user intent
|
||||
status
|
||||
jobs id, request_id, state, attempts, # pipeline execution;
|
||||
current_stage, error, updated_at, # state machine + claim
|
||||
claimed_at
|
||||
candidates id, job_id, source, format, quality, # what Match found
|
||||
track_count, confidence, source_ref,
|
||||
chosen(bool)
|
||||
library_items release_id, path, source, quality, # what is on disk → dedupe
|
||||
imported_at
|
||||
config key, value (+ encrypted credentials) # Qobuz token, slskd url, paths
|
||||
```
|
||||
|
||||
**Flow across the split:**
|
||||
|
||||
1. Next.js writes a `request` + an initial `job (state=requested)`.
|
||||
2. Worker polls `jobs`, claims one (sets `claimed_at`), advances it stage by stage,
|
||||
writing `candidates` and updating `job.state`/`current_stage`.
|
||||
3. Next.js reads `job.state`/`current_stage` to render live progress (poll or SSE).
|
||||
4. On success, worker writes a `library_items` row → future requests dedupe against
|
||||
it.
|
||||
|
||||
**Credentials** (Qobuz token, slskd URL/key) are stored encrypted in `config`,
|
||||
written via the settings UI, read only by the worker.
|
||||
|
||||
## Error handling & edge cases
|
||||
|
||||
The design assumes sources will fail routinely; graceful fall-through is the norm.
|
||||
|
||||
**Fall-through ladder:** within a source, try the next-best candidate; across
|
||||
sources, drop to the next tier; YouTube is the floor; if even that fails →
|
||||
`needs-attention`, never silently dropped.
|
||||
|
||||
| Situation | Handling |
|
||||
|-----------|----------|
|
||||
| No match anywhere | Job → `needs-attention` ("no candidates above threshold"). User may lower threshold or pick manually. |
|
||||
| Soulseek dead/slow peer | Transfer timeout → abandon candidate, try next peer/candidate. Bounded, configurable retries. |
|
||||
| Soulseek partial album | Completeness check fails at Match → candidate deprioritized. v1 does not stitch multi-peer albums. |
|
||||
| Wrong/mislabeled release | Confidence gate on duration + track count catches most; borderline → `needs-attention`. |
|
||||
| Corrupt/truncated download | Integrity check at Tag (track count + non-zero/non-silent) → discard, fall through. |
|
||||
| Already in library | Dedupe at Import against `library_items`, also checked early. Re-grab only if new copy is higher quality (optional upgrade flag). |
|
||||
| Qobuz token expired | `health()` fails fast → job pauses, UI surfaces "re-auth Qobuz". |
|
||||
| Worker crash mid-job | Jobs claimed with `claimed_at`; stale in-progress jobs reclaimed and resumed from `current_stage` (stages idempotent). |
|
||||
| Rate limiting (Qobuz/YT) | Per-adapter backoff + capped retry; exceeding cap → `needs-attention`. |
|
||||
|
||||
**Idempotency principle:** every stage can be safely re-run. Download writes to a
|
||||
temp dir; only Import commits to `/music`. A crash never leaves a half-file in the
|
||||
library.
|
||||
|
||||
## Testing strategy
|
||||
|
||||
Because source complexity is quarantined behind the adapter interface, most of the
|
||||
system is testable without hitting any real API.
|
||||
|
||||
- **Adapter contract tests** — one shared suite every adapter must pass (well-formed
|
||||
`Candidate`s; `download()` reports progress and yields a valid result; `health()`
|
||||
behaves). Run against **recorded fixtures** (cassettes of real Qobuz/slskd/yt-dlp
|
||||
responses): fast, offline, deterministic.
|
||||
- **Ranker/confidence tests** — pure functions, no I/O. Synthetic candidate sets +
|
||||
MB targets; assert the right winner and that the confidence gate rejects wrong
|
||||
releases. Highest-value unit surface.
|
||||
- **Pipeline/state-machine tests** — drive a job through all six stages with **fake
|
||||
adapters** (always-succeeds, always-fails, partial) to prove fall-through,
|
||||
`needs-attention`, and crash-resume.
|
||||
- **Integration smoke test** — small opt-in suite hitting real sources for one known
|
||||
album, gated behind an env flag; not run in normal CI, not needed for development.
|
||||
- **Next.js side** — API route tests against a seeded Postgres; a couple of
|
||||
end-to-end UI checks (submit request → watch it move through states).
|
||||
|
||||
**Goal:** build and test the entire acquisition engine on a laptop with zero real
|
||||
credentials, using fixtures and fakes; real sources enter only for the opt-in smoke
|
||||
test.
|
||||
Reference in New Issue
Block a user