docs: add library-manager design (slice 2)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jonathan
@@ -0,0 +1,325 @@
|
||||
# Lyra — Library Manager (Design)
|
||||
|
||||
**Date:** 2026-07-11
|
||||
**Status:** Approved design, ready for implementation planning
|
||||
**Slice:** 2 of 3 (Library manager)
|
||||
|
||||
## Overview
|
||||
|
||||
Slice 2 turns Lyra from a one-shot acquisition tool into a standing library
|
||||
manager: you follow artists, monitor the specific releases you want, and Lyra
|
||||
keeps working in the background to obtain and upgrade them. It is Lidarr-style
|
||||
monitoring built directly on the slice 1 acquisition engine — the monitor's only
|
||||
output is the same `Request` + `Job` a manual request produces, so the entire
|
||||
existing pipeline is reused unchanged.
|
||||
|
||||
Two loops are equally central and share one monitoring engine:
|
||||
|
||||
1. **Watched artists → auto-grab.** Follow an artist, browse their discography,
|
||||
and monitor the releases you want. Optionally flip a per-artist toggle so
|
||||
*future* releases auto-monitor as they appear.
|
||||
2. **Wanted list → keep retrying.** Every monitored release without an acceptable
|
||||
copy is "wanted" and is re-attempted on a schedule until a good copy lands —
|
||||
and, within a window, upgraded to better quality.
|
||||
|
||||
### Goals
|
||||
|
||||
- Follow artists and monitor releases at release-group granularity; obtain and
|
||||
keep them current automatically.
|
||||
- Reuse the slice 1 pipeline verbatim — monitoring enqueues normal jobs and reads
|
||||
their outcomes; the pipeline itself is untouched.
|
||||
- Never silently drop a want: a monitored release without a copy stays wanted and
|
||||
keeps being retried.
|
||||
- Keep sources and secrets in the worker. The web app performs only read-only,
|
||||
credential-free MusicBrainz *metadata* lookups for interactive search/browse.
|
||||
- Fully developable and testable with zero real credentials via a fake
|
||||
MusicBrainz browser and seeded Postgres.
|
||||
|
||||
### Non-goals (this slice)
|
||||
|
||||
- Notifications / webhooks on grab or needs-attention (a later slice).
|
||||
- Discovery / recommendations (slice 3).
|
||||
- A release-calendar / upcoming view.
|
||||
- Multi-user; this remains a single-user home-server tool.
|
||||
- Any change to how downloading, tagging, or the six-stage pipeline work.
|
||||
|
||||
## Scope
|
||||
|
||||
**In scope:** watched artists (live MusicBrainz search + follow), a per-artist
|
||||
discography detail view with monitor toggles, a per-artist "auto-monitor future
|
||||
releases" toggle, a background monitor sweep (discovery + retry/upgrade), a
|
||||
cross-artist wanted list, and the outcome-reconcile that feeds job results back
|
||||
into monitoring state.
|
||||
|
||||
**Out of scope:** notifications, discovery, calendar. Deferred hardening from
|
||||
slice 1 is unaffected and remains deferred.
|
||||
|
||||
## Key decisions
|
||||
|
||||
- **Default is passive.** Following an artist marks their *entire* discography
|
||||
**unmonitored**. You opt in per release. A per-artist `autoMonitorFuture`
|
||||
toggle (default **off**) makes genuinely-new releases auto-monitor from the
|
||||
moment it is enabled.
|
||||
- **No type filtering when auto-monitoring.** When `autoMonitorFuture` is on, all
|
||||
official release-group types (albums, EPs, singles, live, compilations, …)
|
||||
auto-monitor; you unmonitor the noise by hand. ("Everything, filter later.")
|
||||
- **Quality upgrades within a window.** A quality **cutoff** and an **upgrade
|
||||
window** (N days) govern re-searching: below cutoff and within the window → keep
|
||||
chasing better quality; at/above cutoff, or window expired → frozen (done).
|
||||
- **One table for discography *and* wanted list.** A "wanted" item is simply a
|
||||
monitored release-group without a copy at cutoff. No separate wanted table, no
|
||||
sync problem.
|
||||
- **Monitor lives in the worker.** A scheduled sweep runs inside the existing
|
||||
worker loop. The web app owns CRUD + display, plus read-only MusicBrainz search
|
||||
for the interactive picker.
|
||||
|
||||
## Architecture
|
||||
|
||||
No new containers. Two additions to the existing three-container stack:
|
||||
|
||||
- **Worker gains a monitor.** Beside the job-claim loop, a cheap ~60s tick runs
|
||||
`monitor.sweep()`; after each pipeline run the loop calls
|
||||
`monitor.reconcile()`. All scheduled MusicBrainz work and all enqueuing live
|
||||
here, next to the pipeline it feeds.
|
||||
- **Web gains monitoring UI + read-only MB reads.** New pages/routes for watched
|
||||
artists, discography detail, and the wanted list, plus a thin MusicBrainz read
|
||||
module used only for the live search/browse picker.
|
||||
|
||||
```
|
||||
┌───────────────────────────┐ Postgres ┌──────────────────────────────┐
|
||||
│ Next.js app (UI + API) │──────────────▶│ Python worker │
|
||||
│ │ requests/ │ │
|
||||
│ /artists /artists/[id] │ jobs + │ claim → run_pipeline (S1) │
|
||||
│ /wanted live MB picker │ monitor rows │ monitor.sweep() (~60s tick)│
|
||||
│ │◀──────────────│ monitor.reconcile(job) │
|
||||
│ read-only MB search ─────┼──▶ MusicBrainz│ scheduled MB discovery ─────┼─▶ MB
|
||||
└───────────────────────────┘ (metadata) └──────────────────────────────┘
|
||||
```
|
||||
|
||||
**Separation of concerns.** The web reasons about *watched artists, monitored
|
||||
releases, requests, and jobs*; the worker reasons about *candidates, files, and
|
||||
scheduled acquisition*. The web's only external reach is **read-only, cred-free
|
||||
MusicBrainz metadata** for the picker — every download **source** and every
|
||||
**credential** remains worker-only, as does the scheduled discovery poll and all
|
||||
pipeline-time MB resolution.
|
||||
|
||||
## Data model
|
||||
|
||||
Two new tables, one new nullable column, and config keys in the existing
|
||||
key/value `Config` table (no migration for config).
|
||||
|
||||
```
|
||||
WatchedArtist
|
||||
id
|
||||
mbid (unique) # MusicBrainz artist MBID
|
||||
name
|
||||
autoMonitorFuture Bool default false # auto-monitor new releases going forward
|
||||
monitorFrom DateTime default now() # boundary for "future" when the toggle is on
|
||||
lastPolledAt DateTime? # throttles the discovery poll per artist
|
||||
createdAt
|
||||
monitoredReleases MonitoredRelease[]
|
||||
|
||||
MonitoredRelease # discography row AND wanted-list row
|
||||
id
|
||||
watchedArtistId FK? # null = standalone manual wanted add
|
||||
artistMbid, artistName # denormalized: standalone rows need no join
|
||||
rgMbid (unique) # release-group MBID
|
||||
album
|
||||
primaryType String? # "Album" | "EP" | "Single" | ...
|
||||
secondaryTypes String[] # ["Live", "Compilation", ...]
|
||||
firstReleaseDate String? # "YYYY-MM-DD" as MB reports it
|
||||
monitored Bool default false # does the sweep act on it?
|
||||
state enum(wanted|grabbed|fulfilled) default wanted
|
||||
currentQualityClass Int? # quality of the copy we have (null = none)
|
||||
firstGrabbedAt DateTime? # drives the upgrade window
|
||||
lastSearchedAt DateTime? # throttles retry
|
||||
createdAt
|
||||
|
||||
Request (existing) gains:
|
||||
monitoredReleaseId String? # set when the monitor enqueues; null for manual
|
||||
```
|
||||
|
||||
**Config keys** (worker-read, via existing `Config`):
|
||||
|
||||
| key | default | meaning |
|
||||
|-----|---------|---------|
|
||||
| `monitor.enabled` | `false` | master switch; off → worker behaves exactly as slice 1 |
|
||||
| `monitor.pollIntervalHours` | `24` | how often discovery re-browses each artist |
|
||||
| `monitor.retryIntervalHours` | `6` | how often a wanted item is re-searched |
|
||||
| `monitor.qualityCutoff` | `2` | the `qualityClass` at/above which a release is "done" |
|
||||
| `monitor.upgradeWindowDays` | `14` | days after first grab to keep chasing upgrades |
|
||||
|
||||
`qualityClass` (from slice 1's `quality.py`): **3** = hi-res lossless, **2** = CD
|
||||
lossless, **1** = lossy. The default cutoff `2` means "any lossless satisfies;
|
||||
only a lossy copy stays wanted and keeps chasing an upgrade."
|
||||
|
||||
### How rows come to exist
|
||||
|
||||
- **Follow (live picker):** the web already fetched the artist + discography from
|
||||
MusicBrainz to render the picker. On Follow it writes the `WatchedArtist` and
|
||||
persists the discography as `MonitoredRelease` rows, all **unmonitored**. The
|
||||
detail view is therefore instant.
|
||||
- **Background discovery:** the worker's poll re-browses each watched artist and
|
||||
inserts any *newly appearing* release-groups. If `autoMonitorFuture` is on and
|
||||
the release is new relative to `monitorFrom`, it is inserted **monitored +
|
||||
wanted**; otherwise **unmonitored**.
|
||||
- **Standalone wanted add:** adding an arbitrary album to the wanted list (not via
|
||||
a followed artist) resolves it to a release-group and writes a single
|
||||
`MonitoredRelease` with `watchedArtistId = null`, `monitored = true`.
|
||||
|
||||
## The monitor (worker)
|
||||
|
||||
New module `worker/lyra_worker/monitor.py`. It uses an `MbBrowser` seam (below)
|
||||
and plain SQL over the shared connection; scheduling is driven entirely by
|
||||
timestamps in the tables, so the loop tick is just a cheap gate.
|
||||
|
||||
### Scheduling
|
||||
|
||||
`run_forever()` records the last tick time; each loop iteration, if
|
||||
`monitor.enabled` and ≥ ~60s elapsed, it calls `monitor.sweep(conn, browser,
|
||||
cfg)` before continuing to claim jobs. After every `run_pipeline` it calls
|
||||
`monitor.reconcile(conn, job_id)`. One process, one loop, no new infra.
|
||||
|
||||
### `sweep()` — two phases
|
||||
|
||||
**Phase 1 — Discovery** (per artist where `now - lastPolledAt >
|
||||
pollIntervalHours`):
|
||||
- `browser.browse_release_groups(artist.mbid)` → release-group list.
|
||||
- For each rg not already a `MonitoredRelease`, insert it. `monitored =
|
||||
artist.autoMonitorFuture and firstReleaseDate > artist.monitorFrom`; state
|
||||
`wanted`. (All types eligible — no filtering.)
|
||||
- Set `lastPolledAt = now`.
|
||||
|
||||
**Phase 2 — Retry / enqueue** (rows with `monitored = true`, `state != fulfilled`):
|
||||
- Skip rows with an **active** (non-terminal) job for the same release.
|
||||
- Skip / satisfy via dedupe: if a `LibraryItem` at/above cutoff already exists →
|
||||
set `fulfilled`.
|
||||
- For `state = grabbed`: if `now - firstGrabbedAt >= upgradeWindowDays` **or**
|
||||
`currentQualityClass >= cutoff` → set `fulfilled` and skip; else it is an
|
||||
upgrade candidate.
|
||||
- For due rows (`lastSearchedAt` null or older than `retryIntervalHours`): create
|
||||
`Request(artist, album, monitoredReleaseId=id)` + `Job(state=requested)`; set
|
||||
`lastSearchedAt = now`. The slice 1 pipeline takes it from there.
|
||||
|
||||
### `reconcile(job_id)` — feed outcomes back
|
||||
|
||||
After the pipeline finishes a job that has a `monitoredReleaseId`:
|
||||
- **imported:** set `currentQualityClass` from the winning candidate and
|
||||
`firstGrabbedAt` (if null); set `state = fulfilled` if `qualityClass >= cutoff`,
|
||||
else `grabbed`.
|
||||
- **needs_attention:** leave `state = wanted` (already has `lastSearchedAt`, so it
|
||||
waits out the retry interval before the next attempt).
|
||||
|
||||
Reconcile is a separate call in the main loop, so the pipeline code is untouched
|
||||
and the whole feature is inert when `monitor.enabled = false`.
|
||||
|
||||
### MonitoredRelease state machine
|
||||
|
||||
```
|
||||
discovery / follow
|
||||
├─(autoMonitorFuture & new)─► monitored + wanted ─┐
|
||||
└─(default / back-cat)──────► unmonitored + wanted │ user toggle → monitored
|
||||
▼
|
||||
wanted ──sweep enqueues──► [job] ──imported < cutoff──► grabbed ──window expired / at cutoff──► fulfilled
|
||||
▲ │ │
|
||||
└─────needs_attention──────┘ imported ≥ cutoff───┴──────────────────────────────► fulfilled
|
||||
```
|
||||
|
||||
Unmonitor at any time → dormant (the sweep ignores it). A want is never dropped;
|
||||
it is retried on its interval until satisfied.
|
||||
|
||||
## MusicBrainz access
|
||||
|
||||
The worker's existing `MusicBrainzResolver` handles pipeline-time
|
||||
album resolution. Monitoring adds a small browsing seam.
|
||||
|
||||
**Worker — `MbBrowser` protocol** (real impl via `musicbrainzngs`, lazily
|
||||
imported; faked in tests):
|
||||
- `search_artist(name) -> list[ArtistHit]` — `{mbid, name, disambiguation}`.
|
||||
- `browse_release_groups(artist_mbid) -> list[ReleaseGroupInfo]` — `{rgMbid,
|
||||
title, primaryType, secondaryTypes, firstReleaseDate}`.
|
||||
|
||||
**Web — read-only MB read module** for the live picker only: artist search and
|
||||
discography browse, returning the same shapes the web persists as
|
||||
`MonitoredRelease` rows on Follow. No credentials; read-only; metadata only.
|
||||
|
||||
## Web UI & API
|
||||
|
||||
Three surfaces, thin over the two tables:
|
||||
|
||||
- **`/artists`** — watched artists (name, `#monitored` / `#have`), **Add artist**
|
||||
(live search picker), remove. Per-artist `autoMonitorFuture` toggle.
|
||||
- **`/artists/[id]`** — the discography: every `MonitoredRelease` for the artist,
|
||||
each with a status badge (Have@quality / Wanted / Dormant), a **monitor toggle**,
|
||||
and a **Search now** button (enqueues immediately).
|
||||
- **`/wanted`** — cross-artist list of everything `monitored` and not `fulfilled`
|
||||
(includes standalone manual adds and needs-attention retries), with
|
||||
last-searched time and **Search now**. Plus **Add to wanted** (resolve an
|
||||
arbitrary artist+album → standalone monitored row).
|
||||
|
||||
**API routes (Next.js):**
|
||||
- `GET /api/mb/artists?q=` — live artist search (read-only MB).
|
||||
- `GET /api/mb/artists/[mbid]/releases` — discography browse (read-only MB).
|
||||
- `POST /api/artists` — follow: create `WatchedArtist` + persist discography as
|
||||
unmonitored `MonitoredRelease` rows.
|
||||
- `GET /api/artists`, `GET /api/artists/[id]`, `DELETE /api/artists/[id]`,
|
||||
`PATCH /api/artists/[id]` (toggle `autoMonitorFuture`).
|
||||
- `GET /api/wanted` — monitored, not-fulfilled rows across artists.
|
||||
- `POST /api/wanted` — standalone add (resolve + create row).
|
||||
- `PATCH /api/releases/[id]` — toggle `monitored`.
|
||||
- `POST /api/releases/[id]/search` — enqueue now (`Request` + `Job`).
|
||||
|
||||
"Search now" and standalone adds create the same `Request` + `Job` the monitor
|
||||
does (with `monitoredReleaseId` set), so reconcile handles their outcomes too.
|
||||
|
||||
## Integration touch-point: upgrades must replace
|
||||
|
||||
Quality upgrades re-import an album that is already in the library.
|
||||
`LibraryItem` is `@@unique([artist, album])`, so the import stage must **upsert**
|
||||
the row and overwrite the on-disk files rather than error on a duplicate. Current
|
||||
behavior will be verified during planning; if it errors today, the minimal
|
||||
import-upsert change is included in this slice. This is the only pipeline-adjacent
|
||||
change; the six stages otherwise stand unchanged.
|
||||
|
||||
## Error handling & edge cases
|
||||
|
||||
| Situation | Handling |
|
||||
|-----------|----------|
|
||||
| New release not yet on any source | Job → needs_attention; row stays wanted; retried each interval until a copy appears (natural pre-order behavior). |
|
||||
| Artist name ambiguous | The live picker shows MB `disambiguation`; the user chooses. No blind auto-match. |
|
||||
| Duplicate discovery | `MonitoredRelease.rgMbid` is unique; re-browsing never duplicates rows. |
|
||||
| Piled-up jobs | Sweep skips a release that already has a non-terminal job. |
|
||||
| Already in library | Dedupe against `LibraryItem`; at/above cutoff → `fulfilled` without enqueuing. |
|
||||
| Upgrade never improves | Upgrade window expires → `fulfilled` (frozen); no endless re-searching. |
|
||||
| MB unreachable during sweep | Discovery for that artist is skipped this tick; retried next poll. Web picker surfaces the error to the user. |
|
||||
| Monitor disabled | `monitor.enabled = false` → no sweep, no reconcile effects; worker behaves exactly as slice 1. |
|
||||
|
||||
## Testing strategy
|
||||
|
||||
Same philosophy as slice 1 — logic is testable with zero real credentials.
|
||||
|
||||
- **Monitor unit tests** — `FakeMbBrowser` (canned release-group lists) + seeded
|
||||
Postgres. Assert: discovery sets `monitored` correctly (default off; on only
|
||||
when `autoMonitorFuture` and new-vs-`monitorFrom`); sweep enqueues only due
|
||||
rows; respects poll/retry intervals; skips releases with active jobs; applies
|
||||
cutoff and upgrade-window transitions; dedupes against `LibraryItem`.
|
||||
- **Reconcile tests** — a finished job in each end state updates its
|
||||
`MonitoredRelease` correctly (imported below/above cutoff; needs_attention).
|
||||
- **Web API route tests** — artists/wanted/releases CRUD and toggles against
|
||||
seeded Postgres; the MB read routes tested against a stubbed MB read module.
|
||||
- **Opt-in live test** (`LYRA_LIVE_TESTS`) — browse a real artist's discography
|
||||
and search a real artist name.
|
||||
- **Backward-compat** — with `monitor.enabled = false`, the full existing worker
|
||||
suite passes unchanged and no monitoring side effects occur.
|
||||
|
||||
**Goal:** the entire library manager is built and tested on a laptop with fakes
|
||||
and a seeded database; real MusicBrainz enters only for the opt-in live test.
|
||||
|
||||
## Notes: after this slice
|
||||
|
||||
Slice 2 makes Lyra a standing library manager: follow artists, monitor exactly
|
||||
the releases you want (or auto-monitor future ones per artist), and let the
|
||||
background sweep obtain and upgrade them through the slice 1 pipeline, with a
|
||||
wanted list that never forgets. Deferred to later: notifications, a release
|
||||
calendar, and slice 3 (discovery), each with its own spec → plan → implementation
|
||||
cycle.
|
||||
Reference in New Issue
Block a user