Lyra/docs/superpowers/specs/2026-07-11-lyra-library-manager-design.md

# Lyra — Library Manager (Design)

**Date:** 2026-07-11
**Status:** Approved design, ready for implementation planning
**Slice:** 2 of 3 (Library manager)

## Overview

Slice 2 turns Lyra from a one-shot acquisition tool into a standing library
manager: you follow artists, monitor the specific releases you want, and Lyra
keeps working in the background to obtain and upgrade them. It is Lidarr-style
monitoring built directly on the slice 1 acquisition engine — the monitor's only
output is the same `Request` + `Job` a manual request produces, so the entire
existing pipeline is reused unchanged.

Two loops are equally central and share one monitoring engine:

1. **Watched artists → auto-grab.** Follow an artist, browse their discography,
   and monitor the releases you want. Optionally flip a per-artist toggle so
   *future* releases auto-monitor as they appear.
2. **Wanted list → keep retrying.** Every monitored release without an acceptable
   copy is "wanted" and is re-attempted on a schedule until a good copy lands —
   and, within a window, upgraded to better quality.

### Goals

- Follow artists and monitor releases at release-group granularity; obtain and
  keep them current automatically.
- Reuse the slice 1 pipeline verbatim — monitoring enqueues normal jobs and reads
  their outcomes; the pipeline itself is untouched.
- Never silently drop a want: a monitored release without a copy stays wanted and
  keeps being retried.
- Keep sources and secrets in the worker. The web app performs only read-only,
  credential-free MusicBrainz *metadata* lookups for interactive search/browse.
- Fully developable and testable with zero real credentials via a fake
  MusicBrainz browser and seeded Postgres.

### Non-goals (this slice)

- Notifications / webhooks on grab or needs-attention (a later slice).
- Discovery / recommendations (slice 3).
- A release-calendar / upcoming view.
- Multi-user; this remains a single-user home-server tool.
- Any change to how downloading, tagging, or the six-stage pipeline work.

## Scope

**In scope:** watched artists (live MusicBrainz search + follow), a per-artist
discography detail view with monitor toggles, a per-artist "auto-monitor future
releases" toggle, a background monitor sweep (discovery + retry/upgrade), a
cross-artist wanted list, and the outcome-reconcile that feeds job results back
into monitoring state.

**Out of scope:** notifications, discovery, calendar. Deferred hardening from
slice 1 is unaffected and remains deferred.

## Key decisions

- **Default is passive.** Following an artist marks their *entire* discography
  **unmonitored**. You opt in per release. A per-artist `autoMonitorFuture`
  toggle (default **off**) makes genuinely-new releases auto-monitor from the
  moment it is enabled.
- **No type filtering when auto-monitoring.** When `autoMonitorFuture` is on, all
  official release-group types (albums, EPs, singles, live, compilations, …)
  auto-monitor; you unmonitor the noise by hand. ("Everything, filter later.")
- **Quality upgrades within a window.** A quality **cutoff** and an **upgrade
  window** (N days) govern re-searching: below cutoff and within the window → keep
  chasing better quality; at/above cutoff, or window expired → frozen (done).
- **One table for discography *and* wanted list.** A "wanted" item is simply a
  monitored release-group without a copy at cutoff. No separate wanted table, no
  sync problem.
- **Monitor lives in the worker.** A scheduled sweep runs inside the existing
  worker loop. The web app owns CRUD + display, plus read-only MusicBrainz search
  for the interactive picker.

## Architecture

No new containers. Two additions to the existing three-container stack:

- **Worker gains a monitor.** Beside the job-claim loop, a cheap ~60s tick runs
  `monitor.sweep()`; after each pipeline run the loop calls
  `monitor.reconcile()`. All scheduled MusicBrainz work and all enqueuing live
  here, next to the pipeline it feeds.
- **Web gains monitoring UI + read-only MB reads.** New pages/routes for watched
  artists, discography detail, and the wanted list, plus a thin MusicBrainz read
  module used only for the live search/browse picker.

```
┌───────────────────────────┐   Postgres    ┌──────────────────────────────┐
│  Next.js app (UI + API)   │──────────────▶│  Python worker               │
│                           │  requests/    │                              │
│  /artists  /artists/[id]  │  jobs +       │  claim → run_pipeline (S1)   │
│  /wanted   live MB picker │  monitor rows │  monitor.sweep()  (~60s tick)│
│                           │◀──────────────│  monitor.reconcile(job)      │
│  read-only MB search ─────┼──▶ MusicBrainz│  scheduled MB discovery ─────┼─▶ MB
└───────────────────────────┘  (metadata)   └──────────────────────────────┘
```

**Separation of concerns.** The web reasons about *watched artists, monitored
releases, requests, and jobs*; the worker reasons about *candidates, files, and
scheduled acquisition*. The web's only external reach is **read-only, cred-free
MusicBrainz metadata** for the picker — every download **source** and every
**credential** remains worker-only, as does the scheduled discovery poll and all
pipeline-time MB resolution.

## Data model

Two new tables, one new nullable column, and config keys in the existing
key/value `Config` table (no migration for config).

```
WatchedArtist
  id
  mbid                (unique)                 # MusicBrainz artist MBID
  name
  autoMonitorFuture   Bool  default false      # auto-monitor new releases going forward
  monitorFrom         DateTime default now()   # boundary for "future" when the toggle is on
  lastPolledAt        DateTime?                 # throttles the discovery poll per artist
  createdAt
  monitoredReleases   MonitoredRelease[]

MonitoredRelease                                # discography row AND wanted-list row
  id
  watchedArtistId     FK?                       # null = standalone manual wanted add
  artistMbid, artistName                        # denormalized: standalone rows need no join
  rgMbid              (unique)                   # release-group MBID
  album
  primaryType         String?                    # "Album" | "EP" | "Single" | ...
  secondaryTypes      String[]                   # ["Live", "Compilation", ...]
  firstReleaseDate    String?                    # "YYYY-MM-DD" as MB reports it
  monitored           Bool  default false        # does the sweep act on it?
  state               enum(wanted|grabbed|fulfilled)  default wanted
  currentQualityClass Int?                        # quality of the copy we have (null = none)
  firstGrabbedAt      DateTime?                   # drives the upgrade window
  lastSearchedAt      DateTime?                   # throttles retry
  createdAt

Request  (existing) gains:
  monitoredReleaseId  String?                     # set when the monitor enqueues; null for manual
```

**Config keys** (worker-read, via existing `Config`):

| key | default | meaning |
|-----|---------|---------|
| `monitor.enabled` | `false` | master switch; off → worker behaves exactly as slice 1 |
| `monitor.pollIntervalHours` | `24` | how often discovery re-browses each artist |
| `monitor.retryIntervalHours` | `6` | how often a wanted item is re-searched |
| `monitor.qualityCutoff` | `2` | the `qualityClass` at/above which a release is "done" |
| `monitor.upgradeWindowDays` | `14` | days after first grab to keep chasing upgrades |

`qualityClass` (from slice 1's `quality.py`): **3** = hi-res lossless, **2** = CD
lossless, **1** = lossy. The default cutoff `2` means "any lossless satisfies;
only a lossy copy stays wanted and keeps chasing an upgrade."

### How rows come to exist

- **Follow (live picker):** the web already fetched the artist + discography from
  MusicBrainz to render the picker. On Follow it writes the `WatchedArtist` and
  persists the discography as `MonitoredRelease` rows, all **unmonitored**. The
  detail view is therefore instant.
- **Background discovery:** the worker's poll re-browses each watched artist and
  inserts any *newly appearing* release-groups. If `autoMonitorFuture` is on and
  the release is new relative to `monitorFrom`, it is inserted **monitored +
  wanted**; otherwise **unmonitored**.
- **Standalone wanted add:** adding an arbitrary album to the wanted list (not via
  a followed artist) resolves it to a release-group and writes a single
  `MonitoredRelease` with `watchedArtistId = null`, `monitored = true`.

## The monitor (worker)

New module `worker/lyra_worker/monitor.py`. It uses an `MbBrowser` seam (below)
and plain SQL over the shared connection; scheduling is driven entirely by
timestamps in the tables, so the loop tick is just a cheap gate.

### Scheduling

`run_forever()` records the last tick time; each loop iteration, if
`monitor.enabled` and ≥ ~60s elapsed, it calls `monitor.sweep(conn, browser,
cfg)` before continuing to claim jobs. After every `run_pipeline` it calls
`monitor.reconcile(conn, job_id)`. One process, one loop, no new infra.

### `sweep()` — two phases

**Phase 1 — Discovery** (per artist where `now - lastPolledAt >
pollIntervalHours`):
- `browser.browse_release_groups(artist.mbid)` → release-group list.
- For each rg not already a `MonitoredRelease`, insert it. `monitored =
  artist.autoMonitorFuture and firstReleaseDate > artist.monitorFrom`; state
  `wanted`. (All types eligible — no filtering.)
- Set `lastPolledAt = now`.

**Phase 2 — Retry / enqueue** (rows with `monitored = true`, `state != fulfilled`):
- Skip rows with an **active** (non-terminal) job for the same release.
- Skip / satisfy via dedupe: if a `LibraryItem` at/above cutoff already exists →
  set `fulfilled`.
- For `state = grabbed`: if `now - firstGrabbedAt >= upgradeWindowDays` **or**
  `currentQualityClass >= cutoff` → set `fulfilled` and skip; else it is an
  upgrade candidate.
- For due rows (`lastSearchedAt` null or older than `retryIntervalHours`): create
  `Request(artist, album, monitoredReleaseId=id)` + `Job(state=requested)`; set
  `lastSearchedAt = now`. The slice 1 pipeline takes it from there.

### `reconcile(job_id)` — feed outcomes back

After the pipeline finishes a job that has a `monitoredReleaseId`:
- **imported:** set `currentQualityClass` from the winning candidate and
  `firstGrabbedAt` (if null); set `state = fulfilled` if `qualityClass >= cutoff`,
  else `grabbed`.
- **needs_attention:** leave `state = wanted` (already has `lastSearchedAt`, so it
  waits out the retry interval before the next attempt).

Reconcile is a separate call in the main loop, so the pipeline code is untouched
and the whole feature is inert when `monitor.enabled = false`.

### MonitoredRelease state machine

```
discovery / follow
   ├─(autoMonitorFuture & new)─► monitored + wanted ─┐
   └─(default / back-cat)──────► unmonitored + wanted │  user toggle → monitored
                                                       ▼
   wanted ──sweep enqueues──► [job] ──imported < cutoff──► grabbed ──window expired / at cutoff──► fulfilled
      ▲                          │                            │
      └─────needs_attention──────┘        imported ≥ cutoff───┴──────────────────────────────► fulfilled
```

Unmonitor at any time → dormant (the sweep ignores it). A want is never dropped;
it is retried on its interval until satisfied.

## MusicBrainz access

The worker's existing `MusicBrainzResolver` handles pipeline-time
album resolution. Monitoring adds a small browsing seam.

**Worker — `MbBrowser` protocol** (real impl via `musicbrainzngs`, lazily
imported; faked in tests):
- `search_artist(name) -> list[ArtistHit]` — `{mbid, name, disambiguation}`.
- `browse_release_groups(artist_mbid) -> list[ReleaseGroupInfo]` — `{rgMbid,
  title, primaryType, secondaryTypes, firstReleaseDate}`.

**Web — read-only MB read module** for the live picker only: artist search and
discography browse, returning the same shapes the web persists as
`MonitoredRelease` rows on Follow. No credentials; read-only; metadata only.

## Web UI & API

Three surfaces, thin over the two tables:

- **`/artists`** — watched artists (name, `#monitored` / `#have`), **Add artist**
  (live search picker), remove. Per-artist `autoMonitorFuture` toggle.
- **`/artists/[id]`** — the discography: every `MonitoredRelease` for the artist,
  each with a status badge (Have@quality / Wanted / Dormant), a **monitor toggle**,
  and a **Search now** button (enqueues immediately).
- **`/wanted`** — cross-artist list of everything `monitored` and not `fulfilled`
  (includes standalone manual adds and needs-attention retries), with
  last-searched time and **Search now**. Plus **Add to wanted** (resolve an
  arbitrary artist+album → standalone monitored row).

**API routes (Next.js):**
- `GET /api/mb/artists?q=` — live artist search (read-only MB).
- `GET /api/mb/artists/[mbid]/releases` — discography browse (read-only MB).
- `POST /api/artists` — follow: create `WatchedArtist` + persist discography as
  unmonitored `MonitoredRelease` rows.
- `GET /api/artists`, `GET /api/artists/[id]`, `DELETE /api/artists/[id]`,
  `PATCH /api/artists/[id]` (toggle `autoMonitorFuture`).
- `GET /api/wanted` — monitored, not-fulfilled rows across artists.
- `POST /api/wanted` — standalone add (resolve + create row).
- `PATCH /api/releases/[id]` — toggle `monitored`.
- `POST /api/releases/[id]/search` — enqueue now (`Request` + `Job`).

"Search now" and standalone adds create the same `Request` + `Job` the monitor
does (with `monitoredReleaseId` set), so reconcile handles their outcomes too.

## Integration touch-point: upgrades must replace

Quality upgrades re-import an album that is already in the library.
`LibraryItem` is `@@unique([artist, album])`, so the import stage must **upsert**
the row and overwrite the on-disk files rather than error on a duplicate. Current
behavior will be verified during planning; if it errors today, the minimal
import-upsert change is included in this slice. This is the only pipeline-adjacent
change; the six stages otherwise stand unchanged.

## Error handling & edge cases

| Situation | Handling |
|-----------|----------|
| New release not yet on any source | Job → needs_attention; row stays wanted; retried each interval until a copy appears (natural pre-order behavior). |
| Artist name ambiguous | The live picker shows MB `disambiguation`; the user chooses. No blind auto-match. |
| Duplicate discovery | `MonitoredRelease.rgMbid` is unique; re-browsing never duplicates rows. |
| Piled-up jobs | Sweep skips a release that already has a non-terminal job. |
| Already in library | Dedupe against `LibraryItem`; at/above cutoff → `fulfilled` without enqueuing. |
| Upgrade never improves | Upgrade window expires → `fulfilled` (frozen); no endless re-searching. |
| MB unreachable during sweep | Discovery for that artist is skipped this tick; retried next poll. Web picker surfaces the error to the user. |
| Monitor disabled | `monitor.enabled = false` → no sweep, no reconcile effects; worker behaves exactly as slice 1. |

## Testing strategy

Same philosophy as slice 1 — logic is testable with zero real credentials.

- **Monitor unit tests** — `FakeMbBrowser` (canned release-group lists) + seeded
  Postgres. Assert: discovery sets `monitored` correctly (default off; on only
  when `autoMonitorFuture` and new-vs-`monitorFrom`); sweep enqueues only due
  rows; respects poll/retry intervals; skips releases with active jobs; applies
  cutoff and upgrade-window transitions; dedupes against `LibraryItem`.
- **Reconcile tests** — a finished job in each end state updates its
  `MonitoredRelease` correctly (imported below/above cutoff; needs_attention).
- **Web API route tests** — artists/wanted/releases CRUD and toggles against
  seeded Postgres; the MB read routes tested against a stubbed MB read module.
- **Opt-in live test** (`LYRA_LIVE_TESTS`) — browse a real artist's discography
  and search a real artist name.
- **Backward-compat** — with `monitor.enabled = false`, the full existing worker
  suite passes unchanged and no monitoring side effects occur.

**Goal:** the entire library manager is built and tested on a laptop with fakes
and a seeded database; real MusicBrainz enters only for the opt-in live test.

## Notes: after this slice

Slice 2 makes Lyra a standing library manager: follow artists, monitor exactly
the releases you want (or auto-monitor future ones per artist), and let the
background sweep obtain and upgrade them through the slice 1 pipeline, with a
wanted list that never forgets. Deferred to later: notifications, a release
calendar, and slice 3 (discovery), each with its own spec → plan → implementation
cycle.