diff --git a/docs/superpowers/specs/2026-07-11-lyra-discovery-design.md b/docs/superpowers/specs/2026-07-11-lyra-discovery-design.md new file mode 100644 index 0000000..97f6f93 --- /dev/null +++ b/docs/superpowers/specs/2026-07-11-lyra-discovery-design.md @@ -0,0 +1,319 @@ +# Lyra — Discovery (Design) + +**Date:** 2026-07-11 +**Status:** Approved design, ready for implementation planning +**Slice:** 3 of 3 (Discovery) + +## Overview + +Slice 3 makes Lyra surface music you **don't already follow**. Slices 1–2 +obtain and monitor artists/releases you already know about; discovery closes the +loop by recommending *new* artists and albums similar to what's in your library, +then hands each accepted suggestion straight back into the slice-2 machinery. + +Discovery is deliberately **suggestion-only and manual**: a background sweep (and +an on-demand button) populates a review feed of candidate artists and albums; you +**Follow** an artist (creating a `WatchedArtist` with its discography tracked but +**unmonitored**, exactly like the slice-2 follow) or **Want** an album (creating a +`monitored` wanted `MonitoredRelease`, exactly like the slice-2 wanted list), or +**Dismiss** it. Discovery never grabs anything itself — its only outputs are the +exact rows the existing follow / wanted flows already create, and actual grabbing +still happens only through the slice-2 monitor (off by default). The entire +acquisition + monitoring pipeline is reused unchanged. + +### The source reality that shapes this slice + +The original acquisition-engine spec reserved **Spotify** for "discovery/metadata +only." That premise no longer holds for a newly-registered app: + +- **Nov 2024** — Spotify deprecated **Related Artists**, **Recommendations**, + audio-features, audio-analysis and featured-playlists for any *new* app (new + apps receive 403/404; no waitlist, no path back). +- **Feb 2026** — Spotify additionally removed **Get New Releases**, Get Artist's + Top Tracks and Get Several Albums. + +So Spotify can supply **none** of discovery's signals to a fresh app. Lyra is +already **MusicBrainz-native** (every artist/release is an MBID), so discovery is +instead built on **MetaBrainz**: **ListenBrainz similar-artists** (same family as +MusicBrainz, returns MBIDs directly — zero fuzzy matching) as the engine, with +MusicBrainz for the album sweep. Spotify is dropped entirely from this slice. + +### Goals + +- Recommend new **artists** (and standout **albums**) similar to the library, at + MBID granularity, with no personal-account / listening-history dependency. +- Reuse slice-2 verbatim: **Follow** creates the same `WatchedArtist` + + unmonitored discography the follow flow does; **Want** creates the same + `monitored` wanted `MonitoredRelease` the wanted list does. No new grab logic. +- Manual review only — nothing is followed or grabbed without explicit approval; + **dismissed suggestions never resurface**. +- Keep the design source-agnostic behind a pluggable `SimilaritySource` interface + (mirroring the slice-1 `adapters/` registry), so a second source (Last.fm) or + tag-weighting can drop in later without touching the core. +- Fully developable and testable with **zero credentials** (ListenBrainz labs API + is unauthenticated) via a fake `SimilaritySource` and seeded Postgres. + +### Non-goals (this slice) + +- **No Spotify** anywhere (its similarity/new-release APIs are dead for new apps). +- **No auto-follow / auto-grab** — manual review only. +- **No personal listening data / OAuth** (no ListenBrainz user account required). +- **No second similarity source (Last.fm) and no genre-tag re-weighting** in this + slice — both are left as drop-in extensions behind `SimilaritySource`. +- No notifications; no multi-user. Downloading, tagging, and the six-stage + pipeline are untouched. + +## Scope + +**In scope:** a `SimilaritySource` engine (ListenBrainz), a worker discovery run +that aggregates similar artists across your library's artists and derives album +suggestions, a `DiscoverySuggestion` review feed, per-suggestion Follow / Want / +Dismiss actions that reuse slice-2 operations, an interactive seed-search box, a +scheduled sweep + "Discover now" trigger (off by default), and Discovery settings. + +**Out of scope:** everything under Non-goals. Deferred hardening from slices 1–2 +is unaffected and remains deferred. + +## Key decisions + +- **Web reads, worker persists** — the same split slice 2 established. The + interactive **seed-search box** calls ListenBrainz directly from the web + (unauthenticated, read-only, results not persisted), exactly as `/api/mb/artists` + does live MusicBrainz search today. The **scheduled sweep** and **"Discover + now"** run in the worker and persist `DiscoverySuggestion` rows. +- **Suggestion-only, manual.** Discovery emits suggestions; the *user* turns a + suggestion into a `WatchedArtist` (Follow) or a wanted `MonitoredRelease` + (Want). Discovery has no path that grabs audio. +- **Dismissed is permanent.** A `dedupeKey` unique column plus keeping + dismissed/followed/wanted rows means the sweep's conditional upsert can never + revert them to `pending` — dismissed suggestions never come back. +- **Off by default**, like the monitor: `discover.enabled` unset ⇒ no background + sweep. The on-demand button and seed-search still work regardless. +- **Pluggable engine.** One source (ListenBrainz) ships now behind a + `SimilaritySource` Protocol + `build_similarity_sources()` registry, matching + the existing `SourceAdapter` / `build_adapters` pattern. + +## Architecture + +``` + WEB (read-only, no secrets) WORKER (persists) + /discover page ─┬─ GET /api/discover (feed) ───▶ DiscoverySuggestion (read) + ├─ GET /api/discover/search ──────────▶ MB search + ListenBrainz + │ (interactive seed, not stored) (direct, unauthenticated) + ├─ POST /api/discover/run ────────────▶ Config discover.requested=true + └─ POST /api/discover/[id] (action) ───▶ Follow → WatchedArtist + Want → MonitoredRelease + Dismiss→ status=dismissed + + worker loop tick (discover.enabled, every discover.intervalHours; or on + discover.requested): + run_discovery(conn, sources, browser, cfg): + seeds = WatchedArtist.mbid (throttled by lastDiscoveredAt, capped maxSeeds) + for seed × source: similar_artists(mbid) → aggregate + filter out already-followed / dismissed + upsert ARTIST suggestions (conditional on status='pending') + for top artists: MbBrowser release-groups → core-release filter → + drop library dupes → upsert ALBUM suggestions +``` + +## Data model + +One new table + two enums, plus one column on `WatchedArtist`. + +```prisma +enum DiscoveryKind { artist album } +enum SuggestionStatus { pending followed wanted dismissed } + +model DiscoverySuggestion { + id String @id @default(cuid()) + kind DiscoveryKind + artistMbid String + artistName String + rgMbid String? // album kind only; null for artist + album String? + primaryType String? + secondaryTypes String[] + firstReleaseDate String? + score Float // aggregated similarity across seeds + seedCount Int // how many of your artists pointed here + sources String[] // e.g. ["listenbrainz"] (pluggable) + status SuggestionStatus @default(pending) + dedupeKey String @unique // kind + ':' + artistMbid + ':' + (rgMbid ?? "") + createdAt DateTime @default(now()) + updatedAt DateTime @updatedAt +} +``` + +Plus on `WatchedArtist`: + +```prisma + lastDiscoveredAt DateTime? // per-seed throttle, mirrors lastPolledAt +``` + +**Why `dedupeKey`:** Postgres treats `NULL`s as distinct in a composite unique +index, so a `@@unique([kind, artistMbid, rgMbid])` would let duplicate +`kind=artist` rows through (their `rgMbid` is null). A single non-null +`dedupeKey` string gives a clean `ON CONFLICT` target for the upsert. + +**Conditional upsert (the dismissed-is-permanent guarantee):** + +```sql +INSERT INTO "DiscoverySuggestion" (... , status, "dedupeKey", ...) +VALUES (... , 'pending', %s, ...) +ON CONFLICT ("dedupeKey") DO UPDATE + SET score = EXCLUDED.score, + "seedCount" = EXCLUDED."seedCount", + sources = EXCLUDED.sources, + "updatedAt" = now() +WHERE "DiscoverySuggestion".status = 'pending'; +``` + +A row already `followed` / `wanted` / `dismissed` fails the `WHERE` and is left +untouched — it never returns to the feed. + +## Engine — `worker/lyra_worker/similarity/` + +Mirrors `worker/lyra_worker/adapters/`. + +- **`base.py`** — Protocol: + ```python + @dataclass(frozen=True) + class SimilarArtist: + mbid: str + name: str + score: float + + @runtime_checkable + class SimilaritySource(Protocol): + name: str + def health(self) -> bool: ... + def similar_artists(self, mbid: str) -> list[SimilarArtist]: ... + ``` +- **`_listenbrainz.py`** — `ListenBrainzSource`: GETs the ListenBrainz Labs + similar-artists dataset (`labs.api.listenbrainz.org`, unauthenticated), maps the + response to `SimilarArtist` (MBIDs come back native). `health()` is a + reachability check. A base-URL override reads from config + (`discover.listenBrainzUrl`, default the public host). +- **`registry.build_similarity_sources(config)`** — health-filtered list, exactly + like `build_adapters`. `build_similarity_sources` is added to `registry.py`. + +## Discovery run — `worker/lyra_worker/discovery.py` + +`DiscoveryConfig.from_config(config)` parses the `discover.*` keys (same shape as +`MonitorConfig`). `run_discovery(conn, sources, browser, cfg) -> DiscoveryResult`: + +1. **Seeds** = `WatchedArtist.mbid` where `lastDiscoveredAt IS NULL OR < + now() - intervalHours`, capped at `maxSeeds`. +2. For each seed × source → `similar_artists(mbid)`, keep top `similarPerSeed`. +3. **Aggregate** per candidate MBID: `score = Σ similarity`, `seedCount =` + distinct seeds pointing here, `sources =` union of contributing source names. +4. **Filter** candidates already in `WatchedArtist` (already followed) and any + existing dismissed row; drop below `minScore`. +5. **Upsert artist suggestions** (conditional-on-pending, above). +6. **Album derivation:** for the top surfaced artists, use the existing + **`MbBrowser`** to list their release-groups, keep only **core releases** via + the existing `release-filter` predicate (reused from slice 2), drop any already + in `LibraryItem` or `MonitoredRelease`, and upsert up to `albumsPerArtist` + (default 1, most-recent core) **album** suggestions. +7. Stamp `lastDiscoveredAt = now()` on each processed seed. Return counts. + +Errors from a single source or seed are logged and skipped — a discovery failure +must never kill the worker loop (same discipline as monitor/scan). + +## Worker loop integration — `worker/lyra_worker/main.py` + +- Build similarity sources at startup: `sources = build_similarity_sources(config)`. +- Add a **discover tick** beside the monitor tick: when `discover.enabled` and the + interval has elapsed, call `run_discovery(...)` inside a try/except that rolls + back and logs on failure. Off by default. +- Handle the one-shot **`discover.requested`** flag exactly like `scan.requested`: + run `run_discovery`, write `discover.result` (`"artists N, albums M"`), clear + `discover.requested`. + +## Config keys (`discover.*`, mirroring `monitor.*`) + +| Key | Default | Meaning | +|---|---|---| +| `discover.enabled` | `false` | master switch for the scheduled sweep | +| `discover.intervalHours` | `168` | sweep cadence (weekly) + per-seed throttle window | +| `discover.maxSeeds` | `50` | seed artists processed per sweep | +| `discover.similarPerSeed` | `20` | top-K similar artists pulled per seed | +| `discover.albumsPerArtist` | `1` | album suggestions per surfaced artist | +| `discover.minScore` | `0` | drop candidates below this aggregated score | +| `discover.listenBrainzUrl` | public host | ListenBrainz Labs base URL override | +| `discover.requested` | — | one-shot "Discover now" trigger (worker clears) | +| `discover.result` | — | last-run status string surfaced in the UI | + +No credential key — the ListenBrainz Labs API is unauthenticated. + +## Web + +New page **`/discover`** (`page.tsx` server component + `discover-client.tsx`), +linked in the nav in `layout.tsx`: + +- **"Discover now"** → `POST /api/discover/run` (sets `discover.requested`); + shows `discover.result`. +- **Seed-search box** → `GET /api/discover/search?artist=`: resolve the name + to an MBID (MB search), fetch ListenBrainz similar, return **unpersisted** + results inline; each is actionable via the same Follow/Want endpoints. +- **Feed** → `GET /api/discover`: pending suggestions, artists and albums in two + sections, sorted by `score` desc, each showing *why* (`seedCount`, `sources`, + `score`) and its actions. + +Actions — **reuse existing slice-2 DB operations, no new grab logic:** + +- **Follow** (artist) → create `WatchedArtist` + browse its discography into + `monitored=false` `MonitoredRelease` rows (the same operation as the + `/api/artists` follow POST), then set the suggestion `status='followed'`. +- **Want** (album) → create a `monitored=true` `MonitoredRelease` **directly from + the suggestion's already-stored fields** (`rgMbid`, `album`, types, + `firstReleaseDate`) — no redundant MusicBrainz search, since the sweep already + resolved them — equivalent to what the wanted list stores; set + `status='wanted'`. +- **Dismiss** → set `status='dismissed'`. + +Endpoints (each with a co-located `.test.ts`, matching convention): +`api/discover/route.ts` (feed), `api/discover/run/route.ts`, +`api/discover/search/route.ts`, `api/discover/[id]/route.ts` (action). + +**Settings** — a "Discovery" section in `settings-form.tsx` mirroring the monitor +fields: `enabled` + the tuning knobs above. + +## Error handling + +| Failure | Behavior | +|---|---| +| ListenBrainz unreachable during sweep | source `health()` false ⇒ that source contributes nothing this tick; logged; retried next sweep. Web seed-search returns an error surfaced in the UI. | +| A single seed errors | logged and skipped; the rest of the sweep proceeds. | +| Duplicate suggestion | `dedupeKey` unique + `ON CONFLICT`; re-sweeping never duplicates rows. | +| Candidate already followed / in library | filtered out before insert (checked against `WatchedArtist` / `LibraryItem` / `MonitoredRelease`). | +| Dismissed candidate reappears from source | conditional upsert leaves the dismissed row untouched; it stays out of the feed. | +| Follow/Want race with existing row | reuse slice-2 operations, which already upsert idempotently. | +| Discovery run throws | caught in the loop, `conn.rollback()`, logged; worker keeps running. | + +## Testing (against `lyra_test`, per the DB guard) + +- **Worker — `discovery.py`:** fake `SimilaritySource` + seeded Postgres. Assert + aggregation (score sum, seedCount, sources union), filtering vs `WatchedArtist` + / `LibraryItem` / `MonitoredRelease`, `minScore` cutoff, the **conditional + upsert never reviving a dismissed row**, album derivation through the + core-release predicate, and `lastDiscoveredAt` throttling. +- **Worker — `_listenbrainz.py`:** thin adapter tested with faked HTTP against + recorded response shapes; `health()` true/false paths. +- **Web:** route tests for the feed, each action (Follow/Want/Dismiss with mocked + DB), and seed-search (mocked MB + ListenBrainz). Reuse the `release-filter` + tests for the core-release predicate. + +## Notes / deferred + +- **Second similarity source (Last.fm) and tag-coherent re-weighting** are the + planned extensions behind `SimilaritySource`; adding either is a new source in + the registry + a `sources`-array boost, no core change. +- **Album popularity ranking.** v1 picks the most-recent core release per artist. + A later refinement can rank an artist's release-groups by ListenBrainz + popular-recordings listen counts. +- **Seed scope** is currently all `WatchedArtist` MBIDs. A "top artists only" + scoping (by library album count) is a possible knob if API load becomes an issue. +- **Discovery is synchronous in the worker loop**, like the scan — fine for the + weekly cadence and bounded by `maxSeeds`, but it briefly blocks job-claim during + a sweep. Chunking mirrors the scan's deferred item.