docs: slice 3 discovery design — MetaBrainz-native (ListenBrainz), suggestion feed
Spotify similarity/new-release APIs are dead for new apps (Nov 2024 + Feb 2026), so discovery is built on ListenBrainz similar-artists (MBID-native) behind a pluggable SimilaritySource interface. Manual review feed of similar artists + albums; Follow/Want reuse slice-2 operations, Dismiss is permanent. Off by default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jonathan
@@ -0,0 +1,319 @@
|
||||
# Lyra — Discovery (Design)
|
||||
|
||||
**Date:** 2026-07-11
|
||||
**Status:** Approved design, ready for implementation planning
|
||||
**Slice:** 3 of 3 (Discovery)
|
||||
|
||||
## Overview
|
||||
|
||||
Slice 3 makes Lyra surface music you **don't already follow**. Slices 1–2
|
||||
obtain and monitor artists/releases you already know about; discovery closes the
|
||||
loop by recommending *new* artists and albums similar to what's in your library,
|
||||
then hands each accepted suggestion straight back into the slice-2 machinery.
|
||||
|
||||
Discovery is deliberately **suggestion-only and manual**: a background sweep (and
|
||||
an on-demand button) populates a review feed of candidate artists and albums; you
|
||||
**Follow** an artist (creating a `WatchedArtist` with its discography tracked but
|
||||
**unmonitored**, exactly like the slice-2 follow) or **Want** an album (creating a
|
||||
`monitored` wanted `MonitoredRelease`, exactly like the slice-2 wanted list), or
|
||||
**Dismiss** it. Discovery never grabs anything itself — its only outputs are the
|
||||
exact rows the existing follow / wanted flows already create, and actual grabbing
|
||||
still happens only through the slice-2 monitor (off by default). The entire
|
||||
acquisition + monitoring pipeline is reused unchanged.
|
||||
|
||||
### The source reality that shapes this slice
|
||||
|
||||
The original acquisition-engine spec reserved **Spotify** for "discovery/metadata
|
||||
only." That premise no longer holds for a newly-registered app:
|
||||
|
||||
- **Nov 2024** — Spotify deprecated **Related Artists**, **Recommendations**,
|
||||
audio-features, audio-analysis and featured-playlists for any *new* app (new
|
||||
apps receive 403/404; no waitlist, no path back).
|
||||
- **Feb 2026** — Spotify additionally removed **Get New Releases**, Get Artist's
|
||||
Top Tracks and Get Several Albums.
|
||||
|
||||
So Spotify can supply **none** of discovery's signals to a fresh app. Lyra is
|
||||
already **MusicBrainz-native** (every artist/release is an MBID), so discovery is
|
||||
instead built on **MetaBrainz**: **ListenBrainz similar-artists** (same family as
|
||||
MusicBrainz, returns MBIDs directly — zero fuzzy matching) as the engine, with
|
||||
MusicBrainz for the album sweep. Spotify is dropped entirely from this slice.
|
||||
|
||||
### Goals
|
||||
|
||||
- Recommend new **artists** (and standout **albums**) similar to the library, at
|
||||
MBID granularity, with no personal-account / listening-history dependency.
|
||||
- Reuse slice-2 verbatim: **Follow** creates the same `WatchedArtist` +
|
||||
unmonitored discography the follow flow does; **Want** creates the same
|
||||
`monitored` wanted `MonitoredRelease` the wanted list does. No new grab logic.
|
||||
- Manual review only — nothing is followed or grabbed without explicit approval;
|
||||
**dismissed suggestions never resurface**.
|
||||
- Keep the design source-agnostic behind a pluggable `SimilaritySource` interface
|
||||
(mirroring the slice-1 `adapters/` registry), so a second source (Last.fm) or
|
||||
tag-weighting can drop in later without touching the core.
|
||||
- Fully developable and testable with **zero credentials** (ListenBrainz labs API
|
||||
is unauthenticated) via a fake `SimilaritySource` and seeded Postgres.
|
||||
|
||||
### Non-goals (this slice)
|
||||
|
||||
- **No Spotify** anywhere (its similarity/new-release APIs are dead for new apps).
|
||||
- **No auto-follow / auto-grab** — manual review only.
|
||||
- **No personal listening data / OAuth** (no ListenBrainz user account required).
|
||||
- **No second similarity source (Last.fm) and no genre-tag re-weighting** in this
|
||||
slice — both are left as drop-in extensions behind `SimilaritySource`.
|
||||
- No notifications; no multi-user. Downloading, tagging, and the six-stage
|
||||
pipeline are untouched.
|
||||
|
||||
## Scope
|
||||
|
||||
**In scope:** a `SimilaritySource` engine (ListenBrainz), a worker discovery run
|
||||
that aggregates similar artists across your library's artists and derives album
|
||||
suggestions, a `DiscoverySuggestion` review feed, per-suggestion Follow / Want /
|
||||
Dismiss actions that reuse slice-2 operations, an interactive seed-search box, a
|
||||
scheduled sweep + "Discover now" trigger (off by default), and Discovery settings.
|
||||
|
||||
**Out of scope:** everything under Non-goals. Deferred hardening from slices 1–2
|
||||
is unaffected and remains deferred.
|
||||
|
||||
## Key decisions
|
||||
|
||||
- **Web reads, worker persists** — the same split slice 2 established. The
|
||||
interactive **seed-search box** calls ListenBrainz directly from the web
|
||||
(unauthenticated, read-only, results not persisted), exactly as `/api/mb/artists`
|
||||
does live MusicBrainz search today. The **scheduled sweep** and **"Discover
|
||||
now"** run in the worker and persist `DiscoverySuggestion` rows.
|
||||
- **Suggestion-only, manual.** Discovery emits suggestions; the *user* turns a
|
||||
suggestion into a `WatchedArtist` (Follow) or a wanted `MonitoredRelease`
|
||||
(Want). Discovery has no path that grabs audio.
|
||||
- **Dismissed is permanent.** A `dedupeKey` unique column plus keeping
|
||||
dismissed/followed/wanted rows means the sweep's conditional upsert can never
|
||||
revert them to `pending` — dismissed suggestions never come back.
|
||||
- **Off by default**, like the monitor: `discover.enabled` unset ⇒ no background
|
||||
sweep. The on-demand button and seed-search still work regardless.
|
||||
- **Pluggable engine.** One source (ListenBrainz) ships now behind a
|
||||
`SimilaritySource` Protocol + `build_similarity_sources()` registry, matching
|
||||
the existing `SourceAdapter` / `build_adapters` pattern.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
WEB (read-only, no secrets) WORKER (persists)
|
||||
/discover page ─┬─ GET /api/discover (feed) ───▶ DiscoverySuggestion (read)
|
||||
├─ GET /api/discover/search ──────────▶ MB search + ListenBrainz
|
||||
│ (interactive seed, not stored) (direct, unauthenticated)
|
||||
├─ POST /api/discover/run ────────────▶ Config discover.requested=true
|
||||
└─ POST /api/discover/[id] (action) ───▶ Follow → WatchedArtist
|
||||
Want → MonitoredRelease
|
||||
Dismiss→ status=dismissed
|
||||
|
||||
worker loop tick (discover.enabled, every discover.intervalHours; or on
|
||||
discover.requested):
|
||||
run_discovery(conn, sources, browser, cfg):
|
||||
seeds = WatchedArtist.mbid (throttled by lastDiscoveredAt, capped maxSeeds)
|
||||
for seed × source: similar_artists(mbid) → aggregate
|
||||
filter out already-followed / dismissed
|
||||
upsert ARTIST suggestions (conditional on status='pending')
|
||||
for top artists: MbBrowser release-groups → core-release filter →
|
||||
drop library dupes → upsert ALBUM suggestions
|
||||
```
|
||||
|
||||
## Data model
|
||||
|
||||
One new table + two enums, plus one column on `WatchedArtist`.
|
||||
|
||||
```prisma
|
||||
enum DiscoveryKind { artist album }
|
||||
enum SuggestionStatus { pending followed wanted dismissed }
|
||||
|
||||
model DiscoverySuggestion {
|
||||
id String @id @default(cuid())
|
||||
kind DiscoveryKind
|
||||
artistMbid String
|
||||
artistName String
|
||||
rgMbid String? // album kind only; null for artist
|
||||
album String?
|
||||
primaryType String?
|
||||
secondaryTypes String[]
|
||||
firstReleaseDate String?
|
||||
score Float // aggregated similarity across seeds
|
||||
seedCount Int // how many of your artists pointed here
|
||||
sources String[] // e.g. ["listenbrainz"] (pluggable)
|
||||
status SuggestionStatus @default(pending)
|
||||
dedupeKey String @unique // kind + ':' + artistMbid + ':' + (rgMbid ?? "")
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
}
|
||||
```
|
||||
|
||||
Plus on `WatchedArtist`:
|
||||
|
||||
```prisma
|
||||
lastDiscoveredAt DateTime? // per-seed throttle, mirrors lastPolledAt
|
||||
```
|
||||
|
||||
**Why `dedupeKey`:** Postgres treats `NULL`s as distinct in a composite unique
|
||||
index, so a `@@unique([kind, artistMbid, rgMbid])` would let duplicate
|
||||
`kind=artist` rows through (their `rgMbid` is null). A single non-null
|
||||
`dedupeKey` string gives a clean `ON CONFLICT` target for the upsert.
|
||||
|
||||
**Conditional upsert (the dismissed-is-permanent guarantee):**
|
||||
|
||||
```sql
|
||||
INSERT INTO "DiscoverySuggestion" (... , status, "dedupeKey", ...)
|
||||
VALUES (... , 'pending', %s, ...)
|
||||
ON CONFLICT ("dedupeKey") DO UPDATE
|
||||
SET score = EXCLUDED.score,
|
||||
"seedCount" = EXCLUDED."seedCount",
|
||||
sources = EXCLUDED.sources,
|
||||
"updatedAt" = now()
|
||||
WHERE "DiscoverySuggestion".status = 'pending';
|
||||
```
|
||||
|
||||
A row already `followed` / `wanted` / `dismissed` fails the `WHERE` and is left
|
||||
untouched — it never returns to the feed.
|
||||
|
||||
## Engine — `worker/lyra_worker/similarity/`
|
||||
|
||||
Mirrors `worker/lyra_worker/adapters/`.
|
||||
|
||||
- **`base.py`** — Protocol:
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class SimilarArtist:
|
||||
mbid: str
|
||||
name: str
|
||||
score: float
|
||||
|
||||
@runtime_checkable
|
||||
class SimilaritySource(Protocol):
|
||||
name: str
|
||||
def health(self) -> bool: ...
|
||||
def similar_artists(self, mbid: str) -> list[SimilarArtist]: ...
|
||||
```
|
||||
- **`_listenbrainz.py`** — `ListenBrainzSource`: GETs the ListenBrainz Labs
|
||||
similar-artists dataset (`labs.api.listenbrainz.org`, unauthenticated), maps the
|
||||
response to `SimilarArtist` (MBIDs come back native). `health()` is a
|
||||
reachability check. A base-URL override reads from config
|
||||
(`discover.listenBrainzUrl`, default the public host).
|
||||
- **`registry.build_similarity_sources(config)`** — health-filtered list, exactly
|
||||
like `build_adapters`. `build_similarity_sources` is added to `registry.py`.
|
||||
|
||||
## Discovery run — `worker/lyra_worker/discovery.py`
|
||||
|
||||
`DiscoveryConfig.from_config(config)` parses the `discover.*` keys (same shape as
|
||||
`MonitorConfig`). `run_discovery(conn, sources, browser, cfg) -> DiscoveryResult`:
|
||||
|
||||
1. **Seeds** = `WatchedArtist.mbid` where `lastDiscoveredAt IS NULL OR <
|
||||
now() - intervalHours`, capped at `maxSeeds`.
|
||||
2. For each seed × source → `similar_artists(mbid)`, keep top `similarPerSeed`.
|
||||
3. **Aggregate** per candidate MBID: `score = Σ similarity`, `seedCount =`
|
||||
distinct seeds pointing here, `sources =` union of contributing source names.
|
||||
4. **Filter** candidates already in `WatchedArtist` (already followed) and any
|
||||
existing dismissed row; drop below `minScore`.
|
||||
5. **Upsert artist suggestions** (conditional-on-pending, above).
|
||||
6. **Album derivation:** for the top surfaced artists, use the existing
|
||||
**`MbBrowser`** to list their release-groups, keep only **core releases** via
|
||||
the existing `release-filter` predicate (reused from slice 2), drop any already
|
||||
in `LibraryItem` or `MonitoredRelease`, and upsert up to `albumsPerArtist`
|
||||
(default 1, most-recent core) **album** suggestions.
|
||||
7. Stamp `lastDiscoveredAt = now()` on each processed seed. Return counts.
|
||||
|
||||
Errors from a single source or seed are logged and skipped — a discovery failure
|
||||
must never kill the worker loop (same discipline as monitor/scan).
|
||||
|
||||
## Worker loop integration — `worker/lyra_worker/main.py`
|
||||
|
||||
- Build similarity sources at startup: `sources = build_similarity_sources(config)`.
|
||||
- Add a **discover tick** beside the monitor tick: when `discover.enabled` and the
|
||||
interval has elapsed, call `run_discovery(...)` inside a try/except that rolls
|
||||
back and logs on failure. Off by default.
|
||||
- Handle the one-shot **`discover.requested`** flag exactly like `scan.requested`:
|
||||
run `run_discovery`, write `discover.result` (`"artists N, albums M"`), clear
|
||||
`discover.requested`.
|
||||
|
||||
## Config keys (`discover.*`, mirroring `monitor.*`)
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `discover.enabled` | `false` | master switch for the scheduled sweep |
|
||||
| `discover.intervalHours` | `168` | sweep cadence (weekly) + per-seed throttle window |
|
||||
| `discover.maxSeeds` | `50` | seed artists processed per sweep |
|
||||
| `discover.similarPerSeed` | `20` | top-K similar artists pulled per seed |
|
||||
| `discover.albumsPerArtist` | `1` | album suggestions per surfaced artist |
|
||||
| `discover.minScore` | `0` | drop candidates below this aggregated score |
|
||||
| `discover.listenBrainzUrl` | public host | ListenBrainz Labs base URL override |
|
||||
| `discover.requested` | — | one-shot "Discover now" trigger (worker clears) |
|
||||
| `discover.result` | — | last-run status string surfaced in the UI |
|
||||
|
||||
No credential key — the ListenBrainz Labs API is unauthenticated.
|
||||
|
||||
## Web
|
||||
|
||||
New page **`/discover`** (`page.tsx` server component + `discover-client.tsx`),
|
||||
linked in the nav in `layout.tsx`:
|
||||
|
||||
- **"Discover now"** → `POST /api/discover/run` (sets `discover.requested`);
|
||||
shows `discover.result`.
|
||||
- **Seed-search box** → `GET /api/discover/search?artist=<name>`: resolve the name
|
||||
to an MBID (MB search), fetch ListenBrainz similar, return **unpersisted**
|
||||
results inline; each is actionable via the same Follow/Want endpoints.
|
||||
- **Feed** → `GET /api/discover`: pending suggestions, artists and albums in two
|
||||
sections, sorted by `score` desc, each showing *why* (`seedCount`, `sources`,
|
||||
`score`) and its actions.
|
||||
|
||||
Actions — **reuse existing slice-2 DB operations, no new grab logic:**
|
||||
|
||||
- **Follow** (artist) → create `WatchedArtist` + browse its discography into
|
||||
`monitored=false` `MonitoredRelease` rows (the same operation as the
|
||||
`/api/artists` follow POST), then set the suggestion `status='followed'`.
|
||||
- **Want** (album) → create a `monitored=true` `MonitoredRelease` **directly from
|
||||
the suggestion's already-stored fields** (`rgMbid`, `album`, types,
|
||||
`firstReleaseDate`) — no redundant MusicBrainz search, since the sweep already
|
||||
resolved them — equivalent to what the wanted list stores; set
|
||||
`status='wanted'`.
|
||||
- **Dismiss** → set `status='dismissed'`.
|
||||
|
||||
Endpoints (each with a co-located `.test.ts`, matching convention):
|
||||
`api/discover/route.ts` (feed), `api/discover/run/route.ts`,
|
||||
`api/discover/search/route.ts`, `api/discover/[id]/route.ts` (action).
|
||||
|
||||
**Settings** — a "Discovery" section in `settings-form.tsx` mirroring the monitor
|
||||
fields: `enabled` + the tuning knobs above.
|
||||
|
||||
## Error handling
|
||||
|
||||
| Failure | Behavior |
|
||||
|---|---|
|
||||
| ListenBrainz unreachable during sweep | source `health()` false ⇒ that source contributes nothing this tick; logged; retried next sweep. Web seed-search returns an error surfaced in the UI. |
|
||||
| A single seed errors | logged and skipped; the rest of the sweep proceeds. |
|
||||
| Duplicate suggestion | `dedupeKey` unique + `ON CONFLICT`; re-sweeping never duplicates rows. |
|
||||
| Candidate already followed / in library | filtered out before insert (checked against `WatchedArtist` / `LibraryItem` / `MonitoredRelease`). |
|
||||
| Dismissed candidate reappears from source | conditional upsert leaves the dismissed row untouched; it stays out of the feed. |
|
||||
| Follow/Want race with existing row | reuse slice-2 operations, which already upsert idempotently. |
|
||||
| Discovery run throws | caught in the loop, `conn.rollback()`, logged; worker keeps running. |
|
||||
|
||||
## Testing (against `lyra_test`, per the DB guard)
|
||||
|
||||
- **Worker — `discovery.py`:** fake `SimilaritySource` + seeded Postgres. Assert
|
||||
aggregation (score sum, seedCount, sources union), filtering vs `WatchedArtist`
|
||||
/ `LibraryItem` / `MonitoredRelease`, `minScore` cutoff, the **conditional
|
||||
upsert never reviving a dismissed row**, album derivation through the
|
||||
core-release predicate, and `lastDiscoveredAt` throttling.
|
||||
- **Worker — `_listenbrainz.py`:** thin adapter tested with faked HTTP against
|
||||
recorded response shapes; `health()` true/false paths.
|
||||
- **Web:** route tests for the feed, each action (Follow/Want/Dismiss with mocked
|
||||
DB), and seed-search (mocked MB + ListenBrainz). Reuse the `release-filter`
|
||||
tests for the core-release predicate.
|
||||
|
||||
## Notes / deferred
|
||||
|
||||
- **Second similarity source (Last.fm) and tag-coherent re-weighting** are the
|
||||
planned extensions behind `SimilaritySource`; adding either is a new source in
|
||||
the registry + a `sources`-array boost, no core change.
|
||||
- **Album popularity ranking.** v1 picks the most-recent core release per artist.
|
||||
A later refinement can rank an artist's release-groups by ListenBrainz
|
||||
popular-recordings listen counts.
|
||||
- **Seed scope** is currently all `WatchedArtist` MBIDs. A "top artists only"
|
||||
scoping (by library album count) is a possible knob if API load becomes an issue.
|
||||
- **Discovery is synchronous in the worker loop**, like the scan — fine for the
|
||||
weekly cadence and bounded by `maxSeeds`, but it briefly blocks job-claim during
|
||||
a sweep. Chunking mirrors the scan's deferred item.
|
||||
Reference in New Issue
Block a user