Lyra/docs/superpowers/specs/2026-07-11-lyra-discovery-design.md

# Lyra — Discovery (Design)

**Date:** 2026-07-11
**Status:** Approved design, ready for implementation planning
**Slice:** 3 of 3 (Discovery)

## Overview

Slice 3 makes Lyra surface music you **don't already follow**. Slices 1–2
obtain and monitor artists/releases you already know about; discovery closes the
loop by recommending *new* artists and albums similar to what's in your library,
then hands each accepted suggestion straight back into the slice-2 machinery.

Discovery is deliberately **suggestion-only and manual**: a background sweep (and
an on-demand button) populates a review feed of candidate artists and albums; you
**Follow** an artist (creating a `WatchedArtist` with its discography tracked but
**unmonitored**, exactly like the slice-2 follow) or **Want** an album (creating a
`monitored` wanted `MonitoredRelease`, exactly like the slice-2 wanted list), or
**Dismiss** it. Discovery never grabs anything itself — its only outputs are the
exact rows the existing follow / wanted flows already create, and actual grabbing
still happens only through the slice-2 monitor (off by default). The entire
acquisition + monitoring pipeline is reused unchanged.

### The source reality that shapes this slice

The original acquisition-engine spec reserved **Spotify** for "discovery/metadata
only." That premise no longer holds for a newly-registered app:

- **Nov 2024** — Spotify deprecated **Related Artists**, **Recommendations**,
  audio-features, audio-analysis and featured-playlists for any *new* app (new
  apps receive 403/404; no waitlist, no path back).
- **Feb 2026** — Spotify additionally removed **Get New Releases**, Get Artist's
  Top Tracks and Get Several Albums.

So Spotify can supply **none** of discovery's signals to a fresh app. Lyra is
already **MusicBrainz-native** (every artist/release is an MBID), so discovery is
instead built on **MetaBrainz**: **ListenBrainz similar-artists** (same family as
MusicBrainz, returns MBIDs directly — zero fuzzy matching) as the engine, with
MusicBrainz for the album sweep. Spotify is dropped entirely from this slice.

### Goals

- Recommend new **artists** (and standout **albums**) similar to the library, at
  MBID granularity, with no personal-account / listening-history dependency.
- Reuse slice-2 verbatim: **Follow** creates the same `WatchedArtist` +
  unmonitored discography the follow flow does; **Want** creates the same
  `monitored` wanted `MonitoredRelease` the wanted list does. No new grab logic.
- Manual review only — nothing is followed or grabbed without explicit approval;
  **dismissed suggestions never resurface**.
- Keep the design source-agnostic behind a pluggable `SimilaritySource` interface
  (mirroring the slice-1 `adapters/` registry), so a second source (Last.fm) or
  tag-weighting can drop in later without touching the core.
- Fully developable and testable with **zero credentials** (ListenBrainz labs API
  is unauthenticated) via a fake `SimilaritySource` and seeded Postgres.

### Non-goals (this slice)

- **No Spotify** anywhere (its similarity/new-release APIs are dead for new apps).
- **No auto-follow / auto-grab** — manual review only.
- **No personal listening data / OAuth** (no ListenBrainz user account required).
- **No second similarity source (Last.fm) and no genre-tag re-weighting** in this
  slice — both are left as drop-in extensions behind `SimilaritySource`.
- No notifications; no multi-user. Downloading, tagging, and the six-stage
  pipeline are untouched.

## Scope

**In scope:** a `SimilaritySource` engine (ListenBrainz), a worker discovery run
that aggregates similar artists across your library's artists and derives album
suggestions, a `DiscoverySuggestion` review feed, per-suggestion Follow / Want /
Dismiss actions that reuse slice-2 operations, an interactive seed-search box, a
scheduled sweep + "Discover now" trigger (off by default), and Discovery settings.

**Out of scope:** everything under Non-goals. Deferred hardening from slices 1–2
is unaffected and remains deferred.

## Key decisions

- **Web reads, worker persists** — the same split slice 2 established. The
  interactive **seed-search box** calls ListenBrainz directly from the web
  (unauthenticated, read-only, results not persisted), exactly as `/api/mb/artists`
  does live MusicBrainz search today. The **scheduled sweep** and **"Discover
  now"** run in the worker and persist `DiscoverySuggestion` rows.
- **Suggestion-only, manual.** Discovery emits suggestions; the *user* turns a
  suggestion into a `WatchedArtist` (Follow) or a wanted `MonitoredRelease`
  (Want). Discovery has no path that grabs audio.
- **Dismissed is permanent.** A `dedupeKey` unique column plus keeping
  dismissed/followed/wanted rows means the sweep's conditional upsert can never
  revert them to `pending` — dismissed suggestions never come back.
- **Off by default**, like the monitor: `discover.enabled` unset ⇒ no background
  sweep. The on-demand button and seed-search still work regardless.
- **Pluggable engine.** One source (ListenBrainz) ships now behind a
  `SimilaritySource` Protocol + `build_similarity_sources()` registry, matching
  the existing `SourceAdapter` / `build_adapters` pattern.

## Architecture

```
                       WEB (read-only, no secrets)          WORKER (persists)
  /discover page  ─┬─ GET /api/discover        (feed) ───▶ DiscoverySuggestion (read)
                   ├─ GET /api/discover/search  ──────────▶ MB search + ListenBrainz
                   │        (interactive seed, not stored)   (direct, unauthenticated)
                   ├─ POST /api/discover/run  ────────────▶ Config discover.requested=true
                   └─ POST /api/discover/[id] (action) ───▶ Follow → WatchedArtist
                                                            Want   → MonitoredRelease
                                                            Dismiss→ status=dismissed

  worker loop tick (discover.enabled, every discover.intervalHours; or on
  discover.requested):
    run_discovery(conn, sources, browser, cfg):
      seeds = WatchedArtist.mbid  (throttled by lastDiscoveredAt, capped maxSeeds)
      for seed × source:  similar_artists(mbid)  → aggregate
      filter out already-followed / dismissed
      upsert ARTIST suggestions (conditional on status='pending')
      for top artists: MbBrowser release-groups → core-release filter →
                       drop library dupes → upsert ALBUM suggestions
```

## Data model

One new table + two enums, plus one column on `WatchedArtist`.

```prisma
enum DiscoveryKind    { artist album }
enum SuggestionStatus { pending followed wanted dismissed }

model DiscoverySuggestion {
  id               String   @id @default(cuid())
  kind             DiscoveryKind
  artistMbid       String
  artistName       String
  rgMbid           String?          // album kind only; null for artist
  album            String?
  primaryType      String?
  secondaryTypes   String[]
  firstReleaseDate String?
  score            Float            // aggregated similarity across seeds
  seedCount        Int              // how many of your artists pointed here
  sources          String[]         // e.g. ["listenbrainz"] (pluggable)
  status           SuggestionStatus @default(pending)
  dedupeKey        String   @unique // kind + ':' + artistMbid + ':' + (rgMbid ?? "")
  createdAt        DateTime @default(now())
  updatedAt        DateTime @updatedAt
}
```

Plus on `WatchedArtist`:

```prisma
  lastDiscoveredAt DateTime?   // per-seed throttle, mirrors lastPolledAt
```

**Why `dedupeKey`:** Postgres treats `NULL`s as distinct in a composite unique
index, so a `@@unique([kind, artistMbid, rgMbid])` would let duplicate
`kind=artist` rows through (their `rgMbid` is null). A single non-null
`dedupeKey` string gives a clean `ON CONFLICT` target for the upsert.

**Conditional upsert (the dismissed-is-permanent guarantee):**

```sql
INSERT INTO "DiscoverySuggestion" (... , status, "dedupeKey", ...)
VALUES (... , 'pending', %s, ...)
ON CONFLICT ("dedupeKey") DO UPDATE
  SET score = EXCLUDED.score,
      "seedCount" = EXCLUDED."seedCount",
      sources = EXCLUDED.sources,
      "updatedAt" = now()
WHERE "DiscoverySuggestion".status = 'pending';
```

A row already `followed` / `wanted` / `dismissed` fails the `WHERE` and is left
untouched — it never returns to the feed.

## Engine — `worker/lyra_worker/similarity/`

Mirrors `worker/lyra_worker/adapters/`.

- **`base.py`** — Protocol:
  ```python
  @dataclass(frozen=True)
  class SimilarArtist:
      mbid: str
      name: str
      score: float

  @runtime_checkable
  class SimilaritySource(Protocol):
      name: str
      def health(self) -> bool: ...
      def similar_artists(self, mbid: str) -> list[SimilarArtist]: ...
  ```
- **`_listenbrainz.py`** — `ListenBrainzSource`: GETs the ListenBrainz Labs
  similar-artists dataset (`labs.api.listenbrainz.org`, unauthenticated), maps the
  response to `SimilarArtist` (MBIDs come back native). `health()` is a
  reachability check. A base-URL override reads from config
  (`discover.listenBrainzUrl`, default the public host).
- **`registry.build_similarity_sources(config)`** — health-filtered list, exactly
  like `build_adapters`. `build_similarity_sources` is added to `registry.py`.

## Discovery run — `worker/lyra_worker/discovery.py`

`DiscoveryConfig.from_config(config)` parses the `discover.*` keys (same shape as
`MonitorConfig`). `run_discovery(conn, sources, browser, cfg) -> DiscoveryResult`:

1. **Seeds** = `WatchedArtist.mbid` where `lastDiscoveredAt IS NULL OR <
   now() - intervalHours`, capped at `maxSeeds`.
2. For each seed × source → `similar_artists(mbid)`, keep top `similarPerSeed`.
3. **Aggregate** per candidate MBID: `score = Σ similarity`, `seedCount =`
   distinct seeds pointing here, `sources =` union of contributing source names.
4. **Filter** candidates already in `WatchedArtist` (already followed) and any
   existing dismissed row; drop below `minScore`.
5. **Upsert artist suggestions** (conditional-on-pending, above).
6. **Album derivation:** for the top surfaced artists, use the existing
   **`MbBrowser`** to list their release-groups, keep only **core releases** via
   the existing `release-filter` predicate (reused from slice 2), drop any already
   in `LibraryItem` or `MonitoredRelease`, and upsert up to `albumsPerArtist`
   (default 1, most-recent core) **album** suggestions.
7. Stamp `lastDiscoveredAt = now()` on each processed seed. Return counts.

Errors from a single source or seed are logged and skipped — a discovery failure
must never kill the worker loop (same discipline as monitor/scan).

## Worker loop integration — `worker/lyra_worker/main.py`

- Build similarity sources at startup: `sources = build_similarity_sources(config)`.
- Add a **discover tick** beside the monitor tick: when `discover.enabled` and the
  interval has elapsed, call `run_discovery(...)` inside a try/except that rolls
  back and logs on failure. Off by default.
- Handle the one-shot **`discover.requested`** flag exactly like `scan.requested`:
  run `run_discovery`, write `discover.result` (`"artists N, albums M"`), clear
  `discover.requested`.

## Config keys (`discover.*`, mirroring `monitor.*`)

| Key | Default | Meaning |
|---|---|---|
| `discover.enabled` | `false` | master switch for the scheduled sweep |
| `discover.intervalHours` | `168` | sweep cadence (weekly) + per-seed throttle window |
| `discover.maxSeeds` | `50` | seed artists processed per sweep |
| `discover.similarPerSeed` | `20` | top-K similar artists pulled per seed |
| `discover.albumsPerArtist` | `1` | album suggestions per surfaced artist |
| `discover.minScore` | `0` | drop candidates below this aggregated score |
| `discover.listenBrainzUrl` | public host | ListenBrainz Labs base URL override |
| `discover.requested` | — | one-shot "Discover now" trigger (worker clears) |
| `discover.result` | — | last-run status string surfaced in the UI |

No credential key — the ListenBrainz Labs API is unauthenticated.

## Web

New page **`/discover`** (`page.tsx` server component + `discover-client.tsx`),
linked in the nav in `layout.tsx`:

- **"Discover now"** → `POST /api/discover/run` (sets `discover.requested`);
  shows `discover.result`.
- **Seed-search box** → `GET /api/discover/search?artist=<name>`: resolve the name
  to an MBID (MB search), fetch ListenBrainz similar, return **unpersisted**
  results inline; each is actionable via the same Follow/Want endpoints.
- **Feed** → `GET /api/discover`: pending suggestions, artists and albums in two
  sections, sorted by `score` desc, each showing *why* (`seedCount`, `sources`,
  `score`) and its actions.

Actions — **reuse existing slice-2 DB operations, no new grab logic:**

- **Follow** (artist) → create `WatchedArtist` + browse its discography into
  `monitored=false` `MonitoredRelease` rows (the same operation as the
  `/api/artists` follow POST), then set the suggestion `status='followed'`.
- **Want** (album) → create a `monitored=true` `MonitoredRelease` **directly from
  the suggestion's already-stored fields** (`rgMbid`, `album`, types,
  `firstReleaseDate`) — no redundant MusicBrainz search, since the sweep already
  resolved them — equivalent to what the wanted list stores; set
  `status='wanted'`.
- **Dismiss** → set `status='dismissed'`.

Endpoints (each with a co-located `.test.ts`, matching convention):
`api/discover/route.ts` (feed), `api/discover/run/route.ts`,
`api/discover/search/route.ts`, `api/discover/[id]/route.ts` (action).

**Settings** — a "Discovery" section in `settings-form.tsx` mirroring the monitor
fields: `enabled` + the tuning knobs above.

## Error handling

| Failure | Behavior |
|---|---|
| ListenBrainz unreachable during sweep | source `health()` false ⇒ that source contributes nothing this tick; logged; retried next sweep. Web seed-search returns an error surfaced in the UI. |
| A single seed errors | logged and skipped; the rest of the sweep proceeds. |
| Duplicate suggestion | `dedupeKey` unique + `ON CONFLICT`; re-sweeping never duplicates rows. |
| Candidate already followed / in library | filtered out before insert (checked against `WatchedArtist` / `LibraryItem` / `MonitoredRelease`). |
| Dismissed candidate reappears from source | conditional upsert leaves the dismissed row untouched; it stays out of the feed. |
| Follow/Want race with existing row | reuse slice-2 operations, which already upsert idempotently. |
| Discovery run throws | caught in the loop, `conn.rollback()`, logged; worker keeps running. |

## Testing (against `lyra_test`, per the DB guard)

- **Worker — `discovery.py`:** fake `SimilaritySource` + seeded Postgres. Assert
  aggregation (score sum, seedCount, sources union), filtering vs `WatchedArtist`
  / `LibraryItem` / `MonitoredRelease`, `minScore` cutoff, the **conditional
  upsert never reviving a dismissed row**, album derivation through the
  core-release predicate, and `lastDiscoveredAt` throttling.
- **Worker — `_listenbrainz.py`:** thin adapter tested with faked HTTP against
  recorded response shapes; `health()` true/false paths.
- **Web:** route tests for the feed, each action (Follow/Want/Dismiss with mocked
  DB), and seed-search (mocked MB + ListenBrainz). Reuse the `release-filter`
  tests for the core-release predicate.

## Notes / deferred

- **Second similarity source (Last.fm) and tag-coherent re-weighting** are the
  planned extensions behind `SimilaritySource`; adding either is a new source in
  the registry + a `sources`-array boost, no core change.
- **Album popularity ranking.** v1 picks the most-recent core release per artist.
  A later refinement can rank an artist's release-groups by ListenBrainz
  popular-recordings listen counts.
- **Seed scope** is currently all `WatchedArtist` MBIDs. A "top artists only"
  scoping (by library album count) is a possible knob if API load becomes an issue.
- **Discovery is synchronous in the worker loop**, like the scan — fine for the
  weekly cadence and bounded by `maxSeeds`, but it briefly blocks job-claim during
  a sweep. Chunking mirrors the scan's deferred item.