Lyra/docs/superpowers/specs/2026-07-11-lyra-discovery-design.md

Lyra — Discovery (Design)

Date: 2026-07-11 Status: Approved design, ready for implementation planning Slice: 3 of 3 (Discovery)

Overview

Slice 3 makes Lyra surface music you don't already follow. Slices 1–2 obtain and monitor artists/releases you already know about; discovery closes the loop by recommending new artists and albums similar to what's in your library, then hands each accepted suggestion straight back into the slice-2 machinery.

Discovery is deliberately suggestion-only and manual: a background sweep (and an on-demand button) populates a review feed of candidate artists and albums; you Follow an artist (creating a WatchedArtist with its discography tracked but unmonitored, exactly like the slice-2 follow) or Want an album (creating a monitored wanted MonitoredRelease, exactly like the slice-2 wanted list), or Dismiss it. Discovery never grabs anything itself — its only outputs are the exact rows the existing follow / wanted flows already create, and actual grabbing still happens only through the slice-2 monitor (off by default). The entire acquisition + monitoring pipeline is reused unchanged.

The source reality that shapes this slice

The original acquisition-engine spec reserved Spotify for "discovery/metadata only." That premise no longer holds for a newly-registered app:

• Nov 2024 — Spotify deprecated Related Artists, Recommendations, audio-features, audio-analysis and featured-playlists for any new app (new apps receive 403/404; no waitlist, no path back).
• Feb 2026 — Spotify additionally removed Get New Releases, Get Artist's Top Tracks and Get Several Albums.

So Spotify can supply none of discovery's signals to a fresh app. Lyra is already MusicBrainz-native (every artist/release is an MBID), so discovery is instead built on MetaBrainz: ListenBrainz similar-artists (same family as MusicBrainz, returns MBIDs directly — zero fuzzy matching) as the engine, with MusicBrainz for the album sweep. Spotify is dropped entirely from this slice.

Goals

• Recommend new artists (and standout albums) similar to the library, at MBID granularity, with no personal-account / listening-history dependency.
• Reuse slice-2 verbatim: Follow creates the same WatchedArtist + unmonitored discography the follow flow does; Want creates the same monitored wanted MonitoredRelease the wanted list does. No new grab logic.
• Manual review only — nothing is followed or grabbed without explicit approval; dismissed suggestions never resurface.
• Keep the design source-agnostic behind a pluggable SimilaritySource interface (mirroring the slice-1 adapters/ registry), so a second source (Last.fm) or tag-weighting can drop in later without touching the core.
• Fully developable and testable with zero credentials (ListenBrainz labs API is unauthenticated) via a fake SimilaritySource and seeded Postgres.

Non-goals (this slice)

• No Spotify anywhere (its similarity/new-release APIs are dead for new apps).
• No auto-follow / auto-grab — manual review only.
• No personal listening data / OAuth (no ListenBrainz user account required).
• No second similarity source (Last.fm) and no genre-tag re-weighting in this slice — both are left as drop-in extensions behind SimilaritySource.
• No notifications; no multi-user. Downloading, tagging, and the six-stage pipeline are untouched.

Scope

In scope: a SimilaritySource engine (ListenBrainz), a worker discovery run that aggregates similar artists across your library's artists and derives album suggestions, a DiscoverySuggestion review feed, per-suggestion Follow / Want / Dismiss actions that reuse slice-2 operations, an interactive seed-search box, a scheduled sweep + "Discover now" trigger (off by default), and Discovery settings.

Out of scope: everything under Non-goals. Deferred hardening from slices 1–2 is unaffected and remains deferred.

Key decisions

• Web reads, worker persists — the same split slice 2 established. The interactive seed-search box calls ListenBrainz directly from the web (unauthenticated, read-only, results not persisted), exactly as /api/mb/artists does live MusicBrainz search today. The scheduled sweep and "Discover now" run in the worker and persist DiscoverySuggestion rows.
• Suggestion-only, manual. Discovery emits suggestions; the user turns a suggestion into a WatchedArtist (Follow) or a wanted MonitoredRelease (Want). Discovery has no path that grabs audio.
• Dismissed is permanent. A dedupeKey unique column plus keeping dismissed/followed/wanted rows means the sweep's conditional upsert can never revert them to pending — dismissed suggestions never come back.
• Off by default, like the monitor: discover.enabled unset ⇒ no background sweep. The on-demand button and seed-search still work regardless.
• Pluggable engine. One source (ListenBrainz) ships now behind a SimilaritySource Protocol + build_similarity_sources() registry, matching the existing SourceAdapter / build_adapters pattern.

Architecture

                       WEB (read-only, no secrets)          WORKER (persists)
  /discover page  ─┬─ GET /api/discover        (feed) ───▶ DiscoverySuggestion (read)
                   ├─ GET /api/discover/search  ──────────▶ MB search + ListenBrainz
                   │        (interactive seed, not stored)   (direct, unauthenticated)
                   ├─ POST /api/discover/run  ────────────▶ Config discover.requested=true
                   └─ POST /api/discover/[id] (action) ───▶ Follow → WatchedArtist
                                                            Want   → MonitoredRelease
                                                            Dismiss→ status=dismissed

  worker loop tick (discover.enabled, every discover.intervalHours; or on
  discover.requested):
    run_discovery(conn, sources, browser, cfg):
      seeds = WatchedArtist.mbid  (throttled by lastDiscoveredAt, capped maxSeeds)
      for seed × source:  similar_artists(mbid)  → aggregate
      filter out already-followed / dismissed
      upsert ARTIST suggestions (conditional on status='pending')
      for top artists: MbBrowser release-groups → core-release filter →
                       drop library dupes → upsert ALBUM suggestions

Data model

One new table + two enums, plus one column on WatchedArtist.

enum DiscoveryKind    { artist album }
enum SuggestionStatus { pending followed wanted dismissed }

model DiscoverySuggestion {
  id               String   @id @default(cuid())
  kind             DiscoveryKind
  artistMbid       String
  artistName       String
  rgMbid           String?          // album kind only; null for artist
  album            String?
  primaryType      String?
  secondaryTypes   String[]
  firstReleaseDate String?
  score            Float            // aggregated similarity across seeds
  seedCount        Int              // how many of your artists pointed here
  sources          String[]         // e.g. ["listenbrainz"] (pluggable)
  status           SuggestionStatus @default(pending)
  dedupeKey        String   @unique // kind + ':' + artistMbid + ':' + (rgMbid ?? "")
  createdAt        DateTime @default(now())
  updatedAt        DateTime @updatedAt
}

Plus on WatchedArtist:

  lastDiscoveredAt DateTime?   // per-seed throttle, mirrors lastPolledAt

Why dedupeKey: Postgres treats NULLs as distinct in a composite unique index, so a @@unique([kind, artistMbid, rgMbid]) would let duplicate kind=artist rows through (their rgMbid is null). A single non-null dedupeKey string gives a clean ON CONFLICT target for the upsert.

Conditional upsert (the dismissed-is-permanent guarantee):

INSERT INTO "DiscoverySuggestion" (... , status, "dedupeKey", ...)
VALUES (... , 'pending', %s, ...)
ON CONFLICT ("dedupeKey") DO UPDATE
  SET score = EXCLUDED.score,
      "seedCount" = EXCLUDED."seedCount",
      sources = EXCLUDED.sources,
      "updatedAt" = now()
WHERE "DiscoverySuggestion".status = 'pending';

A row already followed / wanted / dismissed fails the WHERE and is left untouched — it never returns to the feed.

Engine — worker/lyra_worker/similarity/

Mirrors worker/lyra_worker/adapters/.

• base.py — Protocol:

  @dataclass(frozen=True)
  class SimilarArtist:
      mbid: str
      name: str
      score: float
  
  @runtime_checkable
  class SimilaritySource(Protocol):
      name: str
      def health(self) -> bool: ...
      def similar_artists(self, mbid: str) -> list[SimilarArtist]: ...
  

• _listenbrainz.py — ListenBrainzSource: GETs the ListenBrainz Labs similar-artists dataset (labs.api.listenbrainz.org, unauthenticated), maps the response to SimilarArtist (MBIDs come back native). health() is a reachability check. A base-URL override reads from config (discover.listenBrainzUrl, default the public host).
• registry.build_similarity_sources(config) — health-filtered list, exactly like build_adapters. build_similarity_sources is added to registry.py.

Discovery run — worker/lyra_worker/discovery.py

DiscoveryConfig.from_config(config) parses the discover.* keys (same shape as MonitorConfig). run_discovery(conn, sources, browser, cfg) -> DiscoveryResult:

1. Seeds = WatchedArtist.mbid where lastDiscoveredAt IS NULL OR < now() - intervalHours, capped at maxSeeds.
2. For each seed × source → similar_artists(mbid), keep top similarPerSeed.
3. Aggregate per candidate MBID: score = Σ similarity, seedCount = distinct seeds pointing here, sources = union of contributing source names.
4. Filter candidates already in WatchedArtist (already followed) and any existing dismissed row; drop below minScore.
5. Upsert artist suggestions (conditional-on-pending, above).
6. Album derivation: for the top surfaced artists, use the existing MbBrowser to list their release-groups, keep only core releases via the existing release-filter predicate (reused from slice 2), drop any already in LibraryItem or MonitoredRelease, and upsert up to albumsPerArtist (default 1, most-recent core) album suggestions.
7. Stamp lastDiscoveredAt = now() on each processed seed. Return counts.

Errors from a single source or seed are logged and skipped — a discovery failure must never kill the worker loop (same discipline as monitor/scan).

Worker loop integration — worker/lyra_worker/main.py

• Build similarity sources at startup: sources = build_similarity_sources(config).
• Add a discover tick beside the monitor tick: when discover.enabled and the interval has elapsed, call run_discovery(...) inside a try/except that rolls back and logs on failure. Off by default.
• Handle the one-shot discover.requested flag exactly like scan.requested: run run_discovery, write discover.result ("artists N, albums M"), clear discover.requested.

Config keys (discover.*, mirroring monitor.*)

Key	Default	Meaning
discover.enabled	false	master switch for the scheduled sweep
discover.intervalHours	168	sweep cadence (weekly) + per-seed throttle window
discover.maxSeeds	50	seed artists processed per sweep
discover.similarPerSeed	20	top-K similar artists pulled per seed
discover.albumsPerArtist	1	album suggestions per surfaced artist
discover.minScore	0	drop candidates below this aggregated score
discover.listenBrainzUrl	public host	ListenBrainz Labs base URL override
discover.requested	—	one-shot "Discover now" trigger (worker clears)
discover.result	—	last-run status string surfaced in the UI

No credential key — the ListenBrainz Labs API is unauthenticated.

Web

New page /discover (page.tsx server component + discover-client.tsx), linked in the nav in layout.tsx:

• "Discover now" → POST /api/discover/run (sets discover.requested); shows discover.result.
• Seed-search box → GET /api/discover/search?artist=<name>: resolve the name to an MBID (MB search), fetch ListenBrainz similar, return unpersisted results inline; each is actionable via the same Follow/Want endpoints.
• Feed → GET /api/discover: pending suggestions, artists and albums in two sections, sorted by score desc, each showing why (seedCount, sources, score) and its actions.

Actions — reuse existing slice-2 DB operations, no new grab logic:

• Follow (artist) → create WatchedArtist + browse its discography into monitored=false MonitoredRelease rows (the same operation as the /api/artists follow POST), then set the suggestion status='followed'.
• Want (album) → create a monitored=true MonitoredRelease directly from the suggestion's already-stored fields (rgMbid, album, types, firstReleaseDate) — no redundant MusicBrainz search, since the sweep already resolved them — equivalent to what the wanted list stores; set status='wanted'.
• Dismiss → set status='dismissed'.

Endpoints (each with a co-located .test.ts, matching convention): api/discover/route.ts (feed), api/discover/run/route.ts, api/discover/search/route.ts, api/discover/[id]/route.ts (action).

Settings — a "Discovery" section in settings-form.tsx mirroring the monitor fields: enabled + the tuning knobs above.

Error handling

Failure	Behavior
ListenBrainz unreachable during sweep	source health() false ⇒ that source contributes nothing this tick; logged; retried next sweep. Web seed-search returns an error surfaced in the UI.
A single seed errors	logged and skipped; the rest of the sweep proceeds.
Duplicate suggestion	dedupeKey unique + ON CONFLICT; re-sweeping never duplicates rows.
Candidate already followed / in library	filtered out before insert (checked against WatchedArtist / LibraryItem / MonitoredRelease).
Dismissed candidate reappears from source	conditional upsert leaves the dismissed row untouched; it stays out of the feed.
Follow/Want race with existing row	reuse slice-2 operations, which already upsert idempotently.
Discovery run throws	caught in the loop, conn.rollback(), logged; worker keeps running.

Testing (against lyra_test, per the DB guard)

• Worker — discovery.py: fake SimilaritySource + seeded Postgres. Assert aggregation (score sum, seedCount, sources union), filtering vs WatchedArtist / LibraryItem / MonitoredRelease, minScore cutoff, the conditional upsert never reviving a dismissed row, album derivation through the core-release predicate, and lastDiscoveredAt throttling.
• Worker — _listenbrainz.py: thin adapter tested with faked HTTP against recorded response shapes; health() true/false paths.
• Web: route tests for the feed, each action (Follow/Want/Dismiss with mocked DB), and seed-search (mocked MB + ListenBrainz). Reuse the release-filter tests for the core-release predicate.

Notes / deferred

• Second similarity source (Last.fm) and tag-coherent re-weighting are the planned extensions behind SimilaritySource; adding either is a new source in the registry + a sources-array boost, no core change.
• Album popularity ranking. v1 picks the most-recent core release per artist. A later refinement can rank an artist's release-groups by ListenBrainz popular-recordings listen counts.
• Seed scope is currently all WatchedArtist MBIDs. A "top artists only" scoping (by library album count) is a possible knob if API load becomes an issue.
• Discovery is synchronous in the worker loop, like the scan — fine for the weekly cadence and bounded by maxSeeds, but it briefly blocks job-claim during a sweep. Chunking mirrors the scan's deferred item.