Lyra/docs/superpowers/plans/2026-07-14-musicbrainz-cache.md

# MusicBrainz Response Cache (Postgres) — Implementation Plan

Sketched 2026-07-14. A Postgres-backed read-through cache shared by the web app and the
worker, so reopening an artist (discography) or album (tracklist) — and repeat lookups
during scans/discovery — stop re-hitting MusicBrainz. **Postgres only; Redis explicitly
disregarded** (Postgres is already running and shared across web(TS) + worker(Python);
Next's Data Cache is per-instance and the Python worker can't see it, so a shared store is
required and we already have one — no new container/ops for a single-user app). Redis
revisited only if profiling later shows PG cache lookups are a bottleneck (won't be at this
scale) or pub/sub is wanted.

## Context
Both callers hit MB live with **no cache and no throttle**: web `mbGet`
(`web/src/lib/musicbrainz.ts` — bare `fetch`) and worker `_mbbrowser.py`
(`musicbrainzngs`). Reopen an artist → full discography re-fetched; reopen an album →
tracklist re-fetched; search-as-you-type fires a request per keystroke. MB data changes on
the order of days/weeks, so hours-to-days TTLs are safe. Caching also helps stay under MB's
~1 req/s anonymous rate limit (today only the worker scan paces ~1/s; the web side is
unthrottled).

## Confirmed decisions
- **Postgres, not Redis** (rationale above).
- **Key on the LOGICAL operation, not the request URL.** Web uses raw `ws/2` REST paths;
  the worker uses `musicbrainzngs`, which builds its own requests — the two never produce
  the same URL, so URL-keyed caches would never share. Logical keys let a discography the
  worker fetched during a scan warm the web modal (and vice versa). Cost: wrap the
  **high-level** functions, not the low-level fetch.
- **TTL applied at read time from `fetchedAt`** (no `expiresAt` column) so TTLs are tunable
  in code with no migration.
- **Serve stale on MB outage** — if a live fetch throws and a (stale) row exists, return it.
- **TTL-only invalidation** is sufficient for a home app; no explicit invalidation required
  for v1.

## Cross-cutting facts
- Prisma schema `web/prisma/schema.prisma`; migrations reach live `lyra` via the web
  entrypoint's `migrate deploy` on `docker compose up -d --build` (same as every prior
  migration). Postgres `Json` maps to JSONB.
- The worker already threads a `conn` (psycopg) through `maybe_run_scan` / `scan_chunk` /
  `_run_discovery` (`main.py`), and `MusicBrainzBrowser` is built in `registry.build_browser()`
  / used by `LastfmSource`. A conn is available at the monitor/scan call sites.
- Key namespaces (shared verbatim between TS and Python):
  `rg-tracks:{rgMbid}`, `artist-rgs:{artistMbid}`, `artist-name:{artistMbid}`,
  `rg-search:{artist}|{album}`, `artist-search:{query}`.
- Suggested TTLs (all tunable, no migration): tracklists / discography / artist-name **30d**
  (near-immutable), `searchReleaseGroup` **7d**, `searchArtists` type-ahead **1d** (or leave
  uncached). Normalize `rg-search`/`artist-search` keys (trim + lowercase) so casing doesn't
  fragment the cache.
- Tests run against `lyra_test` (guard blocks the live DB); both suites are DB + pure logic,
  no MB network in tests.

## Task 1 — schema + migration
- `web/prisma/schema.prisma` += model:
  ```prisma
  model MbCache {
    key       String   @id
    json      Json
    fetchedAt DateTime @default(now())
  }
  ```
- `npx prisma migrate dev --name add_mb_cache` (dev/`lyra_test`). Ships to live via
  `migrate deploy` on rebuild. No `expiresAt`.
- Verify: `prisma migrate status` clean; table present.

## Task 2 — web read-through (biggest UX win, smallest surface)
- New `web/src/lib/mbcache.ts`:
  ```ts
  import { prisma } from "@/lib/db";
  const DAY = 86_400;
  export async function cachedMb<T>(key: string, ttlSec: number, fetchFresh: () => Promise<T>): Promise<T> {
    const hit = await prisma.mbCache.findUnique({ where: { key } });
    if (hit && (Date.now() - hit.fetchedAt.getTime()) / 1000 < ttlSec) return hit.json as T;
    try {
      const fresh = await fetchFresh();
      await prisma.mbCache.upsert({
        where: { key },
        create: { key, json: fresh as object },
        update: { json: fresh as object, fetchedAt: new Date() },
      });
      return fresh;
    } catch (e) {
      if (hit) return hit.json as T; // serve stale on MB outage
      throw e;
    }
  }
  ```
- Wrap the high-level fns in `web/src/lib/musicbrainz.ts` with logical keys + TTLs:
  `browseReleaseGroupTracks` (`rg-tracks:{rgMbid}`, 30d), `browse discography`
  (`artist-rgs:{artistMbid}`, 30d), `getArtistName` (`artist-name:{artistMbid}`, 30d),
  `searchReleaseGroup` (`rg-search:{artist}|{album}` normalized, 7d), `searchArtists`
  (`artist-search:{query}` normalized, 1d — or skip). Keep each fn's existing MB-mapping
  logic inside the `fetchFresh` closure so cached values are the already-shaped objects.
- **Negative results:** `searchReleaseGroup` can return `null` (no match). Decide: cache
  `null` briefly (e.g. 1d) to avoid re-hammering on edition-qualified misses (ties into the
  Last.fm "No MusicBrainz match" todo), OR don't cache nulls. Recommend caching `null` with a
  short TTL — but only AFTER that todo's title-normalization lands, so we don't cache a miss
  that normalization would have resolved.
- `web/src/lib/mbcache.test.ts` (against `lyra_test`): first call invokes a spy `fetchFresh`,
  second returns cached without calling it; a stale row + a throwing `fetchFresh` serves the
  old value.
- Deploy after this task — cached reopens on artist/album/discover/preview pages land here.

## Task 3 — worker read-through + CachingMbBrowser (makes the cache shared)
- New `worker/lyra_worker/mbcache.py`:
  ```python
  import json
  def cached_mb(conn, key, ttl_sec, fetch_fresh):
      with conn.cursor() as cur:
          cur.execute('SELECT json, extract(epoch from (now() - "fetchedAt")) '
                      'FROM "MbCache" WHERE key = %s', (key,))
          row = cur.fetchone()
      if row and row[1] < ttl_sec:
          return row[0]
      fresh = fetch_fresh()
      with conn.cursor() as cur:
          cur.execute('INSERT INTO "MbCache" (key, json, "fetchedAt") VALUES (%s, %s, now()) '
                      'ON CONFLICT (key) DO UPDATE SET json = EXCLUDED.json, "fetchedAt" = now()',
                      (key, json.dumps(fresh)))
      conn.commit()
      return fresh
  ```
- `CachingMbBrowser(inner: MbBrowser, conn)` decorator implementing the same `MbBrowser`
  protocol, keying on the **same** logical strings as the web side (`artist-rgs:{mbid}`,
  `artist-search:{q}`). `browse_release_groups` returns `ReleaseGroupInfo` dataclasses →
  store plain field dicts and reconstruct on read (small mapping fn mirroring `_mbbrowser.py`).
- `registry.build_browser()` wraps in `CachingMbBrowser` when a conn is available (monitor/scan
  path); `LastfmSource`'s browser can take the same decorator or stay uncached (its own
  try/except already tolerates source failure).
- `worker/tests/test_mbcache.py`: read-through + stale semantics with a fake `fetch_fresh`
  (no MB network). Reconstruction round-trips a `ReleaseGroupInfo`.

## Task 4 — client-side modal cache + pruning (independent cheap wins)
- Module-level `Map<rgMbid, Track[]>` in the album-modal client so reopening the same modal
  within a session is instant with **zero** network (complements the DB cache; no backend).
- Pruning: one line on an existing worker periodic tick —
  `DELETE FROM "MbCache" WHERE "fetchedAt" < now() - interval '90 days'` — keeps the table
  bounded (single-user scale = a few thousand rows regardless).
- Optional force-refresh: on an artist page "search now" / scan re-resolve,
  `DELETE FROM "MbCache" WHERE key = 'artist-rgs:…'` first. Nice-to-have, not v1.

## Rollout order
1. Task 1 (schema + migration).
2. Task 2 (web read-through) → **deploy** — most of the "reopen re-fetches MB" pain gone with
   the smallest surface.
3. Task 3 (worker) → scans/discovery warm the same cache and stop re-fetching discographies.
4. Task 4 (client Map + prune line + optional force-refresh).

## Also-worth-doing (adjacent, not strictly this cache)
- A real **~1 req/s throttle on the web MB client** — caching reduces requests but doesn't
  guarantee pacing under a burst; the worker scan already paces, the web side doesn't.
- Overlaps with **TODO #6** (DB-cached on-disk tracklist): the Library modal should ultimately
  prefer real on-disk tracks; this MB cache covers the non-owned / fallback tracklist path.

## Notes / deferred
- **Name the table/helper generically** — the Last.fm browse page (`/lastfm`) has the same
  problem (live external calls per load) and should ride the same read-through. Consider
  `ApiCache(key, json, fetchedAt)` + `cached(...)` rather than MB-specific names, so Last.fm
  keys (`lfm-top-artists:{user}:{period}:{page}`, etc.) share the table. See the Last.fm
  caching todo. Only difference: Last.fm gets a manual **Refresh** button that busts its keys,
  and own-state annotation stays live (cache only the raw Last.fm payload).
- Redis deferred (see header). Reconsider only on measured PG-cache contention or a need for
  pub/sub / multi-instance.
- `null`/negative caching gated on the Last.fm title-normalization todo landing first.
- Key-normalization (trim+lowercase) for search keys MUST match byte-for-byte between the TS
  and Python sides or the two processes fragment the search cache (browse/tracks/name keys are
  MBID-based so they align naturally).