diff --git a/docs/superpowers/plans/2026-07-14-musicbrainz-cache.md b/docs/superpowers/plans/2026-07-14-musicbrainz-cache.md new file mode 100644 index 0000000..aa5d613 --- /dev/null +++ b/docs/superpowers/plans/2026-07-14-musicbrainz-cache.md @@ -0,0 +1,159 @@ +# MusicBrainz Response Cache (Postgres) — Implementation Plan + +Sketched 2026-07-14. A Postgres-backed read-through cache shared by the web app and the +worker, so reopening an artist (discography) or album (tracklist) — and repeat lookups +during scans/discovery — stop re-hitting MusicBrainz. **Postgres only; Redis explicitly +disregarded** (Postgres is already running and shared across web(TS) + worker(Python); +Next's Data Cache is per-instance and the Python worker can't see it, so a shared store is +required and we already have one — no new container/ops for a single-user app). Redis +revisited only if profiling later shows PG cache lookups are a bottleneck (won't be at this +scale) or pub/sub is wanted. + +## Context +Both callers hit MB live with **no cache and no throttle**: web `mbGet` +(`web/src/lib/musicbrainz.ts` — bare `fetch`) and worker `_mbbrowser.py` +(`musicbrainzngs`). Reopen an artist → full discography re-fetched; reopen an album → +tracklist re-fetched; search-as-you-type fires a request per keystroke. MB data changes on +the order of days/weeks, so hours-to-days TTLs are safe. Caching also helps stay under MB's +~1 req/s anonymous rate limit (today only the worker scan paces ~1/s; the web side is +unthrottled). + +## Confirmed decisions +- **Postgres, not Redis** (rationale above). +- **Key on the LOGICAL operation, not the request URL.** Web uses raw `ws/2` REST paths; + the worker uses `musicbrainzngs`, which builds its own requests — the two never produce + the same URL, so URL-keyed caches would never share. Logical keys let a discography the + worker fetched during a scan warm the web modal (and vice versa). Cost: wrap the + **high-level** functions, not the low-level fetch. +- **TTL applied at read time from `fetchedAt`** (no `expiresAt` column) so TTLs are tunable + in code with no migration. +- **Serve stale on MB outage** — if a live fetch throws and a (stale) row exists, return it. +- **TTL-only invalidation** is sufficient for a home app; no explicit invalidation required + for v1. + +## Cross-cutting facts +- Prisma schema `web/prisma/schema.prisma`; migrations reach live `lyra` via the web + entrypoint's `migrate deploy` on `docker compose up -d --build` (same as every prior + migration). Postgres `Json` maps to JSONB. +- The worker already threads a `conn` (psycopg) through `maybe_run_scan` / `scan_chunk` / + `_run_discovery` (`main.py`), and `MusicBrainzBrowser` is built in `registry.build_browser()` + / used by `LastfmSource`. A conn is available at the monitor/scan call sites. +- Key namespaces (shared verbatim between TS and Python): + `rg-tracks:{rgMbid}`, `artist-rgs:{artistMbid}`, `artist-name:{artistMbid}`, + `rg-search:{artist}|{album}`, `artist-search:{query}`. +- Suggested TTLs (all tunable, no migration): tracklists / discography / artist-name **30d** + (near-immutable), `searchReleaseGroup` **7d**, `searchArtists` type-ahead **1d** (or leave + uncached). Normalize `rg-search`/`artist-search` keys (trim + lowercase) so casing doesn't + fragment the cache. +- Tests run against `lyra_test` (guard blocks the live DB); both suites are DB + pure logic, + no MB network in tests. + +## Task 1 — schema + migration +- `web/prisma/schema.prisma` += model: + ```prisma + model MbCache { + key String @id + json Json + fetchedAt DateTime @default(now()) + } + ``` +- `npx prisma migrate dev --name add_mb_cache` (dev/`lyra_test`). Ships to live via + `migrate deploy` on rebuild. No `expiresAt`. +- Verify: `prisma migrate status` clean; table present. + +## Task 2 — web read-through (biggest UX win, smallest surface) +- New `web/src/lib/mbcache.ts`: + ```ts + import { prisma } from "@/lib/db"; + const DAY = 86_400; + export async function cachedMb(key: string, ttlSec: number, fetchFresh: () => Promise): Promise { + const hit = await prisma.mbCache.findUnique({ where: { key } }); + if (hit && (Date.now() - hit.fetchedAt.getTime()) / 1000 < ttlSec) return hit.json as T; + try { + const fresh = await fetchFresh(); + await prisma.mbCache.upsert({ + where: { key }, + create: { key, json: fresh as object }, + update: { json: fresh as object, fetchedAt: new Date() }, + }); + return fresh; + } catch (e) { + if (hit) return hit.json as T; // serve stale on MB outage + throw e; + } + } + ``` +- Wrap the high-level fns in `web/src/lib/musicbrainz.ts` with logical keys + TTLs: + `browseReleaseGroupTracks` (`rg-tracks:{rgMbid}`, 30d), `browse discography` + (`artist-rgs:{artistMbid}`, 30d), `getArtistName` (`artist-name:{artistMbid}`, 30d), + `searchReleaseGroup` (`rg-search:{artist}|{album}` normalized, 7d), `searchArtists` + (`artist-search:{query}` normalized, 1d — or skip). Keep each fn's existing MB-mapping + logic inside the `fetchFresh` closure so cached values are the already-shaped objects. +- **Negative results:** `searchReleaseGroup` can return `null` (no match). Decide: cache + `null` briefly (e.g. 1d) to avoid re-hammering on edition-qualified misses (ties into the + Last.fm "No MusicBrainz match" todo), OR don't cache nulls. Recommend caching `null` with a + short TTL — but only AFTER that todo's title-normalization lands, so we don't cache a miss + that normalization would have resolved. +- `web/src/lib/mbcache.test.ts` (against `lyra_test`): first call invokes a spy `fetchFresh`, + second returns cached without calling it; a stale row + a throwing `fetchFresh` serves the + old value. +- Deploy after this task — cached reopens on artist/album/discover/preview pages land here. + +## Task 3 — worker read-through + CachingMbBrowser (makes the cache shared) +- New `worker/lyra_worker/mbcache.py`: + ```python + import json + def cached_mb(conn, key, ttl_sec, fetch_fresh): + with conn.cursor() as cur: + cur.execute('SELECT json, extract(epoch from (now() - "fetchedAt")) ' + 'FROM "MbCache" WHERE key = %s', (key,)) + row = cur.fetchone() + if row and row[1] < ttl_sec: + return row[0] + fresh = fetch_fresh() + with conn.cursor() as cur: + cur.execute('INSERT INTO "MbCache" (key, json, "fetchedAt") VALUES (%s, %s, now()) ' + 'ON CONFLICT (key) DO UPDATE SET json = EXCLUDED.json, "fetchedAt" = now()', + (key, json.dumps(fresh))) + conn.commit() + return fresh + ``` +- `CachingMbBrowser(inner: MbBrowser, conn)` decorator implementing the same `MbBrowser` + protocol, keying on the **same** logical strings as the web side (`artist-rgs:{mbid}`, + `artist-search:{q}`). `browse_release_groups` returns `ReleaseGroupInfo` dataclasses → + store plain field dicts and reconstruct on read (small mapping fn mirroring `_mbbrowser.py`). +- `registry.build_browser()` wraps in `CachingMbBrowser` when a conn is available (monitor/scan + path); `LastfmSource`'s browser can take the same decorator or stay uncached (its own + try/except already tolerates source failure). +- `worker/tests/test_mbcache.py`: read-through + stale semantics with a fake `fetch_fresh` + (no MB network). Reconstruction round-trips a `ReleaseGroupInfo`. + +## Task 4 — client-side modal cache + pruning (independent cheap wins) +- Module-level `Map` in the album-modal client so reopening the same modal + within a session is instant with **zero** network (complements the DB cache; no backend). +- Pruning: one line on an existing worker periodic tick — + `DELETE FROM "MbCache" WHERE "fetchedAt" < now() - interval '90 days'` — keeps the table + bounded (single-user scale = a few thousand rows regardless). +- Optional force-refresh: on an artist page "search now" / scan re-resolve, + `DELETE FROM "MbCache" WHERE key = 'artist-rgs:…'` first. Nice-to-have, not v1. + +## Rollout order +1. Task 1 (schema + migration). +2. Task 2 (web read-through) → **deploy** — most of the "reopen re-fetches MB" pain gone with + the smallest surface. +3. Task 3 (worker) → scans/discovery warm the same cache and stop re-fetching discographies. +4. Task 4 (client Map + prune line + optional force-refresh). + +## Also-worth-doing (adjacent, not strictly this cache) +- A real **~1 req/s throttle on the web MB client** — caching reduces requests but doesn't + guarantee pacing under a burst; the worker scan already paces, the web side doesn't. +- Overlaps with **TODO #6** (DB-cached on-disk tracklist): the Library modal should ultimately + prefer real on-disk tracks; this MB cache covers the non-owned / fallback tracklist path. + +## Notes / deferred +- Redis deferred (see header). Reconsider only on measured PG-cache contention or a need for + pub/sub / multi-instance. +- `null`/negative caching gated on the Last.fm title-normalization todo landing first. +- Key-normalization (trim+lowercase) for search keys MUST match byte-for-byte between the TS + and Python sides or the two processes fragment the search cache (browse/tracks/name keys are + MBID-based so they align naturally).