Lyra/docs/superpowers/plans/2026-07-14-musicbrainz-cache.md

MusicBrainz Response Cache (Postgres) — Implementation Plan

Sketched 2026-07-14. A Postgres-backed read-through cache shared by the web app and the worker, so reopening an artist (discography) or album (tracklist) — and repeat lookups during scans/discovery — stop re-hitting MusicBrainz. Postgres only; Redis explicitly disregarded (Postgres is already running and shared across web(TS) + worker(Python); Next's Data Cache is per-instance and the Python worker can't see it, so a shared store is required and we already have one — no new container/ops for a single-user app). Redis revisited only if profiling later shows PG cache lookups are a bottleneck (won't be at this scale) or pub/sub is wanted.

Context

Both callers hit MB live with no cache and no throttle: web mbGet (web/src/lib/musicbrainz.ts — bare fetch) and worker _mbbrowser.py (musicbrainzngs). Reopen an artist → full discography re-fetched; reopen an album → tracklist re-fetched; search-as-you-type fires a request per keystroke. MB data changes on the order of days/weeks, so hours-to-days TTLs are safe. Caching also helps stay under MB's ~1 req/s anonymous rate limit (today only the worker scan paces ~1/s; the web side is unthrottled).

Confirmed decisions

• Postgres, not Redis (rationale above).
• Key on the LOGICAL operation, not the request URL. Web uses raw ws/2 REST paths; the worker uses musicbrainzngs, which builds its own requests — the two never produce the same URL, so URL-keyed caches would never share. Logical keys let a discography the worker fetched during a scan warm the web modal (and vice versa). Cost: wrap the high-level functions, not the low-level fetch.
• TTL applied at read time from fetchedAt (no expiresAt column) so TTLs are tunable in code with no migration.
• Serve stale on MB outage — if a live fetch throws and a (stale) row exists, return it.
• TTL-only invalidation is sufficient for a home app; no explicit invalidation required for v1.

Cross-cutting facts

• Prisma schema web/prisma/schema.prisma; migrations reach live lyra via the web entrypoint's migrate deploy on docker compose up -d --build (same as every prior migration). Postgres Json maps to JSONB.
• The worker already threads a conn (psycopg) through maybe_run_scan / scan_chunk / _run_discovery (main.py), and MusicBrainzBrowser is built in registry.build_browser() / used by LastfmSource. A conn is available at the monitor/scan call sites.
• Key namespaces (shared verbatim between TS and Python): rg-tracks:{rgMbid}, artist-rgs:{artistMbid}, artist-name:{artistMbid}, rg-search:{artist}|{album}, artist-search:{query}.
• Suggested TTLs (all tunable, no migration): tracklists / discography / artist-name 30d (near-immutable), searchReleaseGroup 7d, searchArtists type-ahead 1d (or leave uncached). Normalize rg-search/artist-search keys (trim + lowercase) so casing doesn't fragment the cache.
• Tests run against lyra_test (guard blocks the live DB); both suites are DB + pure logic, no MB network in tests.

Task 1 — schema + migration

• web/prisma/schema.prisma += model:

  model MbCache {
    key       String   @id
    json      Json
    fetchedAt DateTime @default(now())
  }
  

• npx prisma migrate dev --name add_mb_cache (dev/lyra_test). Ships to live via migrate deploy on rebuild. No expiresAt.
• Verify: prisma migrate status clean; table present.

Task 2 — web read-through (biggest UX win, smallest surface)

• New web/src/lib/mbcache.ts:

  import { prisma } from "@/lib/db";
  const DAY = 86_400;
  export async function cachedMb<T>(key: string, ttlSec: number, fetchFresh: () => Promise<T>): Promise<T> {
    const hit = await prisma.mbCache.findUnique({ where: { key } });
    if (hit && (Date.now() - hit.fetchedAt.getTime()) / 1000 < ttlSec) return hit.json as T;
    try {
      const fresh = await fetchFresh();
      await prisma.mbCache.upsert({
        where: { key },
        create: { key, json: fresh as object },
        update: { json: fresh as object, fetchedAt: new Date() },
      });
      return fresh;
    } catch (e) {
      if (hit) return hit.json as T; // serve stale on MB outage
      throw e;
    }
  }
  

• Wrap the high-level fns in web/src/lib/musicbrainz.ts with logical keys + TTLs: browseReleaseGroupTracks (rg-tracks:{rgMbid}, 30d), browse discography (artist-rgs:{artistMbid}, 30d), getArtistName (artist-name:{artistMbid}, 30d), searchReleaseGroup (rg-search:{artist}|{album} normalized, 7d), searchArtists (artist-search:{query} normalized, 1d — or skip). Keep each fn's existing MB-mapping logic inside the fetchFresh closure so cached values are the already-shaped objects.
• Negative results: searchReleaseGroup can return null (no match). Decide: cache null briefly (e.g. 1d) to avoid re-hammering on edition-qualified misses (ties into the Last.fm "No MusicBrainz match" todo), OR don't cache nulls. Recommend caching null with a short TTL — but only AFTER that todo's title-normalization lands, so we don't cache a miss that normalization would have resolved.
• web/src/lib/mbcache.test.ts (against lyra_test): first call invokes a spy fetchFresh, second returns cached without calling it; a stale row + a throwing fetchFresh serves the old value.
• Deploy after this task — cached reopens on artist/album/discover/preview pages land here.

Task 3 — worker read-through + CachingMbBrowser (makes the cache shared)

• New worker/lyra_worker/mbcache.py:

  import json
  def cached_mb(conn, key, ttl_sec, fetch_fresh):
      with conn.cursor() as cur:
          cur.execute('SELECT json, extract(epoch from (now() - "fetchedAt")) '
                      'FROM "MbCache" WHERE key = %s', (key,))
          row = cur.fetchone()
      if row and row[1] < ttl_sec:
          return row[0]
      fresh = fetch_fresh()
      with conn.cursor() as cur:
          cur.execute('INSERT INTO "MbCache" (key, json, "fetchedAt") VALUES (%s, %s, now()) '
                      'ON CONFLICT (key) DO UPDATE SET json = EXCLUDED.json, "fetchedAt" = now()',
                      (key, json.dumps(fresh)))
      conn.commit()
      return fresh
  

• CachingMbBrowser(inner: MbBrowser, conn) decorator implementing the same MbBrowser protocol, keying on the same logical strings as the web side (artist-rgs:{mbid}, artist-search:{q}). browse_release_groups returns ReleaseGroupInfo dataclasses → store plain field dicts and reconstruct on read (small mapping fn mirroring _mbbrowser.py).
• registry.build_browser() wraps in CachingMbBrowser when a conn is available (monitor/scan path); LastfmSource's browser can take the same decorator or stay uncached (its own try/except already tolerates source failure).
• worker/tests/test_mbcache.py: read-through + stale semantics with a fake fetch_fresh (no MB network). Reconstruction round-trips a ReleaseGroupInfo.

Task 4 — client-side modal cache + pruning (independent cheap wins)

• Module-level Map<rgMbid, Track[]> in the album-modal client so reopening the same modal within a session is instant with zero network (complements the DB cache; no backend).
• Pruning: one line on an existing worker periodic tick — DELETE FROM "MbCache" WHERE "fetchedAt" < now() - interval '90 days' — keeps the table bounded (single-user scale = a few thousand rows regardless).
• Optional force-refresh: on an artist page "search now" / scan re-resolve, DELETE FROM "MbCache" WHERE key = 'artist-rgs:…' first. Nice-to-have, not v1.

Rollout order

1. Task 1 (schema + migration).
2. Task 2 (web read-through) → deploy — most of the "reopen re-fetches MB" pain gone with the smallest surface.
3. Task 3 (worker) → scans/discovery warm the same cache and stop re-fetching discographies.
4. Task 4 (client Map + prune line + optional force-refresh).

Also-worth-doing (adjacent, not strictly this cache)

• A real ~1 req/s throttle on the web MB client — caching reduces requests but doesn't guarantee pacing under a burst; the worker scan already paces, the web side doesn't.
• Overlaps with TODO #6 (DB-cached on-disk tracklist): the Library modal should ultimately prefer real on-disk tracks; this MB cache covers the non-owned / fallback tracklist path.

Notes / deferred

• Redis deferred (see header). Reconsider only on measured PG-cache contention or a need for pub/sub / multi-instance.
• null/negative caching gated on the Last.fm title-normalization todo landing first.
• Key-normalization (trim+lowercase) for search keys MUST match byte-for-byte between the TS and Python sides or the two processes fragment the search cache (browse/tracks/name keys are MBID-based so they align naturally).