docs: add MusicBrainz Postgres cache implementation plan

A read-through cache shared by the web app and the worker so reopening an artist/album (and repeat lookups during scans/discovery) stop re-hitting MusicBrainz. Postgres-only (Redis disregarded — PG is already shared across web + worker); keys on the logical operation so both processes share entries; serve-stale-on-outage; TTL-only invalidation. Four tasks: MbCache table → web read-through → worker CachingMbBrowser → client Map + pruning. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 00:33:03 +02:00
parent 8922af757c
commit 2158d988f8
1 changed files with 159 additions and 0 deletions
@@ -0,0 +1,159 @@
+# MusicBrainz Response Cache (Postgres) — Implementation Plan
+
+Sketched 2026-07-14. A Postgres-backed read-through cache shared by the web app and the
+worker, so reopening an artist (discography) or album (tracklist) — and repeat lookups
+during scans/discovery — stop re-hitting MusicBrainz. **Postgres only; Redis explicitly
+disregarded** (Postgres is already running and shared across web(TS) + worker(Python);
+Next's Data Cache is per-instance and the Python worker can't see it, so a shared store is
+required and we already have one — no new container/ops for a single-user app). Redis
+revisited only if profiling later shows PG cache lookups are a bottleneck (won't be at this
+scale) or pub/sub is wanted.
+
+## Context
+Both callers hit MB live with **no cache and no throttle**: web `mbGet`
+(`web/src/lib/musicbrainz.ts` — bare `fetch`) and worker `_mbbrowser.py`
+(`musicbrainzngs`). Reopen an artist → full discography re-fetched; reopen an album →
+tracklist re-fetched; search-as-you-type fires a request per keystroke. MB data changes on
+the order of days/weeks, so hours-to-days TTLs are safe. Caching also helps stay under MB's
+~1 req/s anonymous rate limit (today only the worker scan paces ~1/s; the web side is
+unthrottled).
+
+## Confirmed decisions
+- **Postgres, not Redis** (rationale above).
+- **Key on the LOGICAL operation, not the request URL.** Web uses raw `ws/2` REST paths;
+  the worker uses `musicbrainzngs`, which builds its own requests — the two never produce
+  the same URL, so URL-keyed caches would never share. Logical keys let a discography the
+  worker fetched during a scan warm the web modal (and vice versa). Cost: wrap the
+  **high-level** functions, not the low-level fetch.
+- **TTL applied at read time from `fetchedAt`** (no `expiresAt` column) so TTLs are tunable
+  in code with no migration.
+- **Serve stale on MB outage** — if a live fetch throws and a (stale) row exists, return it.
+- **TTL-only invalidation** is sufficient for a home app; no explicit invalidation required
+  for v1.
+
+## Cross-cutting facts
+- Prisma schema `web/prisma/schema.prisma`; migrations reach live `lyra` via the web
+  entrypoint's `migrate deploy` on `docker compose up -d --build` (same as every prior
+  migration). Postgres `Json` maps to JSONB.
+- The worker already threads a `conn` (psycopg) through `maybe_run_scan` / `scan_chunk` /
+  `_run_discovery` (`main.py`), and `MusicBrainzBrowser` is built in `registry.build_browser()`
+  / used by `LastfmSource`. A conn is available at the monitor/scan call sites.
+- Key namespaces (shared verbatim between TS and Python):
+  `rg-tracks:{rgMbid}`, `artist-rgs:{artistMbid}`, `artist-name:{artistMbid}`,
+  `rg-search:{artist}|{album}`, `artist-search:{query}`.
+- Suggested TTLs (all tunable, no migration): tracklists / discography / artist-name **30d**
+  (near-immutable), `searchReleaseGroup` **7d**, `searchArtists` type-ahead **1d** (or leave
+  uncached). Normalize `rg-search`/`artist-search` keys (trim + lowercase) so casing doesn't
+  fragment the cache.
+- Tests run against `lyra_test` (guard blocks the live DB); both suites are DB + pure logic,
+  no MB network in tests.
+
+## Task 1 — schema + migration
+- `web/prisma/schema.prisma` += model:
+  ```prisma
+  model MbCache {
+    key       String   @id
+    json      Json
+    fetchedAt DateTime @default(now())
+  }
+  ```
+- `npx prisma migrate dev --name add_mb_cache` (dev/`lyra_test`). Ships to live via
+  `migrate deploy` on rebuild. No `expiresAt`.
+- Verify: `prisma migrate status` clean; table present.
+
+## Task 2 — web read-through (biggest UX win, smallest surface)
+- New `web/src/lib/mbcache.ts`:
+  ```ts
+  import { prisma } from "@/lib/db";
+  const DAY = 86_400;
+  export async function cachedMb<T>(key: string, ttlSec: number, fetchFresh: () => Promise<T>): Promise<T> {
+    const hit = await prisma.mbCache.findUnique({ where: { key } });
+    if (hit && (Date.now() - hit.fetchedAt.getTime()) / 1000 < ttlSec) return hit.json as T;
+    try {
+      const fresh = await fetchFresh();
+      await prisma.mbCache.upsert({
+        where: { key },
+        create: { key, json: fresh as object },
+        update: { json: fresh as object, fetchedAt: new Date() },
+      });
+      return fresh;
+    } catch (e) {
+      if (hit) return hit.json as T; // serve stale on MB outage
+      throw e;
+    }
+  }
+  ```
+- Wrap the high-level fns in `web/src/lib/musicbrainz.ts` with logical keys + TTLs:
+  `browseReleaseGroupTracks` (`rg-tracks:{rgMbid}`, 30d), `browse discography`
+  (`artist-rgs:{artistMbid}`, 30d), `getArtistName` (`artist-name:{artistMbid}`, 30d),
+  `searchReleaseGroup` (`rg-search:{artist}|{album}` normalized, 7d), `searchArtists`
+  (`artist-search:{query}` normalized, 1d — or skip). Keep each fn's existing MB-mapping
+  logic inside the `fetchFresh` closure so cached values are the already-shaped objects.
+- **Negative results:** `searchReleaseGroup` can return `null` (no match). Decide: cache
+  `null` briefly (e.g. 1d) to avoid re-hammering on edition-qualified misses (ties into the
+  Last.fm "No MusicBrainz match" todo), OR don't cache nulls. Recommend caching `null` with a
+  short TTL — but only AFTER that todo's title-normalization lands, so we don't cache a miss
+  that normalization would have resolved.
+- `web/src/lib/mbcache.test.ts` (against `lyra_test`): first call invokes a spy `fetchFresh`,
+  second returns cached without calling it; a stale row + a throwing `fetchFresh` serves the
+  old value.
+- Deploy after this task — cached reopens on artist/album/discover/preview pages land here.
+
+## Task 3 — worker read-through + CachingMbBrowser (makes the cache shared)
+- New `worker/lyra_worker/mbcache.py`:
+  ```python
+  import json
+  def cached_mb(conn, key, ttl_sec, fetch_fresh):
+      with conn.cursor() as cur:
+          cur.execute('SELECT json, extract(epoch from (now() - "fetchedAt")) '
+                      'FROM "MbCache" WHERE key = %s', (key,))
+          row = cur.fetchone()
+      if row and row[1] < ttl_sec:
+          return row[0]
+      fresh = fetch_fresh()
+      with conn.cursor() as cur:
+          cur.execute('INSERT INTO "MbCache" (key, json, "fetchedAt") VALUES (%s, %s, now()) '
+                      'ON CONFLICT (key) DO UPDATE SET json = EXCLUDED.json, "fetchedAt" = now()',
+                      (key, json.dumps(fresh)))
+      conn.commit()
+      return fresh
+  ```
+- `CachingMbBrowser(inner: MbBrowser, conn)` decorator implementing the same `MbBrowser`
+  protocol, keying on the **same** logical strings as the web side (`artist-rgs:{mbid}`,
+  `artist-search:{q}`). `browse_release_groups` returns `ReleaseGroupInfo` dataclasses →
+  store plain field dicts and reconstruct on read (small mapping fn mirroring `_mbbrowser.py`).
+- `registry.build_browser()` wraps in `CachingMbBrowser` when a conn is available (monitor/scan
+  path); `LastfmSource`'s browser can take the same decorator or stay uncached (its own
+  try/except already tolerates source failure).
+- `worker/tests/test_mbcache.py`: read-through + stale semantics with a fake `fetch_fresh`
+  (no MB network). Reconstruction round-trips a `ReleaseGroupInfo`.
+
+## Task 4 — client-side modal cache + pruning (independent cheap wins)
+- Module-level `Map<rgMbid, Track[]>` in the album-modal client so reopening the same modal
+  within a session is instant with **zero** network (complements the DB cache; no backend).
+- Pruning: one line on an existing worker periodic tick —
+  `DELETE FROM "MbCache" WHERE "fetchedAt" < now() - interval '90 days'` — keeps the table
+  bounded (single-user scale = a few thousand rows regardless).
+- Optional force-refresh: on an artist page "search now" / scan re-resolve,
+  `DELETE FROM "MbCache" WHERE key = 'artist-rgs:…'` first. Nice-to-have, not v1.
+
+## Rollout order
+1. Task 1 (schema + migration).
+2. Task 2 (web read-through) → **deploy** — most of the "reopen re-fetches MB" pain gone with
+   the smallest surface.
+3. Task 3 (worker) → scans/discovery warm the same cache and stop re-fetching discographies.
+4. Task 4 (client Map + prune line + optional force-refresh).
+
+## Also-worth-doing (adjacent, not strictly this cache)
+- A real **~1 req/s throttle on the web MB client** — caching reduces requests but doesn't
+  guarantee pacing under a burst; the worker scan already paces, the web side doesn't.
+- Overlaps with **TODO #6** (DB-cached on-disk tracklist): the Library modal should ultimately
+  prefer real on-disk tracks; this MB cache covers the non-owned / fallback tracklist path.
+
+## Notes / deferred
+- Redis deferred (see header). Reconsider only on measured PG-cache contention or a need for
+  pub/sub / multi-instance.
+- `null`/negative caching gated on the Last.fm title-normalization todo landing first.
+- Key-normalization (trim+lowercase) for search keys MUST match byte-for-byte between the TS
+  and Python sides or the two processes fragment the search cache (browse/tracks/name keys are
+  MBID-based so they align naturally).