The worker's MB browser (discography + artist search, used by monitor sweeps,
library scans, and Last.fm discovery) re-fetched every time. Route it through
the same ApiCache table + logical keys the web app uses, so a discography one
process fetched warms the other, and repeat lookups stop hitting MB.
- apicache.py cached_api(): read-through, serve-stale-on-outage, never caches
null, best-effort (a cache-DB error degrades to a live fetch, never breaks
scan/sweep/discovery). Its own DEDICATED autocommit connection so it can't
commit the worker's in-flight transaction, and autocommit avoids freezing
Postgres now() (which would break TTL math).
- CachingMbBrowser decorates MusicBrainzBrowser, serializing to the SAME
camelCase JSON shape the web writes (round-trip tested). registry.build_browser
wraps it when a DSN is present; the Last.fm source's browser rides it too.
- Hourly prune drops ApiCache rows > 90d so the table stays bounded.
- worker 213 tests / 7-skip.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.env.example shipped a valid, PUBLIC base64 key — if a user didn't replace it,
every credential got encrypted with a publicly-known key and everything still
"worked," so the footgun was silent.
- .env.example now ships an obvious placeholder (CHANGE_ME_generate_with_...)
with a "you MUST replace this" note and the openssl hint.
- web + worker validate LYRA_SECRET_KEY at startup: missing / placeholder /
wrong-length ⇒ loud FATAL banner and refuse to start (web via the Next.js
instrumentation hook, worker via _require_secret_key before touching the DB).
The old public example key ⇒ a loud WARNING but not fatal, since an existing
install may have encrypted its creds under it (rotating needs re-entry).
- Validation lives in an edge-safe web/src/lib/secret-key.ts (no node:crypto)
so the instrumentation hook bundles for both runtimes; crypto.ts re-exports.
- README emphasizes generating a fresh key + keeping it stable.
Verified live: worker exits 1 on placeholder/unset before any DB call; web
instrumentation throws and refuses to serve on the placeholder. web 138 tests,
worker 204/7-skip, tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two row-leak bugs found in whole-branch review:
- discovery.run_discovery now prunes DiscoverySeedContribution rows whose
seedMbid is no longer a WatchedArtist (unfollow/removal has no FK cascade)
and recomputes the touched candidates' scores this sweep, instead of
letting a dead seed's contribution inflate the aggregate forever.
- maybe_run_scan's exception handler around scan_chunk now calls
clear_worklist and clears scan.id, matching the drain path. Previously
an exception mid-chunk left the already-done rows plus the remaining
worklist for that scan_id orphaned forever, since the next scan mints a
fresh uuid. Also clear scan.id in the build_worklist failure handler for
consistency.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace scan_chunk's O(N^2) tree re-walk-and-skip-to-cursor with a
persisted ScanWorkItem worklist: build_worklist walks the tree once at
scan start and inserts one row per album (idempotent via ON CONFLICT),
scan_chunk pops up to `limit` not-done rows by indexed cursor and marks
them done, and clear_worklist drops the scan's rows once drained.
scan_library is reimplemented on top of the worklist (one-shot scan_id,
drain, clear) with an unchanged signature/behavior. main.py's
maybe_run_scan now drives a generated scan.id through Config
(scan.id/scan.inProgress/scan.progress/scan.result) instead of a
scan.cursor string, so a mid-scan library change no longer risks
skipping or re-walking albums.
Staging is no longer forced under the library. The worker reads STAGING_ROOT
(container path, default /music/.staging) and stages per-job downloads there;
docker-compose mounts ${STAGING_DIR:-${MUSIC_DIR}/.staging} at /staging with
STAGING_ROOT=/staging. Set STAGING_DIR to a fast local disk when the library
(MUSIC_DIR) is a network share so temp download I/O stays off the share.
Safe cross-volume: import_album already assembles into a temp dir on the
library volume and swaps atomically there, so partial downloads never touch
the library and the final swap stays atomic. clear_staging_root now clears the
root's contents (mount-safe) rather than removing the root. Documented in
.env.example; new tests cover the separate-root path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Worker stamps a throttled worker.heartbeat; GET /api/worker reports online
from its updatedAt recency (DB-time); masthead shows a live listening/offline
dot instead of a static decoration.
- /api/requests enriched with rgMbid; Recently Pressed shows cover thumbs.
- Library album count moved from the filter row into a section header.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wanted releases pressed while the monitor is off never got reconciled, so
they lingered on the wanted list. Extract finish_job() and call it for every
finished job (not just when monitor.enabled) so the MonitoredRelease
transitions to fulfilled. Add fulfill_owned_releases(), run once at startup,
to backfill releases already owned at/above the quality cutoff (self-heals
items stuck from before this fix). Both idempotent + tested.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two worker-loop robustness fixes:
- Scan chunking: maybe_run_scan advances the library scan by at most one
scan_chunk (default scan.chunkSize=25 albums) per loop iteration, so a
large library no longer blocks job-claim/monitor for minutes-to-hours.
Progress persists in scan.inProgress/scan.cursor/scan.progress and is
resumable; scan.result (unchanged format) is written on completion.
- DB reconnect: wrap the loop body in `except psycopg.OperationalError`
(AdminShutdown subclasses it) to close the dead handle and reconnect via
wait_for_db() in-process, instead of crashing and relying on the
container restart policy. Verified via a backend-termination test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The scheduled-tick branch and the discover.requested one-shot branch
both read the same stale per-iteration config snapshot, so an enabled+due
sweep coinciding with a pending request ran discovery twice back-to-back
(idempotent but wasteful). Extract maybe_run_discovery, which unifies both
triggers into a single run and is now unit-tested.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds registry.build_similarity_sources() (always constructs
ListenBrainzSource; reachability is a per-run health() concern, not a
startup gate) and wires discovery into the worker's main loop: a
scheduled sweep gated on discover.enabled + DISCOVER_TICK_SECONDS, and
a one-shot discover.requested trigger mirroring the existing
scan.requested pattern.
Also hardens run_discovery's health-check pass: a source whose
health() raises is now logged and skipped instead of aborting the
whole sweep (accepted finding from the Task-3 review).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A followed artist previously showed only the one owned album ("of 1")
because scan_library recorded just the resolved release. Now scan
browses each artist's full MusicBrainz release-group list once per
run and stores it as unmonitored MonitoredRelease rows, matching
web-follow behavior. The owned album keeps its monitored/fulfilled
state via ON CONFLICT DO NOTHING.
The monitor's quality-upgrade path was inert: run_pipeline's intake
short-circuit treated any existing LibraryItem as "done", so a
monitor-driven upgrade job for a below-cutoff copy jumped straight to
imported without downloading. Intake now checks whether the job is
monitor-driven (Request.monitoredReleaseId IS NOT NULL) and, if so,
only short-circuits when the existing copy already meets the quality
cutoff; plain manual requests keep exact slice-1 behavior. Also drops
an unused `field` import in browser.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Jonathan