_derive_albums now also picks the artist's most-played album (Last.fm,
matched into the browsed discography by MBID then normalized title),
alongside the newest. Same release -> one row tagged newest,popular.
albumReason persisted; run_discovery threads an optional popularity provider.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LastfmAlbumPopularity.top_albums(conn, name) ranks an artist's albums by
Last.fm playcount (read-through ApiCache, 12h; errors -> []). registry
build_album_popularity returns it only when lastfm.api_key is set.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Discovery seeded only from followed artists. Now it also seeds from artists
you own albums by (LibraryItem.artistMbid, captured since the MBID own-state
work) but don't follow — so your library shapes suggestions, not just your
follows.
- New DiscoverySeed throttle table (migration add_discovery_seed) mirroring
WatchedArtist.lastDiscoveredAt, so each library-derived seed is swept once
per interval (WatchedArtist seeds keep their own throttle).
- _seed_artists merges both seed sources least-recently-swept first, capped at
chunk_size; run_discovery stamps the right throttle per seed and evicts
contributions from seeds that are neither followed nor owned.
- Artists you already have (followed OR owned) are excluded from the suggestion
candidates — discovery surfaces only genuinely new artists.
- discover.seedFromLibrary config (default true) + a Discovery settings toggle.
NOTE: existing LibraryItem rows have null artistMbid (populated on new
import/scan), so a re-scan is needed before old albums seed discovery.
worker 236, web 187 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Own-state ("in library") relied on an exact artist-NAME match, so an owned
album whose name differed from the MB canonical (punctuation/locale/feat.)
showed as not-owned even when followed. Capture the MusicBrainz artist MBID
on LibraryItem and prefer it for matching.
- New LibraryItem.artistMbid (migration add_library_artist_mbid). Populated at
import (pipeline) + scan, both of which already hold the resolved target;
the pipeline also now stores rgMbid (it previously didn't). COALESCE on
conflict backfills MBIDs on re-scan without clobbering.
- /api/artists "in library, not followed" matches by artistMbid first, name
case-insensitively as fallback (rows predating the column). Library route
prefers the item's own artistMbid over the release join.
Last.fm/discover own-state still use name-match (they work) — a smaller
follow-up. worker 233, web 186 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The verify step checked track COUNT but not playtime, so a truncated file or
a short preview substituted for a real track could import as "complete". Add
a total-duration check: sum the staged audio's playtime (mutagen) and fail
the job when it falls below 85% of MusicBrainz's expected total
(target.total_duration_s, already resolved). Generous tolerance so
edition/encoding differences don't false-positive; bonus tracks (over-long)
always pass.
The duration reader is injected (measure_duration param, default the real
mutagen-based reader) so the fail/pass paths are unit-tested; it returns None
for the offline fakes (no real audio), skipping the check exactly like the
staged-count guard. worker 233 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_record_contributions now normalizes each source's top-N to [0,1] by its own
top score before summing across sources. ListenBrainz co-occurrence counts
(thousands) no longer drown Last.fm's 0–1 match scores, so a small-scale
source contributes real ranking signal instead of only adding candidates.
Discovery score assertions updated to the normalized scale (single-candidate
fakes normalize to 1.0); added a cross-source test showing a small-scale
source's candidate ranks equal to a large-scale one. min_score is now on the
[0, n_sources] scale (default 0.0 → unaffected). worker 231 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add Request.force (migration add_request_force): the pipeline's intake now
skips the "already in library" dedupe for a forced job, so an owned album is
re-downloaded. The import step still keeps the new copy only when it's higher
quality, so a forced upgrade can never downgrade what's on disk.
- Worker: _is_force_job → dedupe bypassed when Request.force.
- POST /api/library/[id]/upgrade finds the matching MonitoredRelease and
enqueues a force request (guards against stacking on an in-flight job).
- Library route exposes monitoredReleaseId; the album modal shows a
"Replace / upgrade" button when a release exists.
Worker + web tests added (pipeline force re-acquire; upgrade route).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add MonitoredRelease.ignored (migration add_release_ignored). The monitor's
enqueue_due skips ignored releases, and they drop off the Wanted list —
independent of `monitored`, so auto-monitor-future can't silently re-grab
an album the user chose to ignore.
- Worker enqueue_due: AND NOT mr.ignored.
- /api/releases/[id] PATCH now accepts `ignored`; /api/wanted filters it out;
artist detail exposes `ignored`.
- UI: Ignore button on Wanted rows; discography shows an "Ignored" badge with
an Ignore/Un-ignore toggle (monitor + search disabled while ignored).
Worker + web tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The library scan silently skipped album folders with no MusicBrainz match
(or an unreadable file). Persist them so they're visible and fixable.
- New UnmatchedAlbum model (migration add_unmatched_album), keyed by path.
- scan_chunk records a skipped album as unmatched (reason "no MusicBrainz
match" or "unreadable: <err>"), clears the row when an album later
resolves, and prunes stale rows (a prior scan's, or a removed folder)
when a scan completes.
- GET /api/library/unmatched lists them; DELETE ?id= dismisses one.
- /library shows a "Couldn't match" section (path + reason + Dismiss).
Worker + web tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Discovery's "Suggested albums" surfaced singles and EPs because
_derive_albums filtered with is_core_release (Album/Single/EP). Add a
discovery-only _album_kind_ok gate that, when discover.albumsOnly (new
config, default true), requires primary_type == "Album" with no secondary
types. is_core_release is untouched — the monitor/scan still use it.
Exposed discover.albumsOnly in the discover config route + a Discovery
settings toggle. Worker + web tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completeness only compared the adapter's self-reported track_count to expected,
so a silently-skipped track (0-byte placeholder — see the Qobuz 24K Magic bug)
imported as a "complete" album. Now the verify also counts the NON-EMPTY audio
actually staged and fails "incomplete download" when fewer than expected are
present (guarded with `staged and ...` so fake adapters that stage nothing keep
using the count path). A real broken download now goes needs_attention (or falls
through to another source) instead of importing a dead track.
Full per-track duration matching against MB recording lengths (the rest of #16)
remains a larger follow-up. worker 224 tests / 7-skip.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docker-compose only health-checked db. Add:
- web: an unauthenticated GET /api/health (excluded from the auth middleware) +
a compose healthcheck that hits it (node global fetch — no curl/wget in image).
- worker: a background daemon thread touches /tmp/worker.alive every ~15s
(independent of the main loop, so it stays fresh through a long download) +
a compose healthcheck on the file's mtime (unhealthy if >90s stale).
Now `docker compose ps` reports real health for all services.
worker 223 tests, web 154, tsc + build + compose config clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bruno Mars "24K Magic" track 1 repeatedly downloaded as a 0-byte file. The
worker log showed the real cause: streamrip's track download hit
HTTPException("got more than 100 headers") — Qobuz's CDN returns >100 response
headers for some tracks and Python's http.client rejects them at the default
_MAXHEADERS=100, so streamrip skips the track and leaves an empty placeholder.
- Raise http.client._MAXHEADERS to 1000 on import of the Qobuz adapter, so those
responses parse and the track downloads.
- _count_staged_audio now skips 0-byte files (a failed-track placeholder is not a
real track) — improves the download-progress estimate and stops counting dead
tracks. (A verify-level "reject 0-byte tracks" safety net is a follow-up under
#16 per-track duration matching — it needs the pipeline fakes to write real
files, deferred to avoid disproportionate churn here.)
worker 223 tests / 7-skip.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live-testing #9 surfaced a pre-existing gap: one truncated file mutagen chokes
on ("file said 4 bytes, read 0 bytes") raised out of _process_album and aborted
the entire library scan (worklist abandoned). Now scan_chunk catches a per-album
error, rolls back, logs it, counts it as skipped, marks the item done, and
continues — so a single bad file can't block the rest of the library.
worker tests: new case asserts a corrupt album is skipped while a good one still
imports (and captures its on-disk trackNames).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Library modal showed MusicBrainz's canonical tracklist, hiding incomplete
downloads / edition or track-order mismatches, and showed nothing for albums
without an rgMbid. Now it shows the REAL files on disk.
- LibraryItem gains trackNames String[] (migration add_library_track_names):
the album's on-disk "## Title.ext" audio filenames.
- Captured at import (pipeline._import lists the final album folder) and at scan
(scan._record_owned now also refreshes trackNames on re-scan via ON CONFLICT
DO UPDATE + xmax=0 to preserve the newly-imported flag). New shared
library.list_audio_files() helper.
- /api/library exposes trackNames; the AlbumModal prefers on-disk tracks when
present (parsed "## Title" → position/title, zero network, labeled "On disk ·
N tracks"), falling back to the MusicBrainz listing otherwise. Items imported
before this column show the MB fallback until re-scanned.
worker 221 tests / 7-skip, web 153, tsc + build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A worker crash/restart mid-pipeline left its job stuck in matching/matched/
downloading/tagging with no auto-recovery — only the manual Retry (needs_attention
only) could rescue it. Now on startup the worker resets any such orphaned job
back to 'requested' so it gets re-claimed.
Safe at startup: this fresh worker owns none of those jobs, staging is already
cleared (clear_staging_root runs first), and the pipeline redoes intake→import
idempotently with an atomic import. Terminal jobs (imported/needs_attention) are
left untouched; claimedAt/error/downloadProgress are reset. Logged on startup.
worker 218 tests / 7-skip (in-flight→requested, terminal untouched, re-claimable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Floor bar used a file-count poller that jumps 0→~99% for small/fast albums
(streamrip reports only 0/1). But yt-dlp and slskd already emit real byte-level
progress via the on_progress callback the pipeline was discarding
(lambda _pct: None).
- Thread a throttled on_progress → Job.downloadProgress (write on a >=1% move or
every 0.5s; clamped 0..1; write failures swallowed). Called synchronously from
adapter.download on the pipeline thread, so it reuses the main conn safely.
- The file-count poller stays as the FALLBACK: it skips writing while a
`reports` flag is set (an adapter is driving real progress), and the flag is
cleared per attempt so a fall-through to streamrip still gets the estimate.
- Smooth bars for YouTube + Soulseek now; Qobuz/streamrip keeps the file-count
estimate until its per-track callback is hooked later.
worker 216 tests / 7-skip (on_progress write/throttle/clamp covered; existing
pipeline tests now exercise the fake's on_progress end-to-end).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The worker's MB browser (discography + artist search, used by monitor sweeps,
library scans, and Last.fm discovery) re-fetched every time. Route it through
the same ApiCache table + logical keys the web app uses, so a discography one
process fetched warms the other, and repeat lookups stop hitting MB.
- apicache.py cached_api(): read-through, serve-stale-on-outage, never caches
null, best-effort (a cache-DB error degrades to a live fetch, never breaks
scan/sweep/discovery). Its own DEDICATED autocommit connection so it can't
commit the worker's in-flight transaction, and autocommit avoids freezing
Postgres now() (which would break TTL math).
- CachingMbBrowser decorates MusicBrainzBrowser, serializing to the SAME
camelCase JSON shape the web writes (round-trip tested). registry.build_browser
wraps it when a DSN is present; the Last.fm source's browser rides it too.
- Hourly prune drops ApiCache rows > 90d so the table stays bounded.
- worker 213 tests / 7-skip.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.env.example shipped a valid, PUBLIC base64 key — if a user didn't replace it,
every credential got encrypted with a publicly-known key and everything still
"worked," so the footgun was silent.
- .env.example now ships an obvious placeholder (CHANGE_ME_generate_with_...)
with a "you MUST replace this" note and the openssl hint.
- web + worker validate LYRA_SECRET_KEY at startup: missing / placeholder /
wrong-length ⇒ loud FATAL banner and refuse to start (web via the Next.js
instrumentation hook, worker via _require_secret_key before touching the DB).
The old public example key ⇒ a loud WARNING but not fatal, since an existing
install may have encrypted its creds under it (rotating needs re-entry).
- Validation lives in an edge-safe web/src/lib/secret-key.ts (no node:crypto)
so the instrumentation hook bundles for both runtimes; crypto.ts re-exports.
- README emphasizes generating a fresh key + keeping it stable.
Verified live: worker exits 1 on placeholder/unset before any DB call; web
instrumentation throws and refuses to serve on the placeholder. web 138 tests,
worker 204/7-skip, tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SlskdClient enqueued the transfer and polled to Completed but never moved any
files into `dest` — slskd saves them to its OWN downloads dir, so staging stayed
empty and every Soulseek job failed "incomplete download". Qobuz/YouTube write
straight into staging and were unaffected.
- SlskdClient._retrieve() moves the finished files out of slskd's downloads dir
into staging after completion, matching tolerantly by basename and preferring
a parent-folder + size match (slskd's on-disk layout varies by version).
- download() returns the moved count as track_count so the pipeline's
completeness check reflects what actually landed; raises loudly if nothing was
found (misconfigured/unmounted downloads dir).
- Downloads dir resolves slskd.downloadsDir Config > SLSKD_DOWNLOADS_ROOT env >
/slskd-downloads default; docker-compose mounts ${SLSKD_DOWNLOADS_DIR} there.
- .env.example documents SLSKD_DOWNLOADS_DIR; retrieval unit-tested offline.
slskd runs as an external daemon, so the shared dir must be a host path both
slskd and the worker mount — set once Lyra runs on the slskd host. Not yet
verified live end-to-end (no slskd access from this VM).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add Job.downloadProgress (0.0-1.0), derived from a worker background
poller that counts audio files landing in staging / expected track
count, since streamrip only reports 0/1. No UI yet (slice B).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each search/download called asyncio.run, creating and closing a fresh event loop.
streamrip's module-global download-concurrency Semaphore binds to the loop on first
use, so the 2nd+ download reused a Semaphore bound to a closed loop -> "Semaphore is
bound to a different event loop", tracks failed, album arrived short -> "incomplete
download" / needs_attention. Run all streamrip coroutines on one persistent loop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_mark_chosen ran only after a download succeeded, so throughout the 'downloading'
state no candidate was chosen — the Floor row's "source · format · tracks" line was
blank (only elapsed showed). Now mark the candidate exclusively at each download
attempt so `chosen` reflects the source in flight (and updates on fall-through).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds LastfmSource, resolving Last.fm's often-empty artist mbid via the
injected MB browser (name->mbid cached) so every emitted SimilarArtist.mbid
is a real MusicBrainz artist MBID. Registered in build_similarity_sources,
gated on lastfm.api_key so the ListenBrainz-only path is unchanged when
unset.
Two row-leak bugs found in whole-branch review:
- discovery.run_discovery now prunes DiscoverySeedContribution rows whose
seedMbid is no longer a WatchedArtist (unfollow/removal has no FK cascade)
and recomputes the touched candidates' scores this sweep, instead of
letting a dead seed's contribution inflate the aggregate forever.
- maybe_run_scan's exception handler around scan_chunk now calls
clear_worklist and clears scan.id, matching the drain path. Previously
an exception mid-chunk left the already-done rows plus the remaining
worklist for that scan_id orphaned forever, since the next scan mints a
fresh uuid. Also clear scan.id in the build_worklist failure handler for
consistency.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace scan_chunk's O(N^2) tree re-walk-and-skip-to-cursor with a
persisted ScanWorkItem worklist: build_worklist walks the tree once at
scan start and inserts one row per album (idempotent via ON CONFLICT),
scan_chunk pops up to `limit` not-done rows by indexed cursor and marks
them done, and clear_worklist drops the scan's rows once drained.
scan_library is reimplemented on top of the worklist (one-shot scan_id,
drain, clear) with an unchanged signature/behavior. main.py's
maybe_run_scan now drives a generated scan.id through Config
(scan.id/scan.inProgress/scan.progress/scan.result) instead of a
scan.cursor string, so a mid-scan library change no longer risks
skipping or re-walking albums.
Reconciled the pre-existing test_config_from_config_parses_and_defaults
test in test_discovery.py, which referenced max_seeds/maxSeeds and would
otherwise have broken.
Adds a crash-recovery preamble that restores `final` from `.old` when a
prior swap was interrupted after rename(final, old) but before
rename(tmp, final), and always drops stale `.old`/`.importing` orphans.
Also guards the swap: if nothing new was staged into tmp (empty/missing
staging) and `final` already exists, leave it untouched instead of
swapping in an empty copy. This is required beyond the literal preamble
because the pre-existing swap logic unconditionally replaced `final`
with `tmp` regardless of whether anything was actually imported, which
would otherwise destroy a just-recovered (or any existing) copy whenever
staging is empty.
Staging is no longer forced under the library. The worker reads STAGING_ROOT
(container path, default /music/.staging) and stages per-job downloads there;
docker-compose mounts ${STAGING_DIR:-${MUSIC_DIR}/.staging} at /staging with
STAGING_ROOT=/staging. Set STAGING_DIR to a fast local disk when the library
(MUSIC_DIR) is a network share so temp download I/O stays off the share.
Safe cross-volume: import_album already assembles into a temp dir on the
library volume and swaps atomically there, so partial downloads never touch
the library and the final swap stays atomic. clear_staging_root now clears the
root's contents (mount-safe) rather than removing the root. Documented in
.env.example; new tests cover the separate-root path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Worker stamps a throttled worker.heartbeat; GET /api/worker reports online
from its updatedAt recency (DB-time); masthead shows a live listening/offline
dot instead of a static decoration.
- /api/requests enriched with rgMbid; Recently Pressed shows cover thumbs.
- Library album count moved from the filter row into a section header.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wanted releases pressed while the monitor is off never got reconciled, so
they lingered on the wanted list. Extract finish_job() and call it for every
finished job (not just when monitor.enabled) so the MonitoredRelease
transitions to fulfilled. Add fulfill_owned_releases(), run once at startup,
to backfill releases already owned at/above the quality cutoff (self-heals
items stuck from before this fix). Both idempotent + tested.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two worker-loop robustness fixes:
- Scan chunking: maybe_run_scan advances the library scan by at most one
scan_chunk (default scan.chunkSize=25 albums) per loop iteration, so a
large library no longer blocks job-claim/monitor for minutes-to-hours.
Progress persists in scan.inProgress/scan.cursor/scan.progress and is
resumable; scan.result (unchanged format) is written on completion.
- DB reconnect: wrap the loop body in `except psycopg.OperationalError`
(AdminShutdown subclasses it) to close the dead handle and reconnect via
wait_for_db() in-process, instead of crashing and relying on the
container restart policy. Verified via a backend-termination test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scan_chunk resolves album folders whose (artist, album) sorts after a
cursor, up to a limit, returning (new_cursor, imported, skipped, done).
scan_library becomes the unbounded case (limit=None), so its behavior and
tests are unchanged. Idempotent: all writes are ON CONFLICT, so an
overlapping/repeated cursor is safe. Enables chunked scanning in the loop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The scheduled-tick branch and the discover.requested one-shot branch
both read the same stale per-iteration config snapshot, so an enabled+due
sweep coinciding with a pending request ran discovery twice back-to-back
(idempotent but wasteful). Extract maybe_run_discovery, which unifies both
triggers into a single run and is now unit-tested.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
autoMonitorFuture flagged any new release, so it could auto-monitor a
newly-released live album or compilation. Gate it on is_core_release so
auto-monitoring matches the studio-only default the UI shows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The placeholder algorithm string (contribution_5_threshold_15_limit_50) is not a
permitted enum member of the LB Labs endpoint — every lookup 400'd, making discovery
inert. Verified against the live API 2026-07-12: the days_7500/contribution_3/
threshold_10/limit_100 variant returns rows shaped {artist_mbid,name,comment,score},
which the existing defensive parser already handles. Fixed in both the worker adapter
and the web seed-search client (kept in sync).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds registry.build_similarity_sources() (always constructs
ListenBrainzSource; reachability is a per-run health() concern, not a
startup gate) and wires discovery into the worker's main loop: a
scheduled sweep gated on discover.enabled + DISCOVER_TICK_SECONDS, and
a one-shot discover.requested trigger mirroring the existing
scan.requested pattern.
Also hardens run_discovery's health-check pass: a source whose
health() raises is now logged and skipped instead of aborting the
whole sweep (accepted finding from the Task-3 review).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MonitoredRelease rows created before an artist's discography was
populated (e.g. an owned album recorded before _populate_discography
ran) never got primaryType/secondaryTypes backfilled, wrongly
excluding them from the studio-only core view. _populate_discography
now DO UPDATEs the MB type metadata on conflict instead of DO NOTHING,
leaving monitored/state/currentQualityClass/firstGrabbedAt untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the per-artist release-type filter toggle: off (default) restricts
a discography view to core studio releases, on shows everything
(live, compilations, etc.).
A followed artist previously showed only the one owned album ("of 1")
because scan_library recorded just the resolved release. Now scan
browses each artist's full MusicBrainz release-group list once per
run and stores it as unmonitored MonitoredRelease rows, matching
web-follow behavior. The owned album keeps its monitored/fulfilled
state via ON CONFLICT DO NOTHING.
Probe each album exactly once and thread the ProbeResult through for
tags, quality, and format instead of hardcoding LibraryItem.format to
"FLAC" and re-probing on the tag-fallback path. Also skip album-level
directories starting with "." (previously only artist-level dirs were
skipped).
Jonathan