Jonathan/Lyra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: Jonathan/Lyra:b1f6408da6a27dad60b67a5481f429939ffe45ee
Branches Tags
Jonathan/Lyra:main Jonathan/Lyra:fix/slskd-resume-download Jonathan/Lyra:fix/slskd-large-album-backstop Jonathan/Lyra:worktree-fix-eta-overflow
..
compare: Jonathan/Lyra:fix/slskd-resume-download
Branches Tags
Jonathan/Lyra:fix/slskd-resume-download Jonathan/Lyra:fix/slskd-large-album-backstop Jonathan/Lyra:main Jonathan/Lyra:worktree-fix-eta-overflow

39 Commits
b1f6408da6 .. fix/slskd-resume-download

Author SHA1 Message Date
Jonathan 67efdedb51 WIP: slskd download resume (idempotent enqueue) — INCOMPLETE, do not deploy 
Adds the _needs_enqueue() helper (a file is (re)enqueued only if it has no
transfer yet or its last one ended Errored/Cancelled/Rejected). This is step 1
of making SlskdClient._download idempotent so a retry/hard-kill resumes instead
of re-downloading the whole album from scratch.

REMAINING (continue here):
1. Add SlskdClient._existing_states(username) -> {filename: state} (GET
   /api/v0/transfers/downloads/{username}, best-effort, {} on error).
2. In _download(): before the POST, compute
     existing = self._existing_states(username)
     to_enqueue = [f for f in files if _needs_enqueue(existing.get(f["filename"], ""))]
   POST only to_enqueue (skip the POST entirely when empty); poll loop unchanged.
   Effect: a completed prior copy is retrieved immediately; a partial one resumes.
3. Tests in test_slskd_cancel.py: resume-skips-completed (no POST, immediate
   retrieve) + resume-only-enqueues-missing (partial). Traced: the 3 existing
   download tests still pass (the extra pre-loop GET consumes poll0 and degrades
   gracefully) — but RUN the suite to confirm.
4. Merge to main, build+push worker image, deploy — but only AFTER the in-flight
   Drake "Scorpion" job (soulseekboy43) has imported, so the deploy's worker
   recreate doesn't interrupt it.

Deferred follow-up (bigger, needs candidate reconstruction): pipeline
peer-stickiness (reuse the persisted chosen candidate on retry instead of
re-searching) + cancel a reclaimed job's orphaned transfers at startup. That is
what fully stops the cross-peer duplicate the user saw.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-21 23:47:35 +02:00
Jonathan 395944f3e1 fix(worker): scale slskd download backstop to album size (stop abandoning big albums) 
The absolute download backstop was a flat _MAX_XFER_POLLS=1200 (~60min). A large
album (e.g. Drake "Scorpion", 521MB / 25 FLACs) from a slow-but-healthy serial
peer (~400KB/s, serving one file at a time) needs ~60-70min, so it was cut off
at 60min and — because an abandoned transfer is cancelled and re-enqueued from
scratch — re-downloaded from zero every attempt, never finishing and eventually
going needs_attention at the attempt cap.

Replace the flat cap with _max_polls(total_bytes): budget the backstop at a
conservative floor throughput (~64KB/s), clamped to [20min, 2h]. Scorpion now
gets the full 2h ceiling instead of 60min. The 90s stall timeout is unchanged,
so a truly dead/queued peer is still abandoned fast.

Follow-up (not in this change): resume across retries (skip files slskd already
completed) so an abandoned large download doesn't restart from zero.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-21 23:27:44 +02:00
Jonathan 10430037fe fix(worker): cap slskd ETA + rollback on progress-write failure (stop txn-abort job churn) 
A near-stalled Soulseek peer drives the EWMA speed toward zero, so
_eta_seconds returned int(remaining/speed) values far past int4 max. Writing
that to Job.downloadEtaSeconds (an integer column) raised "integer out of
range", which aborted the SHARED pipeline connection's transaction. The
on_progress except clause logged but never rolled back, so every subsequent
query cascaded "current transaction is aborted" and the whole job failed and
requeued (9 such failures / 24h observed in prod).

- Cap _eta_seconds at ~100h (359999s), well inside int4.
- Roll back the shared conn in on_progress's except so a failed progress
  write can never poison the pipeline txn, honoring the existing docstring
  promise that "a progress write must never fail the download".
- Regression test for the near-stall cap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-21 20:42:13 +02:00
Jonathan 01657d4e1b fix(web): order Recently Pressed by press time + cap at 10 
Recently Pressed was derived from the request list (ordered by Request.createdAt), so
an album requested long ago but imported just now sorted to the bottom — freshly
pressed albums never appeared near the top ("new albums don't get added"). Order by
press time (job.updatedAt ≈ import time) instead, and show only the 10 most recent.
The "Pressed" stat tile and the "N in library" note still reflect the true total; the
per-row date label now shows when it was pressed, not requested.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-20 23:17:54 +02:00
Jonathan 0ad1fd4cd5 fix(worker): trust the requested artist when it's in the file path (stop MP3-over-FLAC) 
Confidence scoring read the artist from the immediate folder as "Artist - Album", so
common FLAC rip layouts — artist in a parent folder ("Rihanna\(2009) Rated R\…") or
buried ("2009 - Rihanna - Rated R") — parsed with no/garbage artist and scored below
the 0.7 gate. Real FLACs got filtered out and a clean-named MP3 (0.88) won, so Lyra
downloaded lossy when lossless was available (Rihanna "Rated R": 8 FLACs all <0.5).

_parse_search_responses now takes the requested artist and, when it appears anywhere
in the candidate's path (via _path_matches), sets it as the candidate artist so
confidence reflects reality. Especially helps blocklisted-artist fallbacks, whose
album-only results come from varied folder structures. 337 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-20 23:05:38 +02:00
Jonathan 7ed8ca2848 fix(worker): prefer the original release-group over same-titled reissues 
The real root cause of "Hybrid Theory" re-downloading forever: MB lists the 2000
original and a 2023 reissue as SEPARATE same-titled Album release-groups. They tie on
(artist, title, is_album), so _best_release_group kept whichever MB relevance-ordered
first — the 2023 reissue (13 tracks / ~52 min). Its inflated tracklist+runtime then
failed the completeness/duration checks against the standard 12-track album sources
deliver, looping endlessly. Add an earliest-first-release-date tie-break so the
original group wins. (Complements _pick_release, which handles reissue releases
*within* a group.) 335 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-20 22:32:25 +02:00
Jonathan 5f59283bf0 fix(worker): resolve MB to the original edition + loosen duration gate 
Root cause of albums re-downloading forever despite completing (Linkin Park "Hybrid
Theory"): the MB resolver read the tracklist from an arbitrary releases[0], often a
later reissue/deluxe. HT resolved to a 2023 13-track reissue (~52 min); a complete
2000 12-track download (~38 min) then failed _download_problem's duration gate
(2267s < 2877s×0.85) and fell through peer after peer, never importing.

- _pick_release: choose the ORIGINAL standard edition (Official, earliest date)
  instead of releases[0], so completeness/duration are judged against the edition
  sources actually deliver.
- _DURATION_MIN_RATIO 0.85 -> 0.65: a source legitimately delivering a shorter
  edition than MB's release shouldn't be rejected; only genuinely truncated/preview
  downloads (a fraction of expected runtime) should be.

Together with the est-time peer ranking, this stops the download→reject→re-download
loop. 334 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-20 22:27:44 +02:00
Jonathan d2a86932e0 fix(worker): rank Soulseek peers by estimated transfer time, not raw speed 
Peer selection ranked (free-slot, advertised-speed, short-queue), ignoring file
size. But Soulseek FLAC rips of one album vary ~2x in size (a 24-bit/bloated rip vs
a standard CD rip) and Lyra scores them the SAME quality class (the slskd search
exposes no bit-depth, so every FLAC is class 2). So a peer advertising high speed
but serving 40MB/track files would out-rank a smaller standard rip that actually
finishes sooner — then blow past the 60-min per-peer backstop, time out, fall
through, and restart at 0% (Linkin Park "Hybrid Theory" looping for hours).

Rank by estimated transfer time instead: album bytes ÷ advertised speed, after the
free-slot check. Prefers whichever peer FINISHES first — fast peers and/or smaller
standard-edition rips — with no quality loss (all class 2). Equal-size case still
reduces to speed order, so existing ranking behaviour is preserved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-20 22:17:44 +02:00
Jonathan 73f1ae6e2f feat(web): order the In-queue tab next-up-first with a "Next up" marker 
The queue tab showed newest-first (API returns createdAt desc) but the worker claims
the oldest queued job next (claim_next: ORDER BY createdAt ASC) — so the item that ran
next was at the BOTTOM. Sort the queue bucket ascending so it matches processing
order, and badge the first claimable row (oldest, not individually paused) as "Next up".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-20 20:56:42 +02:00
Jonathan c49cee9a20 feat: download ETA on the press bar from measured Soulseek throughput 
Show a live "~Xm left" countdown on the download progress bar. The slskd download
loop already polls byte transfer every 3s; measure the actual throughput (EWMA-
smoothed bytes/sec), compute seconds-remaining from the album's total bytes, and
surface it. New nullable Job.downloadEtaSeconds (migration; web entrypoint runs
prisma migrate deploy), threaded through the on_progress callback (optional 2nd arg,
so other adapters/callers are unaffected — they report no ETA). API exposes it;
queue.tsx renders etaLabel() after the track count. Null when unknown (no sample yet
or a source that doesn't report bytes). Worker 330 tests, web 214 tests, both green;
verified live-rendered as "42% · 6/14 tracks · ~4m left".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-20 20:16:58 +02:00
Jonathan db8b43b0e8 feat(web): tabbed On-the-Press view + kebab menu for press actions 
The press page interleaved active downloads, queued items, and errors in one long
list — hard to see what's actually pressing. Split the non-imported rows into three
tabs (Active / In queue / Errors) with live counts; the default tab follows the work
(Active if anything's running, else Queue, else Errors) until the user picks one.
Move the press-level actions (Pause/Resume, Clear queue, Stop now) off the toolbar
into a vertical "⋮" kebab menu on the right of the header — Clear queue disables when
the queue is empty, Stop now only shows while downloading.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-20 19:41:11 +02:00
Jonathan 434cd2b22c feat(worker): recover blocklisted Soulseek searches via album/artist-only fallback 
The Soulseek server silently drops searches containing certain blocklisted terms
(major-label artists / album titles — Adele, Rihanna, Beyoncé, "Lemonade", …),
returning 0 responses ("TimedOut"). The blocked term poisons the whole query even
though the OTHER field usually isn't blocked and the content is on the network
(e.g. searching "Beyoncé …" → 0, but "I Am… Sasha Fierce" alone → 250 with 136
real Beyoncé folders).

When the primary "{artist} {album}" search returns nothing, retry with each field
alone (normalized to fold accents/punctuation) and keep only results whose file
path still matches the dropped field (_keep_matching), so a same-titled album by a
different artist is never grabbed. Recovers blocklisted albums with a distinctive
title; safely yields nothing when the title is too generic (e.g. Adele "21") rather
than mis-grabbing. Fallback only fires on a 0-response primary, so the happy path is
unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-20 19:06:07 +02:00
Jonathan d7ea29a97a fix(worker): requeue jobs that fail on an unhandled pipeline exception 
An unhandled exception in run_pipeline (e.g. a MusicBrainz TLS drop surfacing as
SSL: UNEXPECTED_EOF, after the MB retries are exhausted) left the job stranded in
its mid-pipeline 'matching' state: the except handler logged "pipeline failed" and
rolled back, but never reset the job. Since claim_next only picks 'requested', the
job was orphaned — stuck showing "Searching sources…" on the press until the next
worker restart's reclaim_stuck_jobs happened to sweep it up.

Add requeue_or_fail: on an unhandled failure, requeue to 'requested' for a bounded
retry (transient network blips clear on retry), then fall to 'needs_attention' at
the attempt cap (5) so a persistently-broken job stays visible instead of looping.
Mirrors reclaim_stuck_jobs (requeue) and pipeline._fail (needs_attention), and
resets Request.status to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-20 18:40:41 +02:00
Jonathan e8483ee22b fix(worker): match library ownership by rgMbid + retire stale failed attempts 
Two related monitor bugs surfaced by owned albums that never leave the press:

1. Ownership was matched on (artist, album) strings, but MusicBrainz canonicalizes
   the artist on import ("Kanye West"->"Ye", "Ne-Yo"->"Ne‐Yo", "The Goo Goo Dolls"
   ->"Goo Goo Dolls"). The strings no longer matched, so reconcile/enqueue/backfill
   never recognized the album as owned — it stayed "wanted" and re-pressed every
   retry interval. Match on the release-group MBID when the library row has one
   (falling back to artist+album for scanned rows without an MBID) across
   _in_library_at_cutoff, fulfill_owned_releases, and reconcile.

2. Each failed attempt left a needs_attention Request behind, so a hard-to-match
   album piled up phantom "needs attention" cards on the press even after a later
   attempt imported it (e.g. Maroon 5 "V": 3 stale cards + 1 success). Retire prior
   failed attempts when a release is fulfilled or re-attempted (_retire_failed_attempts,
   monitor-owned requests only — manual requests have no monitoredReleaseId).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-20 17:04:31 +02:00
Jonathan ca5bbfa9da fix(worker): retry transient MusicBrainz failures with backoff 
The matching stage makes up to three musicbrainzngs calls per job, none of
which retried. A dropped TLS connection (surfacing as NetworkError,
"<urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING]>") or a 503 rate-limit
(ResponseError) failed the entire pipeline for the job, which then re-queued
and re-ran from scratch — ~36% of jobs churned this way over a 6h window.

Wrap the three MB calls in _with_retry (4 attempts, 1s/2s/4s exponential
backoff), catching only the two transient musicbrainzngs error types so real
bugs still propagate. musicbrainzngs stays lazily imported to keep module
import offline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-20 16:45:37 +02:00
Jonathan 675d964bbe fix(worker): smarter Soulseek peer selection + stall/progress-aware transfers 
Draining the queue via Soulseek was stalling: slow peers (~100KB/s) never finished a
full FLAC album within the fixed 10-min timeout, got cancelled at ~89%, and the
fall-through restarted from scratch on the next peer — while single-song folders named
like the album were also being tried.

- Stall detection: abandon a peer with no byte progress for ~90s (queued/dead) instead
  of waiting the full timeout; a peer that IS progressing keeps its slot up to a 60-min
  backstop, so a slow-but-working transfer can finish.
- Rank search candidates by peer: free upload slot, then speed, then shortest queue.
- Drop candidates with far fewer tracks than the release (single-song folders) up front,
  so no download attempt is wasted on them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 19:54:14 +02:00
Jonathan 57f97e9838 feat(worker): defer Qobuz upgrades when it's gated instead of re-grabbing same quality 
With qualityCutoff=3 (want hi-res), a Soulseek-grabbed CD copy stays below cutoff and
is re-attempted for upgrade. Without a guard, when Qobuz is gated by pacing those
attempts would (a) re-search every owned album each monitor cycle and (b) re-download
a same-quality Soulseek copy that keep-if-better discards — pure churn.

Add an upgrade guard: a >= CD-lossless copy can only be improved by hi-res Qobuz, so
when Qobuz is gated, skip the job WITHOUT searching (pre-search skip); and as a safety
net, skip the download when the best-ranked candidate can't beat the existing copy.
Force jobs always proceed. Extracted the shared keep-existing path into a helper.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 19:15:43 +02:00
Jonathan 1fc61a0f5a fix(worker): cancel abandoned slskd transfers + cap download fall-through 
Two compounding causes of 'many downloads of one album from different sources' in
slskd: (1) the slskd client raised on timeout/error WITHOUT cancelling the transfer
it enqueued, so as the pipeline fell through to other peers each left a transfer
running; (2) the incomplete-download fall-through was uncapped, grinding through
every candidate of a popular album (~200 Soulseek peers), amplified now that Qobuz
pacing pushes most jobs to Soulseek.

- SlskdClient.download now cancels its enqueued transfers (DELETE) on any abandon.
- The download loop caps at _MAX_DOWNLOAD_ATTEMPTS (6) candidates.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 19:03:05 +02:00
Jonathan 67f374c470 feat: Qobuz pacing budget gate (human-like volume + cadence) 
Avoid re-triggering a USER_BLOCKED account by capping Qobuz to a human-looking
volume and cadence. A per-album gate (worker/lyra_worker/qobuz_gate.py) allows
Qobuz only within active hours, under today's cap (a warm-up ramp of a steady
cap), and past a randomized spacing since the last Qobuz download; the gap spreads
the day's budget across active hours with jitter. When the gate is closed the
pipeline drops Qobuz candidates so the job falls through to another source (and
monitored albums upgrade to Qobuz later); a Qobuz-only setup is never gated.
Config keys read live each loop; counters reset at the day boundary via the DB
clock. Tunable in Settings -> Qobuz -> Pacing (new /api/qobuz/pacing route).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 18:45:56 +02:00
Jonathan d47ed12e8e feat(worker): fall through to the next source on an incomplete download 
Completeness/duration verification was applied only to the first candidate that
downloaded successfully; if that copy was incomplete or too short the whole job
failed, even when a later-ranked source had a complete copy (e.g. John Mayer 'The
Search for Everything' — Qobuz reliably fails one hi-res track, so it never fell
back to the deluxe edition or Soulseek).

Extract the checks into a pure _download_problem() and run it INSIDE the download
loop: a candidate that downloads but fails verification now falls through to the
next-ranked source, and the job only fails (needs_attention) when every candidate
is incomplete — reporting the last shortfall reason.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 13:41:04 +02:00
Jonathan b0eec41823 fix(worker): judge download completeness by the source's edition, not MB's 
The completeness + duration guards used MusicBrainz's target.track_count /
total_duration_s, but MB picks an arbitrary releases[0] that is often a
deluxe/expanded edition with more tracks (and runtime) than the standard album a
source legitimately delivers. So complete standard-edition downloads were falsely
rejected — The Script 'No Sound Without Silence' (11 vs MB 12) as 'incomplete
download', Skillet 'Unleashed' (12 vs MB's 20-track deluxe) as 'too short'.

Now completeness is judged against the chosen source's own track count
(winner.track_count): a truncated download (fewer files than the source promised)
still fails, and the duration expectation is scaled to the delivered edition's
size. A separate 'implausibly small vs MB' guard (source has <= half MB's tracks)
still rejects a lone-track 'full album' video. Genuine partials (e.g. John Mayer,
a Qobuz track failing mid-download) still correctly fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 13:22:01 +02:00
Jonathan 108114ffa3 fix(web): replace stray null byte with space in Recently-Pressed dedupe key 
A literal NUL had crept into the `${r.artist} ${r.album}` dedupe key on the
Floor, making git treat queue.tsx as a binary file (unreviewable diffs). Restore
the intended space; behavior is unchanged (the key stays unique per album).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 12:44:22 +02:00
Jonathan 6451cf3a3c feat(web): Floor press + per-item controls (pause/resume/clear/stop/remove) 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 12:33:44 +02:00
Jonathan 927266d042 fix(mb): rank release groups by credited artist, not title alone 
The worker MB resolver picked release groups on (title_ratio, is_album)
and never scored the artist — even though resolve() knows the followed
artist. For a generic title, same-titled release groups by *different*
artists tie on title, and the is_album tiebreak (added in 06e7f91 for
Off the Wall) then actively preferred a foreign Album over the correct
release. Concretely, following Lake Street Dive's "Fun Machine" EP
resolved to an unrelated band "Bastards of Melody"'s same-titled Album,
which then propagated as the downloaded/imported artist.

Make credited-artist similarity the primary sort key, above title and
above is_album. It's a soft signal, never a hard filter, so credited-
name variations (feat., punctuation) still resolve; is_album now only
breaks ties within the same artist+title, preserving the Off the Wall
fix. Verified live: resolve("Lake Street Dive", "Fun Machine") now
returns the 6-track LSD EP.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 12:32:12 +02:00
Jonathan c6945bd89e feat(web): inter-job download delay setting (Settings → Monitor) 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 12:24:42 +02:00
Jonathan 3f624c3395 feat(web): DELETE /api/requests/[id] to remove a queued item 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 12:20:16 +02:00
Jonathan ecec0b6073 test(pause-api): harden "reject in-flight" test with durability assertion 
Strengthen test to prove the job's paused state remains unmodified when
pause/unpause is rejected for a downloading job. Tests now seeds paused=true,
attempts paused=false, verifies 400, and fetches from DB to assert paused is
still true — catching regressions that would mutate before the guard check.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 12:18:03 +02:00
Jonathan b78867435a feat(web): per-item pause route for queued jobs 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 12:14:45 +02:00
Jonathan 3d30e7dc8b feat(web): /api/floor pause/resume/clear/stop controls 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 12:09:45 +02:00
Jonathan 8c95ff735b feat(worker): pause gate, inter-job delay, and force-stop watcher in main loop 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 12:06:40 +02:00
Jonathan bff590a00f feat(worker): claim_next skips per-item paused jobs 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 12:01:38 +02:00
Jonathan 20869bdc13 feat(worker): floor pause/delay/kill-switch helpers 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 11:57:34 +02:00
Jonathan 9c2495825f feat(db): add Job.paused column for per-item press pause 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 11:54:19 +02:00
Jonathan 878eac066e docs: implementation plan for Press controls + inter-job delay 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 11:50:54 +02:00
Jonathan 603c62ef87 docs: design spec for Press controls + inter-job delay 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 11:43:09 +02:00
Jonathan 06e7f91897 fix(mb): resolve same-named albums to the Album, not the Single 
MusicBrainz search returns release-groups tied on title similarity in an
arbitrary order; `max(groups, key=_ratio)` kept the first, so Michael
Jackson's "Off the Wall" resolved to the 2-track *single* release-group
(Off the Wall / Get on the Floor) instead of the 10-track album. Because
plan_track_names labels files purely by position, that short tracklist got
mapped onto the full album — track 2 became "Get on the Floor" instead of
"Rock with You", and the real "Off the Wall"/"Get on the Floor" files kept
their streamrip names, yielding duplicate titles.

Extract selection into a pure `_best_release_group()` that breaks title-
similarity ties toward primary-type == "Album". Similarity still dominates;
album preference only settles ties. Live-verified: Off the Wall now resolves
to the 10-track album with track 2 = "Rock With You".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 10:49:26 +02:00
Jonathan 0443f10a3b feat(lastfm): disambiguation picker for same-named MB artists 
Last.fm's artist MBID can point at the wrong same-named band (it mapped
"Looking Glass" to an Australian stoner band, not the US pop group), and the
/lastfm rows trusted it verbatim — so opening/following grabbed the wrong
artist and the grabbed album never showed on the artist page. Now open/follow
resolve via MB search: resolveArtistChoice keeps candidates whose name matches
exactly — 1 match is used directly (even over a disagreeing Last.fm MBID), 2+
surface an ArtistPicker showing each MB disambiguation ("70's US pop group,
track 'Brandy'" vs "Australian stoner rock") with MB's top hit flagged
Suggested. Core decision logic unit-tested; no worker/schema change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 03:43:54 +02:00
Jonathan a8ea3441fc feat(floor): show "preparing…" bar during the pre-download window 
A Qobuz login/metadata/artwork phase (and a Soulseek peer-queue or YouTube
info-fetch) can sit at 0% for a few minutes before byte progress flows, which
read as a stuck bar. While a downloading job's progress is still 0, render the
indeterminate "preparing…" bar (same style as the searching/finishing phases);
it flips to the determinate % bar on the first reported byte and never back
(the progress aggregator only increases from 0).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 03:43:54 +02:00
Jonathan ccaaa41fec docs: slim README for self-hosting, move advanced topics to docs/DEPLOYMENT.md 
Restructure the docs ahead of sharing the project for others to self-host. The
README is now a focused get-it-running path (overview, architecture, prereqs,
quick start, operational notes, security, backup/restore). Moved the fuller
feature tour, the maintainer pre-built-image release flow, and the Gluetun VPN
guide into a new docs/DEPLOYMENT.md, cross-linked both ways. The pre-built-image
section is now labelled a maintainer process with a LYRA_REGISTRY override so a
self-hoster can point at their own registry (or just build from source).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 02:57:45 +02:00
52 changed files with 4604 additions and 308 deletions
Expand all files Collapse all files
README.md
+75 -190
View File
@@ -7,83 +7,54 @@ sources of differing quality: **Qobuz** (lossless/hi-res), **Soulseek** (P2P), a
**ListenBrainz** and **Last.fm** (Spotify's recommendation/related-artists APIs were
retired for new apps, so discovery is MetaBrainz-native and MBID-clean — no Spotify).
## Status
It runs on your own server as a small `docker compose` stack. This README covers getting
it running; see **[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md)** for the full feature tour,
publishing pre-built images, and running behind a VPN.
Built in three slices:
## What it does
1. **Acquisition engine** — ✅ **done**. Resolve a request against MusicBrainz,
match/rank across Qobuz/Soulseek/YouTube, download the best source into an
isolated staging dir, verify completeness, tag (mutagen), and atomically promote
the clean album into `Artist/Album (Year)/## Title.ext`.
2. **Library manager** — ✅ **done**. Watched artists (live MB search + follow),
per-release monitor toggles + auto-monitor-future, a wanted list with retry +
quality-upgrade window, and a worker "monitor" that discovers new releases and
auto-grabs through the acquisition pipeline. Plus a **library scan** that ingests
existing on-disk albums as have + monitored + followed, and a per-artist
studio-only / show-all release-type filter.
3. **Discovery** — ✅ **done**. Recommendation-driven finding of new music via
**ListenBrainz** similar-artists and **Last.fm** `artist.getSimilar` (both
MBID-native), behind a pluggable `SimilaritySource` interface — Last.fm registers
as a second source automatically once its API key is set, and cross-source scores
are summed per candidate. A background sweep (and a "Discover now" button)
aggregates artists similar to the ones you follow into a manual review feed of
suggested **artists** and **albums** at `/discover`; **Follow** and **Want**
reuse the slice-2 follow / wanted flows, **Dismiss** is permanent. Plus a
web-side seed-search box. Every suggestion and seed-search result opens a
read-only **artist preview** (`/discover/artist/[mbid]`) — discography,
per-album tracklists, and "in library" / "monitored" badges — where you can
Follow or Want before committing. No Spotify (its recommendation APIs are dead
for new apps).
- **Acquisition** — resolve a request against MusicBrainz, match/rank across
Qobuz/Soulseek/YouTube, download the best source, verify completeness, tag, and organize
into `Artist/Album (Year)/## Title.ext`.
- **Library manager** — follow artists (live MB search), per-release monitor toggles, a
wanted list with retry + quality-upgrade, a background monitor that auto-grabs new
releases, and a scan that ingests albums already on disk.
- **Discovery** (`/discover`) — recommendation-driven suggestions from ListenBrainz +
Last.fm similar-artists, with Follow / Want / Dismiss and a read-only artist preview.
- **Browse & watch** — Last.fm top artists/albums (`/lastfm`, with inline Follow/Want), a
cover-art **Library** grid (`/library`), and **The Floor** (home): a live queue of
in-flight acquisitions with three-phase progress and a Retry button.
Alongside the three slices:
- **Last.fm browse** (`/lastfm`) — your Last.fm top artists and albums (by period),
with a "hide items already in my library" toggle; each artist opens a modal of your
most-played songs and albums, and every album is one click to **Want**.
- **Library** (`/library`) — a cover-art grid of everything on disk, filterable, each
album opening a tracklist modal.
- **The Floor** (home) — a live queue of in-flight acquisitions with three-phase
progress (**Searching → Downloading → Finishing**), a live download percentage, a
per-job "pressing ticket" (source · format · tracks · elapsed), and a **Retry**
button on anything that needs attention.
The whole UI runs a hand-rolled "Pressing Plant" design system (warm-paper theme, self-hosted
Fraunces serif for prose vs. system mono for machine data, light/dark aware).
**Operational notes:** the monitor and the discovery sweep are both **off by
default** — set `monitor.enabled=true` (or Settings → Monitor, + optional `monitor.*`
tuning) to activate auto-grab/upgrade, and `discover.enabled=true` (or
Settings → Discovery, + optional `discover.*` tuning) to activate the scheduled
discovery sweep. The `/discover` "Discover now" button and seed-search work with the
sweep off. Credentials (Qobuz, Soulseek, Last.fm) are entered in Settings and persist
across `docker compose up -d --build` rebuilds (don't use `down -v`; secrets are
encrypted at rest with `LYRA_SECRET_KEY`). Tests run against a separate `lyra_test`
database (a guard blocks the live DB). Deferred hardening + the full roadmap live in
the implementation-plan "Notes" sections under
[`docs/superpowers/plans/`](docs/superpowers/plans/).
Full detail on each of these is in [docs/DEPLOYMENT.md → Features in depth](docs/DEPLOYMENT.md#features-in-depth).
## Architecture
The `docker compose` stack is three services on a home server:
The `docker compose` stack is a few services on one host:
- **db** — Postgres, the shared source of truth.
- **db** — Postgres, the shared source of truth (all durable state lives here).
- **web** — the Next.js app: UI + API (search, request, queue, progress, settings).
- **worker** — the Python worker: all source integration (streamrip, yt-dlp, slskd
client, mutagen; MusicBrainz + ListenBrainz + Last.fm for metadata and discovery),
running the six-stage acquisition pipeline plus the background monitor and discovery
sweeps.
- **worker** — the Python worker: all source integration (streamrip, yt-dlp, slskd client,
mutagen; MusicBrainz + ListenBrainz + Last.fm for metadata and discovery), running the
acquisition pipeline plus the background monitor and discovery sweeps.
- **db-backup** — a sidecar that `pg_dump`s the database on a schedule (see [Backup &
restore](#backup--restore)).
Soulseek support talks to an **external slskd** daemon (an off-the-shelf headless
Soulseek client) — the worker reaches it via the `slskd.url` + `slskd.api_key`
credentials set in Settings, so run slskd separately rather than as part of this stack.
Soulseek support talks to an **external slskd** daemon (an off-the-shelf headless Soulseek
client) — the worker reaches it via the `slskd.url` + `slskd.api_key` credentials set in
Settings, so run slskd separately rather than as part of this stack.
A shared `/music` volume is the library, with a separate `/staging` volume for
in-progress downloads (point `STAGING_DIR` at a fast local disk when `MUSIC_DIR` is a
network share).
A shared `/music` volume is the library, with a separate `/staging` volume for in-progress
downloads (point `STAGING_DIR` at a fast local disk when `MUSIC_DIR` is a network share).
See [`docs/superpowers/specs/`](docs/superpowers/specs/) for the full design.
## Prerequisites
## Running it
- **Docker** + **Docker Compose**.
- A directory for the music library (set `MUSIC_DIR`).
- Optional, entered in **Settings** after first run: a **Qobuz** account (lossless/hi-res),
a running **slskd** daemon (Soulseek), and a **Last.fm** API key (browse + discovery).
None are required to start — Lyra runs with whatever you configure.
## Quick start
```sh
cp .env.example .env # then edit: set LYRA_SECRET_KEY (openssl rand -base64 32) and MUSIC_DIR
@@ -97,33 +68,26 @@ the placeholder. Once set, keep it stable: changing it makes already-stored cred
undecryptable (you'd re-enter them in Settings). Optionally set `LYRA_PASSWORD` too (see
[Security](#security)).
The web entrypoint runs `prisma migrate deploy` on every rebuild, so schema changes
reach the live DB automatically. Enter Qobuz / Soulseek / Last.fm credentials in
**Settings** (they persist across rebuilds), then follow an artist or request an album
to feed The Floor.
The web entrypoint runs `prisma migrate deploy` on every rebuild, so schema changes reach
the live DB automatically. Enter Qobuz / Soulseek / Last.fm credentials in **Settings** (they
persist across rebuilds), then follow an artist or request an album to feed The Floor.
## Deploying to a server (pre-built images)
`.env.example` documents every setting (`MUSIC_DIR`, `STAGING_DIR`, `SLSKD_DOWNLOADS_DIR`,
backup knobs). Note in particular: point `SLSKD_DOWNLOADS_DIR` at slskd's own downloads
directory (Lyra must run on the same host as slskd for Soulseek delivery to work), and point
`STAGING_DIR` at a fast local disk if `MUSIC_DIR` is a network share.
For a server that should **pull** rather than build, the `web` and `worker` images are
published to the Gitea container registry and consumed by a compose file in the
[`vm-download`](https://git.jger.nl/Jonathan/vm-download) ops repo (`docker/lyra/`).
## Operational notes
Publish a new release from this repo (amd64):
```sh
docker login git.jger.nl # once — Gitea user + a token with package:write
./scripts/build-push.sh # builds + pushes lyra-web / lyra-worker (:latest and :<sha>)
```
Then on the server:
```sh
cd /opt/git/vm-download && git pull
cd docker/lyra && docker compose pull && docker compose up -d
```
`db` (Postgres) and `db-backup` use the stock `postgres:17` image, so only the two custom
images are built here. See `docker/lyra/README.md` in the ops repo for first-time setup.
- The **monitor** and the **discovery sweep** are both **off by default** — set
`monitor.enabled=true` (or Settings → Monitor, + optional `monitor.*` tuning) to activate
auto-grab/upgrade, and `discover.enabled=true` (or Settings → Discovery, + optional
`discover.*` tuning) to activate the scheduled discovery sweep. The `/discover` "Discover
now" button and seed-search work with the sweep off.
- **Credentials persist** across `docker compose up -d --build` rebuilds — don't use
`down -v` (it wipes the Postgres volume); secrets are encrypted at rest with
`LYRA_SECRET_KEY`.
- **Tests** run against a separate `lyra_test` database (a guard blocks the live DB).
## Managing the library
@@ -133,6 +97,24 @@ it — re-Want it from the artist page if you change your mind). This is why the
also bind-mounts `MUSIC_DIR` at `/music` (read-write) — deletion is guarded to only ever
remove folders strictly inside the library root.
## Security
Lyra has **no per-user accounts**. Two layers protect it:
- **Shared-password gate.** Set `LYRA_PASSWORD` in `.env` to require a single shared
password for the entire UI and API. A request without a valid session cookie is
redirected to `/login` (or gets `401` for `/api/*`). The cookie is httpOnly and carries
an HMAC of the password keyed by `LYRA_SECRET_KEY`, so it can't be forged. **If
`LYRA_PASSWORD` is unset, the app is completely open** — anyone who can reach the host
can browse the library, trigger downloads, and read/overwrite the stored credentials in
Settings. Set it for any deployment that isn't on a fully trusted, isolated network.
- **Network.** The shared password is deliberately minimal (single password, bearer
cookie, plain HTTP on the LAN). For anything internet-facing, also put Lyra behind a
reverse proxy with TLS and/or a VPN or auth gateway — don't rely on the shared password
alone. See [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#running-behind-a-vpn-gluetun) for a
Gluetun VPN setup.
## Backup & restore
**All** durable state — library metadata, monitored/wanted lists, discovery data,
@@ -163,105 +145,8 @@ docker compose start web worker
is safe. To restore onto a brand-new host, bring up just `db` first
(`docker compose up -d db`), restore, then `up -d --build` the rest.
## Security
## More
Lyra has **no per-user accounts**. Two layers protect it:
- **Shared-password gate.** Set `LYRA_PASSWORD` in `.env` to require a single shared
password for the entire UI and API. A request without a valid session cookie is
redirected to `/login` (or gets `401` for `/api/*`). The cookie is httpOnly and carries
an HMAC of the password keyed by `LYRA_SECRET_KEY`, so it can't be forged. **If
`LYRA_PASSWORD` is unset, the app is completely open** — anyone who can reach the host
can browse the library, trigger downloads, and read/overwrite the stored credentials in
Settings. Set it for any deployment that isn't on a fully trusted, isolated network.
- **Network.** The shared password is deliberately minimal (single password, bearer
cookie, plain HTTP on the LAN). For anything internet-facing, also put Lyra behind a
reverse proxy with TLS and/or a VPN or auth gateway — don't rely on the shared password
alone.
## Running behind a VPN (Gluetun)
Lyra runs fine with its traffic routed through a [Gluetun](https://github.com/qdm12/gluetun)
VPN container — useful for hiding the download traffic (Soulseek / Qobuz / YouTube). Nothing
in Lyra is VPN-incompatible; the only thing that changes is container networking, because a
service using `network_mode: "service:gluetun"` gives up its own network stack and shares
Gluetun's. Two consequences:
- Containers that share Gluetun's namespace reach each other over **`localhost`**, not by
Docker service name — so `DATABASE_URL` changes from `@db:5432` to **`@localhost:5432`**.
- **Published ports move to the `gluetun` container** (the individual services can no longer
publish their own).
The simplest, most consistent setup is to put the **whole stack** behind Gluetun. Sketch
(merge into your existing Gluetun compose; adjust the VPN provider block to yours):
```yaml
services:
gluetun:
image: qmcgaw/gluetun
cap_add: [NET_ADMIN]
environment:
# ... your VPN provider / credentials ...
FIREWALL_INPUT_PORTS: "3000" # let your browser reach the web UI
FIREWALL_OUTBOUND_SUBNETS: "192.168.0.0/16" # your LAN — so the worker can reach slskd
ports:
- "8770:3000" # Lyra web (published on gluetun, not on `web`)
# - "5432:5432" # only if you want DB access from the LAN
web:
build: ./web
network_mode: "service:gluetun" # no `ports:` / `networks:` here
environment:
DATABASE_URL: postgresql://lyra:lyra@localhost:5432/lyra # `db` -> `localhost`
LYRA_SECRET_KEY: ${LYRA_SECRET_KEY}
LYRA_PASSWORD: ${LYRA_PASSWORD:-}
HOSTNAME: 0.0.0.0
LYRA_LIBRARY_ROOT: /music
volumes:
- ${MUSIC_DIR:-./music}:/music
depends_on:
gluetun: { condition: service_healthy }
db: { condition: service_healthy }
worker:
build: ./worker
network_mode: "service:gluetun"
environment:
DATABASE_URL: postgresql://lyra:lyra@localhost:5432/lyra
LYRA_SECRET_KEY: ${LYRA_SECRET_KEY}
STAGING_ROOT: /staging
SLSKD_DOWNLOADS_ROOT: /slskd-downloads
volumes:
- ${MUSIC_DIR:-./music}:/music
- ${STAGING_DIR:-${MUSIC_DIR:-./music}/.staging}:/staging
- ${SLSKD_DOWNLOADS_DIR:-./slskd-downloads}:/slskd-downloads
depends_on:
gluetun: { condition: service_healthy }
db: { condition: service_healthy }
db:
image: postgres:17
network_mode: "service:gluetun" # shares the namespace so `localhost:5432` resolves
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: ${POSTGRES_DB}
volumes:
- postgres-data:/var/lib/postgresql/data
```
(Give `db-backup` the same `network_mode` and `PGHOST: localhost`.) The `HOSTNAME=0.0.0.0`
and the built-in healthchecks keep working, since everything is in one namespace.
Caveats worth knowing:
- **Qobuz is geo-sensitive** — its catalog/availability varies by country, so pick a VPN
exit in a region where your Qobuz account works, or some downloads will fail to match.
- The worker's `net.ipv6.conf.all.disable_ipv6` sysctl **can't apply behind Gluetun** (a
shared netns can't set sysctls) — harmless, because Gluetun blocks IPv6 anyway, which is
exactly what that sysctl worked around (Qobuz's CDN resolving to an unroutable IPv6).
- MusicBrainz rate-limits per IP; a shared VPN exit is marginally more likely to hit the
1 req/s ceiling, but the built-in response cache makes that a non-issue in normal use.
- Bind mounts (`/music`, `/staging`, `/slskd-downloads`), the Postgres volume, and backups
are filesystem, not network — **completely unaffected** by the VPN.
- **[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md)** — full feature tour, publishing pre-built
images to a registry, and running the whole stack behind a Gluetun VPN.
- **[docs/superpowers/specs/](docs/superpowers/specs/)** — the full design specs.
docs/DEPLOYMENT.md
+177
View File
@@ -0,0 +1,177 @@
# Lyra — deployment & advanced guide
Topics beyond the [README](../README.md) quick-start: the full feature tour, publishing
pre-built images to a registry, and running the whole stack behind a VPN. For a basic
self-hosted setup you only need the README — start there.
- [Features in depth](#features-in-depth)
- [Publishing pre-built images](#publishing-pre-built-images)
- [Running behind a VPN (Gluetun)](#running-behind-a-vpn-gluetun)
---
## Features in depth
Lyra was built in three slices, all complete:
1. **Acquisition engine.** Resolve a request against MusicBrainz, match/rank across
Qobuz/Soulseek/YouTube, download the best source into an isolated staging dir, verify
completeness, tag (mutagen), and atomically promote the clean album into
`Artist/Album (Year)/## Title.ext`.
2. **Library manager.** Watched artists (live MB search + follow), per-release monitor
toggles + auto-monitor-future, a wanted list with retry + quality-upgrade window, and a
worker "monitor" that discovers new releases and auto-grabs through the acquisition
pipeline. Plus a **library scan** that ingests existing on-disk albums as
have + monitored + followed, and a per-artist studio-only / show-all release-type filter.
3. **Discovery.** Recommendation-driven finding of new music via **ListenBrainz**
similar-artists and **Last.fm** `artist.getSimilar` (both MBID-native), behind a
pluggable `SimilaritySource` interface — Last.fm registers as a second source
automatically once its API key is set, and cross-source scores are summed per candidate.
A background sweep (and a "Discover now" button) aggregates artists similar to the ones
you follow into a manual review feed of suggested **artists** and **albums** at
`/discover`; **Follow** and **Want** reuse the slice-2 follow / wanted flows, **Dismiss**
is permanent. Plus a web-side seed-search box. Every suggestion and seed-search result
opens a read-only **artist preview** (`/discover/artist/[mbid]`) — discography, per-album
tracklists, and "in library" / "monitored" badges — where you can Follow or Want before
committing. No Spotify (its recommendation APIs are dead for new apps).
Alongside the three slices:
- **Last.fm browse** (`/lastfm`) — your Last.fm top artists and albums (by period), with a
"hide items already in my library" toggle and inline **Follow** / **Want** buttons on each
row; each artist also opens a modal of your most-played songs and albums.
- **Library** (`/library`) — a cover-art grid of everything on disk, filterable, each album
opening a tracklist modal.
- **The Floor** (home) — a live queue of in-flight acquisitions with three-phase progress
(**Searching → Downloading → Finishing**), a live download percentage, a per-job "pressing
ticket" (source · format · tracks · elapsed), and a **Retry** button on anything that needs
attention.
The whole UI runs a hand-rolled "Pressing Plant" design system (warm-paper theme, self-hosted
Fraunces serif for prose vs. system mono for machine data, light/dark aware).
Deferred hardening and the full roadmap live in the implementation-plan "Notes" sections under
[`docs/superpowers/plans/`](superpowers/plans/); the full design specs are under
[`docs/superpowers/specs/`](superpowers/specs/).
---
## Publishing pre-built images
> **Maintainer release process** — this is how the project maintainer publishes images to a
> private registry. If you're self-hosting your own copy, ignore this and just build from
> source with `docker compose up -d --build` (see the README). You'd only need this if you
> run your own registry and want servers to **pull** rather than build.
The `web` and `worker` images can be published to a container registry and consumed by a
compose file that pulls instead of builds. The maintainer publishes to a Gitea container
registry consumed by a compose file in the
[`vm-download`](https://git.jger.nl/Jonathan/vm-download) ops repo (`docker/lyra/`).
Publish a new release from this repo (amd64):
```sh
docker login <your-registry> # once — registry user + a token with package:write
./scripts/build-push.sh # builds + pushes lyra-web / lyra-worker (:latest and :<sha>)
```
`scripts/build-push.sh` defaults the registry namespace to `git.jger.nl/jonathan`; override it
with `LYRA_REGISTRY=<your-registry>/<namespace>`.
Then on the server:
```sh
cd /opt/git/vm-download && git pull
cd docker/lyra && docker compose pull && docker compose up -d
```
`db` (Postgres) and `db-backup` use the stock `postgres:17` image, so only the two custom
images are built here.
---
## Running behind a VPN (Gluetun)
Lyra runs fine with its traffic routed through a [Gluetun](https://github.com/qdm12/gluetun)
VPN container — useful for hiding the download traffic (Soulseek / Qobuz / YouTube). Nothing
in Lyra is VPN-incompatible; the only thing that changes is container networking, because a
service using `network_mode: "service:gluetun"` gives up its own network stack and shares
Gluetun's. Two consequences:
- Containers that share Gluetun's namespace reach each other over **`localhost`**, not by
Docker service name — so `DATABASE_URL` changes from `@db:5432` to **`@localhost:5432`**.
- **Published ports move to the `gluetun` container** (the individual services can no longer
publish their own).
The simplest, most consistent setup is to put the **whole stack** behind Gluetun. Sketch
(merge into your existing Gluetun compose; adjust the VPN provider block to yours):
```yaml
services:
gluetun:
image: qmcgaw/gluetun
cap_add: [NET_ADMIN]
environment:
# ... your VPN provider / credentials ...
FIREWALL_INPUT_PORTS: "3000" # let your browser reach the web UI
FIREWALL_OUTBOUND_SUBNETS: "192.168.0.0/16" # your LAN — so the worker can reach slskd
ports:
- "8770:3000" # Lyra web (published on gluetun, not on `web`)
# - "5432:5432" # only if you want DB access from the LAN
web:
build: ./web
network_mode: "service:gluetun" # no `ports:` / `networks:` here
environment:
DATABASE_URL: postgresql://lyra:lyra@localhost:5432/lyra # `db` -> `localhost`
LYRA_SECRET_KEY: ${LYRA_SECRET_KEY}
LYRA_PASSWORD: ${LYRA_PASSWORD:-}
HOSTNAME: 0.0.0.0
LYRA_LIBRARY_ROOT: /music
volumes:
- ${MUSIC_DIR:-./music}:/music
depends_on:
gluetun: { condition: service_healthy }
db: { condition: service_healthy }
worker:
build: ./worker
network_mode: "service:gluetun"
environment:
DATABASE_URL: postgresql://lyra:lyra@localhost:5432/lyra
LYRA_SECRET_KEY: ${LYRA_SECRET_KEY}
STAGING_ROOT: /staging
SLSKD_DOWNLOADS_ROOT: /slskd-downloads
volumes:
- ${MUSIC_DIR:-./music}:/music
- ${STAGING_DIR:-${MUSIC_DIR:-./music}/.staging}:/staging
- ${SLSKD_DOWNLOADS_DIR:-./slskd-downloads}:/slskd-downloads
depends_on:
gluetun: { condition: service_healthy }
db: { condition: service_healthy }
db:
image: postgres:17
network_mode: "service:gluetun" # shares the namespace so `localhost:5432` resolves
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: ${POSTGRES_DB}
volumes:
- postgres-data:/var/lib/postgresql/data
```
(Give `db-backup` the same `network_mode` and `PGHOST: localhost`.) The `HOSTNAME=0.0.0.0`
and the built-in healthchecks keep working, since everything is in one namespace.
Caveats worth knowing:
- **Qobuz is geo-sensitive** — its catalog/availability varies by country, so pick a VPN
exit in a region where your Qobuz account works, or some downloads will fail to match.
- The worker's `net.ipv6.conf.all.disable_ipv6` sysctl **can't apply behind Gluetun** (a
shared netns can't set sysctls) — harmless, because Gluetun blocks IPv6 anyway, which is
exactly what that sysctl worked around (Qobuz's CDN resolving to an unroutable IPv6).
- MusicBrainz rate-limits per IP; a shared VPN exit is marginally more likely to hit the
1 req/s ceiling, but the built-in response cache makes that a non-issue in normal use.
- Bind mounts (`/music`, `/staging`, `/slskd-downloads`), the Postgres volume, and backups
are filesystem, not network — **completely unaffected** by the VPN.
docs/superpowers/plans/2026-07-15-press-controls.md
+1189
View File
File diff suppressed because it is too large Load Diff
docs/superpowers/specs/2026-07-15-press-controls-design.md
+179
View File
@@ -0,0 +1,179 @@
# Press controls + inter-job delay — design
Date: 2026-07-15
## Problem
The monitor can enqueue a large backlog of acquisition jobs at once. The worker drains
them serially with **no pacing** (only a 2s idle poll when the queue is empty) and offers
**no runtime control** over the queue — the only per-job action on the Floor is Retry (on
`needs_attention`). The user wants:
1. A **configurable delay between downloads** to spread sustained Qobuz load over time.
2. **Per-item controls** on the Floor ("the Press"): pause and remove a queued item.
3. **Whole-press controls**: pause/resume, clear the queue, and a force-stop of the
currently-downloading album.
## Constraints (from the existing code)
- The worker is a **single, strictly-serial process** (`worker/lyra_worker/main.py:256-298`).
One job runs to completion in a blocking `run_pipeline` call before the loop does anything
else.
- **Config is read fresh every loop iteration** (`main.py:262`, `get_config`), so any Config
change takes effect within ~2s with no restart.
- An **in-flight download cannot be cleanly interrupted** — `run_pipeline` and the adapter
download are synchronous/blocking with only an `on_progress` callback, no cancel token. The
only interruption is process death, after which `reclaim_stuck_jobs` (`claim.py:8-24`)
requeues the in-flight job at startup and `clear_staging_root` (`main.py:233`) wipes the
partial download.
- `claim_next` (`claim.py:27-57`) claims the oldest `state='requested'` job.
- Deleting a `Request` cascades to its `Job` and `Candidate` rows (`schema.prisma:51,66`); the
linked `MonitoredRelease` is `SetNull`, not cascaded.
- `enqueue_due` (`monitor.py:115-147`) re-enqueues a monitored release once it has no live
(non-terminal) job and its `lastSearchedAt` is older than `retryIntervalHours` (default 6).
So a removed/cleared monitored item reappears later on the normal cadence unless `ignored`.
- The web container runs `prisma migrate deploy` on startup (`web/docker-entrypoint.sh`), so a
new migration auto-applies on deploy.
## Decisions (confirmed with the user)
- **Active item:** add a global **force-kill** ("Stop now") that aborts the in-flight download
via worker process-restart. Per-item controls act only on queued items.
- **Remove / Clear semantics:** **cancel-for-now**. Manual one-off requests are gone for good;
monitored/wanted releases stay wanted and may reappear on the monitor's normal cadence.
Permanent skip remains the existing per-release **Ignore**.
- **Pause scope:** **downloads only**. Pausing halts claiming/downloading; the monitor keeps
finding and enqueueing releases (they pile up in the queue) and Resume drains them.
## Data model
- **New column** `Job.paused Boolean @default(false)` — a Prisma migration
(`add_job_paused`). Auto-applies on deploy.
- **Config keys** (key/value `Config` rows, no migration):
- `floor.paused` — `"true"`/`"false"`; global press pause.
- `floor.jobDelaySeconds` — integer string, default `"0"`; delay between downloads.
- `floor.killRequested` — `"true"`/`"false"`; one-shot force-kill signal, cleared by the
worker on consumption.
## Worker changes
`worker/lyra_worker/main.py` and `worker/lyra_worker/claim.py`.
1. **Inter-job delay.** After `finish_job` (`main.py:297`), read `floor.jobDelaySeconds` from
the already-fresh `config` and sleep that many seconds before the next loop iteration.
Sleep in small (~2s) chunks so the loop stays responsive; the force-kill watcher runs on a
separate thread and works regardless. Parse with the existing `_int`-style guard (invalid →
0). A value of 0 preserves today's back-to-back behavior.
2. **Global pause.** Immediately before `claim_next` (`main.py:286`), if `floor.paused` is
truthy (`_TRUE` idiom), skip claiming: `time.sleep(IDLE_SLEEP); continue`. Because the loop
is serial, any in-flight job has already finished before this check — pause stops the
*next* download. Monitor sweep / scan / discovery (above the claim in the loop body)
continue to run, so the queue keeps filling while paused.
3. **Per-item pause.** `claim_next` `WHERE` gains `AND NOT "paused"`. A paused queued job is
skipped by the claim but remains `state='requested'`, so it is not "lost" and resumes
claiming as soon as `paused` is cleared. `reclaim_stuck_jobs` is unaffected (it only touches
in-flight states).
4. **Force-kill watcher.** A daemon thread started in `run_forever` (modeled on
`_liveness_heartbeat`, `main.py:216-226`) polls `floor.killRequested` on its own DB
connection every ~1.5s. When set, it clears the flag (commit) and calls `os._exit(0)`. The
container restart policy brings the worker back; startup `clear_staging_root` +
`reclaim_stuck_jobs` wipe the partial download and requeue the aborted job. The "Stop now"
action also sets `floor.paused=true`, so the worker returns **idle** rather than instantly
re-downloading the aborted album; the user resumes when ready.
## Infrastructure (ops repo)
The prod worker's Docker restart policy is currently **`no`** (`docker inspect` →
`RestartPolicy=no`), so process death — from force-kill *or any crash* — leaves it down. The
force-kill design requires `restart: unless-stopped` on the `worker` service in the ops repo
(`/opt/git/vm-download/docker/lyra/compose.yaml`). The in-repo `docker-compose.yml` already has
this (`docker-compose.yml:53`); this change brings prod in line and is good hygiene
independent of this feature. This is a one-line change in a separate repo and must be applied
before the force-kill feature is usable in prod.
## API (web) — follows existing `retry` / `library DELETE` patterns
- **`GET /api/floor`** → `{ paused: boolean, jobDelaySeconds: number }` (reads the Config
rows). Used by the Floor to render the paused banner and control states.
- **`POST /api/floor`** with `{ action }`:
- `"pause"` / `"resume"` → upsert `floor.paused`.
- `"clear"` → delete every `Request` whose `Job.state = 'requested'` (queued, not yet
claimed). Cascade removes the Jobs and Candidates. The in-flight and terminal jobs are
untouched. Returns the number cleared.
- `"stop"` → upsert `floor.paused='true'` and `floor.killRequested='true'`.
- **`POST /api/requests/[id]/pause`** with `{ paused: boolean }` → set `Job.paused`. Guard:
the job must be `state='requested'` (else `400`, matching the existing retry route), since
pausing an in-flight job has no effect.
- **`DELETE /api/requests/[id]`** → delete the `Request` (cascade Job + Candidates). Guard:
only when `Job.state = 'requested'`, so the active download is never touched (that is the
job of `Stop now`). Mirrors `DELETE /api/library/[id]`.
## Web UI
`web/src/app/queue.tsx` and `web/src/app/_ui/job-row.tsx`; styling via existing `.btn`
variants in `design.css`.
- **Press header** (above the queue list):
- `Pause the press` / `Resume` toggle (from `GET /api/floor`).
- `Clear queue` — confirm dialog showing the queued count; calls `POST /api/floor
{action:"clear"}`.
- `Stop now ⚠` — rendered only when a job is actively downloading (derivable from the
existing `/api/requests` data: any job in `matching`/`matched`/`downloading`/`tagging`).
Confirm dialog ("aborts the current download and pauses the press"); calls `POST /api/floor
{action:"stop"}`.
- A "Press paused — N queued" banner/chip when `paused`.
- **Per-item controls** (queued rows only — `job.state === 'requested'`), placed in the row
`note` slot as `.btn sm ghost`:
- `Pause` / `Resume` → `POST /api/requests/[id]/pause`.
- `Remove` → `DELETE /api/requests/[id]`.
- A per-item paused row shows a subtle "paused" chip and the `Resume` button.
- Active-download and `needs_attention` rows keep their current rendering (Retry unchanged).
- Polling cadence stays 2s; `GET /api/floor` is fetched alongside the existing three calls in
`refresh()`.
## Settings
Add a numeric field **"Delay between downloads (seconds)"** (default 0, min 0) to the settings
form (`web/src/app/settings/settings-form.tsx`), persisted to `floor.jobDelaySeconds` via the
existing settings config-write path (the same mechanism used for the other numeric/monitor
settings). Include short helptext explaining it paces sustained Qobuz load.
## Testing
- **Worker** (pytest, `worker/tests/`):
- `claim_next` skips `paused=true` queued jobs and still claims the next unpaused one.
- Global pause: the loop skips claiming when `floor.paused` is set (unit-test the claim-gate
helper).
- Delay: the delay value is read and applied (unit-test the parse/gate helper; keep the
sleep injectable/mockable).
- Kill watcher: when `floor.killRequested` is set, the watcher clears the flag and triggers
exit (mock `os._exit`, assert flag cleared + exit called).
- **Web** (route tests, mirroring `requests/[id]/retry/route.test.ts` and
`library/bulk/route.test.ts`):
- `POST /api/floor` for each action; `clear` deletes only `requested` jobs; `stop` sets both
flags.
- `POST /api/requests/[id]/pause` toggles `Job.paused`; rejects non-`requested`.
- `DELETE /api/requests/[id]` removes a queued Request; refuses an in-flight/terminal job.
- `GET /api/floor` returns the Config-derived state.
## Edge cases
- **Remove/Clear of monitored items** reappear later by design (cancel-for-now). No extra
handling; the per-release Ignore is the permanent path.
- **Kill with nothing downloading** still restarts the worker (brief blip). Minimized by only
showing `Stop now` while a download is active.
- **Delay during pause:** the delay sleeps only *between* jobs; pause is checked before each
claim, so a paused press never starts a delayed job.
- **Race:** clicking per-item Pause/Remove on a job that was just claimed — the guards
(`state='requested'`) reject it; the UI only renders those buttons on queued rows, and the
2s refresh reconciles.
## Out of scope
- Resuming a partially-downloaded album (no download-resume support; force-killed jobs
re-download from scratch).
- Reordering the queue / per-item priority.
- Any change to the per-release Ignore or the wanted list UI.
docs/superpowers/specs/2026-07-15-qobuz-pacing-design.md
+83
View File
@@ -0,0 +1,83 @@
# Qobuz pacing ("budget gate") — design
Date: 2026-07-15
## Problem
Bulk hi-res downloading got the Qobuz account blocked (`403 USER_BLOCKED`). We need to keep
using Qobuz for hi-res but at a volume and cadence that looks like a human listener, so the
account isn't flagged again — while a large wanted-list still drains promptly via other sources.
## Approach: a per-job Qobuz eligibility gate
One gate decides, **per album**, whether Qobuz is an eligible source *right now*. If the gate is
closed, Qobuz candidates are dropped before ranking, so the worker falls through to the best
non-Qobuz source (Soulseek/YouTube) and keeps moving — no idling. Monitored albums grabbed that
way are upgraded to Qobuz hi-res later, automatically, via the existing quality-upgrade window,
once Qobuz has budget again.
**Qobuz is eligible for a job only if ALL hold:**
1. **Active hours** — current hour ∈ [activeStart, activeEnd) (a daytime window; start < end).
2. **Under today's cap** — Qobuz downloads counted today < today's cap.
3. **Spacing elapsed** — now ≥ the randomized "next allowed" time set after the last Qobuz
download.
After each successful Qobuz download the worker bumps today's counter and sets the next-allowed
time. The counter resets at the day boundary (tracked by a stored date).
**Today's cap = warm-up ramp of the steady cap**, by account age (days since `warmupStartDate`):
| Account age (days) | Cap |
|---|---|
| 0–2 | min(5, steady) |
| 3–6 | min(10, steady) |
| 7–13 | min(20, steady) |
| 14+ | steady |
If `warmupStartDate` is unset, no ramp (steady cap applies).
**Spacing / jitter (combined):** the gap to the next Qobuz download is randomized around
`activeWindowSeconds / todayCap` (so the day's budget spreads evenly across active hours), ±30%,
with a hard floor (≥ 60 s). This *is* the jitter — never a fixed metronome — and it prevents
front-loading the whole daily cap into the first hour.
## Configuration (Config keys; editable in Settings → Qobuz → Pacing)
Settings:
- `qobuz.pacing.enabled` (default `true`)
- `qobuz.pacing.activeStartHour` (default `8`)
- `qobuz.pacing.activeEndHour` (default `23`)
- `qobuz.pacing.dailyCap` (steady cap, default `40`)
- `qobuz.pacing.warmupStartDate` (ISO date; set to the new account's start day)
Worker-managed state (not user-edited):
- `qobuz.pacing.countDay` (date the counter is for)
- `qobuz.pacing.countToday` (int)
- `qobuz.pacing.nextAllowedAt` (epoch seconds)
Current hour / day come from the **DB clock** (`now()`), avoiding container-timezone ambiguity.
Config is read fresh each worker loop, so changes take effect with no restart.
## Worker integration
- New module `worker/lyra_worker/qobuz_gate.py`: a `QobuzPacing` config dataclass
(`from_config`), a pure `today_cap(...)`, a pure `is_allowed(...)` decision, and thin
DB helpers `gate_open(conn, now)` (reads state + decides) and `record_download(conn, pacing,
now)` (bumps counter + sets next-allowed).
- `pipeline.py`: after searching sources, if `gate_open` is false AND at least one non-Qobuz
candidate exists, drop Qobuz candidates before ranking (fall-through). If Qobuz is the *only*
source available, keep it (a Qobuz-only setup isn't gated — holding would strand the album).
- After a successful download where `winner.source == "qobuz"`, call `record_download`.
## Testing
Pure-logic unit tests for `is_allowed` (each gate condition), `today_cap` (ramp boundaries), and
the spacing computation; pipeline tests that a gated Qobuz candidate is dropped in favour of a
non-Qobuz source, that an open gate keeps Qobuz, and that a successful Qobuz download records the
counter/next-allowed.
## Out of scope
- Holding/parking an album for later Qobuz when Qobuz is the only source (fails/last-source as
today). Multi-source setups fall through, which is the common case.
- Timezone-aware active-hours windows that wrap past midnight (start < end only).
web/prisma/migrations/20260715000000_add_job_paused/migration.sql
+2
View File
@@ -0,0 +1,2 @@
-- AlterTable
ALTER TABLE "Job" ADD COLUMN "paused" BOOLEAN NOT NULL DEFAULT false;
web/prisma/migrations/20260720200000_add_job_download_eta/migration.sql
+2
View File
@@ -0,0 +1,2 @@
-- Estimated seconds remaining for the active download (measured throughput). Nullable.
ALTER TABLE "Job" ADD COLUMN "downloadEtaSeconds" INTEGER;
web/prisma/schema.prisma
+4
View File
@@ -58,6 +58,10 @@ model Job {
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
downloadProgress Float @default(0)
// Estimated seconds until the current download finishes, from measured throughput (slskd byte
// transfer rate). Null when unknown (no speed sample yet, or a source that doesn't report bytes).
downloadEtaSeconds Int?
paused Boolean @default(false)
candidates Candidate[]
}
web/src/app/_ui/artist-picker.tsx
+49
View File
@@ -0,0 +1,49 @@
"use client";
import { Modal } from "./modal";
import type { ArtistHit } from "@/lib/artist-resolve";
/** Disambiguation picker: several MusicBrainz artists share a name (e.g. two bands both called
* "Looking Glass"), so let the user choose the right one by its MB disambiguation text instead
* of silently trusting Last.fm's — sometimes wrong — MBID. Candidates arrive in MB relevance
* order, so the first is flagged "Suggested". */
export function ArtistPicker({
open,
name,
candidates,
onPick,
onClose,
}: {
open: boolean;
name: string;
candidates: ArtistHit[];
onPick: (mbid: string, name: string) => void;
onClose: () => void;
}) {
return (
<Modal open={open} onClose={onClose} title="Which artist?">
<p className="muted-note">
More than one artist is named “{name}”. Pick the right one.
</p>
<ul className="list">
{candidates.map((c, i) => (
<li key={c.mbid} className="list-row">
<div className="main">
<div className="rtitle">
<button className="linkish" onClick={() => onPick(c.mbid, c.name)}>
{c.name}
</button>
</div>
<div className="rmeta">
<span>{c.disambiguation || "no description on MusicBrainz"}</span>
</div>
</div>
<div className="actions">
{i === 0 ? <span className="chip working">Suggested</span> : null}
</div>
</li>
))}
</ul>
</Modal>
);
}
web/src/app/_ui/job-row.tsx
+3
View File
@@ -14,6 +14,7 @@ export function JobRow({
album,
kind,
label,
marker,
meta,
note,
bar,
@@ -22,6 +23,7 @@ export function JobRow({
album: string;
kind: Kind;
label: string;
marker?: string;
meta?: ReactNode;
note?: ReactNode;
bar?: ReactNode;
@@ -31,6 +33,7 @@ export function JobRow({
<div className="stripe" />
<div>
<h3 className="title">
{marker ? <span className="nextup">{marker}</span> : null}
{album} <span className="artist">· {artist}</span>
</h3>
{meta ? <div className="meta">{meta}</div> : null}
web/src/app/_ui/kebab-menu.tsx
+47
View File
@@ -0,0 +1,47 @@
"use client";
import { useEffect, useRef, useState, type ReactNode } from "react";
/** A vertical "⋮" trigger that opens a small popup menu. Closes on outside click, Escape, or any
* click inside the popup (so menu items don't each need to manage it). Children are the items —
* use <button className="menu-item">…</button>. */
export function KebabMenu({ label = "Actions", children }: { label?: string; children: ReactNode }) {
const [open, setOpen] = useState(false);
const ref = useRef<HTMLDivElement>(null);
useEffect(() => {
if (!open) return;
function onDoc(e: MouseEvent) {
if (ref.current && !ref.current.contains(e.target as Node)) setOpen(false);
}
function onKey(e: KeyboardEvent) {
if (e.key === "Escape") setOpen(false);
}
document.addEventListener("mousedown", onDoc);
document.addEventListener("keydown", onKey);
return () => {
document.removeEventListener("mousedown", onDoc);
document.removeEventListener("keydown", onKey);
};
}, [open]);
return (
<div className="menu" ref={ref}>
<button
type="button"
className="menu-trigger"
aria-haspopup="menu"
aria-expanded={open}
aria-label={label}
onClick={() => setOpen((o) => !o)}
>
⋮
</button>
{open ? (
<div className="menu-pop" role="menu" onClick={() => setOpen(false)}>
{children}
</div>
) : null}
</div>
);
}
web/src/app/_ui/status.test.ts
+16 -1
View File
@@ -1,5 +1,20 @@
import { describe, it, expect } from "vitest";
import { describeJob, timeAgo } from "./status";
import { describeJob, etaLabel, timeAgo } from "./status";
describe("etaLabel", () => {
it("formats seconds, minutes, and hours", () => {
expect(etaLabel(45)).toBe("~45s left");
expect(etaLabel(240)).toBe("~4m left");
expect(etaLabel(3900)).toBe("~1h 5m left");
});
it("returns null for missing or non-positive input", () => {
expect(etaLabel(null)).toBeNull();
expect(etaLabel(undefined)).toBeNull();
expect(etaLabel(0)).toBeNull();
expect(etaLabel(-5)).toBeNull();
});
});
describe("describeJob", () => {
it("maps pipeline states to literal labels + severity kind", () => {
web/src/app/_ui/status.ts
+10
View File
@@ -27,6 +27,16 @@ export function timeAgo(value: string | Date | null | undefined, now: number): s
return `${Math.floor(h / 24)}d`;
}
/** Compact download ETA like "~4m left", "~45s left", "~1h 5m left". Null/≤0 → no label. */
export function etaLabel(seconds: number | null | undefined): string | null {
if (seconds == null || seconds <= 0) return null;
if (seconds < 60) return `~${seconds}s left`;
const m = Math.round(seconds / 60);
if (m < 60) return `~${m}m left`;
const h = Math.floor(m / 60);
return `~${h}h ${m % 60}m left`;
}
export function describeJob(state: string, stage: string): { label: string; kind: Kind } {
void stage; // stage no longer drives the label; kept for call-site compatibility
return (
web/src/app/api/floor/route.test.ts
+63
View File
@@ -0,0 +1,63 @@
import { describe, it, expect } from "vitest";
import { GET, POST } from "./route";
import { prisma } from "@/lib/db";
function post(action: string) {
return new Request("http://localhost/api/floor", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ action }),
});
}
async function flag(key: string) {
return (await prisma.config.findUnique({ where: { key } }))?.value ?? null;
}
async function queuedRequest(album: string, state = "requested") {
return prisma.request.create({
data: { artist: "A", album, status: "pending", job: { create: { state: state as never } } },
include: { job: true },
});
}
describe("floor API", () => {
it("GET reports paused=false by default", async () => {
const res = await GET();
expect(await res.json()).toEqual({ paused: false });
});
it("pause and resume toggle floor.paused", async () => {
await POST(post("pause"));
expect(await flag("floor.paused")).toBe("true");
const paused = await (await GET()).json();
expect(paused).toEqual({ paused: true });
await POST(post("resume"));
expect(await flag("floor.paused")).toBe("false");
});
it("stop sets both paused and killRequested", async () => {
const res = await POST(post("stop"));
expect(res.status).toBe(200);
expect(await flag("floor.paused")).toBe("true");
expect(await flag("floor.killRequested")).toBe("true");
});
it("clear deletes only queued (requested) requests", async () => {
const queued = await queuedRequest("Queued");
const active = await queuedRequest("Downloading", "downloading");
const res = await POST(post("clear"));
const body = await res.json();
expect(body.cleared).toBe(1);
expect(await prisma.request.findUnique({ where: { id: queued.id } })).toBeNull();
expect(await prisma.request.findUnique({ where: { id: active.id } })).not.toBeNull();
});
it("rejects an unknown action", async () => {
const res = await POST(post("nope"));
expect(res.status).toBe(400);
});
});
web/src/app/api/floor/route.ts
+49
View File
@@ -0,0 +1,49 @@
import { prisma } from "@/lib/db";
async function getFlag(key: string): Promise<boolean> {
const row = await prisma.config.findUnique({ where: { key } });
return row?.value === "true";
}
async function setFlag(key: string, value: boolean): Promise<void> {
const v = value ? "true" : "false";
await prisma.config.upsert({
where: { key },
create: { key, value: v, secret: false },
update: { value: v },
});
}
export async function GET() {
return Response.json({ paused: await getFlag("floor.paused") });
}
export async function POST(request: Request) {
let body: { action?: string };
try {
body = (await request.json()) as { action?: string };
} catch {
return Response.json({ error: "invalid JSON" }, { status: 400 });
}
switch (body.action) {
case "pause":
await setFlag("floor.paused", true);
return Response.json({ ok: true, paused: true });
case "resume":
await setFlag("floor.paused", false);
return Response.json({ ok: true, paused: false });
case "stop":
await setFlag("floor.paused", true);
await setFlag("floor.killRequested", true);
return Response.json({ ok: true, paused: true, stopped: true });
case "clear": {
const { count } = await prisma.request.deleteMany({
where: { job: { is: { state: "requested" } } },
});
return Response.json({ ok: true, cleared: count });
}
default:
return Response.json({ error: "unknown action" }, { status: 400 });
}
}
web/src/app/api/monitor/config/route.test.ts
+20
View File
@@ -34,3 +34,23 @@ describe("monitor config API", () => {
expect(await prisma.config.count()).toBe(before);
});
});
describe("monitor config — floor.jobDelaySeconds", () => {
it("defaults jobDelaySeconds to 0 and round-trips a new value", async () => {
const before = await (await GET()).json();
expect(before["floor.jobDelaySeconds"]).toBe("0");
await PATCH(
new Request("http://localhost/api/monitor/config", {
method: "PATCH",
headers: { "content-type": "application/json" },
body: JSON.stringify({ "floor.jobDelaySeconds": "45" }),
}),
);
const stored = await prisma.config.findUnique({ where: { key: "floor.jobDelaySeconds" } });
expect(stored?.value).toBe("45");
const after = await (await GET()).json();
expect(after["floor.jobDelaySeconds"]).toBe("45");
});
});
web/src/app/api/monitor/config/route.ts
+1
View File
@@ -6,6 +6,7 @@ const DEFAULTS: Record<string, string> = {
"monitor.retryIntervalHours": "6",
"monitor.qualityCutoff": "2",
"monitor.upgradeWindowDays": "14",
"floor.jobDelaySeconds": "0",
};
export async function GET() {
web/src/app/api/qobuz/pacing/route.test.ts
+37
View File
@@ -0,0 +1,37 @@
import { describe, it, expect } from "vitest";
import { GET, PATCH } from "./route";
import { prisma } from "@/lib/db";
function patch(body: Record<string, string>) {
return new Request("http://localhost/api/qobuz/pacing", {
method: "PATCH",
headers: { "content-type": "application/json" },
body: JSON.stringify(body),
});
}
describe("qobuz pacing config API", () => {
it("returns defaults when nothing is stored", async () => {
const cfg = await (await GET()).json();
expect(cfg["qobuz.pacing.enabled"]).toBe("true");
expect(cfg["qobuz.pacing.activeStartHour"]).toBe("8");
expect(cfg["qobuz.pacing.activeEndHour"]).toBe("23");
expect(cfg["qobuz.pacing.dailyCap"]).toBe("40");
expect(cfg["qobuz.pacing.warmupStartDate"]).toBe("");
});
it("round-trips allow-listed settings and persists them", async () => {
await PATCH(patch({ "qobuz.pacing.dailyCap": "25", "qobuz.pacing.activeStartHour": "9" }));
const stored = await prisma.config.findUnique({ where: { key: "qobuz.pacing.dailyCap" } });
expect(stored?.value).toBe("25");
const cfg = await (await GET()).json();
expect(cfg["qobuz.pacing.dailyCap"]).toBe("25");
expect(cfg["qobuz.pacing.activeStartHour"]).toBe("9");
});
it("ignores keys not in the allow-list (e.g. worker-managed counters)", async () => {
await PATCH(patch({ "qobuz.pacing.countToday": "999", "qobuz.pacing.dailyCap": "30" }));
expect(await prisma.config.findUnique({ where: { key: "qobuz.pacing.countToday" } })).toBeNull();
expect((await prisma.config.findUnique({ where: { key: "qobuz.pacing.dailyCap" } }))?.value).toBe("30");
});
});
web/src/app/api/qobuz/pacing/route.ts
+39
View File
@@ -0,0 +1,39 @@
import { prisma } from "@/lib/db";
// Qobuz pacing ("budget gate") settings. The worker reads these Config keys live; state keys
// (countDay/countToday/nextAllowedAt) are worker-managed and intentionally NOT editable here.
const DEFAULTS: Record<string, string> = {
"qobuz.pacing.enabled": "true",
"qobuz.pacing.activeStartHour": "8",
"qobuz.pacing.activeEndHour": "23",
"qobuz.pacing.dailyCap": "40",
"qobuz.pacing.warmupStartDate": "",
};
export async function GET() {
const rows = await prisma.config.findMany({ where: { key: { in: Object.keys(DEFAULTS) } } });
const stored = Object.fromEntries(rows.map((r) => [r.key, r.value]));
return Response.json(
Object.fromEntries(Object.entries(DEFAULTS).map(([k, d]) => [k, stored[k] ?? d])),
);
}
export async function PATCH(request: Request) {
let body: unknown;
try {
body = await request.json();
} catch {
return Response.json({ error: "invalid JSON" }, { status: 400 });
}
const entries = Object.entries((body ?? {}) as Record<string, unknown>).filter(([k]) =>
Object.prototype.hasOwnProperty.call(DEFAULTS, k),
);
for (const [key, value] of entries) {
await prisma.config.upsert({
where: { key },
create: { key, value: String(value), secret: false },
update: { value: String(value) },
});
}
return Response.json({ updated: entries.length });
}
web/src/app/api/requests/[id]/pause/route.test.ts
+49
View File
@@ -0,0 +1,49 @@
import { describe, it, expect } from "vitest";
import { POST } from "./route";
import { prisma } from "@/lib/db";
import type { Prisma } from "@prisma/client";
function req(id: string, paused: boolean) {
return new Request(`http://localhost/api/requests/${id}/pause`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ paused }),
});
}
async function seed(jobData: Prisma.JobCreateWithoutRequestInput) {
return prisma.request.create({
data: { artist: "A", album: "B", status: "pending", job: { create: jobData } },
include: { job: true },
});
}
describe("per-item pause API", () => {
it("pauses a queued job", async () => {
const created = await seed({ state: "requested" });
const res = await POST(req(created.id, true), { params: Promise.resolve({ id: created.id }) });
expect(res.status).toBe(200);
const job = await prisma.job.findUnique({ where: { id: created.job!.id } });
expect(job?.paused).toBe(true);
});
it("resumes a paused job", async () => {
const created = await seed({ state: "requested", paused: true });
await POST(req(created.id, false), { params: Promise.resolve({ id: created.id }) });
const job = await prisma.job.findUnique({ where: { id: created.job!.id } });
expect(job?.paused).toBe(false);
});
it("rejects pausing an in-flight job", async () => {
const created = await seed({ state: "downloading", paused: true });
const res = await POST(req(created.id, false), { params: Promise.resolve({ id: created.id }) });
expect(res.status).toBe(400);
const job = await prisma.job.findUnique({ where: { id: created.job!.id } });
expect(job?.paused).toBe(true);
});
it("404s for a missing request", async () => {
const res = await POST(req("nope", true), { params: Promise.resolve({ id: "nope" }) });
expect(res.status).toBe(404);
});
});
web/src/app/api/requests/[id]/pause/route.ts
+26
View File
@@ -0,0 +1,26 @@
import { prisma } from "@/lib/db";
export async function POST(request: Request, { params }: { params: Promise<{ id: string }> }) {
const { id } = await params;
let body: { paused?: unknown };
try {
body = (await request.json()) as { paused?: unknown };
} catch {
return Response.json({ error: "invalid JSON" }, { status: 400 });
}
if (typeof body.paused !== "boolean") {
return Response.json({ error: "paused must be a boolean" }, { status: 400 });
}
const req = await prisma.request.findUnique({ where: { id }, include: { job: true } });
if (!req || !req.job) {
return Response.json({ error: "not found" }, { status: 404 });
}
if (req.job.state !== "requested") {
return Response.json({ error: "only queued jobs can be paused" }, { status: 400 });
}
await prisma.job.update({ where: { id: req.job.id }, data: { paused: body.paused } });
return Response.json({ ok: true, paused: body.paused });
}
web/src/app/api/requests/[id]/route.test.ts
+37
View File
@@ -0,0 +1,37 @@
import { describe, it, expect } from "vitest";
import { DELETE } from "./route";
import { prisma } from "@/lib/db";
import type { Prisma } from "@prisma/client";
function del(id: string) {
return new Request(`http://localhost/api/requests/${id}`, { method: "DELETE" });
}
async function seed(jobData: Prisma.JobCreateWithoutRequestInput) {
return prisma.request.create({
data: { artist: "A", album: "B", status: "pending", job: { create: jobData } },
include: { job: true },
});
}
describe("remove request API", () => {
it("deletes a queued request and cascades its job", async () => {
const created = await seed({ state: "requested" });
const res = await DELETE(del(created.id), { params: Promise.resolve({ id: created.id }) });
expect(res.status).toBe(200);
expect(await prisma.request.findUnique({ where: { id: created.id } })).toBeNull();
expect(await prisma.job.findUnique({ where: { id: created.job!.id } })).toBeNull();
});
it("refuses to remove an in-flight job", async () => {
const created = await seed({ state: "downloading" });
const res = await DELETE(del(created.id), { params: Promise.resolve({ id: created.id }) });
expect(res.status).toBe(400);
expect(await prisma.request.findUnique({ where: { id: created.id } })).not.toBeNull();
});
it("404s for a missing request", async () => {
const res = await DELETE(del("nope"), { params: Promise.resolve({ id: "nope" }) });
expect(res.status).toBe(404);
});
});
web/src/app/api/requests/[id]/route.ts
+16
View File
@@ -0,0 +1,16 @@
import { prisma } from "@/lib/db";
export async function DELETE(_request: Request, { params }: { params: Promise<{ id: string }> }) {
const { id } = await params;
const req = await prisma.request.findUnique({ where: { id }, include: { job: true } });
if (!req) {
return Response.json({ error: "not found" }, { status: 404 });
}
if (req.job && req.job.state !== "requested") {
return Response.json({ error: "only queued items can be removed" }, { status: 400 });
}
await prisma.request.delete({ where: { id } }); // cascades Job + Candidates
return Response.json({ ok: true });
}
web/src/app/api/requests/route.ts
+2
View File
@@ -64,6 +64,8 @@ export async function GET() {
claimedAt: r.job.claimedAt,
updatedAt: r.job.updatedAt,
downloadProgress: r.job.downloadProgress,
downloadEtaSeconds: r.job.downloadEtaSeconds,
paused: r.job.paused,
candidateCount: cands.length,
chosen: chosen ? { source: chosen.source, format: chosen.format, trackCount: chosen.trackCount } : null,
};
web/src/app/design.css
+36
View File
@@ -245,6 +245,9 @@ nav.contents .sep { flex: 1; }
.btn.ghost { border-color: var(--rule-2); color: var(--graphite); }
.btn.ghost:hover { background: transparent; border-color: var(--ink); color: var(--ink); }
.press-controls { display: flex; gap: 0.5rem; align-items: center; margin: 0 0 1rem; flex-wrap: wrap; }
.item-controls { display: inline-flex; gap: 0.5rem; align-items: center; }
.toggle {
font-family: var(--mono); font-size: 0.64rem; letter-spacing: 0.1em; text-transform: uppercase;
color: var(--graphite); display: inline-flex; align-items: center; gap: 7px; cursor: pointer;
@@ -383,6 +386,39 @@ nav.contents .sep { flex: 1; }
.tab:hover { color: var(--ink); }
.tab.active { color: var(--ink); border-bottom-color: var(--accent); }
.tab .n { color: var(--rule-2); margin-left: 6px; }
.tab.has-errors .n { color: var(--alert); }
/* "Next up" badge on the first queued row (the item the worker claims next) */
.nextup {
display: inline-block; vertical-align: middle; margin-right: 9px; transform: translateY(-2px);
font-family: var(--mono); font-size: 0.56rem; letter-spacing: 0.13em; text-transform: uppercase;
color: var(--accent-ink); background: var(--accent); padding: 2px 7px; border-radius: 4px;
}
/* ── Kebab menu (⋮ popup for press actions) ───────────── */
.dept.has-actions { align-items: center; }
.dept.has-actions .fill { transform: none; }
.menu { position: relative; display: inline-flex; }
.menu-trigger {
font-size: 1.15rem; line-height: 1; color: var(--graphite); background: transparent;
border: 1px solid var(--rule); border-radius: 6px; width: 32px; height: 30px; cursor: pointer;
display: inline-flex; align-items: center; justify-content: center;
}
.menu-trigger:hover { color: var(--ink); border-color: var(--rule-2); }
.menu-pop {
position: absolute; top: calc(100% + 6px); right: 0; z-index: 30; min-width: 190px;
background: var(--paper-2); border: 1px solid var(--rule-2); border-radius: 9px;
box-shadow: 0 10px 30px rgba(0, 0, 0, 0.22); padding: 5px;
display: flex; flex-direction: column; gap: 1px;
}
.menu-item {
font-family: var(--mono); font-size: 0.68rem; letter-spacing: 0.1em; text-transform: uppercase;
text-align: left; background: transparent; border: 0; color: var(--ink);
padding: 10px 11px; border-radius: 6px; cursor: pointer; white-space: nowrap;
}
.menu-item:hover { background: var(--rule); }
.menu-item:disabled { color: var(--rule-2); cursor: default; background: transparent; }
.menu-item.danger { color: var(--alert); }
/* ── Discography row (cover thumb + click-to-modal) ───── */
.disco-open {
web/src/app/lastfm/lastfm-client.tsx
BIN
View File
Binary file not shown.
web/src/app/queue.tsx
BIN
View File
Binary file not shown.
web/src/app/settings/settings-form.tsx
+74
View File
@@ -31,6 +31,7 @@ const MONITOR_FIELDS: [string, string][] = [
["monitor.retryIntervalHours", "Retry interval (hours)"],
["monitor.qualityCutoff", "Quality cutoff"],
["monitor.upgradeWindowDays", "Upgrade window (days)"],
["floor.jobDelaySeconds", "Delay between downloads (seconds)"],
];
export function SettingsForm() {
@@ -53,6 +54,8 @@ export function SettingsForm() {
const [discoverSaved, setDiscoverSaved] = useState(false);
const [monitorCfg, setMonitorCfg] = useState<Record<string, string>>({});
const [monitorSaved, setMonitorSaved] = useState(false);
const [pacingCfg, setPacingCfg] = useState<Record<string, string>>({});
const [pacingSaved, setPacingSaved] = useState(false);
useEffect(() => {
fetch("/api/scan")
@@ -105,6 +108,12 @@ export function SettingsForm() {
.then((c: Record<string, string>) => setMonitorCfg(c));
}, []);
useEffect(() => {
fetch("/api/qobuz/pacing")
.then((r) => r.json())
.then((c: Record<string, string>) => setPacingCfg(c));
}, []);
async function saveCredentials(e: React.FormEvent) {
e.preventDefault();
await fetch("/api/config", {
@@ -159,6 +168,21 @@ export function SettingsForm() {
setTimeout(() => setMonitorSaved(false), 1500);
}
function setPacing(key: string, value: string) {
setPacingCfg((c) => ({ ...c, [key]: value }));
}
async function savePacing(e: React.FormEvent) {
e.preventDefault();
await fetch("/api/qobuz/pacing", {
method: "PATCH",
headers: { "content-type": "application/json" },
body: JSON.stringify(pacingCfg),
});
setPacingSaved(true);
setTimeout(() => setPacingSaved(false), 1500);
}
const secretHint = (set: boolean) => (set ? <span className="set"> · set</span> : null);
async function clearField(field: "qobuzToken" | "slskdApiKey" | "lastfmApiKey") {
@@ -192,6 +216,7 @@ export function SettingsForm() {
</div>
{tab === "Qobuz" ? (
<>
<form className="settings-section" onSubmit={saveCredentials}>
<div className="subhead">
Auth token
@@ -234,6 +259,55 @@ export function SettingsForm() {
{saved ? <span className="saved-note">Saved</span> : null}
</div>
</form>
<form className="settings-section" onSubmit={savePacing}>
<div className="subhead">
Pacing
<HelpButton title="Qobuz pacing">
Throttles Qobuz to a human-looking volume and cadence so the account isn&apos;t flagged
for bulk downloading. Within active hours it downloads up to the daily cap, spaced out
with randomized gaps; when throttled, albums fall through to Soulseek/YouTube and
upgrade to Qobuz hi-res later. The daily cap ramps up over the first two weeks from the
new-account start date.
</HelpButton>
</div>
<label className="field-inline">
<input type="checkbox" aria-label="pace qobuz downloads"
checked={pacingCfg["qobuz.pacing.enabled"] !== "false"}
onChange={(e) => setPacing("qobuz.pacing.enabled", e.target.checked ? "true" : "false")} />
<span>Pace Qobuz downloads</span>
</label>
<div className="field-grid">
<label className="field">
<span>Active hours — start (0–24)</span>
<input type="number" min={0} max={24} aria-label="qobuz active start hour"
value={pacingCfg["qobuz.pacing.activeStartHour"] ?? ""}
onChange={(e) => setPacing("qobuz.pacing.activeStartHour", e.target.value)} />
</label>
<label className="field">
<span>Active hours — end (0–24)</span>
<input type="number" min={0} max={24} aria-label="qobuz active end hour"
value={pacingCfg["qobuz.pacing.activeEndHour"] ?? ""}
onChange={(e) => setPacing("qobuz.pacing.activeEndHour", e.target.value)} />
</label>
<label className="field">
<span>Daily cap (steady)</span>
<input type="number" min={0} aria-label="qobuz daily cap"
value={pacingCfg["qobuz.pacing.dailyCap"] ?? ""}
onChange={(e) => setPacing("qobuz.pacing.dailyCap", e.target.value)} />
</label>
<label className="field">
<span>New-account start (warm-up)</span>
<input type="date" aria-label="qobuz warmup start date"
value={pacingCfg["qobuz.pacing.warmupStartDate"] ?? ""}
onChange={(e) => setPacing("qobuz.pacing.warmupStartDate", e.target.value)} />
</label>
</div>
<div className="save-row">
<button type="submit" className="btn accent">Save</button>
{pacingSaved ? <span className="saved-note">Saved</span> : null}
</div>
</form>
</>
) : null}
{tab === "Soulseek" ? (
web/src/lib/artist-resolve.test.ts
+46
View File
@@ -0,0 +1,46 @@
import { describe, it, expect } from "vitest";
import { resolveArtistChoice, type ArtistHit } from "./artist-resolve";
const US: ArtistHit = { mbid: "us", name: "Looking Glass", disambiguation: "70's US pop group" };
const AU: ArtistHit = { mbid: "au", name: "Looking Glass", disambiguation: "Australian stoner rock" };
describe("resolveArtistChoice", () => {
it("is ambiguous when 2+ candidates share the exact name", () => {
expect(resolveArtistChoice("Looking Glass", [US, AU], "au")).toEqual({
kind: "ambiguous",
candidates: [US, AU],
});
});
it("uses the single exact-name match even over a disagreeing fallback mbid", () => {
const hit: ArtistHit = { mbid: "r", name: "Radiohead", disambiguation: "" };
expect(resolveArtistChoice("Radiohead", [hit], "wrong-lastfm-mbid")).toEqual({
kind: "single",
mbid: "r",
});
});
it("matches exact name case- and whitespace-insensitively", () => {
expect(resolveArtistChoice(" looking GLASS ", [US, AU])).toEqual({
kind: "ambiguous",
candidates: [US, AU],
});
});
it("falls back to the provided mbid when no candidate name matches exactly", () => {
const fuzzy: ArtistHit = { mbid: "x", name: "The Beatles", disambiguation: "" };
expect(resolveArtistChoice("Beatles", [fuzzy], "lastfm-mbid")).toEqual({
kind: "single",
mbid: "lastfm-mbid",
});
});
it("falls back to MB's top hit when there is no exact match and no fallback", () => {
const fuzzy: ArtistHit = { mbid: "top", name: "The Beatles", disambiguation: "" };
expect(resolveArtistChoice("Beatles", [fuzzy])).toEqual({ kind: "single", mbid: "top" });
});
it("is none when there are no candidates and no fallback", () => {
expect(resolveArtistChoice("Nobody Here", [])).toEqual({ kind: "none" });
});
});
web/src/lib/artist-resolve.ts
+34
View File
@@ -0,0 +1,34 @@
export type ArtistHit = { mbid: string; name: string; disambiguation: string };
export type Resolution =
| { kind: "single"; mbid: string }
| { kind: "ambiguous"; candidates: ArtistHit[] }
| { kind: "none" };
const norm = (s: string) => s.trim().toLowerCase();
/**
* Decide which MusicBrainz artist a Last.fm row refers to, given MB's search candidates.
*
* The bug this guards against: Last.fm's own MBID for an ambiguous name can point at the
* wrong same-named band (it mapped "Looking Glass" to an Australian stoner band, not the US
* pop group). So we prefer MB's own resolution:
* - 2+ candidates whose name EXACTLY matches (normalized) → ambiguous; let the user pick.
* - exactly 1 exact-name match → use it, even if it disagrees with `fallbackMbid`.
* - no exact-name match → keep prior behavior: the Last.fm `fallbackMbid`, else MB's top hit.
*
* `candidates` is assumed to arrive in MB relevance order (best first), so `candidates[0]`
* is the suggested default and exact-name matches keep that ordering.
*/
export function resolveArtistChoice(
name: string,
candidates: ArtistHit[],
fallbackMbid?: string | null,
): Resolution {
const exact = candidates.filter((a) => norm(a.name) === norm(name));
if (exact.length >= 2) return { kind: "ambiguous", candidates: exact };
if (exact.length === 1) return { kind: "single", mbid: exact[0].mbid };
if (fallbackMbid) return { kind: "single", mbid: fallbackMbid };
if (candidates.length > 0) return { kind: "single", mbid: candidates[0].mbid };
return { kind: "none" };
}
worker/lyra_worker/_musicbrainz.py
+103 -9
View File
@@ -1,3 +1,4 @@
import time
from difflib import SequenceMatcher
from lyra_worker.types import MBTarget
@@ -5,10 +6,104 @@ from lyra_worker.types import MBTarget
_MIN_TITLE_RATIO = 0.6
def _with_retry(fn, *, attempts: int = 4, base_delay: float = 1.0, sleep=time.sleep):
"""Call `fn`, retrying transient MusicBrainz failures with exponential backoff.
A dropped TLS connection to the MusicBrainz webservice surfaces as
musicbrainzngs.NetworkError (`<urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING]>`),
and MusicBrainz's own 503 rate-limit surfaces as ResponseError; both are
transient, so we retry rather than failing the whole pipeline for the job.
Any other exception (bug, non-transient) propagates immediately.
musicbrainzngs is imported lazily so importing this module stays offline.
"""
import musicbrainzngs
transient = (musicbrainzngs.NetworkError, musicbrainzngs.ResponseError)
for i in range(attempts):
try:
return fn()
except transient:
if i == attempts - 1:
raise
sleep(base_delay * (2**i))
def _ratio(a: str, b: str) -> float:
return SequenceMatcher(None, a.strip().casefold(), b.strip().casefold()).ratio()
def _is_album(g: dict) -> bool:
return (g.get("primary-type") or "").casefold() == "album"
def _rg_year(g: dict) -> int:
"""First-release year of a release-group, or 9999 when unknown (so undated groups lose ties)."""
frd = (g.get("first-release-date") or "")[:4]
return int(frd) if frd.isdigit() else 9999
def _credited_artist(g: dict) -> str:
"""Primary credited artist name of a release-group search result, or '' if absent."""
credit = g.get("artist-credit")
if isinstance(credit, list) and credit and isinstance(credit[0], dict):
return (credit[0].get("artist") or {}).get("name", "") or ""
return ""
def _best_release_group(artist: str, album: str, groups: list) -> dict | None:
"""Choose the best release-group match for `artist`/`album`.
Ranked, highest first, on `(artist_ratio, title_ratio, is_album)`:
* artist match dominates — MusicBrainz relevance ties same-titled release groups by
*different* artists (e.g. "Fun Machine" exists as a Lake Street Dive EP and an
unrelated band's Album), and the artist we followed is trustworthy, so a credited
artist that matches the request outranks everything else;
* title similarity is next;
* an Album primary-type only breaks ties *within* the same artist+title, so a famous
album (Michael Jackson's "Off the Wall") still isn't resolved to its same-named
single whose short tracklist would map positionally onto the album's files;
* finally, the EARLIEST release-group wins — MusicBrainz lists the original album and its
later reissues/anniversary editions as separate same-titled groups (Linkin Park "Hybrid
Theory": a 2000 original and a 2023 13-track reissue), which tie on everything above, so
MB's arbitrary relevance order used to pick a reissue whose longer tracklist/runtime then
failed completeness checks against the standard album a source delivers. Prefer the original.
Artist is a ranking signal, never a hard filter — a low match sinks a candidate but
never drops the release, so credited-name variations (feat., punctuation) still resolve.
Returns None below the title threshold. Pure — no network I/O."""
if not groups:
return None
best = max(
groups,
key=lambda g: (
_ratio(artist, _credited_artist(g)),
_ratio(album, g.get("title", "")),
_is_album(g),
-_rg_year(g), # tie-break: the original (earliest) group over later reissues
),
)
if _ratio(album, best.get("title", "")) < _MIN_TITLE_RATIO:
return None
return best
def _pick_release(releases: list) -> dict | None:
"""Choose which release of a release-group to read the tracklist from. MB returns them in an
arbitrary order and ``releases[0]`` is often a later reissue/deluxe/anniversary edition with
more tracks and a longer runtime than the standard album a source actually delivers — which
then makes a *complete* download fail the pipeline's track-count/duration completeness checks
and re-download forever (Linkin Park "Hybrid Theory": MB's arbitrary pick was a 2023 13-track
reissue at ~52 min vs the 2000 original's 12 tracks / ~38 min). Prefer the ORIGINAL standard
edition: an Official release, then the earliest date, then a dated one. Pure — unit-tested."""
if not releases:
return None
def _key(r: dict) -> tuple:
official = 0 if (r.get("status") == "Official") else 1 # official editions first
date = r.get("date") or ""
return (official, 0 if date else 1, date) # then dated, then earliest date
return min(releases, key=_key)
class MusicBrainzResolver:
"""Real MbResolver using the MusicBrainz webservice via musicbrainzngs.
@@ -26,12 +121,10 @@ class MusicBrainzResolver:
musicbrainzngs.set_useragent(self._app, self._version, self._contact)
res = musicbrainzngs.search_release_groups(query=album, artist=artist, limit=5)
res = _with_retry(lambda: musicbrainzngs.search_release_groups(query=album, artist=artist, limit=5))
groups = res.get("release-group-list", [])
if not groups:
return None
rg = max(groups, key=lambda g: _ratio(album, g.get("title", "")))
if _ratio(album, rg.get("title", "")) < _MIN_TITLE_RATIO:
rg = _best_release_group(artist, album, groups)
if rg is None:
return None
canonical_album = rg.get("title", album)
@@ -53,11 +146,12 @@ class MusicBrainzResolver:
titles: tuple[str, ...] = ()
rgid = rg.get("id")
if rgid:
rgfull = musicbrainzngs.get_release_group_by_id(rgid, includes=["releases"])
rgfull = _with_retry(lambda: musicbrainzngs.get_release_group_by_id(rgid, includes=["releases"]))
releases = rgfull.get("release-group", {}).get("release-list", [])
if releases:
rel = musicbrainzngs.get_release_by_id(
releases[0]["id"], includes=["recordings"]
chosen = _pick_release(releases)
if chosen:
rel = _with_retry(
lambda: musicbrainzngs.get_release_by_id(chosen["id"], includes=["recordings"])
).get("release", {})
tracks = []
for medium in rel.get("medium-list", []):
worker/lyra_worker/adapters/_slskd.py
+211 -42
View File
@@ -2,6 +2,7 @@ import json
import os
import shutil
import time
import unicodedata
from typing import Callable
import requests
@@ -10,9 +11,35 @@ _DEFAULT_DOWNLOADS_ROOT = "/slskd-downloads"
_AUDIO_EXT = {"flac", "mp3", "m4a", "ogg", "opus", "wav", "aac"}
_LOSSLESS_EXT = {"flac", "wav"}
_SEARCH_POLLS = 40 # * 3s ≈ 2 min
_XFER_POLLS = 200 # * 3s ≈ 10 min
_SEARCH_POLLS = 40 # * 3s ≈ 2 min
_STALL_POLLS = 30 # * 3s ≈ 90s of no byte progress → abandon a queued/dead peer and fall through
_POLL_SECONDS = 3
# Absolute backstop for a slow-but-progressing peer. Scaled to the album's total bytes (a flat
# cap starved big albums: a 500MB set from a ~400KB/s serial peer needs ~60-70min and was cut at
# a flat 60min, then re-downloaded from scratch — never finishing). Budget at a conservative floor
# throughput, clamped. The 90s stall timeout still abandons a truly dead/queued peer quickly.
_MIN_XFER_POLLS = 400 # * 3s ≈ 20 min floor (small albums)
_MAX_XFER_POLLS = 2400 # * 3s ≈ 2 h ceiling (bounds worst-case slot hold)
_BACKSTOP_FLOOR_BPS = 64_000 # budget the backstop assuming >= ~64 KB/s sustained
def _max_polls(total_bytes: int) -> int:
"""Poll budget (each _POLL_SECONDS) for the absolute download backstop, scaled to album size at
a conservative floor throughput so a large slow-but-healthy transfer isn't abandoned mid-flight.
Clamped to [_MIN_XFER_POLLS, _MAX_XFER_POLLS]. Pure — unit-tested."""
if total_bytes <= 0:
return _MIN_XFER_POLLS
budget = int(total_bytes / (_BACKSTOP_FLOOR_BPS * _POLL_SECONDS))
return max(_MIN_XFER_POLLS, min(budget, _MAX_XFER_POLLS))
def _needs_enqueue(state: str) -> bool:
"""Whether a file must be (re)enqueued on slskd. True if it has no transfer yet or its last
transfer ended badly (Errored/Cancelled/Rejected); a file already Completed/InProgress/Queued is
left alone so a resumed download continues it instead of restarting from zero. Pure — unit-tested."""
if not state:
return True
return any(bad in state for bad in ("Errored", "Cancelled", "Rejected"))
def _basename(path: str) -> str:
@@ -27,6 +54,114 @@ def _dirname(path: str) -> str:
return ""
_ETA_CAP_SECONDS = 359_999 # ~100h; keeps the value well inside Job.downloadEtaSeconds (int4)
def _eta_seconds(total_bytes: int, bytes_now: int, speed_bps: float) -> int | None:
"""Seconds until the transfer finishes at the current measured rate, or None when it can't be
estimated (no throughput sample yet, unknown total, or already complete). Pure — unit-tested.
A near-stalled peer drives speed_bps toward zero, so the estimate is capped: an uncapped value
can exceed int4 and overflow the downloadEtaSeconds column write (poisoning the pipeline txn)."""
if speed_bps <= 0 or total_bytes <= 0 or bytes_now >= total_bytes:
return None
return min(int((total_bytes - bytes_now) / speed_bps), _ETA_CAP_SECONDS)
def _norm(s: str) -> str:
"""Lowercase, strip diacritics, and reduce to alphanumeric tokens, so 'Beyoncé' matches a
peer's 'Beyonce' folder and punctuation ('I Am… Sasha Fierce') doesn't defeat the match."""
s = unicodedata.normalize("NFKD", s or "")
s = "".join(ch for ch in s if not unicodedata.combining(ch))
s = "".join(ch if ch.isalnum() else " " for ch in s.lower())
return " ".join(s.split())
def _path_matches(needle: str, path: str) -> bool:
"""True if `needle` (an artist or album name) appears in a file path, tolerant of accents and
punctuation. Every significant token (≥3 chars) must be present; for a very short needle
(e.g. 'U2') the whole normalized string must appear."""
npath = _norm(path)
tokens = [t for t in _norm(needle).split() if len(t) >= 3]
if not tokens:
whole = _norm(needle)
return bool(whole) and whole in npath
return all(t in npath for t in tokens)
def _keep_matching(responses: list, needle: str) -> list:
"""Drop files — then now-empty responses — whose path doesn't match `needle`. Makes an album-
or artist-only fallback search precise so we never grab a same-titled release by a different
artist. Pure — unit-tested offline."""
out = []
for resp in responses:
files = [f for f in (resp.get("files") or []) if _path_matches(needle, f.get("filename", ""))]
if files:
out.append({**resp, "files": files})
return out
def _parse_search_responses(responses: list, artist: str = "") -> list[dict]:
"""Turn slskd search responses into album candidates (one per peer+directory), ordered so the
peers that will FINISH first come first: a free upload slot, then the shortest estimated
transfer time (album bytes ÷ the peer's advertised speed), then a shorter queue. Ranking on
estimated time — not raw speed — matters because Soulseek FLAC rips of the same album vary
~2x in size (a 24-bit/bloated rip vs a standard CD rip), and Lyra scores them the SAME quality
class (the search exposes no bit-depth), so a peer advertising high speed but serving huge
files can lose a race — and blow past the download backstop — against a peer with a smaller
standard rip. The pipeline tries candidates in this order, so this is what makes the Soulseek
fall-through converge on a copy that actually completes. Pure — no I/O — unit-tested offline."""
scored: list[tuple] = []
for resp in responses:
username = resp.get("username", "")
has_slot = 0 if resp.get("hasFreeUploadSlot") else 1 # 0 sorts first (free slot = starts now)
speed = int(resp.get("uploadSpeed") or 0)
queue = int(resp.get("queueLength") or 0)
by_dir: dict[str, list] = {}
for f in resp.get("files", []) or []:
name = f.get("filename", "")
ext = name.rsplit(".", 1)[-1].lower() if "." in name else ""
if ext not in _AUDIO_EXT:
continue
by_dir.setdefault(_dirname(name), []).append(
{"filename": name, "size": f.get("size", 0), "ext": ext}
)
for directory, dfiles in by_dir.items():
if not dfiles:
continue
# Estimated seconds to pull this album from this peer at its advertised rate. Unknown/
# zero speed → a low nominal (1 B/s) so total size still orders those peers last.
total_bytes = sum(int(f["size"] or 0) for f in dfiles)
est_seconds = total_bytes / (speed if speed > 0 else 1)
lossless = all(f["ext"] in _LOSSLESS_EXT for f in dfiles)
dirname = _basename(directory)
# Trust the REQUESTED artist when it appears anywhere in the path — many rips put the
# artist in a parent folder ("Rihanna\(2009) Rated R\…") or bury it ("2009 - Rihanna -
# Rated R"), which the "Artist - Album" folder parse gets wrong; that tanked the
# confidence score and lost real FLACs to clean-named MP3s. album stays the folder name
# (fuzzy-matched downstream); artist is what the parse gets wrong.
if artist and _path_matches(artist, dfiles[0]["filename"]):
guess_artist, guess_album = artist, dirname
elif " - " in dirname:
guess_artist, guess_album = (p.strip() for p in dirname.split(" - ", 1))
else:
guess_artist, guess_album = "", dirname
candidate = {
"source_ref": json.dumps({
"username": username,
"files": [{"filename": f["filename"], "size": f["size"]} for f in dfiles],
}),
"title": guess_album,
"artist": guess_artist,
"track_count": len(dfiles),
"format": "FLAC" if lossless else "MP3",
"bitrate": None,
}
scored.append((has_slot, est_seconds, queue, candidate))
# ascending: free slot first, then shortest estimated transfer, then shortest queue
scored.sort(key=lambda t: (t[0], t[1], t[2]))
return [c for *_rest, c in scored]
class SlskdClient:
"""Real Soulseek client talking to an slskd daemon's REST API.
@@ -58,11 +193,12 @@ class SlskdClient:
r.raise_for_status()
return r.json()
def search_album(self, artist: str, album: str) -> list[dict]:
def _run_search(self, text: str) -> list:
"""POST a search, wait for it to complete, and return the raw responses (a list)."""
r = requests.post(
f"{self._url}/api/v0/searches",
headers=self._headers(),
json={"searchText": f"{artist} {album}"},
json={"searchText": text},
timeout=30,
)
r.raise_for_status()
@@ -75,43 +211,27 @@ class SlskdClient:
break
responses = self._get(f"/api/v0/searches/{search_id}/responses")
candidates: list[dict] = []
for resp in responses:
username = resp.get("username", "")
by_dir: dict[str, list] = {}
for f in resp.get("files", []) or []:
name = f.get("filename", "")
ext = name.rsplit(".", 1)[-1].lower() if "." in name else ""
if ext not in _AUDIO_EXT:
return responses if isinstance(responses, list) else []
def search_album(self, artist: str, album: str) -> list[dict]:
responses = self._run_search(f"{artist} {album}")
if not responses:
# The Soulseek server silently drops searches containing certain blocklisted terms
# (major-label artists / album titles like Adele, Rihanna, Beyoncé, "Lemonade") →
# 0 responses. The blocked term poisons the whole query, but the OTHER field usually
# isn't blocked, so retry with each field alone (normalized) and keep only results
# whose path still matches the field we dropped. That recovers e.g. Beyoncé's
# "I Am… Sasha Fierce" via the album title, without ever grabbing a same-titled album
# by someone else.
for query, needle in ((album, artist), (artist, album)):
q = _norm(query)
if not q:
continue
by_dir.setdefault(_dirname(name), []).append(
{"filename": name, "size": f.get("size", 0), "ext": ext}
)
for directory, dfiles in by_dir.items():
if not dfiles:
continue
lossless = all(f["ext"] in _LOSSLESS_EXT for f in dfiles)
dirname = _basename(directory)
if " - " in dirname:
guess_artist, guess_album = (p.strip() for p in dirname.split(" - ", 1))
else:
guess_artist, guess_album = "", dirname
candidates.append(
{
"source_ref": json.dumps(
{
"username": username,
"files": [{"filename": f["filename"], "size": f["size"]} for f in dfiles],
}
),
"title": guess_album,
"artist": guess_artist,
"track_count": len(dfiles),
"format": "FLAC" if lossless else "MP3",
"bitrate": None,
}
)
return candidates
alt = _keep_matching(self._run_search(q), needle)
if alt:
responses = alt
break
return _parse_search_responses(responses, artist)
def download(self, source_ref: str, dest: str, on_progress: Callable[[float], None]) -> dict:
ref = json.loads(source_ref)
@@ -119,7 +239,17 @@ class SlskdClient:
files = ref["files"]
wanted = {f["filename"] for f in files}
os.makedirs(dest, exist_ok=True)
xfer_ids: set[str] = set() # slskd transfer ids we started, for cancel-on-abandon
try:
return self._download(username, files, wanted, dest, on_progress, xfer_ids)
except Exception:
# Abandoning this peer (timeout / error / the pipeline moving on): cancel the transfers
# we enqueued so they don't keep running and pile up in slskd as the job falls through
# to other peers/sources for the same album.
self._cancel_transfers(username, xfer_ids)
raise
def _download(self, username, files, wanted, dest, on_progress, xfer_ids: set) -> dict:
r = requests.post(
f"{self._url}/api/v0/transfers/downloads/{username}",
headers=self._headers(),
@@ -130,23 +260,49 @@ class SlskdClient:
on_progress(0.0)
total = len(files)
total_bytes = sum(int(f.get("size") or 0) for f in files)
completed = False
for _ in range(_XFER_POLLS):
last_bytes = -1
stall = 0
speed = 0.0 # bytes/sec, EWMA-smoothed so the ETA doesn't jitter poll to poll
prev_bytes = 0
prev_t = time.monotonic()
for _ in range(_max_polls(total_bytes)):
time.sleep(_POLL_SECONDS)
data = self._get(f"/api/v0/transfers/downloads/{username}")
states: list[str] = []
bytes_now = 0
for directory in data.get("directories", []) if isinstance(data, dict) else []:
for f in directory.get("files", []):
if f.get("filename") in wanted: # only the files we enqueued
states.append(str(f.get("state", "")))
bytes_now += int(f.get("bytesTransferred") or 0)
if f.get("id"): # remember the transfer so we can cancel it if abandoned
xfer_ids.add(str(f["id"]))
if states:
if any(bad in s for s in states for bad in ("Errored", "Rejected", "Cancelled")):
raise RuntimeError("soulseek transfer failed")
now = time.monotonic()
dt = now - prev_t
if dt > 0 and bytes_now >= prev_bytes:
inst = (bytes_now - prev_bytes) / dt
speed = inst if speed == 0.0 else 0.6 * speed + 0.4 * inst
prev_bytes, prev_t = bytes_now, now
done = sum(1 for s in states if "Completed" in s)
on_progress(min(1.0, done / max(total, 1)))
on_progress(min(1.0, done / max(total, 1)), _eta_seconds(total_bytes, bytes_now, speed))
if done >= total:
completed = True
break
# Stall detection: a peer making byte progress keeps its slot (up to the absolute
# backstop above) so a slow-but-working transfer can finish; one that sends no bytes
# for _STALL_POLLS (~90s) is queued/dead — abandon it fast and fall through.
if bytes_now > last_bytes:
last_bytes = bytes_now
stall = 0
else:
stall += 1
if stall >= _STALL_POLLS:
raise TimeoutError(f"soulseek transfer stalled for {username}")
if not completed:
raise TimeoutError(f"soulseek download did not complete for {username}")
@@ -162,6 +318,19 @@ class SlskdClient:
)
return {"track_count": moved, "path": dest}
def _cancel_transfers(self, username: str, xfer_ids: set) -> None:
"""Cancel the given in-flight slskd transfers for a peer. Best-effort: a cleanup failure
must never mask the original download failure."""
for tid in xfer_ids:
try:
requests.delete(
f"{self._url}/api/v0/transfers/downloads/{username}/{tid}",
headers=self._headers(),
timeout=15,
)
except Exception:
pass
def _retrieve(self, files: list[dict], dest: str) -> int:
"""Move the just-finished slskd files out of its downloads dir into `dest`.
worker/lyra_worker/claim.py
+38 -1
View File
@@ -24,13 +24,50 @@ def reclaim_stuck_jobs(conn: psycopg.Connection) -> int:
return n
_MAX_ATTEMPTS = 5
def requeue_or_fail(
conn: psycopg.Connection, job_id: str, error: str, max_attempts: int = _MAX_ATTEMPTS
) -> None:
"""Recover a job whose pipeline raised an *unhandled* exception (e.g. a dropped MusicBrainz
TLS connection surfacing as an SSL EOF). Without this the job is left in its mid-pipeline
'matching' state; since claim_next only picks 'requested' it is orphaned until the next
restart's reclaim_stuck_jobs — showing "Searching sources…" on the press indefinitely.
Such errors are usually transient, so requeue to 'requested' until the attempt cap, then fall
to 'needs_attention' so a persistently-failing job stays visible instead of looping forever.
`attempts` is bumped by claim_next on every claim, so it already counts tries."""
with conn.cursor() as cur:
cur.execute('SELECT attempts, "requestId" FROM "Job" WHERE id = %s', (job_id,))
row = cur.fetchone()
if row is None:
return
attempts, request_id = row
if attempts >= max_attempts:
cur.execute(
'UPDATE "Job" SET state = \'needs_attention\', error = %s, "claimedAt" = NULL, '
'"updatedAt" = now() WHERE id = %s',
(error, job_id),
)
cur.execute('UPDATE "Request" SET status = \'needs_attention\' WHERE id = %s', (request_id,))
else:
cur.execute(
'UPDATE "Job" SET state = \'requested\', "currentStage" = \'intake\', error = NULL, '
'"claimedAt" = NULL, "downloadProgress" = 0, "updatedAt" = now() WHERE id = %s',
(job_id,),
)
cur.execute('UPDATE "Request" SET status = \'pending\' WHERE id = %s', (request_id,))
conn.commit()
def claim_next(conn: psycopg.Connection) -> str | None:
"""Atomically claim the oldest queued job. Returns its id or None."""
with conn.cursor() as cur:
cur.execute(
"""
SELECT id FROM "Job"
WHERE state = 'requested'
WHERE state = 'requested' AND NOT paused
ORDER BY "createdAt" ASC
FOR UPDATE SKIP LOCKED
LIMIT 1
worker/lyra_worker/floor.py
+42
View File
@@ -0,0 +1,42 @@
import os
from lyra_worker.config import get_config
_TRUE = {"1", "true", "yes", "on"}
def press_paused(config: dict[str, str]) -> bool:
"""True when the whole press is paused — the worker stops claiming new jobs."""
return str(config.get("floor.paused", "")).strip().lower() in _TRUE
def job_delay_seconds(config: dict[str, str]) -> float:
"""Seconds to wait between finishing one download and claiming the next. Invalid or
absent -> 0.0 (back-to-back). Negatives are clamped to 0."""
try:
return max(0.0, float(int(config.get("floor.jobDelaySeconds", "") or 0)))
except (TypeError, ValueError):
return 0.0
def kill_requested(config: dict[str, str]) -> bool:
"""True when a force-stop of the in-flight download has been requested."""
return str(config.get("floor.killRequested", "")).strip().lower() in _TRUE
def consume_kill_switch(conn, exit_fn=os._exit) -> bool:
"""If a force-stop is pending, clear the flag and terminate the process (default
os._exit) so the in-flight download is aborted. The container restart policy brings the
worker back; startup clears staging and requeues the aborted job. Returns whether it
fired. exit_fn is injectable for tests."""
config = get_config(conn)
if not kill_requested(config):
return False
with conn.cursor() as cur:
cur.execute(
'UPDATE "Config" SET value = \'false\', "updatedAt" = now() WHERE key = %s',
("floor.killRequested",),
)
conn.commit()
exit_fn(0)
return True
worker/lyra_worker/main.py
+34 -1
View File
@@ -6,11 +6,12 @@ import uuid
import psycopg
from lyra_worker.claim import claim_next, reclaim_stuck_jobs
from lyra_worker.claim import claim_next, reclaim_stuck_jobs, requeue_or_fail
from lyra_worker.config import get_config
from lyra_worker.crypto import secret_key_problem
from lyra_worker.db import wait_for_db
from lyra_worker.discovery import DiscoveryConfig, DiscoveryResult, reset_pending, run_discovery
from lyra_worker.floor import press_paused, job_delay_seconds, consume_kill_switch
from lyra_worker.library import clear_staging_root
from lyra_worker.monitor import MonitorConfig, fulfill_owned_releases, reconcile, sweep
from lyra_worker.pipeline import run_pipeline
@@ -226,9 +227,30 @@ def _liveness_heartbeat() -> None:
time.sleep(HEARTBEAT_SECONDS)
KILL_POLL_SECONDS = 1.5
def _kill_switch_watch() -> None:
"""Poll floor.killRequested on a background daemon thread (its own DB connection) so a
force-stop aborts the process even while the main loop is blocked in a download. On any
DB hiccup, reconnect and keep watching."""
conn = wait_for_db()
while True:
try:
consume_kill_switch(conn)
except Exception:
try:
conn.close()
except Exception:
pass
conn = wait_for_db()
time.sleep(KILL_POLL_SECONDS)
def run_forever() -> None:
_require_secret_key()
threading.Thread(target=_liveness_heartbeat, daemon=True).start()
threading.Thread(target=_kill_switch_watch, daemon=True).start()
conn = wait_for_db()
clear_staging_root(STAGING_ROOT) # sweep any staging dirs orphaned by a prior crash
reclaimed = reclaim_stuck_jobs(conn) # requeue jobs a prior crash left mid-pipeline
@@ -252,6 +274,7 @@ def run_forever() -> None:
last_discover_tick = 0.0
last_heartbeat = 0.0
last_prune = 0.0
next_job_allowed = 0.0 # monotonic time before which no new job is claimed (inter-job delay)
try:
while True:
# A transient DB connection loss (e.g. AdminShutdown when Postgres restarts) raises
@@ -283,6 +306,12 @@ def run_forever() -> None:
maybe_run_scan(conn, resolver, probe, browser, config)
# Press paused (downloads only) or the inter-job delay hasn't elapsed: keep the
# loop alive (monitor/scan/discovery above still run) but claim nothing.
if press_paused(config) or now < next_job_allowed:
time.sleep(IDLE_SLEEP)
continue
job_id = claim_next(conn)
if job_id is None:
time.sleep(IDLE_SLEEP)
@@ -294,8 +323,12 @@ def run_forever() -> None:
except Exception as e: # one bad job must not take down the worker loop
print(f"worker: pipeline failed for job {job_id}: {e}", flush=True)
conn.rollback()
# Don't orphan the job in its mid-pipeline 'matching' state (claim_next only
# picks 'requested'): requeue for a bounded retry, then needs_attention.
requeue_or_fail(conn, job_id, str(e))
finish_job(conn, job_id, mcfg)
print(f"worker: finished job {job_id}", flush=True)
next_job_allowed = time.monotonic() + job_delay_seconds(config)
except psycopg.OperationalError as e:
print(f"worker: db connection lost ({e}); reconnecting...", flush=True)
try:
worker/lyra_worker/monitor.py
+59 -17
View File
@@ -86,16 +86,40 @@ def _set_state(conn: psycopg.Connection, rel_id: str, state: str) -> None:
cur.execute('UPDATE "MonitoredRelease" SET state = %s WHERE id = %s', (state, rel_id))
def _in_library_at_cutoff(conn: psycopg.Connection, artist: str, album: str, cutoff: int) -> bool:
def _in_library_at_cutoff(
conn: psycopg.Connection, artist: str, album: str, rg_mbid: str, cutoff: int
) -> bool:
"""True if the album is owned at/above `cutoff`. Matches on the release-group MBID when the
library row has one — robust against MusicBrainz artist-name canonicalization (e.g. the
followed 'Kanye West' imports as 'Ye') — and falls back to exact artist+album for scanned
rows that predate MBID capture."""
with conn.cursor() as cur:
cur.execute(
'SELECT 1 FROM "LibraryItem" WHERE artist = %s AND album = %s '
'AND "qualityClass" >= %s LIMIT 1',
(artist, album, cutoff),
'SELECT 1 FROM "LibraryItem" WHERE "qualityClass" >= %s AND ('
' ("rgMbid" IS NOT NULL AND "rgMbid" = %s) '
' OR ("rgMbid" IS NULL AND artist = %s AND album = %s)) LIMIT 1',
(cutoff, rg_mbid, artist, album),
)
return cur.fetchone() is not None
def _retire_failed_attempts(conn: psycopg.Connection, rel_id: str) -> None:
"""Delete prior failed (needs_attention) Request rows for this release (Job/Candidate cascade).
A hard-to-match release is re-pressed every retry interval; each failed attempt used to leave
a needs_attention Request behind, piling up phantom "needs attention" cards on the press even
after a later attempt imported the album. Retiring them when the release is fulfilled or
re-attempted keeps the press showing one live attempt at a time. Manual (non-monitor) requests
have no monitoredReleaseId and are never touched."""
with conn.cursor() as cur:
cur.execute(
'DELETE FROM "Request" r USING "Job" j '
'WHERE j."requestId" = r.id AND r."monitoredReleaseId" = %s '
"AND j.state = 'needs_attention'",
(rel_id,),
)
def _enqueue_one(conn: psycopg.Connection, rel_id: str, artist: str, album: str) -> None:
with conn.cursor() as cur:
cur.execute(
@@ -118,7 +142,8 @@ def enqueue_due(conn: psycopg.Connection, cfg: MonitorConfig) -> int:
cur.execute(
'SELECT mr.id, mr."artistName", mr.album, mr.state, mr."currentQualityClass", '
' (mr."firstGrabbedAt" IS NOT NULL '
' AND mr."firstGrabbedAt" < now() - make_interval(days => %s)) AS window_expired '
' AND mr."firstGrabbedAt" < now() - make_interval(days => %s)) AS window_expired, '
' mr."rgMbid" '
'FROM "MonitoredRelease" mr '
'WHERE mr.monitored = true '
' AND NOT mr.ignored ' # user-ignored releases are never enqueued
@@ -134,8 +159,11 @@ def enqueue_due(conn: psycopg.Connection, cfg: MonitorConfig) -> int:
rows = cur.fetchall()
enqueued = 0
for rel_id, artist, album, state, quality, window_expired in rows:
if _in_library_at_cutoff(conn, artist, album, cfg.quality_cutoff):
for rel_id, artist, album, state, quality, window_expired, rg_mbid in rows:
# A due release is about to be fulfilled or re-attempted; either way any earlier failed
# attempt is superseded, so retire it before deciding.
_retire_failed_attempts(conn, rel_id)
if _in_library_at_cutoff(conn, artist, album, rg_mbid, cfg.quality_cutoff):
_set_state(conn, rel_id, "fulfilled")
continue
if state == "grabbed" and (window_expired or (quality is not None and quality >= cfg.quality_cutoff)):
@@ -154,12 +182,19 @@ def fulfill_owned_releases(conn: psycopg.Connection, cfg: MonitorConfig) -> int:
with conn.cursor() as cur:
cur.execute(
'UPDATE "MonitoredRelease" mr SET state = \'fulfilled\', '
' "currentQualityClass" = li.q, '
' "currentQualityClass" = owned.q, '
' "firstGrabbedAt" = COALESCE(mr."firstGrabbedAt", now()) '
'FROM (SELECT artist, album, max("qualityClass") AS q FROM "LibraryItem" '
' GROUP BY artist, album) li '
'WHERE li.artist = mr."artistName" AND li.album = mr.album '
" AND mr.monitored = true AND mr.state <> 'fulfilled' AND li.q >= %s",
'FROM ('
' SELECT mr2.id AS mid, max(li."qualityClass") AS q '
' FROM "MonitoredRelease" mr2 JOIN "LibraryItem" li ON ('
# match on rgMbid where the library row has one (survives artist-name canonicalization),
# else fall back to exact artist+album for scanned rows lacking an MBID
' (li."rgMbid" IS NOT NULL AND li."rgMbid" = mr2."rgMbid") '
' OR (li."rgMbid" IS NULL AND li.artist = mr2."artistName" AND li.album = mr2.album)) '
" WHERE mr2.monitored = true AND mr2.state <> 'fulfilled' "
' GROUP BY mr2.id HAVING max(li."qualityClass") >= %s'
') owned '
'WHERE mr.id = owned.mid',
(cfg.quality_cutoff,),
)
fixed = cur.rowcount
@@ -171,21 +206,28 @@ def reconcile(conn: psycopg.Connection, job_id: str, cfg: MonitorConfig) -> None
"""Feed a finished job's outcome back into its MonitoredRelease (if any)."""
with conn.cursor() as cur:
cur.execute(
'SELECT j.state, r."monitoredReleaseId", r.artist, r.album '
'FROM "Job" j JOIN "Request" r ON r.id = j."requestId" WHERE j.id = %s',
'SELECT j.state, r."monitoredReleaseId", r.artist, r.album, mr."rgMbid" '
'FROM "Job" j JOIN "Request" r ON r.id = j."requestId" '
'LEFT JOIN "MonitoredRelease" mr ON mr.id = r."monitoredReleaseId" '
'WHERE j.id = %s',
(job_id,),
)
row = cur.fetchone()
if row is None:
return
state, rel_id, artist, album = row
state, rel_id, artist, album, rg_mbid = row
if rel_id is None or state != "imported":
return # needs_attention (or in-flight) leaves the release wanted
# A successful import supersedes any earlier failed attempts for this release.
_retire_failed_attempts(conn, rel_id)
with conn.cursor() as cur:
cur.execute(
'SELECT "qualityClass" FROM "LibraryItem" WHERE artist = %s AND album = %s',
(artist, album),
'SELECT max("qualityClass") FROM "LibraryItem" WHERE '
' ("rgMbid" IS NOT NULL AND "rgMbid" = %s) '
' OR ("rgMbid" IS NULL AND artist = %s AND album = %s)',
(rg_mbid, artist, album),
)
lib = cur.fetchone()
quality = lib[0] if lib else None
worker/lyra_worker/pipeline.py
+142 -43
View File
@@ -1,4 +1,5 @@
import os
import random
import shutil
import threading
import time
@@ -11,6 +12,7 @@ import psycopg
from lyra_worker.adapters.base import SourceAdapter
from lyra_worker.confidence import score_confidence
from lyra_worker.library import album_dir, import_album, list_audio_files, staging_dir
from lyra_worker.qobuz_gate import gate_open, record_download
from lyra_worker.quality import quality_class
from lyra_worker.ranker import rank_candidates
from lyra_worker.types import Candidate, MBTarget
@@ -20,8 +22,19 @@ _AUDIO_EXT = {".flac", ".mp3", ".m4a", ".opus", ".ogg", ".aac", ".wav"}
# Reject a download whose total playtime falls below this fraction of MusicBrainz's expected
# total — catches truncated files / preview clips substituted for real tracks that keep the
# track COUNT right. Generous, so edition/encoding differences don't false-positive; bonus
# tracks (over-long) are always fine.
_DURATION_MIN_RATIO = 0.85
# tracks (over-long) are always fine. Kept well below 1.0 because a source legitimately delivers a
# different edition than MB's resolved release (shorter radio edits, no bonus/live tracks), and only
# a genuinely truncated/preview-filled download (playtime a fraction of expected) should be caught.
_DURATION_MIN_RATIO = 0.65
# Cap how many ranked candidates a single job will actually download+verify before giving up.
# Without this, the incomplete-download fall-through can grind through dozens of sources (a popular
# album returns ~200 Soulseek candidates), each a slow attempt that also leaves an abandoned slskd
# transfer. The best candidates rank first, so 6 attempts is plenty.
_MAX_DOWNLOAD_ATTEMPTS = 6
# The best quality any non-Qobuz source offers (Soulseek FLAC = CD-lossless). A library copy at or
# above this can only be improved by hi-res Qobuz, so when Qobuz is gated by pacing an upgrade of
# such a copy is deferred without even searching.
_CD_LOSSLESS_CLASS = 2
def _measure_staged_duration_s(staging: str) -> float | None:
@@ -70,6 +83,40 @@ def _count_staged_audio(staging: str) -> int:
return n
def _download_problem(result, staging: str, target, winner, measure_duration) -> str | None:
"""Why a completed download is unusable — ``"incomplete download"`` / ``"download too short
(...)"`` — or None if it's good. Completeness is judged against the CHOSEN SOURCE's own track
count (``winner.track_count``), NOT MusicBrainz's canonical release: MB picks an arbitrary
releases[0] that is often a deluxe/expanded edition with more tracks (and runtime) than the
standard album a source legitimately delivers, so gating on ``target.track_count`` falsely
rejects complete standard-edition downloads (The Script "No Sound Without Silence" 11 vs MB 12,
Skillet "Unleashed" 12 vs MB's 20-track deluxe). Pure — no DB I/O — so the download loop can
call it per candidate and fall through to the next source on a problem."""
expected = winner.track_count
# (a) Truncated: fewer files than the source promised (a track failed mid-download). Count the
# NON-EMPTY audio on disk too: a silently-skipped track leaves a 0-byte placeholder the adapter
# still counts. (`staged and ...` keeps fake adapters that stage nothing on the count path.)
staged = _count_staged_audio(staging)
truncated = result.track_count < expected or bool(staged and staged < expected)
# (b) Implausibly small vs MB's album — a lone-track "full album" video / wrong match, as
# distinct from a legitimately smaller edition. A real edition keeps more than half MB's tracks.
too_small = target.track_count is not None and expected * 2 <= target.track_count
if truncated or too_small:
return "incomplete download"
# Duration: a truncated file or short preview substituted for a real track shows up as playtime
# well under the expected total. Scale MB's total to the DELIVERED edition's size so a smaller
# edition isn't judged against a larger one's runtime. Only when the total is known (measured is
# None for the offline fakes).
if target.total_duration_s:
expected_total = target.total_duration_s
if target.track_count:
expected_total *= winner.track_count / target.track_count
measured = measure_duration(staging)
if measured is not None and measured < expected_total * _DURATION_MIN_RATIO:
return f"download too short ({measured / 60:.0f} of ~{expected_total / 60:.0f} min)"
return None
def _poll_download_progress(job_id: str, staging: str, expected: int, stop: "threading.Event",
reports: "threading.Event") -> None:
"""Until `stop`, periodically write Job.downloadProgress = files-in-staging / expected
@@ -114,30 +161,42 @@ def _set_state(conn: psycopg.Connection, job_id: str, state: str, stage: str) ->
conn.commit()
def _set_download_progress(conn: psycopg.Connection, job_id: str, frac: float) -> None:
def _set_download_progress(
conn: psycopg.Connection, job_id: str, frac: float, eta_seconds: int | None = None
) -> None:
with conn.cursor() as cur:
cur.execute('UPDATE "Job" SET "downloadProgress" = %s WHERE id = %s', (frac, job_id))
cur.execute(
'UPDATE "Job" SET "downloadProgress" = %s, "downloadEtaSeconds" = %s WHERE id = %s',
(frac, eta_seconds, job_id),
)
conn.commit()
def _make_on_progress(conn: psycopg.Connection, job_id: str, reports: "threading.Event"):
"""Build the on_progress(pct) callback threaded into adapter.download. It marks the
adapter as reporting real byte-level progress (so the file-count poller yields) and
throttle-writes Job.downloadProgress. Called synchronously from adapter.download on the
pipeline's own thread, so it safely reuses `conn`."""
"""Build the on_progress(pct, eta_seconds=None) callback threaded into adapter.download. It
marks the adapter as reporting real byte-level progress (so the file-count poller yields) and
throttle-writes Job.downloadProgress (+ the download ETA when the adapter measures one — slskd
does; others pass None). Called synchronously from adapter.download on the pipeline's own
thread, so it safely reuses `conn`."""
state = {"frac": -1.0, "t": 0.0}
def on_progress(pct: float) -> None:
def on_progress(pct: float, eta_seconds: int | None = None) -> None:
reports.set()
frac = 0.0 if pct < 0 else 1.0 if pct > 1 else float(pct)
now = time.monotonic()
# throttle DB writes: on a >=1% move or every 0.5s, not once per received byte
if frac - state["frac"] >= 0.01 or (now - state["t"]) >= 0.5:
# throttle DB writes: on a >=1% move, a new ETA, or every 0.5s — not once per received byte
if frac - state["frac"] >= 0.01 or eta_seconds is not None or (now - state["t"]) >= 0.5:
state["frac"] = frac
state["t"] = now
try:
_set_download_progress(conn, job_id, frac)
_set_download_progress(conn, job_id, frac, eta_seconds)
except Exception as e: # a progress write must never fail the download
# Roll back so a failed write can't leave the shared pipeline conn in an aborted
# txn (which would cascade "current transaction is aborted" into every later query).
try:
conn.rollback()
except Exception:
pass
print(f"pipeline: on_progress write failed: {e}", flush=True)
return on_progress
@@ -225,6 +284,17 @@ def _library_quality(conn: psycopg.Connection, target: MBTarget) -> int | None:
return row[0] if row else None
def _keep_existing_and_complete(conn: psycopg.Connection, job_id: str) -> None:
"""Finish a job without importing — keep the existing library copy — and mark its request
completed. Used when a (deferred) upgrade has nothing better to offer right now; the release
stays below the cutoff and is re-attempted later."""
_set_state(conn, job_id, "imported", "import")
with conn.cursor() as cur:
cur.execute("UPDATE \"Request\" SET status = 'completed' WHERE id = %s",
(_request_id(conn, job_id),))
conn.commit()
def _persist_candidates(conn: psycopg.Connection, job_id: str, candidates: list[Candidate]) -> None:
with conn.cursor() as cur:
for c in candidates:
@@ -317,6 +387,16 @@ def run_pipeline(
conn.commit()
return
# Pre-search upgrade skip: a >= CD-lossless copy can only be improved by hi-res Qobuz. If Qobuz
# is gated by pacing, don't even search (no other source can beat it) — defer to when Qobuz
# frees up. This avoids re-searching every already-owned album (~2 min Soulseek search each) on
# every monitor cycle. A user-forced re-acquire always proceeds.
_existing_q = _library_quality(conn, target)
if _existing_q is not None and _existing_q >= _CD_LOSSLESS_CLASS \
and not _is_force_job(conn, job_id) and not gate_open(conn):
_keep_existing_and_complete(conn, job_id)
return
# 2. match — search all adapters CONCURRENTLY (Soulseek's search polls up to ~2 min; run it
# alongside Qobuz's instant search so total match time = the slowest source, not their sum).
# Only Qobuz touches streamrip's shared asyncio loop and there's a single Qobuz adapter, so no
@@ -329,8 +409,21 @@ def run_pipeline(
for adapter, results in per_adapter:
for c in results:
found.append(replace(c, source_tier=adapter.tier, confidence=score_confidence(target, c)))
# Drop candidates that can't be the full album — a result with far fewer tracks than the release
# has (e.g. a single-song Soulseek folder named like the album). Same threshold as the
# completeness 'too small' guard, applied up front so we never waste a download attempt on it.
# Unknown counts (0) and legitimately smaller editions (e.g. 11 vs 12) are kept.
if target.track_count:
found = [c for c in found if c.track_count == 0 or c.track_count * 2 > target.track_count]
_persist_candidates(conn, job_id, found)
# Qobuz pacing: when the budget gate is closed (off-hours / daily cap / spacing), drop Qobuz
# candidates so the job falls through to another source and Qobuz stays under the radar — UNLESS
# Qobuz is the only option (nothing to fall through to, so don't strand the album).
if any(c.source == "qobuz" for c in found) and any(c.source != "qobuz" for c in found):
if not gate_open(conn):
found = [c for c in found if c.source != "qobuz"]
# 3. rank
_set_state(conn, job_id, "matched", "rank")
ranked = rank_candidates(target, found, min_confidence)
@@ -338,12 +431,23 @@ def run_pipeline(
_fail(conn, job_id, "no candidate above confidence threshold")
return
# Upgrade guard: for an album already in the library, skip the download when no currently-
# available source beats the copy we have (ranked[0] is the highest-quality candidate). This
# happens when Qobuz is gated by pacing and only a same-quality Soulseek copy is left — the
# download would be discarded by keep-if-better, so defer it. The release stays below the cutoff
# and is re-attempted when a better source frees up. A user-forced re-acquire always proceeds.
existing_q = _library_quality(conn, target)
if existing_q is not None and not _is_force_job(conn, job_id) \
and quality_class(ranked[0].quality) <= existing_q:
_keep_existing_and_complete(conn, job_id) # keep the existing copy; nothing better available
return
# 4. download (fall-through) into an isolated per-job staging dir
_set_state(conn, job_id, "downloading", "download")
by_source = {a.name: a for a in adapters}
staging = staging_dir(staging_root or f"{dest_root}/.staging", job_id)
winner = None
result = None
last_problem = None # why the most recent downloaded-but-rejected candidate was unusable
_set_download_progress(conn, job_id, 0.0) # reset for this run
expected = target.track_count or (ranked[0].track_count if ranked else 0)
_stop = threading.Event()
@@ -354,46 +458,46 @@ def run_pipeline(
_poller.start()
try:
try:
attempts = 0
for candidate in ranked:
adapter = by_source.get(candidate.source)
if adapter is None:
continue
if attempts >= _MAX_DOWNLOAD_ATTEMPTS:
break # don't grind through every peer/source of a popular album
attempts += 1
_mark_chosen(conn, job_id, candidate.source_ref) # reflect the source now in flight
shutil.rmtree(staging, ignore_errors=True) # clean slate per attempt
_reports.clear() # this attempt hasn't reported yet → poller estimates until it does
result = adapter.download(candidate, staging, _make_on_progress(conn, job_id, _reports))
if result.ok:
if not result.ok:
continue
# Verify completeness on the staging copy. A download that lands incomplete or too
# short falls through to the NEXT-ranked source (e.g. Qobuz repeatedly fails one
# track → try the deluxe edition or Soulseek) rather than failing the whole job on
# the first source's shortfall.
last_problem = _download_problem(result, staging, target, candidate, measure_duration)
if last_problem is None:
winner = candidate
break
finally:
_stop.set()
_poller.join(timeout=3)
if winner is None or result is None or not result.ok:
_fail(conn, job_id, "all downloads failed")
if winner is None:
# No source produced a complete album: surface why the last one was rejected (incomplete
# / too short), or "all downloads failed" if none even downloaded.
_fail(conn, job_id, last_problem or "all downloads failed")
return
_set_download_progress(conn, job_id, 1.0) # download done — UI shows 100% into Finishing
if winner.source == "qobuz":
# Count this Qobuz download against today's budget and set the next randomized spacing.
try:
record_download(conn, random.random())
except Exception as e: # pacing bookkeeping must never fail a good download
print(f"pipeline: qobuz pacing record failed for job {job_id}: {e}", flush=True)
# 5. verify completeness on the staging copy, then promote + tag
# 5. promote + tag the verified winner
_set_state(conn, job_id, "tagging", "tag")
expected = target.track_count if target.track_count is not None else winner.track_count
# Also count the NON-EMPTY audio actually on disk: a silently-skipped track leaves a
# 0-byte placeholder that the adapter still counts, so trust the files when any are
# staged. (`staged and ...` keeps fake adapters that stage nothing on the count path.)
staged = _count_staged_audio(staging)
if result.track_count < expected or (staged and staged < expected):
_fail(conn, job_id, "incomplete download")
return
# Duration check: even with the right track count, a truncated file or a short preview
# substituted for a real track shows up as a total playtime well under MB's expected
# total. Only applies when both are known (measured is None for the offline fakes).
if target.total_duration_s:
measured = measure_duration(staging)
if measured is not None and measured < target.total_duration_s * _DURATION_MIN_RATIO:
_fail(conn, job_id,
f"download too short ({measured / 60:.0f} of ~{target.total_duration_s / 60:.0f} min)")
return
final = album_dir(dest_root, target)
existing_q = _library_quality(conn, target)
new_q = quality_class(winner.quality)
@@ -407,12 +511,7 @@ def run_pipeline(
print(f"pipeline: tagging failed for job {job_id}: {e}", flush=True)
_import(conn, job_id, target, winner, final)
else:
# an existing copy is already at >= this quality — keep it untouched, just complete the request
with conn.cursor() as cur:
cur.execute(
"UPDATE \"Request\" SET status = 'completed' WHERE id = %s",
(_request_id(conn, job_id),),
)
conn.commit()
# an existing copy is already at >= this quality — keep it untouched, just complete
_keep_existing_and_complete(conn, job_id)
finally:
shutil.rmtree(staging, ignore_errors=True) # a partial/failed download never persists
worker/lyra_worker/qobuz_gate.py
+159
View File
@@ -0,0 +1,159 @@
"""Qobuz pacing ("budget gate").
Keep using Qobuz for hi-res, but at a human-looking volume and cadence so the account isn't
flagged for bulk downloading (which got it 403 USER_BLOCKED). One gate decides, per album,
whether Qobuz is an eligible source right now: within active hours, under today's cap (a warm-up
ramp of a steady cap), and past a randomized spacing since the last Qobuz download. When the gate
is closed the pipeline drops Qobuz candidates and falls through to the next source; monitored
albums upgrade to Qobuz later once budget frees up.
Pure decision logic (`today_cap`, `is_allowed`, `next_gap_seconds`) is separated from the two DB
helpers (`gate_open`, `record_download`) so the policy is unit-tested without a clock or DB.
"""
from dataclasses import dataclass
from datetime import date
_TRUE = {"1", "true", "yes", "on"}
_JITTER = 0.3 # ± fraction around the spread interval
_GAP_FLOOR_S = 60.0
@dataclass(frozen=True)
class QobuzPacing:
enabled: bool = True
active_start: int = 8
active_end: int = 23
daily_cap: int = 40
warmup_start_date: str = "" # ISO "YYYY-MM-DD", or "" for no warm-up ramp
@classmethod
def from_config(cls, config: dict) -> "QobuzPacing":
def _int(key, default):
try:
return int(config[key])
except (KeyError, TypeError, ValueError):
return default
return cls(
enabled=str(config.get("qobuz.pacing.enabled", "true")).strip().lower() in _TRUE,
active_start=_int("qobuz.pacing.activeStartHour", 8),
active_end=_int("qobuz.pacing.activeEndHour", 23),
daily_cap=_int("qobuz.pacing.dailyCap", 40),
warmup_start_date=str(config.get("qobuz.pacing.warmupStartDate", "") or "").strip(),
)
def today_cap(pacing: QobuzPacing, today: date) -> int:
"""Today's Qobuz download cap: the steady cap, ramped up by account age (days since
``warmup_start_date``). No warm-up date -> steady cap. A small steady cap is never exceeded."""
steady = max(0, pacing.daily_cap)
if not pacing.warmup_start_date:
return steady
try:
start = date.fromisoformat(pacing.warmup_start_date)
except ValueError:
return steady
days = max(0, (today - start).days)
if days < 3:
return min(5, steady)
if days < 7:
return min(10, steady)
if days < 14:
return min(20, steady)
return steady
def is_allowed(pacing: QobuzPacing, now_hour: int, count_today: int, cap: int,
now_epoch: float, next_allowed_epoch: float) -> bool:
"""Pure gate decision. Qobuz is eligible only within active hours, under today's cap, and past
the randomized spacing since the last download. Pacing disabled -> always allowed."""
if not pacing.enabled:
return True
if not (pacing.active_start <= now_hour < pacing.active_end):
return False
if count_today >= cap:
return False
if now_epoch < next_allowed_epoch:
return False
return True
def next_gap_seconds(pacing: QobuzPacing, cap: int, rand01: float) -> float:
"""Seconds until the next Qobuz download is allowed: the day's budget spread across the active
window (``activeWindow / cap``), ± _JITTER, floored at _GAP_FLOOR_S. ``rand01`` in [0,1] is the
injected randomness (so tests are deterministic; callers pass random.random())."""
window_s = max(1, pacing.active_end - pacing.active_start) * 3600
base = window_s / max(1, cap)
lo, hi = base * (1 - _JITTER), base * (1 + _JITTER)
gap = lo + (hi - lo) * rand01
return max(_GAP_FLOOR_S, gap)
def _now_parts(conn):
"""(hour:int, today:date, epoch:float) from the DB clock — avoids container-TZ ambiguity."""
with conn.cursor() as cur:
cur.execute("SELECT extract(hour from now())::int, current_date, extract(epoch from now())")
hour, today, epoch = cur.fetchone()
return int(hour), today, float(epoch)
def _read(conn, keys):
with conn.cursor() as cur:
cur.execute('SELECT key, value FROM "Config" WHERE key = ANY(%s)', (list(keys),))
return dict(cur.fetchall())
def gate_open(conn) -> bool:
"""Whether Qobuz may be used for a download right now (reads pacing config + counter/spacing
state from Config and the DB clock). A counter from a previous day is treated as 0."""
from lyra_worker.config import get_config
pacing = QobuzPacing.from_config(get_config(conn))
hour, today, epoch = _now_parts(conn)
state = _read(conn, ("qobuz.pacing.countDay", "qobuz.pacing.countToday", "qobuz.pacing.nextAllowedAt"))
count = 0
if state.get("qobuz.pacing.countDay") == today.isoformat():
try:
count = int(state.get("qobuz.pacing.countToday") or 0)
except (TypeError, ValueError):
count = 0
try:
next_allowed = float(state.get("qobuz.pacing.nextAllowedAt") or 0)
except (TypeError, ValueError):
next_allowed = 0.0
return is_allowed(pacing, hour, count, today_cap(pacing, today), epoch, next_allowed)
def record_download(conn, rand01: float) -> None:
"""Record a successful Qobuz download: bump today's counter (resetting it if the stored day is
stale) and set the randomized next-allowed time. ``rand01`` is injected randomness."""
from lyra_worker.config import get_config
pacing = QobuzPacing.from_config(get_config(conn))
_hour, today, epoch = _now_parts(conn)
state = _read(conn, ("qobuz.pacing.countDay", "qobuz.pacing.countToday"))
if state.get("qobuz.pacing.countDay") == today.isoformat():
try:
count = int(state.get("qobuz.pacing.countToday") or 0)
except (TypeError, ValueError):
count = 0
else:
count = 0
count += 1
next_allowed = epoch + next_gap_seconds(pacing, today_cap(pacing, today), rand01)
_write(conn, {
"qobuz.pacing.countDay": today.isoformat(),
"qobuz.pacing.countToday": str(count),
"qobuz.pacing.nextAllowedAt": repr(next_allowed),
})
def _write(conn, kv: dict) -> None:
with conn.cursor() as cur:
for key, value in kv.items():
cur.execute(
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
(key, value),
)
conn.commit()
worker/tests/test_candidate_filter.py
+57
View File
@@ -0,0 +1,57 @@
"""A search result with far fewer tracks than the release (e.g. a single-song Soulseek folder
named like the album) can't be the full album — it must be dropped before we waste a download."""
import os
from lyra_worker.claim import claim_next
from lyra_worker.pipeline import run_pipeline
from lyra_worker.types import Candidate, DownloadResult, MBTarget, Quality
from tests.conftest import insert_request
_Q = Quality(fmt="FLAC", lossless=True, bit_depth=24, sample_rate=96000)
class _TwoCandQobuz:
name = "qobuz"
tier = 0
def __init__(self):
self.downloaded = []
def health(self):
return True
def search(self, target):
return [
Candidate(source="qobuz", source_ref="single", matched_artist=target.artist,
matched_album=target.album, quality=_Q, track_count=1, source_tier=0),
Candidate(source="qobuz", source_ref="full", matched_artist=target.artist,
matched_album=target.album, quality=_Q, track_count=12, source_tier=0),
]
def download(self, candidate, dest, on_progress):
self.downloaded.append(candidate.source_ref)
os.makedirs(dest, exist_ok=True)
for i in range(candidate.track_count):
with open(os.path.join(dest, f"{i + 1:02d}.flac"), "wb") as fh:
fh.write(b"a")
return DownloadResult(ok=True, path=dest, track_count=candidate.track_count)
class _Resolver:
def __init__(self, n):
self.n = n
def resolve(self, artist, album):
return MBTarget(artist=artist, album=album, track_count=self.n)
def test_single_song_candidate_is_dropped(conn):
job_id = insert_request(conn, artist="A", album="B")
claim_next(conn)
adapter = _TwoCandQobuz()
run_pipeline(conn, job_id, [adapter], resolver=_Resolver(12), dest_root="/tmp/lib-cf")
assert adapter.downloaded == ["full"] # only the full album was ever attempted
with conn.cursor() as cur:
cur.execute('SELECT "trackCount" FROM "Candidate" WHERE "jobId" = %s', (job_id,))
assert [r[0] for r in cur.fetchall()] == [12] # the single-song folder wasn't even persisted
worker/tests/test_claim.py
+14
View File
@@ -24,3 +24,17 @@ def test_claim_does_not_repick_claimed_job(conn):
insert_request(conn)
assert claim_next(conn) is not None
assert claim_next(conn) is None
def test_claim_skips_paused_jobs(conn):
paused_job = insert_request(conn, album="Paused Album")
with conn.cursor() as cur:
cur.execute('UPDATE "Job" SET paused = true WHERE id = %s', (paused_job,))
conn.commit()
unpaused_job = insert_request(conn, album="Live Album")
claimed = claim_next(conn)
assert claimed == unpaused_job # the paused one is skipped
# With only the paused job left, nothing is claimable.
assert claim_next(conn) is None
worker/tests/test_floor.py
+50
View File
@@ -0,0 +1,50 @@
from lyra_worker.floor import (
press_paused,
job_delay_seconds,
kill_requested,
consume_kill_switch,
)
def test_press_paused_reads_truthy_values():
assert press_paused({"floor.paused": "true"}) is True
assert press_paused({"floor.paused": "ON"}) is True
assert press_paused({"floor.paused": "false"}) is False
assert press_paused({}) is False
def test_job_delay_seconds_parses_and_defaults():
assert job_delay_seconds({"floor.jobDelaySeconds": "30"}) == 30.0
assert job_delay_seconds({}) == 0.0
assert job_delay_seconds({"floor.jobDelaySeconds": "-5"}) == 0.0
assert job_delay_seconds({"floor.jobDelaySeconds": "junk"}) == 0.0
def test_kill_requested_reads_flag():
assert kill_requested({"floor.killRequested": "true"}) is True
assert kill_requested({}) is False
def test_consume_kill_switch_clears_flag_and_exits(conn):
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "Config"(key, value, secret, "updatedAt") '
"VALUES ('floor.killRequested', 'true', false, now())"
)
conn.commit()
calls = []
fired = consume_kill_switch(conn, exit_fn=lambda code: calls.append(code))
assert fired is True
assert calls == [0]
with conn.cursor() as cur:
cur.execute("SELECT value FROM \"Config\" WHERE key = 'floor.killRequested'")
assert cur.fetchone()[0] == "false"
def test_consume_kill_switch_noop_when_unset(conn):
calls = []
fired = consume_kill_switch(conn, exit_fn=lambda code: calls.append(code))
assert fired is False
assert calls == []
worker/tests/test_incomplete_fallthrough.py
+125
View File
@@ -0,0 +1,125 @@
"""When the top-ranked source downloads but is incomplete (e.g. Qobuz consistently fails one
track), the pipeline must fall through to the next-ranked source instead of failing the whole
job — only giving up (needs_attention) when every candidate is incomplete."""
import os
from lyra_worker.claim import claim_next
from lyra_worker.pipeline import run_pipeline
from lyra_worker.types import Candidate, DownloadResult, Quality
from tests.conftest import insert_request
_HIRES = Quality(fmt="FLAC", lossless=True, bit_depth=24, sample_rate=96000)
_CD = Quality(fmt="FLAC", lossless=True, bit_depth=16, sample_rate=44100)
def _stage(dest, n):
os.makedirs(dest, exist_ok=True)
for i in range(n):
with open(os.path.join(dest, f"{i + 1:02d}.flac"), "wb") as fh:
fh.write(b"audio")
class _Qobuz:
"""Higher-quality (ranks first) Qobuz that promises 3 tracks but delivers `deliver`."""
name = "qobuz"
tier = 0
def __init__(self, deliver):
self._deliver = deliver
def health(self):
return True
def search(self, target):
return [Candidate(source="qobuz", source_ref="q", matched_artist=target.artist,
matched_album=target.album, quality=_HIRES, track_count=3, source_tier=0)]
def download(self, candidate, dest, on_progress):
_stage(dest, self._deliver)
return DownloadResult(ok=True, path=dest, track_count=self._deliver)
class _Soulseek:
"""Lower-quality (ranks second) Soulseek that promises 3 tracks and delivers `deliver`."""
name = "soulseek"
tier = 1
def __init__(self, deliver):
self._deliver = deliver
def health(self):
return True
def search(self, target):
return [Candidate(source="soulseek", source_ref="s", matched_artist=target.artist,
matched_album=target.album, quality=_CD, track_count=3, source_tier=1)]
def download(self, candidate, dest, on_progress):
_stage(dest, self._deliver)
return DownloadResult(ok=True, path=dest, track_count=self._deliver)
def _state_error(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT state, error FROM "Job" WHERE id = %s', (job_id,))
return cur.fetchone()
def _chosen_source(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT source FROM "Candidate" WHERE "jobId" = %s AND chosen = true', (job_id,))
row = cur.fetchone()
return row[0] if row else None
def test_incomplete_top_source_falls_through_to_complete_next(conn, tmp_path):
# Qobuz (ranks first on quality) delivers only 2 of its promised 3 tracks -> incomplete.
# The pipeline must try the next source (Soulseek, complete) rather than failing the job.
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [_Qobuz(deliver=2), _Soulseek(deliver=3)], dest_root=str(tmp_path))
assert _state_error(conn, job_id) == ("imported", None)
assert _chosen_source(conn, job_id) == "soulseek"
def test_all_sources_incomplete_still_needs_attention(conn, tmp_path):
# Every candidate is incomplete -> no fallback succeeds -> needs_attention (incomplete).
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [_Qobuz(deliver=2), _Soulseek(deliver=1)], dest_root=str(tmp_path))
assert _state_error(conn, job_id) == ("needs_attention", "incomplete download")
assert not (tmp_path / "Radiohead" / "In Rainbows").exists() # nothing imported
class _CountingIncompleteQobuz:
"""Offers `n` candidates that all download but come back incomplete; counts download calls."""
name = "qobuz"
tier = 0
def __init__(self, n):
self.n = n
self.calls = 0
def health(self):
return True
def search(self, target):
q = _HIRES
return [Candidate(source="qobuz", source_ref=f"r{i}", matched_artist=target.artist,
matched_album=target.album, quality=q, track_count=3, source_tier=0)
for i in range(self.n)]
def download(self, candidate, dest, on_progress):
self.calls += 1
_stage(dest, 1) # 1 of 3 → always incomplete
return DownloadResult(ok=True, path=dest, track_count=1)
def test_fall_through_is_capped(conn):
from lyra_worker.pipeline import _MAX_DOWNLOAD_ATTEMPTS
a = _CountingIncompleteQobuz(n=20)
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [a], dest_root="/tmp/lib-cap")
assert _state_error(conn, job_id) == ("needs_attention", "incomplete download")
assert a.calls == _MAX_DOWNLOAD_ATTEMPTS # capped, not all 20 peers tried
worker/tests/test_monitor_enqueue.py
+46
View File
@@ -72,6 +72,52 @@ def test_grabbed_past_window_is_fulfilled_not_enqueued(conn):
assert _state(conn, rel_id) == "fulfilled"
def test_owned_by_rgmbid_despite_name_mismatch_is_fulfilled(conn):
# The monitor followed "Kanye West" but the library stores MB's canonical "Ye"; matching on
# rgMbid recognises ownership so the release is fulfilled, not re-pressed.
rel_id = insert_monitored_release(conn, artist_name="Kanye West", album="Graduation",
rg_mbid="rg-grad", monitored=True, state="wanted")
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "LibraryItem" (id, artist, album, path, source, format, '
'"qualityClass", "rgMbid", "importedAt") '
"VALUES (gen_random_uuid()::text, 'Ye', 'Graduation', '/m', 'soulseek', 'FLAC', 2, "
"'rg-grad', now())"
)
conn.commit()
assert enqueue_due(conn, CFG) == 0
assert _state(conn, rel_id) == "fulfilled"
def test_re_enqueue_retires_prior_failed_attempt(conn):
# A hard-to-match release re-pressed each retry interval must not pile up needs_attention
# Request rows; the prior failed attempt is retired when the next one is enqueued.
rel_id = insert_monitored_release(conn, artist_name="X", album="Y", rg_mbid="rg-1",
monitored=True, state="wanted")
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "Request" (id, artist, album, status, "monitoredReleaseId", "createdAt") '
"VALUES (gen_random_uuid()::text, 'X', 'Y', 'needs_attention', %s, now()) RETURNING id",
(rel_id,),
)
req = cur.fetchone()[0]
cur.execute(
'INSERT INTO "Job" (id, "requestId", state, "currentStage", attempts, "createdAt", "updatedAt") '
"VALUES (gen_random_uuid()::text, %s, 'needs_attention', 'rank', 1, now(), now())",
(req,),
)
conn.commit()
assert enqueue_due(conn, CFG) == 1
assert _job_count_for(conn, rel_id) == 1 # old failed attempt retired, one fresh attempt
with conn.cursor() as cur:
cur.execute(
'SELECT j.state FROM "Job" j JOIN "Request" r ON r.id = j."requestId" '
'WHERE r."monitoredReleaseId" = %s',
(rel_id,),
)
assert cur.fetchone()[0] == "requested"
def test_already_in_library_at_cutoff_is_fulfilled(conn):
rel_id = insert_monitored_release(conn, artist_name="A", album="B", rg_mbid="rg-1",
monitored=True, state="wanted")
worker/tests/test_monitor_reconcile.py
+50 -4
View File
@@ -22,17 +22,27 @@ def _make_job(conn, rel_id, artist, album, state):
return req, job
def _add_library(conn, req, artist, album, quality):
def _add_library(conn, req, artist, album, quality, rg_mbid=None):
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "LibraryItem" (id, "requestId", artist, album, path, source, format, '
'"qualityClass", "importedAt") '
"VALUES (gen_random_uuid()::text, %s, %s, %s, '/m', 'qobuz', 'FLAC', %s, now())",
(req, artist, album, quality),
'"qualityClass", "rgMbid", "importedAt") '
"VALUES (gen_random_uuid()::text, %s, %s, %s, '/m', 'qobuz', 'FLAC', %s, %s, now())",
(req, artist, album, quality, rg_mbid),
)
conn.commit()
def _needs_attention_count(conn, rel_id):
with conn.cursor() as cur:
cur.execute(
'SELECT count(*) FROM "Request" r JOIN "Job" j ON j."requestId" = r.id '
"WHERE r.\"monitoredReleaseId\" = %s AND j.state = 'needs_attention'",
(rel_id,),
)
return cur.fetchone()[0]
def _release(conn, rel_id):
with conn.cursor() as cur:
cur.execute(
@@ -106,6 +116,42 @@ def test_fulfill_owned_releases_ignores_below_cutoff(conn):
assert _release(conn, rel)[0] == "wanted"
def test_reconcile_matches_by_rgmbid_when_artist_name_canonicalized(conn):
# Real 'Kanye West' -> 'Ye' bug: the MonitoredRelease keeps the followed name, but the
# imported LibraryItem carries MusicBrainz's canonical artist. Matching on rgMbid lets the
# release reconcile to fulfilled instead of looking un-owned and re-pressing forever.
rel = insert_monitored_release(conn, artist_name="Kanye West", album="Graduation",
rg_mbid="rg-grad", monitored=True, state="wanted")
req, job = _make_job(conn, rel, "Kanye West", "Graduation", "imported")
_add_library(conn, req, "Ye", "Graduation", 3, rg_mbid="rg-grad")
reconcile(conn, job, CFG)
assert _release(conn, rel) == ("fulfilled", 3, True)
def test_reconcile_fulfilled_retires_failed_sibling_attempts(conn):
# 'V' by Maroon 5 phantom: earlier failed attempts leave needs_attention Request rows that
# keep showing on the press even after a later attempt imports. A successful import retires
# those failed siblings so the press stops showing an owned album as needing attention.
rel = insert_monitored_release(conn, artist_name="Maroon 5", album="V", rg_mbid="rg-v",
monitored=True, state="wanted")
_make_job(conn, rel, "Maroon 5", "V", "needs_attention")
_make_job(conn, rel, "Maroon 5", "V", "needs_attention")
req, job = _make_job(conn, rel, "Maroon 5", "V", "imported")
_add_library(conn, req, "Maroon 5", "V", 2)
reconcile(conn, job, CFG)
assert _release(conn, rel)[0] == "fulfilled"
assert _needs_attention_count(conn, rel) == 0 # failed siblings retired
def test_fulfill_owned_releases_matches_by_rgmbid(conn):
# Ne-Yo -> Ne‐Yo (typographic hyphen) name drift: still owned by rgMbid.
rel = insert_monitored_release(conn, artist_name="Ne-Yo", album="Because of You",
rg_mbid="rg-ney", monitored=True, state="grabbed")
_add_library(conn, None, "Ne‐Yo", "Because of You", 2, rg_mbid="rg-ney")
assert fulfill_owned_releases(conn, MonitorConfig(quality_cutoff=2)) == 1
assert _release(conn, rel) == ("fulfilled", 2, True)
def test_unlinked_job_is_ignored(conn):
with conn.cursor() as cur:
cur.execute(
worker/tests/test_musicbrainz.py
+174
View File
@@ -0,0 +1,174 @@
import musicbrainzngs
import pytest
from lyra_worker._musicbrainz import _best_release_group, _pick_release, _with_retry
# Offline unit tests for the pure release-group selection logic. The live resolve()
# path (network) is covered by test_musicbrainz_live.py.
def _rg(title, primary_type, artist=""):
g = {"id": title.lower(), "title": title, "primary-type": primary_type}
if artist:
g["artist-credit"] = [{"artist": {"name": artist, "id": artist.lower()}}]
return g
def test_prefers_album_over_same_titled_single():
# Real-world "Off the Wall" bug: MusicBrainz returns the Single release-group
# first, tied on title with the Album. We must pick the Album so a 10-track
# download isn't tagged from the 2-track single's tracklist.
groups = [
_rg("Off the Wall", "Single", "Michael Jackson"),
_rg("Off the Wall", "Album", "Michael Jackson"),
]
assert _best_release_group("Michael Jackson", "Off the Wall", groups)["primary-type"] == "Album"
def test_album_preference_is_order_independent():
groups = [
_rg("Off the Wall", "Album", "Michael Jackson"),
_rg("Off the Wall", "Single", "Michael Jackson"),
]
assert _best_release_group("Michael Jackson", "Off the Wall", groups)["primary-type"] == "Album"
def test_closer_title_still_wins_over_album_type():
# Title similarity dominates the album-type preference (for a single artist).
groups = [
_rg("Continuum", "Single", "John Mayer"),
_rg("Continuum Deluxe Reissue", "Album", "John Mayer"),
]
assert _best_release_group("John Mayer", "Continuum", groups)["title"] == "Continuum"
def test_below_threshold_returns_none():
groups = [_rg("Something Totally Different", "Album", "Michael Jackson")]
assert _best_release_group("Michael Jackson", "Off the Wall", groups) is None
def test_empty_groups_returns_none():
assert _best_release_group("Michael Jackson", "Off the Wall", []) is None
def test_prefers_original_group_over_later_reissue():
# Real "Hybrid Theory" bug: MB lists a 2000 original and a 2023 reissue as separate same-titled
# Album groups (tie on artist/title/type). The reissue's longer tracklist fails completeness
# checks, so the original (earliest) must win regardless of MB's relevance order.
groups = [
{"id": "reissue", "title": "Hybrid Theory", "primary-type": "Album",
"first-release-date": "2023-07-21",
"artist-credit": [{"artist": {"name": "Linkin Park"}}]},
{"id": "original", "title": "Hybrid Theory", "primary-type": "Album",
"first-release-date": "2000-10-24",
"artist-credit": [{"artist": {"name": "Linkin Park"}}]},
]
assert _best_release_group("Linkin Park", "Hybrid Theory", groups)["id"] == "original"
def test_prefers_credited_artist_over_same_titled_other_artist_album():
# Real-world "Fun Machine" bug: Lake Street Dive's EP is what we followed, but two
# unrelated bands also have a release group literally titled "Fun Machine" as an Album.
# All three tie on title (1.0), so the old is_album tiebreak wrongly grabbed a foreign
# Album. The followed artist must win over title-tied albums by other artists.
groups = [
_rg("Fun Machine", "EP", "Lake Street Dive"),
_rg("Fun Machine", "Album", "Bastards of Melody"),
_rg("Fun Machine", "Album", "Teledildonics 5000"),
]
best = _best_release_group("Lake Street Dive", "Fun Machine", groups)
assert best["artist-credit"][0]["artist"]["name"] == "Lake Street Dive"
# --- _with_retry: transient MusicBrainz failures (SSL EOF, 503 rate-limit) ---
def test_pick_release_prefers_original_official_over_later_reissue():
# Real "Hybrid Theory" bug: MB's arbitrary releases[0] was a 2023 13-track reissue; the 2000
# original is what sources deliver, so completeness checks must reference it.
releases = [
{"id": "reissue", "status": "Official", "date": "2023-06-01"},
{"id": "original", "status": "Official", "date": "2000-10-24"},
{"id": "deluxe", "status": "Official", "date": "2020-10-09"},
]
assert _pick_release(releases)["id"] == "original"
def test_pick_release_skips_non_official_and_undated():
releases = [
{"id": "promo", "status": "Promotion", "date": "1999-01-01"}, # earliest but not official
{"id": "undated", "status": "Official", "date": ""},
{"id": "official", "status": "Official", "date": "2001-03-03"},
]
assert _pick_release(releases)["id"] == "official"
def test_pick_release_empty_is_none():
assert _pick_release([]) is None
def test_retry_returns_value_without_backoff_when_first_call_succeeds():
sleeps: list[float] = []
calls = {"n": 0}
def fn():
calls["n"] += 1
return "ok"
assert _with_retry(fn, sleep=sleeps.append) == "ok"
assert calls["n"] == 1
assert sleeps == [] # no backoff when nothing failed
def test_retry_recovers_from_transient_network_error():
# The real symptom: <urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING]> arrives
# as musicbrainzngs.NetworkError. A couple of retries clears it.
sleeps: list[float] = []
calls = {"n": 0}
def fn():
calls["n"] += 1
if calls["n"] < 3:
raise musicbrainzngs.NetworkError("SSL: UNEXPECTED_EOF_WHILE_READING")
return "recovered"
assert _with_retry(fn, sleep=sleeps.append) == "recovered"
assert calls["n"] == 3
assert sleeps == [1.0, 2.0] # exponential backoff before each retry
def test_retry_recovers_from_transient_response_error():
# MusicBrainz's own 503 rate-limit surfaces as ResponseError.
calls = {"n": 0}
def fn():
calls["n"] += 1
if calls["n"] < 2:
raise musicbrainzngs.ResponseError("503 Service Unavailable")
return "ok"
assert _with_retry(fn, sleep=lambda _: None) == "ok"
assert calls["n"] == 2
def test_retry_reraises_after_exhausting_attempts():
sleeps: list[float] = []
def fn():
raise musicbrainzngs.NetworkError("still down")
with pytest.raises(musicbrainzngs.NetworkError):
_with_retry(fn, attempts=4, sleep=sleeps.append)
assert sleeps == [1.0, 2.0, 4.0] # 3 backoffs across 4 attempts
def test_retry_does_not_swallow_non_transient_errors():
calls = {"n": 0}
def fn():
calls["n"] += 1
raise ValueError("bug, not a network blip")
with pytest.raises(ValueError):
_with_retry(fn, sleep=lambda _: None)
assert calls["n"] == 1 # non-transient errors propagate immediately, no retry
worker/tests/test_qobuz_gate.py
+179
View File
@@ -0,0 +1,179 @@
from datetime import date
from lyra_worker.qobuz_gate import (
QobuzPacing,
today_cap,
is_allowed,
next_gap_seconds,
gate_open,
record_download,
)
# ---- QobuzPacing.from_config -------------------------------------------------
def test_from_config_defaults():
p = QobuzPacing.from_config({})
assert p.enabled is True
assert p.active_start == 8 and p.active_end == 23
assert p.daily_cap == 40
assert p.warmup_start_date == ""
def test_from_config_reads_values():
p = QobuzPacing.from_config({
"qobuz.pacing.enabled": "false",
"qobuz.pacing.activeStartHour": "9",
"qobuz.pacing.activeEndHour": "22",
"qobuz.pacing.dailyCap": "25",
"qobuz.pacing.warmupStartDate": "2026-07-15",
})
assert p.enabled is False
assert p.active_start == 9 and p.active_end == 22
assert p.daily_cap == 25
assert p.warmup_start_date == "2026-07-15"
# ---- today_cap (warm-up ramp) ------------------------------------------------
def test_today_cap_no_warmup_uses_steady():
p = QobuzPacing(daily_cap=40, warmup_start_date="")
assert today_cap(p, date(2026, 7, 20)) == 40
def test_today_cap_ramp_boundaries():
p = QobuzPacing(daily_cap=40, warmup_start_date="2026-07-15")
assert today_cap(p, date(2026, 7, 15)) == 5 # day 0
assert today_cap(p, date(2026, 7, 17)) == 5 # day 2
assert today_cap(p, date(2026, 7, 18)) == 10 # day 3
assert today_cap(p, date(2026, 7, 21)) == 10 # day 6
assert today_cap(p, date(2026, 7, 22)) == 20 # day 7
assert today_cap(p, date(2026, 7, 28)) == 20 # day 13
assert today_cap(p, date(2026, 7, 29)) == 40 # day 14
assert today_cap(p, date(2026, 8, 30)) == 40 # long after
def test_today_cap_steady_below_ramp_step_is_the_ceiling():
# A small steady cap is never exceeded by a ramp step.
p = QobuzPacing(daily_cap=3, warmup_start_date="2026-07-15")
assert today_cap(p, date(2026, 7, 29)) == 3
# ---- is_allowed (the gate decision) -----------------------------------------
_P = QobuzPacing(enabled=True, active_start=8, active_end=23, daily_cap=40)
def test_disabled_always_allows():
p = QobuzPacing(enabled=False, active_start=8, active_end=23)
assert is_allowed(p, now_hour=3, count_today=999, cap=0, now_epoch=0, next_allowed_epoch=1e18) is True
def test_blocked_outside_active_hours():
assert is_allowed(_P, now_hour=7, count_today=0, cap=40, now_epoch=100, next_allowed_epoch=0) is False
assert is_allowed(_P, now_hour=23, count_today=0, cap=40, now_epoch=100, next_allowed_epoch=0) is False
def test_blocked_at_or_over_cap():
assert is_allowed(_P, now_hour=10, count_today=40, cap=40, now_epoch=100, next_allowed_epoch=0) is False
assert is_allowed(_P, now_hour=10, count_today=41, cap=40, now_epoch=100, next_allowed_epoch=0) is False
def test_blocked_before_next_allowed():
assert is_allowed(_P, now_hour=10, count_today=0, cap=40, now_epoch=100, next_allowed_epoch=200) is False
def test_allowed_when_all_conditions_met():
assert is_allowed(_P, now_hour=10, count_today=5, cap=40, now_epoch=300, next_allowed_epoch=200) is True
# ---- next_gap_seconds (spread + jitter) -------------------------------------
def test_next_gap_spreads_cap_across_active_window():
# 15h window (8..23) / cap 40 = 1350s base; ±30%; midpoint rand -> base.
p = QobuzPacing(active_start=8, active_end=23, daily_cap=40)
assert abs(next_gap_seconds(p, cap=40, rand01=0.5) - 1350.0) < 1.0
assert abs(next_gap_seconds(p, cap=40, rand01=0.0) - 1350.0 * 0.7) < 1.0 # low end
assert abs(next_gap_seconds(p, cap=40, rand01=1.0) - 1350.0 * 1.3) < 1.0 # high end
def test_next_gap_has_a_floor():
# A huge cap would give a tiny gap; a 60s floor protects the request rate.
p = QobuzPacing(active_start=8, active_end=23, daily_cap=100000)
assert next_gap_seconds(p, cap=100000, rand01=0.0) == 60.0
# ---- DB helpers: gate_open / record_download --------------------------------
def _set(conn, key, value):
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
(key, value),
)
conn.commit()
def _get(conn, key):
with conn.cursor() as cur:
cur.execute('SELECT value FROM "Config" WHERE key=%s', (key,))
row = cur.fetchone()
return row[0] if row else None
def test_record_download_increments_and_sets_next_allowed(conn):
_set(conn, "qobuz.pacing.dailyCap", "40")
record_download(conn, rand01=0.5)
assert _get(conn, "qobuz.pacing.countToday") == "1"
assert _get(conn, "qobuz.pacing.countDay") is not None
assert _get(conn, "qobuz.pacing.nextAllowedAt") is not None
record_download(conn, rand01=0.5)
assert _get(conn, "qobuz.pacing.countToday") == "2"
def test_record_download_resets_counter_on_new_day(conn):
# A stale count from a previous day must reset to 1, not accumulate.
_set(conn, "qobuz.pacing.dailyCap", "40")
_set(conn, "qobuz.pacing.countDay", "2000-01-01")
_set(conn, "qobuz.pacing.countToday", "39")
record_download(conn, rand01=0.5)
assert _get(conn, "qobuz.pacing.countToday") == "1"
def test_gate_open_with_always_open_window(conn):
# An all-day window (0..24) removes the time-of-day dependency: no prior downloads and no
# next-allowed set → the gate is open regardless of when the test runs.
_set(conn, "qobuz.pacing.activeStartHour", "0")
_set(conn, "qobuz.pacing.activeEndHour", "24")
assert gate_open(conn) is True
def test_gate_closed_by_empty_active_window(conn):
# start == end is an empty window (h >= 5 AND h < 5 is never true) → always closed.
_set(conn, "qobuz.pacing.activeStartHour", "5")
_set(conn, "qobuz.pacing.activeEndHour", "5")
assert gate_open(conn) is False
def test_gate_closed_when_cap_reached_today(conn):
_set(conn, "qobuz.pacing.activeStartHour", "0")
_set(conn, "qobuz.pacing.activeEndHour", "24")
_set(conn, "qobuz.pacing.dailyCap", "3")
# Simulate 3 downloads already today.
with conn.cursor() as cur:
cur.execute("SELECT current_date::text")
today = cur.fetchone()[0]
_set(conn, "qobuz.pacing.countDay", today)
_set(conn, "qobuz.pacing.countToday", "3")
assert gate_open(conn) is False
def test_gate_ignores_stale_count_from_a_previous_day(conn):
# A high count from yesterday must not keep the gate closed today.
_set(conn, "qobuz.pacing.activeStartHour", "0")
_set(conn, "qobuz.pacing.activeEndHour", "24")
_set(conn, "qobuz.pacing.dailyCap", "3")
_set(conn, "qobuz.pacing.countDay", "2000-01-01")
_set(conn, "qobuz.pacing.countToday", "99")
assert gate_open(conn) is True
worker/tests/test_qobuz_gate_pipeline.py
+85
View File
@@ -0,0 +1,85 @@
"""The Qobuz budget gate, wired into the pipeline: when the gate is closed the job falls through
to another source; when open it uses Qobuz and records the download; a Qobuz-only setup is never
gated (nothing to fall through to)."""
from lyra_worker.adapters.fakes import FakeQobuz, FakeSoulseek
from lyra_worker.claim import claim_next
from lyra_worker.pipeline import run_pipeline
from tests.conftest import insert_request
def _cfg(conn, key, value):
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
(key, value),
)
conn.commit()
def _open_gate(conn):
_cfg(conn, "qobuz.pacing.activeStartHour", "0")
_cfg(conn, "qobuz.pacing.activeEndHour", "24")
def _closed_gate(conn):
_cfg(conn, "qobuz.pacing.activeStartHour", "5") # empty window (5..5) → always closed
_cfg(conn, "qobuz.pacing.activeEndHour", "5")
def _state(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT state FROM "Job" WHERE id = %s', (job_id,))
return cur.fetchone()[0]
def _chosen(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT source FROM "Candidate" WHERE "jobId" = %s AND chosen = true', (job_id,))
row = cur.fetchone()
return row[0] if row else None
def _cfg_get(conn, key):
with conn.cursor() as cur:
cur.execute('SELECT value FROM "Config" WHERE key = %s', (key,))
row = cur.fetchone()
return row[0] if row else None
def test_closed_gate_drops_qobuz_and_falls_through(conn):
_closed_gate(conn)
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [FakeQobuz(), FakeSoulseek()], dest_root="/tmp/lib-gate")
assert _state(conn, job_id) == "imported"
assert _chosen(conn, job_id) == "soulseek" # Qobuz dropped by the closed gate
def test_open_gate_uses_qobuz_and_records_the_download(conn):
_open_gate(conn)
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [FakeQobuz(), FakeSoulseek()], dest_root="/tmp/lib-gate")
assert _state(conn, job_id) == "imported"
assert _chosen(conn, job_id) == "qobuz"
assert _cfg_get(conn, "qobuz.pacing.countToday") == "1" # recorded against today's budget
assert _cfg_get(conn, "qobuz.pacing.nextAllowedAt") is not None
def test_qobuz_only_setup_is_never_gated(conn):
_closed_gate(conn)
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [FakeQobuz()], dest_root="/tmp/lib-gate")
assert _state(conn, job_id) == "imported"
assert _chosen(conn, job_id) == "qobuz" # not dropped — nothing to fall through to
def test_non_qobuz_download_does_not_touch_the_counter(conn):
_closed_gate(conn) # forces Soulseek
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [FakeQobuz(), FakeSoulseek()], dest_root="/tmp/lib-gate")
assert _chosen(conn, job_id) == "soulseek"
assert _cfg_get(conn, "qobuz.pacing.countToday") is None # only Qobuz downloads count
worker/tests/test_requeue.py
+69
View File
@@ -0,0 +1,69 @@
from lyra_worker.claim import claim_next, requeue_or_fail
from tests.conftest import insert_request
def _set_attempts(conn, job_id, attempts, state="matching", stage="match"):
with conn.cursor() as cur:
cur.execute(
'UPDATE "Job" SET attempts = %s, state = %s, "currentStage" = %s, "claimedAt" = now() '
'WHERE id = %s',
(attempts, state, stage, job_id),
)
conn.commit()
def _job(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT state, "currentStage", "claimedAt", error FROM "Job" WHERE id = %s', (job_id,))
return cur.fetchone()
def _request_status(conn, job_id):
with conn.cursor() as cur:
cur.execute(
'SELECT status FROM "Request" WHERE id = (SELECT "requestId" FROM "Job" WHERE id = %s)',
(job_id,),
)
return cur.fetchone()[0]
def test_below_cap_requeues_to_requested(conn):
# A transient failure (e.g. MB SSL EOF) mid-'matching' must not strand the job — it returns
# to 'requested' so claim_next picks it up again instead of orphaning it.
jid = insert_request(conn)
_set_attempts(conn, jid, 2)
requeue_or_fail(conn, jid, "SSL: UNEXPECTED_EOF_WHILE_READING")
st, stage, claimed_at, err = _job(conn, jid)
assert st == "requested"
assert stage == "intake"
assert claimed_at is None
assert err is None
assert _request_status(conn, jid) == "pending"
assert claim_next(conn) == jid # immediately re-claimable
def test_at_cap_marks_needs_attention_with_error(conn):
jid = insert_request(conn)
_set_attempts(conn, jid, 5) # hit the cap
requeue_or_fail(conn, jid, "SSL: UNEXPECTED_EOF_WHILE_READING")
st, _stage, claimed_at, err = _job(conn, jid)
assert st == "needs_attention"
assert err == "SSL: UNEXPECTED_EOF_WHILE_READING"
assert claimed_at is None
assert _request_status(conn, jid) == "needs_attention"
assert claim_next(conn) is None # not re-claimed
def test_requeues_until_cap_then_fails(conn):
jid = insert_request(conn)
for attempts in (1, 2, 3, 4):
_set_attempts(conn, jid, attempts)
requeue_or_fail(conn, jid, "boom")
assert _job(conn, jid)[0] == "requested"
_set_attempts(conn, jid, 5)
requeue_or_fail(conn, jid, "boom")
assert _job(conn, jid)[0] == "needs_attention"
def test_missing_job_is_noop(conn):
requeue_or_fail(conn, "does-not-exist", "boom") # must not raise
worker/tests/test_slskd_cancel.py
+218
View File
@@ -0,0 +1,218 @@
"""slskd download behaviour: abandoned transfers are cancelled (no pile-up), a stalled peer is
dropped fast, a slow-but-progressing peer is kept, and search candidates are ranked fast/free
peers first."""
import json
import pytest
import lyra_worker.adapters._slskd as slskd_mod
from lyra_worker.adapters._slskd import (
_ETA_CAP_SECONDS,
_MAX_XFER_POLLS,
_MIN_XFER_POLLS,
_POLL_SECONDS,
SlskdClient,
_eta_seconds,
_max_polls,
_parse_search_responses,
)
class _Resp:
def __init__(self, payload=None):
self._p = payload if payload is not None else {}
def raise_for_status(self):
pass
def json(self):
return self._p
class _SeqRequests:
"""post OK; get() returns successive poll payloads (last one repeats); records DELETEs."""
def __init__(self, polls):
self._polls = list(polls)
self._i = 0
self.deleted = []
def post(self, url, **kw):
return _Resp({})
def get(self, url, **kw):
p = self._polls[min(self._i, len(self._polls) - 1)]
self._i += 1
return _Resp(p)
def delete(self, url, **kw):
self.deleted.append(url)
return _Resp({})
def _poll(state, bytes_transferred, fid="t1"):
return {"directories": [{"files": [
{"filename": "f1.flac", "size": 100, "state": state,
"bytesTransferred": bytes_transferred, "id": fid},
]}]}
def _client(tmp_path):
return SlskdClient({"slskd.url": "http://x", "slskd.api_key": "k",
"slskd.downloadsDir": str(tmp_path)})
_REF = json.dumps({"username": "peerA", "files": [{"filename": "f1.flac", "size": 100}]})
def test_stalled_peer_is_abandoned_and_cancelled(tmp_path, monkeypatch):
# No byte progress across polls → stall → TimeoutError → the transfer is cancelled.
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests([_poll("InProgress", 0)]))
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
monkeypatch.setattr(slskd_mod, "_STALL_POLLS", 3)
fake = slskd_mod.requests
with pytest.raises(TimeoutError):
_client(tmp_path).download(_REF, str(tmp_path / "dest"), lambda *_a: None)
assert any(u.endswith("/api/v0/transfers/downloads/peerA/t1") for u in fake.deleted)
def test_error_state_is_cancelled(tmp_path, monkeypatch):
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests([_poll("Completed, Errored", 0)]))
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
fake = slskd_mod.requests
with pytest.raises(RuntimeError):
_client(tmp_path).download(_REF, str(tmp_path / "dest"), lambda *_a: None)
assert any(u.endswith("/peerA/t1") for u in fake.deleted)
def test_slow_but_progressing_peer_is_not_abandoned(tmp_path, monkeypatch):
# Bytes climb for more polls than _STALL_POLLS, then complete → success, not a premature stall.
(tmp_path / "01.flac").write_bytes(b"audio") # so _retrieve finds the finished file
ref = json.dumps({"username": "peerA", "files": [{"filename": r"d\Album\01.flac", "size": 5}]})
polls = [
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 5,
"state": "InProgress", "bytesTransferred": b, "id": "t1"}]}]}
for b in (10, 20, 30, 40, 50)
] + [
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 5,
"state": "Completed, Succeeded", "bytesTransferred": 5, "id": "t1"}]}]}
]
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests(polls))
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
monkeypatch.setattr(slskd_mod, "_STALL_POLLS", 2) # would fire early if progress were ignored
out = _client(tmp_path).download(ref, str(tmp_path / "dest"), lambda *_a: None)
assert out["track_count"] == 1
assert slskd_mod.requests.deleted == [] # completed cleanly → nothing cancelled
def test_parse_ranks_free_and_fast_peers_first():
responses = [
{"username": "slow", "hasFreeUploadSlot": True, "uploadSpeed": 100, "queueLength": 0,
"files": [{"filename": r"a\Artist - Album\01.flac", "size": 5}]},
{"username": "fast", "hasFreeUploadSlot": True, "uploadSpeed": 9000, "queueLength": 0,
"files": [{"filename": r"b\Artist - Album\01.flac", "size": 5}]},
{"username": "queued", "hasFreeUploadSlot": False, "uploadSpeed": 99999, "queueLength": 40,
"files": [{"filename": r"c\Artist - Album\01.flac", "size": 5}]},
]
peers = [json.loads(c["source_ref"])["username"] for c in _parse_search_responses(responses)]
assert peers == ["fast", "slow", "queued"] # free+fast first; no-slot peer last despite speed
def test_parse_uses_requested_artist_when_present_in_path():
# Real "Rated R" MP3-over-FLAC bug: the artist is in a PARENT folder ("Rihanna\(2009) Rated
# R\…"), so the "Artist - Album" folder parse yields no artist and confidence tanks. When the
# requested artist is in the path, trust it so the candidate scores as Rihanna.
responses = [
{"username": "p", "hasFreeUploadSlot": True, "uploadSpeed": 1000, "queueLength": 0,
"files": [{"filename": r"@@x\Music\Rihanna\(2009) Rated R\01. Mad House.flac", "size": 5}]},
]
c = _parse_search_responses(responses, artist="Rihanna")[0]
assert c["artist"] == "Rihanna"
def test_parse_without_artist_falls_back_to_folder_parse():
responses = [
{"username": "p", "hasFreeUploadSlot": True, "uploadSpeed": 1000, "queueLength": 0,
"files": [{"filename": r"x\Some Band - Some Album\01.flac", "size": 5}]},
]
c = _parse_search_responses(responses)[0] # no artist arg → folder parse
assert c["artist"] == "Some Band"
def test_parse_prefers_shorter_transfer_over_raw_speed():
# The Hybrid Theory case: a peer advertising higher speed but serving a ~2x-larger rip (24-bit
# / bloated FLAC) should lose to a peer with lower speed but a smaller standard rip that
# finishes sooner — both are the same quality class to Lyra, so faster-to-finish wins.
responses = [
{"username": "bloated", "hasFreeUploadSlot": True, "uploadSpeed": 20000, "queueLength": 0,
"files": [{"filename": rf"x\LP - Album\{i:02}.flac", "size": 40_000_000} for i in range(12)]},
{"username": "lean", "hasFreeUploadSlot": True, "uploadSpeed": 15000, "queueLength": 0,
"files": [{"filename": rf"y\LP - Album\{i:02}.flac", "size": 24_000_000} for i in range(12)]},
]
# bloated est = 480MB/20000 = 24000s; lean est = 288MB/15000 = 19200s → lean finishes first
peers = [json.loads(c["source_ref"])["username"] for c in _parse_search_responses(responses)]
assert peers[0] == "lean"
def test_eta_seconds_from_measured_rate():
assert _eta_seconds(1000, 250, 250.0) == 3 # 750 bytes left / 250 B/s
assert _eta_seconds(1000, 0, 100.0) == 10
def test_eta_seconds_unknown_cases():
assert _eta_seconds(1000, 250, 0.0) is None # no speed sample yet
assert _eta_seconds(0, 0, 100.0) is None # unknown total size
assert _eta_seconds(1000, 1000, 100.0) is None # already complete
def test_max_polls_scales_backstop_to_album_size():
# A 521MB album (Drake "Scorpion", 25 FLACs) from a slow serial peer must get far more than
# the old flat 1200-poll/60min budget, or it gets abandoned mid-download and restarts from zero.
scorpion_bytes = 521_000_000
polls = _max_polls(scorpion_bytes)
assert polls > 1200 # more headroom than the old flat cap
assert polls * _POLL_SECONDS >= 90 * 60 # at least ~90 min of wall-clock budget
# A tiny single stays on the floor, not zero.
assert _max_polls(3_000_000) == _MIN_XFER_POLLS
assert _max_polls(0) == _MIN_XFER_POLLS
# An enormous set is clamped to the ceiling (bounds worst-case slot hold).
assert _max_polls(50_000_000_000) == _MAX_XFER_POLLS
# Monotonic: bigger album never gets a smaller budget.
assert _max_polls(200_000_000) <= _max_polls(400_000_000)
def test_eta_seconds_capped_when_peer_nearly_stalls():
# A near-stalled peer (tiny speed) would yield an ETA far past int4, overflowing the
# downloadEtaSeconds column write. The estimate must be clamped to a safe ceiling.
eta = _eta_seconds(30_000_000, 0, 0.001) # 30MB / 0.001 B/s ~= 3e10s uncapped
assert eta == _ETA_CAP_SECONDS
assert eta < 2_147_483_647 # fits in int4
def test_download_reports_shrinking_eta(tmp_path, monkeypatch):
# A 900-byte file arriving 300 bytes per 3s poll → measured speed 100 B/s → ETA counts down
# (6s, 3s) then None on completion. on_progress gets (fraction, eta_seconds).
(tmp_path / "01.flac").write_bytes(b"x" * 900) # so _retrieve finds the finished file
ref = json.dumps({"username": "peerA", "files": [{"filename": r"d\Album\01.flac", "size": 900}]})
polls = [
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 900,
"state": "InProgress", "bytesTransferred": b, "id": "t1"}]}]}
for b in (300, 600)
] + [
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 900,
"state": "Completed, Succeeded", "bytesTransferred": 900, "id": "t1"}]}]}
]
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests(polls))
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
clock = iter(float(i) * 3 for i in range(100)) # 0,3,6,… deterministic monotonic time
monkeypatch.setattr(slskd_mod.time, "monotonic", lambda: next(clock))
calls: list = []
_client(tmp_path).download(ref, str(tmp_path / "dest"), lambda *a: calls.append(a))
etas = [a[1] for a in calls if len(a) > 1]
assert etas == [6, 3, None] # counts down at 100 B/s, then None once complete
worker/tests/test_slskd_search_fallback.py
+150
View File
@@ -0,0 +1,150 @@
"""Soulseek search fallback: the Soulseek server drops searches containing blocklisted terms
(major-label artists / album titles) → 0 responses. search_album retries with album-only then
artist-only, keeping only path-matching results so it never grabs a same-titled foreign album."""
import json
import lyra_worker.adapters._slskd as slskd_mod
from lyra_worker.adapters._slskd import (
SlskdClient,
_keep_matching,
_norm,
_path_matches,
)
# --- pure helpers -----------------------------------------------------------------------------
def test_norm_strips_diacritics_and_punctuation():
assert _norm("Beyoncé") == "beyonce"
assert _norm("I Am… Sasha Fierce") == "i am sasha fierce"
assert _norm("AC/DC") == "ac dc"
def test_path_matches_is_accent_and_punctuation_tolerant():
p = r"@@bckfj\All\Beyonce\I Am... Sasha Fierce CD1\03 Halo.flac"
assert _path_matches("Beyoncé", p) # accent folded
assert _path_matches("I Am… Sasha Fierce", p) # ellipsis / punctuation ignored
assert not _path_matches("Rihanna", p)
def test_path_matches_short_needle_requires_whole_string():
assert _path_matches("U2", r"x\U2 - Achtung Baby\01.flac")
assert not _path_matches("U2", r"x\Blur - 13\01.flac")
def test_keep_matching_drops_foreign_and_empty_responses():
responses = [
{"username": "a", "files": [{"filename": r"x\Rihanna - Unapologetic\01.flac"}]},
{"username": "b", "files": [{"filename": r"x\Someone Else - Unapologetic\01.flac"}]},
]
kept = _keep_matching(responses, "Rihanna")
assert [r["username"] for r in kept] == ["a"]
# --- fallback orchestration (HTTP faked) ------------------------------------------------------
class _Resp:
def __init__(self, payload):
self._p = payload
def raise_for_status(self):
pass
def json(self):
return self._p
class _FakeSearch:
"""Routes slskd search HTTP by URL: POST returns an id, GET on the search returns a completed
state, GET on /responses returns the canned responses for that query. Records queries sent."""
def __init__(self, by_query):
self.by_query = by_query # {searchText: [responses]}
self.queries = [] # searchText values POSTed, in order
self._id_query = {}
self._n = 0
def post(self, url, **kw):
text = kw["json"]["searchText"]
self.queries.append(text)
self._n += 1
sid = f"s{self._n}"
self._id_query[sid] = text
return _Resp({"id": sid})
def get(self, url, **kw):
if url.endswith("/responses"):
sid = url.rsplit("/", 2)[-2]
return _Resp(self.by_query.get(self._id_query[sid], []))
return _Resp({"state": "Completed, TimedOut"})
def _client():
return SlskdClient({"slskd.url": "http://x", "slskd.api_key": "k", "slskd.downloadsDir": "/x"})
def _install(monkeypatch, fake):
monkeypatch.setattr(slskd_mod, "requests", fake)
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
def test_primary_hit_does_not_trigger_fallback(monkeypatch):
fake = _FakeSearch({
"Rihanna Unapologetic": [
{"username": "a", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
"files": [{"filename": r"x\Rihanna - Unapologetic\01.flac", "size": 5}]},
],
})
_install(monkeypatch, fake)
out = _client().search_album("Rihanna", "Unapologetic")
assert len(out) == 1
assert fake.queries == ["Rihanna Unapologetic"] # no fallback searches
def test_album_only_fallback_recovers_blocklisted_artist(monkeypatch):
# Primary (artist blocked) → 0; album-only returns the artist's folder → recovered.
fake = _FakeSearch({
"Beyoncé I Am… Sasha Fierce": [], # blocked
"i am sasha fierce": [
{"username": "a", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
"files": [{"filename": r"x\Beyonce - I Am... Sasha Fierce\01.flac", "size": 5}]},
{"username": "b", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
"files": [{"filename": r"x\Other Artist - I Am Sasha Fierce\01.flac", "size": 5}]},
],
})
_install(monkeypatch, fake)
out = _client().search_album("Beyoncé", "I Am… Sasha Fierce")
assert fake.queries == ["Beyoncé I Am… Sasha Fierce", "i am sasha fierce"]
peers = [json.loads(c["source_ref"])["username"] for c in out]
assert peers == ["a"] # foreign same-titled album dropped
def test_artist_only_fallback_when_album_title_blocked(monkeypatch):
# Album-only also blocked/empty → fall through to artist-only, filtered by album in path.
fake = _FakeSearch({
"Foo Lemonade": [],
"lemonade": [], # album title itself blocked
"foo": [
{"username": "a", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
"files": [{"filename": r"x\Foo - Lemonade\01.flac", "size": 5}]},
],
})
_install(monkeypatch, fake)
out = _client().search_album("Foo", "Lemonade")
assert fake.queries == ["Foo Lemonade", "lemonade", "foo"]
assert len(out) == 1
def test_fallback_yields_nothing_when_title_too_generic(monkeypatch):
# 'Adele 21': artist blocked, album title '21' returns lots of non-Adele content → filtered
# to nothing rather than grabbing a wrong album.
fake = _FakeSearch({
"Adele 21": [],
"21": [
{"username": "z", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
"files": [{"filename": r"x\21 Savage - Savage Mode\01.flac", "size": 5}]},
],
"adele": [], # artist blocked
})
_install(monkeypatch, fake)
out = _client().search_album("Adele", "21")
assert out == []
worker/tests/test_smaller_edition_completeness.py
+101
View File
@@ -0,0 +1,101 @@
"""MusicBrainz picks an arbitrary releases[0] that is often a deluxe/expanded edition with more
tracks (and a longer runtime) than the standard album a source legitimately delivers. Completeness
and the duration guard must be judged against what the CHOSEN SOURCE offered, not MB's canonical
edition — otherwise a complete standard-edition download is falsely rejected as 'incomplete'
(The Script "No Sound Without Silence": 11 delivered vs MB 12) or 'too short' (Skillet "Unleashed":
12 delivered ~40 min vs MB's 20-track deluxe ~73 min). A genuinely short download must still fail.
"""
import os
from dataclasses import replace
from lyra_worker.adapters.fakes import FakeQobuz
from lyra_worker.claim import claim_next
from lyra_worker.pipeline import run_pipeline
from lyra_worker.types import DownloadResult, MBTarget
from tests.conftest import insert_request
class _StandardEditionQobuz(FakeQobuz):
"""Qobuz-like source offering a fixed-size *standard* edition (source_tracks), independent of
MB's possibly-larger deluxe count; stages that many real (non-empty) files and reports them."""
def __init__(self, source_tracks):
super().__init__()
self._n = source_tracks
def search(self, target):
return super().search(replace(target, track_count=self._n))
def download(self, candidate, dest, on_progress):
os.makedirs(dest, exist_ok=True)
for i in range(candidate.track_count):
with open(os.path.join(dest, f"{i + 1:02d} Track.flac"), "wb") as fh:
fh.write(b"audio")
on_progress(1.0)
return DownloadResult(ok=True, path=dest, track_count=candidate.track_count)
class _PartialQobuz(FakeQobuz):
"""Promises `promised` tracks but one fails mid-download: stages and reports `promised - 1`
(the real John Mayer "Roll it on Home" IncompleteRead case)."""
def __init__(self, promised):
super().__init__()
self._n = promised
def search(self, target):
return super().search(replace(target, track_count=self._n))
def download(self, candidate, dest, on_progress):
os.makedirs(dest, exist_ok=True)
delivered = candidate.track_count - 1
for i in range(delivered):
with open(os.path.join(dest, f"{i + 1:02d} Track.flac"), "wb") as fh:
fh.write(b"audio")
return DownloadResult(ok=True, path=dest, track_count=delivered)
class _DeluxeResolver:
"""Resolver whose canonical release is a larger (deluxe) edition than the source delivers."""
def __init__(self, mb_tracks, mb_total_s):
self._t, self._s = mb_tracks, mb_total_s
def resolve(self, artist, album):
return MBTarget(artist=artist, album=album, track_count=self._t, total_duration_s=self._s)
def _state_error(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT state, error FROM "Job" WHERE id = %s', (job_id,))
return cur.fetchone()
def test_standard_edition_not_rejected_as_incomplete(conn):
# The Script: source delivers the 11-track standard edition; MB's canonical release is 12.
job_id = insert_request(conn, artist="The Script", album="No Sound Without Silence")
claim_next(conn)
run_pipeline(conn, job_id, [_StandardEditionQobuz(source_tracks=11)],
resolver=_DeluxeResolver(mb_tracks=12, mb_total_s=2943),
measure_duration=lambda _s: 2700.0, dest_root="/tmp/lib-se")
assert _state_error(conn, job_id) == ("imported", None)
def test_smaller_edition_not_rejected_as_too_short(conn):
# Skillet: source delivers the 12-track standard (~40 min); MB's release is a 20-track deluxe
# (~73 min). The unscaled duration guard falsely rejects; scaling to the delivered edition
# (12/20) must let it pass.
job_id = insert_request(conn, artist="Skillet", album="Unleashed")
claim_next(conn)
run_pipeline(conn, job_id, [_StandardEditionQobuz(source_tracks=12)],
resolver=_DeluxeResolver(mb_tracks=20, mb_total_s=4380),
measure_duration=lambda _s: 2400.0, dest_root="/tmp/lib-se")
assert _state_error(conn, job_id) == ("imported", None)
def test_genuinely_incomplete_still_rejected(conn):
# John Mayer: the source promised 12 tracks but one failed mid-download → only 11 land.
# This is a real partial and must still be rejected (the fix must not swallow it).
job_id = insert_request(conn, artist="John Mayer", album="The Search for Everything")
claim_next(conn)
run_pipeline(conn, job_id, [_PartialQobuz(promised=12)],
resolver=_DeluxeResolver(mb_tracks=12, mb_total_s=2580),
measure_duration=lambda _s: 2400.0, dest_root="/tmp/lib-se")
assert _state_error(conn, job_id) == ("needs_attention", "incomplete download")
worker/tests/test_upgrade_guard.py
+133
View File
@@ -0,0 +1,133 @@
"""Upgrade guard: when an album is already in the library and no currently-available source can
beat it (e.g. Qobuz is gated by pacing and only a same-quality Soulseek copy is left), the pipeline
must NOT waste a download — keep-if-better would discard it anyway. It should skip and let the
release be re-attempted later (when Qobuz frees up). A genuinely better source still upgrades."""
import os
from lyra_worker.claim import claim_next
from lyra_worker.pipeline import run_pipeline
from lyra_worker.types import Candidate, DownloadResult, Quality
_CD = Quality(fmt="FLAC", lossless=True, bit_depth=16, sample_rate=44100) # class 2
_HIRES = Quality(fmt="FLAC", lossless=True, bit_depth=24, sample_rate=96000) # class 3
def _cfg(conn, key, value):
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
(key, value),
)
conn.commit()
class _Counting:
def __init__(self, name, tier, quality):
self.name, self.tier, self._q = name, tier, quality
self.calls = 0
self.searches = 0
def health(self):
return True
def search(self, target):
self.searches += 1
return [Candidate(source=self.name, source_ref=self.name, matched_artist=target.artist,
matched_album=target.album, quality=self._q, track_count=3, source_tier=self.tier)]
def download(self, candidate, dest, on_progress):
self.calls += 1
os.makedirs(dest, exist_ok=True)
for i in range(3):
with open(os.path.join(dest, f"0{i}.flac"), "wb") as fh:
fh.write(b"a")
return DownloadResult(ok=True, path=dest, track_count=3)
def _seed_library(conn, artist, album, quality_class, path):
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "LibraryItem" (id, artist, album, path, source, format, "qualityClass", "importedAt") '
"VALUES (gen_random_uuid()::text, %s, %s, %s, 'soulseek', 'FLAC', %s, now())",
(artist, album, path, quality_class),
)
conn.commit()
def _insert_upgrade_job(conn, artist, album):
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "MonitoredRelease" (id, "artistMbid", "artistName", "rgMbid", album, '
'monitored, state, "secondaryTypes", "createdAt") '
"VALUES (gen_random_uuid()::text, 'm', %s, 'rg', %s, true, 'grabbed', '{}', now()) RETURNING id",
(artist, album),
)
rel_id = cur.fetchone()[0]
cur.execute(
'INSERT INTO "Request" (id, artist, album, status, "monitoredReleaseId", "createdAt") '
"VALUES (gen_random_uuid()::text, %s, %s, 'pending', %s, now()) RETURNING id",
(artist, album, rel_id),
)
req = cur.fetchone()[0]
cur.execute(
'INSERT INTO "Job" (id, "requestId", state, "currentStage", attempts, "createdAt", "updatedAt") '
"VALUES (gen_random_uuid()::text, %s, 'requested', 'intake', 0, now(), now()) RETURNING id",
(req,),
)
job = cur.fetchone()[0]
conn.commit()
return job
def _lib(conn, artist, album):
with conn.cursor() as cur:
cur.execute('SELECT path, "qualityClass" FROM "LibraryItem" WHERE artist=%s AND album=%s', (artist, album))
return cur.fetchone()
def _state(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT state FROM "Job" WHERE id=%s', (job_id,))
return cur.fetchone()[0]
def test_upgrade_skipped_pre_search_when_qobuz_gated(conn):
# Qobuz gated (empty active window) + existing >= CD copy: only hi-res Qobuz could improve it,
# so defer WITHOUT even searching — avoids re-searching every owned album each monitor cycle.
_cfg(conn, "qobuz.pacing.activeStartHour", "5")
_cfg(conn, "qobuz.pacing.activeEndHour", "5") # empty window → gate closed
_seed_library(conn, "A", "B", 2, "/m/existing")
job_id = _insert_upgrade_job(conn, "A", "B")
claim_next(conn)
soulseek = _Counting("soulseek", 1, _CD)
run_pipeline(conn, job_id, [soulseek], dest_root="/tmp/lib-ug", upgrade_cutoff=3)
assert soulseek.searches == 0 # no wasted search
assert soulseek.calls == 0 # no wasted download
assert _lib(conn, "A", "B") == ("/m/existing", 2)
assert _state(conn, job_id) == "imported"
def test_upgrade_skipped_post_search_when_only_same_quality_found(conn):
# Gate OPEN, so it searches, but the only source found is same-quality → skip the download.
_cfg(conn, "qobuz.pacing.activeStartHour", "0")
_cfg(conn, "qobuz.pacing.activeEndHour", "24") # gate open
_seed_library(conn, "A", "B", 2, "/m/existing")
job_id = _insert_upgrade_job(conn, "A", "B")
claim_next(conn)
soulseek = _Counting("soulseek", 1, _CD)
run_pipeline(conn, job_id, [soulseek], dest_root="/tmp/lib-ug", upgrade_cutoff=3)
assert soulseek.searches == 1 # it did search (gate open)
assert soulseek.calls == 0 # but skipped the useless download
assert _lib(conn, "A", "B") == ("/m/existing", 2)
def test_upgrade_proceeds_when_a_better_source_is_available(conn):
# Existing class-2 copy; a class-3 (hi-res) source IS available → download + upgrade.
_seed_library(conn, "A", "B", 2, "/m/existing")
job_id = _insert_upgrade_job(conn, "A", "B")
claim_next(conn)
qobuz = _Counting("qobuz", 0, _HIRES)
run_pipeline(conn, job_id, [qobuz], dest_root="/tmp/lib-ug2", upgrade_cutoff=3)
assert qobuz.calls == 1 # downloaded the better copy
assert _lib(conn, "A", "B")[1] == 3 # upgraded to hi-res