20 Commits

Author SHA1 Message Date
Jonathan 395944f3e1 fix(worker): scale slskd download backstop to album size (stop abandoning big albums)
The absolute download backstop was a flat _MAX_XFER_POLLS=1200 (~60min). A large
album (e.g. Drake "Scorpion", 521MB / 25 FLACs) from a slow-but-healthy serial
peer (~400KB/s, serving one file at a time) needs ~60-70min, so it was cut off
at 60min and — because an abandoned transfer is cancelled and re-enqueued from
scratch — re-downloaded from zero every attempt, never finishing and eventually
going needs_attention at the attempt cap.

Replace the flat cap with _max_polls(total_bytes): budget the backstop at a
conservative floor throughput (~64KB/s), clamped to [20min, 2h]. Scorpion now
gets the full 2h ceiling instead of 60min. The 90s stall timeout is unchanged,
so a truly dead/queued peer is still abandoned fast.

Follow-up (not in this change): resume across retries (skip files slskd already
completed) so an abandoned large download doesn't restart from zero.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 23:27:44 +02:00
Jonathan 10430037fe fix(worker): cap slskd ETA + rollback on progress-write failure (stop txn-abort job churn)
A near-stalled Soulseek peer drives the EWMA speed toward zero, so
_eta_seconds returned int(remaining/speed) values far past int4 max. Writing
that to Job.downloadEtaSeconds (an integer column) raised "integer out of
range", which aborted the SHARED pipeline connection's transaction. The
on_progress except clause logged but never rolled back, so every subsequent
query cascaded "current transaction is aborted" and the whole job failed and
requeued (9 such failures / 24h observed in prod).

- Cap _eta_seconds at ~100h (359999s), well inside int4.
- Roll back the shared conn in on_progress's except so a failed progress
  write can never poison the pipeline txn, honoring the existing docstring
  promise that "a progress write must never fail the download".
- Regression test for the near-stall cap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 20:42:13 +02:00
Jonathan 01657d4e1b fix(web): order Recently Pressed by press time + cap at 10
Recently Pressed was derived from the request list (ordered by Request.createdAt), so
an album requested long ago but imported just now sorted to the bottom — freshly
pressed albums never appeared near the top ("new albums don't get added"). Order by
press time (job.updatedAt ≈ import time) instead, and show only the 10 most recent.
The "Pressed" stat tile and the "N in library" note still reflect the true total; the
per-row date label now shows when it was pressed, not requested.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 23:17:54 +02:00
Jonathan 0ad1fd4cd5 fix(worker): trust the requested artist when it's in the file path (stop MP3-over-FLAC)
Confidence scoring read the artist from the immediate folder as "Artist - Album", so
common FLAC rip layouts — artist in a parent folder ("Rihanna\(2009) Rated R\…") or
buried ("2009 - Rihanna - Rated R") — parsed with no/garbage artist and scored below
the 0.7 gate. Real FLACs got filtered out and a clean-named MP3 (0.88) won, so Lyra
downloaded lossy when lossless was available (Rihanna "Rated R": 8 FLACs all <0.5).

_parse_search_responses now takes the requested artist and, when it appears anywhere
in the candidate's path (via _path_matches), sets it as the candidate artist so
confidence reflects reality. Especially helps blocklisted-artist fallbacks, whose
album-only results come from varied folder structures. 337 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 23:05:38 +02:00
Jonathan 7ed8ca2848 fix(worker): prefer the original release-group over same-titled reissues
The real root cause of "Hybrid Theory" re-downloading forever: MB lists the 2000
original and a 2023 reissue as SEPARATE same-titled Album release-groups. They tie on
(artist, title, is_album), so _best_release_group kept whichever MB relevance-ordered
first — the 2023 reissue (13 tracks / ~52 min). Its inflated tracklist+runtime then
failed the completeness/duration checks against the standard 12-track album sources
deliver, looping endlessly. Add an earliest-first-release-date tie-break so the
original group wins. (Complements _pick_release, which handles reissue releases
*within* a group.) 335 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 22:32:25 +02:00
Jonathan 5f59283bf0 fix(worker): resolve MB to the original edition + loosen duration gate
Root cause of albums re-downloading forever despite completing (Linkin Park "Hybrid
Theory"): the MB resolver read the tracklist from an arbitrary releases[0], often a
later reissue/deluxe. HT resolved to a 2023 13-track reissue (~52 min); a complete
2000 12-track download (~38 min) then failed _download_problem's duration gate
(2267s < 2877s×0.85) and fell through peer after peer, never importing.

- _pick_release: choose the ORIGINAL standard edition (Official, earliest date)
  instead of releases[0], so completeness/duration are judged against the edition
  sources actually deliver.
- _DURATION_MIN_RATIO 0.85 -> 0.65: a source legitimately delivering a shorter
  edition than MB's release shouldn't be rejected; only genuinely truncated/preview
  downloads (a fraction of expected runtime) should be.

Together with the est-time peer ranking, this stops the download→reject→re-download
loop. 334 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 22:27:44 +02:00
Jonathan d2a86932e0 fix(worker): rank Soulseek peers by estimated transfer time, not raw speed
Peer selection ranked (free-slot, advertised-speed, short-queue), ignoring file
size. But Soulseek FLAC rips of one album vary ~2x in size (a 24-bit/bloated rip vs
a standard CD rip) and Lyra scores them the SAME quality class (the slskd search
exposes no bit-depth, so every FLAC is class 2). So a peer advertising high speed
but serving 40MB/track files would out-rank a smaller standard rip that actually
finishes sooner — then blow past the 60-min per-peer backstop, time out, fall
through, and restart at 0% (Linkin Park "Hybrid Theory" looping for hours).

Rank by estimated transfer time instead: album bytes ÷ advertised speed, after the
free-slot check. Prefers whichever peer FINISHES first — fast peers and/or smaller
standard-edition rips — with no quality loss (all class 2). Equal-size case still
reduces to speed order, so existing ranking behaviour is preserved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 22:17:44 +02:00
Jonathan 73f1ae6e2f feat(web): order the In-queue tab next-up-first with a "Next up" marker
The queue tab showed newest-first (API returns createdAt desc) but the worker claims
the oldest queued job next (claim_next: ORDER BY createdAt ASC) — so the item that ran
next was at the BOTTOM. Sort the queue bucket ascending so it matches processing
order, and badge the first claimable row (oldest, not individually paused) as "Next up".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 20:56:42 +02:00
Jonathan c49cee9a20 feat: download ETA on the press bar from measured Soulseek throughput
Show a live "~Xm left" countdown on the download progress bar. The slskd download
loop already polls byte transfer every 3s; measure the actual throughput (EWMA-
smoothed bytes/sec), compute seconds-remaining from the album's total bytes, and
surface it. New nullable Job.downloadEtaSeconds (migration; web entrypoint runs
prisma migrate deploy), threaded through the on_progress callback (optional 2nd arg,
so other adapters/callers are unaffected — they report no ETA). API exposes it;
queue.tsx renders etaLabel() after the track count. Null when unknown (no sample yet
or a source that doesn't report bytes). Worker 330 tests, web 214 tests, both green;
verified live-rendered as "42% · 6/14 tracks · ~4m left".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 20:16:58 +02:00
Jonathan db8b43b0e8 feat(web): tabbed On-the-Press view + kebab menu for press actions
The press page interleaved active downloads, queued items, and errors in one long
list — hard to see what's actually pressing. Split the non-imported rows into three
tabs (Active / In queue / Errors) with live counts; the default tab follows the work
(Active if anything's running, else Queue, else Errors) until the user picks one.
Move the press-level actions (Pause/Resume, Clear queue, Stop now) off the toolbar
into a vertical "⋮" kebab menu on the right of the header — Clear queue disables when
the queue is empty, Stop now only shows while downloading.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 19:41:11 +02:00
Jonathan 434cd2b22c feat(worker): recover blocklisted Soulseek searches via album/artist-only fallback
The Soulseek server silently drops searches containing certain blocklisted terms
(major-label artists / album titles — Adele, Rihanna, Beyoncé, "Lemonade", …),
returning 0 responses ("TimedOut"). The blocked term poisons the whole query even
though the OTHER field usually isn't blocked and the content is on the network
(e.g. searching "Beyoncé …" → 0, but "I Am… Sasha Fierce" alone → 250 with 136
real Beyoncé folders).

When the primary "{artist} {album}" search returns nothing, retry with each field
alone (normalized to fold accents/punctuation) and keep only results whose file
path still matches the dropped field (_keep_matching), so a same-titled album by a
different artist is never grabbed. Recovers blocklisted albums with a distinctive
title; safely yields nothing when the title is too generic (e.g. Adele "21") rather
than mis-grabbing. Fallback only fires on a 0-response primary, so the happy path is
unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 19:06:07 +02:00
Jonathan d7ea29a97a fix(worker): requeue jobs that fail on an unhandled pipeline exception
An unhandled exception in run_pipeline (e.g. a MusicBrainz TLS drop surfacing as
SSL: UNEXPECTED_EOF, after the MB retries are exhausted) left the job stranded in
its mid-pipeline 'matching' state: the except handler logged "pipeline failed" and
rolled back, but never reset the job. Since claim_next only picks 'requested', the
job was orphaned — stuck showing "Searching sources…" on the press until the next
worker restart's reclaim_stuck_jobs happened to sweep it up.

Add requeue_or_fail: on an unhandled failure, requeue to 'requested' for a bounded
retry (transient network blips clear on retry), then fall to 'needs_attention' at
the attempt cap (5) so a persistently-broken job stays visible instead of looping.
Mirrors reclaim_stuck_jobs (requeue) and pipeline._fail (needs_attention), and
resets Request.status to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 18:40:41 +02:00
Jonathan e8483ee22b fix(worker): match library ownership by rgMbid + retire stale failed attempts
Two related monitor bugs surfaced by owned albums that never leave the press:

1. Ownership was matched on (artist, album) strings, but MusicBrainz canonicalizes
   the artist on import ("Kanye West"->"Ye", "Ne-Yo"->"Ne‐Yo", "The Goo Goo Dolls"
   ->"Goo Goo Dolls"). The strings no longer matched, so reconcile/enqueue/backfill
   never recognized the album as owned — it stayed "wanted" and re-pressed every
   retry interval. Match on the release-group MBID when the library row has one
   (falling back to artist+album for scanned rows without an MBID) across
   _in_library_at_cutoff, fulfill_owned_releases, and reconcile.

2. Each failed attempt left a needs_attention Request behind, so a hard-to-match
   album piled up phantom "needs attention" cards on the press even after a later
   attempt imported it (e.g. Maroon 5 "V": 3 stale cards + 1 success). Retire prior
   failed attempts when a release is fulfilled or re-attempted (_retire_failed_attempts,
   monitor-owned requests only — manual requests have no monitoredReleaseId).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 17:04:31 +02:00
Jonathan ca5bbfa9da fix(worker): retry transient MusicBrainz failures with backoff
The matching stage makes up to three musicbrainzngs calls per job, none of
which retried. A dropped TLS connection (surfacing as NetworkError,
"<urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING]>") or a 503 rate-limit
(ResponseError) failed the entire pipeline for the job, which then re-queued
and re-ran from scratch — ~36% of jobs churned this way over a 6h window.

Wrap the three MB calls in _with_retry (4 attempts, 1s/2s/4s exponential
backoff), catching only the two transient musicbrainzngs error types so real
bugs still propagate. musicbrainzngs stays lazily imported to keep module
import offline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 16:45:37 +02:00
Jonathan 675d964bbe fix(worker): smarter Soulseek peer selection + stall/progress-aware transfers
Draining the queue via Soulseek was stalling: slow peers (~100KB/s) never finished a
full FLAC album within the fixed 10-min timeout, got cancelled at ~89%, and the
fall-through restarted from scratch on the next peer — while single-song folders named
like the album were also being tried.

- Stall detection: abandon a peer with no byte progress for ~90s (queued/dead) instead
  of waiting the full timeout; a peer that IS progressing keeps its slot up to a 60-min
  backstop, so a slow-but-working transfer can finish.
- Rank search candidates by peer: free upload slot, then speed, then shortest queue.
- Drop candidates with far fewer tracks than the release (single-song folders) up front,
  so no download attempt is wasted on them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 19:54:14 +02:00
Jonathan 57f97e9838 feat(worker): defer Qobuz upgrades when it's gated instead of re-grabbing same quality
With qualityCutoff=3 (want hi-res), a Soulseek-grabbed CD copy stays below cutoff and
is re-attempted for upgrade. Without a guard, when Qobuz is gated by pacing those
attempts would (a) re-search every owned album each monitor cycle and (b) re-download
a same-quality Soulseek copy that keep-if-better discards — pure churn.

Add an upgrade guard: a >= CD-lossless copy can only be improved by hi-res Qobuz, so
when Qobuz is gated, skip the job WITHOUT searching (pre-search skip); and as a safety
net, skip the download when the best-ranked candidate can't beat the existing copy.
Force jobs always proceed. Extracted the shared keep-existing path into a helper.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 19:15:43 +02:00
Jonathan 1fc61a0f5a fix(worker): cancel abandoned slskd transfers + cap download fall-through
Two compounding causes of 'many downloads of one album from different sources' in
slskd: (1) the slskd client raised on timeout/error WITHOUT cancelling the transfer
it enqueued, so as the pipeline fell through to other peers each left a transfer
running; (2) the incomplete-download fall-through was uncapped, grinding through
every candidate of a popular album (~200 Soulseek peers), amplified now that Qobuz
pacing pushes most jobs to Soulseek.

- SlskdClient.download now cancels its enqueued transfers (DELETE) on any abandon.
- The download loop caps at _MAX_DOWNLOAD_ATTEMPTS (6) candidates.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 19:03:05 +02:00
Jonathan 67f374c470 feat: Qobuz pacing budget gate (human-like volume + cadence)
Avoid re-triggering a USER_BLOCKED account by capping Qobuz to a human-looking
volume and cadence. A per-album gate (worker/lyra_worker/qobuz_gate.py) allows
Qobuz only within active hours, under today's cap (a warm-up ramp of a steady
cap), and past a randomized spacing since the last Qobuz download; the gap spreads
the day's budget across active hours with jitter. When the gate is closed the
pipeline drops Qobuz candidates so the job falls through to another source (and
monitored albums upgrade to Qobuz later); a Qobuz-only setup is never gated.
Config keys read live each loop; counters reset at the day boundary via the DB
clock. Tunable in Settings -> Qobuz -> Pacing (new /api/qobuz/pacing route).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 18:45:56 +02:00
Jonathan d47ed12e8e feat(worker): fall through to the next source on an incomplete download
Completeness/duration verification was applied only to the first candidate that
downloaded successfully; if that copy was incomplete or too short the whole job
failed, even when a later-ranked source had a complete copy (e.g. John Mayer 'The
Search for Everything' — Qobuz reliably fails one hi-res track, so it never fell
back to the deluxe edition or Soulseek).

Extract the checks into a pure _download_problem() and run it INSIDE the download
loop: a candidate that downloads but fails verification now falls through to the
next-ranked source, and the job only fails (needs_attention) when every candidate
is incomplete — reporting the last shortfall reason.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 13:41:04 +02:00
Jonathan b0eec41823 fix(worker): judge download completeness by the source's edition, not MB's
The completeness + duration guards used MusicBrainz's target.track_count /
total_duration_s, but MB picks an arbitrary releases[0] that is often a
deluxe/expanded edition with more tracks (and runtime) than the standard album a
source legitimately delivers. So complete standard-edition downloads were falsely
rejected — The Script 'No Sound Without Silence' (11 vs MB 12) as 'incomplete
download', Skillet 'Unleashed' (12 vs MB's 20-track deluxe) as 'too short'.

Now completeness is judged against the chosen source's own track count
(winner.track_count): a truncated download (fewer files than the source promised)
still fails, and the duration expectation is scaled to the delivered edition's
size. A separate 'implausibly small vs MB' guard (source has <= half MB's tracks)
still rejects a lone-track 'full album' video. Genuine partials (e.g. John Mayer,
a Qobuz track failing mid-download) still correctly fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 13:22:01 +02:00
32 changed files with 2534 additions and 249 deletions
@@ -0,0 +1,83 @@
# Qobuz pacing ("budget gate") — design
Date: 2026-07-15
## Problem
Bulk hi-res downloading got the Qobuz account blocked (`403 USER_BLOCKED`). We need to keep
using Qobuz for hi-res but at a volume and cadence that looks like a human listener, so the
account isn't flagged again — while a large wanted-list still drains promptly via other sources.
## Approach: a per-job Qobuz eligibility gate
One gate decides, **per album**, whether Qobuz is an eligible source *right now*. If the gate is
closed, Qobuz candidates are dropped before ranking, so the worker falls through to the best
non-Qobuz source (Soulseek/YouTube) and keeps moving — no idling. Monitored albums grabbed that
way are upgraded to Qobuz hi-res later, automatically, via the existing quality-upgrade window,
once Qobuz has budget again.
**Qobuz is eligible for a job only if ALL hold:**
1. **Active hours** — current hour ∈ [activeStart, activeEnd) (a daytime window; start < end).
2. **Under today's cap** — Qobuz downloads counted today < today's cap.
3. **Spacing elapsed** — now ≥ the randomized "next allowed" time set after the last Qobuz
download.
After each successful Qobuz download the worker bumps today's counter and sets the next-allowed
time. The counter resets at the day boundary (tracked by a stored date).
**Today's cap = warm-up ramp of the steady cap**, by account age (days since `warmupStartDate`):
| Account age (days) | Cap |
|---|---|
| 02 | min(5, steady) |
| 36 | min(10, steady) |
| 713 | min(20, steady) |
| 14+ | steady |
If `warmupStartDate` is unset, no ramp (steady cap applies).
**Spacing / jitter (combined):** the gap to the next Qobuz download is randomized around
`activeWindowSeconds / todayCap` (so the day's budget spreads evenly across active hours), ±30%,
with a hard floor (≥ 60 s). This *is* the jitter — never a fixed metronome — and it prevents
front-loading the whole daily cap into the first hour.
## Configuration (Config keys; editable in Settings → Qobuz → Pacing)
Settings:
- `qobuz.pacing.enabled` (default `true`)
- `qobuz.pacing.activeStartHour` (default `8`)
- `qobuz.pacing.activeEndHour` (default `23`)
- `qobuz.pacing.dailyCap` (steady cap, default `40`)
- `qobuz.pacing.warmupStartDate` (ISO date; set to the new account's start day)
Worker-managed state (not user-edited):
- `qobuz.pacing.countDay` (date the counter is for)
- `qobuz.pacing.countToday` (int)
- `qobuz.pacing.nextAllowedAt` (epoch seconds)
Current hour / day come from the **DB clock** (`now()`), avoiding container-timezone ambiguity.
Config is read fresh each worker loop, so changes take effect with no restart.
## Worker integration
- New module `worker/lyra_worker/qobuz_gate.py`: a `QobuzPacing` config dataclass
(`from_config`), a pure `today_cap(...)`, a pure `is_allowed(...)` decision, and thin
DB helpers `gate_open(conn, now)` (reads state + decides) and `record_download(conn, pacing,
now)` (bumps counter + sets next-allowed).
- `pipeline.py`: after searching sources, if `gate_open` is false AND at least one non-Qobuz
candidate exists, drop Qobuz candidates before ranking (fall-through). If Qobuz is the *only*
source available, keep it (a Qobuz-only setup isn't gated — holding would strand the album).
- After a successful download where `winner.source == "qobuz"`, call `record_download`.
## Testing
Pure-logic unit tests for `is_allowed` (each gate condition), `today_cap` (ramp boundaries), and
the spacing computation; pipeline tests that a gated Qobuz candidate is dropped in favour of a
non-Qobuz source, that an open gate keeps Qobuz, and that a successful Qobuz download records the
counter/next-allowed.
## Out of scope
- Holding/parking an album for later Qobuz when Qobuz is the only source (fails/last-source as
today). Multi-source setups fall through, which is the common case.
- Timezone-aware active-hours windows that wrap past midnight (start < end only).
@@ -0,0 +1,2 @@
-- Estimated seconds remaining for the active download (measured throughput). Nullable.
ALTER TABLE "Job" ADD COLUMN "downloadEtaSeconds" INTEGER;
+3
View File
@@ -58,6 +58,9 @@ model Job {
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
downloadProgress Float @default(0)
// Estimated seconds until the current download finishes, from measured throughput (slskd byte
// transfer rate). Null when unknown (no speed sample yet, or a source that doesn't report bytes).
downloadEtaSeconds Int?
paused Boolean @default(false)
candidates Candidate[]
}
+3
View File
@@ -14,6 +14,7 @@ export function JobRow({
album,
kind,
label,
marker,
meta,
note,
bar,
@@ -22,6 +23,7 @@ export function JobRow({
album: string;
kind: Kind;
label: string;
marker?: string;
meta?: ReactNode;
note?: ReactNode;
bar?: ReactNode;
@@ -31,6 +33,7 @@ export function JobRow({
<div className="stripe" />
<div>
<h3 className="title">
{marker ? <span className="nextup">{marker}</span> : null}
{album} <span className="artist">· {artist}</span>
</h3>
{meta ? <div className="meta">{meta}</div> : null}
+47
View File
@@ -0,0 +1,47 @@
"use client";
import { useEffect, useRef, useState, type ReactNode } from "react";
/** A vertical "⋮" trigger that opens a small popup menu. Closes on outside click, Escape, or any
* click inside the popup (so menu items don't each need to manage it). Children are the items —
* use <button className="menu-item">…</button>. */
export function KebabMenu({ label = "Actions", children }: { label?: string; children: ReactNode }) {
const [open, setOpen] = useState(false);
const ref = useRef<HTMLDivElement>(null);
useEffect(() => {
if (!open) return;
function onDoc(e: MouseEvent) {
if (ref.current && !ref.current.contains(e.target as Node)) setOpen(false);
}
function onKey(e: KeyboardEvent) {
if (e.key === "Escape") setOpen(false);
}
document.addEventListener("mousedown", onDoc);
document.addEventListener("keydown", onKey);
return () => {
document.removeEventListener("mousedown", onDoc);
document.removeEventListener("keydown", onKey);
};
}, [open]);
return (
<div className="menu" ref={ref}>
<button
type="button"
className="menu-trigger"
aria-haspopup="menu"
aria-expanded={open}
aria-label={label}
onClick={() => setOpen((o) => !o)}
>
</button>
{open ? (
<div className="menu-pop" role="menu" onClick={() => setOpen(false)}>
{children}
</div>
) : null}
</div>
);
}
+16 -1
View File
@@ -1,5 +1,20 @@
import { describe, it, expect } from "vitest";
import { describeJob, timeAgo } from "./status";
import { describeJob, etaLabel, timeAgo } from "./status";
describe("etaLabel", () => {
it("formats seconds, minutes, and hours", () => {
expect(etaLabel(45)).toBe("~45s left");
expect(etaLabel(240)).toBe("~4m left");
expect(etaLabel(3900)).toBe("~1h 5m left");
});
it("returns null for missing or non-positive input", () => {
expect(etaLabel(null)).toBeNull();
expect(etaLabel(undefined)).toBeNull();
expect(etaLabel(0)).toBeNull();
expect(etaLabel(-5)).toBeNull();
});
});
describe("describeJob", () => {
it("maps pipeline states to literal labels + severity kind", () => {
+10
View File
@@ -27,6 +27,16 @@ export function timeAgo(value: string | Date | null | undefined, now: number): s
return `${Math.floor(h / 24)}d`;
}
/** Compact download ETA like "~4m left", "~45s left", "~1h 5m left". Null/≤0 → no label. */
export function etaLabel(seconds: number | null | undefined): string | null {
if (seconds == null || seconds <= 0) return null;
if (seconds < 60) return `~${seconds}s left`;
const m = Math.round(seconds / 60);
if (m < 60) return `~${m}m left`;
const h = Math.floor(m / 60);
return `~${h}h ${m % 60}m left`;
}
export function describeJob(state: string, stage: string): { label: string; kind: Kind } {
void stage; // stage no longer drives the label; kept for call-site compatibility
return (
@@ -0,0 +1,37 @@
import { describe, it, expect } from "vitest";
import { GET, PATCH } from "./route";
import { prisma } from "@/lib/db";
function patch(body: Record<string, string>) {
return new Request("http://localhost/api/qobuz/pacing", {
method: "PATCH",
headers: { "content-type": "application/json" },
body: JSON.stringify(body),
});
}
describe("qobuz pacing config API", () => {
it("returns defaults when nothing is stored", async () => {
const cfg = await (await GET()).json();
expect(cfg["qobuz.pacing.enabled"]).toBe("true");
expect(cfg["qobuz.pacing.activeStartHour"]).toBe("8");
expect(cfg["qobuz.pacing.activeEndHour"]).toBe("23");
expect(cfg["qobuz.pacing.dailyCap"]).toBe("40");
expect(cfg["qobuz.pacing.warmupStartDate"]).toBe("");
});
it("round-trips allow-listed settings and persists them", async () => {
await PATCH(patch({ "qobuz.pacing.dailyCap": "25", "qobuz.pacing.activeStartHour": "9" }));
const stored = await prisma.config.findUnique({ where: { key: "qobuz.pacing.dailyCap" } });
expect(stored?.value).toBe("25");
const cfg = await (await GET()).json();
expect(cfg["qobuz.pacing.dailyCap"]).toBe("25");
expect(cfg["qobuz.pacing.activeStartHour"]).toBe("9");
});
it("ignores keys not in the allow-list (e.g. worker-managed counters)", async () => {
await PATCH(patch({ "qobuz.pacing.countToday": "999", "qobuz.pacing.dailyCap": "30" }));
expect(await prisma.config.findUnique({ where: { key: "qobuz.pacing.countToday" } })).toBeNull();
expect((await prisma.config.findUnique({ where: { key: "qobuz.pacing.dailyCap" } }))?.value).toBe("30");
});
});
+39
View File
@@ -0,0 +1,39 @@
import { prisma } from "@/lib/db";
// Qobuz pacing ("budget gate") settings. The worker reads these Config keys live; state keys
// (countDay/countToday/nextAllowedAt) are worker-managed and intentionally NOT editable here.
const DEFAULTS: Record<string, string> = {
"qobuz.pacing.enabled": "true",
"qobuz.pacing.activeStartHour": "8",
"qobuz.pacing.activeEndHour": "23",
"qobuz.pacing.dailyCap": "40",
"qobuz.pacing.warmupStartDate": "",
};
export async function GET() {
const rows = await prisma.config.findMany({ where: { key: { in: Object.keys(DEFAULTS) } } });
const stored = Object.fromEntries(rows.map((r) => [r.key, r.value]));
return Response.json(
Object.fromEntries(Object.entries(DEFAULTS).map(([k, d]) => [k, stored[k] ?? d])),
);
}
export async function PATCH(request: Request) {
let body: unknown;
try {
body = await request.json();
} catch {
return Response.json({ error: "invalid JSON" }, { status: 400 });
}
const entries = Object.entries((body ?? {}) as Record<string, unknown>).filter(([k]) =>
Object.prototype.hasOwnProperty.call(DEFAULTS, k),
);
for (const [key, value] of entries) {
await prisma.config.upsert({
where: { key },
create: { key, value: String(value), secret: false },
update: { value: String(value) },
});
}
return Response.json({ updated: entries.length });
}
+1
View File
@@ -64,6 +64,7 @@ export async function GET() {
claimedAt: r.job.claimedAt,
updatedAt: r.job.updatedAt,
downloadProgress: r.job.downloadProgress,
downloadEtaSeconds: r.job.downloadEtaSeconds,
paused: r.job.paused,
candidateCount: cands.length,
chosen: chosen ? { source: chosen.source, format: chosen.format, trackCount: chosen.trackCount } : null,
+33
View File
@@ -386,6 +386,39 @@ nav.contents .sep { flex: 1; }
.tab:hover { color: var(--ink); }
.tab.active { color: var(--ink); border-bottom-color: var(--accent); }
.tab .n { color: var(--rule-2); margin-left: 6px; }
.tab.has-errors .n { color: var(--alert); }
/* "Next up" badge on the first queued row (the item the worker claims next) */
.nextup {
display: inline-block; vertical-align: middle; margin-right: 9px; transform: translateY(-2px);
font-family: var(--mono); font-size: 0.56rem; letter-spacing: 0.13em; text-transform: uppercase;
color: var(--accent-ink); background: var(--accent); padding: 2px 7px; border-radius: 4px;
}
/* ── Kebab menu (⋮ popup for press actions) ───────────── */
.dept.has-actions { align-items: center; }
.dept.has-actions .fill { transform: none; }
.menu { position: relative; display: inline-flex; }
.menu-trigger {
font-size: 1.15rem; line-height: 1; color: var(--graphite); background: transparent;
border: 1px solid var(--rule); border-radius: 6px; width: 32px; height: 30px; cursor: pointer;
display: inline-flex; align-items: center; justify-content: center;
}
.menu-trigger:hover { color: var(--ink); border-color: var(--rule-2); }
.menu-pop {
position: absolute; top: calc(100% + 6px); right: 0; z-index: 30; min-width: 190px;
background: var(--paper-2); border: 1px solid var(--rule-2); border-radius: 9px;
box-shadow: 0 10px 30px rgba(0, 0, 0, 0.22); padding: 5px;
display: flex; flex-direction: column; gap: 1px;
}
.menu-item {
font-family: var(--mono); font-size: 0.68rem; letter-spacing: 0.1em; text-transform: uppercase;
text-align: left; background: transparent; border: 0; color: var(--ink);
padding: 10px 11px; border-radius: 6px; cursor: pointer; white-space: nowrap;
}
.menu-item:hover { background: var(--rule); }
.menu-item:disabled { color: var(--rule-2); cursor: default; background: transparent; }
.menu-item.danger { color: var(--alert); }
/* ── Discography row (cover thumb + click-to-modal) ───── */
.disco-open {
+143 -79
View File
@@ -1,14 +1,34 @@
"use client";
import { useEffect, useState } from "react";
import { describeJob, timeAgo } from "./_ui/status";
import { describeJob, etaLabel, timeAgo } from "./_ui/status";
import { StatTiles } from "./_ui/stat-tiles";
import { SectionHeader } from "./_ui/section-header";
import { JobRow } from "./_ui/job-row";
import { ProgressBar } from "./_ui/progress-bar";
import { PressedList } from "./_ui/pressed-list";
import { KebabMenu } from "./_ui/kebab-menu";
import { toast } from "./_ui/toast";
type Tab = "active" | "queue" | "errors";
// Split the on-the-press rows into three honest buckets so active presses aren't buried between
// queued items and errors. Active = being worked (searching/downloading/finishing), Queue =
// waiting to be claimed, Errors = needs attention.
function categoryOf(state: string, stage: string): Tab {
const kind = describeJob(state, stage).kind;
if (kind === "attention") return "errors";
if (state === "requested") return "queue";
return "active";
}
const TAB_LABEL: Record<Tab, string> = { active: "Active", queue: "In queue", errors: "Errors" };
const TAB_EMPTY: Record<Tab, string> = {
active: "Nothing pressing right now.",
queue: "The queue is empty.",
errors: "No errors — everything's flowing.",
};
type Row = {
id: string;
artist: string;
@@ -26,6 +46,7 @@ type Row = {
candidateCount: number;
chosen: { source: string; format: string; trackCount: number } | null;
downloadProgress: number;
downloadEtaSeconds: number | null;
paused: boolean;
} | null;
};
@@ -42,14 +63,16 @@ function jobOf(r: Row) {
candidateCount: 0,
chosen: null,
downloadProgress: 0,
downloadEtaSeconds: null,
paused: false,
}
);
}
function pressedMark(r: Row): string {
if (!r.createdAt) return "Lyra";
const d = new Date(r.createdAt);
const when = r.job?.updatedAt ?? r.createdAt; // when it was pressed, not when it was requested
if (!when) return "Lyra";
const d = new Date(when);
return `Lyra · ${d.toLocaleDateString("en-GB", { day: "2-digit", month: "short" }).toUpperCase()}`;
}
@@ -60,6 +83,9 @@ export function Queue() {
const [artist, setArtist] = useState("");
const [album, setAlbum] = useState("");
const [paused, setPaused] = useState(false);
// null = follow the smart default (Active if anything's active, else Queue, else Errors); once
// the user picks a tab, their choice sticks.
const [tab, setTab] = useState<Tab | null>(null);
async function refresh() {
const [reqRes, wantRes, artRes, floorRes] = await Promise.all([
@@ -159,76 +185,44 @@ export function Queue() {
}
const active = rows.filter((r) => jobOf(r).state !== "imported");
const downloading = rows.some((r) => {
const s = jobOf(r).state;
return s === "matching" || s === "matched" || s === "downloading" || s === "tagging";
});
// Dedupe by artist+album — re-requesting/re-downloading the same album leaves several
// completed Requests; show each album once (rows arrive newest-first, so keep the first).
const buckets: Record<Tab, Row[]> = { active: [], queue: [], errors: [] };
for (const r of active) {
const j = jobOf(r);
buckets[categoryOf(j.state, j.currentStage)].push(r);
}
// The API returns rows newest-first, but the worker claims the OLDEST queued job next
// (claim_next: ORDER BY createdAt ASC). Order the queue tab to match, so the item that runs
// next sits at the top instead of the bottom.
const ts = (r: Row) => (r.createdAt ? new Date(r.createdAt).getTime() : 0);
buckets.queue.sort((a, b) => ts(a) - ts(b));
// "Next up" = the first queued item the worker will actually claim (oldest, not individually
// paused) — mark it so the ordering reads clearly.
const nextUpId = buckets.queue.find((r) => !jobOf(r).paused)?.id ?? null;
const queuedCount = buckets.queue.length;
const downloading = buckets.active.length > 0;
// Smart default: land on the tab that has something worth looking at.
const defaultTab: Tab = buckets.active.length ? "active" : queuedCount ? "queue" : "errors";
const activeTab: Tab = tab ?? defaultTab;
const shown = buckets[activeTab];
// "Recently Pressed": imported albums ordered by when they were PRESSED (job.updatedAt ≈ import
// time), newest first — an album requested long ago but imported just now is recently pressed
// even though its Request is old (rows arrive by request createdAt, which buried fresh imports).
// Dedupe by artist+album (re-downloads leave several completed Requests; keep the most recent).
const seenPressed = new Set<string>();
const pressed = rows.filter((r) => {
if (jobOf(r).state !== "imported") return false;
const pressedAll = rows
.filter((r) => jobOf(r).state === "imported")
.sort((a, b) => new Date(jobOf(b).updatedAt ?? 0).getTime() - new Date(jobOf(a).updatedAt ?? 0).getTime())
.filter((r) => {
const key = `${r.artist} ${r.album}`.toLowerCase();
if (seenPressed.has(key)) return false;
seenPressed.add(key);
return true;
});
const pressed = pressedAll.slice(0, 10); // show only the 10 most recently pressed
return (
<div>
<StatTiles
items={[
{ k: "On the press", v: active.length },
{ k: "Pressed", v: pressed.length, accent: true },
{ k: "Wanted", v: wanted },
{ k: "Watching", v: watching, unit: "artists" },
]}
/>
<SectionHeader title="On the Press" note={`${active.length} running`} />
<div className="press-controls">
{paused ? (
<button className="btn sm accent" onClick={() => floorAction("resume", "Press resumed")}>
Resume the press
</button>
) : (
<button className="btn sm ghost" onClick={() => floorAction("pause", "Press paused")}>
Pause the press
</button>
)}
<button className="btn sm ghost" onClick={clearPress}>
Clear queue
</button>
{downloading ? (
<button className="btn sm" onClick={stopPress}>
Stop now
</button>
) : null}
{paused ? <span className="chip">Press paused</span> : null}
</div>
<form className="request-form" onSubmit={submit}>
<label className="field">
<span>Artist</span>
<input aria-label="artist" placeholder="Artist" value={artist} onChange={(e) => setArtist(e.target.value)} />
</label>
<label className="field">
<span>Album</span>
<input aria-label="album" placeholder="Album" value={album} onChange={(e) => setAlbum(e.target.value)} />
</label>
<button type="submit" className="btn accent">
Request
</button>
</form>
{active.length === 0 ? (
<p className="empty">The press is quiet. Queue a release above, or send one over from Wanted or Discover.</p>
) : (
<section className="floor">
{(() => {
const now = Date.now();
return active.map((r) => {
function renderRow(r: Row) {
const j = jobOf(r);
const d = describeJob(j.state, j.currentStage);
const attention = d.kind === "attention";
@@ -285,20 +279,16 @@ export function Queue() {
} else if (j.state === "downloading") {
const pct = Math.round((j.downloadProgress || 0) * 100);
if (pct <= 0) {
// Bytes haven't started flowing yet — Qobuz login/metadata/artwork, a
// Soulseek peer queue, or a YouTube info-fetch. Show an indeterminate bar
// instead of a stuck-looking 0% (it flips to the % bar on the first byte).
// Bytes haven't started flowing yet — Qobuz login/metadata/artwork, a Soulseek peer
// queue, or a YouTube info-fetch. Show an indeterminate bar instead of a stuck-looking
// 0% (it flips to the % bar on the first byte).
bar = <ProgressBar kind="working" indeterminate caption="preparing…" />;
} else {
const total = j.chosen?.trackCount ?? 0;
const done = total ? Math.round((pct / 100) * total) : 0;
bar = (
<ProgressBar
kind="working"
percent={pct}
caption={total ? `${pct}% · ${done}/${total} tracks` : `${pct}%`}
/>
);
const base = total ? `${pct}% · ${done}/${total} tracks` : `${pct}%`;
const eta = etaLabel(j.downloadEtaSeconds);
bar = <ProgressBar kind="working" percent={pct} caption={eta ? `${base} · ${eta}` : base} />;
}
} else if (j.state === "tagging") {
bar = <ProgressBar kind="verify" indeterminate caption="finishing" />;
@@ -313,19 +303,93 @@ export function Queue() {
album={r.album}
kind={d.kind}
label={d.label}
marker={r.id === nextUpId ? "Next up" : undefined}
meta={meta}
note={note}
bar={bar}
/>
);
});
})()}
</section>
}
return (
<div>
<StatTiles
items={[
{ k: "On the press", v: active.length },
{ k: "Pressed", v: pressedAll.length, accent: true },
{ k: "Wanted", v: wanted },
{ k: "Watching", v: watching, unit: "artists" },
]}
/>
<div className="dept has-actions">
<h2>On the Press</h2>
<span className="fill" />
{paused ? <span className="count">paused</span> : null}
<KebabMenu label="Press actions">
{paused ? (
<button className="menu-item" onClick={() => floorAction("resume", "Press resumed")}>
Resume the press
</button>
) : (
<button className="menu-item" onClick={() => floorAction("pause", "Press paused")}>
Pause the press
</button>
)}
<button className="menu-item" onClick={clearPress} disabled={queuedCount === 0}>
Clear queue
</button>
{downloading ? (
<button className="menu-item danger" onClick={stopPress}>
Stop now
</button>
) : null}
</KebabMenu>
</div>
<form className="request-form" onSubmit={submit}>
<label className="field">
<span>Artist</span>
<input aria-label="artist" placeholder="Artist" value={artist} onChange={(e) => setArtist(e.target.value)} />
</label>
<label className="field">
<span>Album</span>
<input aria-label="album" placeholder="Album" value={album} onChange={(e) => setAlbum(e.target.value)} />
</label>
<button type="submit" className="btn accent">
Request
</button>
</form>
{active.length === 0 ? (
<p className="empty">The press is quiet. Queue a release above, or send one over from Wanted or Discover.</p>
) : (
<>
<div className="tabs" role="tablist">
{(["active", "queue", "errors"] as Tab[]).map((t) => (
<button
key={t}
role="tab"
aria-selected={activeTab === t}
className={`tab${activeTab === t ? " active" : ""}${t === "errors" && buckets.errors.length ? " has-errors" : ""}`}
onClick={() => setTab(t)}
>
{TAB_LABEL[t]}
<span className="n">{buckets[t].length}</span>
</button>
))}
</div>
{shown.length === 0 ? (
<p className="empty">{TAB_EMPTY[activeTab]}</p>
) : (
<section className="floor">{shown.map(renderRow)}</section>
)}
</>
)}
{pressed.length > 0 ? (
<>
<SectionHeader title="Recently Pressed" note={`${pressed.length} in library`} />
<SectionHeader title="Recently Pressed" note={`last ${pressed.length} · ${pressedAll.length} in library`} />
<PressedList
items={pressed.map((r) => ({ id: r.id, artist: r.artist, album: r.album, matrix: pressedMark(r), rgMbid: r.rgMbid ?? null }))}
/>
+73
View File
@@ -54,6 +54,8 @@ export function SettingsForm() {
const [discoverSaved, setDiscoverSaved] = useState(false);
const [monitorCfg, setMonitorCfg] = useState<Record<string, string>>({});
const [monitorSaved, setMonitorSaved] = useState(false);
const [pacingCfg, setPacingCfg] = useState<Record<string, string>>({});
const [pacingSaved, setPacingSaved] = useState(false);
useEffect(() => {
fetch("/api/scan")
@@ -106,6 +108,12 @@ export function SettingsForm() {
.then((c: Record<string, string>) => setMonitorCfg(c));
}, []);
useEffect(() => {
fetch("/api/qobuz/pacing")
.then((r) => r.json())
.then((c: Record<string, string>) => setPacingCfg(c));
}, []);
async function saveCredentials(e: React.FormEvent) {
e.preventDefault();
await fetch("/api/config", {
@@ -160,6 +168,21 @@ export function SettingsForm() {
setTimeout(() => setMonitorSaved(false), 1500);
}
function setPacing(key: string, value: string) {
setPacingCfg((c) => ({ ...c, [key]: value }));
}
async function savePacing(e: React.FormEvent) {
e.preventDefault();
await fetch("/api/qobuz/pacing", {
method: "PATCH",
headers: { "content-type": "application/json" },
body: JSON.stringify(pacingCfg),
});
setPacingSaved(true);
setTimeout(() => setPacingSaved(false), 1500);
}
const secretHint = (set: boolean) => (set ? <span className="set"> · set</span> : null);
async function clearField(field: "qobuzToken" | "slskdApiKey" | "lastfmApiKey") {
@@ -193,6 +216,7 @@ export function SettingsForm() {
</div>
{tab === "Qobuz" ? (
<>
<form className="settings-section" onSubmit={saveCredentials}>
<div className="subhead">
Auth token
@@ -235,6 +259,55 @@ export function SettingsForm() {
{saved ? <span className="saved-note">Saved</span> : null}
</div>
</form>
<form className="settings-section" onSubmit={savePacing}>
<div className="subhead">
Pacing
<HelpButton title="Qobuz pacing">
Throttles Qobuz to a human-looking volume and cadence so the account isn&apos;t flagged
for bulk downloading. Within active hours it downloads up to the daily cap, spaced out
with randomized gaps; when throttled, albums fall through to Soulseek/YouTube and
upgrade to Qobuz hi-res later. The daily cap ramps up over the first two weeks from the
new-account start date.
</HelpButton>
</div>
<label className="field-inline">
<input type="checkbox" aria-label="pace qobuz downloads"
checked={pacingCfg["qobuz.pacing.enabled"] !== "false"}
onChange={(e) => setPacing("qobuz.pacing.enabled", e.target.checked ? "true" : "false")} />
<span>Pace Qobuz downloads</span>
</label>
<div className="field-grid">
<label className="field">
<span>Active hours start (024)</span>
<input type="number" min={0} max={24} aria-label="qobuz active start hour"
value={pacingCfg["qobuz.pacing.activeStartHour"] ?? ""}
onChange={(e) => setPacing("qobuz.pacing.activeStartHour", e.target.value)} />
</label>
<label className="field">
<span>Active hours end (024)</span>
<input type="number" min={0} max={24} aria-label="qobuz active end hour"
value={pacingCfg["qobuz.pacing.activeEndHour"] ?? ""}
onChange={(e) => setPacing("qobuz.pacing.activeEndHour", e.target.value)} />
</label>
<label className="field">
<span>Daily cap (steady)</span>
<input type="number" min={0} aria-label="qobuz daily cap"
value={pacingCfg["qobuz.pacing.dailyCap"] ?? ""}
onChange={(e) => setPacing("qobuz.pacing.dailyCap", e.target.value)} />
</label>
<label className="field">
<span>New-account start (warm-up)</span>
<input type="date" aria-label="qobuz warmup start date"
value={pacingCfg["qobuz.pacing.warmupStartDate"] ?? ""}
onChange={(e) => setPacing("qobuz.pacing.warmupStartDate", e.target.value)} />
</label>
</div>
<div className="save-row">
<button type="submit" className="btn accent">Save</button>
{pacingSaved ? <span className="saved-note">Saved</span> : null}
</div>
</form>
</>
) : null}
{tab === "Soulseek" ? (
+60 -6
View File
@@ -1,3 +1,4 @@
import time
from difflib import SequenceMatcher
from lyra_worker.types import MBTarget
@@ -5,6 +6,29 @@ from lyra_worker.types import MBTarget
_MIN_TITLE_RATIO = 0.6
def _with_retry(fn, *, attempts: int = 4, base_delay: float = 1.0, sleep=time.sleep):
"""Call `fn`, retrying transient MusicBrainz failures with exponential backoff.
A dropped TLS connection to the MusicBrainz webservice surfaces as
musicbrainzngs.NetworkError (`<urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING]>`),
and MusicBrainz's own 503 rate-limit surfaces as ResponseError; both are
transient, so we retry rather than failing the whole pipeline for the job.
Any other exception (bug, non-transient) propagates immediately.
musicbrainzngs is imported lazily so importing this module stays offline.
"""
import musicbrainzngs
transient = (musicbrainzngs.NetworkError, musicbrainzngs.ResponseError)
for i in range(attempts):
try:
return fn()
except transient:
if i == attempts - 1:
raise
sleep(base_delay * (2**i))
def _ratio(a: str, b: str) -> float:
return SequenceMatcher(None, a.strip().casefold(), b.strip().casefold()).ratio()
@@ -13,6 +37,12 @@ def _is_album(g: dict) -> bool:
return (g.get("primary-type") or "").casefold() == "album"
def _rg_year(g: dict) -> int:
"""First-release year of a release-group, or 9999 when unknown (so undated groups lose ties)."""
frd = (g.get("first-release-date") or "")[:4]
return int(frd) if frd.isdigit() else 9999
def _credited_artist(g: dict) -> str:
"""Primary credited artist name of a release-group search result, or '' if absent."""
credit = g.get("artist-credit")
@@ -32,7 +62,12 @@ def _best_release_group(artist: str, album: str, groups: list) -> dict | None:
* title similarity is next;
* an Album primary-type only breaks ties *within* the same artist+title, so a famous
album (Michael Jackson's "Off the Wall") still isn't resolved to its same-named
single whose short tracklist would map positionally onto the album's files.
single whose short tracklist would map positionally onto the album's files;
* finally, the EARLIEST release-group wins — MusicBrainz lists the original album and its
later reissues/anniversary editions as separate same-titled groups (Linkin Park "Hybrid
Theory": a 2000 original and a 2023 13-track reissue), which tie on everything above, so
MB's arbitrary relevance order used to pick a reissue whose longer tracklist/runtime then
failed completeness checks against the standard album a source delivers. Prefer the original.
Artist is a ranking signal, never a hard filter — a low match sinks a candidate but
never drops the release, so credited-name variations (feat., punctuation) still resolve.
Returns None below the title threshold. Pure — no network I/O."""
@@ -44,6 +79,7 @@ def _best_release_group(artist: str, album: str, groups: list) -> dict | None:
_ratio(artist, _credited_artist(g)),
_ratio(album, g.get("title", "")),
_is_album(g),
-_rg_year(g), # tie-break: the original (earliest) group over later reissues
),
)
if _ratio(album, best.get("title", "")) < _MIN_TITLE_RATIO:
@@ -51,6 +87,23 @@ def _best_release_group(artist: str, album: str, groups: list) -> dict | None:
return best
def _pick_release(releases: list) -> dict | None:
"""Choose which release of a release-group to read the tracklist from. MB returns them in an
arbitrary order and ``releases[0]`` is often a later reissue/deluxe/anniversary edition with
more tracks and a longer runtime than the standard album a source actually delivers — which
then makes a *complete* download fail the pipeline's track-count/duration completeness checks
and re-download forever (Linkin Park "Hybrid Theory": MB's arbitrary pick was a 2023 13-track
reissue at ~52 min vs the 2000 original's 12 tracks / ~38 min). Prefer the ORIGINAL standard
edition: an Official release, then the earliest date, then a dated one. Pure — unit-tested."""
if not releases:
return None
def _key(r: dict) -> tuple:
official = 0 if (r.get("status") == "Official") else 1 # official editions first
date = r.get("date") or ""
return (official, 0 if date else 1, date) # then dated, then earliest date
return min(releases, key=_key)
class MusicBrainzResolver:
"""Real MbResolver using the MusicBrainz webservice via musicbrainzngs.
@@ -68,7 +121,7 @@ class MusicBrainzResolver:
musicbrainzngs.set_useragent(self._app, self._version, self._contact)
res = musicbrainzngs.search_release_groups(query=album, artist=artist, limit=5)
res = _with_retry(lambda: musicbrainzngs.search_release_groups(query=album, artist=artist, limit=5))
groups = res.get("release-group-list", [])
rg = _best_release_group(artist, album, groups)
if rg is None:
@@ -93,11 +146,12 @@ class MusicBrainzResolver:
titles: tuple[str, ...] = ()
rgid = rg.get("id")
if rgid:
rgfull = musicbrainzngs.get_release_group_by_id(rgid, includes=["releases"])
rgfull = _with_retry(lambda: musicbrainzngs.get_release_group_by_id(rgid, includes=["releases"]))
releases = rgfull.get("release-group", {}).get("release-list", [])
if releases:
rel = musicbrainzngs.get_release_by_id(
releases[0]["id"], includes=["recordings"]
chosen = _pick_release(releases)
if chosen:
rel = _with_retry(
lambda: musicbrainzngs.get_release_by_id(chosen["id"], includes=["recordings"])
).get("release", {})
tracks = []
for medium in rel.get("medium-list", []):
+201 -41
View File
@@ -2,6 +2,7 @@ import json
import os
import shutil
import time
import unicodedata
from typing import Callable
import requests
@@ -11,8 +12,25 @@ _DEFAULT_DOWNLOADS_ROOT = "/slskd-downloads"
_AUDIO_EXT = {"flac", "mp3", "m4a", "ogg", "opus", "wav", "aac"}
_LOSSLESS_EXT = {"flac", "wav"}
_SEARCH_POLLS = 40 # * 3s ≈ 2 min
_XFER_POLLS = 200 # * 3s ≈ 10 min
_STALL_POLLS = 30 # * 3s ≈ 90s of no byte progress → abandon a queued/dead peer and fall through
_POLL_SECONDS = 3
# Absolute backstop for a slow-but-progressing peer. Scaled to the album's total bytes (a flat
# cap starved big albums: a 500MB set from a ~400KB/s serial peer needs ~60-70min and was cut at
# a flat 60min, then re-downloaded from scratch — never finishing). Budget at a conservative floor
# throughput, clamped. The 90s stall timeout still abandons a truly dead/queued peer quickly.
_MIN_XFER_POLLS = 400 # * 3s ≈ 20 min floor (small albums)
_MAX_XFER_POLLS = 2400 # * 3s ≈ 2 h ceiling (bounds worst-case slot hold)
_BACKSTOP_FLOOR_BPS = 64_000 # budget the backstop assuming >= ~64 KB/s sustained
def _max_polls(total_bytes: int) -> int:
"""Poll budget (each _POLL_SECONDS) for the absolute download backstop, scaled to album size at
a conservative floor throughput so a large slow-but-healthy transfer isn't abandoned mid-flight.
Clamped to [_MIN_XFER_POLLS, _MAX_XFER_POLLS]. Pure — unit-tested."""
if total_bytes <= 0:
return _MIN_XFER_POLLS
budget = int(total_bytes / (_BACKSTOP_FLOOR_BPS * _POLL_SECONDS))
return max(_MIN_XFER_POLLS, min(budget, _MAX_XFER_POLLS))
def _basename(path: str) -> str:
@@ -27,6 +45,114 @@ def _dirname(path: str) -> str:
return ""
_ETA_CAP_SECONDS = 359_999 # ~100h; keeps the value well inside Job.downloadEtaSeconds (int4)
def _eta_seconds(total_bytes: int, bytes_now: int, speed_bps: float) -> int | None:
"""Seconds until the transfer finishes at the current measured rate, or None when it can't be
estimated (no throughput sample yet, unknown total, or already complete). Pure — unit-tested.
A near-stalled peer drives speed_bps toward zero, so the estimate is capped: an uncapped value
can exceed int4 and overflow the downloadEtaSeconds column write (poisoning the pipeline txn)."""
if speed_bps <= 0 or total_bytes <= 0 or bytes_now >= total_bytes:
return None
return min(int((total_bytes - bytes_now) / speed_bps), _ETA_CAP_SECONDS)
def _norm(s: str) -> str:
"""Lowercase, strip diacritics, and reduce to alphanumeric tokens, so 'Beyoncé' matches a
peer's 'Beyonce' folder and punctuation ('I Am… Sasha Fierce') doesn't defeat the match."""
s = unicodedata.normalize("NFKD", s or "")
s = "".join(ch for ch in s if not unicodedata.combining(ch))
s = "".join(ch if ch.isalnum() else " " for ch in s.lower())
return " ".join(s.split())
def _path_matches(needle: str, path: str) -> bool:
"""True if `needle` (an artist or album name) appears in a file path, tolerant of accents and
punctuation. Every significant token (≥3 chars) must be present; for a very short needle
(e.g. 'U2') the whole normalized string must appear."""
npath = _norm(path)
tokens = [t for t in _norm(needle).split() if len(t) >= 3]
if not tokens:
whole = _norm(needle)
return bool(whole) and whole in npath
return all(t in npath for t in tokens)
def _keep_matching(responses: list, needle: str) -> list:
"""Drop files — then now-empty responses — whose path doesn't match `needle`. Makes an album-
or artist-only fallback search precise so we never grab a same-titled release by a different
artist. Pure — unit-tested offline."""
out = []
for resp in responses:
files = [f for f in (resp.get("files") or []) if _path_matches(needle, f.get("filename", ""))]
if files:
out.append({**resp, "files": files})
return out
def _parse_search_responses(responses: list, artist: str = "") -> list[dict]:
"""Turn slskd search responses into album candidates (one per peer+directory), ordered so the
peers that will FINISH first come first: a free upload slot, then the shortest estimated
transfer time (album bytes ÷ the peer's advertised speed), then a shorter queue. Ranking on
estimated time — not raw speed — matters because Soulseek FLAC rips of the same album vary
~2x in size (a 24-bit/bloated rip vs a standard CD rip), and Lyra scores them the SAME quality
class (the search exposes no bit-depth), so a peer advertising high speed but serving huge
files can lose a race — and blow past the download backstop — against a peer with a smaller
standard rip. The pipeline tries candidates in this order, so this is what makes the Soulseek
fall-through converge on a copy that actually completes. Pure — no I/O — unit-tested offline."""
scored: list[tuple] = []
for resp in responses:
username = resp.get("username", "")
has_slot = 0 if resp.get("hasFreeUploadSlot") else 1 # 0 sorts first (free slot = starts now)
speed = int(resp.get("uploadSpeed") or 0)
queue = int(resp.get("queueLength") or 0)
by_dir: dict[str, list] = {}
for f in resp.get("files", []) or []:
name = f.get("filename", "")
ext = name.rsplit(".", 1)[-1].lower() if "." in name else ""
if ext not in _AUDIO_EXT:
continue
by_dir.setdefault(_dirname(name), []).append(
{"filename": name, "size": f.get("size", 0), "ext": ext}
)
for directory, dfiles in by_dir.items():
if not dfiles:
continue
# Estimated seconds to pull this album from this peer at its advertised rate. Unknown/
# zero speed → a low nominal (1 B/s) so total size still orders those peers last.
total_bytes = sum(int(f["size"] or 0) for f in dfiles)
est_seconds = total_bytes / (speed if speed > 0 else 1)
lossless = all(f["ext"] in _LOSSLESS_EXT for f in dfiles)
dirname = _basename(directory)
# Trust the REQUESTED artist when it appears anywhere in the path — many rips put the
# artist in a parent folder ("Rihanna\(2009) Rated R\…") or bury it ("2009 - Rihanna -
# Rated R"), which the "Artist - Album" folder parse gets wrong; that tanked the
# confidence score and lost real FLACs to clean-named MP3s. album stays the folder name
# (fuzzy-matched downstream); artist is what the parse gets wrong.
if artist and _path_matches(artist, dfiles[0]["filename"]):
guess_artist, guess_album = artist, dirname
elif " - " in dirname:
guess_artist, guess_album = (p.strip() for p in dirname.split(" - ", 1))
else:
guess_artist, guess_album = "", dirname
candidate = {
"source_ref": json.dumps({
"username": username,
"files": [{"filename": f["filename"], "size": f["size"]} for f in dfiles],
}),
"title": guess_album,
"artist": guess_artist,
"track_count": len(dfiles),
"format": "FLAC" if lossless else "MP3",
"bitrate": None,
}
scored.append((has_slot, est_seconds, queue, candidate))
# ascending: free slot first, then shortest estimated transfer, then shortest queue
scored.sort(key=lambda t: (t[0], t[1], t[2]))
return [c for *_rest, c in scored]
class SlskdClient:
"""Real Soulseek client talking to an slskd daemon's REST API.
@@ -58,11 +184,12 @@ class SlskdClient:
r.raise_for_status()
return r.json()
def search_album(self, artist: str, album: str) -> list[dict]:
def _run_search(self, text: str) -> list:
"""POST a search, wait for it to complete, and return the raw responses (a list)."""
r = requests.post(
f"{self._url}/api/v0/searches",
headers=self._headers(),
json={"searchText": f"{artist} {album}"},
json={"searchText": text},
timeout=30,
)
r.raise_for_status()
@@ -75,43 +202,27 @@ class SlskdClient:
break
responses = self._get(f"/api/v0/searches/{search_id}/responses")
candidates: list[dict] = []
for resp in responses:
username = resp.get("username", "")
by_dir: dict[str, list] = {}
for f in resp.get("files", []) or []:
name = f.get("filename", "")
ext = name.rsplit(".", 1)[-1].lower() if "." in name else ""
if ext not in _AUDIO_EXT:
return responses if isinstance(responses, list) else []
def search_album(self, artist: str, album: str) -> list[dict]:
responses = self._run_search(f"{artist} {album}")
if not responses:
# The Soulseek server silently drops searches containing certain blocklisted terms
# (major-label artists / album titles like Adele, Rihanna, Beyoncé, "Lemonade") →
# 0 responses. The blocked term poisons the whole query, but the OTHER field usually
# isn't blocked, so retry with each field alone (normalized) and keep only results
# whose path still matches the field we dropped. That recovers e.g. Beyoncé's
# "I Am… Sasha Fierce" via the album title, without ever grabbing a same-titled album
# by someone else.
for query, needle in ((album, artist), (artist, album)):
q = _norm(query)
if not q:
continue
by_dir.setdefault(_dirname(name), []).append(
{"filename": name, "size": f.get("size", 0), "ext": ext}
)
for directory, dfiles in by_dir.items():
if not dfiles:
continue
lossless = all(f["ext"] in _LOSSLESS_EXT for f in dfiles)
dirname = _basename(directory)
if " - " in dirname:
guess_artist, guess_album = (p.strip() for p in dirname.split(" - ", 1))
else:
guess_artist, guess_album = "", dirname
candidates.append(
{
"source_ref": json.dumps(
{
"username": username,
"files": [{"filename": f["filename"], "size": f["size"]} for f in dfiles],
}
),
"title": guess_album,
"artist": guess_artist,
"track_count": len(dfiles),
"format": "FLAC" if lossless else "MP3",
"bitrate": None,
}
)
return candidates
alt = _keep_matching(self._run_search(q), needle)
if alt:
responses = alt
break
return _parse_search_responses(responses, artist)
def download(self, source_ref: str, dest: str, on_progress: Callable[[float], None]) -> dict:
ref = json.loads(source_ref)
@@ -119,7 +230,17 @@ class SlskdClient:
files = ref["files"]
wanted = {f["filename"] for f in files}
os.makedirs(dest, exist_ok=True)
xfer_ids: set[str] = set() # slskd transfer ids we started, for cancel-on-abandon
try:
return self._download(username, files, wanted, dest, on_progress, xfer_ids)
except Exception:
# Abandoning this peer (timeout / error / the pipeline moving on): cancel the transfers
# we enqueued so they don't keep running and pile up in slskd as the job falls through
# to other peers/sources for the same album.
self._cancel_transfers(username, xfer_ids)
raise
def _download(self, username, files, wanted, dest, on_progress, xfer_ids: set) -> dict:
r = requests.post(
f"{self._url}/api/v0/transfers/downloads/{username}",
headers=self._headers(),
@@ -130,23 +251,49 @@ class SlskdClient:
on_progress(0.0)
total = len(files)
total_bytes = sum(int(f.get("size") or 0) for f in files)
completed = False
for _ in range(_XFER_POLLS):
last_bytes = -1
stall = 0
speed = 0.0 # bytes/sec, EWMA-smoothed so the ETA doesn't jitter poll to poll
prev_bytes = 0
prev_t = time.monotonic()
for _ in range(_max_polls(total_bytes)):
time.sleep(_POLL_SECONDS)
data = self._get(f"/api/v0/transfers/downloads/{username}")
states: list[str] = []
bytes_now = 0
for directory in data.get("directories", []) if isinstance(data, dict) else []:
for f in directory.get("files", []):
if f.get("filename") in wanted: # only the files we enqueued
states.append(str(f.get("state", "")))
bytes_now += int(f.get("bytesTransferred") or 0)
if f.get("id"): # remember the transfer so we can cancel it if abandoned
xfer_ids.add(str(f["id"]))
if states:
if any(bad in s for s in states for bad in ("Errored", "Rejected", "Cancelled")):
raise RuntimeError("soulseek transfer failed")
now = time.monotonic()
dt = now - prev_t
if dt > 0 and bytes_now >= prev_bytes:
inst = (bytes_now - prev_bytes) / dt
speed = inst if speed == 0.0 else 0.6 * speed + 0.4 * inst
prev_bytes, prev_t = bytes_now, now
done = sum(1 for s in states if "Completed" in s)
on_progress(min(1.0, done / max(total, 1)))
on_progress(min(1.0, done / max(total, 1)), _eta_seconds(total_bytes, bytes_now, speed))
if done >= total:
completed = True
break
# Stall detection: a peer making byte progress keeps its slot (up to the absolute
# backstop above) so a slow-but-working transfer can finish; one that sends no bytes
# for _STALL_POLLS (~90s) is queued/dead — abandon it fast and fall through.
if bytes_now > last_bytes:
last_bytes = bytes_now
stall = 0
else:
stall += 1
if stall >= _STALL_POLLS:
raise TimeoutError(f"soulseek transfer stalled for {username}")
if not completed:
raise TimeoutError(f"soulseek download did not complete for {username}")
@@ -162,6 +309,19 @@ class SlskdClient:
)
return {"track_count": moved, "path": dest}
def _cancel_transfers(self, username: str, xfer_ids: set) -> None:
"""Cancel the given in-flight slskd transfers for a peer. Best-effort: a cleanup failure
must never mask the original download failure."""
for tid in xfer_ids:
try:
requests.delete(
f"{self._url}/api/v0/transfers/downloads/{username}/{tid}",
headers=self._headers(),
timeout=15,
)
except Exception:
pass
def _retrieve(self, files: list[dict], dest: str) -> int:
"""Move the just-finished slskd files out of its downloads dir into `dest`.
+37
View File
@@ -24,6 +24,43 @@ def reclaim_stuck_jobs(conn: psycopg.Connection) -> int:
return n
_MAX_ATTEMPTS = 5
def requeue_or_fail(
conn: psycopg.Connection, job_id: str, error: str, max_attempts: int = _MAX_ATTEMPTS
) -> None:
"""Recover a job whose pipeline raised an *unhandled* exception (e.g. a dropped MusicBrainz
TLS connection surfacing as an SSL EOF). Without this the job is left in its mid-pipeline
'matching' state; since claim_next only picks 'requested' it is orphaned until the next
restart's reclaim_stuck_jobs — showing "Searching sources…" on the press indefinitely.
Such errors are usually transient, so requeue to 'requested' until the attempt cap, then fall
to 'needs_attention' so a persistently-failing job stays visible instead of looping forever.
`attempts` is bumped by claim_next on every claim, so it already counts tries."""
with conn.cursor() as cur:
cur.execute('SELECT attempts, "requestId" FROM "Job" WHERE id = %s', (job_id,))
row = cur.fetchone()
if row is None:
return
attempts, request_id = row
if attempts >= max_attempts:
cur.execute(
'UPDATE "Job" SET state = \'needs_attention\', error = %s, "claimedAt" = NULL, '
'"updatedAt" = now() WHERE id = %s',
(error, job_id),
)
cur.execute('UPDATE "Request" SET status = \'needs_attention\' WHERE id = %s', (request_id,))
else:
cur.execute(
'UPDATE "Job" SET state = \'requested\', "currentStage" = \'intake\', error = NULL, '
'"claimedAt" = NULL, "downloadProgress" = 0, "updatedAt" = now() WHERE id = %s',
(job_id,),
)
cur.execute('UPDATE "Request" SET status = \'pending\' WHERE id = %s', (request_id,))
conn.commit()
def claim_next(conn: psycopg.Connection) -> str | None:
"""Atomically claim the oldest queued job. Returns its id or None."""
with conn.cursor() as cur:
+4 -1
View File
@@ -6,7 +6,7 @@ import uuid
import psycopg
from lyra_worker.claim import claim_next, reclaim_stuck_jobs
from lyra_worker.claim import claim_next, reclaim_stuck_jobs, requeue_or_fail
from lyra_worker.config import get_config
from lyra_worker.crypto import secret_key_problem
from lyra_worker.db import wait_for_db
@@ -323,6 +323,9 @@ def run_forever() -> None:
except Exception as e: # one bad job must not take down the worker loop
print(f"worker: pipeline failed for job {job_id}: {e}", flush=True)
conn.rollback()
# Don't orphan the job in its mid-pipeline 'matching' state (claim_next only
# picks 'requested'): requeue for a bounded retry, then needs_attention.
requeue_or_fail(conn, job_id, str(e))
finish_job(conn, job_id, mcfg)
print(f"worker: finished job {job_id}", flush=True)
next_job_allowed = time.monotonic() + job_delay_seconds(config)
+59 -17
View File
@@ -86,16 +86,40 @@ def _set_state(conn: psycopg.Connection, rel_id: str, state: str) -> None:
cur.execute('UPDATE "MonitoredRelease" SET state = %s WHERE id = %s', (state, rel_id))
def _in_library_at_cutoff(conn: psycopg.Connection, artist: str, album: str, cutoff: int) -> bool:
def _in_library_at_cutoff(
conn: psycopg.Connection, artist: str, album: str, rg_mbid: str, cutoff: int
) -> bool:
"""True if the album is owned at/above `cutoff`. Matches on the release-group MBID when the
library row has one — robust against MusicBrainz artist-name canonicalization (e.g. the
followed 'Kanye West' imports as 'Ye') — and falls back to exact artist+album for scanned
rows that predate MBID capture."""
with conn.cursor() as cur:
cur.execute(
'SELECT 1 FROM "LibraryItem" WHERE artist = %s AND album = %s '
'AND "qualityClass" >= %s LIMIT 1',
(artist, album, cutoff),
'SELECT 1 FROM "LibraryItem" WHERE "qualityClass" >= %s AND ('
' ("rgMbid" IS NOT NULL AND "rgMbid" = %s) '
' OR ("rgMbid" IS NULL AND artist = %s AND album = %s)) LIMIT 1',
(cutoff, rg_mbid, artist, album),
)
return cur.fetchone() is not None
def _retire_failed_attempts(conn: psycopg.Connection, rel_id: str) -> None:
"""Delete prior failed (needs_attention) Request rows for this release (Job/Candidate cascade).
A hard-to-match release is re-pressed every retry interval; each failed attempt used to leave
a needs_attention Request behind, piling up phantom "needs attention" cards on the press even
after a later attempt imported the album. Retiring them when the release is fulfilled or
re-attempted keeps the press showing one live attempt at a time. Manual (non-monitor) requests
have no monitoredReleaseId and are never touched."""
with conn.cursor() as cur:
cur.execute(
'DELETE FROM "Request" r USING "Job" j '
'WHERE j."requestId" = r.id AND r."monitoredReleaseId" = %s '
"AND j.state = 'needs_attention'",
(rel_id,),
)
def _enqueue_one(conn: psycopg.Connection, rel_id: str, artist: str, album: str) -> None:
with conn.cursor() as cur:
cur.execute(
@@ -118,7 +142,8 @@ def enqueue_due(conn: psycopg.Connection, cfg: MonitorConfig) -> int:
cur.execute(
'SELECT mr.id, mr."artistName", mr.album, mr.state, mr."currentQualityClass", '
' (mr."firstGrabbedAt" IS NOT NULL '
' AND mr."firstGrabbedAt" < now() - make_interval(days => %s)) AS window_expired '
' AND mr."firstGrabbedAt" < now() - make_interval(days => %s)) AS window_expired, '
' mr."rgMbid" '
'FROM "MonitoredRelease" mr '
'WHERE mr.monitored = true '
' AND NOT mr.ignored ' # user-ignored releases are never enqueued
@@ -134,8 +159,11 @@ def enqueue_due(conn: psycopg.Connection, cfg: MonitorConfig) -> int:
rows = cur.fetchall()
enqueued = 0
for rel_id, artist, album, state, quality, window_expired in rows:
if _in_library_at_cutoff(conn, artist, album, cfg.quality_cutoff):
for rel_id, artist, album, state, quality, window_expired, rg_mbid in rows:
# A due release is about to be fulfilled or re-attempted; either way any earlier failed
# attempt is superseded, so retire it before deciding.
_retire_failed_attempts(conn, rel_id)
if _in_library_at_cutoff(conn, artist, album, rg_mbid, cfg.quality_cutoff):
_set_state(conn, rel_id, "fulfilled")
continue
if state == "grabbed" and (window_expired or (quality is not None and quality >= cfg.quality_cutoff)):
@@ -154,12 +182,19 @@ def fulfill_owned_releases(conn: psycopg.Connection, cfg: MonitorConfig) -> int:
with conn.cursor() as cur:
cur.execute(
'UPDATE "MonitoredRelease" mr SET state = \'fulfilled\', '
' "currentQualityClass" = li.q, '
' "currentQualityClass" = owned.q, '
' "firstGrabbedAt" = COALESCE(mr."firstGrabbedAt", now()) '
'FROM (SELECT artist, album, max("qualityClass") AS q FROM "LibraryItem" '
' GROUP BY artist, album) li '
'WHERE li.artist = mr."artistName" AND li.album = mr.album '
" AND mr.monitored = true AND mr.state <> 'fulfilled' AND li.q >= %s",
'FROM ('
' SELECT mr2.id AS mid, max(li."qualityClass") AS q '
' FROM "MonitoredRelease" mr2 JOIN "LibraryItem" li ON ('
# match on rgMbid where the library row has one (survives artist-name canonicalization),
# else fall back to exact artist+album for scanned rows lacking an MBID
' (li."rgMbid" IS NOT NULL AND li."rgMbid" = mr2."rgMbid") '
' OR (li."rgMbid" IS NULL AND li.artist = mr2."artistName" AND li.album = mr2.album)) '
" WHERE mr2.monitored = true AND mr2.state <> 'fulfilled' "
' GROUP BY mr2.id HAVING max(li."qualityClass") >= %s'
') owned '
'WHERE mr.id = owned.mid',
(cfg.quality_cutoff,),
)
fixed = cur.rowcount
@@ -171,21 +206,28 @@ def reconcile(conn: psycopg.Connection, job_id: str, cfg: MonitorConfig) -> None
"""Feed a finished job's outcome back into its MonitoredRelease (if any)."""
with conn.cursor() as cur:
cur.execute(
'SELECT j.state, r."monitoredReleaseId", r.artist, r.album '
'FROM "Job" j JOIN "Request" r ON r.id = j."requestId" WHERE j.id = %s',
'SELECT j.state, r."monitoredReleaseId", r.artist, r.album, mr."rgMbid" '
'FROM "Job" j JOIN "Request" r ON r.id = j."requestId" '
'LEFT JOIN "MonitoredRelease" mr ON mr.id = r."monitoredReleaseId" '
'WHERE j.id = %s',
(job_id,),
)
row = cur.fetchone()
if row is None:
return
state, rel_id, artist, album = row
state, rel_id, artist, album, rg_mbid = row
if rel_id is None or state != "imported":
return # needs_attention (or in-flight) leaves the release wanted
# A successful import supersedes any earlier failed attempts for this release.
_retire_failed_attempts(conn, rel_id)
with conn.cursor() as cur:
cur.execute(
'SELECT "qualityClass" FROM "LibraryItem" WHERE artist = %s AND album = %s',
(artist, album),
'SELECT max("qualityClass") FROM "LibraryItem" WHERE '
' ("rgMbid" IS NOT NULL AND "rgMbid" = %s) '
' OR ("rgMbid" IS NULL AND artist = %s AND album = %s)',
(rg_mbid, artist, album),
)
lib = cur.fetchone()
quality = lib[0] if lib else None
+142 -43
View File
@@ -1,4 +1,5 @@
import os
import random
import shutil
import threading
import time
@@ -11,6 +12,7 @@ import psycopg
from lyra_worker.adapters.base import SourceAdapter
from lyra_worker.confidence import score_confidence
from lyra_worker.library import album_dir, import_album, list_audio_files, staging_dir
from lyra_worker.qobuz_gate import gate_open, record_download
from lyra_worker.quality import quality_class
from lyra_worker.ranker import rank_candidates
from lyra_worker.types import Candidate, MBTarget
@@ -20,8 +22,19 @@ _AUDIO_EXT = {".flac", ".mp3", ".m4a", ".opus", ".ogg", ".aac", ".wav"}
# Reject a download whose total playtime falls below this fraction of MusicBrainz's expected
# total — catches truncated files / preview clips substituted for real tracks that keep the
# track COUNT right. Generous, so edition/encoding differences don't false-positive; bonus
# tracks (over-long) are always fine.
_DURATION_MIN_RATIO = 0.85
# tracks (over-long) are always fine. Kept well below 1.0 because a source legitimately delivers a
# different edition than MB's resolved release (shorter radio edits, no bonus/live tracks), and only
# a genuinely truncated/preview-filled download (playtime a fraction of expected) should be caught.
_DURATION_MIN_RATIO = 0.65
# Cap how many ranked candidates a single job will actually download+verify before giving up.
# Without this, the incomplete-download fall-through can grind through dozens of sources (a popular
# album returns ~200 Soulseek candidates), each a slow attempt that also leaves an abandoned slskd
# transfer. The best candidates rank first, so 6 attempts is plenty.
_MAX_DOWNLOAD_ATTEMPTS = 6
# The best quality any non-Qobuz source offers (Soulseek FLAC = CD-lossless). A library copy at or
# above this can only be improved by hi-res Qobuz, so when Qobuz is gated by pacing an upgrade of
# such a copy is deferred without even searching.
_CD_LOSSLESS_CLASS = 2
def _measure_staged_duration_s(staging: str) -> float | None:
@@ -70,6 +83,40 @@ def _count_staged_audio(staging: str) -> int:
return n
def _download_problem(result, staging: str, target, winner, measure_duration) -> str | None:
"""Why a completed download is unusable — ``"incomplete download"`` / ``"download too short
(...)"`` — or None if it's good. Completeness is judged against the CHOSEN SOURCE's own track
count (``winner.track_count``), NOT MusicBrainz's canonical release: MB picks an arbitrary
releases[0] that is often a deluxe/expanded edition with more tracks (and runtime) than the
standard album a source legitimately delivers, so gating on ``target.track_count`` falsely
rejects complete standard-edition downloads (The Script "No Sound Without Silence" 11 vs MB 12,
Skillet "Unleashed" 12 vs MB's 20-track deluxe). Pure — no DB I/O — so the download loop can
call it per candidate and fall through to the next source on a problem."""
expected = winner.track_count
# (a) Truncated: fewer files than the source promised (a track failed mid-download). Count the
# NON-EMPTY audio on disk too: a silently-skipped track leaves a 0-byte placeholder the adapter
# still counts. (`staged and ...` keeps fake adapters that stage nothing on the count path.)
staged = _count_staged_audio(staging)
truncated = result.track_count < expected or bool(staged and staged < expected)
# (b) Implausibly small vs MB's album — a lone-track "full album" video / wrong match, as
# distinct from a legitimately smaller edition. A real edition keeps more than half MB's tracks.
too_small = target.track_count is not None and expected * 2 <= target.track_count
if truncated or too_small:
return "incomplete download"
# Duration: a truncated file or short preview substituted for a real track shows up as playtime
# well under the expected total. Scale MB's total to the DELIVERED edition's size so a smaller
# edition isn't judged against a larger one's runtime. Only when the total is known (measured is
# None for the offline fakes).
if target.total_duration_s:
expected_total = target.total_duration_s
if target.track_count:
expected_total *= winner.track_count / target.track_count
measured = measure_duration(staging)
if measured is not None and measured < expected_total * _DURATION_MIN_RATIO:
return f"download too short ({measured / 60:.0f} of ~{expected_total / 60:.0f} min)"
return None
def _poll_download_progress(job_id: str, staging: str, expected: int, stop: "threading.Event",
reports: "threading.Event") -> None:
"""Until `stop`, periodically write Job.downloadProgress = files-in-staging / expected
@@ -114,30 +161,42 @@ def _set_state(conn: psycopg.Connection, job_id: str, state: str, stage: str) ->
conn.commit()
def _set_download_progress(conn: psycopg.Connection, job_id: str, frac: float) -> None:
def _set_download_progress(
conn: psycopg.Connection, job_id: str, frac: float, eta_seconds: int | None = None
) -> None:
with conn.cursor() as cur:
cur.execute('UPDATE "Job" SET "downloadProgress" = %s WHERE id = %s', (frac, job_id))
cur.execute(
'UPDATE "Job" SET "downloadProgress" = %s, "downloadEtaSeconds" = %s WHERE id = %s',
(frac, eta_seconds, job_id),
)
conn.commit()
def _make_on_progress(conn: psycopg.Connection, job_id: str, reports: "threading.Event"):
"""Build the on_progress(pct) callback threaded into adapter.download. It marks the
adapter as reporting real byte-level progress (so the file-count poller yields) and
throttle-writes Job.downloadProgress. Called synchronously from adapter.download on the
pipeline's own thread, so it safely reuses `conn`."""
"""Build the on_progress(pct, eta_seconds=None) callback threaded into adapter.download. It
marks the adapter as reporting real byte-level progress (so the file-count poller yields) and
throttle-writes Job.downloadProgress (+ the download ETA when the adapter measures one — slskd
does; others pass None). Called synchronously from adapter.download on the pipeline's own
thread, so it safely reuses `conn`."""
state = {"frac": -1.0, "t": 0.0}
def on_progress(pct: float) -> None:
def on_progress(pct: float, eta_seconds: int | None = None) -> None:
reports.set()
frac = 0.0 if pct < 0 else 1.0 if pct > 1 else float(pct)
now = time.monotonic()
# throttle DB writes: on a >=1% move or every 0.5s, not once per received byte
if frac - state["frac"] >= 0.01 or (now - state["t"]) >= 0.5:
# throttle DB writes: on a >=1% move, a new ETA, or every 0.5s not once per received byte
if frac - state["frac"] >= 0.01 or eta_seconds is not None or (now - state["t"]) >= 0.5:
state["frac"] = frac
state["t"] = now
try:
_set_download_progress(conn, job_id, frac)
_set_download_progress(conn, job_id, frac, eta_seconds)
except Exception as e: # a progress write must never fail the download
# Roll back so a failed write can't leave the shared pipeline conn in an aborted
# txn (which would cascade "current transaction is aborted" into every later query).
try:
conn.rollback()
except Exception:
pass
print(f"pipeline: on_progress write failed: {e}", flush=True)
return on_progress
@@ -225,6 +284,17 @@ def _library_quality(conn: psycopg.Connection, target: MBTarget) -> int | None:
return row[0] if row else None
def _keep_existing_and_complete(conn: psycopg.Connection, job_id: str) -> None:
"""Finish a job without importing — keep the existing library copy — and mark its request
completed. Used when a (deferred) upgrade has nothing better to offer right now; the release
stays below the cutoff and is re-attempted later."""
_set_state(conn, job_id, "imported", "import")
with conn.cursor() as cur:
cur.execute("UPDATE \"Request\" SET status = 'completed' WHERE id = %s",
(_request_id(conn, job_id),))
conn.commit()
def _persist_candidates(conn: psycopg.Connection, job_id: str, candidates: list[Candidate]) -> None:
with conn.cursor() as cur:
for c in candidates:
@@ -317,6 +387,16 @@ def run_pipeline(
conn.commit()
return
# Pre-search upgrade skip: a >= CD-lossless copy can only be improved by hi-res Qobuz. If Qobuz
# is gated by pacing, don't even search (no other source can beat it) — defer to when Qobuz
# frees up. This avoids re-searching every already-owned album (~2 min Soulseek search each) on
# every monitor cycle. A user-forced re-acquire always proceeds.
_existing_q = _library_quality(conn, target)
if _existing_q is not None and _existing_q >= _CD_LOSSLESS_CLASS \
and not _is_force_job(conn, job_id) and not gate_open(conn):
_keep_existing_and_complete(conn, job_id)
return
# 2. match — search all adapters CONCURRENTLY (Soulseek's search polls up to ~2 min; run it
# alongside Qobuz's instant search so total match time = the slowest source, not their sum).
# Only Qobuz touches streamrip's shared asyncio loop and there's a single Qobuz adapter, so no
@@ -329,8 +409,21 @@ def run_pipeline(
for adapter, results in per_adapter:
for c in results:
found.append(replace(c, source_tier=adapter.tier, confidence=score_confidence(target, c)))
# Drop candidates that can't be the full album — a result with far fewer tracks than the release
# has (e.g. a single-song Soulseek folder named like the album). Same threshold as the
# completeness 'too small' guard, applied up front so we never waste a download attempt on it.
# Unknown counts (0) and legitimately smaller editions (e.g. 11 vs 12) are kept.
if target.track_count:
found = [c for c in found if c.track_count == 0 or c.track_count * 2 > target.track_count]
_persist_candidates(conn, job_id, found)
# Qobuz pacing: when the budget gate is closed (off-hours / daily cap / spacing), drop Qobuz
# candidates so the job falls through to another source and Qobuz stays under the radar — UNLESS
# Qobuz is the only option (nothing to fall through to, so don't strand the album).
if any(c.source == "qobuz" for c in found) and any(c.source != "qobuz" for c in found):
if not gate_open(conn):
found = [c for c in found if c.source != "qobuz"]
# 3. rank
_set_state(conn, job_id, "matched", "rank")
ranked = rank_candidates(target, found, min_confidence)
@@ -338,12 +431,23 @@ def run_pipeline(
_fail(conn, job_id, "no candidate above confidence threshold")
return
# Upgrade guard: for an album already in the library, skip the download when no currently-
# available source beats the copy we have (ranked[0] is the highest-quality candidate). This
# happens when Qobuz is gated by pacing and only a same-quality Soulseek copy is left — the
# download would be discarded by keep-if-better, so defer it. The release stays below the cutoff
# and is re-attempted when a better source frees up. A user-forced re-acquire always proceeds.
existing_q = _library_quality(conn, target)
if existing_q is not None and not _is_force_job(conn, job_id) \
and quality_class(ranked[0].quality) <= existing_q:
_keep_existing_and_complete(conn, job_id) # keep the existing copy; nothing better available
return
# 4. download (fall-through) into an isolated per-job staging dir
_set_state(conn, job_id, "downloading", "download")
by_source = {a.name: a for a in adapters}
staging = staging_dir(staging_root or f"{dest_root}/.staging", job_id)
winner = None
result = None
last_problem = None # why the most recent downloaded-but-rejected candidate was unusable
_set_download_progress(conn, job_id, 0.0) # reset for this run
expected = target.track_count or (ranked[0].track_count if ranked else 0)
_stop = threading.Event()
@@ -354,46 +458,46 @@ def run_pipeline(
_poller.start()
try:
try:
attempts = 0
for candidate in ranked:
adapter = by_source.get(candidate.source)
if adapter is None:
continue
if attempts >= _MAX_DOWNLOAD_ATTEMPTS:
break # don't grind through every peer/source of a popular album
attempts += 1
_mark_chosen(conn, job_id, candidate.source_ref) # reflect the source now in flight
shutil.rmtree(staging, ignore_errors=True) # clean slate per attempt
_reports.clear() # this attempt hasn't reported yet → poller estimates until it does
result = adapter.download(candidate, staging, _make_on_progress(conn, job_id, _reports))
if result.ok:
if not result.ok:
continue
# Verify completeness on the staging copy. A download that lands incomplete or too
# short falls through to the NEXT-ranked source (e.g. Qobuz repeatedly fails one
# track → try the deluxe edition or Soulseek) rather than failing the whole job on
# the first source's shortfall.
last_problem = _download_problem(result, staging, target, candidate, measure_duration)
if last_problem is None:
winner = candidate
break
finally:
_stop.set()
_poller.join(timeout=3)
if winner is None or result is None or not result.ok:
_fail(conn, job_id, "all downloads failed")
if winner is None:
# No source produced a complete album: surface why the last one was rejected (incomplete
# / too short), or "all downloads failed" if none even downloaded.
_fail(conn, job_id, last_problem or "all downloads failed")
return
_set_download_progress(conn, job_id, 1.0) # download done — UI shows 100% into Finishing
if winner.source == "qobuz":
# Count this Qobuz download against today's budget and set the next randomized spacing.
try:
record_download(conn, random.random())
except Exception as e: # pacing bookkeeping must never fail a good download
print(f"pipeline: qobuz pacing record failed for job {job_id}: {e}", flush=True)
# 5. verify completeness on the staging copy, then promote + tag
# 5. promote + tag the verified winner
_set_state(conn, job_id, "tagging", "tag")
expected = target.track_count if target.track_count is not None else winner.track_count
# Also count the NON-EMPTY audio actually on disk: a silently-skipped track leaves a
# 0-byte placeholder that the adapter still counts, so trust the files when any are
# staged. (`staged and ...` keeps fake adapters that stage nothing on the count path.)
staged = _count_staged_audio(staging)
if result.track_count < expected or (staged and staged < expected):
_fail(conn, job_id, "incomplete download")
return
# Duration check: even with the right track count, a truncated file or a short preview
# substituted for a real track shows up as a total playtime well under MB's expected
# total. Only applies when both are known (measured is None for the offline fakes).
if target.total_duration_s:
measured = measure_duration(staging)
if measured is not None and measured < target.total_duration_s * _DURATION_MIN_RATIO:
_fail(conn, job_id,
f"download too short ({measured / 60:.0f} of ~{target.total_duration_s / 60:.0f} min)")
return
final = album_dir(dest_root, target)
existing_q = _library_quality(conn, target)
new_q = quality_class(winner.quality)
@@ -407,12 +511,7 @@ def run_pipeline(
print(f"pipeline: tagging failed for job {job_id}: {e}", flush=True)
_import(conn, job_id, target, winner, final)
else:
# an existing copy is already at >= this quality — keep it untouched, just complete the request
with conn.cursor() as cur:
cur.execute(
"UPDATE \"Request\" SET status = 'completed' WHERE id = %s",
(_request_id(conn, job_id),),
)
conn.commit()
# an existing copy is already at >= this quality — keep it untouched, just complete
_keep_existing_and_complete(conn, job_id)
finally:
shutil.rmtree(staging, ignore_errors=True) # a partial/failed download never persists
+159
View File
@@ -0,0 +1,159 @@
"""Qobuz pacing ("budget gate").
Keep using Qobuz for hi-res, but at a human-looking volume and cadence so the account isn't
flagged for bulk downloading (which got it 403 USER_BLOCKED). One gate decides, per album,
whether Qobuz is an eligible source right now: within active hours, under today's cap (a warm-up
ramp of a steady cap), and past a randomized spacing since the last Qobuz download. When the gate
is closed the pipeline drops Qobuz candidates and falls through to the next source; monitored
albums upgrade to Qobuz later once budget frees up.
Pure decision logic (`today_cap`, `is_allowed`, `next_gap_seconds`) is separated from the two DB
helpers (`gate_open`, `record_download`) so the policy is unit-tested without a clock or DB.
"""
from dataclasses import dataclass
from datetime import date
_TRUE = {"1", "true", "yes", "on"}
_JITTER = 0.3 # ± fraction around the spread interval
_GAP_FLOOR_S = 60.0
@dataclass(frozen=True)
class QobuzPacing:
enabled: bool = True
active_start: int = 8
active_end: int = 23
daily_cap: int = 40
warmup_start_date: str = "" # ISO "YYYY-MM-DD", or "" for no warm-up ramp
@classmethod
def from_config(cls, config: dict) -> "QobuzPacing":
def _int(key, default):
try:
return int(config[key])
except (KeyError, TypeError, ValueError):
return default
return cls(
enabled=str(config.get("qobuz.pacing.enabled", "true")).strip().lower() in _TRUE,
active_start=_int("qobuz.pacing.activeStartHour", 8),
active_end=_int("qobuz.pacing.activeEndHour", 23),
daily_cap=_int("qobuz.pacing.dailyCap", 40),
warmup_start_date=str(config.get("qobuz.pacing.warmupStartDate", "") or "").strip(),
)
def today_cap(pacing: QobuzPacing, today: date) -> int:
"""Today's Qobuz download cap: the steady cap, ramped up by account age (days since
``warmup_start_date``). No warm-up date -> steady cap. A small steady cap is never exceeded."""
steady = max(0, pacing.daily_cap)
if not pacing.warmup_start_date:
return steady
try:
start = date.fromisoformat(pacing.warmup_start_date)
except ValueError:
return steady
days = max(0, (today - start).days)
if days < 3:
return min(5, steady)
if days < 7:
return min(10, steady)
if days < 14:
return min(20, steady)
return steady
def is_allowed(pacing: QobuzPacing, now_hour: int, count_today: int, cap: int,
now_epoch: float, next_allowed_epoch: float) -> bool:
"""Pure gate decision. Qobuz is eligible only within active hours, under today's cap, and past
the randomized spacing since the last download. Pacing disabled -> always allowed."""
if not pacing.enabled:
return True
if not (pacing.active_start <= now_hour < pacing.active_end):
return False
if count_today >= cap:
return False
if now_epoch < next_allowed_epoch:
return False
return True
def next_gap_seconds(pacing: QobuzPacing, cap: int, rand01: float) -> float:
"""Seconds until the next Qobuz download is allowed: the day's budget spread across the active
window (``activeWindow / cap``), ± _JITTER, floored at _GAP_FLOOR_S. ``rand01`` in [0,1] is the
injected randomness (so tests are deterministic; callers pass random.random())."""
window_s = max(1, pacing.active_end - pacing.active_start) * 3600
base = window_s / max(1, cap)
lo, hi = base * (1 - _JITTER), base * (1 + _JITTER)
gap = lo + (hi - lo) * rand01
return max(_GAP_FLOOR_S, gap)
def _now_parts(conn):
"""(hour:int, today:date, epoch:float) from the DB clock — avoids container-TZ ambiguity."""
with conn.cursor() as cur:
cur.execute("SELECT extract(hour from now())::int, current_date, extract(epoch from now())")
hour, today, epoch = cur.fetchone()
return int(hour), today, float(epoch)
def _read(conn, keys):
with conn.cursor() as cur:
cur.execute('SELECT key, value FROM "Config" WHERE key = ANY(%s)', (list(keys),))
return dict(cur.fetchall())
def gate_open(conn) -> bool:
"""Whether Qobuz may be used for a download right now (reads pacing config + counter/spacing
state from Config and the DB clock). A counter from a previous day is treated as 0."""
from lyra_worker.config import get_config
pacing = QobuzPacing.from_config(get_config(conn))
hour, today, epoch = _now_parts(conn)
state = _read(conn, ("qobuz.pacing.countDay", "qobuz.pacing.countToday", "qobuz.pacing.nextAllowedAt"))
count = 0
if state.get("qobuz.pacing.countDay") == today.isoformat():
try:
count = int(state.get("qobuz.pacing.countToday") or 0)
except (TypeError, ValueError):
count = 0
try:
next_allowed = float(state.get("qobuz.pacing.nextAllowedAt") or 0)
except (TypeError, ValueError):
next_allowed = 0.0
return is_allowed(pacing, hour, count, today_cap(pacing, today), epoch, next_allowed)
def record_download(conn, rand01: float) -> None:
"""Record a successful Qobuz download: bump today's counter (resetting it if the stored day is
stale) and set the randomized next-allowed time. ``rand01`` is injected randomness."""
from lyra_worker.config import get_config
pacing = QobuzPacing.from_config(get_config(conn))
_hour, today, epoch = _now_parts(conn)
state = _read(conn, ("qobuz.pacing.countDay", "qobuz.pacing.countToday"))
if state.get("qobuz.pacing.countDay") == today.isoformat():
try:
count = int(state.get("qobuz.pacing.countToday") or 0)
except (TypeError, ValueError):
count = 0
else:
count = 0
count += 1
next_allowed = epoch + next_gap_seconds(pacing, today_cap(pacing, today), rand01)
_write(conn, {
"qobuz.pacing.countDay": today.isoformat(),
"qobuz.pacing.countToday": str(count),
"qobuz.pacing.nextAllowedAt": repr(next_allowed),
})
def _write(conn, kv: dict) -> None:
with conn.cursor() as cur:
for key, value in kv.items():
cur.execute(
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
(key, value),
)
conn.commit()
+57
View File
@@ -0,0 +1,57 @@
"""A search result with far fewer tracks than the release (e.g. a single-song Soulseek folder
named like the album) can't be the full album — it must be dropped before we waste a download."""
import os
from lyra_worker.claim import claim_next
from lyra_worker.pipeline import run_pipeline
from lyra_worker.types import Candidate, DownloadResult, MBTarget, Quality
from tests.conftest import insert_request
_Q = Quality(fmt="FLAC", lossless=True, bit_depth=24, sample_rate=96000)
class _TwoCandQobuz:
name = "qobuz"
tier = 0
def __init__(self):
self.downloaded = []
def health(self):
return True
def search(self, target):
return [
Candidate(source="qobuz", source_ref="single", matched_artist=target.artist,
matched_album=target.album, quality=_Q, track_count=1, source_tier=0),
Candidate(source="qobuz", source_ref="full", matched_artist=target.artist,
matched_album=target.album, quality=_Q, track_count=12, source_tier=0),
]
def download(self, candidate, dest, on_progress):
self.downloaded.append(candidate.source_ref)
os.makedirs(dest, exist_ok=True)
for i in range(candidate.track_count):
with open(os.path.join(dest, f"{i + 1:02d}.flac"), "wb") as fh:
fh.write(b"a")
return DownloadResult(ok=True, path=dest, track_count=candidate.track_count)
class _Resolver:
def __init__(self, n):
self.n = n
def resolve(self, artist, album):
return MBTarget(artist=artist, album=album, track_count=self.n)
def test_single_song_candidate_is_dropped(conn):
job_id = insert_request(conn, artist="A", album="B")
claim_next(conn)
adapter = _TwoCandQobuz()
run_pipeline(conn, job_id, [adapter], resolver=_Resolver(12), dest_root="/tmp/lib-cf")
assert adapter.downloaded == ["full"] # only the full album was ever attempted
with conn.cursor() as cur:
cur.execute('SELECT "trackCount" FROM "Candidate" WHERE "jobId" = %s', (job_id,))
assert [r[0] for r in cur.fetchall()] == [12] # the single-song folder wasn't even persisted
+125
View File
@@ -0,0 +1,125 @@
"""When the top-ranked source downloads but is incomplete (e.g. Qobuz consistently fails one
track), the pipeline must fall through to the next-ranked source instead of failing the whole
job — only giving up (needs_attention) when every candidate is incomplete."""
import os
from lyra_worker.claim import claim_next
from lyra_worker.pipeline import run_pipeline
from lyra_worker.types import Candidate, DownloadResult, Quality
from tests.conftest import insert_request
_HIRES = Quality(fmt="FLAC", lossless=True, bit_depth=24, sample_rate=96000)
_CD = Quality(fmt="FLAC", lossless=True, bit_depth=16, sample_rate=44100)
def _stage(dest, n):
os.makedirs(dest, exist_ok=True)
for i in range(n):
with open(os.path.join(dest, f"{i + 1:02d}.flac"), "wb") as fh:
fh.write(b"audio")
class _Qobuz:
"""Higher-quality (ranks first) Qobuz that promises 3 tracks but delivers `deliver`."""
name = "qobuz"
tier = 0
def __init__(self, deliver):
self._deliver = deliver
def health(self):
return True
def search(self, target):
return [Candidate(source="qobuz", source_ref="q", matched_artist=target.artist,
matched_album=target.album, quality=_HIRES, track_count=3, source_tier=0)]
def download(self, candidate, dest, on_progress):
_stage(dest, self._deliver)
return DownloadResult(ok=True, path=dest, track_count=self._deliver)
class _Soulseek:
"""Lower-quality (ranks second) Soulseek that promises 3 tracks and delivers `deliver`."""
name = "soulseek"
tier = 1
def __init__(self, deliver):
self._deliver = deliver
def health(self):
return True
def search(self, target):
return [Candidate(source="soulseek", source_ref="s", matched_artist=target.artist,
matched_album=target.album, quality=_CD, track_count=3, source_tier=1)]
def download(self, candidate, dest, on_progress):
_stage(dest, self._deliver)
return DownloadResult(ok=True, path=dest, track_count=self._deliver)
def _state_error(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT state, error FROM "Job" WHERE id = %s', (job_id,))
return cur.fetchone()
def _chosen_source(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT source FROM "Candidate" WHERE "jobId" = %s AND chosen = true', (job_id,))
row = cur.fetchone()
return row[0] if row else None
def test_incomplete_top_source_falls_through_to_complete_next(conn, tmp_path):
# Qobuz (ranks first on quality) delivers only 2 of its promised 3 tracks -> incomplete.
# The pipeline must try the next source (Soulseek, complete) rather than failing the job.
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [_Qobuz(deliver=2), _Soulseek(deliver=3)], dest_root=str(tmp_path))
assert _state_error(conn, job_id) == ("imported", None)
assert _chosen_source(conn, job_id) == "soulseek"
def test_all_sources_incomplete_still_needs_attention(conn, tmp_path):
# Every candidate is incomplete -> no fallback succeeds -> needs_attention (incomplete).
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [_Qobuz(deliver=2), _Soulseek(deliver=1)], dest_root=str(tmp_path))
assert _state_error(conn, job_id) == ("needs_attention", "incomplete download")
assert not (tmp_path / "Radiohead" / "In Rainbows").exists() # nothing imported
class _CountingIncompleteQobuz:
"""Offers `n` candidates that all download but come back incomplete; counts download calls."""
name = "qobuz"
tier = 0
def __init__(self, n):
self.n = n
self.calls = 0
def health(self):
return True
def search(self, target):
q = _HIRES
return [Candidate(source="qobuz", source_ref=f"r{i}", matched_artist=target.artist,
matched_album=target.album, quality=q, track_count=3, source_tier=0)
for i in range(self.n)]
def download(self, candidate, dest, on_progress):
self.calls += 1
_stage(dest, 1) # 1 of 3 → always incomplete
return DownloadResult(ok=True, path=dest, track_count=1)
def test_fall_through_is_capped(conn):
from lyra_worker.pipeline import _MAX_DOWNLOAD_ATTEMPTS
a = _CountingIncompleteQobuz(n=20)
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [a], dest_root="/tmp/lib-cap")
assert _state_error(conn, job_id) == ("needs_attention", "incomplete download")
assert a.calls == _MAX_DOWNLOAD_ATTEMPTS # capped, not all 20 peers tried
+46
View File
@@ -72,6 +72,52 @@ def test_grabbed_past_window_is_fulfilled_not_enqueued(conn):
assert _state(conn, rel_id) == "fulfilled"
def test_owned_by_rgmbid_despite_name_mismatch_is_fulfilled(conn):
# The monitor followed "Kanye West" but the library stores MB's canonical "Ye"; matching on
# rgMbid recognises ownership so the release is fulfilled, not re-pressed.
rel_id = insert_monitored_release(conn, artist_name="Kanye West", album="Graduation",
rg_mbid="rg-grad", monitored=True, state="wanted")
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "LibraryItem" (id, artist, album, path, source, format, '
'"qualityClass", "rgMbid", "importedAt") '
"VALUES (gen_random_uuid()::text, 'Ye', 'Graduation', '/m', 'soulseek', 'FLAC', 2, "
"'rg-grad', now())"
)
conn.commit()
assert enqueue_due(conn, CFG) == 0
assert _state(conn, rel_id) == "fulfilled"
def test_re_enqueue_retires_prior_failed_attempt(conn):
# A hard-to-match release re-pressed each retry interval must not pile up needs_attention
# Request rows; the prior failed attempt is retired when the next one is enqueued.
rel_id = insert_monitored_release(conn, artist_name="X", album="Y", rg_mbid="rg-1",
monitored=True, state="wanted")
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "Request" (id, artist, album, status, "monitoredReleaseId", "createdAt") '
"VALUES (gen_random_uuid()::text, 'X', 'Y', 'needs_attention', %s, now()) RETURNING id",
(rel_id,),
)
req = cur.fetchone()[0]
cur.execute(
'INSERT INTO "Job" (id, "requestId", state, "currentStage", attempts, "createdAt", "updatedAt") '
"VALUES (gen_random_uuid()::text, %s, 'needs_attention', 'rank', 1, now(), now())",
(req,),
)
conn.commit()
assert enqueue_due(conn, CFG) == 1
assert _job_count_for(conn, rel_id) == 1 # old failed attempt retired, one fresh attempt
with conn.cursor() as cur:
cur.execute(
'SELECT j.state FROM "Job" j JOIN "Request" r ON r.id = j."requestId" '
'WHERE r."monitoredReleaseId" = %s',
(rel_id,),
)
assert cur.fetchone()[0] == "requested"
def test_already_in_library_at_cutoff_is_fulfilled(conn):
rel_id = insert_monitored_release(conn, artist_name="A", album="B", rg_mbid="rg-1",
monitored=True, state="wanted")
+50 -4
View File
@@ -22,17 +22,27 @@ def _make_job(conn, rel_id, artist, album, state):
return req, job
def _add_library(conn, req, artist, album, quality):
def _add_library(conn, req, artist, album, quality, rg_mbid=None):
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "LibraryItem" (id, "requestId", artist, album, path, source, format, '
'"qualityClass", "importedAt") '
"VALUES (gen_random_uuid()::text, %s, %s, %s, '/m', 'qobuz', 'FLAC', %s, now())",
(req, artist, album, quality),
'"qualityClass", "rgMbid", "importedAt") '
"VALUES (gen_random_uuid()::text, %s, %s, %s, '/m', 'qobuz', 'FLAC', %s, %s, now())",
(req, artist, album, quality, rg_mbid),
)
conn.commit()
def _needs_attention_count(conn, rel_id):
with conn.cursor() as cur:
cur.execute(
'SELECT count(*) FROM "Request" r JOIN "Job" j ON j."requestId" = r.id '
"WHERE r.\"monitoredReleaseId\" = %s AND j.state = 'needs_attention'",
(rel_id,),
)
return cur.fetchone()[0]
def _release(conn, rel_id):
with conn.cursor() as cur:
cur.execute(
@@ -106,6 +116,42 @@ def test_fulfill_owned_releases_ignores_below_cutoff(conn):
assert _release(conn, rel)[0] == "wanted"
def test_reconcile_matches_by_rgmbid_when_artist_name_canonicalized(conn):
# Real 'Kanye West' -> 'Ye' bug: the MonitoredRelease keeps the followed name, but the
# imported LibraryItem carries MusicBrainz's canonical artist. Matching on rgMbid lets the
# release reconcile to fulfilled instead of looking un-owned and re-pressing forever.
rel = insert_monitored_release(conn, artist_name="Kanye West", album="Graduation",
rg_mbid="rg-grad", monitored=True, state="wanted")
req, job = _make_job(conn, rel, "Kanye West", "Graduation", "imported")
_add_library(conn, req, "Ye", "Graduation", 3, rg_mbid="rg-grad")
reconcile(conn, job, CFG)
assert _release(conn, rel) == ("fulfilled", 3, True)
def test_reconcile_fulfilled_retires_failed_sibling_attempts(conn):
# 'V' by Maroon 5 phantom: earlier failed attempts leave needs_attention Request rows that
# keep showing on the press even after a later attempt imports. A successful import retires
# those failed siblings so the press stops showing an owned album as needing attention.
rel = insert_monitored_release(conn, artist_name="Maroon 5", album="V", rg_mbid="rg-v",
monitored=True, state="wanted")
_make_job(conn, rel, "Maroon 5", "V", "needs_attention")
_make_job(conn, rel, "Maroon 5", "V", "needs_attention")
req, job = _make_job(conn, rel, "Maroon 5", "V", "imported")
_add_library(conn, req, "Maroon 5", "V", 2)
reconcile(conn, job, CFG)
assert _release(conn, rel)[0] == "fulfilled"
assert _needs_attention_count(conn, rel) == 0 # failed siblings retired
def test_fulfill_owned_releases_matches_by_rgmbid(conn):
# Ne-Yo -> NeYo (typographic hyphen) name drift: still owned by rgMbid.
rel = insert_monitored_release(conn, artist_name="Ne-Yo", album="Because of You",
rg_mbid="rg-ney", monitored=True, state="grabbed")
_add_library(conn, None, "NeYo", "Because of You", 2, rg_mbid="rg-ney")
assert fulfill_owned_releases(conn, MonitorConfig(quality_cutoff=2)) == 1
assert _release(conn, rel) == ("fulfilled", 2, True)
def test_unlinked_job_is_ignored(conn):
with conn.cursor() as cur:
cur.execute(
+113 -1
View File
@@ -1,4 +1,7 @@
from lyra_worker._musicbrainz import _best_release_group
import musicbrainzngs
import pytest
from lyra_worker._musicbrainz import _best_release_group, _pick_release, _with_retry
# Offline unit tests for the pure release-group selection logic. The live resolve()
# path (network) is covered by test_musicbrainz_live.py.
@@ -48,6 +51,21 @@ def test_empty_groups_returns_none():
assert _best_release_group("Michael Jackson", "Off the Wall", []) is None
def test_prefers_original_group_over_later_reissue():
# Real "Hybrid Theory" bug: MB lists a 2000 original and a 2023 reissue as separate same-titled
# Album groups (tie on artist/title/type). The reissue's longer tracklist fails completeness
# checks, so the original (earliest) must win regardless of MB's relevance order.
groups = [
{"id": "reissue", "title": "Hybrid Theory", "primary-type": "Album",
"first-release-date": "2023-07-21",
"artist-credit": [{"artist": {"name": "Linkin Park"}}]},
{"id": "original", "title": "Hybrid Theory", "primary-type": "Album",
"first-release-date": "2000-10-24",
"artist-credit": [{"artist": {"name": "Linkin Park"}}]},
]
assert _best_release_group("Linkin Park", "Hybrid Theory", groups)["id"] == "original"
def test_prefers_credited_artist_over_same_titled_other_artist_album():
# Real-world "Fun Machine" bug: Lake Street Dive's EP is what we followed, but two
# unrelated bands also have a release group literally titled "Fun Machine" as an Album.
@@ -60,3 +78,97 @@ def test_prefers_credited_artist_over_same_titled_other_artist_album():
]
best = _best_release_group("Lake Street Dive", "Fun Machine", groups)
assert best["artist-credit"][0]["artist"]["name"] == "Lake Street Dive"
# --- _with_retry: transient MusicBrainz failures (SSL EOF, 503 rate-limit) ---
def test_pick_release_prefers_original_official_over_later_reissue():
# Real "Hybrid Theory" bug: MB's arbitrary releases[0] was a 2023 13-track reissue; the 2000
# original is what sources deliver, so completeness checks must reference it.
releases = [
{"id": "reissue", "status": "Official", "date": "2023-06-01"},
{"id": "original", "status": "Official", "date": "2000-10-24"},
{"id": "deluxe", "status": "Official", "date": "2020-10-09"},
]
assert _pick_release(releases)["id"] == "original"
def test_pick_release_skips_non_official_and_undated():
releases = [
{"id": "promo", "status": "Promotion", "date": "1999-01-01"}, # earliest but not official
{"id": "undated", "status": "Official", "date": ""},
{"id": "official", "status": "Official", "date": "2001-03-03"},
]
assert _pick_release(releases)["id"] == "official"
def test_pick_release_empty_is_none():
assert _pick_release([]) is None
def test_retry_returns_value_without_backoff_when_first_call_succeeds():
sleeps: list[float] = []
calls = {"n": 0}
def fn():
calls["n"] += 1
return "ok"
assert _with_retry(fn, sleep=sleeps.append) == "ok"
assert calls["n"] == 1
assert sleeps == [] # no backoff when nothing failed
def test_retry_recovers_from_transient_network_error():
# The real symptom: <urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING]> arrives
# as musicbrainzngs.NetworkError. A couple of retries clears it.
sleeps: list[float] = []
calls = {"n": 0}
def fn():
calls["n"] += 1
if calls["n"] < 3:
raise musicbrainzngs.NetworkError("SSL: UNEXPECTED_EOF_WHILE_READING")
return "recovered"
assert _with_retry(fn, sleep=sleeps.append) == "recovered"
assert calls["n"] == 3
assert sleeps == [1.0, 2.0] # exponential backoff before each retry
def test_retry_recovers_from_transient_response_error():
# MusicBrainz's own 503 rate-limit surfaces as ResponseError.
calls = {"n": 0}
def fn():
calls["n"] += 1
if calls["n"] < 2:
raise musicbrainzngs.ResponseError("503 Service Unavailable")
return "ok"
assert _with_retry(fn, sleep=lambda _: None) == "ok"
assert calls["n"] == 2
def test_retry_reraises_after_exhausting_attempts():
sleeps: list[float] = []
def fn():
raise musicbrainzngs.NetworkError("still down")
with pytest.raises(musicbrainzngs.NetworkError):
_with_retry(fn, attempts=4, sleep=sleeps.append)
assert sleeps == [1.0, 2.0, 4.0] # 3 backoffs across 4 attempts
def test_retry_does_not_swallow_non_transient_errors():
calls = {"n": 0}
def fn():
calls["n"] += 1
raise ValueError("bug, not a network blip")
with pytest.raises(ValueError):
_with_retry(fn, sleep=lambda _: None)
assert calls["n"] == 1 # non-transient errors propagate immediately, no retry
+179
View File
@@ -0,0 +1,179 @@
from datetime import date
from lyra_worker.qobuz_gate import (
QobuzPacing,
today_cap,
is_allowed,
next_gap_seconds,
gate_open,
record_download,
)
# ---- QobuzPacing.from_config -------------------------------------------------
def test_from_config_defaults():
p = QobuzPacing.from_config({})
assert p.enabled is True
assert p.active_start == 8 and p.active_end == 23
assert p.daily_cap == 40
assert p.warmup_start_date == ""
def test_from_config_reads_values():
p = QobuzPacing.from_config({
"qobuz.pacing.enabled": "false",
"qobuz.pacing.activeStartHour": "9",
"qobuz.pacing.activeEndHour": "22",
"qobuz.pacing.dailyCap": "25",
"qobuz.pacing.warmupStartDate": "2026-07-15",
})
assert p.enabled is False
assert p.active_start == 9 and p.active_end == 22
assert p.daily_cap == 25
assert p.warmup_start_date == "2026-07-15"
# ---- today_cap (warm-up ramp) ------------------------------------------------
def test_today_cap_no_warmup_uses_steady():
p = QobuzPacing(daily_cap=40, warmup_start_date="")
assert today_cap(p, date(2026, 7, 20)) == 40
def test_today_cap_ramp_boundaries():
p = QobuzPacing(daily_cap=40, warmup_start_date="2026-07-15")
assert today_cap(p, date(2026, 7, 15)) == 5 # day 0
assert today_cap(p, date(2026, 7, 17)) == 5 # day 2
assert today_cap(p, date(2026, 7, 18)) == 10 # day 3
assert today_cap(p, date(2026, 7, 21)) == 10 # day 6
assert today_cap(p, date(2026, 7, 22)) == 20 # day 7
assert today_cap(p, date(2026, 7, 28)) == 20 # day 13
assert today_cap(p, date(2026, 7, 29)) == 40 # day 14
assert today_cap(p, date(2026, 8, 30)) == 40 # long after
def test_today_cap_steady_below_ramp_step_is_the_ceiling():
# A small steady cap is never exceeded by a ramp step.
p = QobuzPacing(daily_cap=3, warmup_start_date="2026-07-15")
assert today_cap(p, date(2026, 7, 29)) == 3
# ---- is_allowed (the gate decision) -----------------------------------------
_P = QobuzPacing(enabled=True, active_start=8, active_end=23, daily_cap=40)
def test_disabled_always_allows():
p = QobuzPacing(enabled=False, active_start=8, active_end=23)
assert is_allowed(p, now_hour=3, count_today=999, cap=0, now_epoch=0, next_allowed_epoch=1e18) is True
def test_blocked_outside_active_hours():
assert is_allowed(_P, now_hour=7, count_today=0, cap=40, now_epoch=100, next_allowed_epoch=0) is False
assert is_allowed(_P, now_hour=23, count_today=0, cap=40, now_epoch=100, next_allowed_epoch=0) is False
def test_blocked_at_or_over_cap():
assert is_allowed(_P, now_hour=10, count_today=40, cap=40, now_epoch=100, next_allowed_epoch=0) is False
assert is_allowed(_P, now_hour=10, count_today=41, cap=40, now_epoch=100, next_allowed_epoch=0) is False
def test_blocked_before_next_allowed():
assert is_allowed(_P, now_hour=10, count_today=0, cap=40, now_epoch=100, next_allowed_epoch=200) is False
def test_allowed_when_all_conditions_met():
assert is_allowed(_P, now_hour=10, count_today=5, cap=40, now_epoch=300, next_allowed_epoch=200) is True
# ---- next_gap_seconds (spread + jitter) -------------------------------------
def test_next_gap_spreads_cap_across_active_window():
# 15h window (8..23) / cap 40 = 1350s base; ±30%; midpoint rand -> base.
p = QobuzPacing(active_start=8, active_end=23, daily_cap=40)
assert abs(next_gap_seconds(p, cap=40, rand01=0.5) - 1350.0) < 1.0
assert abs(next_gap_seconds(p, cap=40, rand01=0.0) - 1350.0 * 0.7) < 1.0 # low end
assert abs(next_gap_seconds(p, cap=40, rand01=1.0) - 1350.0 * 1.3) < 1.0 # high end
def test_next_gap_has_a_floor():
# A huge cap would give a tiny gap; a 60s floor protects the request rate.
p = QobuzPacing(active_start=8, active_end=23, daily_cap=100000)
assert next_gap_seconds(p, cap=100000, rand01=0.0) == 60.0
# ---- DB helpers: gate_open / record_download --------------------------------
def _set(conn, key, value):
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
(key, value),
)
conn.commit()
def _get(conn, key):
with conn.cursor() as cur:
cur.execute('SELECT value FROM "Config" WHERE key=%s', (key,))
row = cur.fetchone()
return row[0] if row else None
def test_record_download_increments_and_sets_next_allowed(conn):
_set(conn, "qobuz.pacing.dailyCap", "40")
record_download(conn, rand01=0.5)
assert _get(conn, "qobuz.pacing.countToday") == "1"
assert _get(conn, "qobuz.pacing.countDay") is not None
assert _get(conn, "qobuz.pacing.nextAllowedAt") is not None
record_download(conn, rand01=0.5)
assert _get(conn, "qobuz.pacing.countToday") == "2"
def test_record_download_resets_counter_on_new_day(conn):
# A stale count from a previous day must reset to 1, not accumulate.
_set(conn, "qobuz.pacing.dailyCap", "40")
_set(conn, "qobuz.pacing.countDay", "2000-01-01")
_set(conn, "qobuz.pacing.countToday", "39")
record_download(conn, rand01=0.5)
assert _get(conn, "qobuz.pacing.countToday") == "1"
def test_gate_open_with_always_open_window(conn):
# An all-day window (0..24) removes the time-of-day dependency: no prior downloads and no
# next-allowed set → the gate is open regardless of when the test runs.
_set(conn, "qobuz.pacing.activeStartHour", "0")
_set(conn, "qobuz.pacing.activeEndHour", "24")
assert gate_open(conn) is True
def test_gate_closed_by_empty_active_window(conn):
# start == end is an empty window (h >= 5 AND h < 5 is never true) → always closed.
_set(conn, "qobuz.pacing.activeStartHour", "5")
_set(conn, "qobuz.pacing.activeEndHour", "5")
assert gate_open(conn) is False
def test_gate_closed_when_cap_reached_today(conn):
_set(conn, "qobuz.pacing.activeStartHour", "0")
_set(conn, "qobuz.pacing.activeEndHour", "24")
_set(conn, "qobuz.pacing.dailyCap", "3")
# Simulate 3 downloads already today.
with conn.cursor() as cur:
cur.execute("SELECT current_date::text")
today = cur.fetchone()[0]
_set(conn, "qobuz.pacing.countDay", today)
_set(conn, "qobuz.pacing.countToday", "3")
assert gate_open(conn) is False
def test_gate_ignores_stale_count_from_a_previous_day(conn):
# A high count from yesterday must not keep the gate closed today.
_set(conn, "qobuz.pacing.activeStartHour", "0")
_set(conn, "qobuz.pacing.activeEndHour", "24")
_set(conn, "qobuz.pacing.dailyCap", "3")
_set(conn, "qobuz.pacing.countDay", "2000-01-01")
_set(conn, "qobuz.pacing.countToday", "99")
assert gate_open(conn) is True
+85
View File
@@ -0,0 +1,85 @@
"""The Qobuz budget gate, wired into the pipeline: when the gate is closed the job falls through
to another source; when open it uses Qobuz and records the download; a Qobuz-only setup is never
gated (nothing to fall through to)."""
from lyra_worker.adapters.fakes import FakeQobuz, FakeSoulseek
from lyra_worker.claim import claim_next
from lyra_worker.pipeline import run_pipeline
from tests.conftest import insert_request
def _cfg(conn, key, value):
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
(key, value),
)
conn.commit()
def _open_gate(conn):
_cfg(conn, "qobuz.pacing.activeStartHour", "0")
_cfg(conn, "qobuz.pacing.activeEndHour", "24")
def _closed_gate(conn):
_cfg(conn, "qobuz.pacing.activeStartHour", "5") # empty window (5..5) → always closed
_cfg(conn, "qobuz.pacing.activeEndHour", "5")
def _state(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT state FROM "Job" WHERE id = %s', (job_id,))
return cur.fetchone()[0]
def _chosen(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT source FROM "Candidate" WHERE "jobId" = %s AND chosen = true', (job_id,))
row = cur.fetchone()
return row[0] if row else None
def _cfg_get(conn, key):
with conn.cursor() as cur:
cur.execute('SELECT value FROM "Config" WHERE key = %s', (key,))
row = cur.fetchone()
return row[0] if row else None
def test_closed_gate_drops_qobuz_and_falls_through(conn):
_closed_gate(conn)
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [FakeQobuz(), FakeSoulseek()], dest_root="/tmp/lib-gate")
assert _state(conn, job_id) == "imported"
assert _chosen(conn, job_id) == "soulseek" # Qobuz dropped by the closed gate
def test_open_gate_uses_qobuz_and_records_the_download(conn):
_open_gate(conn)
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [FakeQobuz(), FakeSoulseek()], dest_root="/tmp/lib-gate")
assert _state(conn, job_id) == "imported"
assert _chosen(conn, job_id) == "qobuz"
assert _cfg_get(conn, "qobuz.pacing.countToday") == "1" # recorded against today's budget
assert _cfg_get(conn, "qobuz.pacing.nextAllowedAt") is not None
def test_qobuz_only_setup_is_never_gated(conn):
_closed_gate(conn)
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [FakeQobuz()], dest_root="/tmp/lib-gate")
assert _state(conn, job_id) == "imported"
assert _chosen(conn, job_id) == "qobuz" # not dropped — nothing to fall through to
def test_non_qobuz_download_does_not_touch_the_counter(conn):
_closed_gate(conn) # forces Soulseek
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
claim_next(conn)
run_pipeline(conn, job_id, [FakeQobuz(), FakeSoulseek()], dest_root="/tmp/lib-gate")
assert _chosen(conn, job_id) == "soulseek"
assert _cfg_get(conn, "qobuz.pacing.countToday") is None # only Qobuz downloads count
+69
View File
@@ -0,0 +1,69 @@
from lyra_worker.claim import claim_next, requeue_or_fail
from tests.conftest import insert_request
def _set_attempts(conn, job_id, attempts, state="matching", stage="match"):
with conn.cursor() as cur:
cur.execute(
'UPDATE "Job" SET attempts = %s, state = %s, "currentStage" = %s, "claimedAt" = now() '
'WHERE id = %s',
(attempts, state, stage, job_id),
)
conn.commit()
def _job(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT state, "currentStage", "claimedAt", error FROM "Job" WHERE id = %s', (job_id,))
return cur.fetchone()
def _request_status(conn, job_id):
with conn.cursor() as cur:
cur.execute(
'SELECT status FROM "Request" WHERE id = (SELECT "requestId" FROM "Job" WHERE id = %s)',
(job_id,),
)
return cur.fetchone()[0]
def test_below_cap_requeues_to_requested(conn):
# A transient failure (e.g. MB SSL EOF) mid-'matching' must not strand the job — it returns
# to 'requested' so claim_next picks it up again instead of orphaning it.
jid = insert_request(conn)
_set_attempts(conn, jid, 2)
requeue_or_fail(conn, jid, "SSL: UNEXPECTED_EOF_WHILE_READING")
st, stage, claimed_at, err = _job(conn, jid)
assert st == "requested"
assert stage == "intake"
assert claimed_at is None
assert err is None
assert _request_status(conn, jid) == "pending"
assert claim_next(conn) == jid # immediately re-claimable
def test_at_cap_marks_needs_attention_with_error(conn):
jid = insert_request(conn)
_set_attempts(conn, jid, 5) # hit the cap
requeue_or_fail(conn, jid, "SSL: UNEXPECTED_EOF_WHILE_READING")
st, _stage, claimed_at, err = _job(conn, jid)
assert st == "needs_attention"
assert err == "SSL: UNEXPECTED_EOF_WHILE_READING"
assert claimed_at is None
assert _request_status(conn, jid) == "needs_attention"
assert claim_next(conn) is None # not re-claimed
def test_requeues_until_cap_then_fails(conn):
jid = insert_request(conn)
for attempts in (1, 2, 3, 4):
_set_attempts(conn, jid, attempts)
requeue_or_fail(conn, jid, "boom")
assert _job(conn, jid)[0] == "requested"
_set_attempts(conn, jid, 5)
requeue_or_fail(conn, jid, "boom")
assert _job(conn, jid)[0] == "needs_attention"
def test_missing_job_is_noop(conn):
requeue_or_fail(conn, "does-not-exist", "boom") # must not raise
+218
View File
@@ -0,0 +1,218 @@
"""slskd download behaviour: abandoned transfers are cancelled (no pile-up), a stalled peer is
dropped fast, a slow-but-progressing peer is kept, and search candidates are ranked fast/free
peers first."""
import json
import pytest
import lyra_worker.adapters._slskd as slskd_mod
from lyra_worker.adapters._slskd import (
_ETA_CAP_SECONDS,
_MAX_XFER_POLLS,
_MIN_XFER_POLLS,
_POLL_SECONDS,
SlskdClient,
_eta_seconds,
_max_polls,
_parse_search_responses,
)
class _Resp:
def __init__(self, payload=None):
self._p = payload if payload is not None else {}
def raise_for_status(self):
pass
def json(self):
return self._p
class _SeqRequests:
"""post OK; get() returns successive poll payloads (last one repeats); records DELETEs."""
def __init__(self, polls):
self._polls = list(polls)
self._i = 0
self.deleted = []
def post(self, url, **kw):
return _Resp({})
def get(self, url, **kw):
p = self._polls[min(self._i, len(self._polls) - 1)]
self._i += 1
return _Resp(p)
def delete(self, url, **kw):
self.deleted.append(url)
return _Resp({})
def _poll(state, bytes_transferred, fid="t1"):
return {"directories": [{"files": [
{"filename": "f1.flac", "size": 100, "state": state,
"bytesTransferred": bytes_transferred, "id": fid},
]}]}
def _client(tmp_path):
return SlskdClient({"slskd.url": "http://x", "slskd.api_key": "k",
"slskd.downloadsDir": str(tmp_path)})
_REF = json.dumps({"username": "peerA", "files": [{"filename": "f1.flac", "size": 100}]})
def test_stalled_peer_is_abandoned_and_cancelled(tmp_path, monkeypatch):
# No byte progress across polls → stall → TimeoutError → the transfer is cancelled.
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests([_poll("InProgress", 0)]))
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
monkeypatch.setattr(slskd_mod, "_STALL_POLLS", 3)
fake = slskd_mod.requests
with pytest.raises(TimeoutError):
_client(tmp_path).download(_REF, str(tmp_path / "dest"), lambda *_a: None)
assert any(u.endswith("/api/v0/transfers/downloads/peerA/t1") for u in fake.deleted)
def test_error_state_is_cancelled(tmp_path, monkeypatch):
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests([_poll("Completed, Errored", 0)]))
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
fake = slskd_mod.requests
with pytest.raises(RuntimeError):
_client(tmp_path).download(_REF, str(tmp_path / "dest"), lambda *_a: None)
assert any(u.endswith("/peerA/t1") for u in fake.deleted)
def test_slow_but_progressing_peer_is_not_abandoned(tmp_path, monkeypatch):
# Bytes climb for more polls than _STALL_POLLS, then complete → success, not a premature stall.
(tmp_path / "01.flac").write_bytes(b"audio") # so _retrieve finds the finished file
ref = json.dumps({"username": "peerA", "files": [{"filename": r"d\Album\01.flac", "size": 5}]})
polls = [
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 5,
"state": "InProgress", "bytesTransferred": b, "id": "t1"}]}]}
for b in (10, 20, 30, 40, 50)
] + [
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 5,
"state": "Completed, Succeeded", "bytesTransferred": 5, "id": "t1"}]}]}
]
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests(polls))
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
monkeypatch.setattr(slskd_mod, "_STALL_POLLS", 2) # would fire early if progress were ignored
out = _client(tmp_path).download(ref, str(tmp_path / "dest"), lambda *_a: None)
assert out["track_count"] == 1
assert slskd_mod.requests.deleted == [] # completed cleanly → nothing cancelled
def test_parse_ranks_free_and_fast_peers_first():
responses = [
{"username": "slow", "hasFreeUploadSlot": True, "uploadSpeed": 100, "queueLength": 0,
"files": [{"filename": r"a\Artist - Album\01.flac", "size": 5}]},
{"username": "fast", "hasFreeUploadSlot": True, "uploadSpeed": 9000, "queueLength": 0,
"files": [{"filename": r"b\Artist - Album\01.flac", "size": 5}]},
{"username": "queued", "hasFreeUploadSlot": False, "uploadSpeed": 99999, "queueLength": 40,
"files": [{"filename": r"c\Artist - Album\01.flac", "size": 5}]},
]
peers = [json.loads(c["source_ref"])["username"] for c in _parse_search_responses(responses)]
assert peers == ["fast", "slow", "queued"] # free+fast first; no-slot peer last despite speed
def test_parse_uses_requested_artist_when_present_in_path():
# Real "Rated R" MP3-over-FLAC bug: the artist is in a PARENT folder ("Rihanna\(2009) Rated
# R\…"), so the "Artist - Album" folder parse yields no artist and confidence tanks. When the
# requested artist is in the path, trust it so the candidate scores as Rihanna.
responses = [
{"username": "p", "hasFreeUploadSlot": True, "uploadSpeed": 1000, "queueLength": 0,
"files": [{"filename": r"@@x\Music\Rihanna\(2009) Rated R\01. Mad House.flac", "size": 5}]},
]
c = _parse_search_responses(responses, artist="Rihanna")[0]
assert c["artist"] == "Rihanna"
def test_parse_without_artist_falls_back_to_folder_parse():
responses = [
{"username": "p", "hasFreeUploadSlot": True, "uploadSpeed": 1000, "queueLength": 0,
"files": [{"filename": r"x\Some Band - Some Album\01.flac", "size": 5}]},
]
c = _parse_search_responses(responses)[0] # no artist arg → folder parse
assert c["artist"] == "Some Band"
def test_parse_prefers_shorter_transfer_over_raw_speed():
# The Hybrid Theory case: a peer advertising higher speed but serving a ~2x-larger rip (24-bit
# / bloated FLAC) should lose to a peer with lower speed but a smaller standard rip that
# finishes sooner — both are the same quality class to Lyra, so faster-to-finish wins.
responses = [
{"username": "bloated", "hasFreeUploadSlot": True, "uploadSpeed": 20000, "queueLength": 0,
"files": [{"filename": rf"x\LP - Album\{i:02}.flac", "size": 40_000_000} for i in range(12)]},
{"username": "lean", "hasFreeUploadSlot": True, "uploadSpeed": 15000, "queueLength": 0,
"files": [{"filename": rf"y\LP - Album\{i:02}.flac", "size": 24_000_000} for i in range(12)]},
]
# bloated est = 480MB/20000 = 24000s; lean est = 288MB/15000 = 19200s → lean finishes first
peers = [json.loads(c["source_ref"])["username"] for c in _parse_search_responses(responses)]
assert peers[0] == "lean"
def test_eta_seconds_from_measured_rate():
assert _eta_seconds(1000, 250, 250.0) == 3 # 750 bytes left / 250 B/s
assert _eta_seconds(1000, 0, 100.0) == 10
def test_eta_seconds_unknown_cases():
assert _eta_seconds(1000, 250, 0.0) is None # no speed sample yet
assert _eta_seconds(0, 0, 100.0) is None # unknown total size
assert _eta_seconds(1000, 1000, 100.0) is None # already complete
def test_max_polls_scales_backstop_to_album_size():
# A 521MB album (Drake "Scorpion", 25 FLACs) from a slow serial peer must get far more than
# the old flat 1200-poll/60min budget, or it gets abandoned mid-download and restarts from zero.
scorpion_bytes = 521_000_000
polls = _max_polls(scorpion_bytes)
assert polls > 1200 # more headroom than the old flat cap
assert polls * _POLL_SECONDS >= 90 * 60 # at least ~90 min of wall-clock budget
# A tiny single stays on the floor, not zero.
assert _max_polls(3_000_000) == _MIN_XFER_POLLS
assert _max_polls(0) == _MIN_XFER_POLLS
# An enormous set is clamped to the ceiling (bounds worst-case slot hold).
assert _max_polls(50_000_000_000) == _MAX_XFER_POLLS
# Monotonic: bigger album never gets a smaller budget.
assert _max_polls(200_000_000) <= _max_polls(400_000_000)
def test_eta_seconds_capped_when_peer_nearly_stalls():
# A near-stalled peer (tiny speed) would yield an ETA far past int4, overflowing the
# downloadEtaSeconds column write. The estimate must be clamped to a safe ceiling.
eta = _eta_seconds(30_000_000, 0, 0.001) # 30MB / 0.001 B/s ~= 3e10s uncapped
assert eta == _ETA_CAP_SECONDS
assert eta < 2_147_483_647 # fits in int4
def test_download_reports_shrinking_eta(tmp_path, monkeypatch):
# A 900-byte file arriving 300 bytes per 3s poll → measured speed 100 B/s → ETA counts down
# (6s, 3s) then None on completion. on_progress gets (fraction, eta_seconds).
(tmp_path / "01.flac").write_bytes(b"x" * 900) # so _retrieve finds the finished file
ref = json.dumps({"username": "peerA", "files": [{"filename": r"d\Album\01.flac", "size": 900}]})
polls = [
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 900,
"state": "InProgress", "bytesTransferred": b, "id": "t1"}]}]}
for b in (300, 600)
] + [
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 900,
"state": "Completed, Succeeded", "bytesTransferred": 900, "id": "t1"}]}]}
]
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests(polls))
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
clock = iter(float(i) * 3 for i in range(100)) # 0,3,6,… deterministic monotonic time
monkeypatch.setattr(slskd_mod.time, "monotonic", lambda: next(clock))
calls: list = []
_client(tmp_path).download(ref, str(tmp_path / "dest"), lambda *a: calls.append(a))
etas = [a[1] for a in calls if len(a) > 1]
assert etas == [6, 3, None] # counts down at 100 B/s, then None once complete
+150
View File
@@ -0,0 +1,150 @@
"""Soulseek search fallback: the Soulseek server drops searches containing blocklisted terms
(major-label artists / album titles) → 0 responses. search_album retries with album-only then
artist-only, keeping only path-matching results so it never grabs a same-titled foreign album."""
import json
import lyra_worker.adapters._slskd as slskd_mod
from lyra_worker.adapters._slskd import (
SlskdClient,
_keep_matching,
_norm,
_path_matches,
)
# --- pure helpers -----------------------------------------------------------------------------
def test_norm_strips_diacritics_and_punctuation():
assert _norm("Beyoncé") == "beyonce"
assert _norm("I Am… Sasha Fierce") == "i am sasha fierce"
assert _norm("AC/DC") == "ac dc"
def test_path_matches_is_accent_and_punctuation_tolerant():
p = r"@@bckfj\All\Beyonce\I Am... Sasha Fierce CD1\03 Halo.flac"
assert _path_matches("Beyoncé", p) # accent folded
assert _path_matches("I Am… Sasha Fierce", p) # ellipsis / punctuation ignored
assert not _path_matches("Rihanna", p)
def test_path_matches_short_needle_requires_whole_string():
assert _path_matches("U2", r"x\U2 - Achtung Baby\01.flac")
assert not _path_matches("U2", r"x\Blur - 13\01.flac")
def test_keep_matching_drops_foreign_and_empty_responses():
responses = [
{"username": "a", "files": [{"filename": r"x\Rihanna - Unapologetic\01.flac"}]},
{"username": "b", "files": [{"filename": r"x\Someone Else - Unapologetic\01.flac"}]},
]
kept = _keep_matching(responses, "Rihanna")
assert [r["username"] for r in kept] == ["a"]
# --- fallback orchestration (HTTP faked) ------------------------------------------------------
class _Resp:
def __init__(self, payload):
self._p = payload
def raise_for_status(self):
pass
def json(self):
return self._p
class _FakeSearch:
"""Routes slskd search HTTP by URL: POST returns an id, GET on the search returns a completed
state, GET on /responses returns the canned responses for that query. Records queries sent."""
def __init__(self, by_query):
self.by_query = by_query # {searchText: [responses]}
self.queries = [] # searchText values POSTed, in order
self._id_query = {}
self._n = 0
def post(self, url, **kw):
text = kw["json"]["searchText"]
self.queries.append(text)
self._n += 1
sid = f"s{self._n}"
self._id_query[sid] = text
return _Resp({"id": sid})
def get(self, url, **kw):
if url.endswith("/responses"):
sid = url.rsplit("/", 2)[-2]
return _Resp(self.by_query.get(self._id_query[sid], []))
return _Resp({"state": "Completed, TimedOut"})
def _client():
return SlskdClient({"slskd.url": "http://x", "slskd.api_key": "k", "slskd.downloadsDir": "/x"})
def _install(monkeypatch, fake):
monkeypatch.setattr(slskd_mod, "requests", fake)
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
def test_primary_hit_does_not_trigger_fallback(monkeypatch):
fake = _FakeSearch({
"Rihanna Unapologetic": [
{"username": "a", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
"files": [{"filename": r"x\Rihanna - Unapologetic\01.flac", "size": 5}]},
],
})
_install(monkeypatch, fake)
out = _client().search_album("Rihanna", "Unapologetic")
assert len(out) == 1
assert fake.queries == ["Rihanna Unapologetic"] # no fallback searches
def test_album_only_fallback_recovers_blocklisted_artist(monkeypatch):
# Primary (artist blocked) → 0; album-only returns the artist's folder → recovered.
fake = _FakeSearch({
"Beyoncé I Am… Sasha Fierce": [], # blocked
"i am sasha fierce": [
{"username": "a", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
"files": [{"filename": r"x\Beyonce - I Am... Sasha Fierce\01.flac", "size": 5}]},
{"username": "b", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
"files": [{"filename": r"x\Other Artist - I Am Sasha Fierce\01.flac", "size": 5}]},
],
})
_install(monkeypatch, fake)
out = _client().search_album("Beyoncé", "I Am… Sasha Fierce")
assert fake.queries == ["Beyoncé I Am… Sasha Fierce", "i am sasha fierce"]
peers = [json.loads(c["source_ref"])["username"] for c in out]
assert peers == ["a"] # foreign same-titled album dropped
def test_artist_only_fallback_when_album_title_blocked(monkeypatch):
# Album-only also blocked/empty → fall through to artist-only, filtered by album in path.
fake = _FakeSearch({
"Foo Lemonade": [],
"lemonade": [], # album title itself blocked
"foo": [
{"username": "a", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
"files": [{"filename": r"x\Foo - Lemonade\01.flac", "size": 5}]},
],
})
_install(monkeypatch, fake)
out = _client().search_album("Foo", "Lemonade")
assert fake.queries == ["Foo Lemonade", "lemonade", "foo"]
assert len(out) == 1
def test_fallback_yields_nothing_when_title_too_generic(monkeypatch):
# 'Adele 21': artist blocked, album title '21' returns lots of non-Adele content → filtered
# to nothing rather than grabbing a wrong album.
fake = _FakeSearch({
"Adele 21": [],
"21": [
{"username": "z", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
"files": [{"filename": r"x\21 Savage - Savage Mode\01.flac", "size": 5}]},
],
"adele": [], # artist blocked
})
_install(monkeypatch, fake)
out = _client().search_album("Adele", "21")
assert out == []
@@ -0,0 +1,101 @@
"""MusicBrainz picks an arbitrary releases[0] that is often a deluxe/expanded edition with more
tracks (and a longer runtime) than the standard album a source legitimately delivers. Completeness
and the duration guard must be judged against what the CHOSEN SOURCE offered, not MB's canonical
edition — otherwise a complete standard-edition download is falsely rejected as 'incomplete'
(The Script "No Sound Without Silence": 11 delivered vs MB 12) or 'too short' (Skillet "Unleashed":
12 delivered ~40 min vs MB's 20-track deluxe ~73 min). A genuinely short download must still fail.
"""
import os
from dataclasses import replace
from lyra_worker.adapters.fakes import FakeQobuz
from lyra_worker.claim import claim_next
from lyra_worker.pipeline import run_pipeline
from lyra_worker.types import DownloadResult, MBTarget
from tests.conftest import insert_request
class _StandardEditionQobuz(FakeQobuz):
"""Qobuz-like source offering a fixed-size *standard* edition (source_tracks), independent of
MB's possibly-larger deluxe count; stages that many real (non-empty) files and reports them."""
def __init__(self, source_tracks):
super().__init__()
self._n = source_tracks
def search(self, target):
return super().search(replace(target, track_count=self._n))
def download(self, candidate, dest, on_progress):
os.makedirs(dest, exist_ok=True)
for i in range(candidate.track_count):
with open(os.path.join(dest, f"{i + 1:02d} Track.flac"), "wb") as fh:
fh.write(b"audio")
on_progress(1.0)
return DownloadResult(ok=True, path=dest, track_count=candidate.track_count)
class _PartialQobuz(FakeQobuz):
"""Promises `promised` tracks but one fails mid-download: stages and reports `promised - 1`
(the real John Mayer "Roll it on Home" IncompleteRead case)."""
def __init__(self, promised):
super().__init__()
self._n = promised
def search(self, target):
return super().search(replace(target, track_count=self._n))
def download(self, candidate, dest, on_progress):
os.makedirs(dest, exist_ok=True)
delivered = candidate.track_count - 1
for i in range(delivered):
with open(os.path.join(dest, f"{i + 1:02d} Track.flac"), "wb") as fh:
fh.write(b"audio")
return DownloadResult(ok=True, path=dest, track_count=delivered)
class _DeluxeResolver:
"""Resolver whose canonical release is a larger (deluxe) edition than the source delivers."""
def __init__(self, mb_tracks, mb_total_s):
self._t, self._s = mb_tracks, mb_total_s
def resolve(self, artist, album):
return MBTarget(artist=artist, album=album, track_count=self._t, total_duration_s=self._s)
def _state_error(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT state, error FROM "Job" WHERE id = %s', (job_id,))
return cur.fetchone()
def test_standard_edition_not_rejected_as_incomplete(conn):
# The Script: source delivers the 11-track standard edition; MB's canonical release is 12.
job_id = insert_request(conn, artist="The Script", album="No Sound Without Silence")
claim_next(conn)
run_pipeline(conn, job_id, [_StandardEditionQobuz(source_tracks=11)],
resolver=_DeluxeResolver(mb_tracks=12, mb_total_s=2943),
measure_duration=lambda _s: 2700.0, dest_root="/tmp/lib-se")
assert _state_error(conn, job_id) == ("imported", None)
def test_smaller_edition_not_rejected_as_too_short(conn):
# Skillet: source delivers the 12-track standard (~40 min); MB's release is a 20-track deluxe
# (~73 min). The unscaled duration guard falsely rejects; scaling to the delivered edition
# (12/20) must let it pass.
job_id = insert_request(conn, artist="Skillet", album="Unleashed")
claim_next(conn)
run_pipeline(conn, job_id, [_StandardEditionQobuz(source_tracks=12)],
resolver=_DeluxeResolver(mb_tracks=20, mb_total_s=4380),
measure_duration=lambda _s: 2400.0, dest_root="/tmp/lib-se")
assert _state_error(conn, job_id) == ("imported", None)
def test_genuinely_incomplete_still_rejected(conn):
# John Mayer: the source promised 12 tracks but one failed mid-download → only 11 land.
# This is a real partial and must still be rejected (the fix must not swallow it).
job_id = insert_request(conn, artist="John Mayer", album="The Search for Everything")
claim_next(conn)
run_pipeline(conn, job_id, [_PartialQobuz(promised=12)],
resolver=_DeluxeResolver(mb_tracks=12, mb_total_s=2580),
measure_duration=lambda _s: 2400.0, dest_root="/tmp/lib-se")
assert _state_error(conn, job_id) == ("needs_attention", "incomplete download")
+133
View File
@@ -0,0 +1,133 @@
"""Upgrade guard: when an album is already in the library and no currently-available source can
beat it (e.g. Qobuz is gated by pacing and only a same-quality Soulseek copy is left), the pipeline
must NOT waste a download — keep-if-better would discard it anyway. It should skip and let the
release be re-attempted later (when Qobuz frees up). A genuinely better source still upgrades."""
import os
from lyra_worker.claim import claim_next
from lyra_worker.pipeline import run_pipeline
from lyra_worker.types import Candidate, DownloadResult, Quality
_CD = Quality(fmt="FLAC", lossless=True, bit_depth=16, sample_rate=44100) # class 2
_HIRES = Quality(fmt="FLAC", lossless=True, bit_depth=24, sample_rate=96000) # class 3
def _cfg(conn, key, value):
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
(key, value),
)
conn.commit()
class _Counting:
def __init__(self, name, tier, quality):
self.name, self.tier, self._q = name, tier, quality
self.calls = 0
self.searches = 0
def health(self):
return True
def search(self, target):
self.searches += 1
return [Candidate(source=self.name, source_ref=self.name, matched_artist=target.artist,
matched_album=target.album, quality=self._q, track_count=3, source_tier=self.tier)]
def download(self, candidate, dest, on_progress):
self.calls += 1
os.makedirs(dest, exist_ok=True)
for i in range(3):
with open(os.path.join(dest, f"0{i}.flac"), "wb") as fh:
fh.write(b"a")
return DownloadResult(ok=True, path=dest, track_count=3)
def _seed_library(conn, artist, album, quality_class, path):
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "LibraryItem" (id, artist, album, path, source, format, "qualityClass", "importedAt") '
"VALUES (gen_random_uuid()::text, %s, %s, %s, 'soulseek', 'FLAC', %s, now())",
(artist, album, path, quality_class),
)
conn.commit()
def _insert_upgrade_job(conn, artist, album):
with conn.cursor() as cur:
cur.execute(
'INSERT INTO "MonitoredRelease" (id, "artistMbid", "artistName", "rgMbid", album, '
'monitored, state, "secondaryTypes", "createdAt") '
"VALUES (gen_random_uuid()::text, 'm', %s, 'rg', %s, true, 'grabbed', '{}', now()) RETURNING id",
(artist, album),
)
rel_id = cur.fetchone()[0]
cur.execute(
'INSERT INTO "Request" (id, artist, album, status, "monitoredReleaseId", "createdAt") '
"VALUES (gen_random_uuid()::text, %s, %s, 'pending', %s, now()) RETURNING id",
(artist, album, rel_id),
)
req = cur.fetchone()[0]
cur.execute(
'INSERT INTO "Job" (id, "requestId", state, "currentStage", attempts, "createdAt", "updatedAt") '
"VALUES (gen_random_uuid()::text, %s, 'requested', 'intake', 0, now(), now()) RETURNING id",
(req,),
)
job = cur.fetchone()[0]
conn.commit()
return job
def _lib(conn, artist, album):
with conn.cursor() as cur:
cur.execute('SELECT path, "qualityClass" FROM "LibraryItem" WHERE artist=%s AND album=%s', (artist, album))
return cur.fetchone()
def _state(conn, job_id):
with conn.cursor() as cur:
cur.execute('SELECT state FROM "Job" WHERE id=%s', (job_id,))
return cur.fetchone()[0]
def test_upgrade_skipped_pre_search_when_qobuz_gated(conn):
# Qobuz gated (empty active window) + existing >= CD copy: only hi-res Qobuz could improve it,
# so defer WITHOUT even searching — avoids re-searching every owned album each monitor cycle.
_cfg(conn, "qobuz.pacing.activeStartHour", "5")
_cfg(conn, "qobuz.pacing.activeEndHour", "5") # empty window → gate closed
_seed_library(conn, "A", "B", 2, "/m/existing")
job_id = _insert_upgrade_job(conn, "A", "B")
claim_next(conn)
soulseek = _Counting("soulseek", 1, _CD)
run_pipeline(conn, job_id, [soulseek], dest_root="/tmp/lib-ug", upgrade_cutoff=3)
assert soulseek.searches == 0 # no wasted search
assert soulseek.calls == 0 # no wasted download
assert _lib(conn, "A", "B") == ("/m/existing", 2)
assert _state(conn, job_id) == "imported"
def test_upgrade_skipped_post_search_when_only_same_quality_found(conn):
# Gate OPEN, so it searches, but the only source found is same-quality → skip the download.
_cfg(conn, "qobuz.pacing.activeStartHour", "0")
_cfg(conn, "qobuz.pacing.activeEndHour", "24") # gate open
_seed_library(conn, "A", "B", 2, "/m/existing")
job_id = _insert_upgrade_job(conn, "A", "B")
claim_next(conn)
soulseek = _Counting("soulseek", 1, _CD)
run_pipeline(conn, job_id, [soulseek], dest_root="/tmp/lib-ug", upgrade_cutoff=3)
assert soulseek.searches == 1 # it did search (gate open)
assert soulseek.calls == 0 # but skipped the useless download
assert _lib(conn, "A", "B") == ("/m/existing", 2)
def test_upgrade_proceeds_when_a_better_source_is_available(conn):
# Existing class-2 copy; a class-3 (hi-res) source IS available → download + upgrade.
_seed_library(conn, "A", "B", 2, "/m/existing")
job_id = _insert_upgrade_job(conn, "A", "B")
claim_next(conn)
qobuz = _Counting("qobuz", 0, _HIRES)
run_pipeline(conn, job_id, [qobuz], dest_root="/tmp/lib-ug2", upgrade_cutoff=3)
assert qobuz.calls == 1 # downloaded the better copy
assert _lib(conn, "A", "B")[1] == 3 # upgraded to hi-res