Compare commits
20 Commits
108114ffa3
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 395944f3e1 | |||
| 10430037fe | |||
| 01657d4e1b | |||
| 0ad1fd4cd5 | |||
| 7ed8ca2848 | |||
| 5f59283bf0 | |||
| d2a86932e0 | |||
| 73f1ae6e2f | |||
| c49cee9a20 | |||
| db8b43b0e8 | |||
| 434cd2b22c | |||
| d7ea29a97a | |||
| e8483ee22b | |||
| ca5bbfa9da | |||
| 675d964bbe | |||
| 57f97e9838 | |||
| 1fc61a0f5a | |||
| 67f374c470 | |||
| d47ed12e8e | |||
| b0eec41823 |
@@ -0,0 +1,83 @@
|
|||||||
|
# Qobuz pacing ("budget gate") — design
|
||||||
|
|
||||||
|
Date: 2026-07-15
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Bulk hi-res downloading got the Qobuz account blocked (`403 USER_BLOCKED`). We need to keep
|
||||||
|
using Qobuz for hi-res but at a volume and cadence that looks like a human listener, so the
|
||||||
|
account isn't flagged again — while a large wanted-list still drains promptly via other sources.
|
||||||
|
|
||||||
|
## Approach: a per-job Qobuz eligibility gate
|
||||||
|
|
||||||
|
One gate decides, **per album**, whether Qobuz is an eligible source *right now*. If the gate is
|
||||||
|
closed, Qobuz candidates are dropped before ranking, so the worker falls through to the best
|
||||||
|
non-Qobuz source (Soulseek/YouTube) and keeps moving — no idling. Monitored albums grabbed that
|
||||||
|
way are upgraded to Qobuz hi-res later, automatically, via the existing quality-upgrade window,
|
||||||
|
once Qobuz has budget again.
|
||||||
|
|
||||||
|
**Qobuz is eligible for a job only if ALL hold:**
|
||||||
|
1. **Active hours** — current hour ∈ [activeStart, activeEnd) (a daytime window; start < end).
|
||||||
|
2. **Under today's cap** — Qobuz downloads counted today < today's cap.
|
||||||
|
3. **Spacing elapsed** — now ≥ the randomized "next allowed" time set after the last Qobuz
|
||||||
|
download.
|
||||||
|
|
||||||
|
After each successful Qobuz download the worker bumps today's counter and sets the next-allowed
|
||||||
|
time. The counter resets at the day boundary (tracked by a stored date).
|
||||||
|
|
||||||
|
**Today's cap = warm-up ramp of the steady cap**, by account age (days since `warmupStartDate`):
|
||||||
|
|
||||||
|
| Account age (days) | Cap |
|
||||||
|
|---|---|
|
||||||
|
| 0–2 | min(5, steady) |
|
||||||
|
| 3–6 | min(10, steady) |
|
||||||
|
| 7–13 | min(20, steady) |
|
||||||
|
| 14+ | steady |
|
||||||
|
|
||||||
|
If `warmupStartDate` is unset, no ramp (steady cap applies).
|
||||||
|
|
||||||
|
**Spacing / jitter (combined):** the gap to the next Qobuz download is randomized around
|
||||||
|
`activeWindowSeconds / todayCap` (so the day's budget spreads evenly across active hours), ±30%,
|
||||||
|
with a hard floor (≥ 60 s). This *is* the jitter — never a fixed metronome — and it prevents
|
||||||
|
front-loading the whole daily cap into the first hour.
|
||||||
|
|
||||||
|
## Configuration (Config keys; editable in Settings → Qobuz → Pacing)
|
||||||
|
|
||||||
|
Settings:
|
||||||
|
- `qobuz.pacing.enabled` (default `true`)
|
||||||
|
- `qobuz.pacing.activeStartHour` (default `8`)
|
||||||
|
- `qobuz.pacing.activeEndHour` (default `23`)
|
||||||
|
- `qobuz.pacing.dailyCap` (steady cap, default `40`)
|
||||||
|
- `qobuz.pacing.warmupStartDate` (ISO date; set to the new account's start day)
|
||||||
|
|
||||||
|
Worker-managed state (not user-edited):
|
||||||
|
- `qobuz.pacing.countDay` (date the counter is for)
|
||||||
|
- `qobuz.pacing.countToday` (int)
|
||||||
|
- `qobuz.pacing.nextAllowedAt` (epoch seconds)
|
||||||
|
|
||||||
|
Current hour / day come from the **DB clock** (`now()`), avoiding container-timezone ambiguity.
|
||||||
|
Config is read fresh each worker loop, so changes take effect with no restart.
|
||||||
|
|
||||||
|
## Worker integration
|
||||||
|
|
||||||
|
- New module `worker/lyra_worker/qobuz_gate.py`: a `QobuzPacing` config dataclass
|
||||||
|
(`from_config`), a pure `today_cap(...)`, a pure `is_allowed(...)` decision, and thin
|
||||||
|
DB helpers `gate_open(conn, now)` (reads state + decides) and `record_download(conn, pacing,
|
||||||
|
now)` (bumps counter + sets next-allowed).
|
||||||
|
- `pipeline.py`: after searching sources, if `gate_open` is false AND at least one non-Qobuz
|
||||||
|
candidate exists, drop Qobuz candidates before ranking (fall-through). If Qobuz is the *only*
|
||||||
|
source available, keep it (a Qobuz-only setup isn't gated — holding would strand the album).
|
||||||
|
- After a successful download where `winner.source == "qobuz"`, call `record_download`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Pure-logic unit tests for `is_allowed` (each gate condition), `today_cap` (ramp boundaries), and
|
||||||
|
the spacing computation; pipeline tests that a gated Qobuz candidate is dropped in favour of a
|
||||||
|
non-Qobuz source, that an open gate keeps Qobuz, and that a successful Qobuz download records the
|
||||||
|
counter/next-allowed.
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- Holding/parking an album for later Qobuz when Qobuz is the only source (fails/last-source as
|
||||||
|
today). Multi-source setups fall through, which is the common case.
|
||||||
|
- Timezone-aware active-hours windows that wrap past midnight (start < end only).
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
-- Estimated seconds remaining for the active download (measured throughput). Nullable.
|
||||||
|
ALTER TABLE "Job" ADD COLUMN "downloadEtaSeconds" INTEGER;
|
||||||
@@ -58,6 +58,9 @@ model Job {
|
|||||||
createdAt DateTime @default(now())
|
createdAt DateTime @default(now())
|
||||||
updatedAt DateTime @updatedAt
|
updatedAt DateTime @updatedAt
|
||||||
downloadProgress Float @default(0)
|
downloadProgress Float @default(0)
|
||||||
|
// Estimated seconds until the current download finishes, from measured throughput (slskd byte
|
||||||
|
// transfer rate). Null when unknown (no speed sample yet, or a source that doesn't report bytes).
|
||||||
|
downloadEtaSeconds Int?
|
||||||
paused Boolean @default(false)
|
paused Boolean @default(false)
|
||||||
candidates Candidate[]
|
candidates Candidate[]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ export function JobRow({
|
|||||||
album,
|
album,
|
||||||
kind,
|
kind,
|
||||||
label,
|
label,
|
||||||
|
marker,
|
||||||
meta,
|
meta,
|
||||||
note,
|
note,
|
||||||
bar,
|
bar,
|
||||||
@@ -22,6 +23,7 @@ export function JobRow({
|
|||||||
album: string;
|
album: string;
|
||||||
kind: Kind;
|
kind: Kind;
|
||||||
label: string;
|
label: string;
|
||||||
|
marker?: string;
|
||||||
meta?: ReactNode;
|
meta?: ReactNode;
|
||||||
note?: ReactNode;
|
note?: ReactNode;
|
||||||
bar?: ReactNode;
|
bar?: ReactNode;
|
||||||
@@ -31,6 +33,7 @@ export function JobRow({
|
|||||||
<div className="stripe" />
|
<div className="stripe" />
|
||||||
<div>
|
<div>
|
||||||
<h3 className="title">
|
<h3 className="title">
|
||||||
|
{marker ? <span className="nextup">{marker}</span> : null}
|
||||||
{album} <span className="artist">· {artist}</span>
|
{album} <span className="artist">· {artist}</span>
|
||||||
</h3>
|
</h3>
|
||||||
{meta ? <div className="meta">{meta}</div> : null}
|
{meta ? <div className="meta">{meta}</div> : null}
|
||||||
|
|||||||
@@ -0,0 +1,47 @@
|
|||||||
|
"use client";
|
||||||
|
|
||||||
|
import { useEffect, useRef, useState, type ReactNode } from "react";
|
||||||
|
|
||||||
|
/** A vertical "⋮" trigger that opens a small popup menu. Closes on outside click, Escape, or any
|
||||||
|
* click inside the popup (so menu items don't each need to manage it). Children are the items —
|
||||||
|
* use <button className="menu-item">…</button>. */
|
||||||
|
export function KebabMenu({ label = "Actions", children }: { label?: string; children: ReactNode }) {
|
||||||
|
const [open, setOpen] = useState(false);
|
||||||
|
const ref = useRef<HTMLDivElement>(null);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!open) return;
|
||||||
|
function onDoc(e: MouseEvent) {
|
||||||
|
if (ref.current && !ref.current.contains(e.target as Node)) setOpen(false);
|
||||||
|
}
|
||||||
|
function onKey(e: KeyboardEvent) {
|
||||||
|
if (e.key === "Escape") setOpen(false);
|
||||||
|
}
|
||||||
|
document.addEventListener("mousedown", onDoc);
|
||||||
|
document.addEventListener("keydown", onKey);
|
||||||
|
return () => {
|
||||||
|
document.removeEventListener("mousedown", onDoc);
|
||||||
|
document.removeEventListener("keydown", onKey);
|
||||||
|
};
|
||||||
|
}, [open]);
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="menu" ref={ref}>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="menu-trigger"
|
||||||
|
aria-haspopup="menu"
|
||||||
|
aria-expanded={open}
|
||||||
|
aria-label={label}
|
||||||
|
onClick={() => setOpen((o) => !o)}
|
||||||
|
>
|
||||||
|
⋮
|
||||||
|
</button>
|
||||||
|
{open ? (
|
||||||
|
<div className="menu-pop" role="menu" onClick={() => setOpen(false)}>
|
||||||
|
{children}
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -1,5 +1,20 @@
|
|||||||
import { describe, it, expect } from "vitest";
|
import { describe, it, expect } from "vitest";
|
||||||
import { describeJob, timeAgo } from "./status";
|
import { describeJob, etaLabel, timeAgo } from "./status";
|
||||||
|
|
||||||
|
describe("etaLabel", () => {
|
||||||
|
it("formats seconds, minutes, and hours", () => {
|
||||||
|
expect(etaLabel(45)).toBe("~45s left");
|
||||||
|
expect(etaLabel(240)).toBe("~4m left");
|
||||||
|
expect(etaLabel(3900)).toBe("~1h 5m left");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("returns null for missing or non-positive input", () => {
|
||||||
|
expect(etaLabel(null)).toBeNull();
|
||||||
|
expect(etaLabel(undefined)).toBeNull();
|
||||||
|
expect(etaLabel(0)).toBeNull();
|
||||||
|
expect(etaLabel(-5)).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe("describeJob", () => {
|
describe("describeJob", () => {
|
||||||
it("maps pipeline states to literal labels + severity kind", () => {
|
it("maps pipeline states to literal labels + severity kind", () => {
|
||||||
|
|||||||
@@ -27,6 +27,16 @@ export function timeAgo(value: string | Date | null | undefined, now: number): s
|
|||||||
return `${Math.floor(h / 24)}d`;
|
return `${Math.floor(h / 24)}d`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Compact download ETA like "~4m left", "~45s left", "~1h 5m left". Null/≤0 → no label. */
|
||||||
|
export function etaLabel(seconds: number | null | undefined): string | null {
|
||||||
|
if (seconds == null || seconds <= 0) return null;
|
||||||
|
if (seconds < 60) return `~${seconds}s left`;
|
||||||
|
const m = Math.round(seconds / 60);
|
||||||
|
if (m < 60) return `~${m}m left`;
|
||||||
|
const h = Math.floor(m / 60);
|
||||||
|
return `~${h}h ${m % 60}m left`;
|
||||||
|
}
|
||||||
|
|
||||||
export function describeJob(state: string, stage: string): { label: string; kind: Kind } {
|
export function describeJob(state: string, stage: string): { label: string; kind: Kind } {
|
||||||
void stage; // stage no longer drives the label; kept for call-site compatibility
|
void stage; // stage no longer drives the label; kept for call-site compatibility
|
||||||
return (
|
return (
|
||||||
|
|||||||
@@ -0,0 +1,37 @@
|
|||||||
|
import { describe, it, expect } from "vitest";
|
||||||
|
import { GET, PATCH } from "./route";
|
||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
|
||||||
|
function patch(body: Record<string, string>) {
|
||||||
|
return new Request("http://localhost/api/qobuz/pacing", {
|
||||||
|
method: "PATCH",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify(body),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("qobuz pacing config API", () => {
|
||||||
|
it("returns defaults when nothing is stored", async () => {
|
||||||
|
const cfg = await (await GET()).json();
|
||||||
|
expect(cfg["qobuz.pacing.enabled"]).toBe("true");
|
||||||
|
expect(cfg["qobuz.pacing.activeStartHour"]).toBe("8");
|
||||||
|
expect(cfg["qobuz.pacing.activeEndHour"]).toBe("23");
|
||||||
|
expect(cfg["qobuz.pacing.dailyCap"]).toBe("40");
|
||||||
|
expect(cfg["qobuz.pacing.warmupStartDate"]).toBe("");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("round-trips allow-listed settings and persists them", async () => {
|
||||||
|
await PATCH(patch({ "qobuz.pacing.dailyCap": "25", "qobuz.pacing.activeStartHour": "9" }));
|
||||||
|
const stored = await prisma.config.findUnique({ where: { key: "qobuz.pacing.dailyCap" } });
|
||||||
|
expect(stored?.value).toBe("25");
|
||||||
|
const cfg = await (await GET()).json();
|
||||||
|
expect(cfg["qobuz.pacing.dailyCap"]).toBe("25");
|
||||||
|
expect(cfg["qobuz.pacing.activeStartHour"]).toBe("9");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("ignores keys not in the allow-list (e.g. worker-managed counters)", async () => {
|
||||||
|
await PATCH(patch({ "qobuz.pacing.countToday": "999", "qobuz.pacing.dailyCap": "30" }));
|
||||||
|
expect(await prisma.config.findUnique({ where: { key: "qobuz.pacing.countToday" } })).toBeNull();
|
||||||
|
expect((await prisma.config.findUnique({ where: { key: "qobuz.pacing.dailyCap" } }))?.value).toBe("30");
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
|
||||||
|
// Qobuz pacing ("budget gate") settings. The worker reads these Config keys live; state keys
|
||||||
|
// (countDay/countToday/nextAllowedAt) are worker-managed and intentionally NOT editable here.
|
||||||
|
const DEFAULTS: Record<string, string> = {
|
||||||
|
"qobuz.pacing.enabled": "true",
|
||||||
|
"qobuz.pacing.activeStartHour": "8",
|
||||||
|
"qobuz.pacing.activeEndHour": "23",
|
||||||
|
"qobuz.pacing.dailyCap": "40",
|
||||||
|
"qobuz.pacing.warmupStartDate": "",
|
||||||
|
};
|
||||||
|
|
||||||
|
export async function GET() {
|
||||||
|
const rows = await prisma.config.findMany({ where: { key: { in: Object.keys(DEFAULTS) } } });
|
||||||
|
const stored = Object.fromEntries(rows.map((r) => [r.key, r.value]));
|
||||||
|
return Response.json(
|
||||||
|
Object.fromEntries(Object.entries(DEFAULTS).map(([k, d]) => [k, stored[k] ?? d])),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function PATCH(request: Request) {
|
||||||
|
let body: unknown;
|
||||||
|
try {
|
||||||
|
body = await request.json();
|
||||||
|
} catch {
|
||||||
|
return Response.json({ error: "invalid JSON" }, { status: 400 });
|
||||||
|
}
|
||||||
|
const entries = Object.entries((body ?? {}) as Record<string, unknown>).filter(([k]) =>
|
||||||
|
Object.prototype.hasOwnProperty.call(DEFAULTS, k),
|
||||||
|
);
|
||||||
|
for (const [key, value] of entries) {
|
||||||
|
await prisma.config.upsert({
|
||||||
|
where: { key },
|
||||||
|
create: { key, value: String(value), secret: false },
|
||||||
|
update: { value: String(value) },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return Response.json({ updated: entries.length });
|
||||||
|
}
|
||||||
@@ -64,6 +64,7 @@ export async function GET() {
|
|||||||
claimedAt: r.job.claimedAt,
|
claimedAt: r.job.claimedAt,
|
||||||
updatedAt: r.job.updatedAt,
|
updatedAt: r.job.updatedAt,
|
||||||
downloadProgress: r.job.downloadProgress,
|
downloadProgress: r.job.downloadProgress,
|
||||||
|
downloadEtaSeconds: r.job.downloadEtaSeconds,
|
||||||
paused: r.job.paused,
|
paused: r.job.paused,
|
||||||
candidateCount: cands.length,
|
candidateCount: cands.length,
|
||||||
chosen: chosen ? { source: chosen.source, format: chosen.format, trackCount: chosen.trackCount } : null,
|
chosen: chosen ? { source: chosen.source, format: chosen.format, trackCount: chosen.trackCount } : null,
|
||||||
|
|||||||
@@ -386,6 +386,39 @@ nav.contents .sep { flex: 1; }
|
|||||||
.tab:hover { color: var(--ink); }
|
.tab:hover { color: var(--ink); }
|
||||||
.tab.active { color: var(--ink); border-bottom-color: var(--accent); }
|
.tab.active { color: var(--ink); border-bottom-color: var(--accent); }
|
||||||
.tab .n { color: var(--rule-2); margin-left: 6px; }
|
.tab .n { color: var(--rule-2); margin-left: 6px; }
|
||||||
|
.tab.has-errors .n { color: var(--alert); }
|
||||||
|
|
||||||
|
/* "Next up" badge on the first queued row (the item the worker claims next) */
|
||||||
|
.nextup {
|
||||||
|
display: inline-block; vertical-align: middle; margin-right: 9px; transform: translateY(-2px);
|
||||||
|
font-family: var(--mono); font-size: 0.56rem; letter-spacing: 0.13em; text-transform: uppercase;
|
||||||
|
color: var(--accent-ink); background: var(--accent); padding: 2px 7px; border-radius: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ── Kebab menu (⋮ popup for press actions) ───────────── */
|
||||||
|
.dept.has-actions { align-items: center; }
|
||||||
|
.dept.has-actions .fill { transform: none; }
|
||||||
|
.menu { position: relative; display: inline-flex; }
|
||||||
|
.menu-trigger {
|
||||||
|
font-size: 1.15rem; line-height: 1; color: var(--graphite); background: transparent;
|
||||||
|
border: 1px solid var(--rule); border-radius: 6px; width: 32px; height: 30px; cursor: pointer;
|
||||||
|
display: inline-flex; align-items: center; justify-content: center;
|
||||||
|
}
|
||||||
|
.menu-trigger:hover { color: var(--ink); border-color: var(--rule-2); }
|
||||||
|
.menu-pop {
|
||||||
|
position: absolute; top: calc(100% + 6px); right: 0; z-index: 30; min-width: 190px;
|
||||||
|
background: var(--paper-2); border: 1px solid var(--rule-2); border-radius: 9px;
|
||||||
|
box-shadow: 0 10px 30px rgba(0, 0, 0, 0.22); padding: 5px;
|
||||||
|
display: flex; flex-direction: column; gap: 1px;
|
||||||
|
}
|
||||||
|
.menu-item {
|
||||||
|
font-family: var(--mono); font-size: 0.68rem; letter-spacing: 0.1em; text-transform: uppercase;
|
||||||
|
text-align: left; background: transparent; border: 0; color: var(--ink);
|
||||||
|
padding: 10px 11px; border-radius: 6px; cursor: pointer; white-space: nowrap;
|
||||||
|
}
|
||||||
|
.menu-item:hover { background: var(--rule); }
|
||||||
|
.menu-item:disabled { color: var(--rule-2); cursor: default; background: transparent; }
|
||||||
|
.menu-item.danger { color: var(--alert); }
|
||||||
|
|
||||||
/* ── Discography row (cover thumb + click-to-modal) ───── */
|
/* ── Discography row (cover thumb + click-to-modal) ───── */
|
||||||
.disco-open {
|
.disco-open {
|
||||||
|
|||||||
+198
-134
@@ -1,14 +1,34 @@
|
|||||||
"use client";
|
"use client";
|
||||||
|
|
||||||
import { useEffect, useState } from "react";
|
import { useEffect, useState } from "react";
|
||||||
import { describeJob, timeAgo } from "./_ui/status";
|
import { describeJob, etaLabel, timeAgo } from "./_ui/status";
|
||||||
import { StatTiles } from "./_ui/stat-tiles";
|
import { StatTiles } from "./_ui/stat-tiles";
|
||||||
import { SectionHeader } from "./_ui/section-header";
|
import { SectionHeader } from "./_ui/section-header";
|
||||||
import { JobRow } from "./_ui/job-row";
|
import { JobRow } from "./_ui/job-row";
|
||||||
import { ProgressBar } from "./_ui/progress-bar";
|
import { ProgressBar } from "./_ui/progress-bar";
|
||||||
import { PressedList } from "./_ui/pressed-list";
|
import { PressedList } from "./_ui/pressed-list";
|
||||||
|
import { KebabMenu } from "./_ui/kebab-menu";
|
||||||
import { toast } from "./_ui/toast";
|
import { toast } from "./_ui/toast";
|
||||||
|
|
||||||
|
type Tab = "active" | "queue" | "errors";
|
||||||
|
|
||||||
|
// Split the on-the-press rows into three honest buckets so active presses aren't buried between
|
||||||
|
// queued items and errors. Active = being worked (searching/downloading/finishing), Queue =
|
||||||
|
// waiting to be claimed, Errors = needs attention.
|
||||||
|
function categoryOf(state: string, stage: string): Tab {
|
||||||
|
const kind = describeJob(state, stage).kind;
|
||||||
|
if (kind === "attention") return "errors";
|
||||||
|
if (state === "requested") return "queue";
|
||||||
|
return "active";
|
||||||
|
}
|
||||||
|
|
||||||
|
const TAB_LABEL: Record<Tab, string> = { active: "Active", queue: "In queue", errors: "Errors" };
|
||||||
|
const TAB_EMPTY: Record<Tab, string> = {
|
||||||
|
active: "Nothing pressing right now.",
|
||||||
|
queue: "The queue is empty.",
|
||||||
|
errors: "No errors — everything's flowing.",
|
||||||
|
};
|
||||||
|
|
||||||
type Row = {
|
type Row = {
|
||||||
id: string;
|
id: string;
|
||||||
artist: string;
|
artist: string;
|
||||||
@@ -26,6 +46,7 @@ type Row = {
|
|||||||
candidateCount: number;
|
candidateCount: number;
|
||||||
chosen: { source: string; format: string; trackCount: number } | null;
|
chosen: { source: string; format: string; trackCount: number } | null;
|
||||||
downloadProgress: number;
|
downloadProgress: number;
|
||||||
|
downloadEtaSeconds: number | null;
|
||||||
paused: boolean;
|
paused: boolean;
|
||||||
} | null;
|
} | null;
|
||||||
};
|
};
|
||||||
@@ -42,14 +63,16 @@ function jobOf(r: Row) {
|
|||||||
candidateCount: 0,
|
candidateCount: 0,
|
||||||
chosen: null,
|
chosen: null,
|
||||||
downloadProgress: 0,
|
downloadProgress: 0,
|
||||||
|
downloadEtaSeconds: null,
|
||||||
paused: false,
|
paused: false,
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
function pressedMark(r: Row): string {
|
function pressedMark(r: Row): string {
|
||||||
if (!r.createdAt) return "Lyra";
|
const when = r.job?.updatedAt ?? r.createdAt; // when it was pressed, not when it was requested
|
||||||
const d = new Date(r.createdAt);
|
if (!when) return "Lyra";
|
||||||
|
const d = new Date(when);
|
||||||
return `Lyra · ${d.toLocaleDateString("en-GB", { day: "2-digit", month: "short" }).toUpperCase()}`;
|
return `Lyra · ${d.toLocaleDateString("en-GB", { day: "2-digit", month: "short" }).toUpperCase()}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -60,6 +83,9 @@ export function Queue() {
|
|||||||
const [artist, setArtist] = useState("");
|
const [artist, setArtist] = useState("");
|
||||||
const [album, setAlbum] = useState("");
|
const [album, setAlbum] = useState("");
|
||||||
const [paused, setPaused] = useState(false);
|
const [paused, setPaused] = useState(false);
|
||||||
|
// null = follow the smart default (Active if anything's active, else Queue, else Errors); once
|
||||||
|
// the user picks a tab, their choice sticks.
|
||||||
|
const [tab, setTab] = useState<Tab | null>(null);
|
||||||
|
|
||||||
async function refresh() {
|
async function refresh() {
|
||||||
const [reqRes, wantRes, artRes, floorRes] = await Promise.all([
|
const [reqRes, wantRes, artRes, floorRes] = await Promise.all([
|
||||||
@@ -159,53 +185,166 @@ export function Queue() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const active = rows.filter((r) => jobOf(r).state !== "imported");
|
const active = rows.filter((r) => jobOf(r).state !== "imported");
|
||||||
const downloading = rows.some((r) => {
|
const buckets: Record<Tab, Row[]> = { active: [], queue: [], errors: [] };
|
||||||
const s = jobOf(r).state;
|
for (const r of active) {
|
||||||
return s === "matching" || s === "matched" || s === "downloading" || s === "tagging";
|
const j = jobOf(r);
|
||||||
});
|
buckets[categoryOf(j.state, j.currentStage)].push(r);
|
||||||
// Dedupe by artist+album — re-requesting/re-downloading the same album leaves several
|
}
|
||||||
// completed Requests; show each album once (rows arrive newest-first, so keep the first).
|
// The API returns rows newest-first, but the worker claims the OLDEST queued job next
|
||||||
|
// (claim_next: ORDER BY createdAt ASC). Order the queue tab to match, so the item that runs
|
||||||
|
// next sits at the top instead of the bottom.
|
||||||
|
const ts = (r: Row) => (r.createdAt ? new Date(r.createdAt).getTime() : 0);
|
||||||
|
buckets.queue.sort((a, b) => ts(a) - ts(b));
|
||||||
|
// "Next up" = the first queued item the worker will actually claim (oldest, not individually
|
||||||
|
// paused) — mark it so the ordering reads clearly.
|
||||||
|
const nextUpId = buckets.queue.find((r) => !jobOf(r).paused)?.id ?? null;
|
||||||
|
const queuedCount = buckets.queue.length;
|
||||||
|
const downloading = buckets.active.length > 0;
|
||||||
|
// Smart default: land on the tab that has something worth looking at.
|
||||||
|
const defaultTab: Tab = buckets.active.length ? "active" : queuedCount ? "queue" : "errors";
|
||||||
|
const activeTab: Tab = tab ?? defaultTab;
|
||||||
|
const shown = buckets[activeTab];
|
||||||
|
|
||||||
|
// "Recently Pressed": imported albums ordered by when they were PRESSED (job.updatedAt ≈ import
|
||||||
|
// time), newest first — an album requested long ago but imported just now is recently pressed
|
||||||
|
// even though its Request is old (rows arrive by request createdAt, which buried fresh imports).
|
||||||
|
// Dedupe by artist+album (re-downloads leave several completed Requests; keep the most recent).
|
||||||
const seenPressed = new Set<string>();
|
const seenPressed = new Set<string>();
|
||||||
const pressed = rows.filter((r) => {
|
const pressedAll = rows
|
||||||
if (jobOf(r).state !== "imported") return false;
|
.filter((r) => jobOf(r).state === "imported")
|
||||||
const key = `${r.artist} ${r.album}`.toLowerCase();
|
.sort((a, b) => new Date(jobOf(b).updatedAt ?? 0).getTime() - new Date(jobOf(a).updatedAt ?? 0).getTime())
|
||||||
if (seenPressed.has(key)) return false;
|
.filter((r) => {
|
||||||
seenPressed.add(key);
|
const key = `${r.artist} ${r.album}`.toLowerCase();
|
||||||
return true;
|
if (seenPressed.has(key)) return false;
|
||||||
});
|
seenPressed.add(key);
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
const pressed = pressedAll.slice(0, 10); // show only the 10 most recently pressed
|
||||||
|
|
||||||
|
const now = Date.now();
|
||||||
|
function renderRow(r: Row) {
|
||||||
|
const j = jobOf(r);
|
||||||
|
const d = describeJob(j.state, j.currentStage);
|
||||||
|
const attention = d.kind === "attention";
|
||||||
|
|
||||||
|
let meta: string | undefined;
|
||||||
|
if (j.state === "downloading" || j.state === "tagging") {
|
||||||
|
const parts: string[] = [];
|
||||||
|
if (j.chosen) parts.push(`${j.chosen.source} · ${j.chosen.format}`);
|
||||||
|
const elapsed = timeAgo(j.claimedAt ?? j.updatedAt, now);
|
||||||
|
if (elapsed) parts.push(elapsed);
|
||||||
|
meta = parts.join(" · ") || undefined;
|
||||||
|
} else if (j.state === "matching" || j.state === "matched") {
|
||||||
|
meta =
|
||||||
|
j.candidateCount > 0
|
||||||
|
? `${j.candidateCount} sources${j.chosen ? ` · best: ${j.chosen.source} ${j.chosen.format}` : ""}`
|
||||||
|
: "Searching sources…";
|
||||||
|
} else if (j.state === "requested") {
|
||||||
|
const elapsed = timeAgo(r.createdAt, now);
|
||||||
|
meta = elapsed ? `In the queue · ${elapsed}` : "In the queue";
|
||||||
|
} else {
|
||||||
|
meta = undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
const note = attention ? (
|
||||||
|
<>
|
||||||
|
{j.error?.trim() || "No source met the quality cutoff, or the pipeline stalled."}{" "}
|
||||||
|
<button className="btn sm" onClick={() => retry(r.id)}>
|
||||||
|
Retry
|
||||||
|
</button>
|
||||||
|
</>
|
||||||
|
) : j.state === "requested" ? (
|
||||||
|
<span className="item-controls">
|
||||||
|
{j.paused ? (
|
||||||
|
<>
|
||||||
|
<span className="chip">Paused</span>{" "}
|
||||||
|
<button className="btn sm ghost" onClick={() => pauseItem(r.id, false)}>
|
||||||
|
Resume
|
||||||
|
</button>
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
<button className="btn sm ghost" onClick={() => pauseItem(r.id, true)}>
|
||||||
|
Pause
|
||||||
|
</button>
|
||||||
|
)}{" "}
|
||||||
|
<button className="btn sm ghost" onClick={() => removeItem(r.id)}>
|
||||||
|
Remove
|
||||||
|
</button>
|
||||||
|
</span>
|
||||||
|
) : undefined;
|
||||||
|
|
||||||
|
let bar: React.ReactNode;
|
||||||
|
if (j.state === "matching" || j.state === "matched") {
|
||||||
|
bar = <ProgressBar kind="working" indeterminate caption="searching" />;
|
||||||
|
} else if (j.state === "downloading") {
|
||||||
|
const pct = Math.round((j.downloadProgress || 0) * 100);
|
||||||
|
if (pct <= 0) {
|
||||||
|
// Bytes haven't started flowing yet — Qobuz login/metadata/artwork, a Soulseek peer
|
||||||
|
// queue, or a YouTube info-fetch. Show an indeterminate bar instead of a stuck-looking
|
||||||
|
// 0% (it flips to the % bar on the first byte).
|
||||||
|
bar = <ProgressBar kind="working" indeterminate caption="preparing…" />;
|
||||||
|
} else {
|
||||||
|
const total = j.chosen?.trackCount ?? 0;
|
||||||
|
const done = total ? Math.round((pct / 100) * total) : 0;
|
||||||
|
const base = total ? `${pct}% · ${done}/${total} tracks` : `${pct}%`;
|
||||||
|
const eta = etaLabel(j.downloadEtaSeconds);
|
||||||
|
bar = <ProgressBar kind="working" percent={pct} caption={eta ? `${base} · ${eta}` : base} />;
|
||||||
|
}
|
||||||
|
} else if (j.state === "tagging") {
|
||||||
|
bar = <ProgressBar kind="verify" indeterminate caption="finishing" />;
|
||||||
|
} else {
|
||||||
|
bar = undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<JobRow
|
||||||
|
key={r.id}
|
||||||
|
artist={r.artist}
|
||||||
|
album={r.album}
|
||||||
|
kind={d.kind}
|
||||||
|
label={d.label}
|
||||||
|
marker={r.id === nextUpId ? "Next up" : undefined}
|
||||||
|
meta={meta}
|
||||||
|
note={note}
|
||||||
|
bar={bar}
|
||||||
|
/>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div>
|
<div>
|
||||||
<StatTiles
|
<StatTiles
|
||||||
items={[
|
items={[
|
||||||
{ k: "On the press", v: active.length },
|
{ k: "On the press", v: active.length },
|
||||||
{ k: "Pressed", v: pressed.length, accent: true },
|
{ k: "Pressed", v: pressedAll.length, accent: true },
|
||||||
{ k: "Wanted", v: wanted },
|
{ k: "Wanted", v: wanted },
|
||||||
{ k: "Watching", v: watching, unit: "artists" },
|
{ k: "Watching", v: watching, unit: "artists" },
|
||||||
]}
|
]}
|
||||||
/>
|
/>
|
||||||
|
|
||||||
<SectionHeader title="On the Press" note={`${active.length} running`} />
|
<div className="dept has-actions">
|
||||||
|
<h2>On the Press</h2>
|
||||||
<div className="press-controls">
|
<span className="fill" />
|
||||||
{paused ? (
|
{paused ? <span className="count">paused</span> : null}
|
||||||
<button className="btn sm accent" onClick={() => floorAction("resume", "Press resumed")}>
|
<KebabMenu label="Press actions">
|
||||||
Resume the press
|
{paused ? (
|
||||||
|
<button className="menu-item" onClick={() => floorAction("resume", "Press resumed")}>
|
||||||
|
Resume the press
|
||||||
|
</button>
|
||||||
|
) : (
|
||||||
|
<button className="menu-item" onClick={() => floorAction("pause", "Press paused")}>
|
||||||
|
Pause the press
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
<button className="menu-item" onClick={clearPress} disabled={queuedCount === 0}>
|
||||||
|
Clear queue
|
||||||
</button>
|
</button>
|
||||||
) : (
|
{downloading ? (
|
||||||
<button className="btn sm ghost" onClick={() => floorAction("pause", "Press paused")}>
|
<button className="menu-item danger" onClick={stopPress}>
|
||||||
Pause the press
|
Stop now
|
||||||
</button>
|
</button>
|
||||||
)}
|
) : null}
|
||||||
<button className="btn sm ghost" onClick={clearPress}>
|
</KebabMenu>
|
||||||
Clear queue
|
|
||||||
</button>
|
|
||||||
{downloading ? (
|
|
||||||
<button className="btn sm" onClick={stopPress}>
|
|
||||||
Stop now
|
|
||||||
</button>
|
|
||||||
) : null}
|
|
||||||
{paused ? <span className="chip">Press paused</span> : null}
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<form className="request-form" onSubmit={submit}>
|
<form className="request-form" onSubmit={submit}>
|
||||||
@@ -225,107 +364,32 @@ export function Queue() {
|
|||||||
{active.length === 0 ? (
|
{active.length === 0 ? (
|
||||||
<p className="empty">The press is quiet. Queue a release above, or send one over from Wanted or Discover.</p>
|
<p className="empty">The press is quiet. Queue a release above, or send one over from Wanted or Discover.</p>
|
||||||
) : (
|
) : (
|
||||||
<section className="floor">
|
<>
|
||||||
{(() => {
|
<div className="tabs" role="tablist">
|
||||||
const now = Date.now();
|
{(["active", "queue", "errors"] as Tab[]).map((t) => (
|
||||||
return active.map((r) => {
|
<button
|
||||||
const j = jobOf(r);
|
key={t}
|
||||||
const d = describeJob(j.state, j.currentStage);
|
role="tab"
|
||||||
const attention = d.kind === "attention";
|
aria-selected={activeTab === t}
|
||||||
|
className={`tab${activeTab === t ? " active" : ""}${t === "errors" && buckets.errors.length ? " has-errors" : ""}`}
|
||||||
let meta: string | undefined;
|
onClick={() => setTab(t)}
|
||||||
if (j.state === "downloading" || j.state === "tagging") {
|
>
|
||||||
const parts: string[] = [];
|
{TAB_LABEL[t]}
|
||||||
if (j.chosen) parts.push(`${j.chosen.source} · ${j.chosen.format}`);
|
<span className="n">{buckets[t].length}</span>
|
||||||
const elapsed = timeAgo(j.claimedAt ?? j.updatedAt, now);
|
</button>
|
||||||
if (elapsed) parts.push(elapsed);
|
))}
|
||||||
meta = parts.join(" · ") || undefined;
|
</div>
|
||||||
} else if (j.state === "matching" || j.state === "matched") {
|
{shown.length === 0 ? (
|
||||||
meta =
|
<p className="empty">{TAB_EMPTY[activeTab]}</p>
|
||||||
j.candidateCount > 0
|
) : (
|
||||||
? `${j.candidateCount} sources${j.chosen ? ` · best: ${j.chosen.source} ${j.chosen.format}` : ""}`
|
<section className="floor">{shown.map(renderRow)}</section>
|
||||||
: "Searching sources…";
|
)}
|
||||||
} else if (j.state === "requested") {
|
</>
|
||||||
const elapsed = timeAgo(r.createdAt, now);
|
|
||||||
meta = elapsed ? `In the queue · ${elapsed}` : "In the queue";
|
|
||||||
} else {
|
|
||||||
meta = undefined;
|
|
||||||
}
|
|
||||||
|
|
||||||
const note = attention ? (
|
|
||||||
<>
|
|
||||||
{j.error?.trim() || "No source met the quality cutoff, or the pipeline stalled."}{" "}
|
|
||||||
<button className="btn sm" onClick={() => retry(r.id)}>
|
|
||||||
Retry
|
|
||||||
</button>
|
|
||||||
</>
|
|
||||||
) : j.state === "requested" ? (
|
|
||||||
<span className="item-controls">
|
|
||||||
{j.paused ? (
|
|
||||||
<>
|
|
||||||
<span className="chip">Paused</span>{" "}
|
|
||||||
<button className="btn sm ghost" onClick={() => pauseItem(r.id, false)}>
|
|
||||||
Resume
|
|
||||||
</button>
|
|
||||||
</>
|
|
||||||
) : (
|
|
||||||
<button className="btn sm ghost" onClick={() => pauseItem(r.id, true)}>
|
|
||||||
Pause
|
|
||||||
</button>
|
|
||||||
)}{" "}
|
|
||||||
<button className="btn sm ghost" onClick={() => removeItem(r.id)}>
|
|
||||||
Remove
|
|
||||||
</button>
|
|
||||||
</span>
|
|
||||||
) : undefined;
|
|
||||||
|
|
||||||
let bar: React.ReactNode;
|
|
||||||
if (j.state === "matching" || j.state === "matched") {
|
|
||||||
bar = <ProgressBar kind="working" indeterminate caption="searching" />;
|
|
||||||
} else if (j.state === "downloading") {
|
|
||||||
const pct = Math.round((j.downloadProgress || 0) * 100);
|
|
||||||
if (pct <= 0) {
|
|
||||||
// Bytes haven't started flowing yet — Qobuz login/metadata/artwork, a
|
|
||||||
// Soulseek peer queue, or a YouTube info-fetch. Show an indeterminate bar
|
|
||||||
// instead of a stuck-looking 0% (it flips to the % bar on the first byte).
|
|
||||||
bar = <ProgressBar kind="working" indeterminate caption="preparing…" />;
|
|
||||||
} else {
|
|
||||||
const total = j.chosen?.trackCount ?? 0;
|
|
||||||
const done = total ? Math.round((pct / 100) * total) : 0;
|
|
||||||
bar = (
|
|
||||||
<ProgressBar
|
|
||||||
kind="working"
|
|
||||||
percent={pct}
|
|
||||||
caption={total ? `${pct}% · ${done}/${total} tracks` : `${pct}%`}
|
|
||||||
/>
|
|
||||||
);
|
|
||||||
}
|
|
||||||
} else if (j.state === "tagging") {
|
|
||||||
bar = <ProgressBar kind="verify" indeterminate caption="finishing" />;
|
|
||||||
} else {
|
|
||||||
bar = undefined;
|
|
||||||
}
|
|
||||||
|
|
||||||
return (
|
|
||||||
<JobRow
|
|
||||||
key={r.id}
|
|
||||||
artist={r.artist}
|
|
||||||
album={r.album}
|
|
||||||
kind={d.kind}
|
|
||||||
label={d.label}
|
|
||||||
meta={meta}
|
|
||||||
note={note}
|
|
||||||
bar={bar}
|
|
||||||
/>
|
|
||||||
);
|
|
||||||
});
|
|
||||||
})()}
|
|
||||||
</section>
|
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{pressed.length > 0 ? (
|
{pressed.length > 0 ? (
|
||||||
<>
|
<>
|
||||||
<SectionHeader title="Recently Pressed" note={`${pressed.length} in library`} />
|
<SectionHeader title="Recently Pressed" note={`last ${pressed.length} · ${pressedAll.length} in library`} />
|
||||||
<PressedList
|
<PressedList
|
||||||
items={pressed.map((r) => ({ id: r.id, artist: r.artist, album: r.album, matrix: pressedMark(r), rgMbid: r.rgMbid ?? null }))}
|
items={pressed.map((r) => ({ id: r.id, artist: r.artist, album: r.album, matrix: pressedMark(r), rgMbid: r.rgMbid ?? null }))}
|
||||||
/>
|
/>
|
||||||
|
|||||||
@@ -54,6 +54,8 @@ export function SettingsForm() {
|
|||||||
const [discoverSaved, setDiscoverSaved] = useState(false);
|
const [discoverSaved, setDiscoverSaved] = useState(false);
|
||||||
const [monitorCfg, setMonitorCfg] = useState<Record<string, string>>({});
|
const [monitorCfg, setMonitorCfg] = useState<Record<string, string>>({});
|
||||||
const [monitorSaved, setMonitorSaved] = useState(false);
|
const [monitorSaved, setMonitorSaved] = useState(false);
|
||||||
|
const [pacingCfg, setPacingCfg] = useState<Record<string, string>>({});
|
||||||
|
const [pacingSaved, setPacingSaved] = useState(false);
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
fetch("/api/scan")
|
fetch("/api/scan")
|
||||||
@@ -106,6 +108,12 @@ export function SettingsForm() {
|
|||||||
.then((c: Record<string, string>) => setMonitorCfg(c));
|
.then((c: Record<string, string>) => setMonitorCfg(c));
|
||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
fetch("/api/qobuz/pacing")
|
||||||
|
.then((r) => r.json())
|
||||||
|
.then((c: Record<string, string>) => setPacingCfg(c));
|
||||||
|
}, []);
|
||||||
|
|
||||||
async function saveCredentials(e: React.FormEvent) {
|
async function saveCredentials(e: React.FormEvent) {
|
||||||
e.preventDefault();
|
e.preventDefault();
|
||||||
await fetch("/api/config", {
|
await fetch("/api/config", {
|
||||||
@@ -160,6 +168,21 @@ export function SettingsForm() {
|
|||||||
setTimeout(() => setMonitorSaved(false), 1500);
|
setTimeout(() => setMonitorSaved(false), 1500);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function setPacing(key: string, value: string) {
|
||||||
|
setPacingCfg((c) => ({ ...c, [key]: value }));
|
||||||
|
}
|
||||||
|
|
||||||
|
async function savePacing(e: React.FormEvent) {
|
||||||
|
e.preventDefault();
|
||||||
|
await fetch("/api/qobuz/pacing", {
|
||||||
|
method: "PATCH",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify(pacingCfg),
|
||||||
|
});
|
||||||
|
setPacingSaved(true);
|
||||||
|
setTimeout(() => setPacingSaved(false), 1500);
|
||||||
|
}
|
||||||
|
|
||||||
const secretHint = (set: boolean) => (set ? <span className="set"> · set</span> : null);
|
const secretHint = (set: boolean) => (set ? <span className="set"> · set</span> : null);
|
||||||
|
|
||||||
async function clearField(field: "qobuzToken" | "slskdApiKey" | "lastfmApiKey") {
|
async function clearField(field: "qobuzToken" | "slskdApiKey" | "lastfmApiKey") {
|
||||||
@@ -193,6 +216,7 @@ export function SettingsForm() {
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
{tab === "Qobuz" ? (
|
{tab === "Qobuz" ? (
|
||||||
|
<>
|
||||||
<form className="settings-section" onSubmit={saveCredentials}>
|
<form className="settings-section" onSubmit={saveCredentials}>
|
||||||
<div className="subhead">
|
<div className="subhead">
|
||||||
Auth token
|
Auth token
|
||||||
@@ -235,6 +259,55 @@ export function SettingsForm() {
|
|||||||
{saved ? <span className="saved-note">Saved</span> : null}
|
{saved ? <span className="saved-note">Saved</span> : null}
|
||||||
</div>
|
</div>
|
||||||
</form>
|
</form>
|
||||||
|
<form className="settings-section" onSubmit={savePacing}>
|
||||||
|
<div className="subhead">
|
||||||
|
Pacing
|
||||||
|
<HelpButton title="Qobuz pacing">
|
||||||
|
Throttles Qobuz to a human-looking volume and cadence so the account isn't flagged
|
||||||
|
for bulk downloading. Within active hours it downloads up to the daily cap, spaced out
|
||||||
|
with randomized gaps; when throttled, albums fall through to Soulseek/YouTube and
|
||||||
|
upgrade to Qobuz hi-res later. The daily cap ramps up over the first two weeks from the
|
||||||
|
new-account start date.
|
||||||
|
</HelpButton>
|
||||||
|
</div>
|
||||||
|
<label className="field-inline">
|
||||||
|
<input type="checkbox" aria-label="pace qobuz downloads"
|
||||||
|
checked={pacingCfg["qobuz.pacing.enabled"] !== "false"}
|
||||||
|
onChange={(e) => setPacing("qobuz.pacing.enabled", e.target.checked ? "true" : "false")} />
|
||||||
|
<span>Pace Qobuz downloads</span>
|
||||||
|
</label>
|
||||||
|
<div className="field-grid">
|
||||||
|
<label className="field">
|
||||||
|
<span>Active hours — start (0–24)</span>
|
||||||
|
<input type="number" min={0} max={24} aria-label="qobuz active start hour"
|
||||||
|
value={pacingCfg["qobuz.pacing.activeStartHour"] ?? ""}
|
||||||
|
onChange={(e) => setPacing("qobuz.pacing.activeStartHour", e.target.value)} />
|
||||||
|
</label>
|
||||||
|
<label className="field">
|
||||||
|
<span>Active hours — end (0–24)</span>
|
||||||
|
<input type="number" min={0} max={24} aria-label="qobuz active end hour"
|
||||||
|
value={pacingCfg["qobuz.pacing.activeEndHour"] ?? ""}
|
||||||
|
onChange={(e) => setPacing("qobuz.pacing.activeEndHour", e.target.value)} />
|
||||||
|
</label>
|
||||||
|
<label className="field">
|
||||||
|
<span>Daily cap (steady)</span>
|
||||||
|
<input type="number" min={0} aria-label="qobuz daily cap"
|
||||||
|
value={pacingCfg["qobuz.pacing.dailyCap"] ?? ""}
|
||||||
|
onChange={(e) => setPacing("qobuz.pacing.dailyCap", e.target.value)} />
|
||||||
|
</label>
|
||||||
|
<label className="field">
|
||||||
|
<span>New-account start (warm-up)</span>
|
||||||
|
<input type="date" aria-label="qobuz warmup start date"
|
||||||
|
value={pacingCfg["qobuz.pacing.warmupStartDate"] ?? ""}
|
||||||
|
onChange={(e) => setPacing("qobuz.pacing.warmupStartDate", e.target.value)} />
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
<div className="save-row">
|
||||||
|
<button type="submit" className="btn accent">Save</button>
|
||||||
|
{pacingSaved ? <span className="saved-note">Saved</span> : null}
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</>
|
||||||
) : null}
|
) : null}
|
||||||
|
|
||||||
{tab === "Soulseek" ? (
|
{tab === "Soulseek" ? (
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import time
|
||||||
from difflib import SequenceMatcher
|
from difflib import SequenceMatcher
|
||||||
|
|
||||||
from lyra_worker.types import MBTarget
|
from lyra_worker.types import MBTarget
|
||||||
@@ -5,6 +6,29 @@ from lyra_worker.types import MBTarget
|
|||||||
_MIN_TITLE_RATIO = 0.6
|
_MIN_TITLE_RATIO = 0.6
|
||||||
|
|
||||||
|
|
||||||
|
def _with_retry(fn, *, attempts: int = 4, base_delay: float = 1.0, sleep=time.sleep):
|
||||||
|
"""Call `fn`, retrying transient MusicBrainz failures with exponential backoff.
|
||||||
|
|
||||||
|
A dropped TLS connection to the MusicBrainz webservice surfaces as
|
||||||
|
musicbrainzngs.NetworkError (`<urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING]>`),
|
||||||
|
and MusicBrainz's own 503 rate-limit surfaces as ResponseError; both are
|
||||||
|
transient, so we retry rather than failing the whole pipeline for the job.
|
||||||
|
Any other exception (bug, non-transient) propagates immediately.
|
||||||
|
|
||||||
|
musicbrainzngs is imported lazily so importing this module stays offline.
|
||||||
|
"""
|
||||||
|
import musicbrainzngs
|
||||||
|
|
||||||
|
transient = (musicbrainzngs.NetworkError, musicbrainzngs.ResponseError)
|
||||||
|
for i in range(attempts):
|
||||||
|
try:
|
||||||
|
return fn()
|
||||||
|
except transient:
|
||||||
|
if i == attempts - 1:
|
||||||
|
raise
|
||||||
|
sleep(base_delay * (2**i))
|
||||||
|
|
||||||
|
|
||||||
def _ratio(a: str, b: str) -> float:
|
def _ratio(a: str, b: str) -> float:
|
||||||
return SequenceMatcher(None, a.strip().casefold(), b.strip().casefold()).ratio()
|
return SequenceMatcher(None, a.strip().casefold(), b.strip().casefold()).ratio()
|
||||||
|
|
||||||
@@ -13,6 +37,12 @@ def _is_album(g: dict) -> bool:
|
|||||||
return (g.get("primary-type") or "").casefold() == "album"
|
return (g.get("primary-type") or "").casefold() == "album"
|
||||||
|
|
||||||
|
|
||||||
|
def _rg_year(g: dict) -> int:
|
||||||
|
"""First-release year of a release-group, or 9999 when unknown (so undated groups lose ties)."""
|
||||||
|
frd = (g.get("first-release-date") or "")[:4]
|
||||||
|
return int(frd) if frd.isdigit() else 9999
|
||||||
|
|
||||||
|
|
||||||
def _credited_artist(g: dict) -> str:
|
def _credited_artist(g: dict) -> str:
|
||||||
"""Primary credited artist name of a release-group search result, or '' if absent."""
|
"""Primary credited artist name of a release-group search result, or '' if absent."""
|
||||||
credit = g.get("artist-credit")
|
credit = g.get("artist-credit")
|
||||||
@@ -32,7 +62,12 @@ def _best_release_group(artist: str, album: str, groups: list) -> dict | None:
|
|||||||
* title similarity is next;
|
* title similarity is next;
|
||||||
* an Album primary-type only breaks ties *within* the same artist+title, so a famous
|
* an Album primary-type only breaks ties *within* the same artist+title, so a famous
|
||||||
album (Michael Jackson's "Off the Wall") still isn't resolved to its same-named
|
album (Michael Jackson's "Off the Wall") still isn't resolved to its same-named
|
||||||
single whose short tracklist would map positionally onto the album's files.
|
single whose short tracklist would map positionally onto the album's files;
|
||||||
|
* finally, the EARLIEST release-group wins — MusicBrainz lists the original album and its
|
||||||
|
later reissues/anniversary editions as separate same-titled groups (Linkin Park "Hybrid
|
||||||
|
Theory": a 2000 original and a 2023 13-track reissue), which tie on everything above, so
|
||||||
|
MB's arbitrary relevance order used to pick a reissue whose longer tracklist/runtime then
|
||||||
|
failed completeness checks against the standard album a source delivers. Prefer the original.
|
||||||
Artist is a ranking signal, never a hard filter — a low match sinks a candidate but
|
Artist is a ranking signal, never a hard filter — a low match sinks a candidate but
|
||||||
never drops the release, so credited-name variations (feat., punctuation) still resolve.
|
never drops the release, so credited-name variations (feat., punctuation) still resolve.
|
||||||
Returns None below the title threshold. Pure — no network I/O."""
|
Returns None below the title threshold. Pure — no network I/O."""
|
||||||
@@ -44,6 +79,7 @@ def _best_release_group(artist: str, album: str, groups: list) -> dict | None:
|
|||||||
_ratio(artist, _credited_artist(g)),
|
_ratio(artist, _credited_artist(g)),
|
||||||
_ratio(album, g.get("title", "")),
|
_ratio(album, g.get("title", "")),
|
||||||
_is_album(g),
|
_is_album(g),
|
||||||
|
-_rg_year(g), # tie-break: the original (earliest) group over later reissues
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
if _ratio(album, best.get("title", "")) < _MIN_TITLE_RATIO:
|
if _ratio(album, best.get("title", "")) < _MIN_TITLE_RATIO:
|
||||||
@@ -51,6 +87,23 @@ def _best_release_group(artist: str, album: str, groups: list) -> dict | None:
|
|||||||
return best
|
return best
|
||||||
|
|
||||||
|
|
||||||
|
def _pick_release(releases: list) -> dict | None:
|
||||||
|
"""Choose which release of a release-group to read the tracklist from. MB returns them in an
|
||||||
|
arbitrary order and ``releases[0]`` is often a later reissue/deluxe/anniversary edition with
|
||||||
|
more tracks and a longer runtime than the standard album a source actually delivers — which
|
||||||
|
then makes a *complete* download fail the pipeline's track-count/duration completeness checks
|
||||||
|
and re-download forever (Linkin Park "Hybrid Theory": MB's arbitrary pick was a 2023 13-track
|
||||||
|
reissue at ~52 min vs the 2000 original's 12 tracks / ~38 min). Prefer the ORIGINAL standard
|
||||||
|
edition: an Official release, then the earliest date, then a dated one. Pure — unit-tested."""
|
||||||
|
if not releases:
|
||||||
|
return None
|
||||||
|
def _key(r: dict) -> tuple:
|
||||||
|
official = 0 if (r.get("status") == "Official") else 1 # official editions first
|
||||||
|
date = r.get("date") or ""
|
||||||
|
return (official, 0 if date else 1, date) # then dated, then earliest date
|
||||||
|
return min(releases, key=_key)
|
||||||
|
|
||||||
|
|
||||||
class MusicBrainzResolver:
|
class MusicBrainzResolver:
|
||||||
"""Real MbResolver using the MusicBrainz webservice via musicbrainzngs.
|
"""Real MbResolver using the MusicBrainz webservice via musicbrainzngs.
|
||||||
|
|
||||||
@@ -68,7 +121,7 @@ class MusicBrainzResolver:
|
|||||||
|
|
||||||
musicbrainzngs.set_useragent(self._app, self._version, self._contact)
|
musicbrainzngs.set_useragent(self._app, self._version, self._contact)
|
||||||
|
|
||||||
res = musicbrainzngs.search_release_groups(query=album, artist=artist, limit=5)
|
res = _with_retry(lambda: musicbrainzngs.search_release_groups(query=album, artist=artist, limit=5))
|
||||||
groups = res.get("release-group-list", [])
|
groups = res.get("release-group-list", [])
|
||||||
rg = _best_release_group(artist, album, groups)
|
rg = _best_release_group(artist, album, groups)
|
||||||
if rg is None:
|
if rg is None:
|
||||||
@@ -93,11 +146,12 @@ class MusicBrainzResolver:
|
|||||||
titles: tuple[str, ...] = ()
|
titles: tuple[str, ...] = ()
|
||||||
rgid = rg.get("id")
|
rgid = rg.get("id")
|
||||||
if rgid:
|
if rgid:
|
||||||
rgfull = musicbrainzngs.get_release_group_by_id(rgid, includes=["releases"])
|
rgfull = _with_retry(lambda: musicbrainzngs.get_release_group_by_id(rgid, includes=["releases"]))
|
||||||
releases = rgfull.get("release-group", {}).get("release-list", [])
|
releases = rgfull.get("release-group", {}).get("release-list", [])
|
||||||
if releases:
|
chosen = _pick_release(releases)
|
||||||
rel = musicbrainzngs.get_release_by_id(
|
if chosen:
|
||||||
releases[0]["id"], includes=["recordings"]
|
rel = _with_retry(
|
||||||
|
lambda: musicbrainzngs.get_release_by_id(chosen["id"], includes=["recordings"])
|
||||||
).get("release", {})
|
).get("release", {})
|
||||||
tracks = []
|
tracks = []
|
||||||
for medium in rel.get("medium-list", []):
|
for medium in rel.get("medium-list", []):
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ import json
|
|||||||
import os
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import time
|
import time
|
||||||
|
import unicodedata
|
||||||
from typing import Callable
|
from typing import Callable
|
||||||
|
|
||||||
import requests
|
import requests
|
||||||
@@ -10,9 +11,26 @@ _DEFAULT_DOWNLOADS_ROOT = "/slskd-downloads"
|
|||||||
|
|
||||||
_AUDIO_EXT = {"flac", "mp3", "m4a", "ogg", "opus", "wav", "aac"}
|
_AUDIO_EXT = {"flac", "mp3", "m4a", "ogg", "opus", "wav", "aac"}
|
||||||
_LOSSLESS_EXT = {"flac", "wav"}
|
_LOSSLESS_EXT = {"flac", "wav"}
|
||||||
_SEARCH_POLLS = 40 # * 3s ≈ 2 min
|
_SEARCH_POLLS = 40 # * 3s ≈ 2 min
|
||||||
_XFER_POLLS = 200 # * 3s ≈ 10 min
|
_STALL_POLLS = 30 # * 3s ≈ 90s of no byte progress → abandon a queued/dead peer and fall through
|
||||||
_POLL_SECONDS = 3
|
_POLL_SECONDS = 3
|
||||||
|
# Absolute backstop for a slow-but-progressing peer. Scaled to the album's total bytes (a flat
|
||||||
|
# cap starved big albums: a 500MB set from a ~400KB/s serial peer needs ~60-70min and was cut at
|
||||||
|
# a flat 60min, then re-downloaded from scratch — never finishing). Budget at a conservative floor
|
||||||
|
# throughput, clamped. The 90s stall timeout still abandons a truly dead/queued peer quickly.
|
||||||
|
_MIN_XFER_POLLS = 400 # * 3s ≈ 20 min floor (small albums)
|
||||||
|
_MAX_XFER_POLLS = 2400 # * 3s ≈ 2 h ceiling (bounds worst-case slot hold)
|
||||||
|
_BACKSTOP_FLOOR_BPS = 64_000 # budget the backstop assuming >= ~64 KB/s sustained
|
||||||
|
|
||||||
|
|
||||||
|
def _max_polls(total_bytes: int) -> int:
|
||||||
|
"""Poll budget (each _POLL_SECONDS) for the absolute download backstop, scaled to album size at
|
||||||
|
a conservative floor throughput so a large slow-but-healthy transfer isn't abandoned mid-flight.
|
||||||
|
Clamped to [_MIN_XFER_POLLS, _MAX_XFER_POLLS]. Pure — unit-tested."""
|
||||||
|
if total_bytes <= 0:
|
||||||
|
return _MIN_XFER_POLLS
|
||||||
|
budget = int(total_bytes / (_BACKSTOP_FLOOR_BPS * _POLL_SECONDS))
|
||||||
|
return max(_MIN_XFER_POLLS, min(budget, _MAX_XFER_POLLS))
|
||||||
|
|
||||||
|
|
||||||
def _basename(path: str) -> str:
|
def _basename(path: str) -> str:
|
||||||
@@ -27,6 +45,114 @@ def _dirname(path: str) -> str:
|
|||||||
return ""
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
_ETA_CAP_SECONDS = 359_999 # ~100h; keeps the value well inside Job.downloadEtaSeconds (int4)
|
||||||
|
|
||||||
|
|
||||||
|
def _eta_seconds(total_bytes: int, bytes_now: int, speed_bps: float) -> int | None:
|
||||||
|
"""Seconds until the transfer finishes at the current measured rate, or None when it can't be
|
||||||
|
estimated (no throughput sample yet, unknown total, or already complete). Pure — unit-tested.
|
||||||
|
A near-stalled peer drives speed_bps toward zero, so the estimate is capped: an uncapped value
|
||||||
|
can exceed int4 and overflow the downloadEtaSeconds column write (poisoning the pipeline txn)."""
|
||||||
|
if speed_bps <= 0 or total_bytes <= 0 or bytes_now >= total_bytes:
|
||||||
|
return None
|
||||||
|
return min(int((total_bytes - bytes_now) / speed_bps), _ETA_CAP_SECONDS)
|
||||||
|
|
||||||
|
|
||||||
|
def _norm(s: str) -> str:
|
||||||
|
"""Lowercase, strip diacritics, and reduce to alphanumeric tokens, so 'Beyoncé' matches a
|
||||||
|
peer's 'Beyonce' folder and punctuation ('I Am… Sasha Fierce') doesn't defeat the match."""
|
||||||
|
s = unicodedata.normalize("NFKD", s or "")
|
||||||
|
s = "".join(ch for ch in s if not unicodedata.combining(ch))
|
||||||
|
s = "".join(ch if ch.isalnum() else " " for ch in s.lower())
|
||||||
|
return " ".join(s.split())
|
||||||
|
|
||||||
|
|
||||||
|
def _path_matches(needle: str, path: str) -> bool:
|
||||||
|
"""True if `needle` (an artist or album name) appears in a file path, tolerant of accents and
|
||||||
|
punctuation. Every significant token (≥3 chars) must be present; for a very short needle
|
||||||
|
(e.g. 'U2') the whole normalized string must appear."""
|
||||||
|
npath = _norm(path)
|
||||||
|
tokens = [t for t in _norm(needle).split() if len(t) >= 3]
|
||||||
|
if not tokens:
|
||||||
|
whole = _norm(needle)
|
||||||
|
return bool(whole) and whole in npath
|
||||||
|
return all(t in npath for t in tokens)
|
||||||
|
|
||||||
|
|
||||||
|
def _keep_matching(responses: list, needle: str) -> list:
|
||||||
|
"""Drop files — then now-empty responses — whose path doesn't match `needle`. Makes an album-
|
||||||
|
or artist-only fallback search precise so we never grab a same-titled release by a different
|
||||||
|
artist. Pure — unit-tested offline."""
|
||||||
|
out = []
|
||||||
|
for resp in responses:
|
||||||
|
files = [f for f in (resp.get("files") or []) if _path_matches(needle, f.get("filename", ""))]
|
||||||
|
if files:
|
||||||
|
out.append({**resp, "files": files})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_search_responses(responses: list, artist: str = "") -> list[dict]:
|
||||||
|
"""Turn slskd search responses into album candidates (one per peer+directory), ordered so the
|
||||||
|
peers that will FINISH first come first: a free upload slot, then the shortest estimated
|
||||||
|
transfer time (album bytes ÷ the peer's advertised speed), then a shorter queue. Ranking on
|
||||||
|
estimated time — not raw speed — matters because Soulseek FLAC rips of the same album vary
|
||||||
|
~2x in size (a 24-bit/bloated rip vs a standard CD rip), and Lyra scores them the SAME quality
|
||||||
|
class (the search exposes no bit-depth), so a peer advertising high speed but serving huge
|
||||||
|
files can lose a race — and blow past the download backstop — against a peer with a smaller
|
||||||
|
standard rip. The pipeline tries candidates in this order, so this is what makes the Soulseek
|
||||||
|
fall-through converge on a copy that actually completes. Pure — no I/O — unit-tested offline."""
|
||||||
|
scored: list[tuple] = []
|
||||||
|
for resp in responses:
|
||||||
|
username = resp.get("username", "")
|
||||||
|
has_slot = 0 if resp.get("hasFreeUploadSlot") else 1 # 0 sorts first (free slot = starts now)
|
||||||
|
speed = int(resp.get("uploadSpeed") or 0)
|
||||||
|
queue = int(resp.get("queueLength") or 0)
|
||||||
|
by_dir: dict[str, list] = {}
|
||||||
|
for f in resp.get("files", []) or []:
|
||||||
|
name = f.get("filename", "")
|
||||||
|
ext = name.rsplit(".", 1)[-1].lower() if "." in name else ""
|
||||||
|
if ext not in _AUDIO_EXT:
|
||||||
|
continue
|
||||||
|
by_dir.setdefault(_dirname(name), []).append(
|
||||||
|
{"filename": name, "size": f.get("size", 0), "ext": ext}
|
||||||
|
)
|
||||||
|
for directory, dfiles in by_dir.items():
|
||||||
|
if not dfiles:
|
||||||
|
continue
|
||||||
|
# Estimated seconds to pull this album from this peer at its advertised rate. Unknown/
|
||||||
|
# zero speed → a low nominal (1 B/s) so total size still orders those peers last.
|
||||||
|
total_bytes = sum(int(f["size"] or 0) for f in dfiles)
|
||||||
|
est_seconds = total_bytes / (speed if speed > 0 else 1)
|
||||||
|
lossless = all(f["ext"] in _LOSSLESS_EXT for f in dfiles)
|
||||||
|
dirname = _basename(directory)
|
||||||
|
# Trust the REQUESTED artist when it appears anywhere in the path — many rips put the
|
||||||
|
# artist in a parent folder ("Rihanna\(2009) Rated R\…") or bury it ("2009 - Rihanna -
|
||||||
|
# Rated R"), which the "Artist - Album" folder parse gets wrong; that tanked the
|
||||||
|
# confidence score and lost real FLACs to clean-named MP3s. album stays the folder name
|
||||||
|
# (fuzzy-matched downstream); artist is what the parse gets wrong.
|
||||||
|
if artist and _path_matches(artist, dfiles[0]["filename"]):
|
||||||
|
guess_artist, guess_album = artist, dirname
|
||||||
|
elif " - " in dirname:
|
||||||
|
guess_artist, guess_album = (p.strip() for p in dirname.split(" - ", 1))
|
||||||
|
else:
|
||||||
|
guess_artist, guess_album = "", dirname
|
||||||
|
candidate = {
|
||||||
|
"source_ref": json.dumps({
|
||||||
|
"username": username,
|
||||||
|
"files": [{"filename": f["filename"], "size": f["size"]} for f in dfiles],
|
||||||
|
}),
|
||||||
|
"title": guess_album,
|
||||||
|
"artist": guess_artist,
|
||||||
|
"track_count": len(dfiles),
|
||||||
|
"format": "FLAC" if lossless else "MP3",
|
||||||
|
"bitrate": None,
|
||||||
|
}
|
||||||
|
scored.append((has_slot, est_seconds, queue, candidate))
|
||||||
|
# ascending: free slot first, then shortest estimated transfer, then shortest queue
|
||||||
|
scored.sort(key=lambda t: (t[0], t[1], t[2]))
|
||||||
|
return [c for *_rest, c in scored]
|
||||||
|
|
||||||
|
|
||||||
class SlskdClient:
|
class SlskdClient:
|
||||||
"""Real Soulseek client talking to an slskd daemon's REST API.
|
"""Real Soulseek client talking to an slskd daemon's REST API.
|
||||||
|
|
||||||
@@ -58,11 +184,12 @@ class SlskdClient:
|
|||||||
r.raise_for_status()
|
r.raise_for_status()
|
||||||
return r.json()
|
return r.json()
|
||||||
|
|
||||||
def search_album(self, artist: str, album: str) -> list[dict]:
|
def _run_search(self, text: str) -> list:
|
||||||
|
"""POST a search, wait for it to complete, and return the raw responses (a list)."""
|
||||||
r = requests.post(
|
r = requests.post(
|
||||||
f"{self._url}/api/v0/searches",
|
f"{self._url}/api/v0/searches",
|
||||||
headers=self._headers(),
|
headers=self._headers(),
|
||||||
json={"searchText": f"{artist} {album}"},
|
json={"searchText": text},
|
||||||
timeout=30,
|
timeout=30,
|
||||||
)
|
)
|
||||||
r.raise_for_status()
|
r.raise_for_status()
|
||||||
@@ -75,43 +202,27 @@ class SlskdClient:
|
|||||||
break
|
break
|
||||||
|
|
||||||
responses = self._get(f"/api/v0/searches/{search_id}/responses")
|
responses = self._get(f"/api/v0/searches/{search_id}/responses")
|
||||||
candidates: list[dict] = []
|
return responses if isinstance(responses, list) else []
|
||||||
for resp in responses:
|
|
||||||
username = resp.get("username", "")
|
def search_album(self, artist: str, album: str) -> list[dict]:
|
||||||
by_dir: dict[str, list] = {}
|
responses = self._run_search(f"{artist} {album}")
|
||||||
for f in resp.get("files", []) or []:
|
if not responses:
|
||||||
name = f.get("filename", "")
|
# The Soulseek server silently drops searches containing certain blocklisted terms
|
||||||
ext = name.rsplit(".", 1)[-1].lower() if "." in name else ""
|
# (major-label artists / album titles like Adele, Rihanna, Beyoncé, "Lemonade") →
|
||||||
if ext not in _AUDIO_EXT:
|
# 0 responses. The blocked term poisons the whole query, but the OTHER field usually
|
||||||
|
# isn't blocked, so retry with each field alone (normalized) and keep only results
|
||||||
|
# whose path still matches the field we dropped. That recovers e.g. Beyoncé's
|
||||||
|
# "I Am… Sasha Fierce" via the album title, without ever grabbing a same-titled album
|
||||||
|
# by someone else.
|
||||||
|
for query, needle in ((album, artist), (artist, album)):
|
||||||
|
q = _norm(query)
|
||||||
|
if not q:
|
||||||
continue
|
continue
|
||||||
by_dir.setdefault(_dirname(name), []).append(
|
alt = _keep_matching(self._run_search(q), needle)
|
||||||
{"filename": name, "size": f.get("size", 0), "ext": ext}
|
if alt:
|
||||||
)
|
responses = alt
|
||||||
for directory, dfiles in by_dir.items():
|
break
|
||||||
if not dfiles:
|
return _parse_search_responses(responses, artist)
|
||||||
continue
|
|
||||||
lossless = all(f["ext"] in _LOSSLESS_EXT for f in dfiles)
|
|
||||||
dirname = _basename(directory)
|
|
||||||
if " - " in dirname:
|
|
||||||
guess_artist, guess_album = (p.strip() for p in dirname.split(" - ", 1))
|
|
||||||
else:
|
|
||||||
guess_artist, guess_album = "", dirname
|
|
||||||
candidates.append(
|
|
||||||
{
|
|
||||||
"source_ref": json.dumps(
|
|
||||||
{
|
|
||||||
"username": username,
|
|
||||||
"files": [{"filename": f["filename"], "size": f["size"]} for f in dfiles],
|
|
||||||
}
|
|
||||||
),
|
|
||||||
"title": guess_album,
|
|
||||||
"artist": guess_artist,
|
|
||||||
"track_count": len(dfiles),
|
|
||||||
"format": "FLAC" if lossless else "MP3",
|
|
||||||
"bitrate": None,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
return candidates
|
|
||||||
|
|
||||||
def download(self, source_ref: str, dest: str, on_progress: Callable[[float], None]) -> dict:
|
def download(self, source_ref: str, dest: str, on_progress: Callable[[float], None]) -> dict:
|
||||||
ref = json.loads(source_ref)
|
ref = json.loads(source_ref)
|
||||||
@@ -119,7 +230,17 @@ class SlskdClient:
|
|||||||
files = ref["files"]
|
files = ref["files"]
|
||||||
wanted = {f["filename"] for f in files}
|
wanted = {f["filename"] for f in files}
|
||||||
os.makedirs(dest, exist_ok=True)
|
os.makedirs(dest, exist_ok=True)
|
||||||
|
xfer_ids: set[str] = set() # slskd transfer ids we started, for cancel-on-abandon
|
||||||
|
try:
|
||||||
|
return self._download(username, files, wanted, dest, on_progress, xfer_ids)
|
||||||
|
except Exception:
|
||||||
|
# Abandoning this peer (timeout / error / the pipeline moving on): cancel the transfers
|
||||||
|
# we enqueued so they don't keep running and pile up in slskd as the job falls through
|
||||||
|
# to other peers/sources for the same album.
|
||||||
|
self._cancel_transfers(username, xfer_ids)
|
||||||
|
raise
|
||||||
|
|
||||||
|
def _download(self, username, files, wanted, dest, on_progress, xfer_ids: set) -> dict:
|
||||||
r = requests.post(
|
r = requests.post(
|
||||||
f"{self._url}/api/v0/transfers/downloads/{username}",
|
f"{self._url}/api/v0/transfers/downloads/{username}",
|
||||||
headers=self._headers(),
|
headers=self._headers(),
|
||||||
@@ -130,23 +251,49 @@ class SlskdClient:
|
|||||||
on_progress(0.0)
|
on_progress(0.0)
|
||||||
|
|
||||||
total = len(files)
|
total = len(files)
|
||||||
|
total_bytes = sum(int(f.get("size") or 0) for f in files)
|
||||||
completed = False
|
completed = False
|
||||||
for _ in range(_XFER_POLLS):
|
last_bytes = -1
|
||||||
|
stall = 0
|
||||||
|
speed = 0.0 # bytes/sec, EWMA-smoothed so the ETA doesn't jitter poll to poll
|
||||||
|
prev_bytes = 0
|
||||||
|
prev_t = time.monotonic()
|
||||||
|
for _ in range(_max_polls(total_bytes)):
|
||||||
time.sleep(_POLL_SECONDS)
|
time.sleep(_POLL_SECONDS)
|
||||||
data = self._get(f"/api/v0/transfers/downloads/{username}")
|
data = self._get(f"/api/v0/transfers/downloads/{username}")
|
||||||
states: list[str] = []
|
states: list[str] = []
|
||||||
|
bytes_now = 0
|
||||||
for directory in data.get("directories", []) if isinstance(data, dict) else []:
|
for directory in data.get("directories", []) if isinstance(data, dict) else []:
|
||||||
for f in directory.get("files", []):
|
for f in directory.get("files", []):
|
||||||
if f.get("filename") in wanted: # only the files we enqueued
|
if f.get("filename") in wanted: # only the files we enqueued
|
||||||
states.append(str(f.get("state", "")))
|
states.append(str(f.get("state", "")))
|
||||||
|
bytes_now += int(f.get("bytesTransferred") or 0)
|
||||||
|
if f.get("id"): # remember the transfer so we can cancel it if abandoned
|
||||||
|
xfer_ids.add(str(f["id"]))
|
||||||
if states:
|
if states:
|
||||||
if any(bad in s for s in states for bad in ("Errored", "Rejected", "Cancelled")):
|
if any(bad in s for s in states for bad in ("Errored", "Rejected", "Cancelled")):
|
||||||
raise RuntimeError("soulseek transfer failed")
|
raise RuntimeError("soulseek transfer failed")
|
||||||
|
now = time.monotonic()
|
||||||
|
dt = now - prev_t
|
||||||
|
if dt > 0 and bytes_now >= prev_bytes:
|
||||||
|
inst = (bytes_now - prev_bytes) / dt
|
||||||
|
speed = inst if speed == 0.0 else 0.6 * speed + 0.4 * inst
|
||||||
|
prev_bytes, prev_t = bytes_now, now
|
||||||
done = sum(1 for s in states if "Completed" in s)
|
done = sum(1 for s in states if "Completed" in s)
|
||||||
on_progress(min(1.0, done / max(total, 1)))
|
on_progress(min(1.0, done / max(total, 1)), _eta_seconds(total_bytes, bytes_now, speed))
|
||||||
if done >= total:
|
if done >= total:
|
||||||
completed = True
|
completed = True
|
||||||
break
|
break
|
||||||
|
# Stall detection: a peer making byte progress keeps its slot (up to the absolute
|
||||||
|
# backstop above) so a slow-but-working transfer can finish; one that sends no bytes
|
||||||
|
# for _STALL_POLLS (~90s) is queued/dead — abandon it fast and fall through.
|
||||||
|
if bytes_now > last_bytes:
|
||||||
|
last_bytes = bytes_now
|
||||||
|
stall = 0
|
||||||
|
else:
|
||||||
|
stall += 1
|
||||||
|
if stall >= _STALL_POLLS:
|
||||||
|
raise TimeoutError(f"soulseek transfer stalled for {username}")
|
||||||
|
|
||||||
if not completed:
|
if not completed:
|
||||||
raise TimeoutError(f"soulseek download did not complete for {username}")
|
raise TimeoutError(f"soulseek download did not complete for {username}")
|
||||||
@@ -162,6 +309,19 @@ class SlskdClient:
|
|||||||
)
|
)
|
||||||
return {"track_count": moved, "path": dest}
|
return {"track_count": moved, "path": dest}
|
||||||
|
|
||||||
|
def _cancel_transfers(self, username: str, xfer_ids: set) -> None:
|
||||||
|
"""Cancel the given in-flight slskd transfers for a peer. Best-effort: a cleanup failure
|
||||||
|
must never mask the original download failure."""
|
||||||
|
for tid in xfer_ids:
|
||||||
|
try:
|
||||||
|
requests.delete(
|
||||||
|
f"{self._url}/api/v0/transfers/downloads/{username}/{tid}",
|
||||||
|
headers=self._headers(),
|
||||||
|
timeout=15,
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
def _retrieve(self, files: list[dict], dest: str) -> int:
|
def _retrieve(self, files: list[dict], dest: str) -> int:
|
||||||
"""Move the just-finished slskd files out of its downloads dir into `dest`.
|
"""Move the just-finished slskd files out of its downloads dir into `dest`.
|
||||||
|
|
||||||
|
|||||||
@@ -24,6 +24,43 @@ def reclaim_stuck_jobs(conn: psycopg.Connection) -> int:
|
|||||||
return n
|
return n
|
||||||
|
|
||||||
|
|
||||||
|
_MAX_ATTEMPTS = 5
|
||||||
|
|
||||||
|
|
||||||
|
def requeue_or_fail(
|
||||||
|
conn: psycopg.Connection, job_id: str, error: str, max_attempts: int = _MAX_ATTEMPTS
|
||||||
|
) -> None:
|
||||||
|
"""Recover a job whose pipeline raised an *unhandled* exception (e.g. a dropped MusicBrainz
|
||||||
|
TLS connection surfacing as an SSL EOF). Without this the job is left in its mid-pipeline
|
||||||
|
'matching' state; since claim_next only picks 'requested' it is orphaned until the next
|
||||||
|
restart's reclaim_stuck_jobs — showing "Searching sources…" on the press indefinitely.
|
||||||
|
|
||||||
|
Such errors are usually transient, so requeue to 'requested' until the attempt cap, then fall
|
||||||
|
to 'needs_attention' so a persistently-failing job stays visible instead of looping forever.
|
||||||
|
`attempts` is bumped by claim_next on every claim, so it already counts tries."""
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT attempts, "requestId" FROM "Job" WHERE id = %s', (job_id,))
|
||||||
|
row = cur.fetchone()
|
||||||
|
if row is None:
|
||||||
|
return
|
||||||
|
attempts, request_id = row
|
||||||
|
if attempts >= max_attempts:
|
||||||
|
cur.execute(
|
||||||
|
'UPDATE "Job" SET state = \'needs_attention\', error = %s, "claimedAt" = NULL, '
|
||||||
|
'"updatedAt" = now() WHERE id = %s',
|
||||||
|
(error, job_id),
|
||||||
|
)
|
||||||
|
cur.execute('UPDATE "Request" SET status = \'needs_attention\' WHERE id = %s', (request_id,))
|
||||||
|
else:
|
||||||
|
cur.execute(
|
||||||
|
'UPDATE "Job" SET state = \'requested\', "currentStage" = \'intake\', error = NULL, '
|
||||||
|
'"claimedAt" = NULL, "downloadProgress" = 0, "updatedAt" = now() WHERE id = %s',
|
||||||
|
(job_id,),
|
||||||
|
)
|
||||||
|
cur.execute('UPDATE "Request" SET status = \'pending\' WHERE id = %s', (request_id,))
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
def claim_next(conn: psycopg.Connection) -> str | None:
|
def claim_next(conn: psycopg.Connection) -> str | None:
|
||||||
"""Atomically claim the oldest queued job. Returns its id or None."""
|
"""Atomically claim the oldest queued job. Returns its id or None."""
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ import uuid
|
|||||||
|
|
||||||
import psycopg
|
import psycopg
|
||||||
|
|
||||||
from lyra_worker.claim import claim_next, reclaim_stuck_jobs
|
from lyra_worker.claim import claim_next, reclaim_stuck_jobs, requeue_or_fail
|
||||||
from lyra_worker.config import get_config
|
from lyra_worker.config import get_config
|
||||||
from lyra_worker.crypto import secret_key_problem
|
from lyra_worker.crypto import secret_key_problem
|
||||||
from lyra_worker.db import wait_for_db
|
from lyra_worker.db import wait_for_db
|
||||||
@@ -323,6 +323,9 @@ def run_forever() -> None:
|
|||||||
except Exception as e: # one bad job must not take down the worker loop
|
except Exception as e: # one bad job must not take down the worker loop
|
||||||
print(f"worker: pipeline failed for job {job_id}: {e}", flush=True)
|
print(f"worker: pipeline failed for job {job_id}: {e}", flush=True)
|
||||||
conn.rollback()
|
conn.rollback()
|
||||||
|
# Don't orphan the job in its mid-pipeline 'matching' state (claim_next only
|
||||||
|
# picks 'requested'): requeue for a bounded retry, then needs_attention.
|
||||||
|
requeue_or_fail(conn, job_id, str(e))
|
||||||
finish_job(conn, job_id, mcfg)
|
finish_job(conn, job_id, mcfg)
|
||||||
print(f"worker: finished job {job_id}", flush=True)
|
print(f"worker: finished job {job_id}", flush=True)
|
||||||
next_job_allowed = time.monotonic() + job_delay_seconds(config)
|
next_job_allowed = time.monotonic() + job_delay_seconds(config)
|
||||||
|
|||||||
@@ -86,16 +86,40 @@ def _set_state(conn: psycopg.Connection, rel_id: str, state: str) -> None:
|
|||||||
cur.execute('UPDATE "MonitoredRelease" SET state = %s WHERE id = %s', (state, rel_id))
|
cur.execute('UPDATE "MonitoredRelease" SET state = %s WHERE id = %s', (state, rel_id))
|
||||||
|
|
||||||
|
|
||||||
def _in_library_at_cutoff(conn: psycopg.Connection, artist: str, album: str, cutoff: int) -> bool:
|
def _in_library_at_cutoff(
|
||||||
|
conn: psycopg.Connection, artist: str, album: str, rg_mbid: str, cutoff: int
|
||||||
|
) -> bool:
|
||||||
|
"""True if the album is owned at/above `cutoff`. Matches on the release-group MBID when the
|
||||||
|
library row has one — robust against MusicBrainz artist-name canonicalization (e.g. the
|
||||||
|
followed 'Kanye West' imports as 'Ye') — and falls back to exact artist+album for scanned
|
||||||
|
rows that predate MBID capture."""
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
'SELECT 1 FROM "LibraryItem" WHERE artist = %s AND album = %s '
|
'SELECT 1 FROM "LibraryItem" WHERE "qualityClass" >= %s AND ('
|
||||||
'AND "qualityClass" >= %s LIMIT 1',
|
' ("rgMbid" IS NOT NULL AND "rgMbid" = %s) '
|
||||||
(artist, album, cutoff),
|
' OR ("rgMbid" IS NULL AND artist = %s AND album = %s)) LIMIT 1',
|
||||||
|
(cutoff, rg_mbid, artist, album),
|
||||||
)
|
)
|
||||||
return cur.fetchone() is not None
|
return cur.fetchone() is not None
|
||||||
|
|
||||||
|
|
||||||
|
def _retire_failed_attempts(conn: psycopg.Connection, rel_id: str) -> None:
|
||||||
|
"""Delete prior failed (needs_attention) Request rows for this release (Job/Candidate cascade).
|
||||||
|
|
||||||
|
A hard-to-match release is re-pressed every retry interval; each failed attempt used to leave
|
||||||
|
a needs_attention Request behind, piling up phantom "needs attention" cards on the press even
|
||||||
|
after a later attempt imported the album. Retiring them when the release is fulfilled or
|
||||||
|
re-attempted keeps the press showing one live attempt at a time. Manual (non-monitor) requests
|
||||||
|
have no monitoredReleaseId and are never touched."""
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'DELETE FROM "Request" r USING "Job" j '
|
||||||
|
'WHERE j."requestId" = r.id AND r."monitoredReleaseId" = %s '
|
||||||
|
"AND j.state = 'needs_attention'",
|
||||||
|
(rel_id,),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _enqueue_one(conn: psycopg.Connection, rel_id: str, artist: str, album: str) -> None:
|
def _enqueue_one(conn: psycopg.Connection, rel_id: str, artist: str, album: str) -> None:
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
@@ -118,7 +142,8 @@ def enqueue_due(conn: psycopg.Connection, cfg: MonitorConfig) -> int:
|
|||||||
cur.execute(
|
cur.execute(
|
||||||
'SELECT mr.id, mr."artistName", mr.album, mr.state, mr."currentQualityClass", '
|
'SELECT mr.id, mr."artistName", mr.album, mr.state, mr."currentQualityClass", '
|
||||||
' (mr."firstGrabbedAt" IS NOT NULL '
|
' (mr."firstGrabbedAt" IS NOT NULL '
|
||||||
' AND mr."firstGrabbedAt" < now() - make_interval(days => %s)) AS window_expired '
|
' AND mr."firstGrabbedAt" < now() - make_interval(days => %s)) AS window_expired, '
|
||||||
|
' mr."rgMbid" '
|
||||||
'FROM "MonitoredRelease" mr '
|
'FROM "MonitoredRelease" mr '
|
||||||
'WHERE mr.monitored = true '
|
'WHERE mr.monitored = true '
|
||||||
' AND NOT mr.ignored ' # user-ignored releases are never enqueued
|
' AND NOT mr.ignored ' # user-ignored releases are never enqueued
|
||||||
@@ -134,8 +159,11 @@ def enqueue_due(conn: psycopg.Connection, cfg: MonitorConfig) -> int:
|
|||||||
rows = cur.fetchall()
|
rows = cur.fetchall()
|
||||||
|
|
||||||
enqueued = 0
|
enqueued = 0
|
||||||
for rel_id, artist, album, state, quality, window_expired in rows:
|
for rel_id, artist, album, state, quality, window_expired, rg_mbid in rows:
|
||||||
if _in_library_at_cutoff(conn, artist, album, cfg.quality_cutoff):
|
# A due release is about to be fulfilled or re-attempted; either way any earlier failed
|
||||||
|
# attempt is superseded, so retire it before deciding.
|
||||||
|
_retire_failed_attempts(conn, rel_id)
|
||||||
|
if _in_library_at_cutoff(conn, artist, album, rg_mbid, cfg.quality_cutoff):
|
||||||
_set_state(conn, rel_id, "fulfilled")
|
_set_state(conn, rel_id, "fulfilled")
|
||||||
continue
|
continue
|
||||||
if state == "grabbed" and (window_expired or (quality is not None and quality >= cfg.quality_cutoff)):
|
if state == "grabbed" and (window_expired or (quality is not None and quality >= cfg.quality_cutoff)):
|
||||||
@@ -154,12 +182,19 @@ def fulfill_owned_releases(conn: psycopg.Connection, cfg: MonitorConfig) -> int:
|
|||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
'UPDATE "MonitoredRelease" mr SET state = \'fulfilled\', '
|
'UPDATE "MonitoredRelease" mr SET state = \'fulfilled\', '
|
||||||
' "currentQualityClass" = li.q, '
|
' "currentQualityClass" = owned.q, '
|
||||||
' "firstGrabbedAt" = COALESCE(mr."firstGrabbedAt", now()) '
|
' "firstGrabbedAt" = COALESCE(mr."firstGrabbedAt", now()) '
|
||||||
'FROM (SELECT artist, album, max("qualityClass") AS q FROM "LibraryItem" '
|
'FROM ('
|
||||||
' GROUP BY artist, album) li '
|
' SELECT mr2.id AS mid, max(li."qualityClass") AS q '
|
||||||
'WHERE li.artist = mr."artistName" AND li.album = mr.album '
|
' FROM "MonitoredRelease" mr2 JOIN "LibraryItem" li ON ('
|
||||||
" AND mr.monitored = true AND mr.state <> 'fulfilled' AND li.q >= %s",
|
# match on rgMbid where the library row has one (survives artist-name canonicalization),
|
||||||
|
# else fall back to exact artist+album for scanned rows lacking an MBID
|
||||||
|
' (li."rgMbid" IS NOT NULL AND li."rgMbid" = mr2."rgMbid") '
|
||||||
|
' OR (li."rgMbid" IS NULL AND li.artist = mr2."artistName" AND li.album = mr2.album)) '
|
||||||
|
" WHERE mr2.monitored = true AND mr2.state <> 'fulfilled' "
|
||||||
|
' GROUP BY mr2.id HAVING max(li."qualityClass") >= %s'
|
||||||
|
') owned '
|
||||||
|
'WHERE mr.id = owned.mid',
|
||||||
(cfg.quality_cutoff,),
|
(cfg.quality_cutoff,),
|
||||||
)
|
)
|
||||||
fixed = cur.rowcount
|
fixed = cur.rowcount
|
||||||
@@ -171,21 +206,28 @@ def reconcile(conn: psycopg.Connection, job_id: str, cfg: MonitorConfig) -> None
|
|||||||
"""Feed a finished job's outcome back into its MonitoredRelease (if any)."""
|
"""Feed a finished job's outcome back into its MonitoredRelease (if any)."""
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
'SELECT j.state, r."monitoredReleaseId", r.artist, r.album '
|
'SELECT j.state, r."monitoredReleaseId", r.artist, r.album, mr."rgMbid" '
|
||||||
'FROM "Job" j JOIN "Request" r ON r.id = j."requestId" WHERE j.id = %s',
|
'FROM "Job" j JOIN "Request" r ON r.id = j."requestId" '
|
||||||
|
'LEFT JOIN "MonitoredRelease" mr ON mr.id = r."monitoredReleaseId" '
|
||||||
|
'WHERE j.id = %s',
|
||||||
(job_id,),
|
(job_id,),
|
||||||
)
|
)
|
||||||
row = cur.fetchone()
|
row = cur.fetchone()
|
||||||
if row is None:
|
if row is None:
|
||||||
return
|
return
|
||||||
state, rel_id, artist, album = row
|
state, rel_id, artist, album, rg_mbid = row
|
||||||
if rel_id is None or state != "imported":
|
if rel_id is None or state != "imported":
|
||||||
return # needs_attention (or in-flight) leaves the release wanted
|
return # needs_attention (or in-flight) leaves the release wanted
|
||||||
|
|
||||||
|
# A successful import supersedes any earlier failed attempts for this release.
|
||||||
|
_retire_failed_attempts(conn, rel_id)
|
||||||
|
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
'SELECT "qualityClass" FROM "LibraryItem" WHERE artist = %s AND album = %s',
|
'SELECT max("qualityClass") FROM "LibraryItem" WHERE '
|
||||||
(artist, album),
|
' ("rgMbid" IS NOT NULL AND "rgMbid" = %s) '
|
||||||
|
' OR ("rgMbid" IS NULL AND artist = %s AND album = %s)',
|
||||||
|
(rg_mbid, artist, album),
|
||||||
)
|
)
|
||||||
lib = cur.fetchone()
|
lib = cur.fetchone()
|
||||||
quality = lib[0] if lib else None
|
quality = lib[0] if lib else None
|
||||||
|
|||||||
+142
-43
@@ -1,4 +1,5 @@
|
|||||||
import os
|
import os
|
||||||
|
import random
|
||||||
import shutil
|
import shutil
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
@@ -11,6 +12,7 @@ import psycopg
|
|||||||
from lyra_worker.adapters.base import SourceAdapter
|
from lyra_worker.adapters.base import SourceAdapter
|
||||||
from lyra_worker.confidence import score_confidence
|
from lyra_worker.confidence import score_confidence
|
||||||
from lyra_worker.library import album_dir, import_album, list_audio_files, staging_dir
|
from lyra_worker.library import album_dir, import_album, list_audio_files, staging_dir
|
||||||
|
from lyra_worker.qobuz_gate import gate_open, record_download
|
||||||
from lyra_worker.quality import quality_class
|
from lyra_worker.quality import quality_class
|
||||||
from lyra_worker.ranker import rank_candidates
|
from lyra_worker.ranker import rank_candidates
|
||||||
from lyra_worker.types import Candidate, MBTarget
|
from lyra_worker.types import Candidate, MBTarget
|
||||||
@@ -20,8 +22,19 @@ _AUDIO_EXT = {".flac", ".mp3", ".m4a", ".opus", ".ogg", ".aac", ".wav"}
|
|||||||
# Reject a download whose total playtime falls below this fraction of MusicBrainz's expected
|
# Reject a download whose total playtime falls below this fraction of MusicBrainz's expected
|
||||||
# total — catches truncated files / preview clips substituted for real tracks that keep the
|
# total — catches truncated files / preview clips substituted for real tracks that keep the
|
||||||
# track COUNT right. Generous, so edition/encoding differences don't false-positive; bonus
|
# track COUNT right. Generous, so edition/encoding differences don't false-positive; bonus
|
||||||
# tracks (over-long) are always fine.
|
# tracks (over-long) are always fine. Kept well below 1.0 because a source legitimately delivers a
|
||||||
_DURATION_MIN_RATIO = 0.85
|
# different edition than MB's resolved release (shorter radio edits, no bonus/live tracks), and only
|
||||||
|
# a genuinely truncated/preview-filled download (playtime a fraction of expected) should be caught.
|
||||||
|
_DURATION_MIN_RATIO = 0.65
|
||||||
|
# Cap how many ranked candidates a single job will actually download+verify before giving up.
|
||||||
|
# Without this, the incomplete-download fall-through can grind through dozens of sources (a popular
|
||||||
|
# album returns ~200 Soulseek candidates), each a slow attempt that also leaves an abandoned slskd
|
||||||
|
# transfer. The best candidates rank first, so 6 attempts is plenty.
|
||||||
|
_MAX_DOWNLOAD_ATTEMPTS = 6
|
||||||
|
# The best quality any non-Qobuz source offers (Soulseek FLAC = CD-lossless). A library copy at or
|
||||||
|
# above this can only be improved by hi-res Qobuz, so when Qobuz is gated by pacing an upgrade of
|
||||||
|
# such a copy is deferred without even searching.
|
||||||
|
_CD_LOSSLESS_CLASS = 2
|
||||||
|
|
||||||
|
|
||||||
def _measure_staged_duration_s(staging: str) -> float | None:
|
def _measure_staged_duration_s(staging: str) -> float | None:
|
||||||
@@ -70,6 +83,40 @@ def _count_staged_audio(staging: str) -> int:
|
|||||||
return n
|
return n
|
||||||
|
|
||||||
|
|
||||||
|
def _download_problem(result, staging: str, target, winner, measure_duration) -> str | None:
|
||||||
|
"""Why a completed download is unusable — ``"incomplete download"`` / ``"download too short
|
||||||
|
(...)"`` — or None if it's good. Completeness is judged against the CHOSEN SOURCE's own track
|
||||||
|
count (``winner.track_count``), NOT MusicBrainz's canonical release: MB picks an arbitrary
|
||||||
|
releases[0] that is often a deluxe/expanded edition with more tracks (and runtime) than the
|
||||||
|
standard album a source legitimately delivers, so gating on ``target.track_count`` falsely
|
||||||
|
rejects complete standard-edition downloads (The Script "No Sound Without Silence" 11 vs MB 12,
|
||||||
|
Skillet "Unleashed" 12 vs MB's 20-track deluxe). Pure — no DB I/O — so the download loop can
|
||||||
|
call it per candidate and fall through to the next source on a problem."""
|
||||||
|
expected = winner.track_count
|
||||||
|
# (a) Truncated: fewer files than the source promised (a track failed mid-download). Count the
|
||||||
|
# NON-EMPTY audio on disk too: a silently-skipped track leaves a 0-byte placeholder the adapter
|
||||||
|
# still counts. (`staged and ...` keeps fake adapters that stage nothing on the count path.)
|
||||||
|
staged = _count_staged_audio(staging)
|
||||||
|
truncated = result.track_count < expected or bool(staged and staged < expected)
|
||||||
|
# (b) Implausibly small vs MB's album — a lone-track "full album" video / wrong match, as
|
||||||
|
# distinct from a legitimately smaller edition. A real edition keeps more than half MB's tracks.
|
||||||
|
too_small = target.track_count is not None and expected * 2 <= target.track_count
|
||||||
|
if truncated or too_small:
|
||||||
|
return "incomplete download"
|
||||||
|
# Duration: a truncated file or short preview substituted for a real track shows up as playtime
|
||||||
|
# well under the expected total. Scale MB's total to the DELIVERED edition's size so a smaller
|
||||||
|
# edition isn't judged against a larger one's runtime. Only when the total is known (measured is
|
||||||
|
# None for the offline fakes).
|
||||||
|
if target.total_duration_s:
|
||||||
|
expected_total = target.total_duration_s
|
||||||
|
if target.track_count:
|
||||||
|
expected_total *= winner.track_count / target.track_count
|
||||||
|
measured = measure_duration(staging)
|
||||||
|
if measured is not None and measured < expected_total * _DURATION_MIN_RATIO:
|
||||||
|
return f"download too short ({measured / 60:.0f} of ~{expected_total / 60:.0f} min)"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _poll_download_progress(job_id: str, staging: str, expected: int, stop: "threading.Event",
|
def _poll_download_progress(job_id: str, staging: str, expected: int, stop: "threading.Event",
|
||||||
reports: "threading.Event") -> None:
|
reports: "threading.Event") -> None:
|
||||||
"""Until `stop`, periodically write Job.downloadProgress = files-in-staging / expected
|
"""Until `stop`, periodically write Job.downloadProgress = files-in-staging / expected
|
||||||
@@ -114,30 +161,42 @@ def _set_state(conn: psycopg.Connection, job_id: str, state: str, stage: str) ->
|
|||||||
conn.commit()
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
def _set_download_progress(conn: psycopg.Connection, job_id: str, frac: float) -> None:
|
def _set_download_progress(
|
||||||
|
conn: psycopg.Connection, job_id: str, frac: float, eta_seconds: int | None = None
|
||||||
|
) -> None:
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute('UPDATE "Job" SET "downloadProgress" = %s WHERE id = %s', (frac, job_id))
|
cur.execute(
|
||||||
|
'UPDATE "Job" SET "downloadProgress" = %s, "downloadEtaSeconds" = %s WHERE id = %s',
|
||||||
|
(frac, eta_seconds, job_id),
|
||||||
|
)
|
||||||
conn.commit()
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
def _make_on_progress(conn: psycopg.Connection, job_id: str, reports: "threading.Event"):
|
def _make_on_progress(conn: psycopg.Connection, job_id: str, reports: "threading.Event"):
|
||||||
"""Build the on_progress(pct) callback threaded into adapter.download. It marks the
|
"""Build the on_progress(pct, eta_seconds=None) callback threaded into adapter.download. It
|
||||||
adapter as reporting real byte-level progress (so the file-count poller yields) and
|
marks the adapter as reporting real byte-level progress (so the file-count poller yields) and
|
||||||
throttle-writes Job.downloadProgress. Called synchronously from adapter.download on the
|
throttle-writes Job.downloadProgress (+ the download ETA when the adapter measures one — slskd
|
||||||
pipeline's own thread, so it safely reuses `conn`."""
|
does; others pass None). Called synchronously from adapter.download on the pipeline's own
|
||||||
|
thread, so it safely reuses `conn`."""
|
||||||
state = {"frac": -1.0, "t": 0.0}
|
state = {"frac": -1.0, "t": 0.0}
|
||||||
|
|
||||||
def on_progress(pct: float) -> None:
|
def on_progress(pct: float, eta_seconds: int | None = None) -> None:
|
||||||
reports.set()
|
reports.set()
|
||||||
frac = 0.0 if pct < 0 else 1.0 if pct > 1 else float(pct)
|
frac = 0.0 if pct < 0 else 1.0 if pct > 1 else float(pct)
|
||||||
now = time.monotonic()
|
now = time.monotonic()
|
||||||
# throttle DB writes: on a >=1% move or every 0.5s, not once per received byte
|
# throttle DB writes: on a >=1% move, a new ETA, or every 0.5s — not once per received byte
|
||||||
if frac - state["frac"] >= 0.01 or (now - state["t"]) >= 0.5:
|
if frac - state["frac"] >= 0.01 or eta_seconds is not None or (now - state["t"]) >= 0.5:
|
||||||
state["frac"] = frac
|
state["frac"] = frac
|
||||||
state["t"] = now
|
state["t"] = now
|
||||||
try:
|
try:
|
||||||
_set_download_progress(conn, job_id, frac)
|
_set_download_progress(conn, job_id, frac, eta_seconds)
|
||||||
except Exception as e: # a progress write must never fail the download
|
except Exception as e: # a progress write must never fail the download
|
||||||
|
# Roll back so a failed write can't leave the shared pipeline conn in an aborted
|
||||||
|
# txn (which would cascade "current transaction is aborted" into every later query).
|
||||||
|
try:
|
||||||
|
conn.rollback()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
print(f"pipeline: on_progress write failed: {e}", flush=True)
|
print(f"pipeline: on_progress write failed: {e}", flush=True)
|
||||||
|
|
||||||
return on_progress
|
return on_progress
|
||||||
@@ -225,6 +284,17 @@ def _library_quality(conn: psycopg.Connection, target: MBTarget) -> int | None:
|
|||||||
return row[0] if row else None
|
return row[0] if row else None
|
||||||
|
|
||||||
|
|
||||||
|
def _keep_existing_and_complete(conn: psycopg.Connection, job_id: str) -> None:
|
||||||
|
"""Finish a job without importing — keep the existing library copy — and mark its request
|
||||||
|
completed. Used when a (deferred) upgrade has nothing better to offer right now; the release
|
||||||
|
stays below the cutoff and is re-attempted later."""
|
||||||
|
_set_state(conn, job_id, "imported", "import")
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute("UPDATE \"Request\" SET status = 'completed' WHERE id = %s",
|
||||||
|
(_request_id(conn, job_id),))
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
def _persist_candidates(conn: psycopg.Connection, job_id: str, candidates: list[Candidate]) -> None:
|
def _persist_candidates(conn: psycopg.Connection, job_id: str, candidates: list[Candidate]) -> None:
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
for c in candidates:
|
for c in candidates:
|
||||||
@@ -317,6 +387,16 @@ def run_pipeline(
|
|||||||
conn.commit()
|
conn.commit()
|
||||||
return
|
return
|
||||||
|
|
||||||
|
# Pre-search upgrade skip: a >= CD-lossless copy can only be improved by hi-res Qobuz. If Qobuz
|
||||||
|
# is gated by pacing, don't even search (no other source can beat it) — defer to when Qobuz
|
||||||
|
# frees up. This avoids re-searching every already-owned album (~2 min Soulseek search each) on
|
||||||
|
# every monitor cycle. A user-forced re-acquire always proceeds.
|
||||||
|
_existing_q = _library_quality(conn, target)
|
||||||
|
if _existing_q is not None and _existing_q >= _CD_LOSSLESS_CLASS \
|
||||||
|
and not _is_force_job(conn, job_id) and not gate_open(conn):
|
||||||
|
_keep_existing_and_complete(conn, job_id)
|
||||||
|
return
|
||||||
|
|
||||||
# 2. match — search all adapters CONCURRENTLY (Soulseek's search polls up to ~2 min; run it
|
# 2. match — search all adapters CONCURRENTLY (Soulseek's search polls up to ~2 min; run it
|
||||||
# alongside Qobuz's instant search so total match time = the slowest source, not their sum).
|
# alongside Qobuz's instant search so total match time = the slowest source, not their sum).
|
||||||
# Only Qobuz touches streamrip's shared asyncio loop and there's a single Qobuz adapter, so no
|
# Only Qobuz touches streamrip's shared asyncio loop and there's a single Qobuz adapter, so no
|
||||||
@@ -329,8 +409,21 @@ def run_pipeline(
|
|||||||
for adapter, results in per_adapter:
|
for adapter, results in per_adapter:
|
||||||
for c in results:
|
for c in results:
|
||||||
found.append(replace(c, source_tier=adapter.tier, confidence=score_confidence(target, c)))
|
found.append(replace(c, source_tier=adapter.tier, confidence=score_confidence(target, c)))
|
||||||
|
# Drop candidates that can't be the full album — a result with far fewer tracks than the release
|
||||||
|
# has (e.g. a single-song Soulseek folder named like the album). Same threshold as the
|
||||||
|
# completeness 'too small' guard, applied up front so we never waste a download attempt on it.
|
||||||
|
# Unknown counts (0) and legitimately smaller editions (e.g. 11 vs 12) are kept.
|
||||||
|
if target.track_count:
|
||||||
|
found = [c for c in found if c.track_count == 0 or c.track_count * 2 > target.track_count]
|
||||||
_persist_candidates(conn, job_id, found)
|
_persist_candidates(conn, job_id, found)
|
||||||
|
|
||||||
|
# Qobuz pacing: when the budget gate is closed (off-hours / daily cap / spacing), drop Qobuz
|
||||||
|
# candidates so the job falls through to another source and Qobuz stays under the radar — UNLESS
|
||||||
|
# Qobuz is the only option (nothing to fall through to, so don't strand the album).
|
||||||
|
if any(c.source == "qobuz" for c in found) and any(c.source != "qobuz" for c in found):
|
||||||
|
if not gate_open(conn):
|
||||||
|
found = [c for c in found if c.source != "qobuz"]
|
||||||
|
|
||||||
# 3. rank
|
# 3. rank
|
||||||
_set_state(conn, job_id, "matched", "rank")
|
_set_state(conn, job_id, "matched", "rank")
|
||||||
ranked = rank_candidates(target, found, min_confidence)
|
ranked = rank_candidates(target, found, min_confidence)
|
||||||
@@ -338,12 +431,23 @@ def run_pipeline(
|
|||||||
_fail(conn, job_id, "no candidate above confidence threshold")
|
_fail(conn, job_id, "no candidate above confidence threshold")
|
||||||
return
|
return
|
||||||
|
|
||||||
|
# Upgrade guard: for an album already in the library, skip the download when no currently-
|
||||||
|
# available source beats the copy we have (ranked[0] is the highest-quality candidate). This
|
||||||
|
# happens when Qobuz is gated by pacing and only a same-quality Soulseek copy is left — the
|
||||||
|
# download would be discarded by keep-if-better, so defer it. The release stays below the cutoff
|
||||||
|
# and is re-attempted when a better source frees up. A user-forced re-acquire always proceeds.
|
||||||
|
existing_q = _library_quality(conn, target)
|
||||||
|
if existing_q is not None and not _is_force_job(conn, job_id) \
|
||||||
|
and quality_class(ranked[0].quality) <= existing_q:
|
||||||
|
_keep_existing_and_complete(conn, job_id) # keep the existing copy; nothing better available
|
||||||
|
return
|
||||||
|
|
||||||
# 4. download (fall-through) into an isolated per-job staging dir
|
# 4. download (fall-through) into an isolated per-job staging dir
|
||||||
_set_state(conn, job_id, "downloading", "download")
|
_set_state(conn, job_id, "downloading", "download")
|
||||||
by_source = {a.name: a for a in adapters}
|
by_source = {a.name: a for a in adapters}
|
||||||
staging = staging_dir(staging_root or f"{dest_root}/.staging", job_id)
|
staging = staging_dir(staging_root or f"{dest_root}/.staging", job_id)
|
||||||
winner = None
|
winner = None
|
||||||
result = None
|
last_problem = None # why the most recent downloaded-but-rejected candidate was unusable
|
||||||
_set_download_progress(conn, job_id, 0.0) # reset for this run
|
_set_download_progress(conn, job_id, 0.0) # reset for this run
|
||||||
expected = target.track_count or (ranked[0].track_count if ranked else 0)
|
expected = target.track_count or (ranked[0].track_count if ranked else 0)
|
||||||
_stop = threading.Event()
|
_stop = threading.Event()
|
||||||
@@ -354,46 +458,46 @@ def run_pipeline(
|
|||||||
_poller.start()
|
_poller.start()
|
||||||
try:
|
try:
|
||||||
try:
|
try:
|
||||||
|
attempts = 0
|
||||||
for candidate in ranked:
|
for candidate in ranked:
|
||||||
adapter = by_source.get(candidate.source)
|
adapter = by_source.get(candidate.source)
|
||||||
if adapter is None:
|
if adapter is None:
|
||||||
continue
|
continue
|
||||||
|
if attempts >= _MAX_DOWNLOAD_ATTEMPTS:
|
||||||
|
break # don't grind through every peer/source of a popular album
|
||||||
|
attempts += 1
|
||||||
_mark_chosen(conn, job_id, candidate.source_ref) # reflect the source now in flight
|
_mark_chosen(conn, job_id, candidate.source_ref) # reflect the source now in flight
|
||||||
shutil.rmtree(staging, ignore_errors=True) # clean slate per attempt
|
shutil.rmtree(staging, ignore_errors=True) # clean slate per attempt
|
||||||
_reports.clear() # this attempt hasn't reported yet → poller estimates until it does
|
_reports.clear() # this attempt hasn't reported yet → poller estimates until it does
|
||||||
result = adapter.download(candidate, staging, _make_on_progress(conn, job_id, _reports))
|
result = adapter.download(candidate, staging, _make_on_progress(conn, job_id, _reports))
|
||||||
if result.ok:
|
if not result.ok:
|
||||||
|
continue
|
||||||
|
# Verify completeness on the staging copy. A download that lands incomplete or too
|
||||||
|
# short falls through to the NEXT-ranked source (e.g. Qobuz repeatedly fails one
|
||||||
|
# track → try the deluxe edition or Soulseek) rather than failing the whole job on
|
||||||
|
# the first source's shortfall.
|
||||||
|
last_problem = _download_problem(result, staging, target, candidate, measure_duration)
|
||||||
|
if last_problem is None:
|
||||||
winner = candidate
|
winner = candidate
|
||||||
break
|
break
|
||||||
finally:
|
finally:
|
||||||
_stop.set()
|
_stop.set()
|
||||||
_poller.join(timeout=3)
|
_poller.join(timeout=3)
|
||||||
if winner is None or result is None or not result.ok:
|
if winner is None:
|
||||||
_fail(conn, job_id, "all downloads failed")
|
# No source produced a complete album: surface why the last one was rejected (incomplete
|
||||||
|
# / too short), or "all downloads failed" if none even downloaded.
|
||||||
|
_fail(conn, job_id, last_problem or "all downloads failed")
|
||||||
return
|
return
|
||||||
_set_download_progress(conn, job_id, 1.0) # download done — UI shows 100% into Finishing
|
_set_download_progress(conn, job_id, 1.0) # download done — UI shows 100% into Finishing
|
||||||
|
if winner.source == "qobuz":
|
||||||
|
# Count this Qobuz download against today's budget and set the next randomized spacing.
|
||||||
|
try:
|
||||||
|
record_download(conn, random.random())
|
||||||
|
except Exception as e: # pacing bookkeeping must never fail a good download
|
||||||
|
print(f"pipeline: qobuz pacing record failed for job {job_id}: {e}", flush=True)
|
||||||
|
|
||||||
# 5. verify completeness on the staging copy, then promote + tag
|
# 5. promote + tag the verified winner
|
||||||
_set_state(conn, job_id, "tagging", "tag")
|
_set_state(conn, job_id, "tagging", "tag")
|
||||||
expected = target.track_count if target.track_count is not None else winner.track_count
|
|
||||||
# Also count the NON-EMPTY audio actually on disk: a silently-skipped track leaves a
|
|
||||||
# 0-byte placeholder that the adapter still counts, so trust the files when any are
|
|
||||||
# staged. (`staged and ...` keeps fake adapters that stage nothing on the count path.)
|
|
||||||
staged = _count_staged_audio(staging)
|
|
||||||
if result.track_count < expected or (staged and staged < expected):
|
|
||||||
_fail(conn, job_id, "incomplete download")
|
|
||||||
return
|
|
||||||
|
|
||||||
# Duration check: even with the right track count, a truncated file or a short preview
|
|
||||||
# substituted for a real track shows up as a total playtime well under MB's expected
|
|
||||||
# total. Only applies when both are known (measured is None for the offline fakes).
|
|
||||||
if target.total_duration_s:
|
|
||||||
measured = measure_duration(staging)
|
|
||||||
if measured is not None and measured < target.total_duration_s * _DURATION_MIN_RATIO:
|
|
||||||
_fail(conn, job_id,
|
|
||||||
f"download too short ({measured / 60:.0f} of ~{target.total_duration_s / 60:.0f} min)")
|
|
||||||
return
|
|
||||||
|
|
||||||
final = album_dir(dest_root, target)
|
final = album_dir(dest_root, target)
|
||||||
existing_q = _library_quality(conn, target)
|
existing_q = _library_quality(conn, target)
|
||||||
new_q = quality_class(winner.quality)
|
new_q = quality_class(winner.quality)
|
||||||
@@ -407,12 +511,7 @@ def run_pipeline(
|
|||||||
print(f"pipeline: tagging failed for job {job_id}: {e}", flush=True)
|
print(f"pipeline: tagging failed for job {job_id}: {e}", flush=True)
|
||||||
_import(conn, job_id, target, winner, final)
|
_import(conn, job_id, target, winner, final)
|
||||||
else:
|
else:
|
||||||
# an existing copy is already at >= this quality — keep it untouched, just complete the request
|
# an existing copy is already at >= this quality — keep it untouched, just complete
|
||||||
with conn.cursor() as cur:
|
_keep_existing_and_complete(conn, job_id)
|
||||||
cur.execute(
|
|
||||||
"UPDATE \"Request\" SET status = 'completed' WHERE id = %s",
|
|
||||||
(_request_id(conn, job_id),),
|
|
||||||
)
|
|
||||||
conn.commit()
|
|
||||||
finally:
|
finally:
|
||||||
shutil.rmtree(staging, ignore_errors=True) # a partial/failed download never persists
|
shutil.rmtree(staging, ignore_errors=True) # a partial/failed download never persists
|
||||||
|
|||||||
@@ -0,0 +1,159 @@
|
|||||||
|
"""Qobuz pacing ("budget gate").
|
||||||
|
|
||||||
|
Keep using Qobuz for hi-res, but at a human-looking volume and cadence so the account isn't
|
||||||
|
flagged for bulk downloading (which got it 403 USER_BLOCKED). One gate decides, per album,
|
||||||
|
whether Qobuz is an eligible source right now: within active hours, under today's cap (a warm-up
|
||||||
|
ramp of a steady cap), and past a randomized spacing since the last Qobuz download. When the gate
|
||||||
|
is closed the pipeline drops Qobuz candidates and falls through to the next source; monitored
|
||||||
|
albums upgrade to Qobuz later once budget frees up.
|
||||||
|
|
||||||
|
Pure decision logic (`today_cap`, `is_allowed`, `next_gap_seconds`) is separated from the two DB
|
||||||
|
helpers (`gate_open`, `record_download`) so the policy is unit-tested without a clock or DB.
|
||||||
|
"""
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import date
|
||||||
|
|
||||||
|
_TRUE = {"1", "true", "yes", "on"}
|
||||||
|
_JITTER = 0.3 # ± fraction around the spread interval
|
||||||
|
_GAP_FLOOR_S = 60.0
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class QobuzPacing:
|
||||||
|
enabled: bool = True
|
||||||
|
active_start: int = 8
|
||||||
|
active_end: int = 23
|
||||||
|
daily_cap: int = 40
|
||||||
|
warmup_start_date: str = "" # ISO "YYYY-MM-DD", or "" for no warm-up ramp
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_config(cls, config: dict) -> "QobuzPacing":
|
||||||
|
def _int(key, default):
|
||||||
|
try:
|
||||||
|
return int(config[key])
|
||||||
|
except (KeyError, TypeError, ValueError):
|
||||||
|
return default
|
||||||
|
|
||||||
|
return cls(
|
||||||
|
enabled=str(config.get("qobuz.pacing.enabled", "true")).strip().lower() in _TRUE,
|
||||||
|
active_start=_int("qobuz.pacing.activeStartHour", 8),
|
||||||
|
active_end=_int("qobuz.pacing.activeEndHour", 23),
|
||||||
|
daily_cap=_int("qobuz.pacing.dailyCap", 40),
|
||||||
|
warmup_start_date=str(config.get("qobuz.pacing.warmupStartDate", "") or "").strip(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def today_cap(pacing: QobuzPacing, today: date) -> int:
|
||||||
|
"""Today's Qobuz download cap: the steady cap, ramped up by account age (days since
|
||||||
|
``warmup_start_date``). No warm-up date -> steady cap. A small steady cap is never exceeded."""
|
||||||
|
steady = max(0, pacing.daily_cap)
|
||||||
|
if not pacing.warmup_start_date:
|
||||||
|
return steady
|
||||||
|
try:
|
||||||
|
start = date.fromisoformat(pacing.warmup_start_date)
|
||||||
|
except ValueError:
|
||||||
|
return steady
|
||||||
|
days = max(0, (today - start).days)
|
||||||
|
if days < 3:
|
||||||
|
return min(5, steady)
|
||||||
|
if days < 7:
|
||||||
|
return min(10, steady)
|
||||||
|
if days < 14:
|
||||||
|
return min(20, steady)
|
||||||
|
return steady
|
||||||
|
|
||||||
|
|
||||||
|
def is_allowed(pacing: QobuzPacing, now_hour: int, count_today: int, cap: int,
|
||||||
|
now_epoch: float, next_allowed_epoch: float) -> bool:
|
||||||
|
"""Pure gate decision. Qobuz is eligible only within active hours, under today's cap, and past
|
||||||
|
the randomized spacing since the last download. Pacing disabled -> always allowed."""
|
||||||
|
if not pacing.enabled:
|
||||||
|
return True
|
||||||
|
if not (pacing.active_start <= now_hour < pacing.active_end):
|
||||||
|
return False
|
||||||
|
if count_today >= cap:
|
||||||
|
return False
|
||||||
|
if now_epoch < next_allowed_epoch:
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def next_gap_seconds(pacing: QobuzPacing, cap: int, rand01: float) -> float:
|
||||||
|
"""Seconds until the next Qobuz download is allowed: the day's budget spread across the active
|
||||||
|
window (``activeWindow / cap``), ± _JITTER, floored at _GAP_FLOOR_S. ``rand01`` in [0,1] is the
|
||||||
|
injected randomness (so tests are deterministic; callers pass random.random())."""
|
||||||
|
window_s = max(1, pacing.active_end - pacing.active_start) * 3600
|
||||||
|
base = window_s / max(1, cap)
|
||||||
|
lo, hi = base * (1 - _JITTER), base * (1 + _JITTER)
|
||||||
|
gap = lo + (hi - lo) * rand01
|
||||||
|
return max(_GAP_FLOOR_S, gap)
|
||||||
|
|
||||||
|
|
||||||
|
def _now_parts(conn):
|
||||||
|
"""(hour:int, today:date, epoch:float) from the DB clock — avoids container-TZ ambiguity."""
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute("SELECT extract(hour from now())::int, current_date, extract(epoch from now())")
|
||||||
|
hour, today, epoch = cur.fetchone()
|
||||||
|
return int(hour), today, float(epoch)
|
||||||
|
|
||||||
|
|
||||||
|
def _read(conn, keys):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT key, value FROM "Config" WHERE key = ANY(%s)', (list(keys),))
|
||||||
|
return dict(cur.fetchall())
|
||||||
|
|
||||||
|
|
||||||
|
def gate_open(conn) -> bool:
|
||||||
|
"""Whether Qobuz may be used for a download right now (reads pacing config + counter/spacing
|
||||||
|
state from Config and the DB clock). A counter from a previous day is treated as 0."""
|
||||||
|
from lyra_worker.config import get_config
|
||||||
|
|
||||||
|
pacing = QobuzPacing.from_config(get_config(conn))
|
||||||
|
hour, today, epoch = _now_parts(conn)
|
||||||
|
state = _read(conn, ("qobuz.pacing.countDay", "qobuz.pacing.countToday", "qobuz.pacing.nextAllowedAt"))
|
||||||
|
count = 0
|
||||||
|
if state.get("qobuz.pacing.countDay") == today.isoformat():
|
||||||
|
try:
|
||||||
|
count = int(state.get("qobuz.pacing.countToday") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
count = 0
|
||||||
|
try:
|
||||||
|
next_allowed = float(state.get("qobuz.pacing.nextAllowedAt") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
next_allowed = 0.0
|
||||||
|
return is_allowed(pacing, hour, count, today_cap(pacing, today), epoch, next_allowed)
|
||||||
|
|
||||||
|
|
||||||
|
def record_download(conn, rand01: float) -> None:
|
||||||
|
"""Record a successful Qobuz download: bump today's counter (resetting it if the stored day is
|
||||||
|
stale) and set the randomized next-allowed time. ``rand01`` is injected randomness."""
|
||||||
|
from lyra_worker.config import get_config
|
||||||
|
|
||||||
|
pacing = QobuzPacing.from_config(get_config(conn))
|
||||||
|
_hour, today, epoch = _now_parts(conn)
|
||||||
|
state = _read(conn, ("qobuz.pacing.countDay", "qobuz.pacing.countToday"))
|
||||||
|
if state.get("qobuz.pacing.countDay") == today.isoformat():
|
||||||
|
try:
|
||||||
|
count = int(state.get("qobuz.pacing.countToday") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
count = 0
|
||||||
|
else:
|
||||||
|
count = 0
|
||||||
|
count += 1
|
||||||
|
next_allowed = epoch + next_gap_seconds(pacing, today_cap(pacing, today), rand01)
|
||||||
|
_write(conn, {
|
||||||
|
"qobuz.pacing.countDay": today.isoformat(),
|
||||||
|
"qobuz.pacing.countToday": str(count),
|
||||||
|
"qobuz.pacing.nextAllowedAt": repr(next_allowed),
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def _write(conn, kv: dict) -> None:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
for key, value in kv.items():
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
|
||||||
|
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
|
||||||
|
(key, value),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
"""A search result with far fewer tracks than the release (e.g. a single-song Soulseek folder
|
||||||
|
named like the album) can't be the full album — it must be dropped before we waste a download."""
|
||||||
|
import os
|
||||||
|
|
||||||
|
from lyra_worker.claim import claim_next
|
||||||
|
from lyra_worker.pipeline import run_pipeline
|
||||||
|
from lyra_worker.types import Candidate, DownloadResult, MBTarget, Quality
|
||||||
|
from tests.conftest import insert_request
|
||||||
|
|
||||||
|
_Q = Quality(fmt="FLAC", lossless=True, bit_depth=24, sample_rate=96000)
|
||||||
|
|
||||||
|
|
||||||
|
class _TwoCandQobuz:
|
||||||
|
name = "qobuz"
|
||||||
|
tier = 0
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.downloaded = []
|
||||||
|
|
||||||
|
def health(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
return [
|
||||||
|
Candidate(source="qobuz", source_ref="single", matched_artist=target.artist,
|
||||||
|
matched_album=target.album, quality=_Q, track_count=1, source_tier=0),
|
||||||
|
Candidate(source="qobuz", source_ref="full", matched_artist=target.artist,
|
||||||
|
matched_album=target.album, quality=_Q, track_count=12, source_tier=0),
|
||||||
|
]
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
self.downloaded.append(candidate.source_ref)
|
||||||
|
os.makedirs(dest, exist_ok=True)
|
||||||
|
for i in range(candidate.track_count):
|
||||||
|
with open(os.path.join(dest, f"{i + 1:02d}.flac"), "wb") as fh:
|
||||||
|
fh.write(b"a")
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=candidate.track_count)
|
||||||
|
|
||||||
|
|
||||||
|
class _Resolver:
|
||||||
|
def __init__(self, n):
|
||||||
|
self.n = n
|
||||||
|
|
||||||
|
def resolve(self, artist, album):
|
||||||
|
return MBTarget(artist=artist, album=album, track_count=self.n)
|
||||||
|
|
||||||
|
|
||||||
|
def test_single_song_candidate_is_dropped(conn):
|
||||||
|
job_id = insert_request(conn, artist="A", album="B")
|
||||||
|
claim_next(conn)
|
||||||
|
adapter = _TwoCandQobuz()
|
||||||
|
run_pipeline(conn, job_id, [adapter], resolver=_Resolver(12), dest_root="/tmp/lib-cf")
|
||||||
|
|
||||||
|
assert adapter.downloaded == ["full"] # only the full album was ever attempted
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT "trackCount" FROM "Candidate" WHERE "jobId" = %s', (job_id,))
|
||||||
|
assert [r[0] for r in cur.fetchall()] == [12] # the single-song folder wasn't even persisted
|
||||||
@@ -0,0 +1,125 @@
|
|||||||
|
"""When the top-ranked source downloads but is incomplete (e.g. Qobuz consistently fails one
|
||||||
|
track), the pipeline must fall through to the next-ranked source instead of failing the whole
|
||||||
|
job — only giving up (needs_attention) when every candidate is incomplete."""
|
||||||
|
import os
|
||||||
|
|
||||||
|
from lyra_worker.claim import claim_next
|
||||||
|
from lyra_worker.pipeline import run_pipeline
|
||||||
|
from lyra_worker.types import Candidate, DownloadResult, Quality
|
||||||
|
from tests.conftest import insert_request
|
||||||
|
|
||||||
|
_HIRES = Quality(fmt="FLAC", lossless=True, bit_depth=24, sample_rate=96000)
|
||||||
|
_CD = Quality(fmt="FLAC", lossless=True, bit_depth=16, sample_rate=44100)
|
||||||
|
|
||||||
|
|
||||||
|
def _stage(dest, n):
|
||||||
|
os.makedirs(dest, exist_ok=True)
|
||||||
|
for i in range(n):
|
||||||
|
with open(os.path.join(dest, f"{i + 1:02d}.flac"), "wb") as fh:
|
||||||
|
fh.write(b"audio")
|
||||||
|
|
||||||
|
|
||||||
|
class _Qobuz:
|
||||||
|
"""Higher-quality (ranks first) Qobuz that promises 3 tracks but delivers `deliver`."""
|
||||||
|
name = "qobuz"
|
||||||
|
tier = 0
|
||||||
|
|
||||||
|
def __init__(self, deliver):
|
||||||
|
self._deliver = deliver
|
||||||
|
|
||||||
|
def health(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
return [Candidate(source="qobuz", source_ref="q", matched_artist=target.artist,
|
||||||
|
matched_album=target.album, quality=_HIRES, track_count=3, source_tier=0)]
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
_stage(dest, self._deliver)
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=self._deliver)
|
||||||
|
|
||||||
|
|
||||||
|
class _Soulseek:
|
||||||
|
"""Lower-quality (ranks second) Soulseek that promises 3 tracks and delivers `deliver`."""
|
||||||
|
name = "soulseek"
|
||||||
|
tier = 1
|
||||||
|
|
||||||
|
def __init__(self, deliver):
|
||||||
|
self._deliver = deliver
|
||||||
|
|
||||||
|
def health(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
return [Candidate(source="soulseek", source_ref="s", matched_artist=target.artist,
|
||||||
|
matched_album=target.album, quality=_CD, track_count=3, source_tier=1)]
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
_stage(dest, self._deliver)
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=self._deliver)
|
||||||
|
|
||||||
|
|
||||||
|
def _state_error(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT state, error FROM "Job" WHERE id = %s', (job_id,))
|
||||||
|
return cur.fetchone()
|
||||||
|
|
||||||
|
|
||||||
|
def _chosen_source(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT source FROM "Candidate" WHERE "jobId" = %s AND chosen = true', (job_id,))
|
||||||
|
row = cur.fetchone()
|
||||||
|
return row[0] if row else None
|
||||||
|
|
||||||
|
|
||||||
|
def test_incomplete_top_source_falls_through_to_complete_next(conn, tmp_path):
|
||||||
|
# Qobuz (ranks first on quality) delivers only 2 of its promised 3 tracks -> incomplete.
|
||||||
|
# The pipeline must try the next source (Soulseek, complete) rather than failing the job.
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [_Qobuz(deliver=2), _Soulseek(deliver=3)], dest_root=str(tmp_path))
|
||||||
|
assert _state_error(conn, job_id) == ("imported", None)
|
||||||
|
assert _chosen_source(conn, job_id) == "soulseek"
|
||||||
|
|
||||||
|
|
||||||
|
def test_all_sources_incomplete_still_needs_attention(conn, tmp_path):
|
||||||
|
# Every candidate is incomplete -> no fallback succeeds -> needs_attention (incomplete).
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [_Qobuz(deliver=2), _Soulseek(deliver=1)], dest_root=str(tmp_path))
|
||||||
|
assert _state_error(conn, job_id) == ("needs_attention", "incomplete download")
|
||||||
|
assert not (tmp_path / "Radiohead" / "In Rainbows").exists() # nothing imported
|
||||||
|
|
||||||
|
|
||||||
|
class _CountingIncompleteQobuz:
|
||||||
|
"""Offers `n` candidates that all download but come back incomplete; counts download calls."""
|
||||||
|
name = "qobuz"
|
||||||
|
tier = 0
|
||||||
|
|
||||||
|
def __init__(self, n):
|
||||||
|
self.n = n
|
||||||
|
self.calls = 0
|
||||||
|
|
||||||
|
def health(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
q = _HIRES
|
||||||
|
return [Candidate(source="qobuz", source_ref=f"r{i}", matched_artist=target.artist,
|
||||||
|
matched_album=target.album, quality=q, track_count=3, source_tier=0)
|
||||||
|
for i in range(self.n)]
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
self.calls += 1
|
||||||
|
_stage(dest, 1) # 1 of 3 → always incomplete
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=1)
|
||||||
|
|
||||||
|
|
||||||
|
def test_fall_through_is_capped(conn):
|
||||||
|
from lyra_worker.pipeline import _MAX_DOWNLOAD_ATTEMPTS
|
||||||
|
a = _CountingIncompleteQobuz(n=20)
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [a], dest_root="/tmp/lib-cap")
|
||||||
|
assert _state_error(conn, job_id) == ("needs_attention", "incomplete download")
|
||||||
|
assert a.calls == _MAX_DOWNLOAD_ATTEMPTS # capped, not all 20 peers tried
|
||||||
@@ -72,6 +72,52 @@ def test_grabbed_past_window_is_fulfilled_not_enqueued(conn):
|
|||||||
assert _state(conn, rel_id) == "fulfilled"
|
assert _state(conn, rel_id) == "fulfilled"
|
||||||
|
|
||||||
|
|
||||||
|
def test_owned_by_rgmbid_despite_name_mismatch_is_fulfilled(conn):
|
||||||
|
# The monitor followed "Kanye West" but the library stores MB's canonical "Ye"; matching on
|
||||||
|
# rgMbid recognises ownership so the release is fulfilled, not re-pressed.
|
||||||
|
rel_id = insert_monitored_release(conn, artist_name="Kanye West", album="Graduation",
|
||||||
|
rg_mbid="rg-grad", monitored=True, state="wanted")
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "LibraryItem" (id, artist, album, path, source, format, '
|
||||||
|
'"qualityClass", "rgMbid", "importedAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, 'Ye', 'Graduation', '/m', 'soulseek', 'FLAC', 2, "
|
||||||
|
"'rg-grad', now())"
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
assert enqueue_due(conn, CFG) == 0
|
||||||
|
assert _state(conn, rel_id) == "fulfilled"
|
||||||
|
|
||||||
|
|
||||||
|
def test_re_enqueue_retires_prior_failed_attempt(conn):
|
||||||
|
# A hard-to-match release re-pressed each retry interval must not pile up needs_attention
|
||||||
|
# Request rows; the prior failed attempt is retired when the next one is enqueued.
|
||||||
|
rel_id = insert_monitored_release(conn, artist_name="X", album="Y", rg_mbid="rg-1",
|
||||||
|
monitored=True, state="wanted")
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Request" (id, artist, album, status, "monitoredReleaseId", "createdAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, 'X', 'Y', 'needs_attention', %s, now()) RETURNING id",
|
||||||
|
(rel_id,),
|
||||||
|
)
|
||||||
|
req = cur.fetchone()[0]
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Job" (id, "requestId", state, "currentStage", attempts, "createdAt", "updatedAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, %s, 'needs_attention', 'rank', 1, now(), now())",
|
||||||
|
(req,),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
assert enqueue_due(conn, CFG) == 1
|
||||||
|
assert _job_count_for(conn, rel_id) == 1 # old failed attempt retired, one fresh attempt
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT j.state FROM "Job" j JOIN "Request" r ON r.id = j."requestId" '
|
||||||
|
'WHERE r."monitoredReleaseId" = %s',
|
||||||
|
(rel_id,),
|
||||||
|
)
|
||||||
|
assert cur.fetchone()[0] == "requested"
|
||||||
|
|
||||||
|
|
||||||
def test_already_in_library_at_cutoff_is_fulfilled(conn):
|
def test_already_in_library_at_cutoff_is_fulfilled(conn):
|
||||||
rel_id = insert_monitored_release(conn, artist_name="A", album="B", rg_mbid="rg-1",
|
rel_id = insert_monitored_release(conn, artist_name="A", album="B", rg_mbid="rg-1",
|
||||||
monitored=True, state="wanted")
|
monitored=True, state="wanted")
|
||||||
|
|||||||
@@ -22,17 +22,27 @@ def _make_job(conn, rel_id, artist, album, state):
|
|||||||
return req, job
|
return req, job
|
||||||
|
|
||||||
|
|
||||||
def _add_library(conn, req, artist, album, quality):
|
def _add_library(conn, req, artist, album, quality, rg_mbid=None):
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
'INSERT INTO "LibraryItem" (id, "requestId", artist, album, path, source, format, '
|
'INSERT INTO "LibraryItem" (id, "requestId", artist, album, path, source, format, '
|
||||||
'"qualityClass", "importedAt") '
|
'"qualityClass", "rgMbid", "importedAt") '
|
||||||
"VALUES (gen_random_uuid()::text, %s, %s, %s, '/m', 'qobuz', 'FLAC', %s, now())",
|
"VALUES (gen_random_uuid()::text, %s, %s, %s, '/m', 'qobuz', 'FLAC', %s, %s, now())",
|
||||||
(req, artist, album, quality),
|
(req, artist, album, quality, rg_mbid),
|
||||||
)
|
)
|
||||||
conn.commit()
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _needs_attention_count(conn, rel_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT count(*) FROM "Request" r JOIN "Job" j ON j."requestId" = r.id '
|
||||||
|
"WHERE r.\"monitoredReleaseId\" = %s AND j.state = 'needs_attention'",
|
||||||
|
(rel_id,),
|
||||||
|
)
|
||||||
|
return cur.fetchone()[0]
|
||||||
|
|
||||||
|
|
||||||
def _release(conn, rel_id):
|
def _release(conn, rel_id):
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
@@ -106,6 +116,42 @@ def test_fulfill_owned_releases_ignores_below_cutoff(conn):
|
|||||||
assert _release(conn, rel)[0] == "wanted"
|
assert _release(conn, rel)[0] == "wanted"
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_matches_by_rgmbid_when_artist_name_canonicalized(conn):
|
||||||
|
# Real 'Kanye West' -> 'Ye' bug: the MonitoredRelease keeps the followed name, but the
|
||||||
|
# imported LibraryItem carries MusicBrainz's canonical artist. Matching on rgMbid lets the
|
||||||
|
# release reconcile to fulfilled instead of looking un-owned and re-pressing forever.
|
||||||
|
rel = insert_monitored_release(conn, artist_name="Kanye West", album="Graduation",
|
||||||
|
rg_mbid="rg-grad", monitored=True, state="wanted")
|
||||||
|
req, job = _make_job(conn, rel, "Kanye West", "Graduation", "imported")
|
||||||
|
_add_library(conn, req, "Ye", "Graduation", 3, rg_mbid="rg-grad")
|
||||||
|
reconcile(conn, job, CFG)
|
||||||
|
assert _release(conn, rel) == ("fulfilled", 3, True)
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_fulfilled_retires_failed_sibling_attempts(conn):
|
||||||
|
# 'V' by Maroon 5 phantom: earlier failed attempts leave needs_attention Request rows that
|
||||||
|
# keep showing on the press even after a later attempt imports. A successful import retires
|
||||||
|
# those failed siblings so the press stops showing an owned album as needing attention.
|
||||||
|
rel = insert_monitored_release(conn, artist_name="Maroon 5", album="V", rg_mbid="rg-v",
|
||||||
|
monitored=True, state="wanted")
|
||||||
|
_make_job(conn, rel, "Maroon 5", "V", "needs_attention")
|
||||||
|
_make_job(conn, rel, "Maroon 5", "V", "needs_attention")
|
||||||
|
req, job = _make_job(conn, rel, "Maroon 5", "V", "imported")
|
||||||
|
_add_library(conn, req, "Maroon 5", "V", 2)
|
||||||
|
reconcile(conn, job, CFG)
|
||||||
|
assert _release(conn, rel)[0] == "fulfilled"
|
||||||
|
assert _needs_attention_count(conn, rel) == 0 # failed siblings retired
|
||||||
|
|
||||||
|
|
||||||
|
def test_fulfill_owned_releases_matches_by_rgmbid(conn):
|
||||||
|
# Ne-Yo -> Ne‐Yo (typographic hyphen) name drift: still owned by rgMbid.
|
||||||
|
rel = insert_monitored_release(conn, artist_name="Ne-Yo", album="Because of You",
|
||||||
|
rg_mbid="rg-ney", monitored=True, state="grabbed")
|
||||||
|
_add_library(conn, None, "Ne‐Yo", "Because of You", 2, rg_mbid="rg-ney")
|
||||||
|
assert fulfill_owned_releases(conn, MonitorConfig(quality_cutoff=2)) == 1
|
||||||
|
assert _release(conn, rel) == ("fulfilled", 2, True)
|
||||||
|
|
||||||
|
|
||||||
def test_unlinked_job_is_ignored(conn):
|
def test_unlinked_job_is_ignored(conn):
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
|
|||||||
@@ -1,4 +1,7 @@
|
|||||||
from lyra_worker._musicbrainz import _best_release_group
|
import musicbrainzngs
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from lyra_worker._musicbrainz import _best_release_group, _pick_release, _with_retry
|
||||||
|
|
||||||
# Offline unit tests for the pure release-group selection logic. The live resolve()
|
# Offline unit tests for the pure release-group selection logic. The live resolve()
|
||||||
# path (network) is covered by test_musicbrainz_live.py.
|
# path (network) is covered by test_musicbrainz_live.py.
|
||||||
@@ -48,6 +51,21 @@ def test_empty_groups_returns_none():
|
|||||||
assert _best_release_group("Michael Jackson", "Off the Wall", []) is None
|
assert _best_release_group("Michael Jackson", "Off the Wall", []) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_prefers_original_group_over_later_reissue():
|
||||||
|
# Real "Hybrid Theory" bug: MB lists a 2000 original and a 2023 reissue as separate same-titled
|
||||||
|
# Album groups (tie on artist/title/type). The reissue's longer tracklist fails completeness
|
||||||
|
# checks, so the original (earliest) must win regardless of MB's relevance order.
|
||||||
|
groups = [
|
||||||
|
{"id": "reissue", "title": "Hybrid Theory", "primary-type": "Album",
|
||||||
|
"first-release-date": "2023-07-21",
|
||||||
|
"artist-credit": [{"artist": {"name": "Linkin Park"}}]},
|
||||||
|
{"id": "original", "title": "Hybrid Theory", "primary-type": "Album",
|
||||||
|
"first-release-date": "2000-10-24",
|
||||||
|
"artist-credit": [{"artist": {"name": "Linkin Park"}}]},
|
||||||
|
]
|
||||||
|
assert _best_release_group("Linkin Park", "Hybrid Theory", groups)["id"] == "original"
|
||||||
|
|
||||||
|
|
||||||
def test_prefers_credited_artist_over_same_titled_other_artist_album():
|
def test_prefers_credited_artist_over_same_titled_other_artist_album():
|
||||||
# Real-world "Fun Machine" bug: Lake Street Dive's EP is what we followed, but two
|
# Real-world "Fun Machine" bug: Lake Street Dive's EP is what we followed, but two
|
||||||
# unrelated bands also have a release group literally titled "Fun Machine" as an Album.
|
# unrelated bands also have a release group literally titled "Fun Machine" as an Album.
|
||||||
@@ -60,3 +78,97 @@ def test_prefers_credited_artist_over_same_titled_other_artist_album():
|
|||||||
]
|
]
|
||||||
best = _best_release_group("Lake Street Dive", "Fun Machine", groups)
|
best = _best_release_group("Lake Street Dive", "Fun Machine", groups)
|
||||||
assert best["artist-credit"][0]["artist"]["name"] == "Lake Street Dive"
|
assert best["artist-credit"][0]["artist"]["name"] == "Lake Street Dive"
|
||||||
|
|
||||||
|
|
||||||
|
# --- _with_retry: transient MusicBrainz failures (SSL EOF, 503 rate-limit) ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_pick_release_prefers_original_official_over_later_reissue():
|
||||||
|
# Real "Hybrid Theory" bug: MB's arbitrary releases[0] was a 2023 13-track reissue; the 2000
|
||||||
|
# original is what sources deliver, so completeness checks must reference it.
|
||||||
|
releases = [
|
||||||
|
{"id": "reissue", "status": "Official", "date": "2023-06-01"},
|
||||||
|
{"id": "original", "status": "Official", "date": "2000-10-24"},
|
||||||
|
{"id": "deluxe", "status": "Official", "date": "2020-10-09"},
|
||||||
|
]
|
||||||
|
assert _pick_release(releases)["id"] == "original"
|
||||||
|
|
||||||
|
|
||||||
|
def test_pick_release_skips_non_official_and_undated():
|
||||||
|
releases = [
|
||||||
|
{"id": "promo", "status": "Promotion", "date": "1999-01-01"}, # earliest but not official
|
||||||
|
{"id": "undated", "status": "Official", "date": ""},
|
||||||
|
{"id": "official", "status": "Official", "date": "2001-03-03"},
|
||||||
|
]
|
||||||
|
assert _pick_release(releases)["id"] == "official"
|
||||||
|
|
||||||
|
|
||||||
|
def test_pick_release_empty_is_none():
|
||||||
|
assert _pick_release([]) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_retry_returns_value_without_backoff_when_first_call_succeeds():
|
||||||
|
sleeps: list[float] = []
|
||||||
|
calls = {"n": 0}
|
||||||
|
|
||||||
|
def fn():
|
||||||
|
calls["n"] += 1
|
||||||
|
return "ok"
|
||||||
|
|
||||||
|
assert _with_retry(fn, sleep=sleeps.append) == "ok"
|
||||||
|
assert calls["n"] == 1
|
||||||
|
assert sleeps == [] # no backoff when nothing failed
|
||||||
|
|
||||||
|
|
||||||
|
def test_retry_recovers_from_transient_network_error():
|
||||||
|
# The real symptom: <urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING]> arrives
|
||||||
|
# as musicbrainzngs.NetworkError. A couple of retries clears it.
|
||||||
|
sleeps: list[float] = []
|
||||||
|
calls = {"n": 0}
|
||||||
|
|
||||||
|
def fn():
|
||||||
|
calls["n"] += 1
|
||||||
|
if calls["n"] < 3:
|
||||||
|
raise musicbrainzngs.NetworkError("SSL: UNEXPECTED_EOF_WHILE_READING")
|
||||||
|
return "recovered"
|
||||||
|
|
||||||
|
assert _with_retry(fn, sleep=sleeps.append) == "recovered"
|
||||||
|
assert calls["n"] == 3
|
||||||
|
assert sleeps == [1.0, 2.0] # exponential backoff before each retry
|
||||||
|
|
||||||
|
|
||||||
|
def test_retry_recovers_from_transient_response_error():
|
||||||
|
# MusicBrainz's own 503 rate-limit surfaces as ResponseError.
|
||||||
|
calls = {"n": 0}
|
||||||
|
|
||||||
|
def fn():
|
||||||
|
calls["n"] += 1
|
||||||
|
if calls["n"] < 2:
|
||||||
|
raise musicbrainzngs.ResponseError("503 Service Unavailable")
|
||||||
|
return "ok"
|
||||||
|
|
||||||
|
assert _with_retry(fn, sleep=lambda _: None) == "ok"
|
||||||
|
assert calls["n"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_retry_reraises_after_exhausting_attempts():
|
||||||
|
sleeps: list[float] = []
|
||||||
|
|
||||||
|
def fn():
|
||||||
|
raise musicbrainzngs.NetworkError("still down")
|
||||||
|
|
||||||
|
with pytest.raises(musicbrainzngs.NetworkError):
|
||||||
|
_with_retry(fn, attempts=4, sleep=sleeps.append)
|
||||||
|
assert sleeps == [1.0, 2.0, 4.0] # 3 backoffs across 4 attempts
|
||||||
|
|
||||||
|
|
||||||
|
def test_retry_does_not_swallow_non_transient_errors():
|
||||||
|
calls = {"n": 0}
|
||||||
|
|
||||||
|
def fn():
|
||||||
|
calls["n"] += 1
|
||||||
|
raise ValueError("bug, not a network blip")
|
||||||
|
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
_with_retry(fn, sleep=lambda _: None)
|
||||||
|
assert calls["n"] == 1 # non-transient errors propagate immediately, no retry
|
||||||
|
|||||||
@@ -0,0 +1,179 @@
|
|||||||
|
from datetime import date
|
||||||
|
|
||||||
|
from lyra_worker.qobuz_gate import (
|
||||||
|
QobuzPacing,
|
||||||
|
today_cap,
|
||||||
|
is_allowed,
|
||||||
|
next_gap_seconds,
|
||||||
|
gate_open,
|
||||||
|
record_download,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---- QobuzPacing.from_config -------------------------------------------------
|
||||||
|
|
||||||
|
def test_from_config_defaults():
|
||||||
|
p = QobuzPacing.from_config({})
|
||||||
|
assert p.enabled is True
|
||||||
|
assert p.active_start == 8 and p.active_end == 23
|
||||||
|
assert p.daily_cap == 40
|
||||||
|
assert p.warmup_start_date == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_from_config_reads_values():
|
||||||
|
p = QobuzPacing.from_config({
|
||||||
|
"qobuz.pacing.enabled": "false",
|
||||||
|
"qobuz.pacing.activeStartHour": "9",
|
||||||
|
"qobuz.pacing.activeEndHour": "22",
|
||||||
|
"qobuz.pacing.dailyCap": "25",
|
||||||
|
"qobuz.pacing.warmupStartDate": "2026-07-15",
|
||||||
|
})
|
||||||
|
assert p.enabled is False
|
||||||
|
assert p.active_start == 9 and p.active_end == 22
|
||||||
|
assert p.daily_cap == 25
|
||||||
|
assert p.warmup_start_date == "2026-07-15"
|
||||||
|
|
||||||
|
|
||||||
|
# ---- today_cap (warm-up ramp) ------------------------------------------------
|
||||||
|
|
||||||
|
def test_today_cap_no_warmup_uses_steady():
|
||||||
|
p = QobuzPacing(daily_cap=40, warmup_start_date="")
|
||||||
|
assert today_cap(p, date(2026, 7, 20)) == 40
|
||||||
|
|
||||||
|
|
||||||
|
def test_today_cap_ramp_boundaries():
|
||||||
|
p = QobuzPacing(daily_cap=40, warmup_start_date="2026-07-15")
|
||||||
|
assert today_cap(p, date(2026, 7, 15)) == 5 # day 0
|
||||||
|
assert today_cap(p, date(2026, 7, 17)) == 5 # day 2
|
||||||
|
assert today_cap(p, date(2026, 7, 18)) == 10 # day 3
|
||||||
|
assert today_cap(p, date(2026, 7, 21)) == 10 # day 6
|
||||||
|
assert today_cap(p, date(2026, 7, 22)) == 20 # day 7
|
||||||
|
assert today_cap(p, date(2026, 7, 28)) == 20 # day 13
|
||||||
|
assert today_cap(p, date(2026, 7, 29)) == 40 # day 14
|
||||||
|
assert today_cap(p, date(2026, 8, 30)) == 40 # long after
|
||||||
|
|
||||||
|
|
||||||
|
def test_today_cap_steady_below_ramp_step_is_the_ceiling():
|
||||||
|
# A small steady cap is never exceeded by a ramp step.
|
||||||
|
p = QobuzPacing(daily_cap=3, warmup_start_date="2026-07-15")
|
||||||
|
assert today_cap(p, date(2026, 7, 29)) == 3
|
||||||
|
|
||||||
|
|
||||||
|
# ---- is_allowed (the gate decision) -----------------------------------------
|
||||||
|
|
||||||
|
_P = QobuzPacing(enabled=True, active_start=8, active_end=23, daily_cap=40)
|
||||||
|
|
||||||
|
|
||||||
|
def test_disabled_always_allows():
|
||||||
|
p = QobuzPacing(enabled=False, active_start=8, active_end=23)
|
||||||
|
assert is_allowed(p, now_hour=3, count_today=999, cap=0, now_epoch=0, next_allowed_epoch=1e18) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_blocked_outside_active_hours():
|
||||||
|
assert is_allowed(_P, now_hour=7, count_today=0, cap=40, now_epoch=100, next_allowed_epoch=0) is False
|
||||||
|
assert is_allowed(_P, now_hour=23, count_today=0, cap=40, now_epoch=100, next_allowed_epoch=0) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_blocked_at_or_over_cap():
|
||||||
|
assert is_allowed(_P, now_hour=10, count_today=40, cap=40, now_epoch=100, next_allowed_epoch=0) is False
|
||||||
|
assert is_allowed(_P, now_hour=10, count_today=41, cap=40, now_epoch=100, next_allowed_epoch=0) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_blocked_before_next_allowed():
|
||||||
|
assert is_allowed(_P, now_hour=10, count_today=0, cap=40, now_epoch=100, next_allowed_epoch=200) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_allowed_when_all_conditions_met():
|
||||||
|
assert is_allowed(_P, now_hour=10, count_today=5, cap=40, now_epoch=300, next_allowed_epoch=200) is True
|
||||||
|
|
||||||
|
|
||||||
|
# ---- next_gap_seconds (spread + jitter) -------------------------------------
|
||||||
|
|
||||||
|
def test_next_gap_spreads_cap_across_active_window():
|
||||||
|
# 15h window (8..23) / cap 40 = 1350s base; ±30%; midpoint rand -> base.
|
||||||
|
p = QobuzPacing(active_start=8, active_end=23, daily_cap=40)
|
||||||
|
assert abs(next_gap_seconds(p, cap=40, rand01=0.5) - 1350.0) < 1.0
|
||||||
|
assert abs(next_gap_seconds(p, cap=40, rand01=0.0) - 1350.0 * 0.7) < 1.0 # low end
|
||||||
|
assert abs(next_gap_seconds(p, cap=40, rand01=1.0) - 1350.0 * 1.3) < 1.0 # high end
|
||||||
|
|
||||||
|
|
||||||
|
def test_next_gap_has_a_floor():
|
||||||
|
# A huge cap would give a tiny gap; a 60s floor protects the request rate.
|
||||||
|
p = QobuzPacing(active_start=8, active_end=23, daily_cap=100000)
|
||||||
|
assert next_gap_seconds(p, cap=100000, rand01=0.0) == 60.0
|
||||||
|
|
||||||
|
|
||||||
|
# ---- DB helpers: gate_open / record_download --------------------------------
|
||||||
|
|
||||||
|
def _set(conn, key, value):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
|
||||||
|
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
|
||||||
|
(key, value),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _get(conn, key):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT value FROM "Config" WHERE key=%s', (key,))
|
||||||
|
row = cur.fetchone()
|
||||||
|
return row[0] if row else None
|
||||||
|
|
||||||
|
|
||||||
|
def test_record_download_increments_and_sets_next_allowed(conn):
|
||||||
|
_set(conn, "qobuz.pacing.dailyCap", "40")
|
||||||
|
record_download(conn, rand01=0.5)
|
||||||
|
assert _get(conn, "qobuz.pacing.countToday") == "1"
|
||||||
|
assert _get(conn, "qobuz.pacing.countDay") is not None
|
||||||
|
assert _get(conn, "qobuz.pacing.nextAllowedAt") is not None
|
||||||
|
record_download(conn, rand01=0.5)
|
||||||
|
assert _get(conn, "qobuz.pacing.countToday") == "2"
|
||||||
|
|
||||||
|
|
||||||
|
def test_record_download_resets_counter_on_new_day(conn):
|
||||||
|
# A stale count from a previous day must reset to 1, not accumulate.
|
||||||
|
_set(conn, "qobuz.pacing.dailyCap", "40")
|
||||||
|
_set(conn, "qobuz.pacing.countDay", "2000-01-01")
|
||||||
|
_set(conn, "qobuz.pacing.countToday", "39")
|
||||||
|
record_download(conn, rand01=0.5)
|
||||||
|
assert _get(conn, "qobuz.pacing.countToday") == "1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_gate_open_with_always_open_window(conn):
|
||||||
|
# An all-day window (0..24) removes the time-of-day dependency: no prior downloads and no
|
||||||
|
# next-allowed set → the gate is open regardless of when the test runs.
|
||||||
|
_set(conn, "qobuz.pacing.activeStartHour", "0")
|
||||||
|
_set(conn, "qobuz.pacing.activeEndHour", "24")
|
||||||
|
assert gate_open(conn) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_gate_closed_by_empty_active_window(conn):
|
||||||
|
# start == end is an empty window (h >= 5 AND h < 5 is never true) → always closed.
|
||||||
|
_set(conn, "qobuz.pacing.activeStartHour", "5")
|
||||||
|
_set(conn, "qobuz.pacing.activeEndHour", "5")
|
||||||
|
assert gate_open(conn) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_gate_closed_when_cap_reached_today(conn):
|
||||||
|
_set(conn, "qobuz.pacing.activeStartHour", "0")
|
||||||
|
_set(conn, "qobuz.pacing.activeEndHour", "24")
|
||||||
|
_set(conn, "qobuz.pacing.dailyCap", "3")
|
||||||
|
# Simulate 3 downloads already today.
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute("SELECT current_date::text")
|
||||||
|
today = cur.fetchone()[0]
|
||||||
|
_set(conn, "qobuz.pacing.countDay", today)
|
||||||
|
_set(conn, "qobuz.pacing.countToday", "3")
|
||||||
|
assert gate_open(conn) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_gate_ignores_stale_count_from_a_previous_day(conn):
|
||||||
|
# A high count from yesterday must not keep the gate closed today.
|
||||||
|
_set(conn, "qobuz.pacing.activeStartHour", "0")
|
||||||
|
_set(conn, "qobuz.pacing.activeEndHour", "24")
|
||||||
|
_set(conn, "qobuz.pacing.dailyCap", "3")
|
||||||
|
_set(conn, "qobuz.pacing.countDay", "2000-01-01")
|
||||||
|
_set(conn, "qobuz.pacing.countToday", "99")
|
||||||
|
assert gate_open(conn) is True
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
"""The Qobuz budget gate, wired into the pipeline: when the gate is closed the job falls through
|
||||||
|
to another source; when open it uses Qobuz and records the download; a Qobuz-only setup is never
|
||||||
|
gated (nothing to fall through to)."""
|
||||||
|
from lyra_worker.adapters.fakes import FakeQobuz, FakeSoulseek
|
||||||
|
from lyra_worker.claim import claim_next
|
||||||
|
from lyra_worker.pipeline import run_pipeline
|
||||||
|
from tests.conftest import insert_request
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg(conn, key, value):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
|
||||||
|
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
|
||||||
|
(key, value),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _open_gate(conn):
|
||||||
|
_cfg(conn, "qobuz.pacing.activeStartHour", "0")
|
||||||
|
_cfg(conn, "qobuz.pacing.activeEndHour", "24")
|
||||||
|
|
||||||
|
|
||||||
|
def _closed_gate(conn):
|
||||||
|
_cfg(conn, "qobuz.pacing.activeStartHour", "5") # empty window (5..5) → always closed
|
||||||
|
_cfg(conn, "qobuz.pacing.activeEndHour", "5")
|
||||||
|
|
||||||
|
|
||||||
|
def _state(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT state FROM "Job" WHERE id = %s', (job_id,))
|
||||||
|
return cur.fetchone()[0]
|
||||||
|
|
||||||
|
|
||||||
|
def _chosen(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT source FROM "Candidate" WHERE "jobId" = %s AND chosen = true', (job_id,))
|
||||||
|
row = cur.fetchone()
|
||||||
|
return row[0] if row else None
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg_get(conn, key):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT value FROM "Config" WHERE key = %s', (key,))
|
||||||
|
row = cur.fetchone()
|
||||||
|
return row[0] if row else None
|
||||||
|
|
||||||
|
|
||||||
|
def test_closed_gate_drops_qobuz_and_falls_through(conn):
|
||||||
|
_closed_gate(conn)
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [FakeQobuz(), FakeSoulseek()], dest_root="/tmp/lib-gate")
|
||||||
|
assert _state(conn, job_id) == "imported"
|
||||||
|
assert _chosen(conn, job_id) == "soulseek" # Qobuz dropped by the closed gate
|
||||||
|
|
||||||
|
|
||||||
|
def test_open_gate_uses_qobuz_and_records_the_download(conn):
|
||||||
|
_open_gate(conn)
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [FakeQobuz(), FakeSoulseek()], dest_root="/tmp/lib-gate")
|
||||||
|
assert _state(conn, job_id) == "imported"
|
||||||
|
assert _chosen(conn, job_id) == "qobuz"
|
||||||
|
assert _cfg_get(conn, "qobuz.pacing.countToday") == "1" # recorded against today's budget
|
||||||
|
assert _cfg_get(conn, "qobuz.pacing.nextAllowedAt") is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_qobuz_only_setup_is_never_gated(conn):
|
||||||
|
_closed_gate(conn)
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [FakeQobuz()], dest_root="/tmp/lib-gate")
|
||||||
|
assert _state(conn, job_id) == "imported"
|
||||||
|
assert _chosen(conn, job_id) == "qobuz" # not dropped — nothing to fall through to
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_qobuz_download_does_not_touch_the_counter(conn):
|
||||||
|
_closed_gate(conn) # forces Soulseek
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [FakeQobuz(), FakeSoulseek()], dest_root="/tmp/lib-gate")
|
||||||
|
assert _chosen(conn, job_id) == "soulseek"
|
||||||
|
assert _cfg_get(conn, "qobuz.pacing.countToday") is None # only Qobuz downloads count
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
from lyra_worker.claim import claim_next, requeue_or_fail
|
||||||
|
from tests.conftest import insert_request
|
||||||
|
|
||||||
|
|
||||||
|
def _set_attempts(conn, job_id, attempts, state="matching", stage="match"):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'UPDATE "Job" SET attempts = %s, state = %s, "currentStage" = %s, "claimedAt" = now() '
|
||||||
|
'WHERE id = %s',
|
||||||
|
(attempts, state, stage, job_id),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _job(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT state, "currentStage", "claimedAt", error FROM "Job" WHERE id = %s', (job_id,))
|
||||||
|
return cur.fetchone()
|
||||||
|
|
||||||
|
|
||||||
|
def _request_status(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT status FROM "Request" WHERE id = (SELECT "requestId" FROM "Job" WHERE id = %s)',
|
||||||
|
(job_id,),
|
||||||
|
)
|
||||||
|
return cur.fetchone()[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_below_cap_requeues_to_requested(conn):
|
||||||
|
# A transient failure (e.g. MB SSL EOF) mid-'matching' must not strand the job — it returns
|
||||||
|
# to 'requested' so claim_next picks it up again instead of orphaning it.
|
||||||
|
jid = insert_request(conn)
|
||||||
|
_set_attempts(conn, jid, 2)
|
||||||
|
requeue_or_fail(conn, jid, "SSL: UNEXPECTED_EOF_WHILE_READING")
|
||||||
|
st, stage, claimed_at, err = _job(conn, jid)
|
||||||
|
assert st == "requested"
|
||||||
|
assert stage == "intake"
|
||||||
|
assert claimed_at is None
|
||||||
|
assert err is None
|
||||||
|
assert _request_status(conn, jid) == "pending"
|
||||||
|
assert claim_next(conn) == jid # immediately re-claimable
|
||||||
|
|
||||||
|
|
||||||
|
def test_at_cap_marks_needs_attention_with_error(conn):
|
||||||
|
jid = insert_request(conn)
|
||||||
|
_set_attempts(conn, jid, 5) # hit the cap
|
||||||
|
requeue_or_fail(conn, jid, "SSL: UNEXPECTED_EOF_WHILE_READING")
|
||||||
|
st, _stage, claimed_at, err = _job(conn, jid)
|
||||||
|
assert st == "needs_attention"
|
||||||
|
assert err == "SSL: UNEXPECTED_EOF_WHILE_READING"
|
||||||
|
assert claimed_at is None
|
||||||
|
assert _request_status(conn, jid) == "needs_attention"
|
||||||
|
assert claim_next(conn) is None # not re-claimed
|
||||||
|
|
||||||
|
|
||||||
|
def test_requeues_until_cap_then_fails(conn):
|
||||||
|
jid = insert_request(conn)
|
||||||
|
for attempts in (1, 2, 3, 4):
|
||||||
|
_set_attempts(conn, jid, attempts)
|
||||||
|
requeue_or_fail(conn, jid, "boom")
|
||||||
|
assert _job(conn, jid)[0] == "requested"
|
||||||
|
_set_attempts(conn, jid, 5)
|
||||||
|
requeue_or_fail(conn, jid, "boom")
|
||||||
|
assert _job(conn, jid)[0] == "needs_attention"
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_job_is_noop(conn):
|
||||||
|
requeue_or_fail(conn, "does-not-exist", "boom") # must not raise
|
||||||
@@ -0,0 +1,218 @@
|
|||||||
|
"""slskd download behaviour: abandoned transfers are cancelled (no pile-up), a stalled peer is
|
||||||
|
dropped fast, a slow-but-progressing peer is kept, and search candidates are ranked fast/free
|
||||||
|
peers first."""
|
||||||
|
import json
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
import lyra_worker.adapters._slskd as slskd_mod
|
||||||
|
from lyra_worker.adapters._slskd import (
|
||||||
|
_ETA_CAP_SECONDS,
|
||||||
|
_MAX_XFER_POLLS,
|
||||||
|
_MIN_XFER_POLLS,
|
||||||
|
_POLL_SECONDS,
|
||||||
|
SlskdClient,
|
||||||
|
_eta_seconds,
|
||||||
|
_max_polls,
|
||||||
|
_parse_search_responses,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _Resp:
|
||||||
|
def __init__(self, payload=None):
|
||||||
|
self._p = payload if payload is not None else {}
|
||||||
|
|
||||||
|
def raise_for_status(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def json(self):
|
||||||
|
return self._p
|
||||||
|
|
||||||
|
|
||||||
|
class _SeqRequests:
|
||||||
|
"""post OK; get() returns successive poll payloads (last one repeats); records DELETEs."""
|
||||||
|
def __init__(self, polls):
|
||||||
|
self._polls = list(polls)
|
||||||
|
self._i = 0
|
||||||
|
self.deleted = []
|
||||||
|
|
||||||
|
def post(self, url, **kw):
|
||||||
|
return _Resp({})
|
||||||
|
|
||||||
|
def get(self, url, **kw):
|
||||||
|
p = self._polls[min(self._i, len(self._polls) - 1)]
|
||||||
|
self._i += 1
|
||||||
|
return _Resp(p)
|
||||||
|
|
||||||
|
def delete(self, url, **kw):
|
||||||
|
self.deleted.append(url)
|
||||||
|
return _Resp({})
|
||||||
|
|
||||||
|
|
||||||
|
def _poll(state, bytes_transferred, fid="t1"):
|
||||||
|
return {"directories": [{"files": [
|
||||||
|
{"filename": "f1.flac", "size": 100, "state": state,
|
||||||
|
"bytesTransferred": bytes_transferred, "id": fid},
|
||||||
|
]}]}
|
||||||
|
|
||||||
|
|
||||||
|
def _client(tmp_path):
|
||||||
|
return SlskdClient({"slskd.url": "http://x", "slskd.api_key": "k",
|
||||||
|
"slskd.downloadsDir": str(tmp_path)})
|
||||||
|
|
||||||
|
|
||||||
|
_REF = json.dumps({"username": "peerA", "files": [{"filename": "f1.flac", "size": 100}]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_stalled_peer_is_abandoned_and_cancelled(tmp_path, monkeypatch):
|
||||||
|
# No byte progress across polls → stall → TimeoutError → the transfer is cancelled.
|
||||||
|
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests([_poll("InProgress", 0)]))
|
||||||
|
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
|
||||||
|
monkeypatch.setattr(slskd_mod, "_STALL_POLLS", 3)
|
||||||
|
fake = slskd_mod.requests
|
||||||
|
|
||||||
|
with pytest.raises(TimeoutError):
|
||||||
|
_client(tmp_path).download(_REF, str(tmp_path / "dest"), lambda *_a: None)
|
||||||
|
assert any(u.endswith("/api/v0/transfers/downloads/peerA/t1") for u in fake.deleted)
|
||||||
|
|
||||||
|
|
||||||
|
def test_error_state_is_cancelled(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests([_poll("Completed, Errored", 0)]))
|
||||||
|
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
|
||||||
|
fake = slskd_mod.requests
|
||||||
|
with pytest.raises(RuntimeError):
|
||||||
|
_client(tmp_path).download(_REF, str(tmp_path / "dest"), lambda *_a: None)
|
||||||
|
assert any(u.endswith("/peerA/t1") for u in fake.deleted)
|
||||||
|
|
||||||
|
|
||||||
|
def test_slow_but_progressing_peer_is_not_abandoned(tmp_path, monkeypatch):
|
||||||
|
# Bytes climb for more polls than _STALL_POLLS, then complete → success, not a premature stall.
|
||||||
|
(tmp_path / "01.flac").write_bytes(b"audio") # so _retrieve finds the finished file
|
||||||
|
ref = json.dumps({"username": "peerA", "files": [{"filename": r"d\Album\01.flac", "size": 5}]})
|
||||||
|
polls = [
|
||||||
|
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 5,
|
||||||
|
"state": "InProgress", "bytesTransferred": b, "id": "t1"}]}]}
|
||||||
|
for b in (10, 20, 30, 40, 50)
|
||||||
|
] + [
|
||||||
|
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 5,
|
||||||
|
"state": "Completed, Succeeded", "bytesTransferred": 5, "id": "t1"}]}]}
|
||||||
|
]
|
||||||
|
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests(polls))
|
||||||
|
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
|
||||||
|
monkeypatch.setattr(slskd_mod, "_STALL_POLLS", 2) # would fire early if progress were ignored
|
||||||
|
|
||||||
|
out = _client(tmp_path).download(ref, str(tmp_path / "dest"), lambda *_a: None)
|
||||||
|
assert out["track_count"] == 1
|
||||||
|
assert slskd_mod.requests.deleted == [] # completed cleanly → nothing cancelled
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_ranks_free_and_fast_peers_first():
|
||||||
|
responses = [
|
||||||
|
{"username": "slow", "hasFreeUploadSlot": True, "uploadSpeed": 100, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"a\Artist - Album\01.flac", "size": 5}]},
|
||||||
|
{"username": "fast", "hasFreeUploadSlot": True, "uploadSpeed": 9000, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"b\Artist - Album\01.flac", "size": 5}]},
|
||||||
|
{"username": "queued", "hasFreeUploadSlot": False, "uploadSpeed": 99999, "queueLength": 40,
|
||||||
|
"files": [{"filename": r"c\Artist - Album\01.flac", "size": 5}]},
|
||||||
|
]
|
||||||
|
peers = [json.loads(c["source_ref"])["username"] for c in _parse_search_responses(responses)]
|
||||||
|
assert peers == ["fast", "slow", "queued"] # free+fast first; no-slot peer last despite speed
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_uses_requested_artist_when_present_in_path():
|
||||||
|
# Real "Rated R" MP3-over-FLAC bug: the artist is in a PARENT folder ("Rihanna\(2009) Rated
|
||||||
|
# R\…"), so the "Artist - Album" folder parse yields no artist and confidence tanks. When the
|
||||||
|
# requested artist is in the path, trust it so the candidate scores as Rihanna.
|
||||||
|
responses = [
|
||||||
|
{"username": "p", "hasFreeUploadSlot": True, "uploadSpeed": 1000, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"@@x\Music\Rihanna\(2009) Rated R\01. Mad House.flac", "size": 5}]},
|
||||||
|
]
|
||||||
|
c = _parse_search_responses(responses, artist="Rihanna")[0]
|
||||||
|
assert c["artist"] == "Rihanna"
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_without_artist_falls_back_to_folder_parse():
|
||||||
|
responses = [
|
||||||
|
{"username": "p", "hasFreeUploadSlot": True, "uploadSpeed": 1000, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"x\Some Band - Some Album\01.flac", "size": 5}]},
|
||||||
|
]
|
||||||
|
c = _parse_search_responses(responses)[0] # no artist arg → folder parse
|
||||||
|
assert c["artist"] == "Some Band"
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_prefers_shorter_transfer_over_raw_speed():
|
||||||
|
# The Hybrid Theory case: a peer advertising higher speed but serving a ~2x-larger rip (24-bit
|
||||||
|
# / bloated FLAC) should lose to a peer with lower speed but a smaller standard rip that
|
||||||
|
# finishes sooner — both are the same quality class to Lyra, so faster-to-finish wins.
|
||||||
|
responses = [
|
||||||
|
{"username": "bloated", "hasFreeUploadSlot": True, "uploadSpeed": 20000, "queueLength": 0,
|
||||||
|
"files": [{"filename": rf"x\LP - Album\{i:02}.flac", "size": 40_000_000} for i in range(12)]},
|
||||||
|
{"username": "lean", "hasFreeUploadSlot": True, "uploadSpeed": 15000, "queueLength": 0,
|
||||||
|
"files": [{"filename": rf"y\LP - Album\{i:02}.flac", "size": 24_000_000} for i in range(12)]},
|
||||||
|
]
|
||||||
|
# bloated est = 480MB/20000 = 24000s; lean est = 288MB/15000 = 19200s → lean finishes first
|
||||||
|
peers = [json.loads(c["source_ref"])["username"] for c in _parse_search_responses(responses)]
|
||||||
|
assert peers[0] == "lean"
|
||||||
|
|
||||||
|
|
||||||
|
def test_eta_seconds_from_measured_rate():
|
||||||
|
assert _eta_seconds(1000, 250, 250.0) == 3 # 750 bytes left / 250 B/s
|
||||||
|
assert _eta_seconds(1000, 0, 100.0) == 10
|
||||||
|
|
||||||
|
|
||||||
|
def test_eta_seconds_unknown_cases():
|
||||||
|
assert _eta_seconds(1000, 250, 0.0) is None # no speed sample yet
|
||||||
|
assert _eta_seconds(0, 0, 100.0) is None # unknown total size
|
||||||
|
assert _eta_seconds(1000, 1000, 100.0) is None # already complete
|
||||||
|
|
||||||
|
|
||||||
|
def test_max_polls_scales_backstop_to_album_size():
|
||||||
|
# A 521MB album (Drake "Scorpion", 25 FLACs) from a slow serial peer must get far more than
|
||||||
|
# the old flat 1200-poll/60min budget, or it gets abandoned mid-download and restarts from zero.
|
||||||
|
scorpion_bytes = 521_000_000
|
||||||
|
polls = _max_polls(scorpion_bytes)
|
||||||
|
assert polls > 1200 # more headroom than the old flat cap
|
||||||
|
assert polls * _POLL_SECONDS >= 90 * 60 # at least ~90 min of wall-clock budget
|
||||||
|
|
||||||
|
# A tiny single stays on the floor, not zero.
|
||||||
|
assert _max_polls(3_000_000) == _MIN_XFER_POLLS
|
||||||
|
assert _max_polls(0) == _MIN_XFER_POLLS
|
||||||
|
|
||||||
|
# An enormous set is clamped to the ceiling (bounds worst-case slot hold).
|
||||||
|
assert _max_polls(50_000_000_000) == _MAX_XFER_POLLS
|
||||||
|
|
||||||
|
# Monotonic: bigger album never gets a smaller budget.
|
||||||
|
assert _max_polls(200_000_000) <= _max_polls(400_000_000)
|
||||||
|
|
||||||
|
|
||||||
|
def test_eta_seconds_capped_when_peer_nearly_stalls():
|
||||||
|
# A near-stalled peer (tiny speed) would yield an ETA far past int4, overflowing the
|
||||||
|
# downloadEtaSeconds column write. The estimate must be clamped to a safe ceiling.
|
||||||
|
eta = _eta_seconds(30_000_000, 0, 0.001) # 30MB / 0.001 B/s ~= 3e10s uncapped
|
||||||
|
assert eta == _ETA_CAP_SECONDS
|
||||||
|
assert eta < 2_147_483_647 # fits in int4
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_reports_shrinking_eta(tmp_path, monkeypatch):
|
||||||
|
# A 900-byte file arriving 300 bytes per 3s poll → measured speed 100 B/s → ETA counts down
|
||||||
|
# (6s, 3s) then None on completion. on_progress gets (fraction, eta_seconds).
|
||||||
|
(tmp_path / "01.flac").write_bytes(b"x" * 900) # so _retrieve finds the finished file
|
||||||
|
ref = json.dumps({"username": "peerA", "files": [{"filename": r"d\Album\01.flac", "size": 900}]})
|
||||||
|
polls = [
|
||||||
|
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 900,
|
||||||
|
"state": "InProgress", "bytesTransferred": b, "id": "t1"}]}]}
|
||||||
|
for b in (300, 600)
|
||||||
|
] + [
|
||||||
|
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 900,
|
||||||
|
"state": "Completed, Succeeded", "bytesTransferred": 900, "id": "t1"}]}]}
|
||||||
|
]
|
||||||
|
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests(polls))
|
||||||
|
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
|
||||||
|
clock = iter(float(i) * 3 for i in range(100)) # 0,3,6,… deterministic monotonic time
|
||||||
|
monkeypatch.setattr(slskd_mod.time, "monotonic", lambda: next(clock))
|
||||||
|
|
||||||
|
calls: list = []
|
||||||
|
_client(tmp_path).download(ref, str(tmp_path / "dest"), lambda *a: calls.append(a))
|
||||||
|
|
||||||
|
etas = [a[1] for a in calls if len(a) > 1]
|
||||||
|
assert etas == [6, 3, None] # counts down at 100 B/s, then None once complete
|
||||||
@@ -0,0 +1,150 @@
|
|||||||
|
"""Soulseek search fallback: the Soulseek server drops searches containing blocklisted terms
|
||||||
|
(major-label artists / album titles) → 0 responses. search_album retries with album-only then
|
||||||
|
artist-only, keeping only path-matching results so it never grabs a same-titled foreign album."""
|
||||||
|
import json
|
||||||
|
|
||||||
|
import lyra_worker.adapters._slskd as slskd_mod
|
||||||
|
from lyra_worker.adapters._slskd import (
|
||||||
|
SlskdClient,
|
||||||
|
_keep_matching,
|
||||||
|
_norm,
|
||||||
|
_path_matches,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- pure helpers -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_norm_strips_diacritics_and_punctuation():
|
||||||
|
assert _norm("Beyoncé") == "beyonce"
|
||||||
|
assert _norm("I Am… Sasha Fierce") == "i am sasha fierce"
|
||||||
|
assert _norm("AC/DC") == "ac dc"
|
||||||
|
|
||||||
|
|
||||||
|
def test_path_matches_is_accent_and_punctuation_tolerant():
|
||||||
|
p = r"@@bckfj\All\Beyonce\I Am... Sasha Fierce CD1\03 Halo.flac"
|
||||||
|
assert _path_matches("Beyoncé", p) # accent folded
|
||||||
|
assert _path_matches("I Am… Sasha Fierce", p) # ellipsis / punctuation ignored
|
||||||
|
assert not _path_matches("Rihanna", p)
|
||||||
|
|
||||||
|
|
||||||
|
def test_path_matches_short_needle_requires_whole_string():
|
||||||
|
assert _path_matches("U2", r"x\U2 - Achtung Baby\01.flac")
|
||||||
|
assert not _path_matches("U2", r"x\Blur - 13\01.flac")
|
||||||
|
|
||||||
|
|
||||||
|
def test_keep_matching_drops_foreign_and_empty_responses():
|
||||||
|
responses = [
|
||||||
|
{"username": "a", "files": [{"filename": r"x\Rihanna - Unapologetic\01.flac"}]},
|
||||||
|
{"username": "b", "files": [{"filename": r"x\Someone Else - Unapologetic\01.flac"}]},
|
||||||
|
]
|
||||||
|
kept = _keep_matching(responses, "Rihanna")
|
||||||
|
assert [r["username"] for r in kept] == ["a"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- fallback orchestration (HTTP faked) ------------------------------------------------------
|
||||||
|
|
||||||
|
class _Resp:
|
||||||
|
def __init__(self, payload):
|
||||||
|
self._p = payload
|
||||||
|
|
||||||
|
def raise_for_status(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def json(self):
|
||||||
|
return self._p
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeSearch:
|
||||||
|
"""Routes slskd search HTTP by URL: POST returns an id, GET on the search returns a completed
|
||||||
|
state, GET on /responses returns the canned responses for that query. Records queries sent."""
|
||||||
|
def __init__(self, by_query):
|
||||||
|
self.by_query = by_query # {searchText: [responses]}
|
||||||
|
self.queries = [] # searchText values POSTed, in order
|
||||||
|
self._id_query = {}
|
||||||
|
self._n = 0
|
||||||
|
|
||||||
|
def post(self, url, **kw):
|
||||||
|
text = kw["json"]["searchText"]
|
||||||
|
self.queries.append(text)
|
||||||
|
self._n += 1
|
||||||
|
sid = f"s{self._n}"
|
||||||
|
self._id_query[sid] = text
|
||||||
|
return _Resp({"id": sid})
|
||||||
|
|
||||||
|
def get(self, url, **kw):
|
||||||
|
if url.endswith("/responses"):
|
||||||
|
sid = url.rsplit("/", 2)[-2]
|
||||||
|
return _Resp(self.by_query.get(self._id_query[sid], []))
|
||||||
|
return _Resp({"state": "Completed, TimedOut"})
|
||||||
|
|
||||||
|
|
||||||
|
def _client():
|
||||||
|
return SlskdClient({"slskd.url": "http://x", "slskd.api_key": "k", "slskd.downloadsDir": "/x"})
|
||||||
|
|
||||||
|
|
||||||
|
def _install(monkeypatch, fake):
|
||||||
|
monkeypatch.setattr(slskd_mod, "requests", fake)
|
||||||
|
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_primary_hit_does_not_trigger_fallback(monkeypatch):
|
||||||
|
fake = _FakeSearch({
|
||||||
|
"Rihanna Unapologetic": [
|
||||||
|
{"username": "a", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"x\Rihanna - Unapologetic\01.flac", "size": 5}]},
|
||||||
|
],
|
||||||
|
})
|
||||||
|
_install(monkeypatch, fake)
|
||||||
|
out = _client().search_album("Rihanna", "Unapologetic")
|
||||||
|
assert len(out) == 1
|
||||||
|
assert fake.queries == ["Rihanna Unapologetic"] # no fallback searches
|
||||||
|
|
||||||
|
|
||||||
|
def test_album_only_fallback_recovers_blocklisted_artist(monkeypatch):
|
||||||
|
# Primary (artist blocked) → 0; album-only returns the artist's folder → recovered.
|
||||||
|
fake = _FakeSearch({
|
||||||
|
"Beyoncé I Am… Sasha Fierce": [], # blocked
|
||||||
|
"i am sasha fierce": [
|
||||||
|
{"username": "a", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"x\Beyonce - I Am... Sasha Fierce\01.flac", "size": 5}]},
|
||||||
|
{"username": "b", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"x\Other Artist - I Am Sasha Fierce\01.flac", "size": 5}]},
|
||||||
|
],
|
||||||
|
})
|
||||||
|
_install(monkeypatch, fake)
|
||||||
|
out = _client().search_album("Beyoncé", "I Am… Sasha Fierce")
|
||||||
|
assert fake.queries == ["Beyoncé I Am… Sasha Fierce", "i am sasha fierce"]
|
||||||
|
peers = [json.loads(c["source_ref"])["username"] for c in out]
|
||||||
|
assert peers == ["a"] # foreign same-titled album dropped
|
||||||
|
|
||||||
|
|
||||||
|
def test_artist_only_fallback_when_album_title_blocked(monkeypatch):
|
||||||
|
# Album-only also blocked/empty → fall through to artist-only, filtered by album in path.
|
||||||
|
fake = _FakeSearch({
|
||||||
|
"Foo Lemonade": [],
|
||||||
|
"lemonade": [], # album title itself blocked
|
||||||
|
"foo": [
|
||||||
|
{"username": "a", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"x\Foo - Lemonade\01.flac", "size": 5}]},
|
||||||
|
],
|
||||||
|
})
|
||||||
|
_install(monkeypatch, fake)
|
||||||
|
out = _client().search_album("Foo", "Lemonade")
|
||||||
|
assert fake.queries == ["Foo Lemonade", "lemonade", "foo"]
|
||||||
|
assert len(out) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_fallback_yields_nothing_when_title_too_generic(monkeypatch):
|
||||||
|
# 'Adele 21': artist blocked, album title '21' returns lots of non-Adele content → filtered
|
||||||
|
# to nothing rather than grabbing a wrong album.
|
||||||
|
fake = _FakeSearch({
|
||||||
|
"Adele 21": [],
|
||||||
|
"21": [
|
||||||
|
{"username": "z", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"x\21 Savage - Savage Mode\01.flac", "size": 5}]},
|
||||||
|
],
|
||||||
|
"adele": [], # artist blocked
|
||||||
|
})
|
||||||
|
_install(monkeypatch, fake)
|
||||||
|
out = _client().search_album("Adele", "21")
|
||||||
|
assert out == []
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
"""MusicBrainz picks an arbitrary releases[0] that is often a deluxe/expanded edition with more
|
||||||
|
tracks (and a longer runtime) than the standard album a source legitimately delivers. Completeness
|
||||||
|
and the duration guard must be judged against what the CHOSEN SOURCE offered, not MB's canonical
|
||||||
|
edition — otherwise a complete standard-edition download is falsely rejected as 'incomplete'
|
||||||
|
(The Script "No Sound Without Silence": 11 delivered vs MB 12) or 'too short' (Skillet "Unleashed":
|
||||||
|
12 delivered ~40 min vs MB's 20-track deluxe ~73 min). A genuinely short download must still fail.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
from dataclasses import replace
|
||||||
|
|
||||||
|
from lyra_worker.adapters.fakes import FakeQobuz
|
||||||
|
from lyra_worker.claim import claim_next
|
||||||
|
from lyra_worker.pipeline import run_pipeline
|
||||||
|
from lyra_worker.types import DownloadResult, MBTarget
|
||||||
|
from tests.conftest import insert_request
|
||||||
|
|
||||||
|
|
||||||
|
class _StandardEditionQobuz(FakeQobuz):
|
||||||
|
"""Qobuz-like source offering a fixed-size *standard* edition (source_tracks), independent of
|
||||||
|
MB's possibly-larger deluxe count; stages that many real (non-empty) files and reports them."""
|
||||||
|
def __init__(self, source_tracks):
|
||||||
|
super().__init__()
|
||||||
|
self._n = source_tracks
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
return super().search(replace(target, track_count=self._n))
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
os.makedirs(dest, exist_ok=True)
|
||||||
|
for i in range(candidate.track_count):
|
||||||
|
with open(os.path.join(dest, f"{i + 1:02d} Track.flac"), "wb") as fh:
|
||||||
|
fh.write(b"audio")
|
||||||
|
on_progress(1.0)
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=candidate.track_count)
|
||||||
|
|
||||||
|
|
||||||
|
class _PartialQobuz(FakeQobuz):
|
||||||
|
"""Promises `promised` tracks but one fails mid-download: stages and reports `promised - 1`
|
||||||
|
(the real John Mayer "Roll it on Home" IncompleteRead case)."""
|
||||||
|
def __init__(self, promised):
|
||||||
|
super().__init__()
|
||||||
|
self._n = promised
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
return super().search(replace(target, track_count=self._n))
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
os.makedirs(dest, exist_ok=True)
|
||||||
|
delivered = candidate.track_count - 1
|
||||||
|
for i in range(delivered):
|
||||||
|
with open(os.path.join(dest, f"{i + 1:02d} Track.flac"), "wb") as fh:
|
||||||
|
fh.write(b"audio")
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=delivered)
|
||||||
|
|
||||||
|
|
||||||
|
class _DeluxeResolver:
|
||||||
|
"""Resolver whose canonical release is a larger (deluxe) edition than the source delivers."""
|
||||||
|
def __init__(self, mb_tracks, mb_total_s):
|
||||||
|
self._t, self._s = mb_tracks, mb_total_s
|
||||||
|
|
||||||
|
def resolve(self, artist, album):
|
||||||
|
return MBTarget(artist=artist, album=album, track_count=self._t, total_duration_s=self._s)
|
||||||
|
|
||||||
|
|
||||||
|
def _state_error(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT state, error FROM "Job" WHERE id = %s', (job_id,))
|
||||||
|
return cur.fetchone()
|
||||||
|
|
||||||
|
|
||||||
|
def test_standard_edition_not_rejected_as_incomplete(conn):
|
||||||
|
# The Script: source delivers the 11-track standard edition; MB's canonical release is 12.
|
||||||
|
job_id = insert_request(conn, artist="The Script", album="No Sound Without Silence")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [_StandardEditionQobuz(source_tracks=11)],
|
||||||
|
resolver=_DeluxeResolver(mb_tracks=12, mb_total_s=2943),
|
||||||
|
measure_duration=lambda _s: 2700.0, dest_root="/tmp/lib-se")
|
||||||
|
assert _state_error(conn, job_id) == ("imported", None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_smaller_edition_not_rejected_as_too_short(conn):
|
||||||
|
# Skillet: source delivers the 12-track standard (~40 min); MB's release is a 20-track deluxe
|
||||||
|
# (~73 min). The unscaled duration guard falsely rejects; scaling to the delivered edition
|
||||||
|
# (12/20) must let it pass.
|
||||||
|
job_id = insert_request(conn, artist="Skillet", album="Unleashed")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [_StandardEditionQobuz(source_tracks=12)],
|
||||||
|
resolver=_DeluxeResolver(mb_tracks=20, mb_total_s=4380),
|
||||||
|
measure_duration=lambda _s: 2400.0, dest_root="/tmp/lib-se")
|
||||||
|
assert _state_error(conn, job_id) == ("imported", None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_genuinely_incomplete_still_rejected(conn):
|
||||||
|
# John Mayer: the source promised 12 tracks but one failed mid-download → only 11 land.
|
||||||
|
# This is a real partial and must still be rejected (the fix must not swallow it).
|
||||||
|
job_id = insert_request(conn, artist="John Mayer", album="The Search for Everything")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [_PartialQobuz(promised=12)],
|
||||||
|
resolver=_DeluxeResolver(mb_tracks=12, mb_total_s=2580),
|
||||||
|
measure_duration=lambda _s: 2400.0, dest_root="/tmp/lib-se")
|
||||||
|
assert _state_error(conn, job_id) == ("needs_attention", "incomplete download")
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
"""Upgrade guard: when an album is already in the library and no currently-available source can
|
||||||
|
beat it (e.g. Qobuz is gated by pacing and only a same-quality Soulseek copy is left), the pipeline
|
||||||
|
must NOT waste a download — keep-if-better would discard it anyway. It should skip and let the
|
||||||
|
release be re-attempted later (when Qobuz frees up). A genuinely better source still upgrades."""
|
||||||
|
import os
|
||||||
|
|
||||||
|
from lyra_worker.claim import claim_next
|
||||||
|
from lyra_worker.pipeline import run_pipeline
|
||||||
|
from lyra_worker.types import Candidate, DownloadResult, Quality
|
||||||
|
|
||||||
|
_CD = Quality(fmt="FLAC", lossless=True, bit_depth=16, sample_rate=44100) # class 2
|
||||||
|
_HIRES = Quality(fmt="FLAC", lossless=True, bit_depth=24, sample_rate=96000) # class 3
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg(conn, key, value):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
|
||||||
|
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
|
||||||
|
(key, value),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
class _Counting:
|
||||||
|
def __init__(self, name, tier, quality):
|
||||||
|
self.name, self.tier, self._q = name, tier, quality
|
||||||
|
self.calls = 0
|
||||||
|
self.searches = 0
|
||||||
|
|
||||||
|
def health(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
self.searches += 1
|
||||||
|
return [Candidate(source=self.name, source_ref=self.name, matched_artist=target.artist,
|
||||||
|
matched_album=target.album, quality=self._q, track_count=3, source_tier=self.tier)]
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
self.calls += 1
|
||||||
|
os.makedirs(dest, exist_ok=True)
|
||||||
|
for i in range(3):
|
||||||
|
with open(os.path.join(dest, f"0{i}.flac"), "wb") as fh:
|
||||||
|
fh.write(b"a")
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=3)
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_library(conn, artist, album, quality_class, path):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "LibraryItem" (id, artist, album, path, source, format, "qualityClass", "importedAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, %s, %s, %s, 'soulseek', 'FLAC', %s, now())",
|
||||||
|
(artist, album, path, quality_class),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _insert_upgrade_job(conn, artist, album):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "MonitoredRelease" (id, "artistMbid", "artistName", "rgMbid", album, '
|
||||||
|
'monitored, state, "secondaryTypes", "createdAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, 'm', %s, 'rg', %s, true, 'grabbed', '{}', now()) RETURNING id",
|
||||||
|
(artist, album),
|
||||||
|
)
|
||||||
|
rel_id = cur.fetchone()[0]
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Request" (id, artist, album, status, "monitoredReleaseId", "createdAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, %s, %s, 'pending', %s, now()) RETURNING id",
|
||||||
|
(artist, album, rel_id),
|
||||||
|
)
|
||||||
|
req = cur.fetchone()[0]
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Job" (id, "requestId", state, "currentStage", attempts, "createdAt", "updatedAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, %s, 'requested', 'intake', 0, now(), now()) RETURNING id",
|
||||||
|
(req,),
|
||||||
|
)
|
||||||
|
job = cur.fetchone()[0]
|
||||||
|
conn.commit()
|
||||||
|
return job
|
||||||
|
|
||||||
|
|
||||||
|
def _lib(conn, artist, album):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT path, "qualityClass" FROM "LibraryItem" WHERE artist=%s AND album=%s', (artist, album))
|
||||||
|
return cur.fetchone()
|
||||||
|
|
||||||
|
|
||||||
|
def _state(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT state FROM "Job" WHERE id=%s', (job_id,))
|
||||||
|
return cur.fetchone()[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_skipped_pre_search_when_qobuz_gated(conn):
|
||||||
|
# Qobuz gated (empty active window) + existing >= CD copy: only hi-res Qobuz could improve it,
|
||||||
|
# so defer WITHOUT even searching — avoids re-searching every owned album each monitor cycle.
|
||||||
|
_cfg(conn, "qobuz.pacing.activeStartHour", "5")
|
||||||
|
_cfg(conn, "qobuz.pacing.activeEndHour", "5") # empty window → gate closed
|
||||||
|
_seed_library(conn, "A", "B", 2, "/m/existing")
|
||||||
|
job_id = _insert_upgrade_job(conn, "A", "B")
|
||||||
|
claim_next(conn)
|
||||||
|
soulseek = _Counting("soulseek", 1, _CD)
|
||||||
|
run_pipeline(conn, job_id, [soulseek], dest_root="/tmp/lib-ug", upgrade_cutoff=3)
|
||||||
|
assert soulseek.searches == 0 # no wasted search
|
||||||
|
assert soulseek.calls == 0 # no wasted download
|
||||||
|
assert _lib(conn, "A", "B") == ("/m/existing", 2)
|
||||||
|
assert _state(conn, job_id) == "imported"
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_skipped_post_search_when_only_same_quality_found(conn):
|
||||||
|
# Gate OPEN, so it searches, but the only source found is same-quality → skip the download.
|
||||||
|
_cfg(conn, "qobuz.pacing.activeStartHour", "0")
|
||||||
|
_cfg(conn, "qobuz.pacing.activeEndHour", "24") # gate open
|
||||||
|
_seed_library(conn, "A", "B", 2, "/m/existing")
|
||||||
|
job_id = _insert_upgrade_job(conn, "A", "B")
|
||||||
|
claim_next(conn)
|
||||||
|
soulseek = _Counting("soulseek", 1, _CD)
|
||||||
|
run_pipeline(conn, job_id, [soulseek], dest_root="/tmp/lib-ug", upgrade_cutoff=3)
|
||||||
|
assert soulseek.searches == 1 # it did search (gate open)
|
||||||
|
assert soulseek.calls == 0 # but skipped the useless download
|
||||||
|
assert _lib(conn, "A", "B") == ("/m/existing", 2)
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_proceeds_when_a_better_source_is_available(conn):
|
||||||
|
# Existing class-2 copy; a class-3 (hi-res) source IS available → download + upgrade.
|
||||||
|
_seed_library(conn, "A", "B", 2, "/m/existing")
|
||||||
|
job_id = _insert_upgrade_job(conn, "A", "B")
|
||||||
|
claim_next(conn)
|
||||||
|
qobuz = _Counting("qobuz", 0, _HIRES)
|
||||||
|
run_pipeline(conn, job_id, [qobuz], dest_root="/tmp/lib-ug2", upgrade_cutoff=3)
|
||||||
|
assert qobuz.calls == 1 # downloaded the better copy
|
||||||
|
assert _lib(conn, "A", "B")[1] == 3 # upgraded to hi-res
|
||||||
Reference in New Issue
Block a user