Compare commits
59 Commits
e04c1c9795
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 395944f3e1 | |||
| 10430037fe | |||
| 01657d4e1b | |||
| 0ad1fd4cd5 | |||
| 7ed8ca2848 | |||
| 5f59283bf0 | |||
| d2a86932e0 | |||
| 73f1ae6e2f | |||
| c49cee9a20 | |||
| db8b43b0e8 | |||
| 434cd2b22c | |||
| d7ea29a97a | |||
| e8483ee22b | |||
| ca5bbfa9da | |||
| 675d964bbe | |||
| 57f97e9838 | |||
| 1fc61a0f5a | |||
| 67f374c470 | |||
| d47ed12e8e | |||
| b0eec41823 | |||
| 108114ffa3 | |||
| 6451cf3a3c | |||
| 927266d042 | |||
| c6945bd89e | |||
| 3f624c3395 | |||
| ecec0b6073 | |||
| b78867435a | |||
| 3d30e7dc8b | |||
| 8c95ff735b | |||
| bff590a00f | |||
| 20869bdc13 | |||
| 9c2495825f | |||
| 878eac066e | |||
| 603c62ef87 | |||
| 06e7f91897 | |||
| 0443f10a3b | |||
| a8ea3441fc | |||
| ccaaa41fec | |||
| b1f6408da6 | |||
| 55cf88f66e | |||
| 0c5e73eb17 | |||
| 0325467aba | |||
| c7bab8e997 | |||
| 53e7f0b7f2 | |||
| f72d193ef7 | |||
| 4a768b7122 | |||
| 9915bfbda2 | |||
| 7e27654b5a | |||
| 288d55269d | |||
| 48d5f287b2 | |||
| 947c86bc8f | |||
| a8c1600111 | |||
| 202fc0db3c | |||
| e8fd4bb0a9 | |||
| 194b4911c7 | |||
| efdd592761 | |||
| a09093f798 | |||
| e74d1d9c7b | |||
| 9dd235c2bc |
@@ -7,83 +7,54 @@ sources of differing quality: **Qobuz** (lossless/hi-res), **Soulseek** (P2P), a
|
|||||||
**ListenBrainz** and **Last.fm** (Spotify's recommendation/related-artists APIs were
|
**ListenBrainz** and **Last.fm** (Spotify's recommendation/related-artists APIs were
|
||||||
retired for new apps, so discovery is MetaBrainz-native and MBID-clean — no Spotify).
|
retired for new apps, so discovery is MetaBrainz-native and MBID-clean — no Spotify).
|
||||||
|
|
||||||
## Status
|
It runs on your own server as a small `docker compose` stack. This README covers getting
|
||||||
|
it running; see **[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md)** for the full feature tour,
|
||||||
|
publishing pre-built images, and running behind a VPN.
|
||||||
|
|
||||||
Built in three slices:
|
## What it does
|
||||||
|
|
||||||
1. **Acquisition engine** — ✅ **done**. Resolve a request against MusicBrainz,
|
- **Acquisition** — resolve a request against MusicBrainz, match/rank across
|
||||||
match/rank across Qobuz/Soulseek/YouTube, download the best source into an
|
Qobuz/Soulseek/YouTube, download the best source, verify completeness, tag, and organize
|
||||||
isolated staging dir, verify completeness, tag (mutagen), and atomically promote
|
into `Artist/Album (Year)/## Title.ext`.
|
||||||
the clean album into `Artist/Album (Year)/## Title.ext`.
|
- **Library manager** — follow artists (live MB search), per-release monitor toggles, a
|
||||||
2. **Library manager** — ✅ **done**. Watched artists (live MB search + follow),
|
wanted list with retry + quality-upgrade, a background monitor that auto-grabs new
|
||||||
per-release monitor toggles + auto-monitor-future, a wanted list with retry +
|
releases, and a scan that ingests albums already on disk.
|
||||||
quality-upgrade window, and a worker "monitor" that discovers new releases and
|
- **Discovery** (`/discover`) — recommendation-driven suggestions from ListenBrainz +
|
||||||
auto-grabs through the acquisition pipeline. Plus a **library scan** that ingests
|
Last.fm similar-artists, with Follow / Want / Dismiss and a read-only artist preview.
|
||||||
existing on-disk albums as have + monitored + followed, and a per-artist
|
- **Browse & watch** — Last.fm top artists/albums (`/lastfm`, with inline Follow/Want), a
|
||||||
studio-only / show-all release-type filter.
|
cover-art **Library** grid (`/library`), and **The Floor** (home): a live queue of
|
||||||
3. **Discovery** — ✅ **done**. Recommendation-driven finding of new music via
|
in-flight acquisitions with three-phase progress and a Retry button.
|
||||||
**ListenBrainz** similar-artists and **Last.fm** `artist.getSimilar` (both
|
|
||||||
MBID-native), behind a pluggable `SimilaritySource` interface — Last.fm registers
|
|
||||||
as a second source automatically once its API key is set, and cross-source scores
|
|
||||||
are summed per candidate. A background sweep (and a "Discover now" button)
|
|
||||||
aggregates artists similar to the ones you follow into a manual review feed of
|
|
||||||
suggested **artists** and **albums** at `/discover`; **Follow** and **Want**
|
|
||||||
reuse the slice-2 follow / wanted flows, **Dismiss** is permanent. Plus a
|
|
||||||
web-side seed-search box. Every suggestion and seed-search result opens a
|
|
||||||
read-only **artist preview** (`/discover/artist/[mbid]`) — discography,
|
|
||||||
per-album tracklists, and "in library" / "monitored" badges — where you can
|
|
||||||
Follow or Want before committing. No Spotify (its recommendation APIs are dead
|
|
||||||
for new apps).
|
|
||||||
|
|
||||||
Alongside the three slices:
|
Full detail on each of these is in [docs/DEPLOYMENT.md → Features in depth](docs/DEPLOYMENT.md#features-in-depth).
|
||||||
|
|
||||||
- **Last.fm browse** (`/lastfm`) — your Last.fm top artists and albums (by period),
|
|
||||||
with a "hide items already in my library" toggle; each artist opens a modal of your
|
|
||||||
most-played songs and albums, and every album is one click to **Want**.
|
|
||||||
- **Library** (`/library`) — a cover-art grid of everything on disk, filterable, each
|
|
||||||
album opening a tracklist modal.
|
|
||||||
- **The Floor** (home) — a live queue of in-flight acquisitions with three-phase
|
|
||||||
progress (**Searching → Downloading → Finishing**), a live download percentage, a
|
|
||||||
per-job "pressing ticket" (source · format · tracks · elapsed), and a **Retry**
|
|
||||||
button on anything that needs attention.
|
|
||||||
|
|
||||||
The whole UI runs a hand-rolled "Pressing Plant" design system (warm-paper theme, self-hosted
|
|
||||||
Fraunces serif for prose vs. system mono for machine data, light/dark aware).
|
|
||||||
|
|
||||||
**Operational notes:** the monitor and the discovery sweep are both **off by
|
|
||||||
default** — set `monitor.enabled=true` (or Settings → Monitor, + optional `monitor.*`
|
|
||||||
tuning) to activate auto-grab/upgrade, and `discover.enabled=true` (or
|
|
||||||
Settings → Discovery, + optional `discover.*` tuning) to activate the scheduled
|
|
||||||
discovery sweep. The `/discover` "Discover now" button and seed-search work with the
|
|
||||||
sweep off. Credentials (Qobuz, Soulseek, Last.fm) are entered in Settings and persist
|
|
||||||
across `docker compose up -d --build` rebuilds (don't use `down -v`; secrets are
|
|
||||||
encrypted at rest with `LYRA_SECRET_KEY`). Tests run against a separate `lyra_test`
|
|
||||||
database (a guard blocks the live DB). Deferred hardening + the full roadmap live in
|
|
||||||
the implementation-plan "Notes" sections under
|
|
||||||
[`docs/superpowers/plans/`](docs/superpowers/plans/).
|
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
The `docker compose` stack is three services on a home server:
|
The `docker compose` stack is a few services on one host:
|
||||||
|
|
||||||
- **db** — Postgres, the shared source of truth.
|
- **db** — Postgres, the shared source of truth (all durable state lives here).
|
||||||
- **web** — the Next.js app: UI + API (search, request, queue, progress, settings).
|
- **web** — the Next.js app: UI + API (search, request, queue, progress, settings).
|
||||||
- **worker** — the Python worker: all source integration (streamrip, yt-dlp, slskd
|
- **worker** — the Python worker: all source integration (streamrip, yt-dlp, slskd client,
|
||||||
client, mutagen; MusicBrainz + ListenBrainz + Last.fm for metadata and discovery),
|
mutagen; MusicBrainz + ListenBrainz + Last.fm for metadata and discovery), running the
|
||||||
running the six-stage acquisition pipeline plus the background monitor and discovery
|
acquisition pipeline plus the background monitor and discovery sweeps.
|
||||||
sweeps.
|
- **db-backup** — a sidecar that `pg_dump`s the database on a schedule (see [Backup &
|
||||||
|
restore](#backup--restore)).
|
||||||
|
|
||||||
Soulseek support talks to an **external slskd** daemon (an off-the-shelf headless
|
Soulseek support talks to an **external slskd** daemon (an off-the-shelf headless Soulseek
|
||||||
Soulseek client) — the worker reaches it via the `slskd.url` + `slskd.api_key`
|
client) — the worker reaches it via the `slskd.url` + `slskd.api_key` credentials set in
|
||||||
credentials set in Settings, so run slskd separately rather than as part of this stack.
|
Settings, so run slskd separately rather than as part of this stack.
|
||||||
|
|
||||||
A shared `/music` volume is the library, with a separate `/staging` volume for
|
A shared `/music` volume is the library, with a separate `/staging` volume for in-progress
|
||||||
in-progress downloads (point `STAGING_DIR` at a fast local disk when `MUSIC_DIR` is a
|
downloads (point `STAGING_DIR` at a fast local disk when `MUSIC_DIR` is a network share).
|
||||||
network share).
|
|
||||||
|
|
||||||
See [`docs/superpowers/specs/`](docs/superpowers/specs/) for the full design.
|
## Prerequisites
|
||||||
|
|
||||||
## Running it
|
- **Docker** + **Docker Compose**.
|
||||||
|
- A directory for the music library (set `MUSIC_DIR`).
|
||||||
|
- Optional, entered in **Settings** after first run: a **Qobuz** account (lossless/hi-res),
|
||||||
|
a running **slskd** daemon (Soulseek), and a **Last.fm** API key (browse + discovery).
|
||||||
|
None are required to start — Lyra runs with whatever you configure.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
cp .env.example .env # then edit: set LYRA_SECRET_KEY (openssl rand -base64 32) and MUSIC_DIR
|
cp .env.example .env # then edit: set LYRA_SECRET_KEY (openssl rand -base64 32) and MUSIC_DIR
|
||||||
@@ -97,10 +68,26 @@ the placeholder. Once set, keep it stable: changing it makes already-stored cred
|
|||||||
undecryptable (you'd re-enter them in Settings). Optionally set `LYRA_PASSWORD` too (see
|
undecryptable (you'd re-enter them in Settings). Optionally set `LYRA_PASSWORD` too (see
|
||||||
[Security](#security)).
|
[Security](#security)).
|
||||||
|
|
||||||
The web entrypoint runs `prisma migrate deploy` on every rebuild, so schema changes
|
The web entrypoint runs `prisma migrate deploy` on every rebuild, so schema changes reach
|
||||||
reach the live DB automatically. Enter Qobuz / Soulseek / Last.fm credentials in
|
the live DB automatically. Enter Qobuz / Soulseek / Last.fm credentials in **Settings** (they
|
||||||
**Settings** (they persist across rebuilds), then follow an artist or request an album
|
persist across rebuilds), then follow an artist or request an album to feed The Floor.
|
||||||
to feed The Floor.
|
|
||||||
|
`.env.example` documents every setting (`MUSIC_DIR`, `STAGING_DIR`, `SLSKD_DOWNLOADS_DIR`,
|
||||||
|
backup knobs). Note in particular: point `SLSKD_DOWNLOADS_DIR` at slskd's own downloads
|
||||||
|
directory (Lyra must run on the same host as slskd for Soulseek delivery to work), and point
|
||||||
|
`STAGING_DIR` at a fast local disk if `MUSIC_DIR` is a network share.
|
||||||
|
|
||||||
|
## Operational notes
|
||||||
|
|
||||||
|
- The **monitor** and the **discovery sweep** are both **off by default** — set
|
||||||
|
`monitor.enabled=true` (or Settings → Monitor, + optional `monitor.*` tuning) to activate
|
||||||
|
auto-grab/upgrade, and `discover.enabled=true` (or Settings → Discovery, + optional
|
||||||
|
`discover.*` tuning) to activate the scheduled discovery sweep. The `/discover` "Discover
|
||||||
|
now" button and seed-search work with the sweep off.
|
||||||
|
- **Credentials persist** across `docker compose up -d --build` rebuilds — don't use
|
||||||
|
`down -v` (it wipes the Postgres volume); secrets are encrypted at rest with
|
||||||
|
`LYRA_SECRET_KEY`.
|
||||||
|
- **Tests** run against a separate `lyra_test` database (a guard blocks the live DB).
|
||||||
|
|
||||||
## Managing the library
|
## Managing the library
|
||||||
|
|
||||||
@@ -110,6 +97,24 @@ it — re-Want it from the artist page if you change your mind). This is why the
|
|||||||
also bind-mounts `MUSIC_DIR` at `/music` (read-write) — deletion is guarded to only ever
|
also bind-mounts `MUSIC_DIR` at `/music` (read-write) — deletion is guarded to only ever
|
||||||
remove folders strictly inside the library root.
|
remove folders strictly inside the library root.
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
Lyra has **no per-user accounts**. Two layers protect it:
|
||||||
|
|
||||||
|
- **Shared-password gate.** Set `LYRA_PASSWORD` in `.env` to require a single shared
|
||||||
|
password for the entire UI and API. A request without a valid session cookie is
|
||||||
|
redirected to `/login` (or gets `401` for `/api/*`). The cookie is httpOnly and carries
|
||||||
|
an HMAC of the password keyed by `LYRA_SECRET_KEY`, so it can't be forged. **If
|
||||||
|
`LYRA_PASSWORD` is unset, the app is completely open** — anyone who can reach the host
|
||||||
|
can browse the library, trigger downloads, and read/overwrite the stored credentials in
|
||||||
|
Settings. Set it for any deployment that isn't on a fully trusted, isolated network.
|
||||||
|
|
||||||
|
- **Network.** The shared password is deliberately minimal (single password, bearer
|
||||||
|
cookie, plain HTTP on the LAN). For anything internet-facing, also put Lyra behind a
|
||||||
|
reverse proxy with TLS and/or a VPN or auth gateway — don't rely on the shared password
|
||||||
|
alone. See [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#running-behind-a-vpn-gluetun) for a
|
||||||
|
Gluetun VPN setup.
|
||||||
|
|
||||||
## Backup & restore
|
## Backup & restore
|
||||||
|
|
||||||
**All** durable state — library metadata, monitored/wanted lists, discovery data,
|
**All** durable state — library metadata, monitored/wanted lists, discovery data,
|
||||||
@@ -140,105 +145,8 @@ docker compose start web worker
|
|||||||
is safe. To restore onto a brand-new host, bring up just `db` first
|
is safe. To restore onto a brand-new host, bring up just `db` first
|
||||||
(`docker compose up -d db`), restore, then `up -d --build` the rest.
|
(`docker compose up -d db`), restore, then `up -d --build` the rest.
|
||||||
|
|
||||||
## Security
|
## More
|
||||||
|
|
||||||
Lyra has **no per-user accounts**. Two layers protect it:
|
- **[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md)** — full feature tour, publishing pre-built
|
||||||
|
images to a registry, and running the whole stack behind a Gluetun VPN.
|
||||||
- **Shared-password gate.** Set `LYRA_PASSWORD` in `.env` to require a single shared
|
- **[docs/superpowers/specs/](docs/superpowers/specs/)** — the full design specs.
|
||||||
password for the entire UI and API. A request without a valid session cookie is
|
|
||||||
redirected to `/login` (or gets `401` for `/api/*`). The cookie is httpOnly and carries
|
|
||||||
an HMAC of the password keyed by `LYRA_SECRET_KEY`, so it can't be forged. **If
|
|
||||||
`LYRA_PASSWORD` is unset, the app is completely open** — anyone who can reach the host
|
|
||||||
can browse the library, trigger downloads, and read/overwrite the stored credentials in
|
|
||||||
Settings. Set it for any deployment that isn't on a fully trusted, isolated network.
|
|
||||||
|
|
||||||
- **Network.** The shared password is deliberately minimal (single password, bearer
|
|
||||||
cookie, plain HTTP on the LAN). For anything internet-facing, also put Lyra behind a
|
|
||||||
reverse proxy with TLS and/or a VPN or auth gateway — don't rely on the shared password
|
|
||||||
alone.
|
|
||||||
|
|
||||||
## Running behind a VPN (Gluetun)
|
|
||||||
|
|
||||||
Lyra runs fine with its traffic routed through a [Gluetun](https://github.com/qdm12/gluetun)
|
|
||||||
VPN container — useful for hiding the download traffic (Soulseek / Qobuz / YouTube). Nothing
|
|
||||||
in Lyra is VPN-incompatible; the only thing that changes is container networking, because a
|
|
||||||
service using `network_mode: "service:gluetun"` gives up its own network stack and shares
|
|
||||||
Gluetun's. Two consequences:
|
|
||||||
|
|
||||||
- Containers that share Gluetun's namespace reach each other over **`localhost`**, not by
|
|
||||||
Docker service name — so `DATABASE_URL` changes from `@db:5432` to **`@localhost:5432`**.
|
|
||||||
- **Published ports move to the `gluetun` container** (the individual services can no longer
|
|
||||||
publish their own).
|
|
||||||
|
|
||||||
The simplest, most consistent setup is to put the **whole stack** behind Gluetun. Sketch
|
|
||||||
(merge into your existing Gluetun compose; adjust the VPN provider block to yours):
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
services:
|
|
||||||
gluetun:
|
|
||||||
image: qmcgaw/gluetun
|
|
||||||
cap_add: [NET_ADMIN]
|
|
||||||
environment:
|
|
||||||
# ... your VPN provider / credentials ...
|
|
||||||
FIREWALL_INPUT_PORTS: "3000" # let your browser reach the web UI
|
|
||||||
FIREWALL_OUTBOUND_SUBNETS: "192.168.0.0/16" # your LAN — so the worker can reach slskd
|
|
||||||
ports:
|
|
||||||
- "8770:3000" # Lyra web (published on gluetun, not on `web`)
|
|
||||||
# - "5432:5432" # only if you want DB access from the LAN
|
|
||||||
|
|
||||||
web:
|
|
||||||
build: ./web
|
|
||||||
network_mode: "service:gluetun" # no `ports:` / `networks:` here
|
|
||||||
environment:
|
|
||||||
DATABASE_URL: postgresql://lyra:lyra@localhost:5432/lyra # `db` -> `localhost`
|
|
||||||
LYRA_SECRET_KEY: ${LYRA_SECRET_KEY}
|
|
||||||
LYRA_PASSWORD: ${LYRA_PASSWORD:-}
|
|
||||||
HOSTNAME: 0.0.0.0
|
|
||||||
LYRA_LIBRARY_ROOT: /music
|
|
||||||
volumes:
|
|
||||||
- ${MUSIC_DIR:-./music}:/music
|
|
||||||
depends_on:
|
|
||||||
gluetun: { condition: service_healthy }
|
|
||||||
db: { condition: service_healthy }
|
|
||||||
|
|
||||||
worker:
|
|
||||||
build: ./worker
|
|
||||||
network_mode: "service:gluetun"
|
|
||||||
environment:
|
|
||||||
DATABASE_URL: postgresql://lyra:lyra@localhost:5432/lyra
|
|
||||||
LYRA_SECRET_KEY: ${LYRA_SECRET_KEY}
|
|
||||||
STAGING_ROOT: /staging
|
|
||||||
SLSKD_DOWNLOADS_ROOT: /slskd-downloads
|
|
||||||
volumes:
|
|
||||||
- ${MUSIC_DIR:-./music}:/music
|
|
||||||
- ${STAGING_DIR:-${MUSIC_DIR:-./music}/.staging}:/staging
|
|
||||||
- ${SLSKD_DOWNLOADS_DIR:-./slskd-downloads}:/slskd-downloads
|
|
||||||
depends_on:
|
|
||||||
gluetun: { condition: service_healthy }
|
|
||||||
db: { condition: service_healthy }
|
|
||||||
|
|
||||||
db:
|
|
||||||
image: postgres:17
|
|
||||||
network_mode: "service:gluetun" # shares the namespace so `localhost:5432` resolves
|
|
||||||
environment:
|
|
||||||
POSTGRES_USER: ${POSTGRES_USER}
|
|
||||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
|
||||||
POSTGRES_DB: ${POSTGRES_DB}
|
|
||||||
volumes:
|
|
||||||
- postgres-data:/var/lib/postgresql/data
|
|
||||||
```
|
|
||||||
|
|
||||||
(Give `db-backup` the same `network_mode` and `PGHOST: localhost`.) The `HOSTNAME=0.0.0.0`
|
|
||||||
and the built-in healthchecks keep working, since everything is in one namespace.
|
|
||||||
|
|
||||||
Caveats worth knowing:
|
|
||||||
|
|
||||||
- **Qobuz is geo-sensitive** — its catalog/availability varies by country, so pick a VPN
|
|
||||||
exit in a region where your Qobuz account works, or some downloads will fail to match.
|
|
||||||
- The worker's `net.ipv6.conf.all.disable_ipv6` sysctl **can't apply behind Gluetun** (a
|
|
||||||
shared netns can't set sysctls) — harmless, because Gluetun blocks IPv6 anyway, which is
|
|
||||||
exactly what that sysctl worked around (Qobuz's CDN resolving to an unroutable IPv6).
|
|
||||||
- MusicBrainz rate-limits per IP; a shared VPN exit is marginally more likely to hit the
|
|
||||||
1 req/s ceiling, but the built-in response cache makes that a non-issue in normal use.
|
|
||||||
- Bind mounts (`/music`, `/staging`, `/slskd-downloads`), the Postgres volume, and backups
|
|
||||||
are filesystem, not network — **completely unaffected** by the VPN.
|
|
||||||
|
|||||||
@@ -0,0 +1,177 @@
|
|||||||
|
# Lyra — deployment & advanced guide
|
||||||
|
|
||||||
|
Topics beyond the [README](../README.md) quick-start: the full feature tour, publishing
|
||||||
|
pre-built images to a registry, and running the whole stack behind a VPN. For a basic
|
||||||
|
self-hosted setup you only need the README — start there.
|
||||||
|
|
||||||
|
- [Features in depth](#features-in-depth)
|
||||||
|
- [Publishing pre-built images](#publishing-pre-built-images)
|
||||||
|
- [Running behind a VPN (Gluetun)](#running-behind-a-vpn-gluetun)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Features in depth
|
||||||
|
|
||||||
|
Lyra was built in three slices, all complete:
|
||||||
|
|
||||||
|
1. **Acquisition engine.** Resolve a request against MusicBrainz, match/rank across
|
||||||
|
Qobuz/Soulseek/YouTube, download the best source into an isolated staging dir, verify
|
||||||
|
completeness, tag (mutagen), and atomically promote the clean album into
|
||||||
|
`Artist/Album (Year)/## Title.ext`.
|
||||||
|
2. **Library manager.** Watched artists (live MB search + follow), per-release monitor
|
||||||
|
toggles + auto-monitor-future, a wanted list with retry + quality-upgrade window, and a
|
||||||
|
worker "monitor" that discovers new releases and auto-grabs through the acquisition
|
||||||
|
pipeline. Plus a **library scan** that ingests existing on-disk albums as
|
||||||
|
have + monitored + followed, and a per-artist studio-only / show-all release-type filter.
|
||||||
|
3. **Discovery.** Recommendation-driven finding of new music via **ListenBrainz**
|
||||||
|
similar-artists and **Last.fm** `artist.getSimilar` (both MBID-native), behind a
|
||||||
|
pluggable `SimilaritySource` interface — Last.fm registers as a second source
|
||||||
|
automatically once its API key is set, and cross-source scores are summed per candidate.
|
||||||
|
A background sweep (and a "Discover now" button) aggregates artists similar to the ones
|
||||||
|
you follow into a manual review feed of suggested **artists** and **albums** at
|
||||||
|
`/discover`; **Follow** and **Want** reuse the slice-2 follow / wanted flows, **Dismiss**
|
||||||
|
is permanent. Plus a web-side seed-search box. Every suggestion and seed-search result
|
||||||
|
opens a read-only **artist preview** (`/discover/artist/[mbid]`) — discography, per-album
|
||||||
|
tracklists, and "in library" / "monitored" badges — where you can Follow or Want before
|
||||||
|
committing. No Spotify (its recommendation APIs are dead for new apps).
|
||||||
|
|
||||||
|
Alongside the three slices:
|
||||||
|
|
||||||
|
- **Last.fm browse** (`/lastfm`) — your Last.fm top artists and albums (by period), with a
|
||||||
|
"hide items already in my library" toggle and inline **Follow** / **Want** buttons on each
|
||||||
|
row; each artist also opens a modal of your most-played songs and albums.
|
||||||
|
- **Library** (`/library`) — a cover-art grid of everything on disk, filterable, each album
|
||||||
|
opening a tracklist modal.
|
||||||
|
- **The Floor** (home) — a live queue of in-flight acquisitions with three-phase progress
|
||||||
|
(**Searching → Downloading → Finishing**), a live download percentage, a per-job "pressing
|
||||||
|
ticket" (source · format · tracks · elapsed), and a **Retry** button on anything that needs
|
||||||
|
attention.
|
||||||
|
|
||||||
|
The whole UI runs a hand-rolled "Pressing Plant" design system (warm-paper theme, self-hosted
|
||||||
|
Fraunces serif for prose vs. system mono for machine data, light/dark aware).
|
||||||
|
|
||||||
|
Deferred hardening and the full roadmap live in the implementation-plan "Notes" sections under
|
||||||
|
[`docs/superpowers/plans/`](superpowers/plans/); the full design specs are under
|
||||||
|
[`docs/superpowers/specs/`](superpowers/specs/).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Publishing pre-built images
|
||||||
|
|
||||||
|
> **Maintainer release process** — this is how the project maintainer publishes images to a
|
||||||
|
> private registry. If you're self-hosting your own copy, ignore this and just build from
|
||||||
|
> source with `docker compose up -d --build` (see the README). You'd only need this if you
|
||||||
|
> run your own registry and want servers to **pull** rather than build.
|
||||||
|
|
||||||
|
The `web` and `worker` images can be published to a container registry and consumed by a
|
||||||
|
compose file that pulls instead of builds. The maintainer publishes to a Gitea container
|
||||||
|
registry consumed by a compose file in the
|
||||||
|
[`vm-download`](https://git.jger.nl/Jonathan/vm-download) ops repo (`docker/lyra/`).
|
||||||
|
|
||||||
|
Publish a new release from this repo (amd64):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker login <your-registry> # once — registry user + a token with package:write
|
||||||
|
./scripts/build-push.sh # builds + pushes lyra-web / lyra-worker (:latest and :<sha>)
|
||||||
|
```
|
||||||
|
|
||||||
|
`scripts/build-push.sh` defaults the registry namespace to `git.jger.nl/jonathan`; override it
|
||||||
|
with `LYRA_REGISTRY=<your-registry>/<namespace>`.
|
||||||
|
|
||||||
|
Then on the server:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd /opt/git/vm-download && git pull
|
||||||
|
cd docker/lyra && docker compose pull && docker compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
`db` (Postgres) and `db-backup` use the stock `postgres:17` image, so only the two custom
|
||||||
|
images are built here.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Running behind a VPN (Gluetun)
|
||||||
|
|
||||||
|
Lyra runs fine with its traffic routed through a [Gluetun](https://github.com/qdm12/gluetun)
|
||||||
|
VPN container — useful for hiding the download traffic (Soulseek / Qobuz / YouTube). Nothing
|
||||||
|
in Lyra is VPN-incompatible; the only thing that changes is container networking, because a
|
||||||
|
service using `network_mode: "service:gluetun"` gives up its own network stack and shares
|
||||||
|
Gluetun's. Two consequences:
|
||||||
|
|
||||||
|
- Containers that share Gluetun's namespace reach each other over **`localhost`**, not by
|
||||||
|
Docker service name — so `DATABASE_URL` changes from `@db:5432` to **`@localhost:5432`**.
|
||||||
|
- **Published ports move to the `gluetun` container** (the individual services can no longer
|
||||||
|
publish their own).
|
||||||
|
|
||||||
|
The simplest, most consistent setup is to put the **whole stack** behind Gluetun. Sketch
|
||||||
|
(merge into your existing Gluetun compose; adjust the VPN provider block to yours):
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
services:
|
||||||
|
gluetun:
|
||||||
|
image: qmcgaw/gluetun
|
||||||
|
cap_add: [NET_ADMIN]
|
||||||
|
environment:
|
||||||
|
# ... your VPN provider / credentials ...
|
||||||
|
FIREWALL_INPUT_PORTS: "3000" # let your browser reach the web UI
|
||||||
|
FIREWALL_OUTBOUND_SUBNETS: "192.168.0.0/16" # your LAN — so the worker can reach slskd
|
||||||
|
ports:
|
||||||
|
- "8770:3000" # Lyra web (published on gluetun, not on `web`)
|
||||||
|
# - "5432:5432" # only if you want DB access from the LAN
|
||||||
|
|
||||||
|
web:
|
||||||
|
build: ./web
|
||||||
|
network_mode: "service:gluetun" # no `ports:` / `networks:` here
|
||||||
|
environment:
|
||||||
|
DATABASE_URL: postgresql://lyra:lyra@localhost:5432/lyra # `db` -> `localhost`
|
||||||
|
LYRA_SECRET_KEY: ${LYRA_SECRET_KEY}
|
||||||
|
LYRA_PASSWORD: ${LYRA_PASSWORD:-}
|
||||||
|
HOSTNAME: 0.0.0.0
|
||||||
|
LYRA_LIBRARY_ROOT: /music
|
||||||
|
volumes:
|
||||||
|
- ${MUSIC_DIR:-./music}:/music
|
||||||
|
depends_on:
|
||||||
|
gluetun: { condition: service_healthy }
|
||||||
|
db: { condition: service_healthy }
|
||||||
|
|
||||||
|
worker:
|
||||||
|
build: ./worker
|
||||||
|
network_mode: "service:gluetun"
|
||||||
|
environment:
|
||||||
|
DATABASE_URL: postgresql://lyra:lyra@localhost:5432/lyra
|
||||||
|
LYRA_SECRET_KEY: ${LYRA_SECRET_KEY}
|
||||||
|
STAGING_ROOT: /staging
|
||||||
|
SLSKD_DOWNLOADS_ROOT: /slskd-downloads
|
||||||
|
volumes:
|
||||||
|
- ${MUSIC_DIR:-./music}:/music
|
||||||
|
- ${STAGING_DIR:-${MUSIC_DIR:-./music}/.staging}:/staging
|
||||||
|
- ${SLSKD_DOWNLOADS_DIR:-./slskd-downloads}:/slskd-downloads
|
||||||
|
depends_on:
|
||||||
|
gluetun: { condition: service_healthy }
|
||||||
|
db: { condition: service_healthy }
|
||||||
|
|
||||||
|
db:
|
||||||
|
image: postgres:17
|
||||||
|
network_mode: "service:gluetun" # shares the namespace so `localhost:5432` resolves
|
||||||
|
environment:
|
||||||
|
POSTGRES_USER: ${POSTGRES_USER}
|
||||||
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||||||
|
POSTGRES_DB: ${POSTGRES_DB}
|
||||||
|
volumes:
|
||||||
|
- postgres-data:/var/lib/postgresql/data
|
||||||
|
```
|
||||||
|
|
||||||
|
(Give `db-backup` the same `network_mode` and `PGHOST: localhost`.) The `HOSTNAME=0.0.0.0`
|
||||||
|
and the built-in healthchecks keep working, since everything is in one namespace.
|
||||||
|
|
||||||
|
Caveats worth knowing:
|
||||||
|
|
||||||
|
- **Qobuz is geo-sensitive** — its catalog/availability varies by country, so pick a VPN
|
||||||
|
exit in a region where your Qobuz account works, or some downloads will fail to match.
|
||||||
|
- The worker's `net.ipv6.conf.all.disable_ipv6` sysctl **can't apply behind Gluetun** (a
|
||||||
|
shared netns can't set sysctls) — harmless, because Gluetun blocks IPv6 anyway, which is
|
||||||
|
exactly what that sysctl worked around (Qobuz's CDN resolving to an unroutable IPv6).
|
||||||
|
- MusicBrainz rate-limits per IP; a shared VPN exit is marginally more likely to hit the
|
||||||
|
1 req/s ceiling, but the built-in response cache makes that a non-issue in normal use.
|
||||||
|
- Bind mounts (`/music`, `/staging`, `/slskd-downloads`), the Postgres volume, and backups
|
||||||
|
are filesystem, not network — **completely unaffected** by the VPN.
|
||||||
@@ -0,0 +1,247 @@
|
|||||||
|
# Discover Unified Rows + Newest/Most-Popular Albums — Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development or superpowers:executing-plans. Steps use checkbox (`- [ ]`) syntax.
|
||||||
|
|
||||||
|
**Goal:** Collapse the Discover Artists|Albums tabs into one artist-centric row that shows each similar artist's newest AND most-popular album inline (album-click → tracklist), with the artist name linking to the full page.
|
||||||
|
|
||||||
|
**Architecture:** Worker derives two albums per surfaced artist (newest = existing; most-popular = Last.fm `artist.getTopAlbums` matched into the browsed MB discography), tagging each `DiscoverySuggestion.albumReason`. The `/api/discover` route groups pending suggestions by artist into rows carrying `albums[]` + `seeds[]`. The client renders one row per artist with album thumbnails.
|
||||||
|
|
||||||
|
**Tech Stack:** Python 3 + psycopg (worker), Next.js 15 App Router + Prisma + vitest (web), Playwright (verify).
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- Popularity source is Last.fm `artist.getTopAlbums`; **graceful fallback to newest-only** when no `lastfm.api_key` or no match. No new required config.
|
||||||
|
- `albumReason` values are exactly `"newest"`, `"popular"`, `"newest,popular"`; badge labels `Newest` / `Most played` / both; `null` → no badge.
|
||||||
|
- Never break the sweep: any Last.fm/popularity error → treat as no popular pick.
|
||||||
|
- Deploy is `docker compose up -d --build web worker` (never `down -v`). Tests: worker `lyra_test`, web `lyra_test`.
|
||||||
|
- Reverts the D2 tab bar (intentional).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: `albumReason` column + migration
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `web/prisma/schema.prisma` (DiscoverySuggestion model)
|
||||||
|
- Create: `web/prisma/migrations/<ts>_add_discovery_album_reason/migration.sql` (via prisma)
|
||||||
|
|
||||||
|
- [ ] **Step 1:** Add `albumReason String?` to `model DiscoverySuggestion` with a comment (nullable, backward compatible; null → no badge).
|
||||||
|
- [ ] **Step 2:** `cd web && DATABASE_URL=postgresql://lyra:lyra@localhost:5432/lyra_test npx prisma migrate dev --name add_discovery_album_reason`. Expected: "Applying migration … add_discovery_album_reason".
|
||||||
|
- [ ] **Step 3:** Commit (`feat(discover): add DiscoverySuggestion.albumReason`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: worker Last.fm album-popularity helper
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `worker/lyra_worker/similarity/_lastfm_albums.py`
|
||||||
|
- Modify: `worker/lyra_worker/registry.py` (add `build_album_popularity`)
|
||||||
|
- Test: `worker/tests/test_lastfm_albums.py`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Produces: `LastfmAlbumPopularity(api_key, base_url=_BASE, timeout=15.0)` with
|
||||||
|
`top_albums(conn, artist_name) -> list[[name:str, mbid:str|None, playcount:int]]`
|
||||||
|
(playcount desc; cached in ApiCache key `lfm-artist-top-albums:{norm}`, 12h). Errors → `[]`.
|
||||||
|
- Produces: `build_album_popularity(config, dsn) -> LastfmAlbumPopularity | None`
|
||||||
|
(None when `lastfm.api_key` unset).
|
||||||
|
|
||||||
|
- [ ] **Step 1: failing test** — parse a canned `artist.gettopalbums` JSON body into
|
||||||
|
`[[name, mbid, playcount], …]` ordered by playcount desc; a `dict` (single album) is
|
||||||
|
wrapped; missing mbid → None; a body with `error` → `[]`. Inject the parse via a helper
|
||||||
|
`_parse_top_albums(data)` so no network is needed. Also assert `build_album_popularity({}, None) is None`.
|
||||||
|
- [ ] **Step 2:** run → FAIL (module missing).
|
||||||
|
- [ ] **Step 3: implement.**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# _lastfm_albums.py
|
||||||
|
import requests
|
||||||
|
from lyra_worker.apicache import cached_api
|
||||||
|
|
||||||
|
_BASE = "https://ws.audioscrobbler.com/2.0/"
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_top_albums(data) -> list[list]:
|
||||||
|
if isinstance(data.get("error"), (int, float)):
|
||||||
|
return []
|
||||||
|
rows = data.get("topalbums", {}).get("album", [])
|
||||||
|
if isinstance(rows, dict):
|
||||||
|
rows = [rows]
|
||||||
|
out = []
|
||||||
|
for r in rows:
|
||||||
|
name = r.get("name") or ""
|
||||||
|
if not name:
|
||||||
|
continue
|
||||||
|
out.append([name, r.get("mbid") or None, int(r.get("playcount") or 0)])
|
||||||
|
out.sort(key=lambda a: a[2], reverse=True)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
class LastfmAlbumPopularity:
|
||||||
|
def __init__(self, api_key, base_url=_BASE, timeout: float = 15.0, limit: int = 25):
|
||||||
|
self._key, self._base, self._timeout, self._limit = api_key, base_url, timeout, limit
|
||||||
|
|
||||||
|
def _fetch(self, artist_name: str) -> list[list]:
|
||||||
|
try:
|
||||||
|
res = requests.get(self._base, params={
|
||||||
|
"method": "artist.gettopalbums", "artist": artist_name,
|
||||||
|
"api_key": self._key, "format": "json", "limit": str(self._limit),
|
||||||
|
"autocorrect": "1",
|
||||||
|
}, timeout=self._timeout)
|
||||||
|
return _parse_top_albums(res.json())
|
||||||
|
except Exception:
|
||||||
|
return []
|
||||||
|
|
||||||
|
def top_albums(self, conn, artist_name: str) -> list[list]:
|
||||||
|
norm = (artist_name or "").strip().lower()
|
||||||
|
if not norm:
|
||||||
|
return []
|
||||||
|
return cached_api(conn, f"lfm-artist-top-albums:{norm}", 43200,
|
||||||
|
lambda: self._fetch(artist_name))
|
||||||
|
```
|
||||||
|
|
||||||
|
```python
|
||||||
|
# registry.py — add
|
||||||
|
from lyra_worker.similarity._lastfm_albums import LastfmAlbumPopularity
|
||||||
|
|
||||||
|
def build_album_popularity(config: dict, dsn: str | None = None):
|
||||||
|
key = config.get("lastfm.api_key")
|
||||||
|
return LastfmAlbumPopularity(api_key=key) if key else None
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4:** run → PASS.
|
||||||
|
- [ ] **Step 5:** commit (`feat(discover): Last.fm album-popularity helper`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: `_derive_albums` picks newest + most-popular
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `worker/lyra_worker/discovery.py` (`_derive_albums`, `_upsert_album`, `run_discovery`)
|
||||||
|
- Test: `worker/tests/test_discovery_albums.py`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `popularity.top_albums(conn, name)` from Task 2 (or None).
|
||||||
|
- Produces: `_derive_albums(conn, browser, surfaced, cfg, popularity=None)`; `run_discovery(conn, sources, browser, cfg, popularity=None)`; `_upsert_album(conn, artist, rg, reason)`.
|
||||||
|
|
||||||
|
- [ ] **Step 1: failing tests** (inject a fake popularity — no live Last.fm):
|
||||||
|
- newest + most-popular → two album rows, `albumReason` `"newest"` / `"popular"`.
|
||||||
|
- popular pick == newest rg → one row, `albumReason == "newest,popular"`.
|
||||||
|
- `popularity=None` → newest-only, `albumReason == "newest"`.
|
||||||
|
- popular match resolves by title when Last.fm mbid absent; skips owned/non-qualifying and falls to next-highest playcount; no match → newest-only.
|
||||||
|
|
||||||
|
```python
|
||||||
|
class FakePopularity:
|
||||||
|
def __init__(self, data): self._d = data # {artist_name: [[name, mbid, plays], …]}
|
||||||
|
def top_albums(self, conn, name): return self._d.get(name, [])
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2:** run → FAIL.
|
||||||
|
- [ ] **Step 3: implement.** In `_derive_albums`, after building `core` (qualifying un-owned, newest-first):
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _norm(s): return (s or "").strip().lower()
|
||||||
|
|
||||||
|
# newest picks (existing)
|
||||||
|
picks = {} # rg_mbid -> reason
|
||||||
|
for rg in core[: cfg.albums_per_artist]:
|
||||||
|
picks[rg.rg_mbid] = "newest"
|
||||||
|
|
||||||
|
# most-popular pick via Last.fm, matched into `core`
|
||||||
|
if popularity is not None and core:
|
||||||
|
by_mbid = {g.rg_mbid: g for g in core}
|
||||||
|
by_title = {_norm(g.title): g for g in core}
|
||||||
|
for name, mbid, _plays in popularity.top_albums(conn, artist["name"]):
|
||||||
|
g = (by_mbid.get(mbid) if mbid else None) or by_title.get(_norm(name))
|
||||||
|
if g is None:
|
||||||
|
continue
|
||||||
|
picks[g.rg_mbid] = "newest,popular" if picks.get(g.rg_mbid) == "newest" else "popular"
|
||||||
|
break
|
||||||
|
|
||||||
|
for rg in core:
|
||||||
|
if rg.rg_mbid in picks:
|
||||||
|
albums += _upsert_album(conn, artist, rg, picks[rg.rg_mbid])
|
||||||
|
have.add(rg.rg_mbid)
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `reason` param to `_upsert_album` → include `"albumReason"` in the INSERT column list +
|
||||||
|
`ON CONFLICT DO UPDATE SET … "albumReason" = EXCLUDED."albumReason"`. Thread `popularity`
|
||||||
|
through `run_discovery` → `_derive_albums`.
|
||||||
|
|
||||||
|
- [ ] **Step 4:** run → PASS; then full worker suite green.
|
||||||
|
- [ ] **Step 5:** commit (`feat(discover): derive newest + most-popular album per artist`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 4: wire popularity into the worker loop
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `worker/lyra_worker/main.py` (`_run_discovery` builds + passes popularity)
|
||||||
|
- Test: existing `test_discovery_trigger.py` still green (popularity defaults None there).
|
||||||
|
|
||||||
|
- [ ] **Step 1:** In `main.py`, import `build_album_popularity`; in `_run_discovery` build it from `get_config(conn)` and pass to `run_discovery(conn, sources, browser, cfg, popularity=pop)`.
|
||||||
|
- [ ] **Step 2:** run worker suite → PASS.
|
||||||
|
- [ ] **Step 3:** commit (`feat(discover): supply Last.fm popularity to the sweep`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 5: `/api/discover` groups by artist
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `web/src/app/api/discover/route.ts`
|
||||||
|
- Test: `web/src/app/api/discover/route.test.ts`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Produces: `GET` returns `{ rows: [{ id, artistMbid, artistName, score, seedCount, sources, seeds, albums: [{ id, rgMbid, album, primaryType, secondaryTypes, firstReleaseDate, albumReason }] }], search? }`. Rows ordered by score desc.
|
||||||
|
|
||||||
|
- [ ] **Step 1: failing test** — seed pending artist + its two album suggestions (`albumReason` newest/popular) sharing `artistMbid`; assert one row with `albums.length === 2`, ordered; seeds present; an album whose artist has no pending artist-suggestion still yields a row.
|
||||||
|
- [ ] **Step 2:** run → FAIL.
|
||||||
|
- [ ] **Step 3: implement.** Fetch all pending suggestions (as today) + the seeds map (as today). Group by `artistMbid`: artist-kind rows seed the row header (id/score/seedCount/sources); album-kind rows push into `albums[]` (include `albumReason`). For an `artistMbid` with only album rows, synthesize a header from the album's `artistName`/`artistMbid` (no `id`, score = max album score). Sort rows by score desc; sort each `albums[]` by `firstReleaseDate` desc. Return `{ rows }`.
|
||||||
|
- [ ] **Step 4:** run → PASS.
|
||||||
|
- [ ] **Step 5:** commit (`feat(discover): group suggestions into artist rows`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 6: dismiss cascade
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `web/src/app/api/discover/[id]/route.ts`
|
||||||
|
- Test: `web/src/app/api/discover/[id]/route.test.ts`
|
||||||
|
|
||||||
|
- [ ] **Step 1: failing test** — dismissing an artist suggestion also marks its pending album suggestions (same `artistMbid`) dismissed; a dismissed album is not returned by `/api/discover`.
|
||||||
|
- [ ] **Step 2:** run → FAIL.
|
||||||
|
- [ ] **Step 3: implement.** In the dismiss branch, after dismissing the artist suggestion, `updateMany({ where: { kind: "album", artistMbid, status: "pending" }, data: { status: "dismissed" } })`. Follow/Want unchanged.
|
||||||
|
- [ ] **Step 4:** run → PASS.
|
||||||
|
- [ ] **Step 5:** commit (`fix(discover): dismissing an artist also dismisses its albums`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 7: unified-row client (drop tabs + ArtistModal)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `web/src/app/discover/discover-client.tsx`
|
||||||
|
- Modify: `web/src/app/design.css` (row + thumb strip + badge styles)
|
||||||
|
|
||||||
|
- [ ] **Step 1:** Replace `Feed`/`tab` state with `rows` from `/api/discover` (`{ rows }`). Remove the tab bar, `ArtistModal` import + `artistModal` state + its JSX + the `isFullAlbum`/albums-tab code.
|
||||||
|
- [ ] **Step 2:** Render one `<li class="list-row disco-row">` per row:
|
||||||
|
- artist block: `<a href={`/discover/artist/${row.artistMbid}`}>{row.artistName}</a>`, `score`, `similarTo(row.seeds)` (keep existing helper).
|
||||||
|
- album strip: `row.albums.map` → a `<button class="disco-thumb" onClick={() => setAlbumModal(album with artistMbid=row.artistMbid)}>`: `<CoverArt>` + title + badge (`badgeFor(albumReason)`) + a `Want` button (`act(album.id, "want")`).
|
||||||
|
- row actions: `Follow` (`act(row.id, "follow")` when `row.id`, else follow-by-mbid via `POST /api/artists`), `Dismiss` (`act(row.id, "dismiss")`, hidden when no `row.id`).
|
||||||
|
- `badgeFor(r)`: `newest`→`Newest`, `popular`→`Most played`, `newest,popular`→`Newest · Most played`, null→none.
|
||||||
|
- [ ] **Step 3:** keep `AlbumModal` (thumb click) + `find-similar` search block (its result artist links become `<a href>` to the full page, no modal). Keep Discover-now/freshness header.
|
||||||
|
- [ ] **Step 4:** CSS: `.disco-row` flex (artist block | thumb strip | actions), `.disco-thumb` (small cover + title + `.badge`), wraps under artist on narrow widths.
|
||||||
|
- [ ] **Step 5:** `npx tsc --noEmit` + `npm test` + `npm run build` → all green.
|
||||||
|
- [ ] **Step 6:** commit (`feat(discover): unified artist rows with inline album thumbnails`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 8: deploy + live verify
|
||||||
|
|
||||||
|
- [ ] **Step 1:** `docker compose up -d --build web worker`; confirm migration applied + all healthy.
|
||||||
|
- [ ] **Step 2:** Playwright on `/discover`: rows render; artist name navigates to `/discover/artist/[mbid]`; album thumbnail opens `AlbumModal` with a tracklist; badges show.
|
||||||
|
- [ ] **Step 3:** Trigger a library **Scan** (populates artistMbid) then **Discover now**; confirm `albumReason` populates (`Most played` badges appear) once a fresh sweep runs. (If a full sweep is impractical to force cheaply, rely on the injected-data Playwright check + unit tests, and note it.)
|
||||||
|
- [ ] **Step 4:** Update `MEMORY.md` / project-state with the shipped feature + commit range.
|
||||||
|
|
||||||
|
## Self-Review
|
||||||
|
|
||||||
|
- **Spec coverage:** Part A → Tasks 2–4; Part B → Task 5; Part C dismiss → Task 6, client/modal → Task 7; schema → Task 1; tests throughout; live verify → Task 8. ✓
|
||||||
|
- **Placeholders:** none — code shown for each implementing step. ✓
|
||||||
|
- **Type consistency:** `top_albums(conn, name) -> list[[name, mbid|None, plays]]` used identically in Tasks 2–3; `_upsert_album(…, reason)` and `_derive_albums(…, popularity=None)` consistent; route emits `rows[]` consumed by Task 7. ✓
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
# Post-deploy UX fixes (3 items) — handoff plan
|
||||||
|
|
||||||
|
> **Context:** Lyra is DEPLOYED + LIVE on the media VM (ssh alias `download` = jonathan@10.10.0.103,
|
||||||
|
> hostname `servarr`, x86_64). Pre-built images from `git.jger.nl/jonathan/lyra-{web,worker}`, whole
|
||||||
|
> stack behind its own gluetun VPN, deploy repo `Jonathan/vm-download` → `docker/lyra/`. Library was
|
||||||
|
> wiped to a blank slate; user rebuilds via Last.fm. Two test presses succeeded end-to-end (John Mayer
|
||||||
|
> Continuum q3 hi-res, Room for Squares q2) — pipeline is solid. These 3 are UX polish the user hit
|
||||||
|
> in real use. See MEMORY.md → project-state.md "VM DEPLOYMENT" for full deploy details.
|
||||||
|
|
||||||
|
**Workflow for EACH fix:** implement in `worker/` → run worker tests (`cd worker && source .venv/bin/activate
|
||||||
|
&& python -m pytest`) → **on the DEV box** `./scripts/build-push.sh` (needs `docker login git.jger.nl`
|
||||||
|
already done this session; a fresh session may need re-login — a **real TTY**, not the `!`-prefix) →
|
||||||
|
on VM `ssh download 'cd /opt/git/vm-download/docker/lyra && docker compose pull && docker compose up -d worker'`
|
||||||
|
→ verify with a test press (queue via SQL, see below) and watch `Job.downloadProgress`/on-disk result.
|
||||||
|
Commit per fix; branch `feat/post-deploy-ux`, ff-merge to main, push.
|
||||||
|
|
||||||
|
**Queue a test press on the VM** (mimics the Wanted "Search now"): pick a `MonitoredRelease` id from
|
||||||
|
`SELECT id,"artistName",album FROM "MonitoredRelease" WHERE monitored AND NOT ignored AND state<>'fulfilled'`,
|
||||||
|
then insert a Request+Job (use a `/tmp/*.sql` file + `docker cp` + `psql -f` to avoid ssh quoting hell):
|
||||||
|
`INSERT INTO "Request"(id,artist,album,"monitoredReleaseId",status,"createdAt") SELECT gen_random_uuid()::text,"artistName",album,id,'pending',now() FROM "MonitoredRelease" WHERE id='<mrid>' RETURNING id` → capture into a Job insert `(id,"requestId",state,"currentStage",attempts,"downloadProgress","createdAt","updatedAt") VALUES (gen_random_uuid()::text,rid,'requested','intake',0,0,now(),now())`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Fix 1 — Smooth Qobuz download progress (byte-level)
|
||||||
|
|
||||||
|
**Problem:** the Floor progress bar sits at 0% for the whole Qobuz download then jumps to 100%.
|
||||||
|
Qobuz/streamrip's per-track byte callback is NOT hooked, so `Job.downloadProgress` comes only from the
|
||||||
|
file-count poller (`pipeline._poll_download_progress`) — and streamrip downloads tracks concurrently, so
|
||||||
|
0 files are "complete" until they all land at once. yt-dlp + slskd already report bytes via `on_progress`
|
||||||
|
(smooth); only Qobuz doesn't.
|
||||||
|
|
||||||
|
**Files:** `worker/lyra_worker/adapters/_streamrip.py` (the Qobuz download), `worker/lyra_worker/pipeline.py`
|
||||||
|
(threads `on_progress` → `_make_on_progress` → `Job.downloadProgress`; poller is the fallback when the
|
||||||
|
per-attempt `reports` Event is set).
|
||||||
|
|
||||||
|
**Approach:** hook streamrip's progress. streamrip drives progress via a callback/console; investigate its
|
||||||
|
API (it uses a `rip.utils`/`Downloadable` byte-progress hook or a `tqdm` callback). Aggregate downloaded
|
||||||
|
bytes / total bytes summed across the album's concurrent tracks → a single fraction → call the pipeline's
|
||||||
|
`on_progress(frac)` (which sets `reports` and writes throttled `Job.downloadProgress`). Keep the file-count
|
||||||
|
poller as the fallback (adapters that don't report). Match the throttle already in `_make_on_progress`
|
||||||
|
(≥1% move or every 0.5s).
|
||||||
|
|
||||||
|
**Test:** hard to unit-test streamrip internals — add a unit test that a fake byte-callback aggregates to a
|
||||||
|
monotonic 0→1 fraction, plus **live-verify** with a test press: `Job.downloadProgress` climbs mid-download
|
||||||
|
instead of 0→100. (Known caveat if streamrip exposes no usable byte hook: fall back to a smoother
|
||||||
|
file-count cadence, or accept the jump — document the finding.)
|
||||||
|
|
||||||
|
## Fix 2 — Parallelize the source searches (Soulseek adds ~2 min)
|
||||||
|
|
||||||
|
**Problem:** every acquisition spends up to ~2 min in the Soulseek search. `SlskdClient.search_album` polls
|
||||||
|
`_SEARCH_POLLS=40 × _POLL_SECONDS=3s = 120s` (early-exits on `isComplete`, but slskd searches genuinely run
|
||||||
|
long). The pipeline searches adapters SEQUENTIALLY (`for adapter in adapters: adapter.search(target)`), so
|
||||||
|
Soulseek's 2 min is added on top of Qobuz's ~instant search.
|
||||||
|
|
||||||
|
**Files:** `worker/lyra_worker/pipeline.py` (the match phase, `for adapter in adapters` loop);
|
||||||
|
optionally `worker/lyra_worker/adapters/_slskd.py` (`_SEARCH_POLLS`).
|
||||||
|
|
||||||
|
**Approach:** run the per-adapter searches **concurrently** (e.g. `ThreadPoolExecutor`, one thread per
|
||||||
|
adapter), collect all candidates when the last returns → total match wall-clock = slowest single search,
|
||||||
|
not the sum. So Qobuz returns instantly and only Soulseek's ~2 min gates. Keep the existing per-adapter
|
||||||
|
try/except (a failing source contributes nothing). **Thread-safety caveat:** streamrip uses a module-level
|
||||||
|
persistent asyncio loop (`_streamrip._run`) — verify concurrent Qobuz search from a worker thread is safe
|
||||||
|
(it runs coroutines on that shared loop; may need a lock or to keep Qobuz on the calling thread while
|
||||||
|
Soulseek/YouTube go to threads). Optionally also drop `_SEARCH_POLLS` to ~20 (60s) with the early-exit.
|
||||||
|
|
||||||
|
**Test:** a pipeline test asserting all adapters' candidates are collected (order-independent) when searched
|
||||||
|
concurrently; existing match/ranking tests stay green. Live-verify a press: match phase finishes in ~the
|
||||||
|
Soulseek time, not Soulseek+Qobuz.
|
||||||
|
|
||||||
|
## Fix 3 — Canonicalize bonus-track filenames on import
|
||||||
|
|
||||||
|
**Problem:** tracks BEYOND the MusicBrainz tracklist keep streamrip's raw name. Real example: Room for
|
||||||
|
Squares imported 12 clean `## Title.flac` + one `13. John Mayer - St. Patrick's Day (Album Version).flac`
|
||||||
|
(MB lists 12; Qobuz had a 13th bonus). The main tracklist gets `## Title` but extras don't.
|
||||||
|
|
||||||
|
**Files:** `worker/lyra_worker/library.py` (`import_album` — how it names/moves audio into
|
||||||
|
`Artist/Album (Year)/## Title.ext`), and/or the tagger `worker/lyra_worker/_mutagen.py`.
|
||||||
|
|
||||||
|
**Approach:** find where files are renamed to `## Title.ext`. For a track with no MB position/title (an
|
||||||
|
extra), derive a clean name from its own embedded tags (track number + title) or by stripping the
|
||||||
|
`^\d+\.?\s*(Artist\s*-\s*)?` prefix from the streamrip filename → `## Title.ext`. Ensure the position keeps
|
||||||
|
counting past the MB list (13, 14, …) and the on-disk `trackNames` capture stays consistent.
|
||||||
|
|
||||||
|
**Test:** unit-test the filename-canonicalizer on inputs like `13. John Mayer - St. Patrick's Day (Album
|
||||||
|
Version).flac` → `13 St. Patrick's Day (Album Version).flac`; live-verify by re-pressing an album with a
|
||||||
|
bonus track (or `docker compose exec worker` a rename dry-run). NOTE: the already-imported Room for Squares
|
||||||
|
track 13 can be fixed by re-pressing (force upgrade) or a one-off rename — mention to the user.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Self-review / notes
|
||||||
|
- All three are worker-only → only the `worker` image rebuilds/redeploys (`docker compose up -d worker`).
|
||||||
|
- Don't disrupt the live stack: the VM is in use; deploy the worker, verify one press, done.
|
||||||
|
- If context/time is tight, do them in priority order: **2 (search speed)** and **3 (filenames)** are
|
||||||
|
self-contained and low-risk; **1 (streamrip progress)** is the deepest (streamrip internals) — timebox it
|
||||||
|
and fall back to documenting if streamrip has no clean byte hook.
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,156 @@
|
|||||||
|
# Discover redesign: unified artist rows + newest/most-popular albums
|
||||||
|
|
||||||
|
**Date:** 2026-07-14
|
||||||
|
**Status:** Approved (design), pending implementation plan
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
`/discover` splits suggested artists and suggested albums into two tabs, but album
|
||||||
|
suggestions are not independent — each one is just "a notable album by a suggested similar
|
||||||
|
artist" (derived in `_derive_albums` from the surfaced artists). The two-tab split fragments a
|
||||||
|
single idea. Two further gaps:
|
||||||
|
|
||||||
|
- Only the **newest** un-owned album is surfaced per similar artist; the album the artist is
|
||||||
|
best known for (their **most popular**) is invisible unless it happens to be newest.
|
||||||
|
- Seeing an album's tracklist requires an extra modal hop, and the artist modal on Discover is
|
||||||
|
a lightweight duplicate of the richer full artist page.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
1. Keep album suggestions artist-mediated (no album-level similarity data exists in our
|
||||||
|
stack), but surface **two** albums per similar artist: **newest** and **most popular**.
|
||||||
|
2. Collapse the Artists|Albums tabs into **one unified list** — one row per suggested artist,
|
||||||
|
carrying that artist's album thumbnails inline.
|
||||||
|
3. On Discover, replace the artist modal with a direct link to the full artist page; keep
|
||||||
|
album-click → the existing `AlbumModal` (cover + tracklist + Want).
|
||||||
|
|
||||||
|
Non-goals: album-to-album similarity (no data source); collection-completion suggestions;
|
||||||
|
changing the Last.fm page's artist modal (it keeps its personal most-played view).
|
||||||
|
|
||||||
|
## Data-source decision: popularity
|
||||||
|
|
||||||
|
- **Chosen:** Last.fm `artist.getTopAlbums` — returns an artist's albums ranked by playcount,
|
||||||
|
each with a name and (usually) an MBID. The user already has Last.fm configured; no new
|
||||||
|
config. Confirmed live: the method exists (error 10 "invalid key" with a bogus key, not
|
||||||
|
error 3 "unknown method").
|
||||||
|
- **Rejected:** ListenBrainz `/1/popularity/release-group` — MBID-native and cleaner in
|
||||||
|
principle, but now requires a ListenBrainz auth token (anti-scraper), which is new config
|
||||||
|
friction. MusicBrainz exposes no popularity signal at all.
|
||||||
|
- **Fallback:** without Last.fm creds, or when no top album matches a qualifying un-owned
|
||||||
|
release group, the artist yields **newest-only** — byte-for-byte today's behavior.
|
||||||
|
|
||||||
|
## Design
|
||||||
|
|
||||||
|
### Part A — Album derivation (worker)
|
||||||
|
|
||||||
|
`_derive_albums(conn, browser, surfaced, cfg, popularity=None)` in
|
||||||
|
`worker/lyra_worker/discovery.py`.
|
||||||
|
|
||||||
|
For each surfaced artist:
|
||||||
|
1. Browse the artist's release groups from MB (already done) → filter to qualifying,
|
||||||
|
un-owned albums (`_album_kind_ok` + not in `_existing_rg_mbids`) — unchanged.
|
||||||
|
2. **Newest:** the newest `cfg.albums_per_artist` (default 1) by `first_release_date` desc —
|
||||||
|
unchanged, tagged `reason="newest"`.
|
||||||
|
3. **Most popular:** if `popularity` is available, fetch the artist's Last.fm top albums, and
|
||||||
|
pick the highest-playcount album that maps to a qualifying un-owned release group in the
|
||||||
|
browsed set — match by MBID first (when Last.fm supplies one that is in the browsed set),
|
||||||
|
else by normalized title (`stripEditionQualifiers`-equivalent, lowercased/trimmed). Tag
|
||||||
|
`reason="popular"`.
|
||||||
|
4. If the popular pick is the same release group as a newest pick, do not insert a second
|
||||||
|
row — the existing row's reason becomes `"newest,popular"`.
|
||||||
|
5. Upsert each as a `DiscoverySuggestion` (kind `album`) as today, additionally writing
|
||||||
|
`albumReason`.
|
||||||
|
|
||||||
|
**Popularity helper (injected, testable — mirrors `#16`'s `measure_duration`).** An object
|
||||||
|
with `top_albums(artist_name) -> list[(name, mbid_or_None, playcount)]`, ordered by playcount
|
||||||
|
desc. Built only when `lastfm.api_key` is set (worker decrypts via `crypto.py`); the fetch is
|
||||||
|
wrapped in `cached_api` (`ApiCache`, ~12h TTL, key `lfm-artist-top-albums:{norm}`). Errors are
|
||||||
|
swallowed → treated as "no popular pick" (newest-only). `run_discovery` constructs it from
|
||||||
|
config and passes it to `_derive_albums`; tests inject a fake or `None`.
|
||||||
|
|
||||||
|
**Cost:** ≤1 Last.fm call per surfaced artist per sweep, cached; sweeps are infrequent and
|
||||||
|
chunked. Only surfaced artists (post `min_score`) are queried.
|
||||||
|
|
||||||
|
### Part B — `/api/discover` groups suggestions by artist
|
||||||
|
|
||||||
|
`web/src/app/api/discover/route.ts` returns **rows** instead of two flat arrays. Build rows
|
||||||
|
from the union of pending artist-suggestions and the distinct artists of pending
|
||||||
|
album-suggestions (so an album is never orphaned), keyed by `artistMbid`:
|
||||||
|
|
||||||
|
```
|
||||||
|
rows: [{
|
||||||
|
id, // the artist suggestion id (for Follow/Dismiss), when present
|
||||||
|
artistMbid, artistName,
|
||||||
|
score, seedCount, sources,
|
||||||
|
seeds: string[], // "why suggested" (existing DiscoverySeedContribution join)
|
||||||
|
albums: [{
|
||||||
|
id, rgMbid, album, primaryType, secondaryTypes, firstReleaseDate,
|
||||||
|
albumReason, // "newest" | "popular" | "newest,popular"
|
||||||
|
}]
|
||||||
|
}]
|
||||||
|
```
|
||||||
|
|
||||||
|
Rows are ordered by `score` desc (artist suggestion score; album-only rows use the album's
|
||||||
|
score). The `find-similar` search flow is unchanged (separate endpoint, no albums).
|
||||||
|
|
||||||
|
### Part C — dismiss cascade
|
||||||
|
|
||||||
|
`POST /api/discover/[id]` (dismiss): when the target is an **artist** suggestion, also mark
|
||||||
|
its pending **album** suggestions (same `artistMbid`) dismissed, so the unified row disappears
|
||||||
|
cleanly rather than leaving stray albums. Follow/Want are unchanged.
|
||||||
|
|
||||||
|
### Part D — `/discover` client (unified rows, no tabs, no modal)
|
||||||
|
|
||||||
|
`web/src/app/discover/discover-client.tsx`:
|
||||||
|
- Remove the Artists|Albums tab bar and the `tab` state; render one `<ul>` of artist rows.
|
||||||
|
- Remove `ArtistModal` import + usage entirely. The artist **name links** to
|
||||||
|
`/discover/artist/{artistMbid}` (an `<a href>`; matches the existing full-page route).
|
||||||
|
- Each row renders its `albums[]` as small `CoverArt` thumbnails, each with:
|
||||||
|
- a **badge** from `albumReason`: `"newest"`→`Newest`, `"popular"`→`Most played`,
|
||||||
|
`"newest,popular"`→both; `null`→no badge;
|
||||||
|
- a **Want** action (`act(album.id, "want")`);
|
||||||
|
- **click → `AlbumModal`** (kept) with the album's `rgMbid`/title/artist → cover + tracklist
|
||||||
|
+ Want.
|
||||||
|
- Row actions stay: **Follow** / **Dismiss** (on `row.id`, the artist suggestion). If a row
|
||||||
|
has no artist suggestion id (album-only, e.g. artist dismissed earlier), Follow resolves by
|
||||||
|
MBID via `POST /api/artists`; Dismiss is hidden for that (degenerate) case.
|
||||||
|
- `find-similar` results keep their existing rendering, minus `ArtistModal` — their artist
|
||||||
|
links also go to the full page (consistency; `ArtistModal` is being removed from this file).
|
||||||
|
- New CSS for the row layout (artist block + horizontal album-thumb strip + actions),
|
||||||
|
responsive: the thumb strip wraps/stacks under the artist on narrow viewports.
|
||||||
|
|
||||||
|
### Schema / migration
|
||||||
|
|
||||||
|
`DiscoverySuggestion.albumReason String?` — nullable, backward compatible. Existing album
|
||||||
|
rows read as `null` (client treats null as no badge) until the next sweep re-derives them.
|
||||||
|
Migration `add_discovery_album_reason`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
**Worker (`test_discovery_albums.py`):**
|
||||||
|
- newest + most-popular produce two rows with correct `albumReason`;
|
||||||
|
- dedupe when newest == popular → one row, `reason="newest,popular"`;
|
||||||
|
- no popularity helper → newest-only (unchanged), `reason="newest"`;
|
||||||
|
- popular pick skips owned / non-qualifying release groups and falls to the next-highest
|
||||||
|
playcount; no match → newest-only;
|
||||||
|
- injected fake popularity (no live Last.fm).
|
||||||
|
|
||||||
|
**Web:**
|
||||||
|
- `/api/discover` groups albums under their artist row; row carries `albums[]` + `seeds[]` +
|
||||||
|
`albumReason`; album-only artist still yields a row.
|
||||||
|
- dismiss cascade: dismissing an artist marks its album suggestions dismissed.
|
||||||
|
- existing route tests updated to the grouped shape.
|
||||||
|
|
||||||
|
**Live verify (Playwright):** unified rows render; artist name navigates to the full page;
|
||||||
|
album thumbnail opens `AlbumModal` with a tracklist; badges show; a fresh "Discover now" sweep
|
||||||
|
populates `albumReason` (needs a re-scan first so owned-artist MBIDs exist — pre-existing
|
||||||
|
caveat from `#18`).
|
||||||
|
|
||||||
|
## Risks / notes
|
||||||
|
|
||||||
|
- Last.fm top-album title↔MB release-group matching can miss (edition wording, localized
|
||||||
|
titles); acceptable — a miss just means newest-only for that artist. Normalized-title
|
||||||
|
matching + MBID-first keeps misses rare.
|
||||||
|
- Adds a Last.fm dependency to the album step of the sweep; fully optional + cached, and the
|
||||||
|
artist step already uses Last.fm as a similarity source.
|
||||||
|
- Reverts the D2 tab bar shipped earlier the same day; intentional per this design.
|
||||||
@@ -0,0 +1,179 @@
|
|||||||
|
# Press controls + inter-job delay — design
|
||||||
|
|
||||||
|
Date: 2026-07-15
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
The monitor can enqueue a large backlog of acquisition jobs at once. The worker drains
|
||||||
|
them serially with **no pacing** (only a 2s idle poll when the queue is empty) and offers
|
||||||
|
**no runtime control** over the queue — the only per-job action on the Floor is Retry (on
|
||||||
|
`needs_attention`). The user wants:
|
||||||
|
|
||||||
|
1. A **configurable delay between downloads** to spread sustained Qobuz load over time.
|
||||||
|
2. **Per-item controls** on the Floor ("the Press"): pause and remove a queued item.
|
||||||
|
3. **Whole-press controls**: pause/resume, clear the queue, and a force-stop of the
|
||||||
|
currently-downloading album.
|
||||||
|
|
||||||
|
## Constraints (from the existing code)
|
||||||
|
|
||||||
|
- The worker is a **single, strictly-serial process** (`worker/lyra_worker/main.py:256-298`).
|
||||||
|
One job runs to completion in a blocking `run_pipeline` call before the loop does anything
|
||||||
|
else.
|
||||||
|
- **Config is read fresh every loop iteration** (`main.py:262`, `get_config`), so any Config
|
||||||
|
change takes effect within ~2s with no restart.
|
||||||
|
- An **in-flight download cannot be cleanly interrupted** — `run_pipeline` and the adapter
|
||||||
|
download are synchronous/blocking with only an `on_progress` callback, no cancel token. The
|
||||||
|
only interruption is process death, after which `reclaim_stuck_jobs` (`claim.py:8-24`)
|
||||||
|
requeues the in-flight job at startup and `clear_staging_root` (`main.py:233`) wipes the
|
||||||
|
partial download.
|
||||||
|
- `claim_next` (`claim.py:27-57`) claims the oldest `state='requested'` job.
|
||||||
|
- Deleting a `Request` cascades to its `Job` and `Candidate` rows (`schema.prisma:51,66`); the
|
||||||
|
linked `MonitoredRelease` is `SetNull`, not cascaded.
|
||||||
|
- `enqueue_due` (`monitor.py:115-147`) re-enqueues a monitored release once it has no live
|
||||||
|
(non-terminal) job and its `lastSearchedAt` is older than `retryIntervalHours` (default 6).
|
||||||
|
So a removed/cleared monitored item reappears later on the normal cadence unless `ignored`.
|
||||||
|
- The web container runs `prisma migrate deploy` on startup (`web/docker-entrypoint.sh`), so a
|
||||||
|
new migration auto-applies on deploy.
|
||||||
|
|
||||||
|
## Decisions (confirmed with the user)
|
||||||
|
|
||||||
|
- **Active item:** add a global **force-kill** ("Stop now") that aborts the in-flight download
|
||||||
|
via worker process-restart. Per-item controls act only on queued items.
|
||||||
|
- **Remove / Clear semantics:** **cancel-for-now**. Manual one-off requests are gone for good;
|
||||||
|
monitored/wanted releases stay wanted and may reappear on the monitor's normal cadence.
|
||||||
|
Permanent skip remains the existing per-release **Ignore**.
|
||||||
|
- **Pause scope:** **downloads only**. Pausing halts claiming/downloading; the monitor keeps
|
||||||
|
finding and enqueueing releases (they pile up in the queue) and Resume drains them.
|
||||||
|
|
||||||
|
## Data model
|
||||||
|
|
||||||
|
- **New column** `Job.paused Boolean @default(false)` — a Prisma migration
|
||||||
|
(`add_job_paused`). Auto-applies on deploy.
|
||||||
|
- **Config keys** (key/value `Config` rows, no migration):
|
||||||
|
- `floor.paused` — `"true"`/`"false"`; global press pause.
|
||||||
|
- `floor.jobDelaySeconds` — integer string, default `"0"`; delay between downloads.
|
||||||
|
- `floor.killRequested` — `"true"`/`"false"`; one-shot force-kill signal, cleared by the
|
||||||
|
worker on consumption.
|
||||||
|
|
||||||
|
## Worker changes
|
||||||
|
|
||||||
|
`worker/lyra_worker/main.py` and `worker/lyra_worker/claim.py`.
|
||||||
|
|
||||||
|
1. **Inter-job delay.** After `finish_job` (`main.py:297`), read `floor.jobDelaySeconds` from
|
||||||
|
the already-fresh `config` and sleep that many seconds before the next loop iteration.
|
||||||
|
Sleep in small (~2s) chunks so the loop stays responsive; the force-kill watcher runs on a
|
||||||
|
separate thread and works regardless. Parse with the existing `_int`-style guard (invalid →
|
||||||
|
0). A value of 0 preserves today's back-to-back behavior.
|
||||||
|
|
||||||
|
2. **Global pause.** Immediately before `claim_next` (`main.py:286`), if `floor.paused` is
|
||||||
|
truthy (`_TRUE` idiom), skip claiming: `time.sleep(IDLE_SLEEP); continue`. Because the loop
|
||||||
|
is serial, any in-flight job has already finished before this check — pause stops the
|
||||||
|
*next* download. Monitor sweep / scan / discovery (above the claim in the loop body)
|
||||||
|
continue to run, so the queue keeps filling while paused.
|
||||||
|
|
||||||
|
3. **Per-item pause.** `claim_next` `WHERE` gains `AND NOT "paused"`. A paused queued job is
|
||||||
|
skipped by the claim but remains `state='requested'`, so it is not "lost" and resumes
|
||||||
|
claiming as soon as `paused` is cleared. `reclaim_stuck_jobs` is unaffected (it only touches
|
||||||
|
in-flight states).
|
||||||
|
|
||||||
|
4. **Force-kill watcher.** A daemon thread started in `run_forever` (modeled on
|
||||||
|
`_liveness_heartbeat`, `main.py:216-226`) polls `floor.killRequested` on its own DB
|
||||||
|
connection every ~1.5s. When set, it clears the flag (commit) and calls `os._exit(0)`. The
|
||||||
|
container restart policy brings the worker back; startup `clear_staging_root` +
|
||||||
|
`reclaim_stuck_jobs` wipe the partial download and requeue the aborted job. The "Stop now"
|
||||||
|
action also sets `floor.paused=true`, so the worker returns **idle** rather than instantly
|
||||||
|
re-downloading the aborted album; the user resumes when ready.
|
||||||
|
|
||||||
|
## Infrastructure (ops repo)
|
||||||
|
|
||||||
|
The prod worker's Docker restart policy is currently **`no`** (`docker inspect` →
|
||||||
|
`RestartPolicy=no`), so process death — from force-kill *or any crash* — leaves it down. The
|
||||||
|
force-kill design requires `restart: unless-stopped` on the `worker` service in the ops repo
|
||||||
|
(`/opt/git/vm-download/docker/lyra/compose.yaml`). The in-repo `docker-compose.yml` already has
|
||||||
|
this (`docker-compose.yml:53`); this change brings prod in line and is good hygiene
|
||||||
|
independent of this feature. This is a one-line change in a separate repo and must be applied
|
||||||
|
before the force-kill feature is usable in prod.
|
||||||
|
|
||||||
|
## API (web) — follows existing `retry` / `library DELETE` patterns
|
||||||
|
|
||||||
|
- **`GET /api/floor`** → `{ paused: boolean, jobDelaySeconds: number }` (reads the Config
|
||||||
|
rows). Used by the Floor to render the paused banner and control states.
|
||||||
|
- **`POST /api/floor`** with `{ action }`:
|
||||||
|
- `"pause"` / `"resume"` → upsert `floor.paused`.
|
||||||
|
- `"clear"` → delete every `Request` whose `Job.state = 'requested'` (queued, not yet
|
||||||
|
claimed). Cascade removes the Jobs and Candidates. The in-flight and terminal jobs are
|
||||||
|
untouched. Returns the number cleared.
|
||||||
|
- `"stop"` → upsert `floor.paused='true'` and `floor.killRequested='true'`.
|
||||||
|
- **`POST /api/requests/[id]/pause`** with `{ paused: boolean }` → set `Job.paused`. Guard:
|
||||||
|
the job must be `state='requested'` (else `400`, matching the existing retry route), since
|
||||||
|
pausing an in-flight job has no effect.
|
||||||
|
- **`DELETE /api/requests/[id]`** → delete the `Request` (cascade Job + Candidates). Guard:
|
||||||
|
only when `Job.state = 'requested'`, so the active download is never touched (that is the
|
||||||
|
job of `Stop now`). Mirrors `DELETE /api/library/[id]`.
|
||||||
|
|
||||||
|
## Web UI
|
||||||
|
|
||||||
|
`web/src/app/queue.tsx` and `web/src/app/_ui/job-row.tsx`; styling via existing `.btn`
|
||||||
|
variants in `design.css`.
|
||||||
|
|
||||||
|
- **Press header** (above the queue list):
|
||||||
|
- `Pause the press` / `Resume` toggle (from `GET /api/floor`).
|
||||||
|
- `Clear queue` — confirm dialog showing the queued count; calls `POST /api/floor
|
||||||
|
{action:"clear"}`.
|
||||||
|
- `Stop now ⚠` — rendered only when a job is actively downloading (derivable from the
|
||||||
|
existing `/api/requests` data: any job in `matching`/`matched`/`downloading`/`tagging`).
|
||||||
|
Confirm dialog ("aborts the current download and pauses the press"); calls `POST /api/floor
|
||||||
|
{action:"stop"}`.
|
||||||
|
- A "Press paused — N queued" banner/chip when `paused`.
|
||||||
|
- **Per-item controls** (queued rows only — `job.state === 'requested'`), placed in the row
|
||||||
|
`note` slot as `.btn sm ghost`:
|
||||||
|
- `Pause` / `Resume` → `POST /api/requests/[id]/pause`.
|
||||||
|
- `Remove` → `DELETE /api/requests/[id]`.
|
||||||
|
- A per-item paused row shows a subtle "paused" chip and the `Resume` button.
|
||||||
|
- Active-download and `needs_attention` rows keep their current rendering (Retry unchanged).
|
||||||
|
- Polling cadence stays 2s; `GET /api/floor` is fetched alongside the existing three calls in
|
||||||
|
`refresh()`.
|
||||||
|
|
||||||
|
## Settings
|
||||||
|
|
||||||
|
Add a numeric field **"Delay between downloads (seconds)"** (default 0, min 0) to the settings
|
||||||
|
form (`web/src/app/settings/settings-form.tsx`), persisted to `floor.jobDelaySeconds` via the
|
||||||
|
existing settings config-write path (the same mechanism used for the other numeric/monitor
|
||||||
|
settings). Include short helptext explaining it paces sustained Qobuz load.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- **Worker** (pytest, `worker/tests/`):
|
||||||
|
- `claim_next` skips `paused=true` queued jobs and still claims the next unpaused one.
|
||||||
|
- Global pause: the loop skips claiming when `floor.paused` is set (unit-test the claim-gate
|
||||||
|
helper).
|
||||||
|
- Delay: the delay value is read and applied (unit-test the parse/gate helper; keep the
|
||||||
|
sleep injectable/mockable).
|
||||||
|
- Kill watcher: when `floor.killRequested` is set, the watcher clears the flag and triggers
|
||||||
|
exit (mock `os._exit`, assert flag cleared + exit called).
|
||||||
|
- **Web** (route tests, mirroring `requests/[id]/retry/route.test.ts` and
|
||||||
|
`library/bulk/route.test.ts`):
|
||||||
|
- `POST /api/floor` for each action; `clear` deletes only `requested` jobs; `stop` sets both
|
||||||
|
flags.
|
||||||
|
- `POST /api/requests/[id]/pause` toggles `Job.paused`; rejects non-`requested`.
|
||||||
|
- `DELETE /api/requests/[id]` removes a queued Request; refuses an in-flight/terminal job.
|
||||||
|
- `GET /api/floor` returns the Config-derived state.
|
||||||
|
|
||||||
|
## Edge cases
|
||||||
|
|
||||||
|
- **Remove/Clear of monitored items** reappear later by design (cancel-for-now). No extra
|
||||||
|
handling; the per-release Ignore is the permanent path.
|
||||||
|
- **Kill with nothing downloading** still restarts the worker (brief blip). Minimized by only
|
||||||
|
showing `Stop now` while a download is active.
|
||||||
|
- **Delay during pause:** the delay sleeps only *between* jobs; pause is checked before each
|
||||||
|
claim, so a paused press never starts a delayed job.
|
||||||
|
- **Race:** clicking per-item Pause/Remove on a job that was just claimed — the guards
|
||||||
|
(`state='requested'`) reject it; the UI only renders those buttons on queued rows, and the
|
||||||
|
2s refresh reconciles.
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- Resuming a partially-downloaded album (no download-resume support; force-killed jobs
|
||||||
|
re-download from scratch).
|
||||||
|
- Reordering the queue / per-item priority.
|
||||||
|
- Any change to the per-release Ignore or the wanted list UI.
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
# Qobuz pacing ("budget gate") — design
|
||||||
|
|
||||||
|
Date: 2026-07-15
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Bulk hi-res downloading got the Qobuz account blocked (`403 USER_BLOCKED`). We need to keep
|
||||||
|
using Qobuz for hi-res but at a volume and cadence that looks like a human listener, so the
|
||||||
|
account isn't flagged again — while a large wanted-list still drains promptly via other sources.
|
||||||
|
|
||||||
|
## Approach: a per-job Qobuz eligibility gate
|
||||||
|
|
||||||
|
One gate decides, **per album**, whether Qobuz is an eligible source *right now*. If the gate is
|
||||||
|
closed, Qobuz candidates are dropped before ranking, so the worker falls through to the best
|
||||||
|
non-Qobuz source (Soulseek/YouTube) and keeps moving — no idling. Monitored albums grabbed that
|
||||||
|
way are upgraded to Qobuz hi-res later, automatically, via the existing quality-upgrade window,
|
||||||
|
once Qobuz has budget again.
|
||||||
|
|
||||||
|
**Qobuz is eligible for a job only if ALL hold:**
|
||||||
|
1. **Active hours** — current hour ∈ [activeStart, activeEnd) (a daytime window; start < end).
|
||||||
|
2. **Under today's cap** — Qobuz downloads counted today < today's cap.
|
||||||
|
3. **Spacing elapsed** — now ≥ the randomized "next allowed" time set after the last Qobuz
|
||||||
|
download.
|
||||||
|
|
||||||
|
After each successful Qobuz download the worker bumps today's counter and sets the next-allowed
|
||||||
|
time. The counter resets at the day boundary (tracked by a stored date).
|
||||||
|
|
||||||
|
**Today's cap = warm-up ramp of the steady cap**, by account age (days since `warmupStartDate`):
|
||||||
|
|
||||||
|
| Account age (days) | Cap |
|
||||||
|
|---|---|
|
||||||
|
| 0–2 | min(5, steady) |
|
||||||
|
| 3–6 | min(10, steady) |
|
||||||
|
| 7–13 | min(20, steady) |
|
||||||
|
| 14+ | steady |
|
||||||
|
|
||||||
|
If `warmupStartDate` is unset, no ramp (steady cap applies).
|
||||||
|
|
||||||
|
**Spacing / jitter (combined):** the gap to the next Qobuz download is randomized around
|
||||||
|
`activeWindowSeconds / todayCap` (so the day's budget spreads evenly across active hours), ±30%,
|
||||||
|
with a hard floor (≥ 60 s). This *is* the jitter — never a fixed metronome — and it prevents
|
||||||
|
front-loading the whole daily cap into the first hour.
|
||||||
|
|
||||||
|
## Configuration (Config keys; editable in Settings → Qobuz → Pacing)
|
||||||
|
|
||||||
|
Settings:
|
||||||
|
- `qobuz.pacing.enabled` (default `true`)
|
||||||
|
- `qobuz.pacing.activeStartHour` (default `8`)
|
||||||
|
- `qobuz.pacing.activeEndHour` (default `23`)
|
||||||
|
- `qobuz.pacing.dailyCap` (steady cap, default `40`)
|
||||||
|
- `qobuz.pacing.warmupStartDate` (ISO date; set to the new account's start day)
|
||||||
|
|
||||||
|
Worker-managed state (not user-edited):
|
||||||
|
- `qobuz.pacing.countDay` (date the counter is for)
|
||||||
|
- `qobuz.pacing.countToday` (int)
|
||||||
|
- `qobuz.pacing.nextAllowedAt` (epoch seconds)
|
||||||
|
|
||||||
|
Current hour / day come from the **DB clock** (`now()`), avoiding container-timezone ambiguity.
|
||||||
|
Config is read fresh each worker loop, so changes take effect with no restart.
|
||||||
|
|
||||||
|
## Worker integration
|
||||||
|
|
||||||
|
- New module `worker/lyra_worker/qobuz_gate.py`: a `QobuzPacing` config dataclass
|
||||||
|
(`from_config`), a pure `today_cap(...)`, a pure `is_allowed(...)` decision, and thin
|
||||||
|
DB helpers `gate_open(conn, now)` (reads state + decides) and `record_download(conn, pacing,
|
||||||
|
now)` (bumps counter + sets next-allowed).
|
||||||
|
- `pipeline.py`: after searching sources, if `gate_open` is false AND at least one non-Qobuz
|
||||||
|
candidate exists, drop Qobuz candidates before ranking (fall-through). If Qobuz is the *only*
|
||||||
|
source available, keep it (a Qobuz-only setup isn't gated — holding would strand the album).
|
||||||
|
- After a successful download where `winner.source == "qobuz"`, call `record_download`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Pure-logic unit tests for `is_allowed` (each gate condition), `today_cap` (ramp boundaries), and
|
||||||
|
the spacing computation; pipeline tests that a gated Qobuz candidate is dropped in favour of a
|
||||||
|
non-Qobuz source, that an open gate keeps Qobuz, and that a successful Qobuz download records the
|
||||||
|
counter/next-allowed.
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- Holding/parking an album for later Qobuz when Qobuz is the only source (fails/last-source as
|
||||||
|
today). Multi-source setups fall through, which is the common case.
|
||||||
|
- Timezone-aware active-hours windows that wrap past midnight (start < end only).
|
||||||
Executable
+29
@@ -0,0 +1,29 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Build the Lyra web + worker images and push them to the Gitea container registry, so the VM
|
||||||
|
# can pull pre-built images instead of building from source.
|
||||||
|
#
|
||||||
|
# Usage (from the Lyra repo root):
|
||||||
|
# docker login git.jger.nl # once — Gitea username + a token with package:write
|
||||||
|
# ./scripts/build-push.sh # builds + pushes :latest and :<short-sha>
|
||||||
|
#
|
||||||
|
# Override the registry namespace with LYRA_REGISTRY (default git.jger.nl/jonathan).
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
REGISTRY="${LYRA_REGISTRY:-git.jger.nl/jonathan}"
|
||||||
|
here="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
sha="$(git -C "$here" rev-parse --short HEAD)"
|
||||||
|
|
||||||
|
for svc in web worker; do
|
||||||
|
img="$REGISTRY/lyra-$svc"
|
||||||
|
echo "==> building $img ($sha)"
|
||||||
|
docker build -t "$img:latest" -t "$img:$sha" "$here/$svc"
|
||||||
|
done
|
||||||
|
|
||||||
|
for svc in web worker; do
|
||||||
|
img="$REGISTRY/lyra-$svc"
|
||||||
|
echo "==> pushing $img:latest and $img:$sha"
|
||||||
|
docker push "$img:latest"
|
||||||
|
docker push "$img:$sha"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "Done. Pulled on the VM with: docker compose pull && docker compose up -d"
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
-- AlterTable
|
||||||
|
ALTER TABLE "DiscoverySuggestion" ADD COLUMN "albumReason" TEXT;
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
-- AlterTable
|
||||||
|
ALTER TABLE "Job" ADD COLUMN "paused" BOOLEAN NOT NULL DEFAULT false;
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
-- Estimated seconds remaining for the active download (measured throughput). Nullable.
|
||||||
|
ALTER TABLE "Job" ADD COLUMN "downloadEtaSeconds" INTEGER;
|
||||||
@@ -58,6 +58,10 @@ model Job {
|
|||||||
createdAt DateTime @default(now())
|
createdAt DateTime @default(now())
|
||||||
updatedAt DateTime @updatedAt
|
updatedAt DateTime @updatedAt
|
||||||
downloadProgress Float @default(0)
|
downloadProgress Float @default(0)
|
||||||
|
// Estimated seconds until the current download finishes, from measured throughput (slskd byte
|
||||||
|
// transfer rate). Null when unknown (no speed sample yet, or a source that doesn't report bytes).
|
||||||
|
downloadEtaSeconds Int?
|
||||||
|
paused Boolean @default(false)
|
||||||
candidates Candidate[]
|
candidates Candidate[]
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -171,6 +175,10 @@ model DiscoverySuggestion {
|
|||||||
score Float
|
score Float
|
||||||
seedCount Int
|
seedCount Int
|
||||||
sources String[]
|
sources String[]
|
||||||
|
// Why this album is suggested (kind='album' only): "newest" (the artist's newest un-owned
|
||||||
|
// release), "popular" (their most-played per Last.fm), or "newest,popular" when they're the
|
||||||
|
// same release. Null for artist suggestions / rows from before this column existed.
|
||||||
|
albumReason String?
|
||||||
status SuggestionStatus @default(pending)
|
status SuggestionStatus @default(pending)
|
||||||
dedupeKey String @unique
|
dedupeKey String @unique
|
||||||
createdAt DateTime @default(now())
|
createdAt DateTime @default(now())
|
||||||
|
|||||||
@@ -0,0 +1,49 @@
|
|||||||
|
"use client";
|
||||||
|
|
||||||
|
import { Modal } from "./modal";
|
||||||
|
import type { ArtistHit } from "@/lib/artist-resolve";
|
||||||
|
|
||||||
|
/** Disambiguation picker: several MusicBrainz artists share a name (e.g. two bands both called
|
||||||
|
* "Looking Glass"), so let the user choose the right one by its MB disambiguation text instead
|
||||||
|
* of silently trusting Last.fm's — sometimes wrong — MBID. Candidates arrive in MB relevance
|
||||||
|
* order, so the first is flagged "Suggested". */
|
||||||
|
export function ArtistPicker({
|
||||||
|
open,
|
||||||
|
name,
|
||||||
|
candidates,
|
||||||
|
onPick,
|
||||||
|
onClose,
|
||||||
|
}: {
|
||||||
|
open: boolean;
|
||||||
|
name: string;
|
||||||
|
candidates: ArtistHit[];
|
||||||
|
onPick: (mbid: string, name: string) => void;
|
||||||
|
onClose: () => void;
|
||||||
|
}) {
|
||||||
|
return (
|
||||||
|
<Modal open={open} onClose={onClose} title="Which artist?">
|
||||||
|
<p className="muted-note">
|
||||||
|
More than one artist is named “{name}”. Pick the right one.
|
||||||
|
</p>
|
||||||
|
<ul className="list">
|
||||||
|
{candidates.map((c, i) => (
|
||||||
|
<li key={c.mbid} className="list-row">
|
||||||
|
<div className="main">
|
||||||
|
<div className="rtitle">
|
||||||
|
<button className="linkish" onClick={() => onPick(c.mbid, c.name)}>
|
||||||
|
{c.name}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
<div className="rmeta">
|
||||||
|
<span>{c.disambiguation || "no description on MusicBrainz"}</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="actions">
|
||||||
|
{i === 0 ? <span className="chip working">Suggested</span> : null}
|
||||||
|
</div>
|
||||||
|
</li>
|
||||||
|
))}
|
||||||
|
</ul>
|
||||||
|
</Modal>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -14,6 +14,7 @@ export function JobRow({
|
|||||||
album,
|
album,
|
||||||
kind,
|
kind,
|
||||||
label,
|
label,
|
||||||
|
marker,
|
||||||
meta,
|
meta,
|
||||||
note,
|
note,
|
||||||
bar,
|
bar,
|
||||||
@@ -22,6 +23,7 @@ export function JobRow({
|
|||||||
album: string;
|
album: string;
|
||||||
kind: Kind;
|
kind: Kind;
|
||||||
label: string;
|
label: string;
|
||||||
|
marker?: string;
|
||||||
meta?: ReactNode;
|
meta?: ReactNode;
|
||||||
note?: ReactNode;
|
note?: ReactNode;
|
||||||
bar?: ReactNode;
|
bar?: ReactNode;
|
||||||
@@ -31,6 +33,7 @@ export function JobRow({
|
|||||||
<div className="stripe" />
|
<div className="stripe" />
|
||||||
<div>
|
<div>
|
||||||
<h3 className="title">
|
<h3 className="title">
|
||||||
|
{marker ? <span className="nextup">{marker}</span> : null}
|
||||||
{album} <span className="artist">· {artist}</span>
|
{album} <span className="artist">· {artist}</span>
|
||||||
</h3>
|
</h3>
|
||||||
{meta ? <div className="meta">{meta}</div> : null}
|
{meta ? <div className="meta">{meta}</div> : null}
|
||||||
|
|||||||
@@ -0,0 +1,47 @@
|
|||||||
|
"use client";
|
||||||
|
|
||||||
|
import { useEffect, useRef, useState, type ReactNode } from "react";
|
||||||
|
|
||||||
|
/** A vertical "⋮" trigger that opens a small popup menu. Closes on outside click, Escape, or any
|
||||||
|
* click inside the popup (so menu items don't each need to manage it). Children are the items —
|
||||||
|
* use <button className="menu-item">…</button>. */
|
||||||
|
export function KebabMenu({ label = "Actions", children }: { label?: string; children: ReactNode }) {
|
||||||
|
const [open, setOpen] = useState(false);
|
||||||
|
const ref = useRef<HTMLDivElement>(null);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!open) return;
|
||||||
|
function onDoc(e: MouseEvent) {
|
||||||
|
if (ref.current && !ref.current.contains(e.target as Node)) setOpen(false);
|
||||||
|
}
|
||||||
|
function onKey(e: KeyboardEvent) {
|
||||||
|
if (e.key === "Escape") setOpen(false);
|
||||||
|
}
|
||||||
|
document.addEventListener("mousedown", onDoc);
|
||||||
|
document.addEventListener("keydown", onKey);
|
||||||
|
return () => {
|
||||||
|
document.removeEventListener("mousedown", onDoc);
|
||||||
|
document.removeEventListener("keydown", onKey);
|
||||||
|
};
|
||||||
|
}, [open]);
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="menu" ref={ref}>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="menu-trigger"
|
||||||
|
aria-haspopup="menu"
|
||||||
|
aria-expanded={open}
|
||||||
|
aria-label={label}
|
||||||
|
onClick={() => setOpen((o) => !o)}
|
||||||
|
>
|
||||||
|
⋮
|
||||||
|
</button>
|
||||||
|
{open ? (
|
||||||
|
<div className="menu-pop" role="menu" onClick={() => setOpen(false)}>
|
||||||
|
{children}
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -1,5 +1,20 @@
|
|||||||
import { describe, it, expect } from "vitest";
|
import { describe, it, expect } from "vitest";
|
||||||
import { describeJob, timeAgo } from "./status";
|
import { describeJob, etaLabel, timeAgo } from "./status";
|
||||||
|
|
||||||
|
describe("etaLabel", () => {
|
||||||
|
it("formats seconds, minutes, and hours", () => {
|
||||||
|
expect(etaLabel(45)).toBe("~45s left");
|
||||||
|
expect(etaLabel(240)).toBe("~4m left");
|
||||||
|
expect(etaLabel(3900)).toBe("~1h 5m left");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("returns null for missing or non-positive input", () => {
|
||||||
|
expect(etaLabel(null)).toBeNull();
|
||||||
|
expect(etaLabel(undefined)).toBeNull();
|
||||||
|
expect(etaLabel(0)).toBeNull();
|
||||||
|
expect(etaLabel(-5)).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe("describeJob", () => {
|
describe("describeJob", () => {
|
||||||
it("maps pipeline states to literal labels + severity kind", () => {
|
it("maps pipeline states to literal labels + severity kind", () => {
|
||||||
|
|||||||
@@ -27,6 +27,16 @@ export function timeAgo(value: string | Date | null | undefined, now: number): s
|
|||||||
return `${Math.floor(h / 24)}d`;
|
return `${Math.floor(h / 24)}d`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Compact download ETA like "~4m left", "~45s left", "~1h 5m left". Null/≤0 → no label. */
|
||||||
|
export function etaLabel(seconds: number | null | undefined): string | null {
|
||||||
|
if (seconds == null || seconds <= 0) return null;
|
||||||
|
if (seconds < 60) return `~${seconds}s left`;
|
||||||
|
const m = Math.round(seconds / 60);
|
||||||
|
if (m < 60) return `~${m}m left`;
|
||||||
|
const h = Math.floor(m / 60);
|
||||||
|
return `~${h}h ${m % 60}m left`;
|
||||||
|
}
|
||||||
|
|
||||||
export function describeJob(state: string, stage: string): { label: string; kind: Kind } {
|
export function describeJob(state: string, stage: string): { label: string; kind: Kind } {
|
||||||
void stage; // stage no longer drives the label; kept for call-site compatibility
|
void stage; // stage no longer drives the label; kept for call-site compatibility
|
||||||
return (
|
return (
|
||||||
|
|||||||
@@ -63,6 +63,20 @@ describe("discover action API", () => {
|
|||||||
expect((await prisma.discoverySuggestion.findUnique({ where: { id: s.id } }))!.status).toBe("followed");
|
expect((await prisma.discoverySuggestion.findUnique({ where: { id: s.id } }))!.status).toBe("followed");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("dismissing an artist also dismisses its pending album suggestions", async () => {
|
||||||
|
const a = await prisma.discoverySuggestion.create({
|
||||||
|
data: { kind: "artist", artistMbid: "shared", artistName: "Cand", score: 1, seedCount: 1,
|
||||||
|
sources: ["listenbrainz"], secondaryTypes: [], dedupeKey: "artist:shared:" },
|
||||||
|
});
|
||||||
|
const alb = await prisma.discoverySuggestion.create({
|
||||||
|
data: { kind: "album", artistMbid: "shared", artistName: "Cand", rgMbid: "rgS", album: "Rec",
|
||||||
|
albumReason: "newest", score: 1, seedCount: 1, sources: ["listenbrainz"],
|
||||||
|
secondaryTypes: [], dedupeKey: "album:shared:rgS" },
|
||||||
|
});
|
||||||
|
await post(a.id, { action: "dismiss" });
|
||||||
|
expect((await prisma.discoverySuggestion.findUnique({ where: { id: alb.id } }))!.status).toBe("dismissed");
|
||||||
|
});
|
||||||
|
|
||||||
it("want upserts a monitored release from the suggestion and marks wanted", async () => {
|
it("want upserts a monitored release from the suggestion and marks wanted", async () => {
|
||||||
const s = await album("rgW");
|
const s = await album("rgW");
|
||||||
const res = await post(s.id, { action: "want" });
|
const res = await post(s.id, { action: "want" });
|
||||||
|
|||||||
@@ -19,6 +19,14 @@ export async function POST(request: Request, { params }: { params: Promise<{ id:
|
|||||||
|
|
||||||
if (action === "dismiss") {
|
if (action === "dismiss") {
|
||||||
await prisma.discoverySuggestion.update({ where: { id }, data: { status: "dismissed" } });
|
await prisma.discoverySuggestion.update({ where: { id }, data: { status: "dismissed" } });
|
||||||
|
// Dismissing an artist also dismisses its album suggestions, so the whole unified row
|
||||||
|
// disappears cleanly instead of leaving the artist's albums stranded.
|
||||||
|
if (s.kind === "artist") {
|
||||||
|
await prisma.discoverySuggestion.updateMany({
|
||||||
|
where: { kind: "album", artistMbid: s.artistMbid, status: "pending" },
|
||||||
|
data: { status: "dismissed" },
|
||||||
|
});
|
||||||
|
}
|
||||||
return Response.json({ id, status: "dismissed" });
|
return Response.json({ id, status: "dismissed" });
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,17 @@
|
|||||||
|
import { describe, it, expect } from "vitest";
|
||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
import { POST } from "./route";
|
||||||
|
|
||||||
|
describe("POST /api/discover/reset", () => {
|
||||||
|
it("sets the reset + requested flags so the worker rebuilds on the next sweep", async () => {
|
||||||
|
const res = await POST();
|
||||||
|
expect(res.status).toBe(202);
|
||||||
|
expect((await res.json()).reset).toBe(true);
|
||||||
|
const cfg = Object.fromEntries(
|
||||||
|
(await prisma.config.findMany({ where: { key: { in: ["discover.resetRequested", "discover.requested"] } } }))
|
||||||
|
.map((c) => [c.key, c.value]),
|
||||||
|
);
|
||||||
|
expect(cfg["discover.resetRequested"]).toBe("true");
|
||||||
|
expect(cfg["discover.requested"]).toBe("true");
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
|
||||||
|
// POST /api/discover/reset — request a full rebuild: the worker clears the pending
|
||||||
|
// suggestions, contributions, and seed throttles at the start of the next sweep, then
|
||||||
|
// regenerates everything with the current logic. Kicks a sweep immediately.
|
||||||
|
export async function POST() {
|
||||||
|
await Promise.all(
|
||||||
|
["discover.resetRequested", "discover.requested"].map((key) =>
|
||||||
|
prisma.config.upsert({
|
||||||
|
where: { key },
|
||||||
|
create: { key, value: "true", secret: false },
|
||||||
|
update: { value: "true" },
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
return Response.json({ reset: true }, { status: 202 });
|
||||||
|
}
|
||||||
@@ -3,51 +3,61 @@ import { prisma } from "@/lib/db";
|
|||||||
import { GET } from "./route";
|
import { GET } from "./route";
|
||||||
|
|
||||||
describe("discover feed API", () => {
|
describe("discover feed API", () => {
|
||||||
it("returns pending artists and albums split, highest score first, excluding non-pending", async () => {
|
it("groups pending suggestions into artist rows with their albums, highest score first", async () => {
|
||||||
await prisma.discoverySuggestion.createMany({
|
await prisma.discoverySuggestion.createMany({
|
||||||
data: [
|
data: [
|
||||||
{ kind: "artist", artistMbid: "a1", artistName: "Low", score: 0.2, seedCount: 1,
|
{ kind: "artist", artistMbid: "a1", artistName: "Low", score: 0.2, seedCount: 1,
|
||||||
sources: ["listenbrainz"], secondaryTypes: [], dedupeKey: "artist:a1:" },
|
sources: ["listenbrainz"], secondaryTypes: [], dedupeKey: "artist:a1:" },
|
||||||
{ kind: "artist", artistMbid: "a2", artistName: "High", score: 0.9, seedCount: 2,
|
{ kind: "artist", artistMbid: "a2", artistName: "High", score: 0.9, seedCount: 2,
|
||||||
sources: ["listenbrainz"], secondaryTypes: [], dedupeKey: "artist:a2:" },
|
sources: ["listenbrainz"], secondaryTypes: [], dedupeKey: "artist:a2:" },
|
||||||
{ kind: "album", artistMbid: "a3", artistName: "Band", rgMbid: "rg1", album: "Rec",
|
{ kind: "album", artistMbid: "a2", artistName: "High", rgMbid: "rg-new", album: "Latest",
|
||||||
score: 0.5, seedCount: 1, sources: ["listenbrainz"], secondaryTypes: [], dedupeKey: "album:a3:rg1" },
|
firstReleaseDate: "2022-01-01", albumReason: "newest", score: 0.9, seedCount: 2,
|
||||||
{ kind: "artist", artistMbid: "a4", artistName: "Gone", score: 5, seedCount: 1,
|
sources: ["listenbrainz"], secondaryTypes: [], dedupeKey: "album:a2:rg-new" },
|
||||||
sources: ["listenbrainz"], secondaryTypes: [], status: "dismissed", dedupeKey: "artist:a4:" },
|
{ kind: "album", artistMbid: "a2", artistName: "High", rgMbid: "rg-hit", album: "Big Hit",
|
||||||
|
firstReleaseDate: "2018-01-01", albumReason: "popular", score: 0.9, seedCount: 2,
|
||||||
|
sources: ["listenbrainz"], secondaryTypes: [], dedupeKey: "album:a2:rg-hit" },
|
||||||
|
{ kind: "artist", artistMbid: "a3", artistName: "Gone", score: 5, seedCount: 1,
|
||||||
|
sources: ["listenbrainz"], secondaryTypes: [], status: "dismissed", dedupeKey: "artist:a3:" },
|
||||||
],
|
],
|
||||||
});
|
});
|
||||||
const body = await (await GET()).json();
|
const body = await (await GET()).json();
|
||||||
expect(body.artists.map((a: any) => a.artistName)).toEqual(["High", "Low"]); // dismissed excluded
|
expect(body.rows.map((r: { artistName: string }) => r.artistName)).toEqual(["High", "Low"]); // dismissed excluded, score desc
|
||||||
expect(body.albums.map((a: any) => a.album)).toEqual(["Rec"]);
|
const high = body.rows[0];
|
||||||
expect(body.artists[0]).toMatchObject({ artistMbid: "a2", seedCount: 2, sources: ["listenbrainz"] });
|
expect(high).toMatchObject({ id: expect.any(String), artistMbid: "a2", seedCount: 2 });
|
||||||
|
// albums sorted newest-first, each carrying its reason
|
||||||
|
expect(high.albums.map((a: { album: string; albumReason: string }) => [a.album, a.albumReason]))
|
||||||
|
.toEqual([["Latest", "newest"], ["Big Hit", "popular"]]);
|
||||||
|
expect(body.rows[1].albums).toEqual([]); // "Low" has no album suggestions
|
||||||
});
|
});
|
||||||
|
|
||||||
it("annotates each suggestion with the followed-artist seeds that surfaced it", async () => {
|
it("yields a row for an album whose artist has no pending artist suggestion", async () => {
|
||||||
await prisma.watchedArtist.createMany({
|
await prisma.discoverySuggestion.create({
|
||||||
data: [
|
data: { kind: "album", artistMbid: "orphan", artistName: "Orphan Band", rgMbid: "rgX",
|
||||||
{ mbid: "seed-am", name: "Arctic Monkeys" },
|
album: "Stray", albumReason: "newest", score: 0.4, seedCount: 1, sources: ["lastfm"],
|
||||||
{ mbid: "seed-cp", name: "Coldplay" },
|
secondaryTypes: [], dedupeKey: "album:orphan:rgX" },
|
||||||
{ mbid: "seed-gone", name: "Unfollowed" },
|
|
||||||
],
|
|
||||||
});
|
});
|
||||||
await prisma.discoverySuggestion.createMany({
|
const body = await (await GET()).json();
|
||||||
data: [
|
expect(body.rows).toHaveLength(1);
|
||||||
{ kind: "artist", artistMbid: "cand1", artistName: "The Strokes", score: 0.9, seedCount: 2,
|
expect(body.rows[0]).toMatchObject({ id: null, artistName: "Orphan Band" });
|
||||||
sources: ["listenbrainz"], secondaryTypes: [], dedupeKey: "artist:cand1:" },
|
expect(body.rows[0].albums[0].album).toBe("Stray");
|
||||||
{ kind: "album", artistMbid: "cand1", artistName: "The Strokes", rgMbid: "rgX", album: "Room on Fire",
|
});
|
||||||
score: 0.7, seedCount: 2, sources: ["listenbrainz"], secondaryTypes: [], dedupeKey: "album:cand1:rgX" },
|
|
||||||
],
|
it("annotates rows with the followed-artist seeds that surfaced them", async () => {
|
||||||
|
await prisma.watchedArtist.createMany({
|
||||||
|
data: [{ mbid: "seed-am", name: "Arctic Monkeys" }, { mbid: "seed-cp", name: "Coldplay" }],
|
||||||
|
});
|
||||||
|
await prisma.discoverySuggestion.create({
|
||||||
|
data: { kind: "artist", artistMbid: "cand1", artistName: "The Strokes", score: 0.9, seedCount: 2,
|
||||||
|
sources: ["listenbrainz"], secondaryTypes: [], dedupeKey: "artist:cand1:" },
|
||||||
});
|
});
|
||||||
await prisma.discoverySeedContribution.createMany({
|
await prisma.discoverySeedContribution.createMany({
|
||||||
data: [
|
data: [
|
||||||
{ candidateMbid: "cand1", candidateName: "The Strokes", seedMbid: "seed-am", score: 0.6, sources: ["listenbrainz"] },
|
{ candidateMbid: "cand1", candidateName: "The Strokes", seedMbid: "seed-am", score: 0.6, sources: ["listenbrainz"] },
|
||||||
{ candidateMbid: "cand1", candidateName: "The Strokes", seedMbid: "seed-cp", score: 0.3, sources: ["listenbrainz"] },
|
{ candidateMbid: "cand1", candidateName: "The Strokes", seedMbid: "seed-cp", score: 0.3, sources: ["listenbrainz"] },
|
||||||
{ candidateMbid: "cand1", candidateName: "The Strokes", seedMbid: "no-such-seed", score: 0.9, sources: ["listenbrainz"] },
|
{ candidateMbid: "cand1", candidateName: "The Strokes", seedMbid: "gone", score: 0.9, sources: ["listenbrainz"] },
|
||||||
],
|
],
|
||||||
});
|
});
|
||||||
const body = await (await GET()).json();
|
const body = await (await GET()).json();
|
||||||
// highest-scoring seed first; an unresolved/unfollowed seed is dropped
|
expect(body.rows[0].seeds).toEqual(["Arctic Monkeys", "Coldplay"]); // score desc, unfollowed dropped
|
||||||
expect(body.artists[0].seeds).toEqual(["Arctic Monkeys", "Coldplay"]);
|
|
||||||
expect(body.albums[0].seeds).toEqual(["Arctic Monkeys", "Coldplay"]); // album inherits its artist's seeds
|
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -1,15 +1,35 @@
|
|||||||
import { prisma } from "@/lib/db";
|
import { prisma } from "@/lib/db";
|
||||||
|
|
||||||
|
// One Discover row = a suggested artist plus that artist's suggested albums (newest /
|
||||||
|
// most-played), since album suggestions are always derived from a surfaced artist. Pending
|
||||||
|
// suggestions are grouped by artistMbid so the UI shows a single artist-centric row.
|
||||||
|
type Row = {
|
||||||
|
id: string | null; // the artist suggestion id (Follow/Dismiss); null if only albums exist
|
||||||
|
artistMbid: string;
|
||||||
|
artistName: string;
|
||||||
|
score: number;
|
||||||
|
seedCount: number;
|
||||||
|
sources: string[];
|
||||||
|
seeds: string[];
|
||||||
|
albums: {
|
||||||
|
id: string;
|
||||||
|
rgMbid: string | null;
|
||||||
|
album: string | null;
|
||||||
|
primaryType: string | null;
|
||||||
|
secondaryTypes: string[];
|
||||||
|
firstReleaseDate: string | null;
|
||||||
|
albumReason: string | null;
|
||||||
|
}[];
|
||||||
|
};
|
||||||
|
|
||||||
export async function GET() {
|
export async function GET() {
|
||||||
const rows = await prisma.discoverySuggestion.findMany({
|
const rows = await prisma.discoverySuggestion.findMany({
|
||||||
where: { status: "pending" },
|
where: { status: "pending" },
|
||||||
orderBy: [{ score: "desc" }, { createdAt: "asc" }],
|
orderBy: [{ score: "desc" }, { createdAt: "asc" }],
|
||||||
});
|
});
|
||||||
|
|
||||||
// "Why suggested": the followed artists (seeds) that surfaced each candidate. A suggestion's
|
// "Why suggested": the followed artists (seeds) that surfaced each candidate, from the
|
||||||
// candidate is its artistMbid (for both artist and album kinds — the album's artist is what
|
// per-seed contributions joined to the current watched roster (an unfollowed seed drops out).
|
||||||
// was found similar). Join the per-seed contributions to the current watched roster so an
|
|
||||||
// unfollowed seed drops out. Highest-scoring seed first.
|
|
||||||
const candidateMbids = [...new Set(rows.map((r) => r.artistMbid))];
|
const candidateMbids = [...new Set(rows.map((r) => r.artistMbid))];
|
||||||
const contribs = candidateMbids.length
|
const contribs = candidateMbids.length
|
||||||
? await prisma.discoverySeedContribution.findMany({
|
? await prisma.discoverySeedContribution.findMany({
|
||||||
@@ -23,28 +43,57 @@ export async function GET() {
|
|||||||
const seedsByCandidate = new Map<string, string[]>();
|
const seedsByCandidate = new Map<string, string[]>();
|
||||||
for (const c of contribs) {
|
for (const c of contribs) {
|
||||||
const name = seedNames.get(c.seedMbid);
|
const name = seedNames.get(c.seedMbid);
|
||||||
if (!name) continue; // seed no longer followed
|
if (!name) continue;
|
||||||
const list = seedsByCandidate.get(c.candidateMbid) ?? [];
|
const list = seedsByCandidate.get(c.candidateMbid) ?? [];
|
||||||
if (!list.includes(name)) list.push(name);
|
if (!list.includes(name)) list.push(name);
|
||||||
seedsByCandidate.set(c.candidateMbid, list);
|
seedsByCandidate.set(c.candidateMbid, list);
|
||||||
}
|
}
|
||||||
|
|
||||||
const map = (r: (typeof rows)[number]) => ({
|
// Group by artist. An artist suggestion sets the row header; album suggestions fill albums[].
|
||||||
id: r.id,
|
// An album whose artist has no pending artist suggestion still yields a header (never orphaned).
|
||||||
artistMbid: r.artistMbid,
|
const byArtist = new Map<string, Row>();
|
||||||
artistName: r.artistName,
|
const ensure = (r: (typeof rows)[number]): Row => {
|
||||||
rgMbid: r.rgMbid,
|
let row = byArtist.get(r.artistMbid);
|
||||||
album: r.album,
|
if (!row) {
|
||||||
primaryType: r.primaryType,
|
row = {
|
||||||
secondaryTypes: r.secondaryTypes,
|
id: null,
|
||||||
firstReleaseDate: r.firstReleaseDate,
|
artistMbid: r.artistMbid,
|
||||||
score: r.score,
|
artistName: r.artistName,
|
||||||
seedCount: r.seedCount,
|
score: r.score,
|
||||||
sources: r.sources,
|
seedCount: r.seedCount,
|
||||||
seeds: seedsByCandidate.get(r.artistMbid) ?? [],
|
sources: r.sources,
|
||||||
});
|
seeds: seedsByCandidate.get(r.artistMbid) ?? [],
|
||||||
return Response.json({
|
albums: [],
|
||||||
artists: rows.filter((r) => r.kind === "artist").map(map),
|
};
|
||||||
albums: rows.filter((r) => r.kind === "album").map(map),
|
byArtist.set(r.artistMbid, row);
|
||||||
});
|
}
|
||||||
|
return row;
|
||||||
|
};
|
||||||
|
|
||||||
|
for (const r of rows) {
|
||||||
|
const row = ensure(r);
|
||||||
|
if (r.kind === "artist") {
|
||||||
|
row.id = r.id;
|
||||||
|
row.score = r.score;
|
||||||
|
row.seedCount = r.seedCount;
|
||||||
|
row.sources = r.sources;
|
||||||
|
} else {
|
||||||
|
row.albums.push({
|
||||||
|
id: r.id,
|
||||||
|
rgMbid: r.rgMbid,
|
||||||
|
album: r.album,
|
||||||
|
primaryType: r.primaryType,
|
||||||
|
secondaryTypes: r.secondaryTypes,
|
||||||
|
firstReleaseDate: r.firstReleaseDate,
|
||||||
|
albumReason: r.albumReason,
|
||||||
|
});
|
||||||
|
row.score = Math.max(row.score, r.score); // album-only row scores by its best album
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const row of byArtist.values()) {
|
||||||
|
row.albums.sort((a, b) => (b.firstReleaseDate ?? "").localeCompare(a.firstReleaseDate ?? ""));
|
||||||
|
}
|
||||||
|
const out = [...byArtist.values()].sort((a, b) => b.score - a.score);
|
||||||
|
return Response.json({ rows: out });
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
import { describe, it, expect } from "vitest";
|
||||||
|
import { GET, POST } from "./route";
|
||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
|
||||||
|
function post(action: string) {
|
||||||
|
return new Request("http://localhost/api/floor", {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify({ action }),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async function flag(key: string) {
|
||||||
|
return (await prisma.config.findUnique({ where: { key } }))?.value ?? null;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function queuedRequest(album: string, state = "requested") {
|
||||||
|
return prisma.request.create({
|
||||||
|
data: { artist: "A", album, status: "pending", job: { create: { state: state as never } } },
|
||||||
|
include: { job: true },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("floor API", () => {
|
||||||
|
it("GET reports paused=false by default", async () => {
|
||||||
|
const res = await GET();
|
||||||
|
expect(await res.json()).toEqual({ paused: false });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("pause and resume toggle floor.paused", async () => {
|
||||||
|
await POST(post("pause"));
|
||||||
|
expect(await flag("floor.paused")).toBe("true");
|
||||||
|
const paused = await (await GET()).json();
|
||||||
|
expect(paused).toEqual({ paused: true });
|
||||||
|
|
||||||
|
await POST(post("resume"));
|
||||||
|
expect(await flag("floor.paused")).toBe("false");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("stop sets both paused and killRequested", async () => {
|
||||||
|
const res = await POST(post("stop"));
|
||||||
|
expect(res.status).toBe(200);
|
||||||
|
expect(await flag("floor.paused")).toBe("true");
|
||||||
|
expect(await flag("floor.killRequested")).toBe("true");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("clear deletes only queued (requested) requests", async () => {
|
||||||
|
const queued = await queuedRequest("Queued");
|
||||||
|
const active = await queuedRequest("Downloading", "downloading");
|
||||||
|
|
||||||
|
const res = await POST(post("clear"));
|
||||||
|
const body = await res.json();
|
||||||
|
expect(body.cleared).toBe(1);
|
||||||
|
|
||||||
|
expect(await prisma.request.findUnique({ where: { id: queued.id } })).toBeNull();
|
||||||
|
expect(await prisma.request.findUnique({ where: { id: active.id } })).not.toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rejects an unknown action", async () => {
|
||||||
|
const res = await POST(post("nope"));
|
||||||
|
expect(res.status).toBe(400);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
|
||||||
|
async function getFlag(key: string): Promise<boolean> {
|
||||||
|
const row = await prisma.config.findUnique({ where: { key } });
|
||||||
|
return row?.value === "true";
|
||||||
|
}
|
||||||
|
|
||||||
|
async function setFlag(key: string, value: boolean): Promise<void> {
|
||||||
|
const v = value ? "true" : "false";
|
||||||
|
await prisma.config.upsert({
|
||||||
|
where: { key },
|
||||||
|
create: { key, value: v, secret: false },
|
||||||
|
update: { value: v },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function GET() {
|
||||||
|
return Response.json({ paused: await getFlag("floor.paused") });
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function POST(request: Request) {
|
||||||
|
let body: { action?: string };
|
||||||
|
try {
|
||||||
|
body = (await request.json()) as { action?: string };
|
||||||
|
} catch {
|
||||||
|
return Response.json({ error: "invalid JSON" }, { status: 400 });
|
||||||
|
}
|
||||||
|
|
||||||
|
switch (body.action) {
|
||||||
|
case "pause":
|
||||||
|
await setFlag("floor.paused", true);
|
||||||
|
return Response.json({ ok: true, paused: true });
|
||||||
|
case "resume":
|
||||||
|
await setFlag("floor.paused", false);
|
||||||
|
return Response.json({ ok: true, paused: false });
|
||||||
|
case "stop":
|
||||||
|
await setFlag("floor.paused", true);
|
||||||
|
await setFlag("floor.killRequested", true);
|
||||||
|
return Response.json({ ok: true, paused: true, stopped: true });
|
||||||
|
case "clear": {
|
||||||
|
const { count } = await prisma.request.deleteMany({
|
||||||
|
where: { job: { is: { state: "requested" } } },
|
||||||
|
});
|
||||||
|
return Response.json({ ok: true, cleared: count });
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
return Response.json({ error: "unknown action" }, { status: 400 });
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -34,3 +34,23 @@ describe("monitor config API", () => {
|
|||||||
expect(await prisma.config.count()).toBe(before);
|
expect(await prisma.config.count()).toBe(before);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe("monitor config — floor.jobDelaySeconds", () => {
|
||||||
|
it("defaults jobDelaySeconds to 0 and round-trips a new value", async () => {
|
||||||
|
const before = await (await GET()).json();
|
||||||
|
expect(before["floor.jobDelaySeconds"]).toBe("0");
|
||||||
|
|
||||||
|
await PATCH(
|
||||||
|
new Request("http://localhost/api/monitor/config", {
|
||||||
|
method: "PATCH",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify({ "floor.jobDelaySeconds": "45" }),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
const stored = await prisma.config.findUnique({ where: { key: "floor.jobDelaySeconds" } });
|
||||||
|
expect(stored?.value).toBe("45");
|
||||||
|
const after = await (await GET()).json();
|
||||||
|
expect(after["floor.jobDelaySeconds"]).toBe("45");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ const DEFAULTS: Record<string, string> = {
|
|||||||
"monitor.retryIntervalHours": "6",
|
"monitor.retryIntervalHours": "6",
|
||||||
"monitor.qualityCutoff": "2",
|
"monitor.qualityCutoff": "2",
|
||||||
"monitor.upgradeWindowDays": "14",
|
"monitor.upgradeWindowDays": "14",
|
||||||
|
"floor.jobDelaySeconds": "0",
|
||||||
};
|
};
|
||||||
|
|
||||||
export async function GET() {
|
export async function GET() {
|
||||||
|
|||||||
@@ -0,0 +1,37 @@
|
|||||||
|
import { describe, it, expect } from "vitest";
|
||||||
|
import { GET, PATCH } from "./route";
|
||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
|
||||||
|
function patch(body: Record<string, string>) {
|
||||||
|
return new Request("http://localhost/api/qobuz/pacing", {
|
||||||
|
method: "PATCH",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify(body),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("qobuz pacing config API", () => {
|
||||||
|
it("returns defaults when nothing is stored", async () => {
|
||||||
|
const cfg = await (await GET()).json();
|
||||||
|
expect(cfg["qobuz.pacing.enabled"]).toBe("true");
|
||||||
|
expect(cfg["qobuz.pacing.activeStartHour"]).toBe("8");
|
||||||
|
expect(cfg["qobuz.pacing.activeEndHour"]).toBe("23");
|
||||||
|
expect(cfg["qobuz.pacing.dailyCap"]).toBe("40");
|
||||||
|
expect(cfg["qobuz.pacing.warmupStartDate"]).toBe("");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("round-trips allow-listed settings and persists them", async () => {
|
||||||
|
await PATCH(patch({ "qobuz.pacing.dailyCap": "25", "qobuz.pacing.activeStartHour": "9" }));
|
||||||
|
const stored = await prisma.config.findUnique({ where: { key: "qobuz.pacing.dailyCap" } });
|
||||||
|
expect(stored?.value).toBe("25");
|
||||||
|
const cfg = await (await GET()).json();
|
||||||
|
expect(cfg["qobuz.pacing.dailyCap"]).toBe("25");
|
||||||
|
expect(cfg["qobuz.pacing.activeStartHour"]).toBe("9");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("ignores keys not in the allow-list (e.g. worker-managed counters)", async () => {
|
||||||
|
await PATCH(patch({ "qobuz.pacing.countToday": "999", "qobuz.pacing.dailyCap": "30" }));
|
||||||
|
expect(await prisma.config.findUnique({ where: { key: "qobuz.pacing.countToday" } })).toBeNull();
|
||||||
|
expect((await prisma.config.findUnique({ where: { key: "qobuz.pacing.dailyCap" } }))?.value).toBe("30");
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
|
||||||
|
// Qobuz pacing ("budget gate") settings. The worker reads these Config keys live; state keys
|
||||||
|
// (countDay/countToday/nextAllowedAt) are worker-managed and intentionally NOT editable here.
|
||||||
|
const DEFAULTS: Record<string, string> = {
|
||||||
|
"qobuz.pacing.enabled": "true",
|
||||||
|
"qobuz.pacing.activeStartHour": "8",
|
||||||
|
"qobuz.pacing.activeEndHour": "23",
|
||||||
|
"qobuz.pacing.dailyCap": "40",
|
||||||
|
"qobuz.pacing.warmupStartDate": "",
|
||||||
|
};
|
||||||
|
|
||||||
|
export async function GET() {
|
||||||
|
const rows = await prisma.config.findMany({ where: { key: { in: Object.keys(DEFAULTS) } } });
|
||||||
|
const stored = Object.fromEntries(rows.map((r) => [r.key, r.value]));
|
||||||
|
return Response.json(
|
||||||
|
Object.fromEntries(Object.entries(DEFAULTS).map(([k, d]) => [k, stored[k] ?? d])),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function PATCH(request: Request) {
|
||||||
|
let body: unknown;
|
||||||
|
try {
|
||||||
|
body = await request.json();
|
||||||
|
} catch {
|
||||||
|
return Response.json({ error: "invalid JSON" }, { status: 400 });
|
||||||
|
}
|
||||||
|
const entries = Object.entries((body ?? {}) as Record<string, unknown>).filter(([k]) =>
|
||||||
|
Object.prototype.hasOwnProperty.call(DEFAULTS, k),
|
||||||
|
);
|
||||||
|
for (const [key, value] of entries) {
|
||||||
|
await prisma.config.upsert({
|
||||||
|
where: { key },
|
||||||
|
create: { key, value: String(value), secret: false },
|
||||||
|
update: { value: String(value) },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return Response.json({ updated: entries.length });
|
||||||
|
}
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
import { describe, it, expect } from "vitest";
|
||||||
|
import { POST } from "./route";
|
||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
import type { Prisma } from "@prisma/client";
|
||||||
|
|
||||||
|
function req(id: string, paused: boolean) {
|
||||||
|
return new Request(`http://localhost/api/requests/${id}/pause`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify({ paused }),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async function seed(jobData: Prisma.JobCreateWithoutRequestInput) {
|
||||||
|
return prisma.request.create({
|
||||||
|
data: { artist: "A", album: "B", status: "pending", job: { create: jobData } },
|
||||||
|
include: { job: true },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("per-item pause API", () => {
|
||||||
|
it("pauses a queued job", async () => {
|
||||||
|
const created = await seed({ state: "requested" });
|
||||||
|
const res = await POST(req(created.id, true), { params: Promise.resolve({ id: created.id }) });
|
||||||
|
expect(res.status).toBe(200);
|
||||||
|
const job = await prisma.job.findUnique({ where: { id: created.job!.id } });
|
||||||
|
expect(job?.paused).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("resumes a paused job", async () => {
|
||||||
|
const created = await seed({ state: "requested", paused: true });
|
||||||
|
await POST(req(created.id, false), { params: Promise.resolve({ id: created.id }) });
|
||||||
|
const job = await prisma.job.findUnique({ where: { id: created.job!.id } });
|
||||||
|
expect(job?.paused).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rejects pausing an in-flight job", async () => {
|
||||||
|
const created = await seed({ state: "downloading", paused: true });
|
||||||
|
const res = await POST(req(created.id, false), { params: Promise.resolve({ id: created.id }) });
|
||||||
|
expect(res.status).toBe(400);
|
||||||
|
const job = await prisma.job.findUnique({ where: { id: created.job!.id } });
|
||||||
|
expect(job?.paused).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("404s for a missing request", async () => {
|
||||||
|
const res = await POST(req("nope", true), { params: Promise.resolve({ id: "nope" }) });
|
||||||
|
expect(res.status).toBe(404);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
|
||||||
|
export async function POST(request: Request, { params }: { params: Promise<{ id: string }> }) {
|
||||||
|
const { id } = await params;
|
||||||
|
|
||||||
|
let body: { paused?: unknown };
|
||||||
|
try {
|
||||||
|
body = (await request.json()) as { paused?: unknown };
|
||||||
|
} catch {
|
||||||
|
return Response.json({ error: "invalid JSON" }, { status: 400 });
|
||||||
|
}
|
||||||
|
if (typeof body.paused !== "boolean") {
|
||||||
|
return Response.json({ error: "paused must be a boolean" }, { status: 400 });
|
||||||
|
}
|
||||||
|
|
||||||
|
const req = await prisma.request.findUnique({ where: { id }, include: { job: true } });
|
||||||
|
if (!req || !req.job) {
|
||||||
|
return Response.json({ error: "not found" }, { status: 404 });
|
||||||
|
}
|
||||||
|
if (req.job.state !== "requested") {
|
||||||
|
return Response.json({ error: "only queued jobs can be paused" }, { status: 400 });
|
||||||
|
}
|
||||||
|
|
||||||
|
await prisma.job.update({ where: { id: req.job.id }, data: { paused: body.paused } });
|
||||||
|
return Response.json({ ok: true, paused: body.paused });
|
||||||
|
}
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
import { describe, it, expect } from "vitest";
|
||||||
|
import { DELETE } from "./route";
|
||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
import type { Prisma } from "@prisma/client";
|
||||||
|
|
||||||
|
function del(id: string) {
|
||||||
|
return new Request(`http://localhost/api/requests/${id}`, { method: "DELETE" });
|
||||||
|
}
|
||||||
|
|
||||||
|
async function seed(jobData: Prisma.JobCreateWithoutRequestInput) {
|
||||||
|
return prisma.request.create({
|
||||||
|
data: { artist: "A", album: "B", status: "pending", job: { create: jobData } },
|
||||||
|
include: { job: true },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("remove request API", () => {
|
||||||
|
it("deletes a queued request and cascades its job", async () => {
|
||||||
|
const created = await seed({ state: "requested" });
|
||||||
|
const res = await DELETE(del(created.id), { params: Promise.resolve({ id: created.id }) });
|
||||||
|
expect(res.status).toBe(200);
|
||||||
|
expect(await prisma.request.findUnique({ where: { id: created.id } })).toBeNull();
|
||||||
|
expect(await prisma.job.findUnique({ where: { id: created.job!.id } })).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses to remove an in-flight job", async () => {
|
||||||
|
const created = await seed({ state: "downloading" });
|
||||||
|
const res = await DELETE(del(created.id), { params: Promise.resolve({ id: created.id }) });
|
||||||
|
expect(res.status).toBe(400);
|
||||||
|
expect(await prisma.request.findUnique({ where: { id: created.id } })).not.toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("404s for a missing request", async () => {
|
||||||
|
const res = await DELETE(del("nope"), { params: Promise.resolve({ id: "nope" }) });
|
||||||
|
expect(res.status).toBe(404);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
|
||||||
|
export async function DELETE(_request: Request, { params }: { params: Promise<{ id: string }> }) {
|
||||||
|
const { id } = await params;
|
||||||
|
|
||||||
|
const req = await prisma.request.findUnique({ where: { id }, include: { job: true } });
|
||||||
|
if (!req) {
|
||||||
|
return Response.json({ error: "not found" }, { status: 404 });
|
||||||
|
}
|
||||||
|
if (req.job && req.job.state !== "requested") {
|
||||||
|
return Response.json({ error: "only queued items can be removed" }, { status: 400 });
|
||||||
|
}
|
||||||
|
|
||||||
|
await prisma.request.delete({ where: { id } }); // cascades Job + Candidates
|
||||||
|
return Response.json({ ok: true });
|
||||||
|
}
|
||||||
@@ -64,6 +64,8 @@ export async function GET() {
|
|||||||
claimedAt: r.job.claimedAt,
|
claimedAt: r.job.claimedAt,
|
||||||
updatedAt: r.job.updatedAt,
|
updatedAt: r.job.updatedAt,
|
||||||
downloadProgress: r.job.downloadProgress,
|
downloadProgress: r.job.downloadProgress,
|
||||||
|
downloadEtaSeconds: r.job.downloadEtaSeconds,
|
||||||
|
paused: r.job.paused,
|
||||||
candidateCount: cands.length,
|
candidateCount: cands.length,
|
||||||
chosen: chosen ? { source: chosen.source, format: chosen.format, trackCount: chosen.trackCount } : null,
|
chosen: chosen ? { source: chosen.source, format: chosen.format, trackCount: chosen.trackCount } : null,
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -245,6 +245,9 @@ nav.contents .sep { flex: 1; }
|
|||||||
.btn.ghost { border-color: var(--rule-2); color: var(--graphite); }
|
.btn.ghost { border-color: var(--rule-2); color: var(--graphite); }
|
||||||
.btn.ghost:hover { background: transparent; border-color: var(--ink); color: var(--ink); }
|
.btn.ghost:hover { background: transparent; border-color: var(--ink); color: var(--ink); }
|
||||||
|
|
||||||
|
.press-controls { display: flex; gap: 0.5rem; align-items: center; margin: 0 0 1rem; flex-wrap: wrap; }
|
||||||
|
.item-controls { display: inline-flex; gap: 0.5rem; align-items: center; }
|
||||||
|
|
||||||
.toggle {
|
.toggle {
|
||||||
font-family: var(--mono); font-size: 0.64rem; letter-spacing: 0.1em; text-transform: uppercase;
|
font-family: var(--mono); font-size: 0.64rem; letter-spacing: 0.1em; text-transform: uppercase;
|
||||||
color: var(--graphite); display: inline-flex; align-items: center; gap: 7px; cursor: pointer;
|
color: var(--graphite); display: inline-flex; align-items: center; gap: 7px; cursor: pointer;
|
||||||
@@ -286,6 +289,20 @@ nav.contents .sep { flex: 1; }
|
|||||||
.save-row { display: flex; align-items: center; gap: 14px; margin-top: 8px; }
|
.save-row { display: flex; align-items: center; gap: 14px; margin-top: 8px; }
|
||||||
.edit-meta { display: flex; flex-direction: column; gap: 8px; width: 100%; }
|
.edit-meta { display: flex; flex-direction: column; gap: 8px; width: 100%; }
|
||||||
.rmeta.why { font-style: italic; color: var(--graphite); margin-top: 2px; }
|
.rmeta.why { font-style: italic; color: var(--graphite); margin-top: 2px; }
|
||||||
|
|
||||||
|
/* Discover unified row: artist block | compact album list (stacked) | actions.
|
||||||
|
Each album is a small horizontal item (cover + title/badge on one line) so two stack
|
||||||
|
vertically without making the row taller than the artist block. */
|
||||||
|
.disco-row { align-items: center; flex-wrap: wrap; gap: 14px 20px; }
|
||||||
|
.disco-row .main { flex: 1 1 200px; }
|
||||||
|
.disco-albums { display: flex; flex-direction: column; gap: 6px; flex: 0 1 300px; min-width: 220px; }
|
||||||
|
.disco-album { display: flex; align-items: center; gap: 8px; }
|
||||||
|
.disco-thumb { display: flex; align-items: center; gap: 8px; flex: 1; min-width: 0; background: none; border: 0; padding: 0; cursor: pointer; text-align: left; }
|
||||||
|
.disco-thumb .da-cover { width: 30px; height: 30px; flex-shrink: 0; }
|
||||||
|
.da-text { display: flex; flex-direction: column; min-width: 0; line-height: 1.15; }
|
||||||
|
.disco-album .da-title { font-size: 12px; color: var(--ink); white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
|
||||||
|
.disco-album .badge { font-family: var(--mono); font-size: 9px; text-transform: uppercase; letter-spacing: 0.04em; color: var(--graphite); }
|
||||||
|
.disco-album .btn.sm.want { padding: 1px 8px; flex-shrink: 0; }
|
||||||
.sel-tools { display: inline-flex; align-items: center; gap: 10px; }
|
.sel-tools { display: inline-flex; align-items: center; gap: 10px; }
|
||||||
.bulk-bar { display: flex; align-items: center; gap: 12px; flex-wrap: wrap; margin: 4px 0 12px; padding: 8px 12px; border: 1px solid var(--rule); background: var(--paper-2); }
|
.bulk-bar { display: flex; align-items: center; gap: 12px; flex-wrap: wrap; margin: 4px 0 12px; padding: 8px 12px; border: 1px solid var(--rule); background: var(--paper-2); }
|
||||||
.album-card { position: relative; }
|
.album-card { position: relative; }
|
||||||
@@ -369,6 +386,39 @@ nav.contents .sep { flex: 1; }
|
|||||||
.tab:hover { color: var(--ink); }
|
.tab:hover { color: var(--ink); }
|
||||||
.tab.active { color: var(--ink); border-bottom-color: var(--accent); }
|
.tab.active { color: var(--ink); border-bottom-color: var(--accent); }
|
||||||
.tab .n { color: var(--rule-2); margin-left: 6px; }
|
.tab .n { color: var(--rule-2); margin-left: 6px; }
|
||||||
|
.tab.has-errors .n { color: var(--alert); }
|
||||||
|
|
||||||
|
/* "Next up" badge on the first queued row (the item the worker claims next) */
|
||||||
|
.nextup {
|
||||||
|
display: inline-block; vertical-align: middle; margin-right: 9px; transform: translateY(-2px);
|
||||||
|
font-family: var(--mono); font-size: 0.56rem; letter-spacing: 0.13em; text-transform: uppercase;
|
||||||
|
color: var(--accent-ink); background: var(--accent); padding: 2px 7px; border-radius: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ── Kebab menu (⋮ popup for press actions) ───────────── */
|
||||||
|
.dept.has-actions { align-items: center; }
|
||||||
|
.dept.has-actions .fill { transform: none; }
|
||||||
|
.menu { position: relative; display: inline-flex; }
|
||||||
|
.menu-trigger {
|
||||||
|
font-size: 1.15rem; line-height: 1; color: var(--graphite); background: transparent;
|
||||||
|
border: 1px solid var(--rule); border-radius: 6px; width: 32px; height: 30px; cursor: pointer;
|
||||||
|
display: inline-flex; align-items: center; justify-content: center;
|
||||||
|
}
|
||||||
|
.menu-trigger:hover { color: var(--ink); border-color: var(--rule-2); }
|
||||||
|
.menu-pop {
|
||||||
|
position: absolute; top: calc(100% + 6px); right: 0; z-index: 30; min-width: 190px;
|
||||||
|
background: var(--paper-2); border: 1px solid var(--rule-2); border-radius: 9px;
|
||||||
|
box-shadow: 0 10px 30px rgba(0, 0, 0, 0.22); padding: 5px;
|
||||||
|
display: flex; flex-direction: column; gap: 1px;
|
||||||
|
}
|
||||||
|
.menu-item {
|
||||||
|
font-family: var(--mono); font-size: 0.68rem; letter-spacing: 0.1em; text-transform: uppercase;
|
||||||
|
text-align: left; background: transparent; border: 0; color: var(--ink);
|
||||||
|
padding: 10px 11px; border-radius: 6px; cursor: pointer; white-space: nowrap;
|
||||||
|
}
|
||||||
|
.menu-item:hover { background: var(--rule); }
|
||||||
|
.menu-item:disabled { color: var(--rule-2); cursor: default; background: transparent; }
|
||||||
|
.menu-item.danger { color: var(--alert); }
|
||||||
|
|
||||||
/* ── Discography row (cover thumb + click-to-modal) ───── */
|
/* ── Discography row (cover thumb + click-to-modal) ───── */
|
||||||
.disco-open {
|
.disco-open {
|
||||||
|
|||||||
@@ -4,23 +4,28 @@ import { useCallback, useEffect, useState } from "react";
|
|||||||
import { PageHead } from "../_ui/page-head";
|
import { PageHead } from "../_ui/page-head";
|
||||||
import { timeAgo } from "../_ui/status";
|
import { timeAgo } from "../_ui/status";
|
||||||
import { SectionHeader } from "../_ui/section-header";
|
import { SectionHeader } from "../_ui/section-header";
|
||||||
import { ArtistModal } from "../_ui/artist-modal";
|
|
||||||
import { AlbumModal } from "../_ui/album-modal";
|
import { AlbumModal } from "../_ui/album-modal";
|
||||||
import { CoverArt } from "../_ui/cover-art";
|
import { CoverArt } from "../_ui/cover-art";
|
||||||
import { toast } from "../_ui/toast";
|
import { toast } from "../_ui/toast";
|
||||||
|
|
||||||
type Suggestion = {
|
type Album = {
|
||||||
id: string;
|
id: string;
|
||||||
artistMbid: string;
|
|
||||||
artistName: string;
|
|
||||||
rgMbid: string | null;
|
rgMbid: string | null;
|
||||||
album: string | null;
|
album: string | null;
|
||||||
primaryType: string | null;
|
primaryType: string | null;
|
||||||
secondaryTypes: string[];
|
secondaryTypes: string[];
|
||||||
|
firstReleaseDate: string | null;
|
||||||
|
albumReason: string | null;
|
||||||
|
};
|
||||||
|
type Row = {
|
||||||
|
id: string | null;
|
||||||
|
artistMbid: string;
|
||||||
|
artistName: string;
|
||||||
score: number;
|
score: number;
|
||||||
seedCount: number;
|
seedCount: number;
|
||||||
sources: string[];
|
sources: string[];
|
||||||
seeds: string[];
|
seeds: string[];
|
||||||
|
albums: Album[];
|
||||||
};
|
};
|
||||||
|
|
||||||
/** "Similar to Radiohead, Coldplay +2" — the followed artists that surfaced this suggestion. */
|
/** "Similar to Radiohead, Coldplay +2" — the followed artists that surfaced this suggestion. */
|
||||||
@@ -31,13 +36,18 @@ function similarTo(seeds: string[]): string | null {
|
|||||||
return `Similar to ${shown}${extra > 0 ? ` +${extra} more` : ""}`;
|
return `Similar to ${shown}${extra > 0 ? ` +${extra} more` : ""}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
type Tab = "Artists" | "Albums";
|
/** Full-length albums only — hide singles/EPs and secondary-type releases (live/comp) that
|
||||||
|
* pre-date the albums-only derivation and still linger in the DB. */
|
||||||
|
function isFullAlbum(a: Album): boolean {
|
||||||
|
return (a.primaryType ?? "Album") === "Album" && (a.secondaryTypes?.length ?? 0) === 0;
|
||||||
|
}
|
||||||
|
|
||||||
/** Only full-length albums belong in Suggested albums — hide singles/EPs and
|
/** Human badge for why an album is surfaced. */
|
||||||
* secondary-type releases (live/comp). New sweeps already filter server-side
|
function badgeFor(reason: string | null): string | null {
|
||||||
* (discover.albumsOnly); this also sweeps any pre-filter rows still in the DB. */
|
if (reason === "newest") return "Newest";
|
||||||
function isFullAlbum(s: Suggestion): boolean {
|
if (reason === "popular") return "Most played";
|
||||||
return (s.primaryType ?? "Album") === "Album" && (s.secondaryTypes?.length ?? 0) === 0;
|
if (reason === "newest,popular") return "Newest · Most played";
|
||||||
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** "just now" / "3h ago" / "2d ago" — timeAgo has no suffix, so add one (except "just now"). */
|
/** "just now" / "3h ago" / "2d ago" — timeAgo has no suffix, so add one (except "just now"). */
|
||||||
@@ -46,33 +56,29 @@ function agoLabel(value: string): string {
|
|||||||
return !a || a === "just now" ? a || "recently" : `${a} ago`;
|
return !a || a === "just now" ? a || "recently" : `${a} ago`;
|
||||||
}
|
}
|
||||||
|
|
||||||
type Feed = { artists: Suggestion[]; albums: Suggestion[] };
|
|
||||||
type SearchResult = {
|
type SearchResult = {
|
||||||
seed: { mbid: string; name: string } | null;
|
seed: { mbid: string; name: string } | null;
|
||||||
similar: { mbid: string; name: string; score: number }[];
|
similar: { mbid: string; name: string; score: number }[];
|
||||||
};
|
};
|
||||||
|
|
||||||
export function DiscoverClient() {
|
export function DiscoverClient() {
|
||||||
const [feed, setFeed] = useState<Feed>({ artists: [], albums: [] });
|
const [rows, setRows] = useState<Row[]>([]);
|
||||||
const [result, setResult] = useState<string | null>(null);
|
const [result, setResult] = useState<string | null>(null);
|
||||||
const [lastRunAt, setLastRunAt] = useState<string | null>(null);
|
const [lastRunAt, setLastRunAt] = useState<string | null>(null);
|
||||||
const [running, setRunning] = useState(false);
|
const [running, setRunning] = useState(false);
|
||||||
const [query, setQuery] = useState("");
|
const [query, setQuery] = useState("");
|
||||||
const [search, setSearch] = useState<SearchResult | null>(null);
|
const [search, setSearch] = useState<SearchResult | null>(null);
|
||||||
const [busy, setBusy] = useState(false);
|
const [busy, setBusy] = useState(false);
|
||||||
|
const [confirmReset, setConfirmReset] = useState(false);
|
||||||
const [followedSimilar, setFollowedSimilar] = useState<Set<string>>(new Set());
|
const [followedSimilar, setFollowedSimilar] = useState<Set<string>>(new Set());
|
||||||
const [artistModal, setArtistModal] = useState<{ mbid: string; name: string } | null>(null);
|
const [albumModal, setAlbumModal] = useState<{ a: Album; artistName: string; artistMbid: string } | null>(null);
|
||||||
const [albumModal, setAlbumModal] = useState<Suggestion | null>(null);
|
|
||||||
const [tab, setTab] = useState<Tab>("Artists");
|
|
||||||
|
|
||||||
const albums = feed.albums.filter(isFullAlbum);
|
const loadRows = useCallback(async () => {
|
||||||
|
setRows((await (await fetch("/api/discover")).json()).rows ?? []);
|
||||||
const loadFeed = useCallback(async () => {
|
|
||||||
setFeed(await (await fetch("/api/discover")).json());
|
|
||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
loadFeed();
|
loadRows();
|
||||||
fetch("/api/discover/run")
|
fetch("/api/discover/run")
|
||||||
.then((r) => r.json())
|
.then((r) => r.json())
|
||||||
.then((s: { result: string | null; requested: boolean; inProgress: boolean; lastRunAt: string | null }) => {
|
.then((s: { result: string | null; requested: boolean; inProgress: boolean; lastRunAt: string | null }) => {
|
||||||
@@ -80,7 +86,7 @@ export function DiscoverClient() {
|
|||||||
setLastRunAt(s.lastRunAt);
|
setLastRunAt(s.lastRunAt);
|
||||||
setRunning(s.requested || s.inProgress);
|
setRunning(s.requested || s.inProgress);
|
||||||
});
|
});
|
||||||
}, [loadFeed]);
|
}, [loadRows]);
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (!running) return;
|
if (!running) return;
|
||||||
@@ -89,19 +95,30 @@ export function DiscoverClient() {
|
|||||||
setResult(s.result);
|
setResult(s.result);
|
||||||
setLastRunAt(s.lastRunAt);
|
setLastRunAt(s.lastRunAt);
|
||||||
if (!s.requested && !s.inProgress) {
|
if (!s.requested && !s.inProgress) {
|
||||||
// the sweep finished (neither queued nor in progress) — refresh the feed
|
|
||||||
setRunning(false);
|
setRunning(false);
|
||||||
loadFeed();
|
loadRows();
|
||||||
}
|
}
|
||||||
}, 2000);
|
}, 2000);
|
||||||
return () => clearInterval(id);
|
return () => clearInterval(id);
|
||||||
}, [running, loadFeed]);
|
}, [running, loadRows]);
|
||||||
|
|
||||||
async function discoverNow() {
|
async function discoverNow() {
|
||||||
await fetch("/api/discover/run", { method: "POST" });
|
await fetch("/api/discover/run", { method: "POST" });
|
||||||
setRunning(true);
|
setRunning(true);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function resetDiscover() {
|
||||||
|
setConfirmReset(false);
|
||||||
|
const res = await fetch("/api/discover/reset", { method: "POST" }).catch(() => null);
|
||||||
|
if (res && (res.ok || res.status === 202)) {
|
||||||
|
setRows([]); // clears immediately; the rebuild repopulates
|
||||||
|
setRunning(true);
|
||||||
|
toast("Rebuilding suggestions from scratch…");
|
||||||
|
} else {
|
||||||
|
toast("Couldn't reset");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function runSearch(e: React.FormEvent) {
|
async function runSearch(e: React.FormEvent) {
|
||||||
e.preventDefault();
|
e.preventDefault();
|
||||||
if (!query.trim()) return;
|
if (!query.trim()) return;
|
||||||
@@ -113,18 +130,26 @@ export function DiscoverClient() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async function followSimilar(a: { mbid: string; name: string }) {
|
async function followByMbid(mbid: string, name: string) {
|
||||||
const res = await fetch("/api/artists", {
|
const res = await fetch("/api/artists", {
|
||||||
method: "POST",
|
method: "POST",
|
||||||
headers: { "content-type": "application/json" },
|
headers: { "content-type": "application/json" },
|
||||||
body: JSON.stringify({ mbid: a.mbid, name: a.name }),
|
body: JSON.stringify({ mbid, name }),
|
||||||
});
|
}).catch(() => null);
|
||||||
if (res.ok || res.status === 409) {
|
return !!res && (res.ok || res.status === 409);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function followSimilar(a: { mbid: string; name: string }) {
|
||||||
|
if (await followByMbid(a.mbid, a.name)) {
|
||||||
setFollowedSimilar((s) => new Set(s).add(a.mbid));
|
setFollowedSimilar((s) => new Set(s).add(a.mbid));
|
||||||
toast(`Following ${a.name}`);
|
toast(`Following ${a.name}`);
|
||||||
|
} else {
|
||||||
|
toast(`Couldn't follow ${a.name}`);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Suggestion actions key off a suggestion id. A row with no artist-suggestion id (album-only,
|
||||||
|
// e.g. the artist was dismissed earlier) falls back to a plain follow-by-MBID.
|
||||||
async function act(id: string, action: "follow" | "want" | "dismiss") {
|
async function act(id: string, action: "follow" | "want" | "dismiss") {
|
||||||
const res = await fetch(`/api/discover/${id}`, {
|
const res = await fetch(`/api/discover/${id}`, {
|
||||||
method: "POST",
|
method: "POST",
|
||||||
@@ -136,7 +161,17 @@ export function DiscoverClient() {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
toast(action === "follow" ? "Following" : action === "want" ? "Added to wanted" : "Dismissed");
|
toast(action === "follow" ? "Following" : action === "want" ? "Added to wanted" : "Dismissed");
|
||||||
loadFeed();
|
loadRows();
|
||||||
|
}
|
||||||
|
|
||||||
|
async function followRow(row: Row) {
|
||||||
|
if (row.id) return act(row.id, "follow");
|
||||||
|
if (await followByMbid(row.artistMbid, row.artistName)) {
|
||||||
|
toast(`Following ${row.artistName}`);
|
||||||
|
loadRows();
|
||||||
|
} else {
|
||||||
|
toast(`Couldn't follow ${row.artistName}`);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return (
|
return (
|
||||||
@@ -147,13 +182,26 @@ export function DiscoverClient() {
|
|||||||
<button className="btn accent" onClick={discoverNow} disabled={running}>
|
<button className="btn accent" onClick={discoverNow} disabled={running}>
|
||||||
{running ? "Discovering…" : "Discover now"}
|
{running ? "Discovering…" : "Discover now"}
|
||||||
</button>
|
</button>
|
||||||
<span className="rmeta">
|
{confirmReset ? (
|
||||||
{running
|
<>
|
||||||
? "Sweeping your followed artists for new suggestions…"
|
<span className="rmeta">Clear all suggestions and rebuild from scratch?</span>
|
||||||
: lastRunAt
|
<button className="btn sm accent" onClick={resetDiscover}>Reset</button>
|
||||||
? `Last discovered ${agoLabel(lastRunAt)}${result ? ` · ${result}` : ""}`
|
<button className="btn sm ghost" onClick={() => setConfirmReset(false)}>Cancel</button>
|
||||||
: "Not yet run — press Discover now to sweep your followed artists"}
|
</>
|
||||||
</span>
|
) : (
|
||||||
|
<>
|
||||||
|
<button className="btn sm ghost" onClick={() => setConfirmReset(true)} disabled={running}>
|
||||||
|
Reset
|
||||||
|
</button>
|
||||||
|
<span className="rmeta">
|
||||||
|
{running
|
||||||
|
? "Sweeping your followed artists for new suggestions…"
|
||||||
|
: lastRunAt
|
||||||
|
? `Last discovered ${agoLabel(lastRunAt)}${result ? ` · ${result}` : ""}`
|
||||||
|
: "Not yet run — press Discover now to sweep your followed artists"}
|
||||||
|
</span>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<SectionHeader title="Find similar" />
|
<SectionHeader title="Find similar" />
|
||||||
@@ -173,9 +221,7 @@ export function DiscoverClient() {
|
|||||||
<li key={s.mbid} className="list-row">
|
<li key={s.mbid} className="list-row">
|
||||||
<div className="main">
|
<div className="main">
|
||||||
<div className="rtitle">
|
<div className="rtitle">
|
||||||
<button className="linkish" onClick={() => setArtistModal({ mbid: s.mbid, name: s.name })}>
|
<a href={`/discover/artist/${s.mbid}?name=${encodeURIComponent(s.name)}`}>{s.name}</a>
|
||||||
{s.name}
|
|
||||||
</button>
|
|
||||||
</div>
|
</div>
|
||||||
<div className="rmeta">
|
<div className="rmeta">
|
||||||
<span className="score">similarity {s.score.toFixed(1)}</span>
|
<span className="score">similarity {s.score.toFixed(1)}</span>
|
||||||
@@ -195,96 +241,85 @@ export function DiscoverClient() {
|
|||||||
</ul>
|
</ul>
|
||||||
) : null}
|
) : null}
|
||||||
|
|
||||||
<div className="tabs">
|
<SectionHeader title="Suggested artists" note={`${rows.length}`} />
|
||||||
<button className={`tab${tab === "Artists" ? " active" : ""}`} onClick={() => setTab("Artists")}>
|
{rows.length === 0 ? (
|
||||||
Suggested artists
|
|
||||||
<span className="n">{feed.artists.length}</span>
|
|
||||||
</button>
|
|
||||||
<button className={`tab${tab === "Albums" ? " active" : ""}`} onClick={() => setTab("Albums")}>
|
|
||||||
Suggested albums
|
|
||||||
<span className="n">{albums.length}</span>
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
{tab === "Artists" ? (
|
|
||||||
feed.artists.length === 0 ? (
|
|
||||||
<p className="muted-note">No suggestions yet — run Discover, or follow a few artists to seed it.</p>
|
<p className="muted-note">No suggestions yet — run Discover, or follow a few artists to seed it.</p>
|
||||||
) : (
|
) : (
|
||||||
<ul className="list">
|
<ul className="list">
|
||||||
{feed.artists.map((s) => (
|
{rows.map((row) => {
|
||||||
<li key={s.id} className="list-row">
|
const albums = row.albums.filter(isFullAlbum);
|
||||||
|
return (
|
||||||
|
<li key={row.artistMbid} className="list-row disco-row">
|
||||||
<div className="main">
|
<div className="main">
|
||||||
<div className="rtitle">
|
<div className="rtitle">
|
||||||
<button className="linkish" onClick={() => setArtistModal({ mbid: s.artistMbid, name: s.artistName })}>
|
<a href={`/discover/artist/${row.artistMbid}?name=${encodeURIComponent(row.artistName)}`}>
|
||||||
{s.artistName}
|
{row.artistName}
|
||||||
</button>
|
</a>
|
||||||
</div>
|
</div>
|
||||||
<div className="rmeta">
|
<div className="rmeta">
|
||||||
<span className="score">score {s.score.toFixed(2)}</span>
|
<span className="score">score {row.score.toFixed(2)}</span>
|
||||||
<span className="dot">·</span>
|
<span className="dot">·</span>
|
||||||
<span>{s.seedCount} seed{s.seedCount === 1 ? "" : "s"}</span>
|
<span>{row.sources.join(", ")}</span>
|
||||||
<span className="dot">·</span>
|
|
||||||
<span>{s.sources.join(", ")}</span>
|
|
||||||
</div>
|
</div>
|
||||||
{similarTo(s.seeds) ? <div className="rmeta why">{similarTo(s.seeds)}</div> : null}
|
{similarTo(row.seeds) ? <div className="rmeta why">{similarTo(row.seeds)}</div> : null}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{albums.length > 0 ? (
|
||||||
|
<div className="disco-albums">
|
||||||
|
{albums.map((a) => (
|
||||||
|
<div key={a.id} className="disco-album">
|
||||||
|
<button
|
||||||
|
className="disco-thumb"
|
||||||
|
onClick={() => setAlbumModal({ a, artistName: row.artistName, artistMbid: row.artistMbid })}
|
||||||
|
aria-label={`Open ${a.album ?? ""}`}
|
||||||
|
>
|
||||||
|
<CoverArt rgMbid={a.rgMbid} alt={a.album ?? ""} className="da-cover" />
|
||||||
|
<span className="da-text">
|
||||||
|
<span className="da-title">{a.album}</span>
|
||||||
|
{badgeFor(a.albumReason) ? <span className="badge">{badgeFor(a.albumReason)}</span> : null}
|
||||||
|
</span>
|
||||||
|
</button>
|
||||||
|
<button className="btn sm ghost want" onClick={() => act(a.id, "want")}>
|
||||||
|
Want
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
|
||||||
<div className="actions">
|
<div className="actions">
|
||||||
<button className="btn sm" onClick={() => act(s.id, "follow")}>
|
<button className="btn sm" onClick={() => followRow(row)}>
|
||||||
Follow
|
Follow
|
||||||
</button>
|
</button>
|
||||||
<button className="btn sm ghost" onClick={() => act(s.id, "dismiss")}>
|
{row.id ? (
|
||||||
Dismiss
|
<button className="btn sm ghost" onClick={() => act(row.id!, "dismiss")}>
|
||||||
</button>
|
Dismiss
|
||||||
|
</button>
|
||||||
|
) : null}
|
||||||
</div>
|
</div>
|
||||||
</li>
|
</li>
|
||||||
))}
|
);
|
||||||
</ul>
|
})}
|
||||||
)
|
|
||||||
) : albums.length === 0 ? (
|
|
||||||
<p className="muted-note">No album suggestions yet.</p>
|
|
||||||
) : (
|
|
||||||
<ul className="list">
|
|
||||||
{albums.map((s) => (
|
|
||||||
<li key={s.id} className="list-row">
|
|
||||||
<div className="main">
|
|
||||||
<button className="disco-open" onClick={() => setAlbumModal(s)} aria-label={`Open ${s.album ?? ""}`}>
|
|
||||||
<CoverArt rgMbid={s.rgMbid} alt={s.album ?? ""} className="thumb" />
|
|
||||||
<span>
|
|
||||||
<span className="rtitle">
|
|
||||||
{s.album} <span className="artist">· {s.artistName}</span>
|
|
||||||
</span>
|
|
||||||
<span className="rmeta">
|
|
||||||
<span className="score">score {s.score.toFixed(2)}</span>
|
|
||||||
</span>
|
|
||||||
{similarTo(s.seeds) ? <span className="rmeta why">{similarTo(s.seeds)}</span> : null}
|
|
||||||
</span>
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
<div className="actions">
|
|
||||||
<button className="btn sm" onClick={() => act(s.id, "want")}>
|
|
||||||
Want
|
|
||||||
</button>
|
|
||||||
<button className="btn sm ghost" onClick={() => act(s.id, "dismiss")}>
|
|
||||||
Dismiss
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
</li>
|
|
||||||
))}
|
|
||||||
</ul>
|
</ul>
|
||||||
)}
|
)}
|
||||||
{artistModal ? (
|
|
||||||
<ArtistModal open onClose={() => setArtistModal(null)} mbid={artistModal.mbid} initialName={artistModal.name} />
|
|
||||||
) : null}
|
|
||||||
{albumModal ? (
|
{albumModal ? (
|
||||||
<AlbumModal
|
<AlbumModal
|
||||||
open
|
open
|
||||||
onClose={() => setAlbumModal(null)}
|
onClose={() => setAlbumModal(null)}
|
||||||
album={{ rgMbid: albumModal.rgMbid, album: albumModal.album ?? "", artist: albumModal.artistName, year: null, type: null, artistMbid: albumModal.artistMbid }}
|
album={{
|
||||||
|
rgMbid: albumModal.a.rgMbid,
|
||||||
|
album: albumModal.a.album ?? "",
|
||||||
|
artist: albumModal.artistName,
|
||||||
|
year: albumModal.a.firstReleaseDate?.slice(0, 4) ?? null,
|
||||||
|
type: albumModal.a.primaryType,
|
||||||
|
artistMbid: albumModal.artistMbid,
|
||||||
|
}}
|
||||||
actions={
|
actions={
|
||||||
<button
|
<button
|
||||||
className="btn sm"
|
className="btn sm"
|
||||||
onClick={() => {
|
onClick={() => {
|
||||||
act(albumModal.id, "want");
|
act(albumModal.a.id, "want");
|
||||||
setAlbumModal(null);
|
setAlbumModal(null);
|
||||||
}}
|
}}
|
||||||
>
|
>
|
||||||
|
|||||||
Binary file not shown.
Binary file not shown.
@@ -31,6 +31,7 @@ const MONITOR_FIELDS: [string, string][] = [
|
|||||||
["monitor.retryIntervalHours", "Retry interval (hours)"],
|
["monitor.retryIntervalHours", "Retry interval (hours)"],
|
||||||
["monitor.qualityCutoff", "Quality cutoff"],
|
["monitor.qualityCutoff", "Quality cutoff"],
|
||||||
["monitor.upgradeWindowDays", "Upgrade window (days)"],
|
["monitor.upgradeWindowDays", "Upgrade window (days)"],
|
||||||
|
["floor.jobDelaySeconds", "Delay between downloads (seconds)"],
|
||||||
];
|
];
|
||||||
|
|
||||||
export function SettingsForm() {
|
export function SettingsForm() {
|
||||||
@@ -53,6 +54,8 @@ export function SettingsForm() {
|
|||||||
const [discoverSaved, setDiscoverSaved] = useState(false);
|
const [discoverSaved, setDiscoverSaved] = useState(false);
|
||||||
const [monitorCfg, setMonitorCfg] = useState<Record<string, string>>({});
|
const [monitorCfg, setMonitorCfg] = useState<Record<string, string>>({});
|
||||||
const [monitorSaved, setMonitorSaved] = useState(false);
|
const [monitorSaved, setMonitorSaved] = useState(false);
|
||||||
|
const [pacingCfg, setPacingCfg] = useState<Record<string, string>>({});
|
||||||
|
const [pacingSaved, setPacingSaved] = useState(false);
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
fetch("/api/scan")
|
fetch("/api/scan")
|
||||||
@@ -105,6 +108,12 @@ export function SettingsForm() {
|
|||||||
.then((c: Record<string, string>) => setMonitorCfg(c));
|
.then((c: Record<string, string>) => setMonitorCfg(c));
|
||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
fetch("/api/qobuz/pacing")
|
||||||
|
.then((r) => r.json())
|
||||||
|
.then((c: Record<string, string>) => setPacingCfg(c));
|
||||||
|
}, []);
|
||||||
|
|
||||||
async function saveCredentials(e: React.FormEvent) {
|
async function saveCredentials(e: React.FormEvent) {
|
||||||
e.preventDefault();
|
e.preventDefault();
|
||||||
await fetch("/api/config", {
|
await fetch("/api/config", {
|
||||||
@@ -159,6 +168,21 @@ export function SettingsForm() {
|
|||||||
setTimeout(() => setMonitorSaved(false), 1500);
|
setTimeout(() => setMonitorSaved(false), 1500);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function setPacing(key: string, value: string) {
|
||||||
|
setPacingCfg((c) => ({ ...c, [key]: value }));
|
||||||
|
}
|
||||||
|
|
||||||
|
async function savePacing(e: React.FormEvent) {
|
||||||
|
e.preventDefault();
|
||||||
|
await fetch("/api/qobuz/pacing", {
|
||||||
|
method: "PATCH",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify(pacingCfg),
|
||||||
|
});
|
||||||
|
setPacingSaved(true);
|
||||||
|
setTimeout(() => setPacingSaved(false), 1500);
|
||||||
|
}
|
||||||
|
|
||||||
const secretHint = (set: boolean) => (set ? <span className="set"> · set</span> : null);
|
const secretHint = (set: boolean) => (set ? <span className="set"> · set</span> : null);
|
||||||
|
|
||||||
async function clearField(field: "qobuzToken" | "slskdApiKey" | "lastfmApiKey") {
|
async function clearField(field: "qobuzToken" | "slskdApiKey" | "lastfmApiKey") {
|
||||||
@@ -192,6 +216,7 @@ export function SettingsForm() {
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
{tab === "Qobuz" ? (
|
{tab === "Qobuz" ? (
|
||||||
|
<>
|
||||||
<form className="settings-section" onSubmit={saveCredentials}>
|
<form className="settings-section" onSubmit={saveCredentials}>
|
||||||
<div className="subhead">
|
<div className="subhead">
|
||||||
Auth token
|
Auth token
|
||||||
@@ -234,6 +259,55 @@ export function SettingsForm() {
|
|||||||
{saved ? <span className="saved-note">Saved</span> : null}
|
{saved ? <span className="saved-note">Saved</span> : null}
|
||||||
</div>
|
</div>
|
||||||
</form>
|
</form>
|
||||||
|
<form className="settings-section" onSubmit={savePacing}>
|
||||||
|
<div className="subhead">
|
||||||
|
Pacing
|
||||||
|
<HelpButton title="Qobuz pacing">
|
||||||
|
Throttles Qobuz to a human-looking volume and cadence so the account isn't flagged
|
||||||
|
for bulk downloading. Within active hours it downloads up to the daily cap, spaced out
|
||||||
|
with randomized gaps; when throttled, albums fall through to Soulseek/YouTube and
|
||||||
|
upgrade to Qobuz hi-res later. The daily cap ramps up over the first two weeks from the
|
||||||
|
new-account start date.
|
||||||
|
</HelpButton>
|
||||||
|
</div>
|
||||||
|
<label className="field-inline">
|
||||||
|
<input type="checkbox" aria-label="pace qobuz downloads"
|
||||||
|
checked={pacingCfg["qobuz.pacing.enabled"] !== "false"}
|
||||||
|
onChange={(e) => setPacing("qobuz.pacing.enabled", e.target.checked ? "true" : "false")} />
|
||||||
|
<span>Pace Qobuz downloads</span>
|
||||||
|
</label>
|
||||||
|
<div className="field-grid">
|
||||||
|
<label className="field">
|
||||||
|
<span>Active hours — start (0–24)</span>
|
||||||
|
<input type="number" min={0} max={24} aria-label="qobuz active start hour"
|
||||||
|
value={pacingCfg["qobuz.pacing.activeStartHour"] ?? ""}
|
||||||
|
onChange={(e) => setPacing("qobuz.pacing.activeStartHour", e.target.value)} />
|
||||||
|
</label>
|
||||||
|
<label className="field">
|
||||||
|
<span>Active hours — end (0–24)</span>
|
||||||
|
<input type="number" min={0} max={24} aria-label="qobuz active end hour"
|
||||||
|
value={pacingCfg["qobuz.pacing.activeEndHour"] ?? ""}
|
||||||
|
onChange={(e) => setPacing("qobuz.pacing.activeEndHour", e.target.value)} />
|
||||||
|
</label>
|
||||||
|
<label className="field">
|
||||||
|
<span>Daily cap (steady)</span>
|
||||||
|
<input type="number" min={0} aria-label="qobuz daily cap"
|
||||||
|
value={pacingCfg["qobuz.pacing.dailyCap"] ?? ""}
|
||||||
|
onChange={(e) => setPacing("qobuz.pacing.dailyCap", e.target.value)} />
|
||||||
|
</label>
|
||||||
|
<label className="field">
|
||||||
|
<span>New-account start (warm-up)</span>
|
||||||
|
<input type="date" aria-label="qobuz warmup start date"
|
||||||
|
value={pacingCfg["qobuz.pacing.warmupStartDate"] ?? ""}
|
||||||
|
onChange={(e) => setPacing("qobuz.pacing.warmupStartDate", e.target.value)} />
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
<div className="save-row">
|
||||||
|
<button type="submit" className="btn accent">Save</button>
|
||||||
|
{pacingSaved ? <span className="saved-note">Saved</span> : null}
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</>
|
||||||
) : null}
|
) : null}
|
||||||
|
|
||||||
{tab === "Soulseek" ? (
|
{tab === "Soulseek" ? (
|
||||||
|
|||||||
@@ -0,0 +1,46 @@
|
|||||||
|
import { describe, it, expect } from "vitest";
|
||||||
|
import { resolveArtistChoice, type ArtistHit } from "./artist-resolve";
|
||||||
|
|
||||||
|
const US: ArtistHit = { mbid: "us", name: "Looking Glass", disambiguation: "70's US pop group" };
|
||||||
|
const AU: ArtistHit = { mbid: "au", name: "Looking Glass", disambiguation: "Australian stoner rock" };
|
||||||
|
|
||||||
|
describe("resolveArtistChoice", () => {
|
||||||
|
it("is ambiguous when 2+ candidates share the exact name", () => {
|
||||||
|
expect(resolveArtistChoice("Looking Glass", [US, AU], "au")).toEqual({
|
||||||
|
kind: "ambiguous",
|
||||||
|
candidates: [US, AU],
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("uses the single exact-name match even over a disagreeing fallback mbid", () => {
|
||||||
|
const hit: ArtistHit = { mbid: "r", name: "Radiohead", disambiguation: "" };
|
||||||
|
expect(resolveArtistChoice("Radiohead", [hit], "wrong-lastfm-mbid")).toEqual({
|
||||||
|
kind: "single",
|
||||||
|
mbid: "r",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("matches exact name case- and whitespace-insensitively", () => {
|
||||||
|
expect(resolveArtistChoice(" looking GLASS ", [US, AU])).toEqual({
|
||||||
|
kind: "ambiguous",
|
||||||
|
candidates: [US, AU],
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("falls back to the provided mbid when no candidate name matches exactly", () => {
|
||||||
|
const fuzzy: ArtistHit = { mbid: "x", name: "The Beatles", disambiguation: "" };
|
||||||
|
expect(resolveArtistChoice("Beatles", [fuzzy], "lastfm-mbid")).toEqual({
|
||||||
|
kind: "single",
|
||||||
|
mbid: "lastfm-mbid",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("falls back to MB's top hit when there is no exact match and no fallback", () => {
|
||||||
|
const fuzzy: ArtistHit = { mbid: "top", name: "The Beatles", disambiguation: "" };
|
||||||
|
expect(resolveArtistChoice("Beatles", [fuzzy])).toEqual({ kind: "single", mbid: "top" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is none when there are no candidates and no fallback", () => {
|
||||||
|
expect(resolveArtistChoice("Nobody Here", [])).toEqual({ kind: "none" });
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
export type ArtistHit = { mbid: string; name: string; disambiguation: string };
|
||||||
|
|
||||||
|
export type Resolution =
|
||||||
|
| { kind: "single"; mbid: string }
|
||||||
|
| { kind: "ambiguous"; candidates: ArtistHit[] }
|
||||||
|
| { kind: "none" };
|
||||||
|
|
||||||
|
const norm = (s: string) => s.trim().toLowerCase();
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Decide which MusicBrainz artist a Last.fm row refers to, given MB's search candidates.
|
||||||
|
*
|
||||||
|
* The bug this guards against: Last.fm's own MBID for an ambiguous name can point at the
|
||||||
|
* wrong same-named band (it mapped "Looking Glass" to an Australian stoner band, not the US
|
||||||
|
* pop group). So we prefer MB's own resolution:
|
||||||
|
* - 2+ candidates whose name EXACTLY matches (normalized) → ambiguous; let the user pick.
|
||||||
|
* - exactly 1 exact-name match → use it, even if it disagrees with `fallbackMbid`.
|
||||||
|
* - no exact-name match → keep prior behavior: the Last.fm `fallbackMbid`, else MB's top hit.
|
||||||
|
*
|
||||||
|
* `candidates` is assumed to arrive in MB relevance order (best first), so `candidates[0]`
|
||||||
|
* is the suggested default and exact-name matches keep that ordering.
|
||||||
|
*/
|
||||||
|
export function resolveArtistChoice(
|
||||||
|
name: string,
|
||||||
|
candidates: ArtistHit[],
|
||||||
|
fallbackMbid?: string | null,
|
||||||
|
): Resolution {
|
||||||
|
const exact = candidates.filter((a) => norm(a.name) === norm(name));
|
||||||
|
if (exact.length >= 2) return { kind: "ambiguous", candidates: exact };
|
||||||
|
if (exact.length === 1) return { kind: "single", mbid: exact[0].mbid };
|
||||||
|
if (fallbackMbid) return { kind: "single", mbid: fallbackMbid };
|
||||||
|
if (candidates.length > 0) return { kind: "single", mbid: candidates[0].mbid };
|
||||||
|
return { kind: "none" };
|
||||||
|
}
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import time
|
||||||
from difflib import SequenceMatcher
|
from difflib import SequenceMatcher
|
||||||
|
|
||||||
from lyra_worker.types import MBTarget
|
from lyra_worker.types import MBTarget
|
||||||
@@ -5,10 +6,104 @@ from lyra_worker.types import MBTarget
|
|||||||
_MIN_TITLE_RATIO = 0.6
|
_MIN_TITLE_RATIO = 0.6
|
||||||
|
|
||||||
|
|
||||||
|
def _with_retry(fn, *, attempts: int = 4, base_delay: float = 1.0, sleep=time.sleep):
|
||||||
|
"""Call `fn`, retrying transient MusicBrainz failures with exponential backoff.
|
||||||
|
|
||||||
|
A dropped TLS connection to the MusicBrainz webservice surfaces as
|
||||||
|
musicbrainzngs.NetworkError (`<urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING]>`),
|
||||||
|
and MusicBrainz's own 503 rate-limit surfaces as ResponseError; both are
|
||||||
|
transient, so we retry rather than failing the whole pipeline for the job.
|
||||||
|
Any other exception (bug, non-transient) propagates immediately.
|
||||||
|
|
||||||
|
musicbrainzngs is imported lazily so importing this module stays offline.
|
||||||
|
"""
|
||||||
|
import musicbrainzngs
|
||||||
|
|
||||||
|
transient = (musicbrainzngs.NetworkError, musicbrainzngs.ResponseError)
|
||||||
|
for i in range(attempts):
|
||||||
|
try:
|
||||||
|
return fn()
|
||||||
|
except transient:
|
||||||
|
if i == attempts - 1:
|
||||||
|
raise
|
||||||
|
sleep(base_delay * (2**i))
|
||||||
|
|
||||||
|
|
||||||
def _ratio(a: str, b: str) -> float:
|
def _ratio(a: str, b: str) -> float:
|
||||||
return SequenceMatcher(None, a.strip().casefold(), b.strip().casefold()).ratio()
|
return SequenceMatcher(None, a.strip().casefold(), b.strip().casefold()).ratio()
|
||||||
|
|
||||||
|
|
||||||
|
def _is_album(g: dict) -> bool:
|
||||||
|
return (g.get("primary-type") or "").casefold() == "album"
|
||||||
|
|
||||||
|
|
||||||
|
def _rg_year(g: dict) -> int:
|
||||||
|
"""First-release year of a release-group, or 9999 when unknown (so undated groups lose ties)."""
|
||||||
|
frd = (g.get("first-release-date") or "")[:4]
|
||||||
|
return int(frd) if frd.isdigit() else 9999
|
||||||
|
|
||||||
|
|
||||||
|
def _credited_artist(g: dict) -> str:
|
||||||
|
"""Primary credited artist name of a release-group search result, or '' if absent."""
|
||||||
|
credit = g.get("artist-credit")
|
||||||
|
if isinstance(credit, list) and credit and isinstance(credit[0], dict):
|
||||||
|
return (credit[0].get("artist") or {}).get("name", "") or ""
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def _best_release_group(artist: str, album: str, groups: list) -> dict | None:
|
||||||
|
"""Choose the best release-group match for `artist`/`album`.
|
||||||
|
|
||||||
|
Ranked, highest first, on `(artist_ratio, title_ratio, is_album)`:
|
||||||
|
* artist match dominates — MusicBrainz relevance ties same-titled release groups by
|
||||||
|
*different* artists (e.g. "Fun Machine" exists as a Lake Street Dive EP and an
|
||||||
|
unrelated band's Album), and the artist we followed is trustworthy, so a credited
|
||||||
|
artist that matches the request outranks everything else;
|
||||||
|
* title similarity is next;
|
||||||
|
* an Album primary-type only breaks ties *within* the same artist+title, so a famous
|
||||||
|
album (Michael Jackson's "Off the Wall") still isn't resolved to its same-named
|
||||||
|
single whose short tracklist would map positionally onto the album's files;
|
||||||
|
* finally, the EARLIEST release-group wins — MusicBrainz lists the original album and its
|
||||||
|
later reissues/anniversary editions as separate same-titled groups (Linkin Park "Hybrid
|
||||||
|
Theory": a 2000 original and a 2023 13-track reissue), which tie on everything above, so
|
||||||
|
MB's arbitrary relevance order used to pick a reissue whose longer tracklist/runtime then
|
||||||
|
failed completeness checks against the standard album a source delivers. Prefer the original.
|
||||||
|
Artist is a ranking signal, never a hard filter — a low match sinks a candidate but
|
||||||
|
never drops the release, so credited-name variations (feat., punctuation) still resolve.
|
||||||
|
Returns None below the title threshold. Pure — no network I/O."""
|
||||||
|
if not groups:
|
||||||
|
return None
|
||||||
|
best = max(
|
||||||
|
groups,
|
||||||
|
key=lambda g: (
|
||||||
|
_ratio(artist, _credited_artist(g)),
|
||||||
|
_ratio(album, g.get("title", "")),
|
||||||
|
_is_album(g),
|
||||||
|
-_rg_year(g), # tie-break: the original (earliest) group over later reissues
|
||||||
|
),
|
||||||
|
)
|
||||||
|
if _ratio(album, best.get("title", "")) < _MIN_TITLE_RATIO:
|
||||||
|
return None
|
||||||
|
return best
|
||||||
|
|
||||||
|
|
||||||
|
def _pick_release(releases: list) -> dict | None:
|
||||||
|
"""Choose which release of a release-group to read the tracklist from. MB returns them in an
|
||||||
|
arbitrary order and ``releases[0]`` is often a later reissue/deluxe/anniversary edition with
|
||||||
|
more tracks and a longer runtime than the standard album a source actually delivers — which
|
||||||
|
then makes a *complete* download fail the pipeline's track-count/duration completeness checks
|
||||||
|
and re-download forever (Linkin Park "Hybrid Theory": MB's arbitrary pick was a 2023 13-track
|
||||||
|
reissue at ~52 min vs the 2000 original's 12 tracks / ~38 min). Prefer the ORIGINAL standard
|
||||||
|
edition: an Official release, then the earliest date, then a dated one. Pure — unit-tested."""
|
||||||
|
if not releases:
|
||||||
|
return None
|
||||||
|
def _key(r: dict) -> tuple:
|
||||||
|
official = 0 if (r.get("status") == "Official") else 1 # official editions first
|
||||||
|
date = r.get("date") or ""
|
||||||
|
return (official, 0 if date else 1, date) # then dated, then earliest date
|
||||||
|
return min(releases, key=_key)
|
||||||
|
|
||||||
|
|
||||||
class MusicBrainzResolver:
|
class MusicBrainzResolver:
|
||||||
"""Real MbResolver using the MusicBrainz webservice via musicbrainzngs.
|
"""Real MbResolver using the MusicBrainz webservice via musicbrainzngs.
|
||||||
|
|
||||||
@@ -26,12 +121,10 @@ class MusicBrainzResolver:
|
|||||||
|
|
||||||
musicbrainzngs.set_useragent(self._app, self._version, self._contact)
|
musicbrainzngs.set_useragent(self._app, self._version, self._contact)
|
||||||
|
|
||||||
res = musicbrainzngs.search_release_groups(query=album, artist=artist, limit=5)
|
res = _with_retry(lambda: musicbrainzngs.search_release_groups(query=album, artist=artist, limit=5))
|
||||||
groups = res.get("release-group-list", [])
|
groups = res.get("release-group-list", [])
|
||||||
if not groups:
|
rg = _best_release_group(artist, album, groups)
|
||||||
return None
|
if rg is None:
|
||||||
rg = max(groups, key=lambda g: _ratio(album, g.get("title", "")))
|
|
||||||
if _ratio(album, rg.get("title", "")) < _MIN_TITLE_RATIO:
|
|
||||||
return None
|
return None
|
||||||
|
|
||||||
canonical_album = rg.get("title", album)
|
canonical_album = rg.get("title", album)
|
||||||
@@ -53,11 +146,12 @@ class MusicBrainzResolver:
|
|||||||
titles: tuple[str, ...] = ()
|
titles: tuple[str, ...] = ()
|
||||||
rgid = rg.get("id")
|
rgid = rg.get("id")
|
||||||
if rgid:
|
if rgid:
|
||||||
rgfull = musicbrainzngs.get_release_group_by_id(rgid, includes=["releases"])
|
rgfull = _with_retry(lambda: musicbrainzngs.get_release_group_by_id(rgid, includes=["releases"]))
|
||||||
releases = rgfull.get("release-group", {}).get("release-list", [])
|
releases = rgfull.get("release-group", {}).get("release-list", [])
|
||||||
if releases:
|
chosen = _pick_release(releases)
|
||||||
rel = musicbrainzngs.get_release_by_id(
|
if chosen:
|
||||||
releases[0]["id"], includes=["recordings"]
|
rel = _with_retry(
|
||||||
|
lambda: musicbrainzngs.get_release_by_id(chosen["id"], includes=["recordings"])
|
||||||
).get("release", {})
|
).get("release", {})
|
||||||
tracks = []
|
tracks = []
|
||||||
for medium in rel.get("medium-list", []):
|
for medium in rel.get("medium-list", []):
|
||||||
|
|||||||
@@ -1,21 +1,48 @@
|
|||||||
import os
|
import os
|
||||||
|
import re
|
||||||
|
|
||||||
from lyra_worker.library import safe_name
|
from lyra_worker.library import safe_name
|
||||||
|
|
||||||
_AUDIO_EXT = {".flac", ".mp3", ".m4a", ".opus", ".ogg", ".aac", ".wav"}
|
_AUDIO_EXT = {".flac", ".mp3", ".m4a", ".opus", ".ogg", ".aac", ".wav"}
|
||||||
|
|
||||||
|
# Leading "13.", "13 -", "13)" style track-number prefix streamrip bakes into a bonus-track
|
||||||
|
# filename. Requires an explicit delimiter so a real title that merely starts with a number
|
||||||
|
# (e.g. "99 Luftballons") is left alone.
|
||||||
|
_NUM_PREFIX = re.compile(r"^\s*\d+\s*[.):\-]\s*")
|
||||||
|
|
||||||
def plan_track_names(names: list[str], tracklist: tuple[str, ...]) -> list[tuple[str, str, str, int]]:
|
|
||||||
|
def _clean_extra_title(stem: str, artist: str = "") -> str:
|
||||||
|
"""Derive a display title for a track BEYOND the MusicBrainz tracklist (a bonus track that
|
||||||
|
streamrip named itself, e.g. `13. John Mayer - St. Patrick's Day (Album Version)`). Strip a
|
||||||
|
leading track-number prefix, then a leading `<artist> - ` if present, so the example yields
|
||||||
|
`St. Patrick's Day (Album Version)`. Falls back to the stem if nothing strips."""
|
||||||
|
s = _NUM_PREFIX.sub("", stem).strip()
|
||||||
|
if artist:
|
||||||
|
s = re.sub(r"^" + re.escape(artist) + r"\s*-\s*", "", s, flags=re.IGNORECASE).strip()
|
||||||
|
return s or stem
|
||||||
|
|
||||||
|
|
||||||
|
def plan_track_names(
|
||||||
|
names: list[str], tracklist: tuple[str, ...], artist: str = ""
|
||||||
|
) -> list[tuple[str, str, str, int]]:
|
||||||
"""Map audio filenames -> (old_name, new_name, title, track_number).
|
"""Map audio filenames -> (old_name, new_name, title, track_number).
|
||||||
|
|
||||||
Files are sorted; titles come from `tracklist` by position when available,
|
Files are sorted; titles come from `tracklist` by position when available. A file BEYOND
|
||||||
else the filename stem. Pure — no filesystem or tag I/O.
|
the tracklist (a bonus track streamrip named itself) is canonicalized via `_clean_extra_title`
|
||||||
|
so it gets a clean `## Title.ext` name too; with no tracklist at all the on-disk stem is kept.
|
||||||
|
Pure — no filesystem or tag I/O.
|
||||||
"""
|
"""
|
||||||
plan: list[tuple[str, str, str, int]] = []
|
plan: list[tuple[str, str, str, int]] = []
|
||||||
audio = sorted(n for n in names if os.path.splitext(n)[1].lower() in _AUDIO_EXT)
|
audio = sorted(n for n in names if os.path.splitext(n)[1].lower() in _AUDIO_EXT)
|
||||||
for i, name in enumerate(audio):
|
for i, name in enumerate(audio):
|
||||||
ext = os.path.splitext(name)[1].lower()
|
ext = os.path.splitext(name)[1].lower()
|
||||||
title = tracklist[i] if i < len(tracklist) else os.path.splitext(name)[0]
|
stem = os.path.splitext(name)[0]
|
||||||
|
if i < len(tracklist):
|
||||||
|
title = tracklist[i]
|
||||||
|
elif tracklist: # a genuine bonus track past a known tracklist -> canonicalize its name
|
||||||
|
title = _clean_extra_title(stem, artist)
|
||||||
|
else: # no tracklist resolved at all -> keep the on-disk stem (renumbered)
|
||||||
|
title = stem
|
||||||
new_name = f"{i + 1:02d} {safe_name(title)}{ext}"
|
new_name = f"{i + 1:02d} {safe_name(title)}{ext}"
|
||||||
plan.append((name, new_name, title, i + 1))
|
plan.append((name, new_name, title, i + 1))
|
||||||
return plan
|
return plan
|
||||||
@@ -29,7 +56,7 @@ class MutagenTagger:
|
|||||||
import mutagen
|
import mutagen
|
||||||
|
|
||||||
names = os.listdir(album_dir)
|
names = os.listdir(album_dir)
|
||||||
for old_name, new_name, title, number in plan_track_names(names, target.tracklist):
|
for old_name, new_name, title, number in plan_track_names(names, target.tracklist, target.artist):
|
||||||
try:
|
try:
|
||||||
path = os.path.join(album_dir, old_name)
|
path = os.path.join(album_dir, old_name)
|
||||||
audio = mutagen.File(path, easy=True)
|
audio = mutagen.File(path, easy=True)
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ import json
|
|||||||
import os
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import time
|
import time
|
||||||
|
import unicodedata
|
||||||
from typing import Callable
|
from typing import Callable
|
||||||
|
|
||||||
import requests
|
import requests
|
||||||
@@ -10,9 +11,26 @@ _DEFAULT_DOWNLOADS_ROOT = "/slskd-downloads"
|
|||||||
|
|
||||||
_AUDIO_EXT = {"flac", "mp3", "m4a", "ogg", "opus", "wav", "aac"}
|
_AUDIO_EXT = {"flac", "mp3", "m4a", "ogg", "opus", "wav", "aac"}
|
||||||
_LOSSLESS_EXT = {"flac", "wav"}
|
_LOSSLESS_EXT = {"flac", "wav"}
|
||||||
_SEARCH_POLLS = 40 # * 3s ≈ 2 min
|
_SEARCH_POLLS = 40 # * 3s ≈ 2 min
|
||||||
_XFER_POLLS = 200 # * 3s ≈ 10 min
|
_STALL_POLLS = 30 # * 3s ≈ 90s of no byte progress → abandon a queued/dead peer and fall through
|
||||||
_POLL_SECONDS = 3
|
_POLL_SECONDS = 3
|
||||||
|
# Absolute backstop for a slow-but-progressing peer. Scaled to the album's total bytes (a flat
|
||||||
|
# cap starved big albums: a 500MB set from a ~400KB/s serial peer needs ~60-70min and was cut at
|
||||||
|
# a flat 60min, then re-downloaded from scratch — never finishing). Budget at a conservative floor
|
||||||
|
# throughput, clamped. The 90s stall timeout still abandons a truly dead/queued peer quickly.
|
||||||
|
_MIN_XFER_POLLS = 400 # * 3s ≈ 20 min floor (small albums)
|
||||||
|
_MAX_XFER_POLLS = 2400 # * 3s ≈ 2 h ceiling (bounds worst-case slot hold)
|
||||||
|
_BACKSTOP_FLOOR_BPS = 64_000 # budget the backstop assuming >= ~64 KB/s sustained
|
||||||
|
|
||||||
|
|
||||||
|
def _max_polls(total_bytes: int) -> int:
|
||||||
|
"""Poll budget (each _POLL_SECONDS) for the absolute download backstop, scaled to album size at
|
||||||
|
a conservative floor throughput so a large slow-but-healthy transfer isn't abandoned mid-flight.
|
||||||
|
Clamped to [_MIN_XFER_POLLS, _MAX_XFER_POLLS]. Pure — unit-tested."""
|
||||||
|
if total_bytes <= 0:
|
||||||
|
return _MIN_XFER_POLLS
|
||||||
|
budget = int(total_bytes / (_BACKSTOP_FLOOR_BPS * _POLL_SECONDS))
|
||||||
|
return max(_MIN_XFER_POLLS, min(budget, _MAX_XFER_POLLS))
|
||||||
|
|
||||||
|
|
||||||
def _basename(path: str) -> str:
|
def _basename(path: str) -> str:
|
||||||
@@ -27,6 +45,114 @@ def _dirname(path: str) -> str:
|
|||||||
return ""
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
_ETA_CAP_SECONDS = 359_999 # ~100h; keeps the value well inside Job.downloadEtaSeconds (int4)
|
||||||
|
|
||||||
|
|
||||||
|
def _eta_seconds(total_bytes: int, bytes_now: int, speed_bps: float) -> int | None:
|
||||||
|
"""Seconds until the transfer finishes at the current measured rate, or None when it can't be
|
||||||
|
estimated (no throughput sample yet, unknown total, or already complete). Pure — unit-tested.
|
||||||
|
A near-stalled peer drives speed_bps toward zero, so the estimate is capped: an uncapped value
|
||||||
|
can exceed int4 and overflow the downloadEtaSeconds column write (poisoning the pipeline txn)."""
|
||||||
|
if speed_bps <= 0 or total_bytes <= 0 or bytes_now >= total_bytes:
|
||||||
|
return None
|
||||||
|
return min(int((total_bytes - bytes_now) / speed_bps), _ETA_CAP_SECONDS)
|
||||||
|
|
||||||
|
|
||||||
|
def _norm(s: str) -> str:
|
||||||
|
"""Lowercase, strip diacritics, and reduce to alphanumeric tokens, so 'Beyoncé' matches a
|
||||||
|
peer's 'Beyonce' folder and punctuation ('I Am… Sasha Fierce') doesn't defeat the match."""
|
||||||
|
s = unicodedata.normalize("NFKD", s or "")
|
||||||
|
s = "".join(ch for ch in s if not unicodedata.combining(ch))
|
||||||
|
s = "".join(ch if ch.isalnum() else " " for ch in s.lower())
|
||||||
|
return " ".join(s.split())
|
||||||
|
|
||||||
|
|
||||||
|
def _path_matches(needle: str, path: str) -> bool:
|
||||||
|
"""True if `needle` (an artist or album name) appears in a file path, tolerant of accents and
|
||||||
|
punctuation. Every significant token (≥3 chars) must be present; for a very short needle
|
||||||
|
(e.g. 'U2') the whole normalized string must appear."""
|
||||||
|
npath = _norm(path)
|
||||||
|
tokens = [t for t in _norm(needle).split() if len(t) >= 3]
|
||||||
|
if not tokens:
|
||||||
|
whole = _norm(needle)
|
||||||
|
return bool(whole) and whole in npath
|
||||||
|
return all(t in npath for t in tokens)
|
||||||
|
|
||||||
|
|
||||||
|
def _keep_matching(responses: list, needle: str) -> list:
|
||||||
|
"""Drop files — then now-empty responses — whose path doesn't match `needle`. Makes an album-
|
||||||
|
or artist-only fallback search precise so we never grab a same-titled release by a different
|
||||||
|
artist. Pure — unit-tested offline."""
|
||||||
|
out = []
|
||||||
|
for resp in responses:
|
||||||
|
files = [f for f in (resp.get("files") or []) if _path_matches(needle, f.get("filename", ""))]
|
||||||
|
if files:
|
||||||
|
out.append({**resp, "files": files})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_search_responses(responses: list, artist: str = "") -> list[dict]:
|
||||||
|
"""Turn slskd search responses into album candidates (one per peer+directory), ordered so the
|
||||||
|
peers that will FINISH first come first: a free upload slot, then the shortest estimated
|
||||||
|
transfer time (album bytes ÷ the peer's advertised speed), then a shorter queue. Ranking on
|
||||||
|
estimated time — not raw speed — matters because Soulseek FLAC rips of the same album vary
|
||||||
|
~2x in size (a 24-bit/bloated rip vs a standard CD rip), and Lyra scores them the SAME quality
|
||||||
|
class (the search exposes no bit-depth), so a peer advertising high speed but serving huge
|
||||||
|
files can lose a race — and blow past the download backstop — against a peer with a smaller
|
||||||
|
standard rip. The pipeline tries candidates in this order, so this is what makes the Soulseek
|
||||||
|
fall-through converge on a copy that actually completes. Pure — no I/O — unit-tested offline."""
|
||||||
|
scored: list[tuple] = []
|
||||||
|
for resp in responses:
|
||||||
|
username = resp.get("username", "")
|
||||||
|
has_slot = 0 if resp.get("hasFreeUploadSlot") else 1 # 0 sorts first (free slot = starts now)
|
||||||
|
speed = int(resp.get("uploadSpeed") or 0)
|
||||||
|
queue = int(resp.get("queueLength") or 0)
|
||||||
|
by_dir: dict[str, list] = {}
|
||||||
|
for f in resp.get("files", []) or []:
|
||||||
|
name = f.get("filename", "")
|
||||||
|
ext = name.rsplit(".", 1)[-1].lower() if "." in name else ""
|
||||||
|
if ext not in _AUDIO_EXT:
|
||||||
|
continue
|
||||||
|
by_dir.setdefault(_dirname(name), []).append(
|
||||||
|
{"filename": name, "size": f.get("size", 0), "ext": ext}
|
||||||
|
)
|
||||||
|
for directory, dfiles in by_dir.items():
|
||||||
|
if not dfiles:
|
||||||
|
continue
|
||||||
|
# Estimated seconds to pull this album from this peer at its advertised rate. Unknown/
|
||||||
|
# zero speed → a low nominal (1 B/s) so total size still orders those peers last.
|
||||||
|
total_bytes = sum(int(f["size"] or 0) for f in dfiles)
|
||||||
|
est_seconds = total_bytes / (speed if speed > 0 else 1)
|
||||||
|
lossless = all(f["ext"] in _LOSSLESS_EXT for f in dfiles)
|
||||||
|
dirname = _basename(directory)
|
||||||
|
# Trust the REQUESTED artist when it appears anywhere in the path — many rips put the
|
||||||
|
# artist in a parent folder ("Rihanna\(2009) Rated R\…") or bury it ("2009 - Rihanna -
|
||||||
|
# Rated R"), which the "Artist - Album" folder parse gets wrong; that tanked the
|
||||||
|
# confidence score and lost real FLACs to clean-named MP3s. album stays the folder name
|
||||||
|
# (fuzzy-matched downstream); artist is what the parse gets wrong.
|
||||||
|
if artist and _path_matches(artist, dfiles[0]["filename"]):
|
||||||
|
guess_artist, guess_album = artist, dirname
|
||||||
|
elif " - " in dirname:
|
||||||
|
guess_artist, guess_album = (p.strip() for p in dirname.split(" - ", 1))
|
||||||
|
else:
|
||||||
|
guess_artist, guess_album = "", dirname
|
||||||
|
candidate = {
|
||||||
|
"source_ref": json.dumps({
|
||||||
|
"username": username,
|
||||||
|
"files": [{"filename": f["filename"], "size": f["size"]} for f in dfiles],
|
||||||
|
}),
|
||||||
|
"title": guess_album,
|
||||||
|
"artist": guess_artist,
|
||||||
|
"track_count": len(dfiles),
|
||||||
|
"format": "FLAC" if lossless else "MP3",
|
||||||
|
"bitrate": None,
|
||||||
|
}
|
||||||
|
scored.append((has_slot, est_seconds, queue, candidate))
|
||||||
|
# ascending: free slot first, then shortest estimated transfer, then shortest queue
|
||||||
|
scored.sort(key=lambda t: (t[0], t[1], t[2]))
|
||||||
|
return [c for *_rest, c in scored]
|
||||||
|
|
||||||
|
|
||||||
class SlskdClient:
|
class SlskdClient:
|
||||||
"""Real Soulseek client talking to an slskd daemon's REST API.
|
"""Real Soulseek client talking to an slskd daemon's REST API.
|
||||||
|
|
||||||
@@ -58,11 +184,12 @@ class SlskdClient:
|
|||||||
r.raise_for_status()
|
r.raise_for_status()
|
||||||
return r.json()
|
return r.json()
|
||||||
|
|
||||||
def search_album(self, artist: str, album: str) -> list[dict]:
|
def _run_search(self, text: str) -> list:
|
||||||
|
"""POST a search, wait for it to complete, and return the raw responses (a list)."""
|
||||||
r = requests.post(
|
r = requests.post(
|
||||||
f"{self._url}/api/v0/searches",
|
f"{self._url}/api/v0/searches",
|
||||||
headers=self._headers(),
|
headers=self._headers(),
|
||||||
json={"searchText": f"{artist} {album}"},
|
json={"searchText": text},
|
||||||
timeout=30,
|
timeout=30,
|
||||||
)
|
)
|
||||||
r.raise_for_status()
|
r.raise_for_status()
|
||||||
@@ -75,43 +202,27 @@ class SlskdClient:
|
|||||||
break
|
break
|
||||||
|
|
||||||
responses = self._get(f"/api/v0/searches/{search_id}/responses")
|
responses = self._get(f"/api/v0/searches/{search_id}/responses")
|
||||||
candidates: list[dict] = []
|
return responses if isinstance(responses, list) else []
|
||||||
for resp in responses:
|
|
||||||
username = resp.get("username", "")
|
def search_album(self, artist: str, album: str) -> list[dict]:
|
||||||
by_dir: dict[str, list] = {}
|
responses = self._run_search(f"{artist} {album}")
|
||||||
for f in resp.get("files", []) or []:
|
if not responses:
|
||||||
name = f.get("filename", "")
|
# The Soulseek server silently drops searches containing certain blocklisted terms
|
||||||
ext = name.rsplit(".", 1)[-1].lower() if "." in name else ""
|
# (major-label artists / album titles like Adele, Rihanna, Beyoncé, "Lemonade") →
|
||||||
if ext not in _AUDIO_EXT:
|
# 0 responses. The blocked term poisons the whole query, but the OTHER field usually
|
||||||
|
# isn't blocked, so retry with each field alone (normalized) and keep only results
|
||||||
|
# whose path still matches the field we dropped. That recovers e.g. Beyoncé's
|
||||||
|
# "I Am… Sasha Fierce" via the album title, without ever grabbing a same-titled album
|
||||||
|
# by someone else.
|
||||||
|
for query, needle in ((album, artist), (artist, album)):
|
||||||
|
q = _norm(query)
|
||||||
|
if not q:
|
||||||
continue
|
continue
|
||||||
by_dir.setdefault(_dirname(name), []).append(
|
alt = _keep_matching(self._run_search(q), needle)
|
||||||
{"filename": name, "size": f.get("size", 0), "ext": ext}
|
if alt:
|
||||||
)
|
responses = alt
|
||||||
for directory, dfiles in by_dir.items():
|
break
|
||||||
if not dfiles:
|
return _parse_search_responses(responses, artist)
|
||||||
continue
|
|
||||||
lossless = all(f["ext"] in _LOSSLESS_EXT for f in dfiles)
|
|
||||||
dirname = _basename(directory)
|
|
||||||
if " - " in dirname:
|
|
||||||
guess_artist, guess_album = (p.strip() for p in dirname.split(" - ", 1))
|
|
||||||
else:
|
|
||||||
guess_artist, guess_album = "", dirname
|
|
||||||
candidates.append(
|
|
||||||
{
|
|
||||||
"source_ref": json.dumps(
|
|
||||||
{
|
|
||||||
"username": username,
|
|
||||||
"files": [{"filename": f["filename"], "size": f["size"]} for f in dfiles],
|
|
||||||
}
|
|
||||||
),
|
|
||||||
"title": guess_album,
|
|
||||||
"artist": guess_artist,
|
|
||||||
"track_count": len(dfiles),
|
|
||||||
"format": "FLAC" if lossless else "MP3",
|
|
||||||
"bitrate": None,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
return candidates
|
|
||||||
|
|
||||||
def download(self, source_ref: str, dest: str, on_progress: Callable[[float], None]) -> dict:
|
def download(self, source_ref: str, dest: str, on_progress: Callable[[float], None]) -> dict:
|
||||||
ref = json.loads(source_ref)
|
ref = json.loads(source_ref)
|
||||||
@@ -119,7 +230,17 @@ class SlskdClient:
|
|||||||
files = ref["files"]
|
files = ref["files"]
|
||||||
wanted = {f["filename"] for f in files}
|
wanted = {f["filename"] for f in files}
|
||||||
os.makedirs(dest, exist_ok=True)
|
os.makedirs(dest, exist_ok=True)
|
||||||
|
xfer_ids: set[str] = set() # slskd transfer ids we started, for cancel-on-abandon
|
||||||
|
try:
|
||||||
|
return self._download(username, files, wanted, dest, on_progress, xfer_ids)
|
||||||
|
except Exception:
|
||||||
|
# Abandoning this peer (timeout / error / the pipeline moving on): cancel the transfers
|
||||||
|
# we enqueued so they don't keep running and pile up in slskd as the job falls through
|
||||||
|
# to other peers/sources for the same album.
|
||||||
|
self._cancel_transfers(username, xfer_ids)
|
||||||
|
raise
|
||||||
|
|
||||||
|
def _download(self, username, files, wanted, dest, on_progress, xfer_ids: set) -> dict:
|
||||||
r = requests.post(
|
r = requests.post(
|
||||||
f"{self._url}/api/v0/transfers/downloads/{username}",
|
f"{self._url}/api/v0/transfers/downloads/{username}",
|
||||||
headers=self._headers(),
|
headers=self._headers(),
|
||||||
@@ -130,23 +251,49 @@ class SlskdClient:
|
|||||||
on_progress(0.0)
|
on_progress(0.0)
|
||||||
|
|
||||||
total = len(files)
|
total = len(files)
|
||||||
|
total_bytes = sum(int(f.get("size") or 0) for f in files)
|
||||||
completed = False
|
completed = False
|
||||||
for _ in range(_XFER_POLLS):
|
last_bytes = -1
|
||||||
|
stall = 0
|
||||||
|
speed = 0.0 # bytes/sec, EWMA-smoothed so the ETA doesn't jitter poll to poll
|
||||||
|
prev_bytes = 0
|
||||||
|
prev_t = time.monotonic()
|
||||||
|
for _ in range(_max_polls(total_bytes)):
|
||||||
time.sleep(_POLL_SECONDS)
|
time.sleep(_POLL_SECONDS)
|
||||||
data = self._get(f"/api/v0/transfers/downloads/{username}")
|
data = self._get(f"/api/v0/transfers/downloads/{username}")
|
||||||
states: list[str] = []
|
states: list[str] = []
|
||||||
|
bytes_now = 0
|
||||||
for directory in data.get("directories", []) if isinstance(data, dict) else []:
|
for directory in data.get("directories", []) if isinstance(data, dict) else []:
|
||||||
for f in directory.get("files", []):
|
for f in directory.get("files", []):
|
||||||
if f.get("filename") in wanted: # only the files we enqueued
|
if f.get("filename") in wanted: # only the files we enqueued
|
||||||
states.append(str(f.get("state", "")))
|
states.append(str(f.get("state", "")))
|
||||||
|
bytes_now += int(f.get("bytesTransferred") or 0)
|
||||||
|
if f.get("id"): # remember the transfer so we can cancel it if abandoned
|
||||||
|
xfer_ids.add(str(f["id"]))
|
||||||
if states:
|
if states:
|
||||||
if any(bad in s for s in states for bad in ("Errored", "Rejected", "Cancelled")):
|
if any(bad in s for s in states for bad in ("Errored", "Rejected", "Cancelled")):
|
||||||
raise RuntimeError("soulseek transfer failed")
|
raise RuntimeError("soulseek transfer failed")
|
||||||
|
now = time.monotonic()
|
||||||
|
dt = now - prev_t
|
||||||
|
if dt > 0 and bytes_now >= prev_bytes:
|
||||||
|
inst = (bytes_now - prev_bytes) / dt
|
||||||
|
speed = inst if speed == 0.0 else 0.6 * speed + 0.4 * inst
|
||||||
|
prev_bytes, prev_t = bytes_now, now
|
||||||
done = sum(1 for s in states if "Completed" in s)
|
done = sum(1 for s in states if "Completed" in s)
|
||||||
on_progress(min(1.0, done / max(total, 1)))
|
on_progress(min(1.0, done / max(total, 1)), _eta_seconds(total_bytes, bytes_now, speed))
|
||||||
if done >= total:
|
if done >= total:
|
||||||
completed = True
|
completed = True
|
||||||
break
|
break
|
||||||
|
# Stall detection: a peer making byte progress keeps its slot (up to the absolute
|
||||||
|
# backstop above) so a slow-but-working transfer can finish; one that sends no bytes
|
||||||
|
# for _STALL_POLLS (~90s) is queued/dead — abandon it fast and fall through.
|
||||||
|
if bytes_now > last_bytes:
|
||||||
|
last_bytes = bytes_now
|
||||||
|
stall = 0
|
||||||
|
else:
|
||||||
|
stall += 1
|
||||||
|
if stall >= _STALL_POLLS:
|
||||||
|
raise TimeoutError(f"soulseek transfer stalled for {username}")
|
||||||
|
|
||||||
if not completed:
|
if not completed:
|
||||||
raise TimeoutError(f"soulseek download did not complete for {username}")
|
raise TimeoutError(f"soulseek download did not complete for {username}")
|
||||||
@@ -162,6 +309,19 @@ class SlskdClient:
|
|||||||
)
|
)
|
||||||
return {"track_count": moved, "path": dest}
|
return {"track_count": moved, "path": dest}
|
||||||
|
|
||||||
|
def _cancel_transfers(self, username: str, xfer_ids: set) -> None:
|
||||||
|
"""Cancel the given in-flight slskd transfers for a peer. Best-effort: a cleanup failure
|
||||||
|
must never mask the original download failure."""
|
||||||
|
for tid in xfer_ids:
|
||||||
|
try:
|
||||||
|
requests.delete(
|
||||||
|
f"{self._url}/api/v0/transfers/downloads/{username}/{tid}",
|
||||||
|
headers=self._headers(),
|
||||||
|
timeout=15,
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
def _retrieve(self, files: list[dict], dest: str) -> int:
|
def _retrieve(self, files: list[dict], dest: str) -> int:
|
||||||
"""Move the just-finished slskd files out of its downloads dir into `dest`.
|
"""Move the just-finished slskd files out of its downloads dir into `dest`.
|
||||||
|
|
||||||
|
|||||||
@@ -34,6 +34,53 @@ def _run(coro):
|
|||||||
return _loop.run_until_complete(coro)
|
return _loop.run_until_complete(coro)
|
||||||
|
|
||||||
|
|
||||||
|
class _ProgressAggregator:
|
||||||
|
"""Turn streamrip's per-track byte callbacks into one album-level 0..1 fraction.
|
||||||
|
|
||||||
|
streamrip downloads an album's tracks concurrently and reports progress only per track
|
||||||
|
(a byte delta per chunk), so the file-count poller sees 0 complete files until they all
|
||||||
|
land at once — the 0%→100% jump. Summing downloaded bytes over an *estimated* album total
|
||||||
|
(average started-track size × track count) gives a smooth, generally-climbing fraction.
|
||||||
|
Track sizes are similar, so the estimate is stable once a track or two has started; capped
|
||||||
|
at 0.99 because the pipeline sets 1.0 once the whole download returns. All streamrip
|
||||||
|
coroutines run on one event loop on the calling thread, so no locking is needed."""
|
||||||
|
|
||||||
|
def __init__(self, track_count: int, on_progress: Callable[[float], None]):
|
||||||
|
self._n = track_count or 0
|
||||||
|
self._on_progress = on_progress
|
||||||
|
self._seen_total = 0 # summed byte-size of tracks that have started downloading
|
||||||
|
self._seen_count = 0 # number of tracks that have started
|
||||||
|
self._done_bytes = 0 # summed bytes downloaded across all started tracks
|
||||||
|
|
||||||
|
def start_track(self, total) -> None:
|
||||||
|
self._seen_total += max(int(total or 0), 0)
|
||||||
|
self._seen_count += 1
|
||||||
|
|
||||||
|
def advance(self, delta) -> None:
|
||||||
|
self._done_bytes += max(int(delta or 0), 0)
|
||||||
|
if self._seen_count == 0:
|
||||||
|
return
|
||||||
|
avg = self._seen_total / self._seen_count
|
||||||
|
n = self._n if self._n > 0 else self._seen_count
|
||||||
|
est_total = avg * n
|
||||||
|
if est_total > 0:
|
||||||
|
self._on_progress(min(self._done_bytes / est_total, 0.99))
|
||||||
|
|
||||||
|
|
||||||
|
class _ProgressHandle:
|
||||||
|
"""streamrip's `Track.download` uses `with get_progress_callback(...) as cb:`; `cb` must be
|
||||||
|
the per-chunk byte callback. This stand-in yields that callback and no-ops on exit."""
|
||||||
|
|
||||||
|
def __init__(self, update: Callable[[int], None]):
|
||||||
|
self.update = update
|
||||||
|
|
||||||
|
def __enter__(self):
|
||||||
|
return self.update
|
||||||
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
def _flatten_audio(dest: str) -> int:
|
def _flatten_audio(dest: str) -> int:
|
||||||
"""streamrip writes into a nested `Artist - Album [...]/` subfolder. Move any audio
|
"""streamrip writes into a nested `Artist - Album [...]/` subfolder. Move any audio
|
||||||
files up into `dest`, remove the now-empty subfolders, and return the count of audio
|
files up into `dest`, remove the now-empty subfolders, and return the count of audio
|
||||||
@@ -136,6 +183,7 @@ class StreamripClient:
|
|||||||
|
|
||||||
async def _download(self, album_id: str, dest: str, on_progress: Callable[[float], None]) -> None:
|
async def _download(self, album_id: str, dest: str, on_progress: Callable[[float], None]) -> None:
|
||||||
os.makedirs(dest, exist_ok=True)
|
os.makedirs(dest, exist_ok=True)
|
||||||
|
import streamrip.media.track as _track_mod
|
||||||
from streamrip.client import QobuzClient
|
from streamrip.client import QobuzClient
|
||||||
from streamrip.db import Database, Dummy
|
from streamrip.db import Database, Dummy
|
||||||
from streamrip.media import PendingAlbum
|
from streamrip.media import PendingAlbum
|
||||||
@@ -153,7 +201,21 @@ class StreamripClient:
|
|||||||
album = await pending.resolve()
|
album = await pending.resolve()
|
||||||
if album is None:
|
if album is None:
|
||||||
raise RuntimeError(f"could not resolve Qobuz album {album_id}")
|
raise RuntimeError(f"could not resolve Qobuz album {album_id}")
|
||||||
await album.rip()
|
# Hook streamrip's per-track byte progress into a single album-level fraction so the
|
||||||
|
# Floor bar climbs smoothly instead of jumping 0→100. `Track.download` binds
|
||||||
|
# get_progress_callback at import, so patch it on the track module (not progress.py).
|
||||||
|
agg = _ProgressAggregator(len(getattr(album, "tracks", []) or []), on_progress)
|
||||||
|
_orig_cb = _track_mod.get_progress_callback
|
||||||
|
|
||||||
|
def _hook(_enabled, total, _desc):
|
||||||
|
agg.start_track(total)
|
||||||
|
return _ProgressHandle(agg.advance)
|
||||||
|
|
||||||
|
_track_mod.get_progress_callback = _hook
|
||||||
|
try:
|
||||||
|
await album.rip()
|
||||||
|
finally:
|
||||||
|
_track_mod.get_progress_callback = _orig_cb
|
||||||
on_progress(1.0)
|
on_progress(1.0)
|
||||||
finally:
|
finally:
|
||||||
await client.session.close()
|
await client.session.close()
|
||||||
|
|||||||
@@ -72,17 +72,24 @@ class FailingAdapter(_BaseFake):
|
|||||||
|
|
||||||
|
|
||||||
class FakeMbBrowser:
|
class FakeMbBrowser:
|
||||||
"""In-memory MbBrowser for tests. `releases` maps artist mbid -> [ReleaseGroupInfo]."""
|
"""In-memory MbBrowser for tests. `releases` maps artist mbid -> [ReleaseGroupInfo].
|
||||||
|
`default_core=True` returns a synthetic core Album for any artist not in `releases`, so
|
||||||
|
scoring tests don't trip the "no core discography → drop the suggestion" filter."""
|
||||||
|
|
||||||
def __init__(self, artists=None, releases=None):
|
def __init__(self, artists=None, releases=None, default_core=False):
|
||||||
self._artists = list(artists or [])
|
self._artists = list(artists or [])
|
||||||
self._releases = dict(releases or {})
|
self._releases = dict(releases or {})
|
||||||
|
self._default_core = default_core
|
||||||
|
|
||||||
def search_artist(self, name: str) -> list[ArtistHit]:
|
def search_artist(self, name: str) -> list[ArtistHit]:
|
||||||
return list(self._artists)
|
return list(self._artists)
|
||||||
|
|
||||||
def browse_release_groups(self, artist_mbid: str) -> list[ReleaseGroupInfo]:
|
def browse_release_groups(self, artist_mbid: str) -> list[ReleaseGroupInfo]:
|
||||||
return list(self._releases.get(artist_mbid, []))
|
if artist_mbid in self._releases:
|
||||||
|
return list(self._releases[artist_mbid])
|
||||||
|
if self._default_core:
|
||||||
|
return [ReleaseGroupInfo(f"rg-{artist_mbid}", "Album", "Album", (), "2000-01-01")]
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
class FakeAudioProbe:
|
class FakeAudioProbe:
|
||||||
|
|||||||
@@ -24,13 +24,50 @@ def reclaim_stuck_jobs(conn: psycopg.Connection) -> int:
|
|||||||
return n
|
return n
|
||||||
|
|
||||||
|
|
||||||
|
_MAX_ATTEMPTS = 5
|
||||||
|
|
||||||
|
|
||||||
|
def requeue_or_fail(
|
||||||
|
conn: psycopg.Connection, job_id: str, error: str, max_attempts: int = _MAX_ATTEMPTS
|
||||||
|
) -> None:
|
||||||
|
"""Recover a job whose pipeline raised an *unhandled* exception (e.g. a dropped MusicBrainz
|
||||||
|
TLS connection surfacing as an SSL EOF). Without this the job is left in its mid-pipeline
|
||||||
|
'matching' state; since claim_next only picks 'requested' it is orphaned until the next
|
||||||
|
restart's reclaim_stuck_jobs — showing "Searching sources…" on the press indefinitely.
|
||||||
|
|
||||||
|
Such errors are usually transient, so requeue to 'requested' until the attempt cap, then fall
|
||||||
|
to 'needs_attention' so a persistently-failing job stays visible instead of looping forever.
|
||||||
|
`attempts` is bumped by claim_next on every claim, so it already counts tries."""
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT attempts, "requestId" FROM "Job" WHERE id = %s', (job_id,))
|
||||||
|
row = cur.fetchone()
|
||||||
|
if row is None:
|
||||||
|
return
|
||||||
|
attempts, request_id = row
|
||||||
|
if attempts >= max_attempts:
|
||||||
|
cur.execute(
|
||||||
|
'UPDATE "Job" SET state = \'needs_attention\', error = %s, "claimedAt" = NULL, '
|
||||||
|
'"updatedAt" = now() WHERE id = %s',
|
||||||
|
(error, job_id),
|
||||||
|
)
|
||||||
|
cur.execute('UPDATE "Request" SET status = \'needs_attention\' WHERE id = %s', (request_id,))
|
||||||
|
else:
|
||||||
|
cur.execute(
|
||||||
|
'UPDATE "Job" SET state = \'requested\', "currentStage" = \'intake\', error = NULL, '
|
||||||
|
'"claimedAt" = NULL, "downloadProgress" = 0, "updatedAt" = now() WHERE id = %s',
|
||||||
|
(job_id,),
|
||||||
|
)
|
||||||
|
cur.execute('UPDATE "Request" SET status = \'pending\' WHERE id = %s', (request_id,))
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
def claim_next(conn: psycopg.Connection) -> str | None:
|
def claim_next(conn: psycopg.Connection) -> str | None:
|
||||||
"""Atomically claim the oldest queued job. Returns its id or None."""
|
"""Atomically claim the oldest queued job. Returns its id or None."""
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
"""
|
"""
|
||||||
SELECT id FROM "Job"
|
SELECT id FROM "Job"
|
||||||
WHERE state = 'requested'
|
WHERE state = 'requested' AND NOT paused
|
||||||
ORDER BY "createdAt" ASC
|
ORDER BY "createdAt" ASC
|
||||||
FOR UPDATE SKIP LOCKED
|
FOR UPDATE SKIP LOCKED
|
||||||
LIMIT 1
|
LIMIT 1
|
||||||
|
|||||||
@@ -143,21 +143,21 @@ def _existing_rg_mbids(conn: psycopg.Connection) -> set[str]:
|
|||||||
return {r[0] for r in cur.fetchall()}
|
return {r[0] for r in cur.fetchall()}
|
||||||
|
|
||||||
|
|
||||||
def _upsert_album(conn: psycopg.Connection, artist: dict, rg) -> int:
|
def _upsert_album(conn: psycopg.Connection, artist: dict, rg, reason: str) -> int:
|
||||||
dedupe = f"album:{artist['mbid']}:{rg.rg_mbid}"
|
dedupe = f"album:{artist['mbid']}:{rg.rg_mbid}"
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
'INSERT INTO "DiscoverySuggestion" (id, kind, "artistMbid", "artistName", '
|
'INSERT INTO "DiscoverySuggestion" (id, kind, "artistMbid", "artistName", '
|
||||||
'"rgMbid", album, "primaryType", "secondaryTypes", "firstReleaseDate", '
|
'"rgMbid", album, "primaryType", "secondaryTypes", "firstReleaseDate", '
|
||||||
'score, "seedCount", sources, status, "dedupeKey", "createdAt", "updatedAt") '
|
'score, "seedCount", sources, "albumReason", status, "dedupeKey", "createdAt", "updatedAt") '
|
||||||
"VALUES (gen_random_uuid()::text, 'album', %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, "
|
"VALUES (gen_random_uuid()::text, 'album', %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, "
|
||||||
"'pending', %s, now(), now()) "
|
"'pending', %s, now(), now()) "
|
||||||
'ON CONFLICT ("dedupeKey") DO UPDATE SET score = EXCLUDED.score, '
|
'ON CONFLICT ("dedupeKey") DO UPDATE SET score = EXCLUDED.score, '
|
||||||
'"updatedAt" = now() '
|
'"albumReason" = EXCLUDED."albumReason", "updatedAt" = now() '
|
||||||
"WHERE \"DiscoverySuggestion\".status = 'pending'",
|
"WHERE \"DiscoverySuggestion\".status = 'pending'",
|
||||||
(artist["mbid"], artist["name"], rg.rg_mbid, rg.title,
|
(artist["mbid"], artist["name"], rg.rg_mbid, rg.title,
|
||||||
rg.primary_type or None, list(rg.secondary_types), rg.first_release_date or None,
|
rg.primary_type or None, list(rg.secondary_types), rg.first_release_date or None,
|
||||||
artist["score"], artist["seed_count"], sorted(artist["sources"]), dedupe),
|
artist["score"], artist["seed_count"], sorted(artist["sources"]), reason, dedupe),
|
||||||
)
|
)
|
||||||
return cur.rowcount
|
return cur.rowcount
|
||||||
|
|
||||||
@@ -174,8 +174,26 @@ def _album_kind_ok(g, albums_only: bool) -> bool:
|
|||||||
return is_core_release(g.primary_type, g.secondary_types)
|
return is_core_release(g.primary_type, g.secondary_types)
|
||||||
|
|
||||||
|
|
||||||
|
def _norm_title(s: str) -> str:
|
||||||
|
return (s or "").strip().lower()
|
||||||
|
|
||||||
|
|
||||||
|
def _popular_pick(conn, popularity, artist_name: str, core: list) -> str | None:
|
||||||
|
"""The rg_mbid of the artist's most-played qualifying un-owned album (Last.fm), matched
|
||||||
|
into the already-browsed `core` by MBID first then normalized title. None if unavailable."""
|
||||||
|
if popularity is None or not core:
|
||||||
|
return None
|
||||||
|
by_mbid = {g.rg_mbid: g for g in core}
|
||||||
|
by_title = {_norm_title(g.title): g for g in core}
|
||||||
|
for name, mbid, _plays in popularity.top_albums(conn, artist_name):
|
||||||
|
g = (by_mbid.get(mbid) if mbid else None) or by_title.get(_norm_title(name))
|
||||||
|
if g is not None:
|
||||||
|
return g.rg_mbid
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _derive_albums(conn: psycopg.Connection, browser: MbBrowser, surfaced: list[dict],
|
def _derive_albums(conn: psycopg.Connection, browser: MbBrowser, surfaced: list[dict],
|
||||||
cfg: DiscoveryConfig) -> int:
|
cfg: DiscoveryConfig, popularity=None) -> int:
|
||||||
if cfg.albums_per_artist <= 0:
|
if cfg.albums_per_artist <= 0:
|
||||||
return 0
|
return 0
|
||||||
have = _existing_rg_mbids(conn)
|
have = _existing_rg_mbids(conn)
|
||||||
@@ -186,12 +204,44 @@ def _derive_albums(conn: psycopg.Connection, browser: MbBrowser, surfaced: list[
|
|||||||
except Exception as e: # one artist's browse failure must not abort the sweep
|
except Exception as e: # one artist's browse failure must not abort the sweep
|
||||||
print(f"worker: discovery album browse failed for {artist['mbid']}: {e}", flush=True)
|
print(f"worker: discovery album browse failed for {artist['mbid']}: {e}", flush=True)
|
||||||
continue
|
continue
|
||||||
|
# Drop suggestions for "artists" with no real discography of their own — producers,
|
||||||
|
# session players, band members, soundtrack-only credits. ListenBrainz surfaces them
|
||||||
|
# (they co-occur in listens) but they have no core release-group to follow or want, and
|
||||||
|
# their page is empty. Require at least one core release (Album/Single/EP, no secondary
|
||||||
|
# type); on browse failure we keep the suggestion rather than drop a real artist.
|
||||||
|
if not any(is_core_release(g.primary_type, g.secondary_types) for g in groups):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute("DELETE FROM \"DiscoverySuggestion\" WHERE status = 'pending' "
|
||||||
|
'AND "artistMbid" = %s', (artist["mbid"],))
|
||||||
|
continue
|
||||||
core = [g for g in groups
|
core = [g for g in groups
|
||||||
if _album_kind_ok(g, cfg.albums_only) and g.rg_mbid not in have]
|
if _album_kind_ok(g, cfg.albums_only) and g.rg_mbid not in have]
|
||||||
core.sort(key=lambda g: g.first_release_date or "", reverse=True)
|
core.sort(key=lambda g: g.first_release_date or "", reverse=True)
|
||||||
|
|
||||||
|
# newest picks (existing behavior) + the single most-played album (Last.fm); merged so
|
||||||
|
# an album that is both newest AND most-played is one row tagged "newest,popular".
|
||||||
|
reasons: dict[str, str] = {}
|
||||||
for rg in core[: cfg.albums_per_artist]:
|
for rg in core[: cfg.albums_per_artist]:
|
||||||
albums += _upsert_album(conn, artist, rg)
|
reasons[rg.rg_mbid] = "newest"
|
||||||
have.add(rg.rg_mbid)
|
popular = _popular_pick(conn, popularity, artist["name"], core)
|
||||||
|
if popular is not None:
|
||||||
|
reasons[popular] = "newest,popular" if reasons.get(popular) == "newest" else "popular"
|
||||||
|
|
||||||
|
# Re-deriving replaces this artist's album set: drop any pending album suggestion of
|
||||||
|
# theirs that is no longer a pick (stale singles/EPs from older sweeps, or an album now
|
||||||
|
# owned), keeping wanted/dismissed rows untouched. Then upsert the current picks.
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
"DELETE FROM \"DiscoverySuggestion\" WHERE kind = 'album' AND status = 'pending' "
|
||||||
|
'AND "artistMbid" = %s AND NOT ("rgMbid" = ANY(%s))',
|
||||||
|
(artist["mbid"], list(reasons.keys())),
|
||||||
|
)
|
||||||
|
|
||||||
|
for rg in core: # preserve newest-first order among the chosen
|
||||||
|
reason = reasons.get(rg.rg_mbid)
|
||||||
|
if reason is not None:
|
||||||
|
albums += _upsert_album(conn, artist, rg, reason)
|
||||||
|
have.add(rg.rg_mbid)
|
||||||
return albums
|
return albums
|
||||||
|
|
||||||
|
|
||||||
@@ -275,8 +325,20 @@ def _recompute_suggestions(conn: psycopg.Connection, affected: set[str],
|
|||||||
return surfaced
|
return surfaced
|
||||||
|
|
||||||
|
|
||||||
|
def reset_pending(conn: psycopg.Connection) -> None:
|
||||||
|
"""Clear discovery state so the next sweep rebuilds from scratch with current logic. Drops
|
||||||
|
all PENDING suggestions (followed/wanted/dismissed user decisions are kept), every seed
|
||||||
|
contribution + library-seed throttle, and marks each watched artist due again."""
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute("DELETE FROM \"DiscoverySuggestion\" WHERE status = 'pending'")
|
||||||
|
cur.execute('DELETE FROM "DiscoverySeedContribution"')
|
||||||
|
cur.execute('DELETE FROM "DiscoverySeed"')
|
||||||
|
cur.execute('UPDATE "WatchedArtist" SET "lastDiscoveredAt" = NULL')
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
def run_discovery(conn: psycopg.Connection, sources: list[SimilaritySource],
|
def run_discovery(conn: psycopg.Connection, sources: list[SimilaritySource],
|
||||||
browser: MbBrowser, cfg: DiscoveryConfig) -> DiscoveryResult:
|
browser: MbBrowser, cfg: DiscoveryConfig, popularity=None) -> DiscoveryResult:
|
||||||
"""Record each seed's similar-artist contributions, then recompute affected
|
"""Record each seed's similar-artist contributions, then recompute affected
|
||||||
suggestions from the summed contributions — so a candidate's score is the sum of
|
suggestions from the summed contributions — so a candidate's score is the sum of
|
||||||
every seed's latest contribution, independent of chunk and sweep boundaries."""
|
every seed's latest contribution, independent of chunk and sweep boundaries."""
|
||||||
@@ -304,7 +366,7 @@ def run_discovery(conn: psycopg.Connection, sources: list[SimilaritySource],
|
|||||||
affected |= _record_contributions(conn, seed_mbid, live, known, cfg)
|
affected |= _record_contributions(conn, seed_mbid, live, known, cfg)
|
||||||
|
|
||||||
surfaced = _recompute_suggestions(conn, affected, cfg)
|
surfaced = _recompute_suggestions(conn, affected, cfg)
|
||||||
albums = _derive_albums(conn, browser, surfaced, cfg)
|
albums = _derive_albums(conn, browser, surfaced, cfg, popularity)
|
||||||
|
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
for seed_mbid, _seed_name, source in seeds:
|
for seed_mbid, _seed_name, source in seeds:
|
||||||
|
|||||||
@@ -0,0 +1,42 @@
|
|||||||
|
import os
|
||||||
|
|
||||||
|
from lyra_worker.config import get_config
|
||||||
|
|
||||||
|
_TRUE = {"1", "true", "yes", "on"}
|
||||||
|
|
||||||
|
|
||||||
|
def press_paused(config: dict[str, str]) -> bool:
|
||||||
|
"""True when the whole press is paused — the worker stops claiming new jobs."""
|
||||||
|
return str(config.get("floor.paused", "")).strip().lower() in _TRUE
|
||||||
|
|
||||||
|
|
||||||
|
def job_delay_seconds(config: dict[str, str]) -> float:
|
||||||
|
"""Seconds to wait between finishing one download and claiming the next. Invalid or
|
||||||
|
absent -> 0.0 (back-to-back). Negatives are clamped to 0."""
|
||||||
|
try:
|
||||||
|
return max(0.0, float(int(config.get("floor.jobDelaySeconds", "") or 0)))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def kill_requested(config: dict[str, str]) -> bool:
|
||||||
|
"""True when a force-stop of the in-flight download has been requested."""
|
||||||
|
return str(config.get("floor.killRequested", "")).strip().lower() in _TRUE
|
||||||
|
|
||||||
|
|
||||||
|
def consume_kill_switch(conn, exit_fn=os._exit) -> bool:
|
||||||
|
"""If a force-stop is pending, clear the flag and terminate the process (default
|
||||||
|
os._exit) so the in-flight download is aborted. The container restart policy brings the
|
||||||
|
worker back; startup clears staging and requeues the aborted job. Returns whether it
|
||||||
|
fired. exit_fn is injectable for tests."""
|
||||||
|
config = get_config(conn)
|
||||||
|
if not kill_requested(config):
|
||||||
|
return False
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'UPDATE "Config" SET value = \'false\', "updatedAt" = now() WHERE key = %s',
|
||||||
|
("floor.killRequested",),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
exit_fn(0)
|
||||||
|
return True
|
||||||
@@ -6,16 +6,17 @@ import uuid
|
|||||||
|
|
||||||
import psycopg
|
import psycopg
|
||||||
|
|
||||||
from lyra_worker.claim import claim_next, reclaim_stuck_jobs
|
from lyra_worker.claim import claim_next, reclaim_stuck_jobs, requeue_or_fail
|
||||||
from lyra_worker.config import get_config
|
from lyra_worker.config import get_config
|
||||||
from lyra_worker.crypto import secret_key_problem
|
from lyra_worker.crypto import secret_key_problem
|
||||||
from lyra_worker.db import wait_for_db
|
from lyra_worker.db import wait_for_db
|
||||||
from lyra_worker.discovery import DiscoveryConfig, DiscoveryResult, run_discovery
|
from lyra_worker.discovery import DiscoveryConfig, DiscoveryResult, reset_pending, run_discovery
|
||||||
|
from lyra_worker.floor import press_paused, job_delay_seconds, consume_kill_switch
|
||||||
from lyra_worker.library import clear_staging_root
|
from lyra_worker.library import clear_staging_root
|
||||||
from lyra_worker.monitor import MonitorConfig, fulfill_owned_releases, reconcile, sweep
|
from lyra_worker.monitor import MonitorConfig, fulfill_owned_releases, reconcile, sweep
|
||||||
from lyra_worker.pipeline import run_pipeline
|
from lyra_worker.pipeline import run_pipeline
|
||||||
from lyra_worker.registry import (
|
from lyra_worker.registry import (
|
||||||
build_adapters, build_browser, build_probe, build_resolver,
|
build_adapters, build_album_popularity, build_browser, build_probe, build_resolver,
|
||||||
build_similarity_sources, build_tagger,
|
build_similarity_sources, build_tagger,
|
||||||
)
|
)
|
||||||
from lyra_worker.scan import build_worklist, clear_worklist, scan_chunk
|
from lyra_worker.scan import build_worklist, clear_worklist, scan_chunk
|
||||||
@@ -115,8 +116,10 @@ def maybe_run_scan(conn, resolver, probe, browser, config, dest_root: str = DEST
|
|||||||
|
|
||||||
|
|
||||||
def _run_discovery(conn, sources, browser, config=None) -> DiscoveryResult:
|
def _run_discovery(conn, sources, browser, config=None) -> DiscoveryResult:
|
||||||
cfg = DiscoveryConfig.from_config(config if config is not None else get_config(conn))
|
resolved = config if config is not None else get_config(conn)
|
||||||
return run_discovery(conn, sources, browser, cfg)
|
cfg = DiscoveryConfig.from_config(resolved)
|
||||||
|
popularity = build_album_popularity(resolved)
|
||||||
|
return run_discovery(conn, sources, browser, cfg, popularity=popularity)
|
||||||
|
|
||||||
|
|
||||||
def finish_job(conn, job_id, mcfg) -> None:
|
def finish_job(conn, job_id, mcfg) -> None:
|
||||||
@@ -145,6 +148,10 @@ def maybe_run_discovery(conn, sources, browser, config, dcfg, now, last_discover
|
|||||||
|
|
||||||
starting = not in_progress
|
starting = not in_progress
|
||||||
if starting: # begin a fresh sweep
|
if starting: # begin a fresh sweep
|
||||||
|
if str(config.get("discover.resetRequested", "")).strip().lower() in _TRUE:
|
||||||
|
reset_pending(conn) # wipe stale pending suggestions before rebuilding
|
||||||
|
_set_config(conn, "discover.resetRequested", "false")
|
||||||
|
print("worker: discovery reset — cleared pending suggestions, rebuilding", flush=True)
|
||||||
_set_config(conn, "discover.inProgress", "true")
|
_set_config(conn, "discover.inProgress", "true")
|
||||||
_set_config(conn, "discover.requested", "false")
|
_set_config(conn, "discover.requested", "false")
|
||||||
|
|
||||||
@@ -220,9 +227,30 @@ def _liveness_heartbeat() -> None:
|
|||||||
time.sleep(HEARTBEAT_SECONDS)
|
time.sleep(HEARTBEAT_SECONDS)
|
||||||
|
|
||||||
|
|
||||||
|
KILL_POLL_SECONDS = 1.5
|
||||||
|
|
||||||
|
|
||||||
|
def _kill_switch_watch() -> None:
|
||||||
|
"""Poll floor.killRequested on a background daemon thread (its own DB connection) so a
|
||||||
|
force-stop aborts the process even while the main loop is blocked in a download. On any
|
||||||
|
DB hiccup, reconnect and keep watching."""
|
||||||
|
conn = wait_for_db()
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
consume_kill_switch(conn)
|
||||||
|
except Exception:
|
||||||
|
try:
|
||||||
|
conn.close()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
conn = wait_for_db()
|
||||||
|
time.sleep(KILL_POLL_SECONDS)
|
||||||
|
|
||||||
|
|
||||||
def run_forever() -> None:
|
def run_forever() -> None:
|
||||||
_require_secret_key()
|
_require_secret_key()
|
||||||
threading.Thread(target=_liveness_heartbeat, daemon=True).start()
|
threading.Thread(target=_liveness_heartbeat, daemon=True).start()
|
||||||
|
threading.Thread(target=_kill_switch_watch, daemon=True).start()
|
||||||
conn = wait_for_db()
|
conn = wait_for_db()
|
||||||
clear_staging_root(STAGING_ROOT) # sweep any staging dirs orphaned by a prior crash
|
clear_staging_root(STAGING_ROOT) # sweep any staging dirs orphaned by a prior crash
|
||||||
reclaimed = reclaim_stuck_jobs(conn) # requeue jobs a prior crash left mid-pipeline
|
reclaimed = reclaim_stuck_jobs(conn) # requeue jobs a prior crash left mid-pipeline
|
||||||
@@ -246,6 +274,7 @@ def run_forever() -> None:
|
|||||||
last_discover_tick = 0.0
|
last_discover_tick = 0.0
|
||||||
last_heartbeat = 0.0
|
last_heartbeat = 0.0
|
||||||
last_prune = 0.0
|
last_prune = 0.0
|
||||||
|
next_job_allowed = 0.0 # monotonic time before which no new job is claimed (inter-job delay)
|
||||||
try:
|
try:
|
||||||
while True:
|
while True:
|
||||||
# A transient DB connection loss (e.g. AdminShutdown when Postgres restarts) raises
|
# A transient DB connection loss (e.g. AdminShutdown when Postgres restarts) raises
|
||||||
@@ -277,6 +306,12 @@ def run_forever() -> None:
|
|||||||
|
|
||||||
maybe_run_scan(conn, resolver, probe, browser, config)
|
maybe_run_scan(conn, resolver, probe, browser, config)
|
||||||
|
|
||||||
|
# Press paused (downloads only) or the inter-job delay hasn't elapsed: keep the
|
||||||
|
# loop alive (monitor/scan/discovery above still run) but claim nothing.
|
||||||
|
if press_paused(config) or now < next_job_allowed:
|
||||||
|
time.sleep(IDLE_SLEEP)
|
||||||
|
continue
|
||||||
|
|
||||||
job_id = claim_next(conn)
|
job_id = claim_next(conn)
|
||||||
if job_id is None:
|
if job_id is None:
|
||||||
time.sleep(IDLE_SLEEP)
|
time.sleep(IDLE_SLEEP)
|
||||||
@@ -288,8 +323,12 @@ def run_forever() -> None:
|
|||||||
except Exception as e: # one bad job must not take down the worker loop
|
except Exception as e: # one bad job must not take down the worker loop
|
||||||
print(f"worker: pipeline failed for job {job_id}: {e}", flush=True)
|
print(f"worker: pipeline failed for job {job_id}: {e}", flush=True)
|
||||||
conn.rollback()
|
conn.rollback()
|
||||||
|
# Don't orphan the job in its mid-pipeline 'matching' state (claim_next only
|
||||||
|
# picks 'requested'): requeue for a bounded retry, then needs_attention.
|
||||||
|
requeue_or_fail(conn, job_id, str(e))
|
||||||
finish_job(conn, job_id, mcfg)
|
finish_job(conn, job_id, mcfg)
|
||||||
print(f"worker: finished job {job_id}", flush=True)
|
print(f"worker: finished job {job_id}", flush=True)
|
||||||
|
next_job_allowed = time.monotonic() + job_delay_seconds(config)
|
||||||
except psycopg.OperationalError as e:
|
except psycopg.OperationalError as e:
|
||||||
print(f"worker: db connection lost ({e}); reconnecting...", flush=True)
|
print(f"worker: db connection lost ({e}); reconnecting...", flush=True)
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -86,16 +86,40 @@ def _set_state(conn: psycopg.Connection, rel_id: str, state: str) -> None:
|
|||||||
cur.execute('UPDATE "MonitoredRelease" SET state = %s WHERE id = %s', (state, rel_id))
|
cur.execute('UPDATE "MonitoredRelease" SET state = %s WHERE id = %s', (state, rel_id))
|
||||||
|
|
||||||
|
|
||||||
def _in_library_at_cutoff(conn: psycopg.Connection, artist: str, album: str, cutoff: int) -> bool:
|
def _in_library_at_cutoff(
|
||||||
|
conn: psycopg.Connection, artist: str, album: str, rg_mbid: str, cutoff: int
|
||||||
|
) -> bool:
|
||||||
|
"""True if the album is owned at/above `cutoff`. Matches on the release-group MBID when the
|
||||||
|
library row has one — robust against MusicBrainz artist-name canonicalization (e.g. the
|
||||||
|
followed 'Kanye West' imports as 'Ye') — and falls back to exact artist+album for scanned
|
||||||
|
rows that predate MBID capture."""
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
'SELECT 1 FROM "LibraryItem" WHERE artist = %s AND album = %s '
|
'SELECT 1 FROM "LibraryItem" WHERE "qualityClass" >= %s AND ('
|
||||||
'AND "qualityClass" >= %s LIMIT 1',
|
' ("rgMbid" IS NOT NULL AND "rgMbid" = %s) '
|
||||||
(artist, album, cutoff),
|
' OR ("rgMbid" IS NULL AND artist = %s AND album = %s)) LIMIT 1',
|
||||||
|
(cutoff, rg_mbid, artist, album),
|
||||||
)
|
)
|
||||||
return cur.fetchone() is not None
|
return cur.fetchone() is not None
|
||||||
|
|
||||||
|
|
||||||
|
def _retire_failed_attempts(conn: psycopg.Connection, rel_id: str) -> None:
|
||||||
|
"""Delete prior failed (needs_attention) Request rows for this release (Job/Candidate cascade).
|
||||||
|
|
||||||
|
A hard-to-match release is re-pressed every retry interval; each failed attempt used to leave
|
||||||
|
a needs_attention Request behind, piling up phantom "needs attention" cards on the press even
|
||||||
|
after a later attempt imported the album. Retiring them when the release is fulfilled or
|
||||||
|
re-attempted keeps the press showing one live attempt at a time. Manual (non-monitor) requests
|
||||||
|
have no monitoredReleaseId and are never touched."""
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'DELETE FROM "Request" r USING "Job" j '
|
||||||
|
'WHERE j."requestId" = r.id AND r."monitoredReleaseId" = %s '
|
||||||
|
"AND j.state = 'needs_attention'",
|
||||||
|
(rel_id,),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _enqueue_one(conn: psycopg.Connection, rel_id: str, artist: str, album: str) -> None:
|
def _enqueue_one(conn: psycopg.Connection, rel_id: str, artist: str, album: str) -> None:
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
@@ -118,7 +142,8 @@ def enqueue_due(conn: psycopg.Connection, cfg: MonitorConfig) -> int:
|
|||||||
cur.execute(
|
cur.execute(
|
||||||
'SELECT mr.id, mr."artistName", mr.album, mr.state, mr."currentQualityClass", '
|
'SELECT mr.id, mr."artistName", mr.album, mr.state, mr."currentQualityClass", '
|
||||||
' (mr."firstGrabbedAt" IS NOT NULL '
|
' (mr."firstGrabbedAt" IS NOT NULL '
|
||||||
' AND mr."firstGrabbedAt" < now() - make_interval(days => %s)) AS window_expired '
|
' AND mr."firstGrabbedAt" < now() - make_interval(days => %s)) AS window_expired, '
|
||||||
|
' mr."rgMbid" '
|
||||||
'FROM "MonitoredRelease" mr '
|
'FROM "MonitoredRelease" mr '
|
||||||
'WHERE mr.monitored = true '
|
'WHERE mr.monitored = true '
|
||||||
' AND NOT mr.ignored ' # user-ignored releases are never enqueued
|
' AND NOT mr.ignored ' # user-ignored releases are never enqueued
|
||||||
@@ -134,8 +159,11 @@ def enqueue_due(conn: psycopg.Connection, cfg: MonitorConfig) -> int:
|
|||||||
rows = cur.fetchall()
|
rows = cur.fetchall()
|
||||||
|
|
||||||
enqueued = 0
|
enqueued = 0
|
||||||
for rel_id, artist, album, state, quality, window_expired in rows:
|
for rel_id, artist, album, state, quality, window_expired, rg_mbid in rows:
|
||||||
if _in_library_at_cutoff(conn, artist, album, cfg.quality_cutoff):
|
# A due release is about to be fulfilled or re-attempted; either way any earlier failed
|
||||||
|
# attempt is superseded, so retire it before deciding.
|
||||||
|
_retire_failed_attempts(conn, rel_id)
|
||||||
|
if _in_library_at_cutoff(conn, artist, album, rg_mbid, cfg.quality_cutoff):
|
||||||
_set_state(conn, rel_id, "fulfilled")
|
_set_state(conn, rel_id, "fulfilled")
|
||||||
continue
|
continue
|
||||||
if state == "grabbed" and (window_expired or (quality is not None and quality >= cfg.quality_cutoff)):
|
if state == "grabbed" and (window_expired or (quality is not None and quality >= cfg.quality_cutoff)):
|
||||||
@@ -154,12 +182,19 @@ def fulfill_owned_releases(conn: psycopg.Connection, cfg: MonitorConfig) -> int:
|
|||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
'UPDATE "MonitoredRelease" mr SET state = \'fulfilled\', '
|
'UPDATE "MonitoredRelease" mr SET state = \'fulfilled\', '
|
||||||
' "currentQualityClass" = li.q, '
|
' "currentQualityClass" = owned.q, '
|
||||||
' "firstGrabbedAt" = COALESCE(mr."firstGrabbedAt", now()) '
|
' "firstGrabbedAt" = COALESCE(mr."firstGrabbedAt", now()) '
|
||||||
'FROM (SELECT artist, album, max("qualityClass") AS q FROM "LibraryItem" '
|
'FROM ('
|
||||||
' GROUP BY artist, album) li '
|
' SELECT mr2.id AS mid, max(li."qualityClass") AS q '
|
||||||
'WHERE li.artist = mr."artistName" AND li.album = mr.album '
|
' FROM "MonitoredRelease" mr2 JOIN "LibraryItem" li ON ('
|
||||||
" AND mr.monitored = true AND mr.state <> 'fulfilled' AND li.q >= %s",
|
# match on rgMbid where the library row has one (survives artist-name canonicalization),
|
||||||
|
# else fall back to exact artist+album for scanned rows lacking an MBID
|
||||||
|
' (li."rgMbid" IS NOT NULL AND li."rgMbid" = mr2."rgMbid") '
|
||||||
|
' OR (li."rgMbid" IS NULL AND li.artist = mr2."artistName" AND li.album = mr2.album)) '
|
||||||
|
" WHERE mr2.monitored = true AND mr2.state <> 'fulfilled' "
|
||||||
|
' GROUP BY mr2.id HAVING max(li."qualityClass") >= %s'
|
||||||
|
') owned '
|
||||||
|
'WHERE mr.id = owned.mid',
|
||||||
(cfg.quality_cutoff,),
|
(cfg.quality_cutoff,),
|
||||||
)
|
)
|
||||||
fixed = cur.rowcount
|
fixed = cur.rowcount
|
||||||
@@ -171,21 +206,28 @@ def reconcile(conn: psycopg.Connection, job_id: str, cfg: MonitorConfig) -> None
|
|||||||
"""Feed a finished job's outcome back into its MonitoredRelease (if any)."""
|
"""Feed a finished job's outcome back into its MonitoredRelease (if any)."""
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
'SELECT j.state, r."monitoredReleaseId", r.artist, r.album '
|
'SELECT j.state, r."monitoredReleaseId", r.artist, r.album, mr."rgMbid" '
|
||||||
'FROM "Job" j JOIN "Request" r ON r.id = j."requestId" WHERE j.id = %s',
|
'FROM "Job" j JOIN "Request" r ON r.id = j."requestId" '
|
||||||
|
'LEFT JOIN "MonitoredRelease" mr ON mr.id = r."monitoredReleaseId" '
|
||||||
|
'WHERE j.id = %s',
|
||||||
(job_id,),
|
(job_id,),
|
||||||
)
|
)
|
||||||
row = cur.fetchone()
|
row = cur.fetchone()
|
||||||
if row is None:
|
if row is None:
|
||||||
return
|
return
|
||||||
state, rel_id, artist, album = row
|
state, rel_id, artist, album, rg_mbid = row
|
||||||
if rel_id is None or state != "imported":
|
if rel_id is None or state != "imported":
|
||||||
return # needs_attention (or in-flight) leaves the release wanted
|
return # needs_attention (or in-flight) leaves the release wanted
|
||||||
|
|
||||||
|
# A successful import supersedes any earlier failed attempts for this release.
|
||||||
|
_retire_failed_attempts(conn, rel_id)
|
||||||
|
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
'SELECT "qualityClass" FROM "LibraryItem" WHERE artist = %s AND album = %s',
|
'SELECT max("qualityClass") FROM "LibraryItem" WHERE '
|
||||||
(artist, album),
|
' ("rgMbid" IS NOT NULL AND "rgMbid" = %s) '
|
||||||
|
' OR ("rgMbid" IS NULL AND artist = %s AND album = %s)',
|
||||||
|
(rg_mbid, artist, album),
|
||||||
)
|
)
|
||||||
lib = cur.fetchone()
|
lib = cur.fetchone()
|
||||||
quality = lib[0] if lib else None
|
quality = lib[0] if lib else None
|
||||||
|
|||||||
+161
-50
@@ -1,7 +1,9 @@
|
|||||||
import os
|
import os
|
||||||
|
import random
|
||||||
import shutil
|
import shutil
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
|
from concurrent.futures import ThreadPoolExecutor
|
||||||
from dataclasses import replace
|
from dataclasses import replace
|
||||||
from typing import Callable, Sequence
|
from typing import Callable, Sequence
|
||||||
|
|
||||||
@@ -10,6 +12,7 @@ import psycopg
|
|||||||
from lyra_worker.adapters.base import SourceAdapter
|
from lyra_worker.adapters.base import SourceAdapter
|
||||||
from lyra_worker.confidence import score_confidence
|
from lyra_worker.confidence import score_confidence
|
||||||
from lyra_worker.library import album_dir, import_album, list_audio_files, staging_dir
|
from lyra_worker.library import album_dir, import_album, list_audio_files, staging_dir
|
||||||
|
from lyra_worker.qobuz_gate import gate_open, record_download
|
||||||
from lyra_worker.quality import quality_class
|
from lyra_worker.quality import quality_class
|
||||||
from lyra_worker.ranker import rank_candidates
|
from lyra_worker.ranker import rank_candidates
|
||||||
from lyra_worker.types import Candidate, MBTarget
|
from lyra_worker.types import Candidate, MBTarget
|
||||||
@@ -19,8 +22,19 @@ _AUDIO_EXT = {".flac", ".mp3", ".m4a", ".opus", ".ogg", ".aac", ".wav"}
|
|||||||
# Reject a download whose total playtime falls below this fraction of MusicBrainz's expected
|
# Reject a download whose total playtime falls below this fraction of MusicBrainz's expected
|
||||||
# total — catches truncated files / preview clips substituted for real tracks that keep the
|
# total — catches truncated files / preview clips substituted for real tracks that keep the
|
||||||
# track COUNT right. Generous, so edition/encoding differences don't false-positive; bonus
|
# track COUNT right. Generous, so edition/encoding differences don't false-positive; bonus
|
||||||
# tracks (over-long) are always fine.
|
# tracks (over-long) are always fine. Kept well below 1.0 because a source legitimately delivers a
|
||||||
_DURATION_MIN_RATIO = 0.85
|
# different edition than MB's resolved release (shorter radio edits, no bonus/live tracks), and only
|
||||||
|
# a genuinely truncated/preview-filled download (playtime a fraction of expected) should be caught.
|
||||||
|
_DURATION_MIN_RATIO = 0.65
|
||||||
|
# Cap how many ranked candidates a single job will actually download+verify before giving up.
|
||||||
|
# Without this, the incomplete-download fall-through can grind through dozens of sources (a popular
|
||||||
|
# album returns ~200 Soulseek candidates), each a slow attempt that also leaves an abandoned slskd
|
||||||
|
# transfer. The best candidates rank first, so 6 attempts is plenty.
|
||||||
|
_MAX_DOWNLOAD_ATTEMPTS = 6
|
||||||
|
# The best quality any non-Qobuz source offers (Soulseek FLAC = CD-lossless). A library copy at or
|
||||||
|
# above this can only be improved by hi-res Qobuz, so when Qobuz is gated by pacing an upgrade of
|
||||||
|
# such a copy is deferred without even searching.
|
||||||
|
_CD_LOSSLESS_CLASS = 2
|
||||||
|
|
||||||
|
|
||||||
def _measure_staged_duration_s(staging: str) -> float | None:
|
def _measure_staged_duration_s(staging: str) -> float | None:
|
||||||
@@ -69,6 +83,40 @@ def _count_staged_audio(staging: str) -> int:
|
|||||||
return n
|
return n
|
||||||
|
|
||||||
|
|
||||||
|
def _download_problem(result, staging: str, target, winner, measure_duration) -> str | None:
|
||||||
|
"""Why a completed download is unusable — ``"incomplete download"`` / ``"download too short
|
||||||
|
(...)"`` — or None if it's good. Completeness is judged against the CHOSEN SOURCE's own track
|
||||||
|
count (``winner.track_count``), NOT MusicBrainz's canonical release: MB picks an arbitrary
|
||||||
|
releases[0] that is often a deluxe/expanded edition with more tracks (and runtime) than the
|
||||||
|
standard album a source legitimately delivers, so gating on ``target.track_count`` falsely
|
||||||
|
rejects complete standard-edition downloads (The Script "No Sound Without Silence" 11 vs MB 12,
|
||||||
|
Skillet "Unleashed" 12 vs MB's 20-track deluxe). Pure — no DB I/O — so the download loop can
|
||||||
|
call it per candidate and fall through to the next source on a problem."""
|
||||||
|
expected = winner.track_count
|
||||||
|
# (a) Truncated: fewer files than the source promised (a track failed mid-download). Count the
|
||||||
|
# NON-EMPTY audio on disk too: a silently-skipped track leaves a 0-byte placeholder the adapter
|
||||||
|
# still counts. (`staged and ...` keeps fake adapters that stage nothing on the count path.)
|
||||||
|
staged = _count_staged_audio(staging)
|
||||||
|
truncated = result.track_count < expected or bool(staged and staged < expected)
|
||||||
|
# (b) Implausibly small vs MB's album — a lone-track "full album" video / wrong match, as
|
||||||
|
# distinct from a legitimately smaller edition. A real edition keeps more than half MB's tracks.
|
||||||
|
too_small = target.track_count is not None and expected * 2 <= target.track_count
|
||||||
|
if truncated or too_small:
|
||||||
|
return "incomplete download"
|
||||||
|
# Duration: a truncated file or short preview substituted for a real track shows up as playtime
|
||||||
|
# well under the expected total. Scale MB's total to the DELIVERED edition's size so a smaller
|
||||||
|
# edition isn't judged against a larger one's runtime. Only when the total is known (measured is
|
||||||
|
# None for the offline fakes).
|
||||||
|
if target.total_duration_s:
|
||||||
|
expected_total = target.total_duration_s
|
||||||
|
if target.track_count:
|
||||||
|
expected_total *= winner.track_count / target.track_count
|
||||||
|
measured = measure_duration(staging)
|
||||||
|
if measured is not None and measured < expected_total * _DURATION_MIN_RATIO:
|
||||||
|
return f"download too short ({measured / 60:.0f} of ~{expected_total / 60:.0f} min)"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _poll_download_progress(job_id: str, staging: str, expected: int, stop: "threading.Event",
|
def _poll_download_progress(job_id: str, staging: str, expected: int, stop: "threading.Event",
|
||||||
reports: "threading.Event") -> None:
|
reports: "threading.Event") -> None:
|
||||||
"""Until `stop`, periodically write Job.downloadProgress = files-in-staging / expected
|
"""Until `stop`, periodically write Job.downloadProgress = files-in-staging / expected
|
||||||
@@ -94,6 +142,16 @@ def _poll_download_progress(job_id: str, staging: str, expected: int, stop: "thr
|
|||||||
conn.close()
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _search_adapter(adapter: SourceAdapter, target: MBTarget) -> list[Candidate]:
|
||||||
|
"""Search one adapter, swallowing failures — a down source contributes no candidates rather
|
||||||
|
than crashing the job. Runs on a worker thread (searches are parallelized across sources)."""
|
||||||
|
try:
|
||||||
|
return list(adapter.search(target))
|
||||||
|
except Exception as e:
|
||||||
|
print(f"pipeline: adapter {adapter.name} search failed: {e}", flush=True)
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
def _set_state(conn: psycopg.Connection, job_id: str, state: str, stage: str) -> None:
|
def _set_state(conn: psycopg.Connection, job_id: str, state: str, stage: str) -> None:
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
@@ -103,30 +161,42 @@ def _set_state(conn: psycopg.Connection, job_id: str, state: str, stage: str) ->
|
|||||||
conn.commit()
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
def _set_download_progress(conn: psycopg.Connection, job_id: str, frac: float) -> None:
|
def _set_download_progress(
|
||||||
|
conn: psycopg.Connection, job_id: str, frac: float, eta_seconds: int | None = None
|
||||||
|
) -> None:
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute('UPDATE "Job" SET "downloadProgress" = %s WHERE id = %s', (frac, job_id))
|
cur.execute(
|
||||||
|
'UPDATE "Job" SET "downloadProgress" = %s, "downloadEtaSeconds" = %s WHERE id = %s',
|
||||||
|
(frac, eta_seconds, job_id),
|
||||||
|
)
|
||||||
conn.commit()
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
def _make_on_progress(conn: psycopg.Connection, job_id: str, reports: "threading.Event"):
|
def _make_on_progress(conn: psycopg.Connection, job_id: str, reports: "threading.Event"):
|
||||||
"""Build the on_progress(pct) callback threaded into adapter.download. It marks the
|
"""Build the on_progress(pct, eta_seconds=None) callback threaded into adapter.download. It
|
||||||
adapter as reporting real byte-level progress (so the file-count poller yields) and
|
marks the adapter as reporting real byte-level progress (so the file-count poller yields) and
|
||||||
throttle-writes Job.downloadProgress. Called synchronously from adapter.download on the
|
throttle-writes Job.downloadProgress (+ the download ETA when the adapter measures one — slskd
|
||||||
pipeline's own thread, so it safely reuses `conn`."""
|
does; others pass None). Called synchronously from adapter.download on the pipeline's own
|
||||||
|
thread, so it safely reuses `conn`."""
|
||||||
state = {"frac": -1.0, "t": 0.0}
|
state = {"frac": -1.0, "t": 0.0}
|
||||||
|
|
||||||
def on_progress(pct: float) -> None:
|
def on_progress(pct: float, eta_seconds: int | None = None) -> None:
|
||||||
reports.set()
|
reports.set()
|
||||||
frac = 0.0 if pct < 0 else 1.0 if pct > 1 else float(pct)
|
frac = 0.0 if pct < 0 else 1.0 if pct > 1 else float(pct)
|
||||||
now = time.monotonic()
|
now = time.monotonic()
|
||||||
# throttle DB writes: on a >=1% move or every 0.5s, not once per received byte
|
# throttle DB writes: on a >=1% move, a new ETA, or every 0.5s — not once per received byte
|
||||||
if frac - state["frac"] >= 0.01 or (now - state["t"]) >= 0.5:
|
if frac - state["frac"] >= 0.01 or eta_seconds is not None or (now - state["t"]) >= 0.5:
|
||||||
state["frac"] = frac
|
state["frac"] = frac
|
||||||
state["t"] = now
|
state["t"] = now
|
||||||
try:
|
try:
|
||||||
_set_download_progress(conn, job_id, frac)
|
_set_download_progress(conn, job_id, frac, eta_seconds)
|
||||||
except Exception as e: # a progress write must never fail the download
|
except Exception as e: # a progress write must never fail the download
|
||||||
|
# Roll back so a failed write can't leave the shared pipeline conn in an aborted
|
||||||
|
# txn (which would cascade "current transaction is aborted" into every later query).
|
||||||
|
try:
|
||||||
|
conn.rollback()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
print(f"pipeline: on_progress write failed: {e}", flush=True)
|
print(f"pipeline: on_progress write failed: {e}", flush=True)
|
||||||
|
|
||||||
return on_progress
|
return on_progress
|
||||||
@@ -214,6 +284,17 @@ def _library_quality(conn: psycopg.Connection, target: MBTarget) -> int | None:
|
|||||||
return row[0] if row else None
|
return row[0] if row else None
|
||||||
|
|
||||||
|
|
||||||
|
def _keep_existing_and_complete(conn: psycopg.Connection, job_id: str) -> None:
|
||||||
|
"""Finish a job without importing — keep the existing library copy — and mark its request
|
||||||
|
completed. Used when a (deferred) upgrade has nothing better to offer right now; the release
|
||||||
|
stays below the cutoff and is re-attempted later."""
|
||||||
|
_set_state(conn, job_id, "imported", "import")
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute("UPDATE \"Request\" SET status = 'completed' WHERE id = %s",
|
||||||
|
(_request_id(conn, job_id),))
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
def _persist_candidates(conn: psycopg.Connection, job_id: str, candidates: list[Candidate]) -> None:
|
def _persist_candidates(conn: psycopg.Connection, job_id: str, candidates: list[Candidate]) -> None:
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
for c in candidates:
|
for c in candidates:
|
||||||
@@ -306,19 +387,43 @@ def run_pipeline(
|
|||||||
conn.commit()
|
conn.commit()
|
||||||
return
|
return
|
||||||
|
|
||||||
# 2. match — search all adapters; score + persist EVERY candidate found
|
# Pre-search upgrade skip: a >= CD-lossless copy can only be improved by hi-res Qobuz. If Qobuz
|
||||||
|
# is gated by pacing, don't even search (no other source can beat it) — defer to when Qobuz
|
||||||
|
# frees up. This avoids re-searching every already-owned album (~2 min Soulseek search each) on
|
||||||
|
# every monitor cycle. A user-forced re-acquire always proceeds.
|
||||||
|
_existing_q = _library_quality(conn, target)
|
||||||
|
if _existing_q is not None and _existing_q >= _CD_LOSSLESS_CLASS \
|
||||||
|
and not _is_force_job(conn, job_id) and not gate_open(conn):
|
||||||
|
_keep_existing_and_complete(conn, job_id)
|
||||||
|
return
|
||||||
|
|
||||||
|
# 2. match — search all adapters CONCURRENTLY (Soulseek's search polls up to ~2 min; run it
|
||||||
|
# alongside Qobuz's instant search so total match time = the slowest source, not their sum).
|
||||||
|
# Only Qobuz touches streamrip's shared asyncio loop and there's a single Qobuz adapter, so no
|
||||||
|
# two threads drive that loop at once. Results are collected in adapter order (map preserves
|
||||||
|
# input order) so candidate ordering stays deterministic.
|
||||||
_set_state(conn, job_id, "matching", "match")
|
_set_state(conn, job_id, "matching", "match")
|
||||||
found: list[Candidate] = []
|
found: list[Candidate] = []
|
||||||
for adapter in adapters:
|
with ThreadPoolExecutor(max_workers=max(1, len(adapters))) as pool:
|
||||||
try:
|
per_adapter = list(pool.map(lambda a: (a, _search_adapter(a, target)), adapters))
|
||||||
results = adapter.search(target)
|
for adapter, results in per_adapter:
|
||||||
except Exception as e: # a down source contributes no candidates, never crashes the job
|
|
||||||
print(f"pipeline: adapter {adapter.name} search failed: {e}", flush=True)
|
|
||||||
continue
|
|
||||||
for c in results:
|
for c in results:
|
||||||
found.append(replace(c, source_tier=adapter.tier, confidence=score_confidence(target, c)))
|
found.append(replace(c, source_tier=adapter.tier, confidence=score_confidence(target, c)))
|
||||||
|
# Drop candidates that can't be the full album — a result with far fewer tracks than the release
|
||||||
|
# has (e.g. a single-song Soulseek folder named like the album). Same threshold as the
|
||||||
|
# completeness 'too small' guard, applied up front so we never waste a download attempt on it.
|
||||||
|
# Unknown counts (0) and legitimately smaller editions (e.g. 11 vs 12) are kept.
|
||||||
|
if target.track_count:
|
||||||
|
found = [c for c in found if c.track_count == 0 or c.track_count * 2 > target.track_count]
|
||||||
_persist_candidates(conn, job_id, found)
|
_persist_candidates(conn, job_id, found)
|
||||||
|
|
||||||
|
# Qobuz pacing: when the budget gate is closed (off-hours / daily cap / spacing), drop Qobuz
|
||||||
|
# candidates so the job falls through to another source and Qobuz stays under the radar — UNLESS
|
||||||
|
# Qobuz is the only option (nothing to fall through to, so don't strand the album).
|
||||||
|
if any(c.source == "qobuz" for c in found) and any(c.source != "qobuz" for c in found):
|
||||||
|
if not gate_open(conn):
|
||||||
|
found = [c for c in found if c.source != "qobuz"]
|
||||||
|
|
||||||
# 3. rank
|
# 3. rank
|
||||||
_set_state(conn, job_id, "matched", "rank")
|
_set_state(conn, job_id, "matched", "rank")
|
||||||
ranked = rank_candidates(target, found, min_confidence)
|
ranked = rank_candidates(target, found, min_confidence)
|
||||||
@@ -326,12 +431,23 @@ def run_pipeline(
|
|||||||
_fail(conn, job_id, "no candidate above confidence threshold")
|
_fail(conn, job_id, "no candidate above confidence threshold")
|
||||||
return
|
return
|
||||||
|
|
||||||
|
# Upgrade guard: for an album already in the library, skip the download when no currently-
|
||||||
|
# available source beats the copy we have (ranked[0] is the highest-quality candidate). This
|
||||||
|
# happens when Qobuz is gated by pacing and only a same-quality Soulseek copy is left — the
|
||||||
|
# download would be discarded by keep-if-better, so defer it. The release stays below the cutoff
|
||||||
|
# and is re-attempted when a better source frees up. A user-forced re-acquire always proceeds.
|
||||||
|
existing_q = _library_quality(conn, target)
|
||||||
|
if existing_q is not None and not _is_force_job(conn, job_id) \
|
||||||
|
and quality_class(ranked[0].quality) <= existing_q:
|
||||||
|
_keep_existing_and_complete(conn, job_id) # keep the existing copy; nothing better available
|
||||||
|
return
|
||||||
|
|
||||||
# 4. download (fall-through) into an isolated per-job staging dir
|
# 4. download (fall-through) into an isolated per-job staging dir
|
||||||
_set_state(conn, job_id, "downloading", "download")
|
_set_state(conn, job_id, "downloading", "download")
|
||||||
by_source = {a.name: a for a in adapters}
|
by_source = {a.name: a for a in adapters}
|
||||||
staging = staging_dir(staging_root or f"{dest_root}/.staging", job_id)
|
staging = staging_dir(staging_root or f"{dest_root}/.staging", job_id)
|
||||||
winner = None
|
winner = None
|
||||||
result = None
|
last_problem = None # why the most recent downloaded-but-rejected candidate was unusable
|
||||||
_set_download_progress(conn, job_id, 0.0) # reset for this run
|
_set_download_progress(conn, job_id, 0.0) # reset for this run
|
||||||
expected = target.track_count or (ranked[0].track_count if ranked else 0)
|
expected = target.track_count or (ranked[0].track_count if ranked else 0)
|
||||||
_stop = threading.Event()
|
_stop = threading.Event()
|
||||||
@@ -342,46 +458,46 @@ def run_pipeline(
|
|||||||
_poller.start()
|
_poller.start()
|
||||||
try:
|
try:
|
||||||
try:
|
try:
|
||||||
|
attempts = 0
|
||||||
for candidate in ranked:
|
for candidate in ranked:
|
||||||
adapter = by_source.get(candidate.source)
|
adapter = by_source.get(candidate.source)
|
||||||
if adapter is None:
|
if adapter is None:
|
||||||
continue
|
continue
|
||||||
|
if attempts >= _MAX_DOWNLOAD_ATTEMPTS:
|
||||||
|
break # don't grind through every peer/source of a popular album
|
||||||
|
attempts += 1
|
||||||
_mark_chosen(conn, job_id, candidate.source_ref) # reflect the source now in flight
|
_mark_chosen(conn, job_id, candidate.source_ref) # reflect the source now in flight
|
||||||
shutil.rmtree(staging, ignore_errors=True) # clean slate per attempt
|
shutil.rmtree(staging, ignore_errors=True) # clean slate per attempt
|
||||||
_reports.clear() # this attempt hasn't reported yet → poller estimates until it does
|
_reports.clear() # this attempt hasn't reported yet → poller estimates until it does
|
||||||
result = adapter.download(candidate, staging, _make_on_progress(conn, job_id, _reports))
|
result = adapter.download(candidate, staging, _make_on_progress(conn, job_id, _reports))
|
||||||
if result.ok:
|
if not result.ok:
|
||||||
|
continue
|
||||||
|
# Verify completeness on the staging copy. A download that lands incomplete or too
|
||||||
|
# short falls through to the NEXT-ranked source (e.g. Qobuz repeatedly fails one
|
||||||
|
# track → try the deluxe edition or Soulseek) rather than failing the whole job on
|
||||||
|
# the first source's shortfall.
|
||||||
|
last_problem = _download_problem(result, staging, target, candidate, measure_duration)
|
||||||
|
if last_problem is None:
|
||||||
winner = candidate
|
winner = candidate
|
||||||
break
|
break
|
||||||
finally:
|
finally:
|
||||||
_stop.set()
|
_stop.set()
|
||||||
_poller.join(timeout=3)
|
_poller.join(timeout=3)
|
||||||
if winner is None or result is None or not result.ok:
|
if winner is None:
|
||||||
_fail(conn, job_id, "all downloads failed")
|
# No source produced a complete album: surface why the last one was rejected (incomplete
|
||||||
|
# / too short), or "all downloads failed" if none even downloaded.
|
||||||
|
_fail(conn, job_id, last_problem or "all downloads failed")
|
||||||
return
|
return
|
||||||
_set_download_progress(conn, job_id, 1.0) # download done — UI shows 100% into Finishing
|
_set_download_progress(conn, job_id, 1.0) # download done — UI shows 100% into Finishing
|
||||||
|
if winner.source == "qobuz":
|
||||||
|
# Count this Qobuz download against today's budget and set the next randomized spacing.
|
||||||
|
try:
|
||||||
|
record_download(conn, random.random())
|
||||||
|
except Exception as e: # pacing bookkeeping must never fail a good download
|
||||||
|
print(f"pipeline: qobuz pacing record failed for job {job_id}: {e}", flush=True)
|
||||||
|
|
||||||
# 5. verify completeness on the staging copy, then promote + tag
|
# 5. promote + tag the verified winner
|
||||||
_set_state(conn, job_id, "tagging", "tag")
|
_set_state(conn, job_id, "tagging", "tag")
|
||||||
expected = target.track_count if target.track_count is not None else winner.track_count
|
|
||||||
# Also count the NON-EMPTY audio actually on disk: a silently-skipped track leaves a
|
|
||||||
# 0-byte placeholder that the adapter still counts, so trust the files when any are
|
|
||||||
# staged. (`staged and ...` keeps fake adapters that stage nothing on the count path.)
|
|
||||||
staged = _count_staged_audio(staging)
|
|
||||||
if result.track_count < expected or (staged and staged < expected):
|
|
||||||
_fail(conn, job_id, "incomplete download")
|
|
||||||
return
|
|
||||||
|
|
||||||
# Duration check: even with the right track count, a truncated file or a short preview
|
|
||||||
# substituted for a real track shows up as a total playtime well under MB's expected
|
|
||||||
# total. Only applies when both are known (measured is None for the offline fakes).
|
|
||||||
if target.total_duration_s:
|
|
||||||
measured = measure_duration(staging)
|
|
||||||
if measured is not None and measured < target.total_duration_s * _DURATION_MIN_RATIO:
|
|
||||||
_fail(conn, job_id,
|
|
||||||
f"download too short ({measured / 60:.0f} of ~{target.total_duration_s / 60:.0f} min)")
|
|
||||||
return
|
|
||||||
|
|
||||||
final = album_dir(dest_root, target)
|
final = album_dir(dest_root, target)
|
||||||
existing_q = _library_quality(conn, target)
|
existing_q = _library_quality(conn, target)
|
||||||
new_q = quality_class(winner.quality)
|
new_q = quality_class(winner.quality)
|
||||||
@@ -395,12 +511,7 @@ def run_pipeline(
|
|||||||
print(f"pipeline: tagging failed for job {job_id}: {e}", flush=True)
|
print(f"pipeline: tagging failed for job {job_id}: {e}", flush=True)
|
||||||
_import(conn, job_id, target, winner, final)
|
_import(conn, job_id, target, winner, final)
|
||||||
else:
|
else:
|
||||||
# an existing copy is already at >= this quality — keep it untouched, just complete the request
|
# an existing copy is already at >= this quality — keep it untouched, just complete
|
||||||
with conn.cursor() as cur:
|
_keep_existing_and_complete(conn, job_id)
|
||||||
cur.execute(
|
|
||||||
"UPDATE \"Request\" SET status = 'completed' WHERE id = %s",
|
|
||||||
(_request_id(conn, job_id),),
|
|
||||||
)
|
|
||||||
conn.commit()
|
|
||||||
finally:
|
finally:
|
||||||
shutil.rmtree(staging, ignore_errors=True) # a partial/failed download never persists
|
shutil.rmtree(staging, ignore_errors=True) # a partial/failed download never persists
|
||||||
|
|||||||
@@ -0,0 +1,159 @@
|
|||||||
|
"""Qobuz pacing ("budget gate").
|
||||||
|
|
||||||
|
Keep using Qobuz for hi-res, but at a human-looking volume and cadence so the account isn't
|
||||||
|
flagged for bulk downloading (which got it 403 USER_BLOCKED). One gate decides, per album,
|
||||||
|
whether Qobuz is an eligible source right now: within active hours, under today's cap (a warm-up
|
||||||
|
ramp of a steady cap), and past a randomized spacing since the last Qobuz download. When the gate
|
||||||
|
is closed the pipeline drops Qobuz candidates and falls through to the next source; monitored
|
||||||
|
albums upgrade to Qobuz later once budget frees up.
|
||||||
|
|
||||||
|
Pure decision logic (`today_cap`, `is_allowed`, `next_gap_seconds`) is separated from the two DB
|
||||||
|
helpers (`gate_open`, `record_download`) so the policy is unit-tested without a clock or DB.
|
||||||
|
"""
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import date
|
||||||
|
|
||||||
|
_TRUE = {"1", "true", "yes", "on"}
|
||||||
|
_JITTER = 0.3 # ± fraction around the spread interval
|
||||||
|
_GAP_FLOOR_S = 60.0
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class QobuzPacing:
|
||||||
|
enabled: bool = True
|
||||||
|
active_start: int = 8
|
||||||
|
active_end: int = 23
|
||||||
|
daily_cap: int = 40
|
||||||
|
warmup_start_date: str = "" # ISO "YYYY-MM-DD", or "" for no warm-up ramp
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_config(cls, config: dict) -> "QobuzPacing":
|
||||||
|
def _int(key, default):
|
||||||
|
try:
|
||||||
|
return int(config[key])
|
||||||
|
except (KeyError, TypeError, ValueError):
|
||||||
|
return default
|
||||||
|
|
||||||
|
return cls(
|
||||||
|
enabled=str(config.get("qobuz.pacing.enabled", "true")).strip().lower() in _TRUE,
|
||||||
|
active_start=_int("qobuz.pacing.activeStartHour", 8),
|
||||||
|
active_end=_int("qobuz.pacing.activeEndHour", 23),
|
||||||
|
daily_cap=_int("qobuz.pacing.dailyCap", 40),
|
||||||
|
warmup_start_date=str(config.get("qobuz.pacing.warmupStartDate", "") or "").strip(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def today_cap(pacing: QobuzPacing, today: date) -> int:
|
||||||
|
"""Today's Qobuz download cap: the steady cap, ramped up by account age (days since
|
||||||
|
``warmup_start_date``). No warm-up date -> steady cap. A small steady cap is never exceeded."""
|
||||||
|
steady = max(0, pacing.daily_cap)
|
||||||
|
if not pacing.warmup_start_date:
|
||||||
|
return steady
|
||||||
|
try:
|
||||||
|
start = date.fromisoformat(pacing.warmup_start_date)
|
||||||
|
except ValueError:
|
||||||
|
return steady
|
||||||
|
days = max(0, (today - start).days)
|
||||||
|
if days < 3:
|
||||||
|
return min(5, steady)
|
||||||
|
if days < 7:
|
||||||
|
return min(10, steady)
|
||||||
|
if days < 14:
|
||||||
|
return min(20, steady)
|
||||||
|
return steady
|
||||||
|
|
||||||
|
|
||||||
|
def is_allowed(pacing: QobuzPacing, now_hour: int, count_today: int, cap: int,
|
||||||
|
now_epoch: float, next_allowed_epoch: float) -> bool:
|
||||||
|
"""Pure gate decision. Qobuz is eligible only within active hours, under today's cap, and past
|
||||||
|
the randomized spacing since the last download. Pacing disabled -> always allowed."""
|
||||||
|
if not pacing.enabled:
|
||||||
|
return True
|
||||||
|
if not (pacing.active_start <= now_hour < pacing.active_end):
|
||||||
|
return False
|
||||||
|
if count_today >= cap:
|
||||||
|
return False
|
||||||
|
if now_epoch < next_allowed_epoch:
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def next_gap_seconds(pacing: QobuzPacing, cap: int, rand01: float) -> float:
|
||||||
|
"""Seconds until the next Qobuz download is allowed: the day's budget spread across the active
|
||||||
|
window (``activeWindow / cap``), ± _JITTER, floored at _GAP_FLOOR_S. ``rand01`` in [0,1] is the
|
||||||
|
injected randomness (so tests are deterministic; callers pass random.random())."""
|
||||||
|
window_s = max(1, pacing.active_end - pacing.active_start) * 3600
|
||||||
|
base = window_s / max(1, cap)
|
||||||
|
lo, hi = base * (1 - _JITTER), base * (1 + _JITTER)
|
||||||
|
gap = lo + (hi - lo) * rand01
|
||||||
|
return max(_GAP_FLOOR_S, gap)
|
||||||
|
|
||||||
|
|
||||||
|
def _now_parts(conn):
|
||||||
|
"""(hour:int, today:date, epoch:float) from the DB clock — avoids container-TZ ambiguity."""
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute("SELECT extract(hour from now())::int, current_date, extract(epoch from now())")
|
||||||
|
hour, today, epoch = cur.fetchone()
|
||||||
|
return int(hour), today, float(epoch)
|
||||||
|
|
||||||
|
|
||||||
|
def _read(conn, keys):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT key, value FROM "Config" WHERE key = ANY(%s)', (list(keys),))
|
||||||
|
return dict(cur.fetchall())
|
||||||
|
|
||||||
|
|
||||||
|
def gate_open(conn) -> bool:
|
||||||
|
"""Whether Qobuz may be used for a download right now (reads pacing config + counter/spacing
|
||||||
|
state from Config and the DB clock). A counter from a previous day is treated as 0."""
|
||||||
|
from lyra_worker.config import get_config
|
||||||
|
|
||||||
|
pacing = QobuzPacing.from_config(get_config(conn))
|
||||||
|
hour, today, epoch = _now_parts(conn)
|
||||||
|
state = _read(conn, ("qobuz.pacing.countDay", "qobuz.pacing.countToday", "qobuz.pacing.nextAllowedAt"))
|
||||||
|
count = 0
|
||||||
|
if state.get("qobuz.pacing.countDay") == today.isoformat():
|
||||||
|
try:
|
||||||
|
count = int(state.get("qobuz.pacing.countToday") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
count = 0
|
||||||
|
try:
|
||||||
|
next_allowed = float(state.get("qobuz.pacing.nextAllowedAt") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
next_allowed = 0.0
|
||||||
|
return is_allowed(pacing, hour, count, today_cap(pacing, today), epoch, next_allowed)
|
||||||
|
|
||||||
|
|
||||||
|
def record_download(conn, rand01: float) -> None:
|
||||||
|
"""Record a successful Qobuz download: bump today's counter (resetting it if the stored day is
|
||||||
|
stale) and set the randomized next-allowed time. ``rand01`` is injected randomness."""
|
||||||
|
from lyra_worker.config import get_config
|
||||||
|
|
||||||
|
pacing = QobuzPacing.from_config(get_config(conn))
|
||||||
|
_hour, today, epoch = _now_parts(conn)
|
||||||
|
state = _read(conn, ("qobuz.pacing.countDay", "qobuz.pacing.countToday"))
|
||||||
|
if state.get("qobuz.pacing.countDay") == today.isoformat():
|
||||||
|
try:
|
||||||
|
count = int(state.get("qobuz.pacing.countToday") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
count = 0
|
||||||
|
else:
|
||||||
|
count = 0
|
||||||
|
count += 1
|
||||||
|
next_allowed = epoch + next_gap_seconds(pacing, today_cap(pacing, today), rand01)
|
||||||
|
_write(conn, {
|
||||||
|
"qobuz.pacing.countDay": today.isoformat(),
|
||||||
|
"qobuz.pacing.countToday": str(count),
|
||||||
|
"qobuz.pacing.nextAllowedAt": repr(next_allowed),
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def _write(conn, kv: dict) -> None:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
for key, value in kv.items():
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
|
||||||
|
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
|
||||||
|
(key, value),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
@@ -13,6 +13,7 @@ from lyra_worker.browser import MbBrowser
|
|||||||
from lyra_worker.probe import AudioProbe
|
from lyra_worker.probe import AudioProbe
|
||||||
from lyra_worker.resolver import MbResolver
|
from lyra_worker.resolver import MbResolver
|
||||||
from lyra_worker.similarity._lastfm import LastfmSource
|
from lyra_worker.similarity._lastfm import LastfmSource
|
||||||
|
from lyra_worker.similarity._lastfm_albums import LastfmAlbumPopularity
|
||||||
from lyra_worker.similarity._listenbrainz import ListenBrainzSource
|
from lyra_worker.similarity._listenbrainz import ListenBrainzSource
|
||||||
from lyra_worker.similarity.base import SimilaritySource
|
from lyra_worker.similarity.base import SimilaritySource
|
||||||
from lyra_worker.tagger import Tagger
|
from lyra_worker.tagger import Tagger
|
||||||
@@ -67,3 +68,10 @@ def build_similarity_sources(config: dict, dsn: str | None = None) -> list[Simil
|
|||||||
if key:
|
if key:
|
||||||
sources.append(LastfmSource(api_key=key, browser=build_browser(dsn)))
|
sources.append(LastfmSource(api_key=key, browser=build_browser(dsn)))
|
||||||
return sources
|
return sources
|
||||||
|
|
||||||
|
|
||||||
|
def build_album_popularity(config: dict, dsn: str | None = None):
|
||||||
|
"""Last.fm album-popularity provider for discovery, or None when no Last.fm key is set
|
||||||
|
(discovery then surfaces newest albums only)."""
|
||||||
|
key = config.get("lastfm.api_key")
|
||||||
|
return LastfmAlbumPopularity(api_key=key) if key else None
|
||||||
|
|||||||
@@ -31,6 +31,7 @@ class LastfmSource:
|
|||||||
params={"method": method, "api_key": self._key, "format": "json", **params},
|
params={"method": method, "api_key": self._key, "format": "json", **params},
|
||||||
timeout=self._timeout,
|
timeout=self._timeout,
|
||||||
)
|
)
|
||||||
|
res.raise_for_status() # a 4xx/5xx (rate limit, auth) → RequestException, handled by callers
|
||||||
return res.json()
|
return res.json()
|
||||||
|
|
||||||
def health(self) -> bool:
|
def health(self) -> bool:
|
||||||
|
|||||||
@@ -0,0 +1,54 @@
|
|||||||
|
import requests
|
||||||
|
|
||||||
|
from lyra_worker.apicache import cached_api
|
||||||
|
|
||||||
|
_BASE = "https://ws.audioscrobbler.com/2.0/"
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_top_albums(data) -> list[list]:
|
||||||
|
"""Parse a Last.fm artist.gettopalbums response into [[name, mbid|None, playcount], …]
|
||||||
|
ordered by playcount desc. A single-album body arrives as a dict; an error body → []."""
|
||||||
|
if isinstance(data.get("error"), (int, float)):
|
||||||
|
return []
|
||||||
|
rows = data.get("topalbums", {}).get("album", [])
|
||||||
|
if isinstance(rows, dict):
|
||||||
|
rows = [rows]
|
||||||
|
out: list[list] = []
|
||||||
|
for r in rows:
|
||||||
|
name = r.get("name") or ""
|
||||||
|
if not name:
|
||||||
|
continue
|
||||||
|
out.append([name, r.get("mbid") or None, int(r.get("playcount") or 0)])
|
||||||
|
out.sort(key=lambda a: a[2], reverse=True)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
class LastfmAlbumPopularity:
|
||||||
|
"""Ranks an artist's albums by Last.fm playcount, so discovery can surface each similar
|
||||||
|
artist's most-played album (alongside their newest). Read-through cached in ApiCache; any
|
||||||
|
failure yields [] so the sweep just falls back to newest-only."""
|
||||||
|
|
||||||
|
def __init__(self, api_key, base_url=_BASE, timeout: float = 15.0, limit: int = 25):
|
||||||
|
self._key = api_key
|
||||||
|
self._base = base_url
|
||||||
|
self._timeout = timeout
|
||||||
|
self._limit = limit
|
||||||
|
|
||||||
|
def _fetch(self, artist_name: str) -> list[list]:
|
||||||
|
try:
|
||||||
|
res = requests.get(self._base, params={
|
||||||
|
"method": "artist.gettopalbums", "artist": artist_name,
|
||||||
|
"api_key": self._key, "format": "json", "limit": str(self._limit),
|
||||||
|
"autocorrect": "1",
|
||||||
|
}, timeout=self._timeout)
|
||||||
|
res.raise_for_status()
|
||||||
|
return _parse_top_albums(res.json())
|
||||||
|
except Exception: # a Last.fm outage must never abort the sweep
|
||||||
|
return []
|
||||||
|
|
||||||
|
def top_albums(self, conn, artist_name: str) -> list[list]:
|
||||||
|
norm = (artist_name or "").strip().lower()
|
||||||
|
if not norm:
|
||||||
|
return []
|
||||||
|
return cached_api(conn, f"lfm-artist-top-albums:{norm}", 43200,
|
||||||
|
lambda: self._fetch(artist_name))
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
"""A search result with far fewer tracks than the release (e.g. a single-song Soulseek folder
|
||||||
|
named like the album) can't be the full album — it must be dropped before we waste a download."""
|
||||||
|
import os
|
||||||
|
|
||||||
|
from lyra_worker.claim import claim_next
|
||||||
|
from lyra_worker.pipeline import run_pipeline
|
||||||
|
from lyra_worker.types import Candidate, DownloadResult, MBTarget, Quality
|
||||||
|
from tests.conftest import insert_request
|
||||||
|
|
||||||
|
_Q = Quality(fmt="FLAC", lossless=True, bit_depth=24, sample_rate=96000)
|
||||||
|
|
||||||
|
|
||||||
|
class _TwoCandQobuz:
|
||||||
|
name = "qobuz"
|
||||||
|
tier = 0
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.downloaded = []
|
||||||
|
|
||||||
|
def health(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
return [
|
||||||
|
Candidate(source="qobuz", source_ref="single", matched_artist=target.artist,
|
||||||
|
matched_album=target.album, quality=_Q, track_count=1, source_tier=0),
|
||||||
|
Candidate(source="qobuz", source_ref="full", matched_artist=target.artist,
|
||||||
|
matched_album=target.album, quality=_Q, track_count=12, source_tier=0),
|
||||||
|
]
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
self.downloaded.append(candidate.source_ref)
|
||||||
|
os.makedirs(dest, exist_ok=True)
|
||||||
|
for i in range(candidate.track_count):
|
||||||
|
with open(os.path.join(dest, f"{i + 1:02d}.flac"), "wb") as fh:
|
||||||
|
fh.write(b"a")
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=candidate.track_count)
|
||||||
|
|
||||||
|
|
||||||
|
class _Resolver:
|
||||||
|
def __init__(self, n):
|
||||||
|
self.n = n
|
||||||
|
|
||||||
|
def resolve(self, artist, album):
|
||||||
|
return MBTarget(artist=artist, album=album, track_count=self.n)
|
||||||
|
|
||||||
|
|
||||||
|
def test_single_song_candidate_is_dropped(conn):
|
||||||
|
job_id = insert_request(conn, artist="A", album="B")
|
||||||
|
claim_next(conn)
|
||||||
|
adapter = _TwoCandQobuz()
|
||||||
|
run_pipeline(conn, job_id, [adapter], resolver=_Resolver(12), dest_root="/tmp/lib-cf")
|
||||||
|
|
||||||
|
assert adapter.downloaded == ["full"] # only the full album was ever attempted
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT "trackCount" FROM "Candidate" WHERE "jobId" = %s', (job_id,))
|
||||||
|
assert [r[0] for r in cur.fetchall()] == [12] # the single-song folder wasn't even persisted
|
||||||
@@ -24,3 +24,17 @@ def test_claim_does_not_repick_claimed_job(conn):
|
|||||||
insert_request(conn)
|
insert_request(conn)
|
||||||
assert claim_next(conn) is not None
|
assert claim_next(conn) is not None
|
||||||
assert claim_next(conn) is None
|
assert claim_next(conn) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_claim_skips_paused_jobs(conn):
|
||||||
|
paused_job = insert_request(conn, album="Paused Album")
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('UPDATE "Job" SET paused = true WHERE id = %s', (paused_job,))
|
||||||
|
conn.commit()
|
||||||
|
unpaused_job = insert_request(conn, album="Live Album")
|
||||||
|
|
||||||
|
claimed = claim_next(conn)
|
||||||
|
assert claimed == unpaused_job # the paused one is skipped
|
||||||
|
|
||||||
|
# With only the paused job left, nothing is claimable.
|
||||||
|
assert claim_next(conn) is None
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ def test_seeds_discovery_from_owned_album_artists(conn):
|
|||||||
# An artist you own an album by but don't follow becomes a discovery seed.
|
# An artist you own an album by but don't follow becomes a discovery seed.
|
||||||
_insert_library_item(conn, "Owned Artist", "lib-seed")
|
_insert_library_item(conn, "Owned Artist", "lib-seed")
|
||||||
src = FakeSimilaritySource(similar={"lib-seed": [SimilarArtist("c1", "New Find", 0.9)]})
|
src = FakeSimilaritySource(similar={"lib-seed": [SimilarArtist("c1", "New Find", 0.9)]})
|
||||||
result = run_discovery(conn, [src], FakeMbBrowser(), DiscoveryConfig())
|
result = run_discovery(conn, [src], FakeMbBrowser(default_core=True), DiscoveryConfig())
|
||||||
assert result.seeds == 1
|
assert result.seeds == 1
|
||||||
assert [r[0] for r in _artist_rows(conn)] == ["c1"] # surfaced from the library seed
|
assert [r[0] for r in _artist_rows(conn)] == ["c1"] # surfaced from the library seed
|
||||||
with conn.cursor() as cur: # library seed throttled via DiscoverySeed, not WatchedArtist
|
with conn.cursor() as cur: # library seed throttled via DiscoverySeed, not WatchedArtist
|
||||||
@@ -37,7 +37,7 @@ def test_seeds_discovery_from_owned_album_artists(conn):
|
|||||||
def test_library_seeding_can_be_disabled(conn):
|
def test_library_seeding_can_be_disabled(conn):
|
||||||
_insert_library_item(conn, "Owned Artist", "lib-seed")
|
_insert_library_item(conn, "Owned Artist", "lib-seed")
|
||||||
src = FakeSimilaritySource(similar={"lib-seed": [SimilarArtist("c1", "New Find", 0.9)]})
|
src = FakeSimilaritySource(similar={"lib-seed": [SimilarArtist("c1", "New Find", 0.9)]})
|
||||||
result = run_discovery(conn, [src], FakeMbBrowser(), DiscoveryConfig(seed_from_library=False))
|
result = run_discovery(conn, [src], FakeMbBrowser(default_core=True), DiscoveryConfig(seed_from_library=False))
|
||||||
assert result.seeds == 0 and _artist_rows(conn) == [] # library not seeded
|
assert result.seeds == 0 and _artist_rows(conn) == [] # library not seeded
|
||||||
|
|
||||||
|
|
||||||
@@ -49,7 +49,7 @@ def test_owned_artist_is_not_suggested_as_a_candidate(conn):
|
|||||||
"s1": [SimilarArtist("c1", "Owned", 0.9), SimilarArtist("c2", "New", 0.8)],
|
"s1": [SimilarArtist("c1", "Owned", 0.9), SimilarArtist("c2", "New", 0.8)],
|
||||||
"c1": [], # the library seed itself finds nothing new here
|
"c1": [], # the library seed itself finds nothing new here
|
||||||
})
|
})
|
||||||
run_discovery(conn, [src], FakeMbBrowser(), DiscoveryConfig())
|
run_discovery(conn, [src], FakeMbBrowser(default_core=True), DiscoveryConfig())
|
||||||
assert [r[0] for r in _artist_rows(conn)] == ["c2"] # c1 excluded (already owned)
|
assert [r[0] for r in _artist_rows(conn)] == ["c2"] # c1 excluded (already owned)
|
||||||
|
|
||||||
|
|
||||||
@@ -60,7 +60,7 @@ def test_aggregates_scores_across_seeds(conn):
|
|||||||
"s1": [SimilarArtist("c1", "Shared", 0.6), SimilarArtist("c2", "Only1", 0.4)],
|
"s1": [SimilarArtist("c1", "Shared", 0.6), SimilarArtist("c2", "Only1", 0.4)],
|
||||||
"s2": [SimilarArtist("c1", "Shared", 0.5)],
|
"s2": [SimilarArtist("c1", "Shared", 0.5)],
|
||||||
})
|
})
|
||||||
result = run_discovery(conn, [src], FakeMbBrowser(), DiscoveryConfig())
|
result = run_discovery(conn, [src], FakeMbBrowser(default_core=True), DiscoveryConfig())
|
||||||
assert result.artists == 2
|
assert result.artists == 2
|
||||||
rows = _artist_rows(conn)
|
rows = _artist_rows(conn)
|
||||||
# c1 recommended by both seeds; each seed max-normalizes its top score to 1.0, so
|
# c1 recommended by both seeds; each seed max-normalizes its top score to 1.0, so
|
||||||
@@ -81,7 +81,7 @@ def test_normalizes_across_sources_of_different_scales(conn):
|
|||||||
lfm = FakeSimilaritySource(similar={"s1": [
|
lfm = FakeSimilaritySource(similar={"s1": [
|
||||||
SimilarArtist("c_lfm", "LFM Only", 0.9), SimilarArtist("c_shared", "Shared", 0.5)]})
|
SimilarArtist("c_lfm", "LFM Only", 0.9), SimilarArtist("c_shared", "Shared", 0.5)]})
|
||||||
lfm.name = "lastfm"
|
lfm.name = "lastfm"
|
||||||
run_discovery(conn, [lb, lfm], FakeMbBrowser(), DiscoveryConfig())
|
run_discovery(conn, [lb, lfm], FakeMbBrowser(default_core=True), DiscoveryConfig())
|
||||||
|
|
||||||
lb_only = _artist_score(conn, "c_lb")[0] # 5000/5000 = 1.0
|
lb_only = _artist_score(conn, "c_lb")[0] # 5000/5000 = 1.0
|
||||||
lfm_only = _artist_score(conn, "c_lfm")[0] # 0.9/0.9 = 1.0
|
lfm_only = _artist_score(conn, "c_lfm")[0] # 0.9/0.9 = 1.0
|
||||||
@@ -95,7 +95,7 @@ def test_skips_already_followed_artists(conn):
|
|||||||
insert_watched_artist(conn, mbid="c1", name="Already Followed")
|
insert_watched_artist(conn, mbid="c1", name="Already Followed")
|
||||||
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "X", 0.9),
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "X", 0.9),
|
||||||
SimilarArtist("c2", "New", 0.8)]})
|
SimilarArtist("c2", "New", 0.8)]})
|
||||||
run_discovery(conn, [src], FakeMbBrowser(), DiscoveryConfig())
|
run_discovery(conn, [src], FakeMbBrowser(default_core=True), DiscoveryConfig())
|
||||||
assert [r[0] for r in _artist_rows(conn)] == ["c2"] # c1 filtered (followed)
|
assert [r[0] for r in _artist_rows(conn)] == ["c2"] # c1 filtered (followed)
|
||||||
|
|
||||||
|
|
||||||
@@ -103,7 +103,7 @@ def test_min_score_filters_weak_candidates(conn):
|
|||||||
insert_watched_artist(conn, mbid="s1", name="Seed")
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Strong", 0.9),
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Strong", 0.9),
|
||||||
SimilarArtist("c2", "Weak", 0.1)]})
|
SimilarArtist("c2", "Weak", 0.1)]})
|
||||||
run_discovery(conn, [src], FakeMbBrowser(), DiscoveryConfig(min_score=0.5))
|
run_discovery(conn, [src], FakeMbBrowser(default_core=True), DiscoveryConfig(min_score=0.5))
|
||||||
assert [r[0] for r in _artist_rows(conn)] == ["c1"]
|
assert [r[0] for r in _artist_rows(conn)] == ["c1"]
|
||||||
|
|
||||||
|
|
||||||
@@ -111,7 +111,7 @@ def test_dismissed_suggestion_is_never_revived(conn):
|
|||||||
insert_watched_artist(conn, mbid="s1", name="Seed")
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
insert_discovery_suggestion(conn, kind="artist", artist_mbid="c1", status="dismissed", score=0.0)
|
insert_discovery_suggestion(conn, kind="artist", artist_mbid="c1", status="dismissed", score=0.0)
|
||||||
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "X", 0.9)]})
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "X", 0.9)]})
|
||||||
run_discovery(conn, [src], FakeMbBrowser(), DiscoveryConfig())
|
run_discovery(conn, [src], FakeMbBrowser(default_core=True), DiscoveryConfig())
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute('SELECT status::text, score FROM "DiscoverySuggestion" WHERE "artistMbid" = %s', ("c1",))
|
cur.execute('SELECT status::text, score FROM "DiscoverySuggestion" WHERE "artistMbid" = %s', ("c1",))
|
||||||
assert cur.fetchone() == ("dismissed", 0.0) # untouched — WHERE status='pending' blocked the update
|
assert cur.fetchone() == ("dismissed", 0.0) # untouched — WHERE status='pending' blocked the update
|
||||||
@@ -120,18 +120,18 @@ def test_dismissed_suggestion_is_never_revived(conn):
|
|||||||
def test_stamps_last_discovered_and_throttles(conn):
|
def test_stamps_last_discovered_and_throttles(conn):
|
||||||
insert_watched_artist(conn, mbid="s1", name="Seed")
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "X", 0.9)]})
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "X", 0.9)]})
|
||||||
assert run_discovery(conn, [src], FakeMbBrowser(), DiscoveryConfig()).artists == 1
|
assert run_discovery(conn, [src], FakeMbBrowser(default_core=True), DiscoveryConfig()).artists == 1
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute('SELECT "lastDiscoveredAt" IS NOT NULL FROM "WatchedArtist" WHERE mbid = %s', ("s1",))
|
cur.execute('SELECT "lastDiscoveredAt" IS NOT NULL FROM "WatchedArtist" WHERE mbid = %s', ("s1",))
|
||||||
assert cur.fetchone()[0] is True
|
assert cur.fetchone()[0] is True
|
||||||
# second run: seed not due (lastDiscoveredAt recent) -> no seeds -> nothing new
|
# second run: seed not due (lastDiscoveredAt recent) -> no seeds -> nothing new
|
||||||
assert run_discovery(conn, [src], FakeMbBrowser(), DiscoveryConfig()).artists == 0
|
assert run_discovery(conn, [src], FakeMbBrowser(default_core=True), DiscoveryConfig()).artists == 0
|
||||||
|
|
||||||
|
|
||||||
def test_unhealthy_source_contributes_nothing(conn):
|
def test_unhealthy_source_contributes_nothing(conn):
|
||||||
insert_watched_artist(conn, mbid="s1", name="Seed")
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "X", 0.9)]}, healthy=False)
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "X", 0.9)]}, healthy=False)
|
||||||
assert run_discovery(conn, [src], FakeMbBrowser(), DiscoveryConfig()).artists == 0
|
assert run_discovery(conn, [src], FakeMbBrowser(default_core=True), DiscoveryConfig()).artists == 0
|
||||||
assert _artist_rows(conn) == []
|
assert _artist_rows(conn) == []
|
||||||
|
|
||||||
|
|
||||||
@@ -146,7 +146,7 @@ def test_source_with_raising_health_is_skipped_not_fatal(conn):
|
|||||||
return []
|
return []
|
||||||
|
|
||||||
good = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "X", 0.9)]})
|
good = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "X", 0.9)]})
|
||||||
result = run_discovery(conn, [BoomSource(), good], FakeMbBrowser(), DiscoveryConfig())
|
result = run_discovery(conn, [BoomSource(), good], FakeMbBrowser(default_core=True), DiscoveryConfig())
|
||||||
assert result.artists == 1 # good source still processed; boom skipped, no crash
|
assert result.artists == 1 # good source still processed; boom skipped, no crash
|
||||||
|
|
||||||
|
|
||||||
@@ -176,8 +176,8 @@ def test_score_sums_across_separate_chunks(conn):
|
|||||||
"s2": [SimilarArtist("c1", "Shared", 0.5)],
|
"s2": [SimilarArtist("c1", "Shared", 0.5)],
|
||||||
})
|
})
|
||||||
cfg = DiscoveryConfig(chunk_size=1)
|
cfg = DiscoveryConfig(chunk_size=1)
|
||||||
run_discovery(conn, [src], FakeMbBrowser(), cfg) # processes s1 (older)
|
run_discovery(conn, [src], FakeMbBrowser(default_core=True), cfg) # processes s1 (older)
|
||||||
run_discovery(conn, [src], FakeMbBrowser(), cfg) # processes s2
|
run_discovery(conn, [src], FakeMbBrowser(default_core=True), cfg) # processes s2
|
||||||
# replace-not-accumulate would leave 0.5/seedCount1; contributions give the sum.
|
# replace-not-accumulate would leave 0.5/seedCount1; contributions give the sum.
|
||||||
assert _artist_score(conn, "c1") == (2.0, 2) # each seed max-normalizes to 1.0
|
assert _artist_score(conn, "c1") == (2.0, 2) # each seed max-normalizes to 1.0
|
||||||
|
|
||||||
@@ -193,7 +193,7 @@ def test_partial_sweep_preserves_other_seeds_contribution(conn):
|
|||||||
"s1": [SimilarArtist("c1", "Shared", 0.6)],
|
"s1": [SimilarArtist("c1", "Shared", 0.6)],
|
||||||
"s2": [SimilarArtist("c1", "Shared", 0.5)],
|
"s2": [SimilarArtist("c1", "Shared", 0.5)],
|
||||||
})
|
})
|
||||||
run_discovery(conn, [src], FakeMbBrowser(), DiscoveryConfig()) # both -> 1.1/2
|
run_discovery(conn, [src], FakeMbBrowser(default_core=True), DiscoveryConfig()) # both -> 1.1/2
|
||||||
assert _artist_score(conn, "c1") == (2.0, 2) # each seed max-normalizes to 1.0
|
assert _artist_score(conn, "c1") == (2.0, 2) # each seed max-normalizes to 1.0
|
||||||
|
|
||||||
# s1 due again (older than interval), s2 fresh; s1 now scores c1 at 0.7.
|
# s1 due again (older than interval), s2 fresh; s1 now scores c1 at 0.7.
|
||||||
@@ -202,7 +202,7 @@ def test_partial_sweep_preserves_other_seeds_contribution(conn):
|
|||||||
cur.execute('UPDATE "WatchedArtist" SET "lastDiscoveredAt" = now() WHERE mbid = %s', ("s2",))
|
cur.execute('UPDATE "WatchedArtist" SET "lastDiscoveredAt" = now() WHERE mbid = %s', ("s2",))
|
||||||
conn.commit()
|
conn.commit()
|
||||||
src2 = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Shared", 0.7)]})
|
src2 = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Shared", 0.7)]})
|
||||||
run_discovery(conn, [src2], FakeMbBrowser(), DiscoveryConfig())
|
run_discovery(conn, [src2], FakeMbBrowser(default_core=True), DiscoveryConfig())
|
||||||
assert _artist_score(conn, "c1") == (2.0, 2) # 1.0 (new s1) + 1.0 (kept s2), max-normalized
|
assert _artist_score(conn, "c1") == (2.0, 2) # 1.0 (new s1) + 1.0 (kept s2), max-normalized
|
||||||
|
|
||||||
|
|
||||||
@@ -215,7 +215,7 @@ def test_seed_dropoff_lowers_seedcount(conn):
|
|||||||
"s1": [SimilarArtist("c1", "Shared", 0.6)],
|
"s1": [SimilarArtist("c1", "Shared", 0.6)],
|
||||||
"s2": [SimilarArtist("c1", "Shared", 0.5)],
|
"s2": [SimilarArtist("c1", "Shared", 0.5)],
|
||||||
})
|
})
|
||||||
run_discovery(conn, [src], FakeMbBrowser(), DiscoveryConfig())
|
run_discovery(conn, [src], FakeMbBrowser(default_core=True), DiscoveryConfig())
|
||||||
assert _artist_score(conn, "c1") == (2.0, 2) # each seed max-normalizes to 1.0
|
assert _artist_score(conn, "c1") == (2.0, 2) # each seed max-normalizes to 1.0
|
||||||
|
|
||||||
# s1 re-swept but no longer finds c1 -> its contribution is removed, c1 recomputed.
|
# s1 re-swept but no longer finds c1 -> its contribution is removed, c1 recomputed.
|
||||||
@@ -224,7 +224,7 @@ def test_seed_dropoff_lowers_seedcount(conn):
|
|||||||
cur.execute('UPDATE "WatchedArtist" SET "lastDiscoveredAt" = now() WHERE mbid = %s', ("s2",))
|
cur.execute('UPDATE "WatchedArtist" SET "lastDiscoveredAt" = now() WHERE mbid = %s', ("s2",))
|
||||||
conn.commit()
|
conn.commit()
|
||||||
src2 = FakeSimilaritySource(similar={"s1": [SimilarArtist("c9", "Other", 0.9)]})
|
src2 = FakeSimilaritySource(similar={"s1": [SimilarArtist("c9", "Other", 0.9)]})
|
||||||
run_discovery(conn, [src2], FakeMbBrowser(), DiscoveryConfig())
|
run_discovery(conn, [src2], FakeMbBrowser(default_core=True), DiscoveryConfig())
|
||||||
assert _artist_score(conn, "c1") == (1.0, 1) # only s2 remains (max-normalized to 1.0)
|
assert _artist_score(conn, "c1") == (1.0, 1) # only s2 remains (max-normalized to 1.0)
|
||||||
|
|
||||||
|
|
||||||
@@ -241,7 +241,7 @@ def test_evicts_contributions_from_removed_seed(conn):
|
|||||||
"s1": [SimilarArtist("c1", "Shared", 0.6)],
|
"s1": [SimilarArtist("c1", "Shared", 0.6)],
|
||||||
"s2": [SimilarArtist("c1", "Shared", 0.5)],
|
"s2": [SimilarArtist("c1", "Shared", 0.5)],
|
||||||
})
|
})
|
||||||
run_discovery(conn, [src], FakeMbBrowser(), DiscoveryConfig())
|
run_discovery(conn, [src], FakeMbBrowser(default_core=True), DiscoveryConfig())
|
||||||
assert _artist_score(conn, "c1") == (2.0, 2) # each seed max-normalizes to 1.0
|
assert _artist_score(conn, "c1") == (2.0, 2) # each seed max-normalizes to 1.0
|
||||||
|
|
||||||
# Unfollow s1 (no FK cascade on DiscoverySeedContribution).
|
# Unfollow s1 (no FK cascade on DiscoverySeedContribution).
|
||||||
@@ -254,7 +254,7 @@ def test_evicts_contributions_from_removed_seed(conn):
|
|||||||
cur.execute('UPDATE "WatchedArtist" SET "lastDiscoveredAt" = now() - interval \'200 hours\' WHERE mbid = %s', ("s2",))
|
cur.execute('UPDATE "WatchedArtist" SET "lastDiscoveredAt" = now() - interval \'200 hours\' WHERE mbid = %s', ("s2",))
|
||||||
conn.commit()
|
conn.commit()
|
||||||
src2 = FakeSimilaritySource(similar={"s2": [SimilarArtist("c1", "Shared", 0.5)]})
|
src2 = FakeSimilaritySource(similar={"s2": [SimilarArtist("c1", "Shared", 0.5)]})
|
||||||
run_discovery(conn, [src2], FakeMbBrowser(), DiscoveryConfig())
|
run_discovery(conn, [src2], FakeMbBrowser(default_core=True), DiscoveryConfig())
|
||||||
|
|
||||||
assert _artist_score(conn, "c1") == (1.0, 1) # s1's stale contribution evicted; s2 alone -> 1.0
|
assert _artist_score(conn, "c1") == (1.0, 1) # s1's stale contribution evicted; s2 alone -> 1.0
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
@@ -271,9 +271,38 @@ def test_run_discovery_processes_at_most_chunk_size_seeds(conn):
|
|||||||
insert_watched_artist(conn, mbid=f"s{i}", name=f"Seed{i}")
|
insert_watched_artist(conn, mbid=f"s{i}", name=f"Seed{i}")
|
||||||
src = FakeSimilaritySource(similar={f"s{i}": [SimilarArtist(f"c{i}", f"Cand{i}", 0.9)] for i in range(4)})
|
src = FakeSimilaritySource(similar={f"s{i}": [SimilarArtist(f"c{i}", f"Cand{i}", 0.9)] for i in range(4)})
|
||||||
cfg = DiscoveryConfig.from_config({"discover.chunkSize": "2"})
|
cfg = DiscoveryConfig.from_config({"discover.chunkSize": "2"})
|
||||||
result = run_discovery(conn, [src], FakeMbBrowser(), cfg)
|
result = run_discovery(conn, [src], FakeMbBrowser(default_core=True), cfg)
|
||||||
assert result.seeds == 2 # only chunk_size seeds this call
|
assert result.seeds == 2 # only chunk_size seeds this call
|
||||||
# exactly 2 seeds now have lastDiscoveredAt set
|
# exactly 2 seeds now have lastDiscoveredAt set
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute('SELECT count(*) FROM "WatchedArtist" WHERE "lastDiscoveredAt" IS NOT NULL')
|
cur.execute('SELECT count(*) FROM "WatchedArtist" WHERE "lastDiscoveredAt" IS NOT NULL')
|
||||||
assert cur.fetchone()[0] == 2
|
assert cur.fetchone()[0] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_reset_pending_clears_state_and_marks_seeds_due(conn):
|
||||||
|
from lyra_worker.discovery import reset_pending
|
||||||
|
from tests.conftest import insert_discovery_suggestion
|
||||||
|
insert_watched_artist(conn, mbid="s1", name="Seed", last_discovered_sql="now()")
|
||||||
|
insert_discovery_suggestion(conn, kind="artist", artist_mbid="c1", status="pending")
|
||||||
|
insert_discovery_suggestion(conn, kind="artist", artist_mbid="c2", status="dismissed")
|
||||||
|
insert_discovery_suggestion(conn, kind="artist", artist_mbid="c3", status="wanted")
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('INSERT INTO "DiscoverySeedContribution" (id, "candidateMbid", "candidateName", '
|
||||||
|
'"seedMbid", score, sources, "updatedAt") VALUES '
|
||||||
|
"(gen_random_uuid()::text, 'c1', 'C1', 's1', 1.0, '{}', now())")
|
||||||
|
cur.execute('INSERT INTO "DiscoverySeed" (mbid, "lastDiscoveredAt") VALUES (%s, now())', ("libseed",))
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
reset_pending(conn)
|
||||||
|
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute("SELECT count(*) FROM \"DiscoverySuggestion\" WHERE status='pending'")
|
||||||
|
assert cur.fetchone()[0] == 0 # pending cleared
|
||||||
|
cur.execute("SELECT count(*) FROM \"DiscoverySuggestion\" WHERE status IN ('dismissed','wanted')")
|
||||||
|
assert cur.fetchone()[0] == 2 # user decisions kept
|
||||||
|
cur.execute('SELECT count(*) FROM "DiscoverySeedContribution"')
|
||||||
|
assert cur.fetchone()[0] == 0
|
||||||
|
cur.execute('SELECT count(*) FROM "DiscoverySeed"')
|
||||||
|
assert cur.fetchone()[0] == 0
|
||||||
|
cur.execute('SELECT count(*) FROM "WatchedArtist" WHERE "lastDiscoveredAt" IS NULL')
|
||||||
|
assert cur.fetchone()[0] == 1 # seed due again
|
||||||
|
|||||||
@@ -2,7 +2,7 @@ from lyra_worker.adapters.fakes import FakeMbBrowser, FakeSimilaritySource
|
|||||||
from lyra_worker.browser import ReleaseGroupInfo
|
from lyra_worker.browser import ReleaseGroupInfo
|
||||||
from lyra_worker.discovery import DiscoveryConfig, run_discovery
|
from lyra_worker.discovery import DiscoveryConfig, run_discovery
|
||||||
from lyra_worker.similarity.base import SimilarArtist
|
from lyra_worker.similarity.base import SimilarArtist
|
||||||
from tests.conftest import insert_monitored_release, insert_watched_artist
|
from tests.conftest import insert_discovery_suggestion, insert_monitored_release, insert_watched_artist
|
||||||
|
|
||||||
|
|
||||||
def _album_rows(conn):
|
def _album_rows(conn):
|
||||||
@@ -12,6 +12,28 @@ def _album_rows(conn):
|
|||||||
return cur.fetchall()
|
return cur.fetchall()
|
||||||
|
|
||||||
|
|
||||||
|
def _artist_rows(conn):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute("SELECT \"artistMbid\" FROM \"DiscoverySuggestion\" "
|
||||||
|
"WHERE kind = 'artist' AND status = 'pending' ORDER BY \"artistMbid\"")
|
||||||
|
return cur.fetchall()
|
||||||
|
|
||||||
|
|
||||||
|
def _reasons(conn):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT "rgMbid", "albumReason" FROM "DiscoverySuggestion" '
|
||||||
|
'WHERE kind = \'album\' ORDER BY "rgMbid"')
|
||||||
|
return dict(cur.fetchall())
|
||||||
|
|
||||||
|
|
||||||
|
class FakePopularity:
|
||||||
|
"""Maps artist NAME -> [[album_name, mbid_or_None, playcount], …] (Last.fm shape)."""
|
||||||
|
def __init__(self, data):
|
||||||
|
self._d = data
|
||||||
|
def top_albums(self, conn, name):
|
||||||
|
return self._d.get(name, [])
|
||||||
|
|
||||||
|
|
||||||
def test_derives_most_recent_core_album_for_surfaced_artist(conn):
|
def test_derives_most_recent_core_album_for_surfaced_artist(conn):
|
||||||
insert_watched_artist(conn, mbid="s1", name="Seed")
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Cand", 0.9)]})
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Cand", 0.9)]})
|
||||||
@@ -37,6 +59,107 @@ def test_album_skips_release_groups_already_in_library(conn):
|
|||||||
assert _album_rows(conn) == [("c1", "rg-fresh", "Fresh", "Album")] # owned rg excluded
|
assert _album_rows(conn) == [("c1", "rg-fresh", "Fresh", "Album")] # owned rg excluded
|
||||||
|
|
||||||
|
|
||||||
|
def test_derives_newest_and_most_popular_albums(conn):
|
||||||
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Cand", 0.9)]})
|
||||||
|
browser = FakeMbBrowser(releases={"c1": [
|
||||||
|
ReleaseGroupInfo("rg-hit", "Big Hit", "Album", (), "2005-01-01"),
|
||||||
|
ReleaseGroupInfo("rg-new", "Latest", "Album", (), "2022-01-01"),
|
||||||
|
]})
|
||||||
|
pop = FakePopularity({"Cand": [["Big Hit", "rg-hit", 90000], ["Latest", "rg-new", 100]]})
|
||||||
|
result = run_discovery(conn, [src], browser, DiscoveryConfig(albums_per_artist=1), popularity=pop)
|
||||||
|
assert result.albums == 2
|
||||||
|
assert _reasons(conn) == {"rg-new": "newest", "rg-hit": "popular"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_newest_and_popular_the_same_album_is_one_row(conn):
|
||||||
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Cand", 0.9)]})
|
||||||
|
browser = FakeMbBrowser(releases={"c1": [
|
||||||
|
ReleaseGroupInfo("rg-only", "Only", "Album", (), "2022-01-01"),
|
||||||
|
]})
|
||||||
|
pop = FakePopularity({"Cand": [["Only", "rg-only", 90000]]})
|
||||||
|
result = run_discovery(conn, [src], browser, DiscoveryConfig(albums_per_artist=1), popularity=pop)
|
||||||
|
assert result.albums == 1
|
||||||
|
assert _reasons(conn) == {"rg-only": "newest,popular"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_popular_matches_by_title_when_lastfm_mbid_missing_and_skips_owned(conn):
|
||||||
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
|
insert_monitored_release(conn, artist_mbid="c1", rg_mbid="rg-owned", album="Owned Hit")
|
||||||
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Cand", 0.9)]})
|
||||||
|
browser = FakeMbBrowser(releases={"c1": [
|
||||||
|
ReleaseGroupInfo("rg-owned", "Owned Hit", "Album", (), "2010-01-01"), # excluded (owned)
|
||||||
|
ReleaseGroupInfo("rg-match", "Second Hit", "Album", (), "2012-01-01"),
|
||||||
|
ReleaseGroupInfo("rg-new", "Latest", "Album", (), "2022-01-01"),
|
||||||
|
]})
|
||||||
|
# top album is owned (skipped), next has no mbid -> matched by title into core
|
||||||
|
pop = FakePopularity({"Cand": [["Owned Hit", "rg-owned", 99999], ["Second Hit", None, 80000]]})
|
||||||
|
run_discovery(conn, [src], browser, DiscoveryConfig(albums_per_artist=1), popularity=pop)
|
||||||
|
assert _reasons(conn) == {"rg-new": "newest", "rg-match": "popular"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_rederiving_replaces_stale_pending_album_suggestions(conn):
|
||||||
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
|
# a stale pending album suggestion for c1 that is no longer a pick (e.g. an old single)
|
||||||
|
insert_discovery_suggestion(conn, kind="album", artist_mbid="c1", rg_mbid="rg-old-single",
|
||||||
|
album="Old Single", status="pending")
|
||||||
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Cand", 0.9)]})
|
||||||
|
browser = FakeMbBrowser(releases={"c1": [
|
||||||
|
ReleaseGroupInfo("rg-new", "New Album", "Album", (), "2022-01-01"),
|
||||||
|
]})
|
||||||
|
run_discovery(conn, [src], browser, DiscoveryConfig(albums_per_artist=1))
|
||||||
|
# the stale single is gone; only the freshly-derived album remains
|
||||||
|
assert [r[1] for r in _album_rows(conn)] == ["rg-new"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_rederiving_keeps_wanted_album_suggestions(conn):
|
||||||
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
|
insert_discovery_suggestion(conn, kind="album", artist_mbid="c1", rg_mbid="rg-wanted",
|
||||||
|
album="Wanted One", status="wanted")
|
||||||
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Cand", 0.9)]})
|
||||||
|
browser = FakeMbBrowser(releases={"c1": [
|
||||||
|
ReleaseGroupInfo("rg-new", "New Album", "Album", (), "2022-01-01"),
|
||||||
|
]})
|
||||||
|
run_discovery(conn, [src], browser, DiscoveryConfig(albums_per_artist=1))
|
||||||
|
with conn.cursor() as cur: # wanted row untouched (only pending stale rows are replaced)
|
||||||
|
cur.execute('SELECT count(*) FROM "DiscoverySuggestion" WHERE "rgMbid" = %s AND status = \'wanted\'', ("rg-wanted",))
|
||||||
|
assert cur.fetchone()[0] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_drops_artist_with_no_core_discography(conn):
|
||||||
|
# Two candidates: one real (has a core album), one a "producer" whose only release is a
|
||||||
|
# secondary-typed soundtrack (like Max Martin's & Juliet) — the latter must be dropped.
|
||||||
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
|
src = FakeSimilaritySource(similar={"s1": [
|
||||||
|
SimilarArtist("real", "Real Artist", 0.9), SimilarArtist("producer", "Producer", 0.8)]})
|
||||||
|
browser = FakeMbBrowser(releases={
|
||||||
|
"real": [ReleaseGroupInfo("rg-r", "Record", "Album", (), "2020-01-01")],
|
||||||
|
"producer": [ReleaseGroupInfo("rg-st", "A Musical", "Album", ("Soundtrack",), "2019-01-01")],
|
||||||
|
})
|
||||||
|
run_discovery(conn, [src], browser, DiscoveryConfig(albums_per_artist=1))
|
||||||
|
assert [r[0] for r in _artist_rows(conn)] == ["real"] # producer dropped
|
||||||
|
|
||||||
|
|
||||||
|
def test_drops_artist_with_empty_discography(conn):
|
||||||
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("member", "Band Member", 0.9)]})
|
||||||
|
browser = FakeMbBrowser(releases={"member": []}) # no release groups at all
|
||||||
|
run_discovery(conn, [src], browser, DiscoveryConfig(albums_per_artist=1))
|
||||||
|
assert _artist_rows(conn) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_popularity_provider_yields_newest_only(conn):
|
||||||
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Cand", 0.9)]})
|
||||||
|
browser = FakeMbBrowser(releases={"c1": [
|
||||||
|
ReleaseGroupInfo("rg-old", "Old", "Album", (), "2005-01-01"),
|
||||||
|
ReleaseGroupInfo("rg-new", "New", "Album", (), "2022-01-01"),
|
||||||
|
]})
|
||||||
|
run_discovery(conn, [src], browser, DiscoveryConfig(albums_per_artist=1)) # popularity=None
|
||||||
|
assert _reasons(conn) == {"rg-new": "newest"}
|
||||||
|
|
||||||
|
|
||||||
def test_albums_only_excludes_singles_and_eps_by_default(conn):
|
def test_albums_only_excludes_singles_and_eps_by_default(conn):
|
||||||
insert_watched_artist(conn, mbid="s1", name="Seed")
|
insert_watched_artist(conn, mbid="s1", name="Seed")
|
||||||
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Cand", 0.9)]})
|
src = FakeSimilaritySource(similar={"s1": [SimilarArtist("c1", "Cand", 0.9)]})
|
||||||
|
|||||||
@@ -0,0 +1,50 @@
|
|||||||
|
from lyra_worker.floor import (
|
||||||
|
press_paused,
|
||||||
|
job_delay_seconds,
|
||||||
|
kill_requested,
|
||||||
|
consume_kill_switch,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_press_paused_reads_truthy_values():
|
||||||
|
assert press_paused({"floor.paused": "true"}) is True
|
||||||
|
assert press_paused({"floor.paused": "ON"}) is True
|
||||||
|
assert press_paused({"floor.paused": "false"}) is False
|
||||||
|
assert press_paused({}) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_job_delay_seconds_parses_and_defaults():
|
||||||
|
assert job_delay_seconds({"floor.jobDelaySeconds": "30"}) == 30.0
|
||||||
|
assert job_delay_seconds({}) == 0.0
|
||||||
|
assert job_delay_seconds({"floor.jobDelaySeconds": "-5"}) == 0.0
|
||||||
|
assert job_delay_seconds({"floor.jobDelaySeconds": "junk"}) == 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_kill_requested_reads_flag():
|
||||||
|
assert kill_requested({"floor.killRequested": "true"}) is True
|
||||||
|
assert kill_requested({}) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_consume_kill_switch_clears_flag_and_exits(conn):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Config"(key, value, secret, "updatedAt") '
|
||||||
|
"VALUES ('floor.killRequested', 'true', false, now())"
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
calls = []
|
||||||
|
fired = consume_kill_switch(conn, exit_fn=lambda code: calls.append(code))
|
||||||
|
|
||||||
|
assert fired is True
|
||||||
|
assert calls == [0]
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute("SELECT value FROM \"Config\" WHERE key = 'floor.killRequested'")
|
||||||
|
assert cur.fetchone()[0] == "false"
|
||||||
|
|
||||||
|
|
||||||
|
def test_consume_kill_switch_noop_when_unset(conn):
|
||||||
|
calls = []
|
||||||
|
fired = consume_kill_switch(conn, exit_fn=lambda code: calls.append(code))
|
||||||
|
assert fired is False
|
||||||
|
assert calls == []
|
||||||
@@ -0,0 +1,125 @@
|
|||||||
|
"""When the top-ranked source downloads but is incomplete (e.g. Qobuz consistently fails one
|
||||||
|
track), the pipeline must fall through to the next-ranked source instead of failing the whole
|
||||||
|
job — only giving up (needs_attention) when every candidate is incomplete."""
|
||||||
|
import os
|
||||||
|
|
||||||
|
from lyra_worker.claim import claim_next
|
||||||
|
from lyra_worker.pipeline import run_pipeline
|
||||||
|
from lyra_worker.types import Candidate, DownloadResult, Quality
|
||||||
|
from tests.conftest import insert_request
|
||||||
|
|
||||||
|
_HIRES = Quality(fmt="FLAC", lossless=True, bit_depth=24, sample_rate=96000)
|
||||||
|
_CD = Quality(fmt="FLAC", lossless=True, bit_depth=16, sample_rate=44100)
|
||||||
|
|
||||||
|
|
||||||
|
def _stage(dest, n):
|
||||||
|
os.makedirs(dest, exist_ok=True)
|
||||||
|
for i in range(n):
|
||||||
|
with open(os.path.join(dest, f"{i + 1:02d}.flac"), "wb") as fh:
|
||||||
|
fh.write(b"audio")
|
||||||
|
|
||||||
|
|
||||||
|
class _Qobuz:
|
||||||
|
"""Higher-quality (ranks first) Qobuz that promises 3 tracks but delivers `deliver`."""
|
||||||
|
name = "qobuz"
|
||||||
|
tier = 0
|
||||||
|
|
||||||
|
def __init__(self, deliver):
|
||||||
|
self._deliver = deliver
|
||||||
|
|
||||||
|
def health(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
return [Candidate(source="qobuz", source_ref="q", matched_artist=target.artist,
|
||||||
|
matched_album=target.album, quality=_HIRES, track_count=3, source_tier=0)]
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
_stage(dest, self._deliver)
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=self._deliver)
|
||||||
|
|
||||||
|
|
||||||
|
class _Soulseek:
|
||||||
|
"""Lower-quality (ranks second) Soulseek that promises 3 tracks and delivers `deliver`."""
|
||||||
|
name = "soulseek"
|
||||||
|
tier = 1
|
||||||
|
|
||||||
|
def __init__(self, deliver):
|
||||||
|
self._deliver = deliver
|
||||||
|
|
||||||
|
def health(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
return [Candidate(source="soulseek", source_ref="s", matched_artist=target.artist,
|
||||||
|
matched_album=target.album, quality=_CD, track_count=3, source_tier=1)]
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
_stage(dest, self._deliver)
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=self._deliver)
|
||||||
|
|
||||||
|
|
||||||
|
def _state_error(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT state, error FROM "Job" WHERE id = %s', (job_id,))
|
||||||
|
return cur.fetchone()
|
||||||
|
|
||||||
|
|
||||||
|
def _chosen_source(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT source FROM "Candidate" WHERE "jobId" = %s AND chosen = true', (job_id,))
|
||||||
|
row = cur.fetchone()
|
||||||
|
return row[0] if row else None
|
||||||
|
|
||||||
|
|
||||||
|
def test_incomplete_top_source_falls_through_to_complete_next(conn, tmp_path):
|
||||||
|
# Qobuz (ranks first on quality) delivers only 2 of its promised 3 tracks -> incomplete.
|
||||||
|
# The pipeline must try the next source (Soulseek, complete) rather than failing the job.
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [_Qobuz(deliver=2), _Soulseek(deliver=3)], dest_root=str(tmp_path))
|
||||||
|
assert _state_error(conn, job_id) == ("imported", None)
|
||||||
|
assert _chosen_source(conn, job_id) == "soulseek"
|
||||||
|
|
||||||
|
|
||||||
|
def test_all_sources_incomplete_still_needs_attention(conn, tmp_path):
|
||||||
|
# Every candidate is incomplete -> no fallback succeeds -> needs_attention (incomplete).
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [_Qobuz(deliver=2), _Soulseek(deliver=1)], dest_root=str(tmp_path))
|
||||||
|
assert _state_error(conn, job_id) == ("needs_attention", "incomplete download")
|
||||||
|
assert not (tmp_path / "Radiohead" / "In Rainbows").exists() # nothing imported
|
||||||
|
|
||||||
|
|
||||||
|
class _CountingIncompleteQobuz:
|
||||||
|
"""Offers `n` candidates that all download but come back incomplete; counts download calls."""
|
||||||
|
name = "qobuz"
|
||||||
|
tier = 0
|
||||||
|
|
||||||
|
def __init__(self, n):
|
||||||
|
self.n = n
|
||||||
|
self.calls = 0
|
||||||
|
|
||||||
|
def health(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
q = _HIRES
|
||||||
|
return [Candidate(source="qobuz", source_ref=f"r{i}", matched_artist=target.artist,
|
||||||
|
matched_album=target.album, quality=q, track_count=3, source_tier=0)
|
||||||
|
for i in range(self.n)]
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
self.calls += 1
|
||||||
|
_stage(dest, 1) # 1 of 3 → always incomplete
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=1)
|
||||||
|
|
||||||
|
|
||||||
|
def test_fall_through_is_capped(conn):
|
||||||
|
from lyra_worker.pipeline import _MAX_DOWNLOAD_ATTEMPTS
|
||||||
|
a = _CountingIncompleteQobuz(n=20)
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [a], dest_root="/tmp/lib-cap")
|
||||||
|
assert _state_error(conn, job_id) == ("needs_attention", "incomplete download")
|
||||||
|
assert a.calls == _MAX_DOWNLOAD_ATTEMPTS # capped, not all 20 peers tried
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
from lyra_worker.registry import build_album_popularity
|
||||||
|
from lyra_worker.similarity._lastfm_albums import _parse_top_albums
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_orders_by_playcount_and_keeps_mbids():
|
||||||
|
data = {"topalbums": {"album": [
|
||||||
|
{"name": "Older Hit", "mbid": "rg-old", "playcount": "5000"},
|
||||||
|
{"name": "Newer", "mbid": "", "playcount": "9000"},
|
||||||
|
{"name": "No Plays", "playcount": "0"},
|
||||||
|
]}}
|
||||||
|
assert _parse_top_albums(data) == [
|
||||||
|
["Newer", None, 9000], # highest playcount first; empty mbid -> None
|
||||||
|
["Older Hit", "rg-old", 5000],
|
||||||
|
["No Plays", None, 0],
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_wraps_a_single_album_dict():
|
||||||
|
data = {"topalbums": {"album": {"name": "Only One", "mbid": "rg1", "playcount": "3"}}}
|
||||||
|
assert _parse_top_albums(data) == [["Only One", "rg1", 3]]
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_returns_empty_on_error_body():
|
||||||
|
assert _parse_top_albums({"error": 6, "message": "bad"}) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_album_popularity_none_without_key():
|
||||||
|
assert build_album_popularity({}) is None
|
||||||
|
assert build_album_popularity({"lastfm.api_key": "abc"}) is not None
|
||||||
@@ -72,6 +72,52 @@ def test_grabbed_past_window_is_fulfilled_not_enqueued(conn):
|
|||||||
assert _state(conn, rel_id) == "fulfilled"
|
assert _state(conn, rel_id) == "fulfilled"
|
||||||
|
|
||||||
|
|
||||||
|
def test_owned_by_rgmbid_despite_name_mismatch_is_fulfilled(conn):
|
||||||
|
# The monitor followed "Kanye West" but the library stores MB's canonical "Ye"; matching on
|
||||||
|
# rgMbid recognises ownership so the release is fulfilled, not re-pressed.
|
||||||
|
rel_id = insert_monitored_release(conn, artist_name="Kanye West", album="Graduation",
|
||||||
|
rg_mbid="rg-grad", monitored=True, state="wanted")
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "LibraryItem" (id, artist, album, path, source, format, '
|
||||||
|
'"qualityClass", "rgMbid", "importedAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, 'Ye', 'Graduation', '/m', 'soulseek', 'FLAC', 2, "
|
||||||
|
"'rg-grad', now())"
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
assert enqueue_due(conn, CFG) == 0
|
||||||
|
assert _state(conn, rel_id) == "fulfilled"
|
||||||
|
|
||||||
|
|
||||||
|
def test_re_enqueue_retires_prior_failed_attempt(conn):
|
||||||
|
# A hard-to-match release re-pressed each retry interval must not pile up needs_attention
|
||||||
|
# Request rows; the prior failed attempt is retired when the next one is enqueued.
|
||||||
|
rel_id = insert_monitored_release(conn, artist_name="X", album="Y", rg_mbid="rg-1",
|
||||||
|
monitored=True, state="wanted")
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Request" (id, artist, album, status, "monitoredReleaseId", "createdAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, 'X', 'Y', 'needs_attention', %s, now()) RETURNING id",
|
||||||
|
(rel_id,),
|
||||||
|
)
|
||||||
|
req = cur.fetchone()[0]
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Job" (id, "requestId", state, "currentStage", attempts, "createdAt", "updatedAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, %s, 'needs_attention', 'rank', 1, now(), now())",
|
||||||
|
(req,),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
assert enqueue_due(conn, CFG) == 1
|
||||||
|
assert _job_count_for(conn, rel_id) == 1 # old failed attempt retired, one fresh attempt
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT j.state FROM "Job" j JOIN "Request" r ON r.id = j."requestId" '
|
||||||
|
'WHERE r."monitoredReleaseId" = %s',
|
||||||
|
(rel_id,),
|
||||||
|
)
|
||||||
|
assert cur.fetchone()[0] == "requested"
|
||||||
|
|
||||||
|
|
||||||
def test_already_in_library_at_cutoff_is_fulfilled(conn):
|
def test_already_in_library_at_cutoff_is_fulfilled(conn):
|
||||||
rel_id = insert_monitored_release(conn, artist_name="A", album="B", rg_mbid="rg-1",
|
rel_id = insert_monitored_release(conn, artist_name="A", album="B", rg_mbid="rg-1",
|
||||||
monitored=True, state="wanted")
|
monitored=True, state="wanted")
|
||||||
|
|||||||
@@ -22,17 +22,27 @@ def _make_job(conn, rel_id, artist, album, state):
|
|||||||
return req, job
|
return req, job
|
||||||
|
|
||||||
|
|
||||||
def _add_library(conn, req, artist, album, quality):
|
def _add_library(conn, req, artist, album, quality, rg_mbid=None):
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
'INSERT INTO "LibraryItem" (id, "requestId", artist, album, path, source, format, '
|
'INSERT INTO "LibraryItem" (id, "requestId", artist, album, path, source, format, '
|
||||||
'"qualityClass", "importedAt") '
|
'"qualityClass", "rgMbid", "importedAt") '
|
||||||
"VALUES (gen_random_uuid()::text, %s, %s, %s, '/m', 'qobuz', 'FLAC', %s, now())",
|
"VALUES (gen_random_uuid()::text, %s, %s, %s, '/m', 'qobuz', 'FLAC', %s, %s, now())",
|
||||||
(req, artist, album, quality),
|
(req, artist, album, quality, rg_mbid),
|
||||||
)
|
)
|
||||||
conn.commit()
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _needs_attention_count(conn, rel_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT count(*) FROM "Request" r JOIN "Job" j ON j."requestId" = r.id '
|
||||||
|
"WHERE r.\"monitoredReleaseId\" = %s AND j.state = 'needs_attention'",
|
||||||
|
(rel_id,),
|
||||||
|
)
|
||||||
|
return cur.fetchone()[0]
|
||||||
|
|
||||||
|
|
||||||
def _release(conn, rel_id):
|
def _release(conn, rel_id):
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
@@ -106,6 +116,42 @@ def test_fulfill_owned_releases_ignores_below_cutoff(conn):
|
|||||||
assert _release(conn, rel)[0] == "wanted"
|
assert _release(conn, rel)[0] == "wanted"
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_matches_by_rgmbid_when_artist_name_canonicalized(conn):
|
||||||
|
# Real 'Kanye West' -> 'Ye' bug: the MonitoredRelease keeps the followed name, but the
|
||||||
|
# imported LibraryItem carries MusicBrainz's canonical artist. Matching on rgMbid lets the
|
||||||
|
# release reconcile to fulfilled instead of looking un-owned and re-pressing forever.
|
||||||
|
rel = insert_monitored_release(conn, artist_name="Kanye West", album="Graduation",
|
||||||
|
rg_mbid="rg-grad", monitored=True, state="wanted")
|
||||||
|
req, job = _make_job(conn, rel, "Kanye West", "Graduation", "imported")
|
||||||
|
_add_library(conn, req, "Ye", "Graduation", 3, rg_mbid="rg-grad")
|
||||||
|
reconcile(conn, job, CFG)
|
||||||
|
assert _release(conn, rel) == ("fulfilled", 3, True)
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_fulfilled_retires_failed_sibling_attempts(conn):
|
||||||
|
# 'V' by Maroon 5 phantom: earlier failed attempts leave needs_attention Request rows that
|
||||||
|
# keep showing on the press even after a later attempt imports. A successful import retires
|
||||||
|
# those failed siblings so the press stops showing an owned album as needing attention.
|
||||||
|
rel = insert_monitored_release(conn, artist_name="Maroon 5", album="V", rg_mbid="rg-v",
|
||||||
|
monitored=True, state="wanted")
|
||||||
|
_make_job(conn, rel, "Maroon 5", "V", "needs_attention")
|
||||||
|
_make_job(conn, rel, "Maroon 5", "V", "needs_attention")
|
||||||
|
req, job = _make_job(conn, rel, "Maroon 5", "V", "imported")
|
||||||
|
_add_library(conn, req, "Maroon 5", "V", 2)
|
||||||
|
reconcile(conn, job, CFG)
|
||||||
|
assert _release(conn, rel)[0] == "fulfilled"
|
||||||
|
assert _needs_attention_count(conn, rel) == 0 # failed siblings retired
|
||||||
|
|
||||||
|
|
||||||
|
def test_fulfill_owned_releases_matches_by_rgmbid(conn):
|
||||||
|
# Ne-Yo -> Ne‐Yo (typographic hyphen) name drift: still owned by rgMbid.
|
||||||
|
rel = insert_monitored_release(conn, artist_name="Ne-Yo", album="Because of You",
|
||||||
|
rg_mbid="rg-ney", monitored=True, state="grabbed")
|
||||||
|
_add_library(conn, None, "Ne‐Yo", "Because of You", 2, rg_mbid="rg-ney")
|
||||||
|
assert fulfill_owned_releases(conn, MonitorConfig(quality_cutoff=2)) == 1
|
||||||
|
assert _release(conn, rel) == ("fulfilled", 2, True)
|
||||||
|
|
||||||
|
|
||||||
def test_unlinked_job_is_ignored(conn):
|
def test_unlinked_job_is_ignored(conn):
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
cur.execute(
|
cur.execute(
|
||||||
|
|||||||
@@ -0,0 +1,174 @@
|
|||||||
|
import musicbrainzngs
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from lyra_worker._musicbrainz import _best_release_group, _pick_release, _with_retry
|
||||||
|
|
||||||
|
# Offline unit tests for the pure release-group selection logic. The live resolve()
|
||||||
|
# path (network) is covered by test_musicbrainz_live.py.
|
||||||
|
|
||||||
|
|
||||||
|
def _rg(title, primary_type, artist=""):
|
||||||
|
g = {"id": title.lower(), "title": title, "primary-type": primary_type}
|
||||||
|
if artist:
|
||||||
|
g["artist-credit"] = [{"artist": {"name": artist, "id": artist.lower()}}]
|
||||||
|
return g
|
||||||
|
|
||||||
|
|
||||||
|
def test_prefers_album_over_same_titled_single():
|
||||||
|
# Real-world "Off the Wall" bug: MusicBrainz returns the Single release-group
|
||||||
|
# first, tied on title with the Album. We must pick the Album so a 10-track
|
||||||
|
# download isn't tagged from the 2-track single's tracklist.
|
||||||
|
groups = [
|
||||||
|
_rg("Off the Wall", "Single", "Michael Jackson"),
|
||||||
|
_rg("Off the Wall", "Album", "Michael Jackson"),
|
||||||
|
]
|
||||||
|
assert _best_release_group("Michael Jackson", "Off the Wall", groups)["primary-type"] == "Album"
|
||||||
|
|
||||||
|
|
||||||
|
def test_album_preference_is_order_independent():
|
||||||
|
groups = [
|
||||||
|
_rg("Off the Wall", "Album", "Michael Jackson"),
|
||||||
|
_rg("Off the Wall", "Single", "Michael Jackson"),
|
||||||
|
]
|
||||||
|
assert _best_release_group("Michael Jackson", "Off the Wall", groups)["primary-type"] == "Album"
|
||||||
|
|
||||||
|
|
||||||
|
def test_closer_title_still_wins_over_album_type():
|
||||||
|
# Title similarity dominates the album-type preference (for a single artist).
|
||||||
|
groups = [
|
||||||
|
_rg("Continuum", "Single", "John Mayer"),
|
||||||
|
_rg("Continuum Deluxe Reissue", "Album", "John Mayer"),
|
||||||
|
]
|
||||||
|
assert _best_release_group("John Mayer", "Continuum", groups)["title"] == "Continuum"
|
||||||
|
|
||||||
|
|
||||||
|
def test_below_threshold_returns_none():
|
||||||
|
groups = [_rg("Something Totally Different", "Album", "Michael Jackson")]
|
||||||
|
assert _best_release_group("Michael Jackson", "Off the Wall", groups) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_groups_returns_none():
|
||||||
|
assert _best_release_group("Michael Jackson", "Off the Wall", []) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_prefers_original_group_over_later_reissue():
|
||||||
|
# Real "Hybrid Theory" bug: MB lists a 2000 original and a 2023 reissue as separate same-titled
|
||||||
|
# Album groups (tie on artist/title/type). The reissue's longer tracklist fails completeness
|
||||||
|
# checks, so the original (earliest) must win regardless of MB's relevance order.
|
||||||
|
groups = [
|
||||||
|
{"id": "reissue", "title": "Hybrid Theory", "primary-type": "Album",
|
||||||
|
"first-release-date": "2023-07-21",
|
||||||
|
"artist-credit": [{"artist": {"name": "Linkin Park"}}]},
|
||||||
|
{"id": "original", "title": "Hybrid Theory", "primary-type": "Album",
|
||||||
|
"first-release-date": "2000-10-24",
|
||||||
|
"artist-credit": [{"artist": {"name": "Linkin Park"}}]},
|
||||||
|
]
|
||||||
|
assert _best_release_group("Linkin Park", "Hybrid Theory", groups)["id"] == "original"
|
||||||
|
|
||||||
|
|
||||||
|
def test_prefers_credited_artist_over_same_titled_other_artist_album():
|
||||||
|
# Real-world "Fun Machine" bug: Lake Street Dive's EP is what we followed, but two
|
||||||
|
# unrelated bands also have a release group literally titled "Fun Machine" as an Album.
|
||||||
|
# All three tie on title (1.0), so the old is_album tiebreak wrongly grabbed a foreign
|
||||||
|
# Album. The followed artist must win over title-tied albums by other artists.
|
||||||
|
groups = [
|
||||||
|
_rg("Fun Machine", "EP", "Lake Street Dive"),
|
||||||
|
_rg("Fun Machine", "Album", "Bastards of Melody"),
|
||||||
|
_rg("Fun Machine", "Album", "Teledildonics 5000"),
|
||||||
|
]
|
||||||
|
best = _best_release_group("Lake Street Dive", "Fun Machine", groups)
|
||||||
|
assert best["artist-credit"][0]["artist"]["name"] == "Lake Street Dive"
|
||||||
|
|
||||||
|
|
||||||
|
# --- _with_retry: transient MusicBrainz failures (SSL EOF, 503 rate-limit) ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_pick_release_prefers_original_official_over_later_reissue():
|
||||||
|
# Real "Hybrid Theory" bug: MB's arbitrary releases[0] was a 2023 13-track reissue; the 2000
|
||||||
|
# original is what sources deliver, so completeness checks must reference it.
|
||||||
|
releases = [
|
||||||
|
{"id": "reissue", "status": "Official", "date": "2023-06-01"},
|
||||||
|
{"id": "original", "status": "Official", "date": "2000-10-24"},
|
||||||
|
{"id": "deluxe", "status": "Official", "date": "2020-10-09"},
|
||||||
|
]
|
||||||
|
assert _pick_release(releases)["id"] == "original"
|
||||||
|
|
||||||
|
|
||||||
|
def test_pick_release_skips_non_official_and_undated():
|
||||||
|
releases = [
|
||||||
|
{"id": "promo", "status": "Promotion", "date": "1999-01-01"}, # earliest but not official
|
||||||
|
{"id": "undated", "status": "Official", "date": ""},
|
||||||
|
{"id": "official", "status": "Official", "date": "2001-03-03"},
|
||||||
|
]
|
||||||
|
assert _pick_release(releases)["id"] == "official"
|
||||||
|
|
||||||
|
|
||||||
|
def test_pick_release_empty_is_none():
|
||||||
|
assert _pick_release([]) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_retry_returns_value_without_backoff_when_first_call_succeeds():
|
||||||
|
sleeps: list[float] = []
|
||||||
|
calls = {"n": 0}
|
||||||
|
|
||||||
|
def fn():
|
||||||
|
calls["n"] += 1
|
||||||
|
return "ok"
|
||||||
|
|
||||||
|
assert _with_retry(fn, sleep=sleeps.append) == "ok"
|
||||||
|
assert calls["n"] == 1
|
||||||
|
assert sleeps == [] # no backoff when nothing failed
|
||||||
|
|
||||||
|
|
||||||
|
def test_retry_recovers_from_transient_network_error():
|
||||||
|
# The real symptom: <urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING]> arrives
|
||||||
|
# as musicbrainzngs.NetworkError. A couple of retries clears it.
|
||||||
|
sleeps: list[float] = []
|
||||||
|
calls = {"n": 0}
|
||||||
|
|
||||||
|
def fn():
|
||||||
|
calls["n"] += 1
|
||||||
|
if calls["n"] < 3:
|
||||||
|
raise musicbrainzngs.NetworkError("SSL: UNEXPECTED_EOF_WHILE_READING")
|
||||||
|
return "recovered"
|
||||||
|
|
||||||
|
assert _with_retry(fn, sleep=sleeps.append) == "recovered"
|
||||||
|
assert calls["n"] == 3
|
||||||
|
assert sleeps == [1.0, 2.0] # exponential backoff before each retry
|
||||||
|
|
||||||
|
|
||||||
|
def test_retry_recovers_from_transient_response_error():
|
||||||
|
# MusicBrainz's own 503 rate-limit surfaces as ResponseError.
|
||||||
|
calls = {"n": 0}
|
||||||
|
|
||||||
|
def fn():
|
||||||
|
calls["n"] += 1
|
||||||
|
if calls["n"] < 2:
|
||||||
|
raise musicbrainzngs.ResponseError("503 Service Unavailable")
|
||||||
|
return "ok"
|
||||||
|
|
||||||
|
assert _with_retry(fn, sleep=lambda _: None) == "ok"
|
||||||
|
assert calls["n"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_retry_reraises_after_exhausting_attempts():
|
||||||
|
sleeps: list[float] = []
|
||||||
|
|
||||||
|
def fn():
|
||||||
|
raise musicbrainzngs.NetworkError("still down")
|
||||||
|
|
||||||
|
with pytest.raises(musicbrainzngs.NetworkError):
|
||||||
|
_with_retry(fn, attempts=4, sleep=sleeps.append)
|
||||||
|
assert sleeps == [1.0, 2.0, 4.0] # 3 backoffs across 4 attempts
|
||||||
|
|
||||||
|
|
||||||
|
def test_retry_does_not_swallow_non_transient_errors():
|
||||||
|
calls = {"n": 0}
|
||||||
|
|
||||||
|
def fn():
|
||||||
|
calls["n"] += 1
|
||||||
|
raise ValueError("bug, not a network blip")
|
||||||
|
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
_with_retry(fn, sleep=lambda _: None)
|
||||||
|
assert calls["n"] == 1 # non-transient errors propagate immediately, no retry
|
||||||
@@ -1,4 +1,34 @@
|
|||||||
from lyra_worker._mutagen import plan_track_names
|
from lyra_worker._mutagen import _clean_extra_title, plan_track_names
|
||||||
|
|
||||||
|
|
||||||
|
def test_clean_extra_title_strips_number_and_artist_prefix():
|
||||||
|
assert _clean_extra_title(
|
||||||
|
"13. John Mayer - St. Patrick's Day (Album Version)", "John Mayer"
|
||||||
|
) == "St. Patrick's Day (Album Version)"
|
||||||
|
|
||||||
|
|
||||||
|
def test_clean_extra_title_strips_number_without_artist_prefix():
|
||||||
|
assert _clean_extra_title("13. St. Patrick's Day", "John Mayer") == "St. Patrick's Day"
|
||||||
|
|
||||||
|
|
||||||
|
def test_clean_extra_title_leaves_plain_title_and_numeric_titles():
|
||||||
|
assert _clean_extra_title("Hidden Track", "John Mayer") == "Hidden Track"
|
||||||
|
# "99 Luftballons" has no delimiter after the number -> not a track-number prefix
|
||||||
|
assert _clean_extra_title("99 Luftballons", "Nena") == "99 Luftballons"
|
||||||
|
|
||||||
|
|
||||||
|
def test_plan_canonicalizes_bonus_track_beyond_tracklist():
|
||||||
|
names = [f"{i:02d} Track{i}.flac" for i in range(1, 13)] + [
|
||||||
|
"13. John Mayer - St. Patrick's Day (Album Version).flac"
|
||||||
|
]
|
||||||
|
tracklist = tuple(f"Track{i}" for i in range(1, 13))
|
||||||
|
plan = plan_track_names(names, tracklist, artist="John Mayer")
|
||||||
|
assert plan[12] == (
|
||||||
|
"13. John Mayer - St. Patrick's Day (Album Version).flac",
|
||||||
|
"13 St. Patrick's Day (Album Version).flac",
|
||||||
|
"St. Patrick's Day (Album Version)",
|
||||||
|
13,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_plan_uses_tracklist_titles_and_numbers():
|
def test_plan_uses_tracklist_titles_and_numbers():
|
||||||
|
|||||||
@@ -195,6 +195,33 @@ def test_below_threshold_candidates_are_persisted_for_diagnosis(conn):
|
|||||||
assert cur.fetchone()[0] >= 1 # found-but-rejected candidate retained for diagnosis
|
assert cur.fetchone()[0] >= 1 # found-but-rejected candidate retained for diagnosis
|
||||||
|
|
||||||
|
|
||||||
|
def test_searches_run_concurrently_across_adapters(conn):
|
||||||
|
# Two sources that each take ~0.5s to search must run in parallel: total match time is ~the
|
||||||
|
# slowest single search (~0.5s), not the sum (~1.0s). Also verifies concurrency drops no
|
||||||
|
# candidates — both sources' contributions are persisted.
|
||||||
|
import time as _t
|
||||||
|
|
||||||
|
class _SlowQobuz(FakeQobuz):
|
||||||
|
def search(self, target):
|
||||||
|
_t.sleep(0.5)
|
||||||
|
return super().search(target)
|
||||||
|
|
||||||
|
class _SlowSoulseek(FakeSoulseek):
|
||||||
|
def search(self, target):
|
||||||
|
_t.sleep(0.5)
|
||||||
|
return super().search(target)
|
||||||
|
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
start = _t.monotonic()
|
||||||
|
run_pipeline(conn, job_id, [_SlowQobuz(), _SlowSoulseek()], dest_root="/tmp/lib")
|
||||||
|
elapsed = _t.monotonic() - start
|
||||||
|
assert elapsed < 0.9 # concurrent (~0.5s), not sequential (~1.0s)
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT count(DISTINCT source) FROM "Candidate" WHERE "jobId" = %s', (job_id,))
|
||||||
|
assert cur.fetchone()[0] == 2 # both adapters contributed candidates
|
||||||
|
|
||||||
|
|
||||||
def test_duplicate_adapter_names_raise(conn):
|
def test_duplicate_adapter_names_raise(conn):
|
||||||
import pytest
|
import pytest
|
||||||
from lyra_worker.adapters.fakes import FakeQobuz
|
from lyra_worker.adapters.fakes import FakeQobuz
|
||||||
|
|||||||
@@ -0,0 +1,179 @@
|
|||||||
|
from datetime import date
|
||||||
|
|
||||||
|
from lyra_worker.qobuz_gate import (
|
||||||
|
QobuzPacing,
|
||||||
|
today_cap,
|
||||||
|
is_allowed,
|
||||||
|
next_gap_seconds,
|
||||||
|
gate_open,
|
||||||
|
record_download,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---- QobuzPacing.from_config -------------------------------------------------
|
||||||
|
|
||||||
|
def test_from_config_defaults():
|
||||||
|
p = QobuzPacing.from_config({})
|
||||||
|
assert p.enabled is True
|
||||||
|
assert p.active_start == 8 and p.active_end == 23
|
||||||
|
assert p.daily_cap == 40
|
||||||
|
assert p.warmup_start_date == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_from_config_reads_values():
|
||||||
|
p = QobuzPacing.from_config({
|
||||||
|
"qobuz.pacing.enabled": "false",
|
||||||
|
"qobuz.pacing.activeStartHour": "9",
|
||||||
|
"qobuz.pacing.activeEndHour": "22",
|
||||||
|
"qobuz.pacing.dailyCap": "25",
|
||||||
|
"qobuz.pacing.warmupStartDate": "2026-07-15",
|
||||||
|
})
|
||||||
|
assert p.enabled is False
|
||||||
|
assert p.active_start == 9 and p.active_end == 22
|
||||||
|
assert p.daily_cap == 25
|
||||||
|
assert p.warmup_start_date == "2026-07-15"
|
||||||
|
|
||||||
|
|
||||||
|
# ---- today_cap (warm-up ramp) ------------------------------------------------
|
||||||
|
|
||||||
|
def test_today_cap_no_warmup_uses_steady():
|
||||||
|
p = QobuzPacing(daily_cap=40, warmup_start_date="")
|
||||||
|
assert today_cap(p, date(2026, 7, 20)) == 40
|
||||||
|
|
||||||
|
|
||||||
|
def test_today_cap_ramp_boundaries():
|
||||||
|
p = QobuzPacing(daily_cap=40, warmup_start_date="2026-07-15")
|
||||||
|
assert today_cap(p, date(2026, 7, 15)) == 5 # day 0
|
||||||
|
assert today_cap(p, date(2026, 7, 17)) == 5 # day 2
|
||||||
|
assert today_cap(p, date(2026, 7, 18)) == 10 # day 3
|
||||||
|
assert today_cap(p, date(2026, 7, 21)) == 10 # day 6
|
||||||
|
assert today_cap(p, date(2026, 7, 22)) == 20 # day 7
|
||||||
|
assert today_cap(p, date(2026, 7, 28)) == 20 # day 13
|
||||||
|
assert today_cap(p, date(2026, 7, 29)) == 40 # day 14
|
||||||
|
assert today_cap(p, date(2026, 8, 30)) == 40 # long after
|
||||||
|
|
||||||
|
|
||||||
|
def test_today_cap_steady_below_ramp_step_is_the_ceiling():
|
||||||
|
# A small steady cap is never exceeded by a ramp step.
|
||||||
|
p = QobuzPacing(daily_cap=3, warmup_start_date="2026-07-15")
|
||||||
|
assert today_cap(p, date(2026, 7, 29)) == 3
|
||||||
|
|
||||||
|
|
||||||
|
# ---- is_allowed (the gate decision) -----------------------------------------
|
||||||
|
|
||||||
|
_P = QobuzPacing(enabled=True, active_start=8, active_end=23, daily_cap=40)
|
||||||
|
|
||||||
|
|
||||||
|
def test_disabled_always_allows():
|
||||||
|
p = QobuzPacing(enabled=False, active_start=8, active_end=23)
|
||||||
|
assert is_allowed(p, now_hour=3, count_today=999, cap=0, now_epoch=0, next_allowed_epoch=1e18) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_blocked_outside_active_hours():
|
||||||
|
assert is_allowed(_P, now_hour=7, count_today=0, cap=40, now_epoch=100, next_allowed_epoch=0) is False
|
||||||
|
assert is_allowed(_P, now_hour=23, count_today=0, cap=40, now_epoch=100, next_allowed_epoch=0) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_blocked_at_or_over_cap():
|
||||||
|
assert is_allowed(_P, now_hour=10, count_today=40, cap=40, now_epoch=100, next_allowed_epoch=0) is False
|
||||||
|
assert is_allowed(_P, now_hour=10, count_today=41, cap=40, now_epoch=100, next_allowed_epoch=0) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_blocked_before_next_allowed():
|
||||||
|
assert is_allowed(_P, now_hour=10, count_today=0, cap=40, now_epoch=100, next_allowed_epoch=200) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_allowed_when_all_conditions_met():
|
||||||
|
assert is_allowed(_P, now_hour=10, count_today=5, cap=40, now_epoch=300, next_allowed_epoch=200) is True
|
||||||
|
|
||||||
|
|
||||||
|
# ---- next_gap_seconds (spread + jitter) -------------------------------------
|
||||||
|
|
||||||
|
def test_next_gap_spreads_cap_across_active_window():
|
||||||
|
# 15h window (8..23) / cap 40 = 1350s base; ±30%; midpoint rand -> base.
|
||||||
|
p = QobuzPacing(active_start=8, active_end=23, daily_cap=40)
|
||||||
|
assert abs(next_gap_seconds(p, cap=40, rand01=0.5) - 1350.0) < 1.0
|
||||||
|
assert abs(next_gap_seconds(p, cap=40, rand01=0.0) - 1350.0 * 0.7) < 1.0 # low end
|
||||||
|
assert abs(next_gap_seconds(p, cap=40, rand01=1.0) - 1350.0 * 1.3) < 1.0 # high end
|
||||||
|
|
||||||
|
|
||||||
|
def test_next_gap_has_a_floor():
|
||||||
|
# A huge cap would give a tiny gap; a 60s floor protects the request rate.
|
||||||
|
p = QobuzPacing(active_start=8, active_end=23, daily_cap=100000)
|
||||||
|
assert next_gap_seconds(p, cap=100000, rand01=0.0) == 60.0
|
||||||
|
|
||||||
|
|
||||||
|
# ---- DB helpers: gate_open / record_download --------------------------------
|
||||||
|
|
||||||
|
def _set(conn, key, value):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
|
||||||
|
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
|
||||||
|
(key, value),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _get(conn, key):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT value FROM "Config" WHERE key=%s', (key,))
|
||||||
|
row = cur.fetchone()
|
||||||
|
return row[0] if row else None
|
||||||
|
|
||||||
|
|
||||||
|
def test_record_download_increments_and_sets_next_allowed(conn):
|
||||||
|
_set(conn, "qobuz.pacing.dailyCap", "40")
|
||||||
|
record_download(conn, rand01=0.5)
|
||||||
|
assert _get(conn, "qobuz.pacing.countToday") == "1"
|
||||||
|
assert _get(conn, "qobuz.pacing.countDay") is not None
|
||||||
|
assert _get(conn, "qobuz.pacing.nextAllowedAt") is not None
|
||||||
|
record_download(conn, rand01=0.5)
|
||||||
|
assert _get(conn, "qobuz.pacing.countToday") == "2"
|
||||||
|
|
||||||
|
|
||||||
|
def test_record_download_resets_counter_on_new_day(conn):
|
||||||
|
# A stale count from a previous day must reset to 1, not accumulate.
|
||||||
|
_set(conn, "qobuz.pacing.dailyCap", "40")
|
||||||
|
_set(conn, "qobuz.pacing.countDay", "2000-01-01")
|
||||||
|
_set(conn, "qobuz.pacing.countToday", "39")
|
||||||
|
record_download(conn, rand01=0.5)
|
||||||
|
assert _get(conn, "qobuz.pacing.countToday") == "1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_gate_open_with_always_open_window(conn):
|
||||||
|
# An all-day window (0..24) removes the time-of-day dependency: no prior downloads and no
|
||||||
|
# next-allowed set → the gate is open regardless of when the test runs.
|
||||||
|
_set(conn, "qobuz.pacing.activeStartHour", "0")
|
||||||
|
_set(conn, "qobuz.pacing.activeEndHour", "24")
|
||||||
|
assert gate_open(conn) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_gate_closed_by_empty_active_window(conn):
|
||||||
|
# start == end is an empty window (h >= 5 AND h < 5 is never true) → always closed.
|
||||||
|
_set(conn, "qobuz.pacing.activeStartHour", "5")
|
||||||
|
_set(conn, "qobuz.pacing.activeEndHour", "5")
|
||||||
|
assert gate_open(conn) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_gate_closed_when_cap_reached_today(conn):
|
||||||
|
_set(conn, "qobuz.pacing.activeStartHour", "0")
|
||||||
|
_set(conn, "qobuz.pacing.activeEndHour", "24")
|
||||||
|
_set(conn, "qobuz.pacing.dailyCap", "3")
|
||||||
|
# Simulate 3 downloads already today.
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute("SELECT current_date::text")
|
||||||
|
today = cur.fetchone()[0]
|
||||||
|
_set(conn, "qobuz.pacing.countDay", today)
|
||||||
|
_set(conn, "qobuz.pacing.countToday", "3")
|
||||||
|
assert gate_open(conn) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_gate_ignores_stale_count_from_a_previous_day(conn):
|
||||||
|
# A high count from yesterday must not keep the gate closed today.
|
||||||
|
_set(conn, "qobuz.pacing.activeStartHour", "0")
|
||||||
|
_set(conn, "qobuz.pacing.activeEndHour", "24")
|
||||||
|
_set(conn, "qobuz.pacing.dailyCap", "3")
|
||||||
|
_set(conn, "qobuz.pacing.countDay", "2000-01-01")
|
||||||
|
_set(conn, "qobuz.pacing.countToday", "99")
|
||||||
|
assert gate_open(conn) is True
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
"""The Qobuz budget gate, wired into the pipeline: when the gate is closed the job falls through
|
||||||
|
to another source; when open it uses Qobuz and records the download; a Qobuz-only setup is never
|
||||||
|
gated (nothing to fall through to)."""
|
||||||
|
from lyra_worker.adapters.fakes import FakeQobuz, FakeSoulseek
|
||||||
|
from lyra_worker.claim import claim_next
|
||||||
|
from lyra_worker.pipeline import run_pipeline
|
||||||
|
from tests.conftest import insert_request
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg(conn, key, value):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
|
||||||
|
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
|
||||||
|
(key, value),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _open_gate(conn):
|
||||||
|
_cfg(conn, "qobuz.pacing.activeStartHour", "0")
|
||||||
|
_cfg(conn, "qobuz.pacing.activeEndHour", "24")
|
||||||
|
|
||||||
|
|
||||||
|
def _closed_gate(conn):
|
||||||
|
_cfg(conn, "qobuz.pacing.activeStartHour", "5") # empty window (5..5) → always closed
|
||||||
|
_cfg(conn, "qobuz.pacing.activeEndHour", "5")
|
||||||
|
|
||||||
|
|
||||||
|
def _state(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT state FROM "Job" WHERE id = %s', (job_id,))
|
||||||
|
return cur.fetchone()[0]
|
||||||
|
|
||||||
|
|
||||||
|
def _chosen(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT source FROM "Candidate" WHERE "jobId" = %s AND chosen = true', (job_id,))
|
||||||
|
row = cur.fetchone()
|
||||||
|
return row[0] if row else None
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg_get(conn, key):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT value FROM "Config" WHERE key = %s', (key,))
|
||||||
|
row = cur.fetchone()
|
||||||
|
return row[0] if row else None
|
||||||
|
|
||||||
|
|
||||||
|
def test_closed_gate_drops_qobuz_and_falls_through(conn):
|
||||||
|
_closed_gate(conn)
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [FakeQobuz(), FakeSoulseek()], dest_root="/tmp/lib-gate")
|
||||||
|
assert _state(conn, job_id) == "imported"
|
||||||
|
assert _chosen(conn, job_id) == "soulseek" # Qobuz dropped by the closed gate
|
||||||
|
|
||||||
|
|
||||||
|
def test_open_gate_uses_qobuz_and_records_the_download(conn):
|
||||||
|
_open_gate(conn)
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [FakeQobuz(), FakeSoulseek()], dest_root="/tmp/lib-gate")
|
||||||
|
assert _state(conn, job_id) == "imported"
|
||||||
|
assert _chosen(conn, job_id) == "qobuz"
|
||||||
|
assert _cfg_get(conn, "qobuz.pacing.countToday") == "1" # recorded against today's budget
|
||||||
|
assert _cfg_get(conn, "qobuz.pacing.nextAllowedAt") is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_qobuz_only_setup_is_never_gated(conn):
|
||||||
|
_closed_gate(conn)
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [FakeQobuz()], dest_root="/tmp/lib-gate")
|
||||||
|
assert _state(conn, job_id) == "imported"
|
||||||
|
assert _chosen(conn, job_id) == "qobuz" # not dropped — nothing to fall through to
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_qobuz_download_does_not_touch_the_counter(conn):
|
||||||
|
_closed_gate(conn) # forces Soulseek
|
||||||
|
job_id = insert_request(conn, artist="Radiohead", album="In Rainbows")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [FakeQobuz(), FakeSoulseek()], dest_root="/tmp/lib-gate")
|
||||||
|
assert _chosen(conn, job_id) == "soulseek"
|
||||||
|
assert _cfg_get(conn, "qobuz.pacing.countToday") is None # only Qobuz downloads count
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
from lyra_worker.claim import claim_next, requeue_or_fail
|
||||||
|
from tests.conftest import insert_request
|
||||||
|
|
||||||
|
|
||||||
|
def _set_attempts(conn, job_id, attempts, state="matching", stage="match"):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'UPDATE "Job" SET attempts = %s, state = %s, "currentStage" = %s, "claimedAt" = now() '
|
||||||
|
'WHERE id = %s',
|
||||||
|
(attempts, state, stage, job_id),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _job(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT state, "currentStage", "claimedAt", error FROM "Job" WHERE id = %s', (job_id,))
|
||||||
|
return cur.fetchone()
|
||||||
|
|
||||||
|
|
||||||
|
def _request_status(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'SELECT status FROM "Request" WHERE id = (SELECT "requestId" FROM "Job" WHERE id = %s)',
|
||||||
|
(job_id,),
|
||||||
|
)
|
||||||
|
return cur.fetchone()[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_below_cap_requeues_to_requested(conn):
|
||||||
|
# A transient failure (e.g. MB SSL EOF) mid-'matching' must not strand the job — it returns
|
||||||
|
# to 'requested' so claim_next picks it up again instead of orphaning it.
|
||||||
|
jid = insert_request(conn)
|
||||||
|
_set_attempts(conn, jid, 2)
|
||||||
|
requeue_or_fail(conn, jid, "SSL: UNEXPECTED_EOF_WHILE_READING")
|
||||||
|
st, stage, claimed_at, err = _job(conn, jid)
|
||||||
|
assert st == "requested"
|
||||||
|
assert stage == "intake"
|
||||||
|
assert claimed_at is None
|
||||||
|
assert err is None
|
||||||
|
assert _request_status(conn, jid) == "pending"
|
||||||
|
assert claim_next(conn) == jid # immediately re-claimable
|
||||||
|
|
||||||
|
|
||||||
|
def test_at_cap_marks_needs_attention_with_error(conn):
|
||||||
|
jid = insert_request(conn)
|
||||||
|
_set_attempts(conn, jid, 5) # hit the cap
|
||||||
|
requeue_or_fail(conn, jid, "SSL: UNEXPECTED_EOF_WHILE_READING")
|
||||||
|
st, _stage, claimed_at, err = _job(conn, jid)
|
||||||
|
assert st == "needs_attention"
|
||||||
|
assert err == "SSL: UNEXPECTED_EOF_WHILE_READING"
|
||||||
|
assert claimed_at is None
|
||||||
|
assert _request_status(conn, jid) == "needs_attention"
|
||||||
|
assert claim_next(conn) is None # not re-claimed
|
||||||
|
|
||||||
|
|
||||||
|
def test_requeues_until_cap_then_fails(conn):
|
||||||
|
jid = insert_request(conn)
|
||||||
|
for attempts in (1, 2, 3, 4):
|
||||||
|
_set_attempts(conn, jid, attempts)
|
||||||
|
requeue_or_fail(conn, jid, "boom")
|
||||||
|
assert _job(conn, jid)[0] == "requested"
|
||||||
|
_set_attempts(conn, jid, 5)
|
||||||
|
requeue_or_fail(conn, jid, "boom")
|
||||||
|
assert _job(conn, jid)[0] == "needs_attention"
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_job_is_noop(conn):
|
||||||
|
requeue_or_fail(conn, "does-not-exist", "boom") # must not raise
|
||||||
@@ -0,0 +1,218 @@
|
|||||||
|
"""slskd download behaviour: abandoned transfers are cancelled (no pile-up), a stalled peer is
|
||||||
|
dropped fast, a slow-but-progressing peer is kept, and search candidates are ranked fast/free
|
||||||
|
peers first."""
|
||||||
|
import json
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
import lyra_worker.adapters._slskd as slskd_mod
|
||||||
|
from lyra_worker.adapters._slskd import (
|
||||||
|
_ETA_CAP_SECONDS,
|
||||||
|
_MAX_XFER_POLLS,
|
||||||
|
_MIN_XFER_POLLS,
|
||||||
|
_POLL_SECONDS,
|
||||||
|
SlskdClient,
|
||||||
|
_eta_seconds,
|
||||||
|
_max_polls,
|
||||||
|
_parse_search_responses,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _Resp:
|
||||||
|
def __init__(self, payload=None):
|
||||||
|
self._p = payload if payload is not None else {}
|
||||||
|
|
||||||
|
def raise_for_status(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def json(self):
|
||||||
|
return self._p
|
||||||
|
|
||||||
|
|
||||||
|
class _SeqRequests:
|
||||||
|
"""post OK; get() returns successive poll payloads (last one repeats); records DELETEs."""
|
||||||
|
def __init__(self, polls):
|
||||||
|
self._polls = list(polls)
|
||||||
|
self._i = 0
|
||||||
|
self.deleted = []
|
||||||
|
|
||||||
|
def post(self, url, **kw):
|
||||||
|
return _Resp({})
|
||||||
|
|
||||||
|
def get(self, url, **kw):
|
||||||
|
p = self._polls[min(self._i, len(self._polls) - 1)]
|
||||||
|
self._i += 1
|
||||||
|
return _Resp(p)
|
||||||
|
|
||||||
|
def delete(self, url, **kw):
|
||||||
|
self.deleted.append(url)
|
||||||
|
return _Resp({})
|
||||||
|
|
||||||
|
|
||||||
|
def _poll(state, bytes_transferred, fid="t1"):
|
||||||
|
return {"directories": [{"files": [
|
||||||
|
{"filename": "f1.flac", "size": 100, "state": state,
|
||||||
|
"bytesTransferred": bytes_transferred, "id": fid},
|
||||||
|
]}]}
|
||||||
|
|
||||||
|
|
||||||
|
def _client(tmp_path):
|
||||||
|
return SlskdClient({"slskd.url": "http://x", "slskd.api_key": "k",
|
||||||
|
"slskd.downloadsDir": str(tmp_path)})
|
||||||
|
|
||||||
|
|
||||||
|
_REF = json.dumps({"username": "peerA", "files": [{"filename": "f1.flac", "size": 100}]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_stalled_peer_is_abandoned_and_cancelled(tmp_path, monkeypatch):
|
||||||
|
# No byte progress across polls → stall → TimeoutError → the transfer is cancelled.
|
||||||
|
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests([_poll("InProgress", 0)]))
|
||||||
|
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
|
||||||
|
monkeypatch.setattr(slskd_mod, "_STALL_POLLS", 3)
|
||||||
|
fake = slskd_mod.requests
|
||||||
|
|
||||||
|
with pytest.raises(TimeoutError):
|
||||||
|
_client(tmp_path).download(_REF, str(tmp_path / "dest"), lambda *_a: None)
|
||||||
|
assert any(u.endswith("/api/v0/transfers/downloads/peerA/t1") for u in fake.deleted)
|
||||||
|
|
||||||
|
|
||||||
|
def test_error_state_is_cancelled(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests([_poll("Completed, Errored", 0)]))
|
||||||
|
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
|
||||||
|
fake = slskd_mod.requests
|
||||||
|
with pytest.raises(RuntimeError):
|
||||||
|
_client(tmp_path).download(_REF, str(tmp_path / "dest"), lambda *_a: None)
|
||||||
|
assert any(u.endswith("/peerA/t1") for u in fake.deleted)
|
||||||
|
|
||||||
|
|
||||||
|
def test_slow_but_progressing_peer_is_not_abandoned(tmp_path, monkeypatch):
|
||||||
|
# Bytes climb for more polls than _STALL_POLLS, then complete → success, not a premature stall.
|
||||||
|
(tmp_path / "01.flac").write_bytes(b"audio") # so _retrieve finds the finished file
|
||||||
|
ref = json.dumps({"username": "peerA", "files": [{"filename": r"d\Album\01.flac", "size": 5}]})
|
||||||
|
polls = [
|
||||||
|
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 5,
|
||||||
|
"state": "InProgress", "bytesTransferred": b, "id": "t1"}]}]}
|
||||||
|
for b in (10, 20, 30, 40, 50)
|
||||||
|
] + [
|
||||||
|
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 5,
|
||||||
|
"state": "Completed, Succeeded", "bytesTransferred": 5, "id": "t1"}]}]}
|
||||||
|
]
|
||||||
|
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests(polls))
|
||||||
|
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
|
||||||
|
monkeypatch.setattr(slskd_mod, "_STALL_POLLS", 2) # would fire early if progress were ignored
|
||||||
|
|
||||||
|
out = _client(tmp_path).download(ref, str(tmp_path / "dest"), lambda *_a: None)
|
||||||
|
assert out["track_count"] == 1
|
||||||
|
assert slskd_mod.requests.deleted == [] # completed cleanly → nothing cancelled
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_ranks_free_and_fast_peers_first():
|
||||||
|
responses = [
|
||||||
|
{"username": "slow", "hasFreeUploadSlot": True, "uploadSpeed": 100, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"a\Artist - Album\01.flac", "size": 5}]},
|
||||||
|
{"username": "fast", "hasFreeUploadSlot": True, "uploadSpeed": 9000, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"b\Artist - Album\01.flac", "size": 5}]},
|
||||||
|
{"username": "queued", "hasFreeUploadSlot": False, "uploadSpeed": 99999, "queueLength": 40,
|
||||||
|
"files": [{"filename": r"c\Artist - Album\01.flac", "size": 5}]},
|
||||||
|
]
|
||||||
|
peers = [json.loads(c["source_ref"])["username"] for c in _parse_search_responses(responses)]
|
||||||
|
assert peers == ["fast", "slow", "queued"] # free+fast first; no-slot peer last despite speed
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_uses_requested_artist_when_present_in_path():
|
||||||
|
# Real "Rated R" MP3-over-FLAC bug: the artist is in a PARENT folder ("Rihanna\(2009) Rated
|
||||||
|
# R\…"), so the "Artist - Album" folder parse yields no artist and confidence tanks. When the
|
||||||
|
# requested artist is in the path, trust it so the candidate scores as Rihanna.
|
||||||
|
responses = [
|
||||||
|
{"username": "p", "hasFreeUploadSlot": True, "uploadSpeed": 1000, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"@@x\Music\Rihanna\(2009) Rated R\01. Mad House.flac", "size": 5}]},
|
||||||
|
]
|
||||||
|
c = _parse_search_responses(responses, artist="Rihanna")[0]
|
||||||
|
assert c["artist"] == "Rihanna"
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_without_artist_falls_back_to_folder_parse():
|
||||||
|
responses = [
|
||||||
|
{"username": "p", "hasFreeUploadSlot": True, "uploadSpeed": 1000, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"x\Some Band - Some Album\01.flac", "size": 5}]},
|
||||||
|
]
|
||||||
|
c = _parse_search_responses(responses)[0] # no artist arg → folder parse
|
||||||
|
assert c["artist"] == "Some Band"
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_prefers_shorter_transfer_over_raw_speed():
|
||||||
|
# The Hybrid Theory case: a peer advertising higher speed but serving a ~2x-larger rip (24-bit
|
||||||
|
# / bloated FLAC) should lose to a peer with lower speed but a smaller standard rip that
|
||||||
|
# finishes sooner — both are the same quality class to Lyra, so faster-to-finish wins.
|
||||||
|
responses = [
|
||||||
|
{"username": "bloated", "hasFreeUploadSlot": True, "uploadSpeed": 20000, "queueLength": 0,
|
||||||
|
"files": [{"filename": rf"x\LP - Album\{i:02}.flac", "size": 40_000_000} for i in range(12)]},
|
||||||
|
{"username": "lean", "hasFreeUploadSlot": True, "uploadSpeed": 15000, "queueLength": 0,
|
||||||
|
"files": [{"filename": rf"y\LP - Album\{i:02}.flac", "size": 24_000_000} for i in range(12)]},
|
||||||
|
]
|
||||||
|
# bloated est = 480MB/20000 = 24000s; lean est = 288MB/15000 = 19200s → lean finishes first
|
||||||
|
peers = [json.loads(c["source_ref"])["username"] for c in _parse_search_responses(responses)]
|
||||||
|
assert peers[0] == "lean"
|
||||||
|
|
||||||
|
|
||||||
|
def test_eta_seconds_from_measured_rate():
|
||||||
|
assert _eta_seconds(1000, 250, 250.0) == 3 # 750 bytes left / 250 B/s
|
||||||
|
assert _eta_seconds(1000, 0, 100.0) == 10
|
||||||
|
|
||||||
|
|
||||||
|
def test_eta_seconds_unknown_cases():
|
||||||
|
assert _eta_seconds(1000, 250, 0.0) is None # no speed sample yet
|
||||||
|
assert _eta_seconds(0, 0, 100.0) is None # unknown total size
|
||||||
|
assert _eta_seconds(1000, 1000, 100.0) is None # already complete
|
||||||
|
|
||||||
|
|
||||||
|
def test_max_polls_scales_backstop_to_album_size():
|
||||||
|
# A 521MB album (Drake "Scorpion", 25 FLACs) from a slow serial peer must get far more than
|
||||||
|
# the old flat 1200-poll/60min budget, or it gets abandoned mid-download and restarts from zero.
|
||||||
|
scorpion_bytes = 521_000_000
|
||||||
|
polls = _max_polls(scorpion_bytes)
|
||||||
|
assert polls > 1200 # more headroom than the old flat cap
|
||||||
|
assert polls * _POLL_SECONDS >= 90 * 60 # at least ~90 min of wall-clock budget
|
||||||
|
|
||||||
|
# A tiny single stays on the floor, not zero.
|
||||||
|
assert _max_polls(3_000_000) == _MIN_XFER_POLLS
|
||||||
|
assert _max_polls(0) == _MIN_XFER_POLLS
|
||||||
|
|
||||||
|
# An enormous set is clamped to the ceiling (bounds worst-case slot hold).
|
||||||
|
assert _max_polls(50_000_000_000) == _MAX_XFER_POLLS
|
||||||
|
|
||||||
|
# Monotonic: bigger album never gets a smaller budget.
|
||||||
|
assert _max_polls(200_000_000) <= _max_polls(400_000_000)
|
||||||
|
|
||||||
|
|
||||||
|
def test_eta_seconds_capped_when_peer_nearly_stalls():
|
||||||
|
# A near-stalled peer (tiny speed) would yield an ETA far past int4, overflowing the
|
||||||
|
# downloadEtaSeconds column write. The estimate must be clamped to a safe ceiling.
|
||||||
|
eta = _eta_seconds(30_000_000, 0, 0.001) # 30MB / 0.001 B/s ~= 3e10s uncapped
|
||||||
|
assert eta == _ETA_CAP_SECONDS
|
||||||
|
assert eta < 2_147_483_647 # fits in int4
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_reports_shrinking_eta(tmp_path, monkeypatch):
|
||||||
|
# A 900-byte file arriving 300 bytes per 3s poll → measured speed 100 B/s → ETA counts down
|
||||||
|
# (6s, 3s) then None on completion. on_progress gets (fraction, eta_seconds).
|
||||||
|
(tmp_path / "01.flac").write_bytes(b"x" * 900) # so _retrieve finds the finished file
|
||||||
|
ref = json.dumps({"username": "peerA", "files": [{"filename": r"d\Album\01.flac", "size": 900}]})
|
||||||
|
polls = [
|
||||||
|
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 900,
|
||||||
|
"state": "InProgress", "bytesTransferred": b, "id": "t1"}]}]}
|
||||||
|
for b in (300, 600)
|
||||||
|
] + [
|
||||||
|
{"directories": [{"files": [{"filename": r"d\Album\01.flac", "size": 900,
|
||||||
|
"state": "Completed, Succeeded", "bytesTransferred": 900, "id": "t1"}]}]}
|
||||||
|
]
|
||||||
|
monkeypatch.setattr(slskd_mod, "requests", _SeqRequests(polls))
|
||||||
|
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
|
||||||
|
clock = iter(float(i) * 3 for i in range(100)) # 0,3,6,… deterministic monotonic time
|
||||||
|
monkeypatch.setattr(slskd_mod.time, "monotonic", lambda: next(clock))
|
||||||
|
|
||||||
|
calls: list = []
|
||||||
|
_client(tmp_path).download(ref, str(tmp_path / "dest"), lambda *a: calls.append(a))
|
||||||
|
|
||||||
|
etas = [a[1] for a in calls if len(a) > 1]
|
||||||
|
assert etas == [6, 3, None] # counts down at 100 B/s, then None once complete
|
||||||
@@ -0,0 +1,150 @@
|
|||||||
|
"""Soulseek search fallback: the Soulseek server drops searches containing blocklisted terms
|
||||||
|
(major-label artists / album titles) → 0 responses. search_album retries with album-only then
|
||||||
|
artist-only, keeping only path-matching results so it never grabs a same-titled foreign album."""
|
||||||
|
import json
|
||||||
|
|
||||||
|
import lyra_worker.adapters._slskd as slskd_mod
|
||||||
|
from lyra_worker.adapters._slskd import (
|
||||||
|
SlskdClient,
|
||||||
|
_keep_matching,
|
||||||
|
_norm,
|
||||||
|
_path_matches,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- pure helpers -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_norm_strips_diacritics_and_punctuation():
|
||||||
|
assert _norm("Beyoncé") == "beyonce"
|
||||||
|
assert _norm("I Am… Sasha Fierce") == "i am sasha fierce"
|
||||||
|
assert _norm("AC/DC") == "ac dc"
|
||||||
|
|
||||||
|
|
||||||
|
def test_path_matches_is_accent_and_punctuation_tolerant():
|
||||||
|
p = r"@@bckfj\All\Beyonce\I Am... Sasha Fierce CD1\03 Halo.flac"
|
||||||
|
assert _path_matches("Beyoncé", p) # accent folded
|
||||||
|
assert _path_matches("I Am… Sasha Fierce", p) # ellipsis / punctuation ignored
|
||||||
|
assert not _path_matches("Rihanna", p)
|
||||||
|
|
||||||
|
|
||||||
|
def test_path_matches_short_needle_requires_whole_string():
|
||||||
|
assert _path_matches("U2", r"x\U2 - Achtung Baby\01.flac")
|
||||||
|
assert not _path_matches("U2", r"x\Blur - 13\01.flac")
|
||||||
|
|
||||||
|
|
||||||
|
def test_keep_matching_drops_foreign_and_empty_responses():
|
||||||
|
responses = [
|
||||||
|
{"username": "a", "files": [{"filename": r"x\Rihanna - Unapologetic\01.flac"}]},
|
||||||
|
{"username": "b", "files": [{"filename": r"x\Someone Else - Unapologetic\01.flac"}]},
|
||||||
|
]
|
||||||
|
kept = _keep_matching(responses, "Rihanna")
|
||||||
|
assert [r["username"] for r in kept] == ["a"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- fallback orchestration (HTTP faked) ------------------------------------------------------
|
||||||
|
|
||||||
|
class _Resp:
|
||||||
|
def __init__(self, payload):
|
||||||
|
self._p = payload
|
||||||
|
|
||||||
|
def raise_for_status(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def json(self):
|
||||||
|
return self._p
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeSearch:
|
||||||
|
"""Routes slskd search HTTP by URL: POST returns an id, GET on the search returns a completed
|
||||||
|
state, GET on /responses returns the canned responses for that query. Records queries sent."""
|
||||||
|
def __init__(self, by_query):
|
||||||
|
self.by_query = by_query # {searchText: [responses]}
|
||||||
|
self.queries = [] # searchText values POSTed, in order
|
||||||
|
self._id_query = {}
|
||||||
|
self._n = 0
|
||||||
|
|
||||||
|
def post(self, url, **kw):
|
||||||
|
text = kw["json"]["searchText"]
|
||||||
|
self.queries.append(text)
|
||||||
|
self._n += 1
|
||||||
|
sid = f"s{self._n}"
|
||||||
|
self._id_query[sid] = text
|
||||||
|
return _Resp({"id": sid})
|
||||||
|
|
||||||
|
def get(self, url, **kw):
|
||||||
|
if url.endswith("/responses"):
|
||||||
|
sid = url.rsplit("/", 2)[-2]
|
||||||
|
return _Resp(self.by_query.get(self._id_query[sid], []))
|
||||||
|
return _Resp({"state": "Completed, TimedOut"})
|
||||||
|
|
||||||
|
|
||||||
|
def _client():
|
||||||
|
return SlskdClient({"slskd.url": "http://x", "slskd.api_key": "k", "slskd.downloadsDir": "/x"})
|
||||||
|
|
||||||
|
|
||||||
|
def _install(monkeypatch, fake):
|
||||||
|
monkeypatch.setattr(slskd_mod, "requests", fake)
|
||||||
|
monkeypatch.setattr(slskd_mod.time, "sleep", lambda _s: None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_primary_hit_does_not_trigger_fallback(monkeypatch):
|
||||||
|
fake = _FakeSearch({
|
||||||
|
"Rihanna Unapologetic": [
|
||||||
|
{"username": "a", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"x\Rihanna - Unapologetic\01.flac", "size": 5}]},
|
||||||
|
],
|
||||||
|
})
|
||||||
|
_install(monkeypatch, fake)
|
||||||
|
out = _client().search_album("Rihanna", "Unapologetic")
|
||||||
|
assert len(out) == 1
|
||||||
|
assert fake.queries == ["Rihanna Unapologetic"] # no fallback searches
|
||||||
|
|
||||||
|
|
||||||
|
def test_album_only_fallback_recovers_blocklisted_artist(monkeypatch):
|
||||||
|
# Primary (artist blocked) → 0; album-only returns the artist's folder → recovered.
|
||||||
|
fake = _FakeSearch({
|
||||||
|
"Beyoncé I Am… Sasha Fierce": [], # blocked
|
||||||
|
"i am sasha fierce": [
|
||||||
|
{"username": "a", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"x\Beyonce - I Am... Sasha Fierce\01.flac", "size": 5}]},
|
||||||
|
{"username": "b", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"x\Other Artist - I Am Sasha Fierce\01.flac", "size": 5}]},
|
||||||
|
],
|
||||||
|
})
|
||||||
|
_install(monkeypatch, fake)
|
||||||
|
out = _client().search_album("Beyoncé", "I Am… Sasha Fierce")
|
||||||
|
assert fake.queries == ["Beyoncé I Am… Sasha Fierce", "i am sasha fierce"]
|
||||||
|
peers = [json.loads(c["source_ref"])["username"] for c in out]
|
||||||
|
assert peers == ["a"] # foreign same-titled album dropped
|
||||||
|
|
||||||
|
|
||||||
|
def test_artist_only_fallback_when_album_title_blocked(monkeypatch):
|
||||||
|
# Album-only also blocked/empty → fall through to artist-only, filtered by album in path.
|
||||||
|
fake = _FakeSearch({
|
||||||
|
"Foo Lemonade": [],
|
||||||
|
"lemonade": [], # album title itself blocked
|
||||||
|
"foo": [
|
||||||
|
{"username": "a", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"x\Foo - Lemonade\01.flac", "size": 5}]},
|
||||||
|
],
|
||||||
|
})
|
||||||
|
_install(monkeypatch, fake)
|
||||||
|
out = _client().search_album("Foo", "Lemonade")
|
||||||
|
assert fake.queries == ["Foo Lemonade", "lemonade", "foo"]
|
||||||
|
assert len(out) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_fallback_yields_nothing_when_title_too_generic(monkeypatch):
|
||||||
|
# 'Adele 21': artist blocked, album title '21' returns lots of non-Adele content → filtered
|
||||||
|
# to nothing rather than grabbing a wrong album.
|
||||||
|
fake = _FakeSearch({
|
||||||
|
"Adele 21": [],
|
||||||
|
"21": [
|
||||||
|
{"username": "z", "hasFreeUploadSlot": True, "uploadSpeed": 1, "queueLength": 0,
|
||||||
|
"files": [{"filename": r"x\21 Savage - Savage Mode\01.flac", "size": 5}]},
|
||||||
|
],
|
||||||
|
"adele": [], # artist blocked
|
||||||
|
})
|
||||||
|
_install(monkeypatch, fake)
|
||||||
|
out = _client().search_album("Adele", "21")
|
||||||
|
assert out == []
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
"""MusicBrainz picks an arbitrary releases[0] that is often a deluxe/expanded edition with more
|
||||||
|
tracks (and a longer runtime) than the standard album a source legitimately delivers. Completeness
|
||||||
|
and the duration guard must be judged against what the CHOSEN SOURCE offered, not MB's canonical
|
||||||
|
edition — otherwise a complete standard-edition download is falsely rejected as 'incomplete'
|
||||||
|
(The Script "No Sound Without Silence": 11 delivered vs MB 12) or 'too short' (Skillet "Unleashed":
|
||||||
|
12 delivered ~40 min vs MB's 20-track deluxe ~73 min). A genuinely short download must still fail.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
from dataclasses import replace
|
||||||
|
|
||||||
|
from lyra_worker.adapters.fakes import FakeQobuz
|
||||||
|
from lyra_worker.claim import claim_next
|
||||||
|
from lyra_worker.pipeline import run_pipeline
|
||||||
|
from lyra_worker.types import DownloadResult, MBTarget
|
||||||
|
from tests.conftest import insert_request
|
||||||
|
|
||||||
|
|
||||||
|
class _StandardEditionQobuz(FakeQobuz):
|
||||||
|
"""Qobuz-like source offering a fixed-size *standard* edition (source_tracks), independent of
|
||||||
|
MB's possibly-larger deluxe count; stages that many real (non-empty) files and reports them."""
|
||||||
|
def __init__(self, source_tracks):
|
||||||
|
super().__init__()
|
||||||
|
self._n = source_tracks
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
return super().search(replace(target, track_count=self._n))
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
os.makedirs(dest, exist_ok=True)
|
||||||
|
for i in range(candidate.track_count):
|
||||||
|
with open(os.path.join(dest, f"{i + 1:02d} Track.flac"), "wb") as fh:
|
||||||
|
fh.write(b"audio")
|
||||||
|
on_progress(1.0)
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=candidate.track_count)
|
||||||
|
|
||||||
|
|
||||||
|
class _PartialQobuz(FakeQobuz):
|
||||||
|
"""Promises `promised` tracks but one fails mid-download: stages and reports `promised - 1`
|
||||||
|
(the real John Mayer "Roll it on Home" IncompleteRead case)."""
|
||||||
|
def __init__(self, promised):
|
||||||
|
super().__init__()
|
||||||
|
self._n = promised
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
return super().search(replace(target, track_count=self._n))
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
os.makedirs(dest, exist_ok=True)
|
||||||
|
delivered = candidate.track_count - 1
|
||||||
|
for i in range(delivered):
|
||||||
|
with open(os.path.join(dest, f"{i + 1:02d} Track.flac"), "wb") as fh:
|
||||||
|
fh.write(b"audio")
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=delivered)
|
||||||
|
|
||||||
|
|
||||||
|
class _DeluxeResolver:
|
||||||
|
"""Resolver whose canonical release is a larger (deluxe) edition than the source delivers."""
|
||||||
|
def __init__(self, mb_tracks, mb_total_s):
|
||||||
|
self._t, self._s = mb_tracks, mb_total_s
|
||||||
|
|
||||||
|
def resolve(self, artist, album):
|
||||||
|
return MBTarget(artist=artist, album=album, track_count=self._t, total_duration_s=self._s)
|
||||||
|
|
||||||
|
|
||||||
|
def _state_error(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT state, error FROM "Job" WHERE id = %s', (job_id,))
|
||||||
|
return cur.fetchone()
|
||||||
|
|
||||||
|
|
||||||
|
def test_standard_edition_not_rejected_as_incomplete(conn):
|
||||||
|
# The Script: source delivers the 11-track standard edition; MB's canonical release is 12.
|
||||||
|
job_id = insert_request(conn, artist="The Script", album="No Sound Without Silence")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [_StandardEditionQobuz(source_tracks=11)],
|
||||||
|
resolver=_DeluxeResolver(mb_tracks=12, mb_total_s=2943),
|
||||||
|
measure_duration=lambda _s: 2700.0, dest_root="/tmp/lib-se")
|
||||||
|
assert _state_error(conn, job_id) == ("imported", None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_smaller_edition_not_rejected_as_too_short(conn):
|
||||||
|
# Skillet: source delivers the 12-track standard (~40 min); MB's release is a 20-track deluxe
|
||||||
|
# (~73 min). The unscaled duration guard falsely rejects; scaling to the delivered edition
|
||||||
|
# (12/20) must let it pass.
|
||||||
|
job_id = insert_request(conn, artist="Skillet", album="Unleashed")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [_StandardEditionQobuz(source_tracks=12)],
|
||||||
|
resolver=_DeluxeResolver(mb_tracks=20, mb_total_s=4380),
|
||||||
|
measure_duration=lambda _s: 2400.0, dest_root="/tmp/lib-se")
|
||||||
|
assert _state_error(conn, job_id) == ("imported", None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_genuinely_incomplete_still_rejected(conn):
|
||||||
|
# John Mayer: the source promised 12 tracks but one failed mid-download → only 11 land.
|
||||||
|
# This is a real partial and must still be rejected (the fix must not swallow it).
|
||||||
|
job_id = insert_request(conn, artist="John Mayer", album="The Search for Everything")
|
||||||
|
claim_next(conn)
|
||||||
|
run_pipeline(conn, job_id, [_PartialQobuz(promised=12)],
|
||||||
|
resolver=_DeluxeResolver(mb_tracks=12, mb_total_s=2580),
|
||||||
|
measure_duration=lambda _s: 2400.0, dest_root="/tmp/lib-se")
|
||||||
|
assert _state_error(conn, job_id) == ("needs_attention", "incomplete download")
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
from lyra_worker.adapters._streamrip import _ProgressAggregator
|
||||||
|
|
||||||
|
|
||||||
|
def _record():
|
||||||
|
seen: list[float] = []
|
||||||
|
return seen, seen.append
|
||||||
|
|
||||||
|
|
||||||
|
def test_equal_size_tracks_aggregate_monotonically_to_near_one():
|
||||||
|
# 4 tracks of 100 bytes each, all started up front (concurrency covers the album), then
|
||||||
|
# each downloaded in 25-byte chunks. Fraction must climb monotonically 0→~1.
|
||||||
|
seen, cb = _record()
|
||||||
|
agg = _ProgressAggregator(4, cb)
|
||||||
|
for _ in range(4):
|
||||||
|
agg.start_track(100)
|
||||||
|
for _ in range(16): # 16 chunks * 25 bytes = 400 bytes = full album
|
||||||
|
agg.advance(25)
|
||||||
|
|
||||||
|
assert seen == sorted(seen) # monotonic non-decreasing
|
||||||
|
assert all(0.0 <= f <= 0.99 for f in seen) # never negative, capped at 0.99
|
||||||
|
assert seen[-1] == 0.99 # reaches the cap when all bytes are in
|
||||||
|
assert seen[len(seen) // 2] > 0.3 # genuinely climbs mid-download (no 0→100 jump)
|
||||||
|
|
||||||
|
|
||||||
|
def test_estimate_uses_track_count_before_all_tracks_start():
|
||||||
|
# Only the first of 4 tracks has started; with a known track count the estimate already
|
||||||
|
# spans the whole album, so finishing track 1 reports ~1/4, not ~1/1.
|
||||||
|
seen, cb = _record()
|
||||||
|
agg = _ProgressAggregator(4, cb)
|
||||||
|
agg.start_track(100)
|
||||||
|
agg.advance(100) # track 1 fully downloaded
|
||||||
|
assert abs(seen[-1] - 0.25) < 0.01
|
||||||
|
|
||||||
|
|
||||||
|
def test_zero_or_missing_sizes_never_emit_and_never_divide_by_zero():
|
||||||
|
seen, cb = _record()
|
||||||
|
agg = _ProgressAggregator(0, cb)
|
||||||
|
agg.advance(50) # no track started yet -> no emit
|
||||||
|
agg.start_track(0) # a track with unknown size
|
||||||
|
agg.advance(50) # est_total still 0 -> still no emit, no crash
|
||||||
|
assert seen == []
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
"""Upgrade guard: when an album is already in the library and no currently-available source can
|
||||||
|
beat it (e.g. Qobuz is gated by pacing and only a same-quality Soulseek copy is left), the pipeline
|
||||||
|
must NOT waste a download — keep-if-better would discard it anyway. It should skip and let the
|
||||||
|
release be re-attempted later (when Qobuz frees up). A genuinely better source still upgrades."""
|
||||||
|
import os
|
||||||
|
|
||||||
|
from lyra_worker.claim import claim_next
|
||||||
|
from lyra_worker.pipeline import run_pipeline
|
||||||
|
from lyra_worker.types import Candidate, DownloadResult, Quality
|
||||||
|
|
||||||
|
_CD = Quality(fmt="FLAC", lossless=True, bit_depth=16, sample_rate=44100) # class 2
|
||||||
|
_HIRES = Quality(fmt="FLAC", lossless=True, bit_depth=24, sample_rate=96000) # class 3
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg(conn, key, value):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Config"(key,value,secret,"updatedAt") VALUES (%s,%s,false,now()) '
|
||||||
|
'ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, "updatedAt"=now()',
|
||||||
|
(key, value),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
class _Counting:
|
||||||
|
def __init__(self, name, tier, quality):
|
||||||
|
self.name, self.tier, self._q = name, tier, quality
|
||||||
|
self.calls = 0
|
||||||
|
self.searches = 0
|
||||||
|
|
||||||
|
def health(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
def search(self, target):
|
||||||
|
self.searches += 1
|
||||||
|
return [Candidate(source=self.name, source_ref=self.name, matched_artist=target.artist,
|
||||||
|
matched_album=target.album, quality=self._q, track_count=3, source_tier=self.tier)]
|
||||||
|
|
||||||
|
def download(self, candidate, dest, on_progress):
|
||||||
|
self.calls += 1
|
||||||
|
os.makedirs(dest, exist_ok=True)
|
||||||
|
for i in range(3):
|
||||||
|
with open(os.path.join(dest, f"0{i}.flac"), "wb") as fh:
|
||||||
|
fh.write(b"a")
|
||||||
|
return DownloadResult(ok=True, path=dest, track_count=3)
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_library(conn, artist, album, quality_class, path):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "LibraryItem" (id, artist, album, path, source, format, "qualityClass", "importedAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, %s, %s, %s, 'soulseek', 'FLAC', %s, now())",
|
||||||
|
(artist, album, path, quality_class),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _insert_upgrade_job(conn, artist, album):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "MonitoredRelease" (id, "artistMbid", "artistName", "rgMbid", album, '
|
||||||
|
'monitored, state, "secondaryTypes", "createdAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, 'm', %s, 'rg', %s, true, 'grabbed', '{}', now()) RETURNING id",
|
||||||
|
(artist, album),
|
||||||
|
)
|
||||||
|
rel_id = cur.fetchone()[0]
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Request" (id, artist, album, status, "monitoredReleaseId", "createdAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, %s, %s, 'pending', %s, now()) RETURNING id",
|
||||||
|
(artist, album, rel_id),
|
||||||
|
)
|
||||||
|
req = cur.fetchone()[0]
|
||||||
|
cur.execute(
|
||||||
|
'INSERT INTO "Job" (id, "requestId", state, "currentStage", attempts, "createdAt", "updatedAt") '
|
||||||
|
"VALUES (gen_random_uuid()::text, %s, 'requested', 'intake', 0, now(), now()) RETURNING id",
|
||||||
|
(req,),
|
||||||
|
)
|
||||||
|
job = cur.fetchone()[0]
|
||||||
|
conn.commit()
|
||||||
|
return job
|
||||||
|
|
||||||
|
|
||||||
|
def _lib(conn, artist, album):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT path, "qualityClass" FROM "LibraryItem" WHERE artist=%s AND album=%s', (artist, album))
|
||||||
|
return cur.fetchone()
|
||||||
|
|
||||||
|
|
||||||
|
def _state(conn, job_id):
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT state FROM "Job" WHERE id=%s', (job_id,))
|
||||||
|
return cur.fetchone()[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_skipped_pre_search_when_qobuz_gated(conn):
|
||||||
|
# Qobuz gated (empty active window) + existing >= CD copy: only hi-res Qobuz could improve it,
|
||||||
|
# so defer WITHOUT even searching — avoids re-searching every owned album each monitor cycle.
|
||||||
|
_cfg(conn, "qobuz.pacing.activeStartHour", "5")
|
||||||
|
_cfg(conn, "qobuz.pacing.activeEndHour", "5") # empty window → gate closed
|
||||||
|
_seed_library(conn, "A", "B", 2, "/m/existing")
|
||||||
|
job_id = _insert_upgrade_job(conn, "A", "B")
|
||||||
|
claim_next(conn)
|
||||||
|
soulseek = _Counting("soulseek", 1, _CD)
|
||||||
|
run_pipeline(conn, job_id, [soulseek], dest_root="/tmp/lib-ug", upgrade_cutoff=3)
|
||||||
|
assert soulseek.searches == 0 # no wasted search
|
||||||
|
assert soulseek.calls == 0 # no wasted download
|
||||||
|
assert _lib(conn, "A", "B") == ("/m/existing", 2)
|
||||||
|
assert _state(conn, job_id) == "imported"
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_skipped_post_search_when_only_same_quality_found(conn):
|
||||||
|
# Gate OPEN, so it searches, but the only source found is same-quality → skip the download.
|
||||||
|
_cfg(conn, "qobuz.pacing.activeStartHour", "0")
|
||||||
|
_cfg(conn, "qobuz.pacing.activeEndHour", "24") # gate open
|
||||||
|
_seed_library(conn, "A", "B", 2, "/m/existing")
|
||||||
|
job_id = _insert_upgrade_job(conn, "A", "B")
|
||||||
|
claim_next(conn)
|
||||||
|
soulseek = _Counting("soulseek", 1, _CD)
|
||||||
|
run_pipeline(conn, job_id, [soulseek], dest_root="/tmp/lib-ug", upgrade_cutoff=3)
|
||||||
|
assert soulseek.searches == 1 # it did search (gate open)
|
||||||
|
assert soulseek.calls == 0 # but skipped the useless download
|
||||||
|
assert _lib(conn, "A", "B") == ("/m/existing", 2)
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_proceeds_when_a_better_source_is_available(conn):
|
||||||
|
# Existing class-2 copy; a class-3 (hi-res) source IS available → download + upgrade.
|
||||||
|
_seed_library(conn, "A", "B", 2, "/m/existing")
|
||||||
|
job_id = _insert_upgrade_job(conn, "A", "B")
|
||||||
|
claim_next(conn)
|
||||||
|
qobuz = _Counting("qobuz", 0, _HIRES)
|
||||||
|
run_pipeline(conn, job_id, [qobuz], dest_root="/tmp/lib-ug2", upgrade_cutoff=3)
|
||||||
|
assert qobuz.calls == 1 # downloaded the better copy
|
||||||
|
assert _lib(conn, "A", "B")[1] == 3 # upgraded to hi-res
|
||||||
Reference in New Issue
Block a user