diff --git a/README.md b/README.md index 3c2bb4a..738082e 100644 --- a/README.md +++ b/README.md @@ -7,83 +7,54 @@ sources of differing quality: **Qobuz** (lossless/hi-res), **Soulseek** (P2P), a **ListenBrainz** and **Last.fm** (Spotify's recommendation/related-artists APIs were retired for new apps, so discovery is MetaBrainz-native and MBID-clean — no Spotify). -## Status +It runs on your own server as a small `docker compose` stack. This README covers getting +it running; see **[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md)** for the full feature tour, +publishing pre-built images, and running behind a VPN. -Built in three slices: +## What it does -1. **Acquisition engine** — ✅ **done**. Resolve a request against MusicBrainz, - match/rank across Qobuz/Soulseek/YouTube, download the best source into an - isolated staging dir, verify completeness, tag (mutagen), and atomically promote - the clean album into `Artist/Album (Year)/## Title.ext`. -2. **Library manager** — ✅ **done**. Watched artists (live MB search + follow), - per-release monitor toggles + auto-monitor-future, a wanted list with retry + - quality-upgrade window, and a worker "monitor" that discovers new releases and - auto-grabs through the acquisition pipeline. Plus a **library scan** that ingests - existing on-disk albums as have + monitored + followed, and a per-artist - studio-only / show-all release-type filter. -3. **Discovery** — ✅ **done**. Recommendation-driven finding of new music via - **ListenBrainz** similar-artists and **Last.fm** `artist.getSimilar` (both - MBID-native), behind a pluggable `SimilaritySource` interface — Last.fm registers - as a second source automatically once its API key is set, and cross-source scores - are summed per candidate. A background sweep (and a "Discover now" button) - aggregates artists similar to the ones you follow into a manual review feed of - suggested **artists** and **albums** at `/discover`; **Follow** and **Want** - reuse the slice-2 follow / wanted flows, **Dismiss** is permanent. Plus a - web-side seed-search box. Every suggestion and seed-search result opens a - read-only **artist preview** (`/discover/artist/[mbid]`) — discography, - per-album tracklists, and "in library" / "monitored" badges — where you can - Follow or Want before committing. No Spotify (its recommendation APIs are dead - for new apps). +- **Acquisition** — resolve a request against MusicBrainz, match/rank across + Qobuz/Soulseek/YouTube, download the best source, verify completeness, tag, and organize + into `Artist/Album (Year)/## Title.ext`. +- **Library manager** — follow artists (live MB search), per-release monitor toggles, a + wanted list with retry + quality-upgrade, a background monitor that auto-grabs new + releases, and a scan that ingests albums already on disk. +- **Discovery** (`/discover`) — recommendation-driven suggestions from ListenBrainz + + Last.fm similar-artists, with Follow / Want / Dismiss and a read-only artist preview. +- **Browse & watch** — Last.fm top artists/albums (`/lastfm`, with inline Follow/Want), a + cover-art **Library** grid (`/library`), and **The Floor** (home): a live queue of + in-flight acquisitions with three-phase progress and a Retry button. -Alongside the three slices: - -- **Last.fm browse** (`/lastfm`) — your Last.fm top artists and albums (by period), - with a "hide items already in my library" toggle; each artist opens a modal of your - most-played songs and albums, and every album is one click to **Want**. -- **Library** (`/library`) — a cover-art grid of everything on disk, filterable, each - album opening a tracklist modal. -- **The Floor** (home) — a live queue of in-flight acquisitions with three-phase - progress (**Searching → Downloading → Finishing**), a live download percentage, a - per-job "pressing ticket" (source · format · tracks · elapsed), and a **Retry** - button on anything that needs attention. - -The whole UI runs a hand-rolled "Pressing Plant" design system (warm-paper theme, self-hosted -Fraunces serif for prose vs. system mono for machine data, light/dark aware). - -**Operational notes:** the monitor and the discovery sweep are both **off by -default** — set `monitor.enabled=true` (or Settings → Monitor, + optional `monitor.*` -tuning) to activate auto-grab/upgrade, and `discover.enabled=true` (or -Settings → Discovery, + optional `discover.*` tuning) to activate the scheduled -discovery sweep. The `/discover` "Discover now" button and seed-search work with the -sweep off. Credentials (Qobuz, Soulseek, Last.fm) are entered in Settings and persist -across `docker compose up -d --build` rebuilds (don't use `down -v`; secrets are -encrypted at rest with `LYRA_SECRET_KEY`). Tests run against a separate `lyra_test` -database (a guard blocks the live DB). Deferred hardening + the full roadmap live in -the implementation-plan "Notes" sections under -[`docs/superpowers/plans/`](docs/superpowers/plans/). +Full detail on each of these is in [docs/DEPLOYMENT.md → Features in depth](docs/DEPLOYMENT.md#features-in-depth). ## Architecture -The `docker compose` stack is three services on a home server: +The `docker compose` stack is a few services on one host: -- **db** — Postgres, the shared source of truth. +- **db** — Postgres, the shared source of truth (all durable state lives here). - **web** — the Next.js app: UI + API (search, request, queue, progress, settings). -- **worker** — the Python worker: all source integration (streamrip, yt-dlp, slskd - client, mutagen; MusicBrainz + ListenBrainz + Last.fm for metadata and discovery), - running the six-stage acquisition pipeline plus the background monitor and discovery - sweeps. +- **worker** — the Python worker: all source integration (streamrip, yt-dlp, slskd client, + mutagen; MusicBrainz + ListenBrainz + Last.fm for metadata and discovery), running the + acquisition pipeline plus the background monitor and discovery sweeps. +- **db-backup** — a sidecar that `pg_dump`s the database on a schedule (see [Backup & + restore](#backup--restore)). -Soulseek support talks to an **external slskd** daemon (an off-the-shelf headless -Soulseek client) — the worker reaches it via the `slskd.url` + `slskd.api_key` -credentials set in Settings, so run slskd separately rather than as part of this stack. +Soulseek support talks to an **external slskd** daemon (an off-the-shelf headless Soulseek +client) — the worker reaches it via the `slskd.url` + `slskd.api_key` credentials set in +Settings, so run slskd separately rather than as part of this stack. -A shared `/music` volume is the library, with a separate `/staging` volume for -in-progress downloads (point `STAGING_DIR` at a fast local disk when `MUSIC_DIR` is a -network share). +A shared `/music` volume is the library, with a separate `/staging` volume for in-progress +downloads (point `STAGING_DIR` at a fast local disk when `MUSIC_DIR` is a network share). -See [`docs/superpowers/specs/`](docs/superpowers/specs/) for the full design. +## Prerequisites -## Running it +- **Docker** + **Docker Compose**. +- A directory for the music library (set `MUSIC_DIR`). +- Optional, entered in **Settings** after first run: a **Qobuz** account (lossless/hi-res), + a running **slskd** daemon (Soulseek), and a **Last.fm** API key (browse + discovery). + None are required to start — Lyra runs with whatever you configure. + +## Quick start ```sh cp .env.example .env # then edit: set LYRA_SECRET_KEY (openssl rand -base64 32) and MUSIC_DIR @@ -97,33 +68,26 @@ the placeholder. Once set, keep it stable: changing it makes already-stored cred undecryptable (you'd re-enter them in Settings). Optionally set `LYRA_PASSWORD` too (see [Security](#security)). -The web entrypoint runs `prisma migrate deploy` on every rebuild, so schema changes -reach the live DB automatically. Enter Qobuz / Soulseek / Last.fm credentials in -**Settings** (they persist across rebuilds), then follow an artist or request an album -to feed The Floor. +The web entrypoint runs `prisma migrate deploy` on every rebuild, so schema changes reach +the live DB automatically. Enter Qobuz / Soulseek / Last.fm credentials in **Settings** (they +persist across rebuilds), then follow an artist or request an album to feed The Floor. -## Deploying to a server (pre-built images) +`.env.example` documents every setting (`MUSIC_DIR`, `STAGING_DIR`, `SLSKD_DOWNLOADS_DIR`, +backup knobs). Note in particular: point `SLSKD_DOWNLOADS_DIR` at slskd's own downloads +directory (Lyra must run on the same host as slskd for Soulseek delivery to work), and point +`STAGING_DIR` at a fast local disk if `MUSIC_DIR` is a network share. -For a server that should **pull** rather than build, the `web` and `worker` images are -published to the Gitea container registry and consumed by a compose file in the -[`vm-download`](https://git.jger.nl/Jonathan/vm-download) ops repo (`docker/lyra/`). +## Operational notes -Publish a new release from this repo (amd64): - -```sh -docker login git.jger.nl # once — Gitea user + a token with package:write -./scripts/build-push.sh # builds + pushes lyra-web / lyra-worker (:latest and :) -``` - -Then on the server: - -```sh -cd /opt/git/vm-download && git pull -cd docker/lyra && docker compose pull && docker compose up -d -``` - -`db` (Postgres) and `db-backup` use the stock `postgres:17` image, so only the two custom -images are built here. See `docker/lyra/README.md` in the ops repo for first-time setup. +- The **monitor** and the **discovery sweep** are both **off by default** — set + `monitor.enabled=true` (or Settings → Monitor, + optional `monitor.*` tuning) to activate + auto-grab/upgrade, and `discover.enabled=true` (or Settings → Discovery, + optional + `discover.*` tuning) to activate the scheduled discovery sweep. The `/discover` "Discover + now" button and seed-search work with the sweep off. +- **Credentials persist** across `docker compose up -d --build` rebuilds — don't use + `down -v` (it wipes the Postgres volume); secrets are encrypted at rest with + `LYRA_SECRET_KEY`. +- **Tests** run against a separate `lyra_test` database (a guard blocks the live DB). ## Managing the library @@ -133,6 +97,24 @@ it — re-Want it from the artist page if you change your mind). This is why the also bind-mounts `MUSIC_DIR` at `/music` (read-write) — deletion is guarded to only ever remove folders strictly inside the library root. +## Security + +Lyra has **no per-user accounts**. Two layers protect it: + +- **Shared-password gate.** Set `LYRA_PASSWORD` in `.env` to require a single shared + password for the entire UI and API. A request without a valid session cookie is + redirected to `/login` (or gets `401` for `/api/*`). The cookie is httpOnly and carries + an HMAC of the password keyed by `LYRA_SECRET_KEY`, so it can't be forged. **If + `LYRA_PASSWORD` is unset, the app is completely open** — anyone who can reach the host + can browse the library, trigger downloads, and read/overwrite the stored credentials in + Settings. Set it for any deployment that isn't on a fully trusted, isolated network. + +- **Network.** The shared password is deliberately minimal (single password, bearer + cookie, plain HTTP on the LAN). For anything internet-facing, also put Lyra behind a + reverse proxy with TLS and/or a VPN or auth gateway — don't rely on the shared password + alone. See [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#running-behind-a-vpn-gluetun) for a + Gluetun VPN setup. + ## Backup & restore **All** durable state — library metadata, monitored/wanted lists, discovery data, @@ -163,105 +145,8 @@ docker compose start web worker is safe. To restore onto a brand-new host, bring up just `db` first (`docker compose up -d db`), restore, then `up -d --build` the rest. -## Security +## More -Lyra has **no per-user accounts**. Two layers protect it: - -- **Shared-password gate.** Set `LYRA_PASSWORD` in `.env` to require a single shared - password for the entire UI and API. A request without a valid session cookie is - redirected to `/login` (or gets `401` for `/api/*`). The cookie is httpOnly and carries - an HMAC of the password keyed by `LYRA_SECRET_KEY`, so it can't be forged. **If - `LYRA_PASSWORD` is unset, the app is completely open** — anyone who can reach the host - can browse the library, trigger downloads, and read/overwrite the stored credentials in - Settings. Set it for any deployment that isn't on a fully trusted, isolated network. - -- **Network.** The shared password is deliberately minimal (single password, bearer - cookie, plain HTTP on the LAN). For anything internet-facing, also put Lyra behind a - reverse proxy with TLS and/or a VPN or auth gateway — don't rely on the shared password - alone. - -## Running behind a VPN (Gluetun) - -Lyra runs fine with its traffic routed through a [Gluetun](https://github.com/qdm12/gluetun) -VPN container — useful for hiding the download traffic (Soulseek / Qobuz / YouTube). Nothing -in Lyra is VPN-incompatible; the only thing that changes is container networking, because a -service using `network_mode: "service:gluetun"` gives up its own network stack and shares -Gluetun's. Two consequences: - -- Containers that share Gluetun's namespace reach each other over **`localhost`**, not by - Docker service name — so `DATABASE_URL` changes from `@db:5432` to **`@localhost:5432`**. -- **Published ports move to the `gluetun` container** (the individual services can no longer - publish their own). - -The simplest, most consistent setup is to put the **whole stack** behind Gluetun. Sketch -(merge into your existing Gluetun compose; adjust the VPN provider block to yours): - -```yaml -services: - gluetun: - image: qmcgaw/gluetun - cap_add: [NET_ADMIN] - environment: - # ... your VPN provider / credentials ... - FIREWALL_INPUT_PORTS: "3000" # let your browser reach the web UI - FIREWALL_OUTBOUND_SUBNETS: "192.168.0.0/16" # your LAN — so the worker can reach slskd - ports: - - "8770:3000" # Lyra web (published on gluetun, not on `web`) - # - "5432:5432" # only if you want DB access from the LAN - - web: - build: ./web - network_mode: "service:gluetun" # no `ports:` / `networks:` here - environment: - DATABASE_URL: postgresql://lyra:lyra@localhost:5432/lyra # `db` -> `localhost` - LYRA_SECRET_KEY: ${LYRA_SECRET_KEY} - LYRA_PASSWORD: ${LYRA_PASSWORD:-} - HOSTNAME: 0.0.0.0 - LYRA_LIBRARY_ROOT: /music - volumes: - - ${MUSIC_DIR:-./music}:/music - depends_on: - gluetun: { condition: service_healthy } - db: { condition: service_healthy } - - worker: - build: ./worker - network_mode: "service:gluetun" - environment: - DATABASE_URL: postgresql://lyra:lyra@localhost:5432/lyra - LYRA_SECRET_KEY: ${LYRA_SECRET_KEY} - STAGING_ROOT: /staging - SLSKD_DOWNLOADS_ROOT: /slskd-downloads - volumes: - - ${MUSIC_DIR:-./music}:/music - - ${STAGING_DIR:-${MUSIC_DIR:-./music}/.staging}:/staging - - ${SLSKD_DOWNLOADS_DIR:-./slskd-downloads}:/slskd-downloads - depends_on: - gluetun: { condition: service_healthy } - db: { condition: service_healthy } - - db: - image: postgres:17 - network_mode: "service:gluetun" # shares the namespace so `localhost:5432` resolves - environment: - POSTGRES_USER: ${POSTGRES_USER} - POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} - POSTGRES_DB: ${POSTGRES_DB} - volumes: - - postgres-data:/var/lib/postgresql/data -``` - -(Give `db-backup` the same `network_mode` and `PGHOST: localhost`.) The `HOSTNAME=0.0.0.0` -and the built-in healthchecks keep working, since everything is in one namespace. - -Caveats worth knowing: - -- **Qobuz is geo-sensitive** — its catalog/availability varies by country, so pick a VPN - exit in a region where your Qobuz account works, or some downloads will fail to match. -- The worker's `net.ipv6.conf.all.disable_ipv6` sysctl **can't apply behind Gluetun** (a - shared netns can't set sysctls) — harmless, because Gluetun blocks IPv6 anyway, which is - exactly what that sysctl worked around (Qobuz's CDN resolving to an unroutable IPv6). -- MusicBrainz rate-limits per IP; a shared VPN exit is marginally more likely to hit the - 1 req/s ceiling, but the built-in response cache makes that a non-issue in normal use. -- Bind mounts (`/music`, `/staging`, `/slskd-downloads`), the Postgres volume, and backups - are filesystem, not network — **completely unaffected** by the VPN. +- **[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md)** — full feature tour, publishing pre-built + images to a registry, and running the whole stack behind a Gluetun VPN. +- **[docs/superpowers/specs/](docs/superpowers/specs/)** — the full design specs. diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md new file mode 100644 index 0000000..25c4fcc --- /dev/null +++ b/docs/DEPLOYMENT.md @@ -0,0 +1,177 @@ +# Lyra — deployment & advanced guide + +Topics beyond the [README](../README.md) quick-start: the full feature tour, publishing +pre-built images to a registry, and running the whole stack behind a VPN. For a basic +self-hosted setup you only need the README — start there. + +- [Features in depth](#features-in-depth) +- [Publishing pre-built images](#publishing-pre-built-images) +- [Running behind a VPN (Gluetun)](#running-behind-a-vpn-gluetun) + +--- + +## Features in depth + +Lyra was built in three slices, all complete: + +1. **Acquisition engine.** Resolve a request against MusicBrainz, match/rank across + Qobuz/Soulseek/YouTube, download the best source into an isolated staging dir, verify + completeness, tag (mutagen), and atomically promote the clean album into + `Artist/Album (Year)/## Title.ext`. +2. **Library manager.** Watched artists (live MB search + follow), per-release monitor + toggles + auto-monitor-future, a wanted list with retry + quality-upgrade window, and a + worker "monitor" that discovers new releases and auto-grabs through the acquisition + pipeline. Plus a **library scan** that ingests existing on-disk albums as + have + monitored + followed, and a per-artist studio-only / show-all release-type filter. +3. **Discovery.** Recommendation-driven finding of new music via **ListenBrainz** + similar-artists and **Last.fm** `artist.getSimilar` (both MBID-native), behind a + pluggable `SimilaritySource` interface — Last.fm registers as a second source + automatically once its API key is set, and cross-source scores are summed per candidate. + A background sweep (and a "Discover now" button) aggregates artists similar to the ones + you follow into a manual review feed of suggested **artists** and **albums** at + `/discover`; **Follow** and **Want** reuse the slice-2 follow / wanted flows, **Dismiss** + is permanent. Plus a web-side seed-search box. Every suggestion and seed-search result + opens a read-only **artist preview** (`/discover/artist/[mbid]`) — discography, per-album + tracklists, and "in library" / "monitored" badges — where you can Follow or Want before + committing. No Spotify (its recommendation APIs are dead for new apps). + +Alongside the three slices: + +- **Last.fm browse** (`/lastfm`) — your Last.fm top artists and albums (by period), with a + "hide items already in my library" toggle and inline **Follow** / **Want** buttons on each + row; each artist also opens a modal of your most-played songs and albums. +- **Library** (`/library`) — a cover-art grid of everything on disk, filterable, each album + opening a tracklist modal. +- **The Floor** (home) — a live queue of in-flight acquisitions with three-phase progress + (**Searching → Downloading → Finishing**), a live download percentage, a per-job "pressing + ticket" (source · format · tracks · elapsed), and a **Retry** button on anything that needs + attention. + +The whole UI runs a hand-rolled "Pressing Plant" design system (warm-paper theme, self-hosted +Fraunces serif for prose vs. system mono for machine data, light/dark aware). + +Deferred hardening and the full roadmap live in the implementation-plan "Notes" sections under +[`docs/superpowers/plans/`](superpowers/plans/); the full design specs are under +[`docs/superpowers/specs/`](superpowers/specs/). + +--- + +## Publishing pre-built images + +> **Maintainer release process** — this is how the project maintainer publishes images to a +> private registry. If you're self-hosting your own copy, ignore this and just build from +> source with `docker compose up -d --build` (see the README). You'd only need this if you +> run your own registry and want servers to **pull** rather than build. + +The `web` and `worker` images can be published to a container registry and consumed by a +compose file that pulls instead of builds. The maintainer publishes to a Gitea container +registry consumed by a compose file in the +[`vm-download`](https://git.jger.nl/Jonathan/vm-download) ops repo (`docker/lyra/`). + +Publish a new release from this repo (amd64): + +```sh +docker login # once — registry user + a token with package:write +./scripts/build-push.sh # builds + pushes lyra-web / lyra-worker (:latest and :) +``` + +`scripts/build-push.sh` defaults the registry namespace to `git.jger.nl/jonathan`; override it +with `LYRA_REGISTRY=/`. + +Then on the server: + +```sh +cd /opt/git/vm-download && git pull +cd docker/lyra && docker compose pull && docker compose up -d +``` + +`db` (Postgres) and `db-backup` use the stock `postgres:17` image, so only the two custom +images are built here. + +--- + +## Running behind a VPN (Gluetun) + +Lyra runs fine with its traffic routed through a [Gluetun](https://github.com/qdm12/gluetun) +VPN container — useful for hiding the download traffic (Soulseek / Qobuz / YouTube). Nothing +in Lyra is VPN-incompatible; the only thing that changes is container networking, because a +service using `network_mode: "service:gluetun"` gives up its own network stack and shares +Gluetun's. Two consequences: + +- Containers that share Gluetun's namespace reach each other over **`localhost`**, not by + Docker service name — so `DATABASE_URL` changes from `@db:5432` to **`@localhost:5432`**. +- **Published ports move to the `gluetun` container** (the individual services can no longer + publish their own). + +The simplest, most consistent setup is to put the **whole stack** behind Gluetun. Sketch +(merge into your existing Gluetun compose; adjust the VPN provider block to yours): + +```yaml +services: + gluetun: + image: qmcgaw/gluetun + cap_add: [NET_ADMIN] + environment: + # ... your VPN provider / credentials ... + FIREWALL_INPUT_PORTS: "3000" # let your browser reach the web UI + FIREWALL_OUTBOUND_SUBNETS: "192.168.0.0/16" # your LAN — so the worker can reach slskd + ports: + - "8770:3000" # Lyra web (published on gluetun, not on `web`) + # - "5432:5432" # only if you want DB access from the LAN + + web: + build: ./web + network_mode: "service:gluetun" # no `ports:` / `networks:` here + environment: + DATABASE_URL: postgresql://lyra:lyra@localhost:5432/lyra # `db` -> `localhost` + LYRA_SECRET_KEY: ${LYRA_SECRET_KEY} + LYRA_PASSWORD: ${LYRA_PASSWORD:-} + HOSTNAME: 0.0.0.0 + LYRA_LIBRARY_ROOT: /music + volumes: + - ${MUSIC_DIR:-./music}:/music + depends_on: + gluetun: { condition: service_healthy } + db: { condition: service_healthy } + + worker: + build: ./worker + network_mode: "service:gluetun" + environment: + DATABASE_URL: postgresql://lyra:lyra@localhost:5432/lyra + LYRA_SECRET_KEY: ${LYRA_SECRET_KEY} + STAGING_ROOT: /staging + SLSKD_DOWNLOADS_ROOT: /slskd-downloads + volumes: + - ${MUSIC_DIR:-./music}:/music + - ${STAGING_DIR:-${MUSIC_DIR:-./music}/.staging}:/staging + - ${SLSKD_DOWNLOADS_DIR:-./slskd-downloads}:/slskd-downloads + depends_on: + gluetun: { condition: service_healthy } + db: { condition: service_healthy } + + db: + image: postgres:17 + network_mode: "service:gluetun" # shares the namespace so `localhost:5432` resolves + environment: + POSTGRES_USER: ${POSTGRES_USER} + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} + POSTGRES_DB: ${POSTGRES_DB} + volumes: + - postgres-data:/var/lib/postgresql/data +``` + +(Give `db-backup` the same `network_mode` and `PGHOST: localhost`.) The `HOSTNAME=0.0.0.0` +and the built-in healthchecks keep working, since everything is in one namespace. + +Caveats worth knowing: + +- **Qobuz is geo-sensitive** — its catalog/availability varies by country, so pick a VPN + exit in a region where your Qobuz account works, or some downloads will fail to match. +- The worker's `net.ipv6.conf.all.disable_ipv6` sysctl **can't apply behind Gluetun** (a + shared netns can't set sysctls) — harmless, because Gluetun blocks IPv6 anyway, which is + exactly what that sysctl worked around (Qobuz's CDN resolving to an unroutable IPv6). +- MusicBrainz rate-limits per IP; a shared VPN exit is marginally more likely to hit the + 1 req/s ceiling, but the built-in response cache makes that a non-issue in normal use. +- Bind mounts (`/music`, `/staging`, `/slskd-downloads`), the Postgres volume, and backups + are filesystem, not network — **completely unaffected** by the VPN.