docs: add library-scan implementation plan
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jonathan
@@ -0,0 +1,938 @@
|
||||
# Lyra Library Scan Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** A manual "Scan library" action that walks `/music`, matches each album to MusicBrainz, and records it as **have** (`LibraryItem`) + **on monitor** (`MonitoredRelease`, fulfilled) with the artist **followed** (`WatchedArtist`) — so the library manager reflects what's already on disk.
|
||||
|
||||
**Architecture:** The scan runs in the worker (owns `/music`, the MB resolver, mutagen), triggered via a `Config` flag the web sets. A new `scan.py` walks the library, probes each album's tags/quality via an injectable `AudioProbe` seam, resolves it against MusicBrainz (extended to return `rgMbid`/`artistMbid`), and idempotently upserts the rows. `LibraryItem.requestId` becomes nullable so scanned "have" records need no originating request. The web adds a Settings button + `POST/GET /api/scan`.
|
||||
|
||||
**Tech Stack:** Python 3.12 worker (psycopg, mutagen lazy), Prisma/Postgres, Next.js 15 web. Tests: pytest with a fake `AudioProbe` + fake resolver + temp `/music` tree + seeded `lyra_test`; vitest for the web routes. Zero real MusicBrainz/downloads offline.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **Tests MUST target `lyra_test`** (a guard in conftest/setup refuses non-`*_test` DBs). Worker: `DATABASE_URL=postgresql://lyra:lyra@localhost:5432/lyra_test .venv/bin/python -m pytest` (or leave `DATABASE_URL` unset — conftest defaults to `lyra_test`). Web: `npm test` (already points at `lyra_test`). Run worker cmds from `/home/jonathan/Projects/lyra/worker` using `.venv/bin/python`; web cmds from `/home/jonathan/Projects/lyra/web`.
|
||||
- Worker: psycopg raw SQL against Prisma tables (quoted PascalCase, camelCase columns, `gen_random_uuid()::text`, SQL-side `now()`), enum literals as plain strings, `text[]` as Python lists.
|
||||
- The scan is READ-ONLY on `/music` — no downloads, tagging, moves, or file writes. mutagen is imported lazily inside the real probe (construction stays offline).
|
||||
- Idempotent: re-scanning updates-or-leaves rows, never duplicates, and never flips a real download's `LibraryItem.source` to `"scan"`.
|
||||
- Backward compatibility: the nullable-`requestId` migration is additive; every existing worker + web test stays green. `MBTarget` gains defaulted fields (existing constructions unaffected).
|
||||
- Every task ends with a commit.
|
||||
|
||||
## Shared interfaces (defined by tasks below)
|
||||
|
||||
```python
|
||||
# worker/lyra_worker/types.py (Task 1): MBTarget gains rg_mbid: str = "", artist_mbid: str = ""
|
||||
# worker/lyra_worker/_musicbrainz.py (Task 1): resolver fills rg_mbid + artist_mbid
|
||||
|
||||
# worker/lyra_worker/probe.py (Task 3)
|
||||
@dataclass(frozen=True)
|
||||
class ProbeResult: artist: str; album: str; fmt: str; quality_class: int
|
||||
class AudioProbe(Protocol):
|
||||
def probe(self, path: str) -> ProbeResult: ...
|
||||
|
||||
# worker/lyra_worker/scan.py (Task 3)
|
||||
@dataclass(frozen=True)
|
||||
class ScanResult: imported: int; skipped: int
|
||||
def scan_library(conn, resolver, probe: AudioProbe, dest_root: str = "/music") -> ScanResult: ...
|
||||
|
||||
# worker/lyra_worker/registry.py (Task 3): build_probe() -> AudioProbe
|
||||
# worker/lyra_worker/main.py (Task 4): loop runs scan_library when Config "scan.requested" is truthy
|
||||
# web: POST /api/scan (set scan.requested), GET /api/scan (read scan.result) (Task 5); Settings button (Task 6)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 1: MBTarget + resolver expose `rgMbid` / `artistMbid`
|
||||
|
||||
**Files:**
|
||||
- Modify: `worker/lyra_worker/types.py`
|
||||
- Modify: `worker/lyra_worker/_musicbrainz.py`
|
||||
- Test: `worker/tests/test_types_mbtarget.py`
|
||||
- Test: `worker/tests/test_musicbrainz_live.py` (extend the opt-in live assertion)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `MBTarget.rg_mbid: str = ""`, `MBTarget.artist_mbid: str = ""`; `MusicBrainzResolver.resolve` fills both from the matched release-group and its artist-credit.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
`worker/tests/test_types_mbtarget.py`:
|
||||
```python
|
||||
from lyra_worker.types import MBTarget
|
||||
|
||||
|
||||
def test_mbtarget_has_rg_and_artist_mbid_defaults():
|
||||
t = MBTarget(artist="A", album="B")
|
||||
assert t.rg_mbid == "" and t.artist_mbid == ""
|
||||
t2 = MBTarget(artist="A", album="B", rg_mbid="rg1", artist_mbid="a1")
|
||||
assert t2.rg_mbid == "rg1" and t2.artist_mbid == "a1"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run it to verify it fails**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/worker
|
||||
.venv/bin/python -m pytest tests/test_types_mbtarget.py -v
|
||||
```
|
||||
Expected: FAIL — `MBTarget` has no `rg_mbid`.
|
||||
|
||||
- [ ] **Step 3: Add the fields**
|
||||
|
||||
In `worker/lyra_worker/types.py`, add two fields to `MBTarget` (after `tracklist`):
|
||||
```python
|
||||
tracklist: tuple[str, ...] = ()
|
||||
rg_mbid: str = ""
|
||||
artist_mbid: str = ""
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Fill them in the resolver**
|
||||
|
||||
In `worker/lyra_worker/_musicbrainz.py`, in `resolve`, capture the artist MBID next to the name, and the release-group id, and pass both to `MBTarget(...)`.
|
||||
|
||||
Where `canonical_artist` is computed from the credit, also capture the id:
|
||||
```python
|
||||
canonical_artist = artist
|
||||
artist_mbid = ""
|
||||
credit = rg.get("artist-credit")
|
||||
if isinstance(credit, list) and credit and isinstance(credit[0], dict):
|
||||
_artist = credit[0].get("artist") or {}
|
||||
canonical_artist = _artist.get("name", artist)
|
||||
artist_mbid = _artist.get("id", "") or ""
|
||||
```
|
||||
And in the returned `MBTarget(...)`, add:
|
||||
```python
|
||||
return MBTarget(
|
||||
artist=canonical_artist,
|
||||
album=canonical_album,
|
||||
track_count=track_count,
|
||||
total_duration_s=total_duration_s,
|
||||
year=year,
|
||||
tracklist=titles,
|
||||
rg_mbid=rg.get("id", "") or "",
|
||||
artist_mbid=artist_mbid,
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Extend the opt-in live test**
|
||||
|
||||
In `worker/tests/test_musicbrainz_live.py`, in the existing live resolve test (the one gated by `LYRA_LIVE_TESTS`), add after the target is resolved:
|
||||
```python
|
||||
assert target.rg_mbid # a real release-group MBID
|
||||
assert target.artist_mbid # a real artist MBID
|
||||
```
|
||||
(If the test's variable is not named `target`, use whatever it binds the resolved `MBTarget` to.)
|
||||
|
||||
- [ ] **Step 6: Run the offline tests + full suite**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/worker
|
||||
.venv/bin/python -m pytest tests/test_types_mbtarget.py -v
|
||||
.venv/bin/python -m pytest -q
|
||||
```
|
||||
Expected: the new test passes; full suite green (live test still skipped); output pristine.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra
|
||||
git add worker/lyra_worker/types.py worker/lyra_worker/_musicbrainz.py worker/tests/test_types_mbtarget.py worker/tests/test_musicbrainz_live.py
|
||||
git commit -m "feat: MBTarget exposes rgMbid + artistMbid from the resolver"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: `LibraryItem.requestId` nullable + test cleanup
|
||||
|
||||
**Files:**
|
||||
- Modify: `web/prisma/schema.prisma`
|
||||
- Create: (generated) migration under `web/prisma/migrations/`
|
||||
- Modify: `worker/tests/conftest.py` (clean `LibraryItem`)
|
||||
- Modify: `web/src/test/setup.ts` (clean `libraryItem`)
|
||||
- Test: `worker/tests/test_library_item_optional_request.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `LibraryItem.requestId` is optional; a `LibraryItem` can be inserted with `requestId = NULL`. Test fixtures clean `LibraryItem` (scanned rows have no `Request` to cascade from).
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
`worker/tests/test_library_item_optional_request.py`:
|
||||
```python
|
||||
def test_library_item_can_have_null_request(conn):
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
'INSERT INTO "LibraryItem" (id, "requestId", artist, album, path, source, format, '
|
||||
'"qualityClass", "importedAt") '
|
||||
"VALUES (gen_random_uuid()::text, NULL, 'A', 'B', '/m/A/B', 'scan', 'FLAC', 2, now()) "
|
||||
"RETURNING id"
|
||||
)
|
||||
lib_id = cur.fetchone()[0]
|
||||
conn.commit()
|
||||
with conn.cursor() as cur:
|
||||
cur.execute('SELECT "requestId" FROM "LibraryItem" WHERE id = %s', (lib_id,))
|
||||
assert cur.fetchone()[0] is None
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run it to verify it fails**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/worker
|
||||
.venv/bin/python -m pytest tests/test_library_item_optional_request.py -v
|
||||
```
|
||||
Expected: FAIL — `null value in column "requestId" violates not-null constraint`.
|
||||
|
||||
- [ ] **Step 3: Make the relation optional**
|
||||
|
||||
In `web/prisma/schema.prisma`, `model LibraryItem`, change the request field + fk to optional:
|
||||
```prisma
|
||||
request Request? @relation(fields: [requestId], references: [id], onDelete: Cascade)
|
||||
requestId String? @unique
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Generate + apply the migration (into lyra_test)**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/web
|
||||
DATABASE_URL=postgresql://lyra:lyra@localhost:5432/lyra_test npx prisma migrate dev --name library_item_optional_request
|
||||
```
|
||||
Expected: a migration with `ALTER TABLE "LibraryItem" ALTER COLUMN "requestId" DROP NOT NULL;` is created and applied to `lyra_test`. (The live `lyra` DB gets it via `migrate deploy` on the next `docker compose up --build`.)
|
||||
|
||||
- [ ] **Step 5: Clean `LibraryItem` in the test fixtures**
|
||||
|
||||
Scanned `LibraryItem`s have `requestId = NULL`, so they no longer cascade-delete when `Request` rows are cleared. Add explicit cleanup.
|
||||
|
||||
In `worker/tests/conftest.py`, in BOTH cleanup blocks of the `conn` fixture, add a `LibraryItem` delete after `Request`:
|
||||
```python
|
||||
cur.execute('DELETE FROM "Job"')
|
||||
cur.execute('DELETE FROM "Request"')
|
||||
cur.execute('DELETE FROM "LibraryItem"')
|
||||
cur.execute('DELETE FROM "MonitoredRelease"')
|
||||
cur.execute('DELETE FROM "WatchedArtist"')
|
||||
cur.execute('DELETE FROM "Config"')
|
||||
```
|
||||
|
||||
In `web/src/test/setup.ts`, add `libraryItem` to the `beforeEach` cleanup (after `request`):
|
||||
```ts
|
||||
await prisma.job.deleteMany();
|
||||
await prisma.request.deleteMany();
|
||||
await prisma.libraryItem.deleteMany();
|
||||
await prisma.monitoredRelease.deleteMany();
|
||||
await prisma.watchedArtist.deleteMany();
|
||||
await prisma.config.deleteMany();
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the test + full worker suite + web suite**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/worker
|
||||
.venv/bin/python -m pytest tests/test_library_item_optional_request.py -v
|
||||
.venv/bin/python -m pytest -q
|
||||
cd /home/jonathan/Projects/lyra/web
|
||||
npm test --silent 2>&1 | tail -4
|
||||
```
|
||||
Expected: new test passes; full worker suite green; web suite green (the `libraryItem` cleanup is harmless to existing tests).
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra
|
||||
git add web/prisma/schema.prisma web/prisma/migrations worker/tests/conftest.py web/src/test/setup.ts worker/tests/test_library_item_optional_request.py
|
||||
git commit -m "feat: LibraryItem.requestId nullable (scanned haves need no request) + test cleanup"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: The scan — `AudioProbe` seam + `scan_library`
|
||||
|
||||
**Files:**
|
||||
- Create: `worker/lyra_worker/probe.py` (seam)
|
||||
- Create: `worker/lyra_worker/_probe.py` (real `MutagenProbe`)
|
||||
- Create: `worker/lyra_worker/scan.py`
|
||||
- Modify: `worker/lyra_worker/registry.py` (`build_probe`)
|
||||
- Modify: `worker/lyra_worker/adapters/fakes.py` (add `FakeAudioProbe`)
|
||||
- Test: `worker/tests/test_scan.py`
|
||||
- Test: `worker/tests/test_probe_live.py` (opt-in)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `MBTarget` (with `rg_mbid`/`artist_mbid`), an `MbResolver`, `AudioProbe`.
|
||||
- Produces: `ProbeResult`, `AudioProbe` (Protocol), `MutagenProbe`, `FakeAudioProbe`, `ScanResult`, `scan_library(conn, resolver, probe, dest_root)`, `registry.build_probe()`.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
`worker/tests/test_scan.py`:
|
||||
```python
|
||||
from lyra_worker.probe import ProbeResult
|
||||
from lyra_worker.scan import scan_library
|
||||
from lyra_worker.types import MBTarget
|
||||
|
||||
|
||||
class FakeProbe:
|
||||
"""Returns a canned ProbeResult regardless of path (tests control identification via folders)."""
|
||||
def __init__(self, artist="", album="", fmt="FLAC", quality_class=3):
|
||||
self._r = ProbeResult(artist=artist, album=album, fmt=fmt, quality_class=quality_class)
|
||||
def probe(self, path):
|
||||
return self._r
|
||||
|
||||
|
||||
class FakeResolver:
|
||||
"""resolve(artist, album) -> MBTarget or None, keyed by (artist, album)."""
|
||||
def __init__(self, table):
|
||||
self._t = dict(table)
|
||||
def resolve(self, artist, album):
|
||||
return self._t.get((artist, album))
|
||||
|
||||
|
||||
def _album(tmp_path, artist, album_folder, files=("01 Track.flac",)):
|
||||
d = tmp_path / artist / album_folder
|
||||
d.mkdir(parents=True)
|
||||
for f in files:
|
||||
(d / f).write_bytes(b"")
|
||||
return d
|
||||
|
||||
|
||||
def _counts(conn):
|
||||
with conn.cursor() as cur:
|
||||
out = {}
|
||||
for t in ("LibraryItem", "MonitoredRelease", "WatchedArtist"):
|
||||
cur.execute(f'SELECT count(*) FROM "{t}"')
|
||||
out[t] = cur.fetchone()[0]
|
||||
return out
|
||||
|
||||
|
||||
def test_scan_records_have_monitored_and_follows_artist(conn, tmp_path):
|
||||
_album(tmp_path, "John Mayer", "Continuum (2006)")
|
||||
resolver = FakeResolver({
|
||||
("John Mayer", "Continuum"): MBTarget(
|
||||
artist="John Mayer", album="Continuum", year=2006, rg_mbid="rg1", artist_mbid="a1"),
|
||||
})
|
||||
result = scan_library(conn, resolver, FakeProbe(quality_class=3), dest_root=str(tmp_path))
|
||||
|
||||
assert (result.imported, result.skipped) == (1, 0)
|
||||
assert _counts(conn) == {"LibraryItem": 1, "MonitoredRelease": 1, "WatchedArtist": 1}
|
||||
with conn.cursor() as cur:
|
||||
cur.execute('SELECT source, "qualityClass" FROM "LibraryItem" WHERE artist=%s AND album=%s',
|
||||
("John Mayer", "Continuum"))
|
||||
assert cur.fetchone() == ("scan", 3)
|
||||
cur.execute('SELECT monitored, state, "currentQualityClass" FROM "MonitoredRelease" WHERE "rgMbid"=%s',
|
||||
("rg1",))
|
||||
assert cur.fetchone() == (True, "fulfilled", 3)
|
||||
cur.execute('SELECT "autoMonitorFuture" FROM "WatchedArtist" WHERE mbid=%s', ("a1",))
|
||||
assert cur.fetchone()[0] is False
|
||||
|
||||
|
||||
def test_scan_skips_unmatched_albums(conn, tmp_path):
|
||||
_album(tmp_path, "Obscure", "Nope (1999)")
|
||||
result = scan_library(conn, FakeResolver({}), FakeProbe(), dest_root=str(tmp_path))
|
||||
assert (result.imported, result.skipped) == (0, 1)
|
||||
assert _counts(conn) == {"LibraryItem": 0, "MonitoredRelease": 0, "WatchedArtist": 0}
|
||||
|
||||
|
||||
def test_scan_falls_back_to_tags_when_folder_does_not_resolve(conn, tmp_path):
|
||||
_album(tmp_path, "GARBLE", "CD1") # folder names won't resolve
|
||||
resolver = FakeResolver({
|
||||
("Radiohead", "In Rainbows"): MBTarget(
|
||||
artist="Radiohead", album="In Rainbows", year=2007, rg_mbid="rg9", artist_mbid="a9"),
|
||||
})
|
||||
probe = FakeProbe(artist="Radiohead", album="In Rainbows", quality_class=3) # embedded tags
|
||||
result = scan_library(conn, resolver, probe, dest_root=str(tmp_path))
|
||||
assert (result.imported, result.skipped) == (1, 0)
|
||||
with conn.cursor() as cur:
|
||||
cur.execute('SELECT count(*) FROM "MonitoredRelease" WHERE "rgMbid"=%s', ("rg9",))
|
||||
assert cur.fetchone()[0] == 1
|
||||
|
||||
|
||||
def test_scan_skips_folders_without_audio(conn, tmp_path):
|
||||
d = tmp_path / "Artist" / "Album"
|
||||
d.mkdir(parents=True)
|
||||
(d / "cover.jpg").write_bytes(b"") # no audio
|
||||
result = scan_library(conn, FakeResolver({}), FakeProbe(), dest_root=str(tmp_path))
|
||||
assert (result.imported, result.skipped) == (0, 0)
|
||||
|
||||
|
||||
def test_scan_is_idempotent(conn, tmp_path):
|
||||
_album(tmp_path, "John Mayer", "Continuum (2006)")
|
||||
resolver = FakeResolver({
|
||||
("John Mayer", "Continuum"): MBTarget(
|
||||
artist="John Mayer", album="Continuum", year=2006, rg_mbid="rg1", artist_mbid="a1"),
|
||||
})
|
||||
scan_library(conn, resolver, FakeProbe(quality_class=3), dest_root=str(tmp_path))
|
||||
second = scan_library(conn, resolver, FakeProbe(quality_class=3), dest_root=str(tmp_path))
|
||||
assert second.imported == 0 # already recorded
|
||||
assert _counts(conn) == {"LibraryItem": 1, "MonitoredRelease": 1, "WatchedArtist": 1}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run it to verify it fails**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/worker
|
||||
.venv/bin/python -m pytest tests/test_scan.py -v
|
||||
```
|
||||
Expected: FAIL — `No module named 'lyra_worker.scan'`.
|
||||
|
||||
- [ ] **Step 3: Add the probe seam**
|
||||
|
||||
`worker/lyra_worker/probe.py`:
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
from typing import Protocol
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ProbeResult:
|
||||
artist: str
|
||||
album: str
|
||||
fmt: str
|
||||
quality_class: int
|
||||
|
||||
|
||||
class AudioProbe(Protocol):
|
||||
def probe(self, path: str) -> ProbeResult:
|
||||
"""Read an audio file's artist/album tags + format/quality."""
|
||||
...
|
||||
```
|
||||
|
||||
`worker/lyra_worker/_probe.py`:
|
||||
```python
|
||||
import os
|
||||
|
||||
from lyra_worker.probe import ProbeResult
|
||||
from lyra_worker.quality import quality_class
|
||||
from lyra_worker.types import Quality
|
||||
|
||||
_LOSSLESS_EXT = {".flac", ".wav", ".alac", ".aiff", ".aif"}
|
||||
|
||||
|
||||
class MutagenProbe:
|
||||
"""Real AudioProbe via mutagen (imported lazily; not unit-tested offline)."""
|
||||
|
||||
def probe(self, path: str) -> ProbeResult:
|
||||
import mutagen
|
||||
|
||||
ext = os.path.splitext(path)[1].lower()
|
||||
audio = mutagen.File(path, easy=True)
|
||||
artist = album = ""
|
||||
bit_depth = sample_rate = None
|
||||
if audio is not None:
|
||||
artist = (audio.get("artist") or [""])[0]
|
||||
album = (audio.get("album") or [""])[0]
|
||||
info = getattr(audio, "info", None)
|
||||
bit_depth = getattr(info, "bits_per_sample", None)
|
||||
sample_rate = getattr(info, "sample_rate", None)
|
||||
q = quality_class(Quality(
|
||||
fmt=ext.lstrip(".").upper(),
|
||||
lossless=ext in _LOSSLESS_EXT,
|
||||
bit_depth=bit_depth,
|
||||
sample_rate=sample_rate,
|
||||
))
|
||||
return ProbeResult(artist=artist, album=album, fmt=ext.lstrip(".").upper(), quality_class=q)
|
||||
```
|
||||
|
||||
Append to `worker/lyra_worker/adapters/fakes.py`:
|
||||
```python
|
||||
from lyra_worker.probe import ProbeResult
|
||||
|
||||
|
||||
class FakeAudioProbe:
|
||||
"""In-memory AudioProbe for tests."""
|
||||
def __init__(self, artist="", album="", fmt="FLAC", quality_class=3):
|
||||
self._r = ProbeResult(artist=artist, album=album, fmt=fmt, quality_class=quality_class)
|
||||
def probe(self, path: str) -> ProbeResult:
|
||||
return self._r
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Implement `scan_library`**
|
||||
|
||||
`worker/lyra_worker/scan.py`:
|
||||
```python
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
import psycopg
|
||||
|
||||
from lyra_worker.probe import AudioProbe
|
||||
|
||||
_AUDIO_EXT = {".flac", ".mp3", ".m4a", ".opus", ".ogg", ".aac", ".wav"}
|
||||
_YEAR = re.compile(r"^(.*) \((\d{4})\)$")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ScanResult:
|
||||
imported: int
|
||||
skipped: int
|
||||
|
||||
|
||||
def _parse_album_dir(name: str) -> str:
|
||||
m = _YEAR.match(name)
|
||||
return m.group(1) if m else name
|
||||
|
||||
|
||||
def _first_audio(album_path: str) -> str | None:
|
||||
for name in sorted(os.listdir(album_path)):
|
||||
if os.path.splitext(name)[1].lower() in _AUDIO_EXT:
|
||||
return os.path.join(album_path, name)
|
||||
return None
|
||||
|
||||
|
||||
def _upsert(conn, target, quality_class: int, path: str) -> bool:
|
||||
"""Record have + monitored + followed for a resolved album. Returns True if newly imported."""
|
||||
with conn.cursor() as cur:
|
||||
watched_id = None
|
||||
if target.artist_mbid:
|
||||
cur.execute(
|
||||
'INSERT INTO "WatchedArtist" (id, mbid, name, "autoMonitorFuture", "monitorFrom", "createdAt") '
|
||||
"VALUES (gen_random_uuid()::text, %s, %s, false, now(), now()) "
|
||||
'ON CONFLICT (mbid) DO NOTHING',
|
||||
(target.artist_mbid, target.artist),
|
||||
)
|
||||
cur.execute('SELECT id FROM "WatchedArtist" WHERE mbid = %s', (target.artist_mbid,))
|
||||
row = cur.fetchone()
|
||||
watched_id = row[0] if row else None
|
||||
|
||||
if target.rg_mbid:
|
||||
cur.execute(
|
||||
'INSERT INTO "MonitoredRelease" (id, "watchedArtistId", "artistMbid", "artistName", '
|
||||
'"rgMbid", album, "secondaryTypes", monitored, state, "currentQualityClass", '
|
||||
'"firstGrabbedAt", "createdAt") '
|
||||
"VALUES (gen_random_uuid()::text, %s, %s, %s, %s, %s, '{}', true, 'fulfilled', %s, now(), now()) "
|
||||
'ON CONFLICT ("rgMbid") DO UPDATE SET monitored = true, state = \'fulfilled\', '
|
||||
' "currentQualityClass" = EXCLUDED."currentQualityClass", '
|
||||
' "firstGrabbedAt" = COALESCE("MonitoredRelease"."firstGrabbedAt", now()), '
|
||||
' "watchedArtistId" = COALESCE("MonitoredRelease"."watchedArtistId", EXCLUDED."watchedArtistId")',
|
||||
(watched_id, target.artist_mbid, target.artist, target.rg_mbid, target.album, quality_class),
|
||||
)
|
||||
|
||||
cur.execute(
|
||||
'INSERT INTO "LibraryItem" (id, "requestId", artist, album, path, source, format, '
|
||||
'"qualityClass", "importedAt") '
|
||||
"VALUES (gen_random_uuid()::text, NULL, %s, %s, %s, 'scan', %s, %s, now()) "
|
||||
'ON CONFLICT (artist, album) DO NOTHING',
|
||||
(target.artist, target.album, path, "FLAC", quality_class),
|
||||
)
|
||||
newly = cur.rowcount == 1 # a new LibraryItem means this album wasn't recorded before
|
||||
conn.commit()
|
||||
return newly
|
||||
|
||||
|
||||
def scan_library(conn: psycopg.Connection, resolver, probe: AudioProbe, dest_root: str = "/music") -> ScanResult:
|
||||
"""Walk `{dest_root}/{Artist}/{Album}/`, resolve each album against MusicBrainz, and record
|
||||
matched albums as have + monitored + followed. Unmatched or audio-less folders are skipped."""
|
||||
imported = 0
|
||||
skipped = 0
|
||||
if not os.path.isdir(dest_root):
|
||||
return ScanResult(0, 0)
|
||||
for artist_name in sorted(os.listdir(dest_root)):
|
||||
artist_path = os.path.join(dest_root, artist_name)
|
||||
if artist_name.startswith(".") or not os.path.isdir(artist_path):
|
||||
continue
|
||||
for album_folder in sorted(os.listdir(artist_path)):
|
||||
album_path = os.path.join(artist_path, album_folder)
|
||||
if not os.path.isdir(album_path):
|
||||
continue
|
||||
audio = _first_audio(album_path)
|
||||
if audio is None:
|
||||
continue # no audio → not an album
|
||||
album_name = _parse_album_dir(album_folder)
|
||||
target = resolver.resolve(artist_name, album_name) if artist_name and album_name else None
|
||||
if target is None: # fall back to the file's embedded tags
|
||||
pr = probe.probe(audio)
|
||||
if pr.artist and pr.album and (pr.artist, pr.album) != (artist_name, album_name):
|
||||
target = resolver.resolve(pr.artist, pr.album)
|
||||
if target is None:
|
||||
skipped += 1
|
||||
continue
|
||||
q = probe.probe(audio).quality_class
|
||||
if _upsert(conn, target, q, album_path):
|
||||
imported += 1
|
||||
return ScanResult(imported, skipped)
|
||||
```
|
||||
|
||||
Add to `worker/lyra_worker/registry.py`:
|
||||
```python
|
||||
from lyra_worker._probe import MutagenProbe
|
||||
from lyra_worker.probe import AudioProbe
|
||||
|
||||
|
||||
def build_probe() -> AudioProbe:
|
||||
"""The audio probe used by the library scan."""
|
||||
return MutagenProbe()
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run the scan tests to verify they pass**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/worker
|
||||
.venv/bin/python -m pytest tests/test_scan.py -v
|
||||
```
|
||||
Expected: PASS — all five tests green.
|
||||
|
||||
- [ ] **Step 6: Add the opt-in real-probe test**
|
||||
|
||||
`worker/tests/test_probe_live.py`:
|
||||
```python
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not (os.environ.get("LYRA_LIVE_TESTS") and os.environ.get("LYRA_SAMPLE_AUDIO")),
|
||||
reason="reads a real audio file; set LYRA_LIVE_TESTS=1 + LYRA_SAMPLE_AUDIO=/path/to/a.flac to run",
|
||||
)
|
||||
|
||||
|
||||
def test_probe_reads_a_real_file():
|
||||
from lyra_worker._probe import MutagenProbe
|
||||
|
||||
r = MutagenProbe().probe(os.environ["LYRA_SAMPLE_AUDIO"])
|
||||
assert r.quality_class in (1, 2, 3)
|
||||
assert r.fmt
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Verify lazy import + full suite**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/worker
|
||||
.venv/bin/python -c "import sys; from lyra_worker._probe import MutagenProbe; from lyra_worker.scan import scan_library; print('import ok; mutagen loaded:', 'mutagen' in sys.modules)"
|
||||
.venv/bin/python -m pytest -q
|
||||
```
|
||||
Expected: prints `import ok; mutagen loaded: False`; full suite green, `test_probe_live.py` skipped, output pristine.
|
||||
|
||||
- [ ] **Step 8: Commit**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra
|
||||
git add worker/lyra_worker/probe.py worker/lyra_worker/_probe.py worker/lyra_worker/scan.py worker/lyra_worker/registry.py worker/lyra_worker/adapters/fakes.py worker/tests/test_scan.py worker/tests/test_probe_live.py
|
||||
git commit -m "feat: library scan — probe seam + scan_library (have + monitored + followed)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Wire the scan trigger into the worker loop
|
||||
|
||||
**Files:**
|
||||
- Modify: `worker/lyra_worker/main.py`
|
||||
- Test: `worker/tests/test_scan_trigger.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `scan_library`, `build_probe`, the `Config` table.
|
||||
- Produces: `run_forever()` reads config once per loop; when `Config` `scan.requested` is truthy it runs `scan_library`, writes a `scan.result` summary, and clears `scan.requested`. A `_run_scan(conn, resolver, probe)` helper is unit-tested.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
`worker/tests/test_scan_trigger.py`:
|
||||
```python
|
||||
from lyra_worker.main import _run_scan
|
||||
from lyra_worker.types import MBTarget
|
||||
|
||||
|
||||
class FakeProbe:
|
||||
def probe(self, path):
|
||||
from lyra_worker.probe import ProbeResult
|
||||
return ProbeResult(artist="", album="", fmt="FLAC", quality_class=3)
|
||||
|
||||
|
||||
class FakeResolver:
|
||||
def resolve(self, artist, album):
|
||||
return MBTarget(artist=artist, album=album, rg_mbid="rg1", artist_mbid="a1")
|
||||
|
||||
|
||||
def _set(conn, key, value):
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
'INSERT INTO "Config"(key, value, secret, "updatedAt") VALUES (%s, %s, false, now()) '
|
||||
'ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value, "updatedAt" = now()',
|
||||
(key, value),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def _get(conn, key):
|
||||
with conn.cursor() as cur:
|
||||
cur.execute('SELECT value FROM "Config" WHERE key = %s', (key,))
|
||||
row = cur.fetchone()
|
||||
return row[0] if row else None
|
||||
|
||||
|
||||
def test_run_scan_writes_result_and_clears_flag(conn, tmp_path):
|
||||
(tmp_path / "John Mayer" / "Continuum (2006)").mkdir(parents=True)
|
||||
(tmp_path / "John Mayer" / "Continuum (2006)" / "01.flac").write_bytes(b"")
|
||||
_set(conn, "scan.requested", "true")
|
||||
|
||||
_run_scan(conn, FakeResolver(), FakeProbe(), dest_root=str(tmp_path))
|
||||
|
||||
assert _get(conn, "scan.requested") == "false" # flag cleared
|
||||
assert "imported" in (_get(conn, "scan.result") or "") # summary written
|
||||
with conn.cursor() as cur:
|
||||
cur.execute('SELECT count(*) FROM "LibraryItem"')
|
||||
assert cur.fetchone()[0] == 1
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run it to verify it fails**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/worker
|
||||
.venv/bin/python -m pytest tests/test_scan_trigger.py -v
|
||||
```
|
||||
Expected: FAIL — `cannot import name '_run_scan'`.
|
||||
|
||||
- [ ] **Step 3: Add `_run_scan` + wire the loop**
|
||||
|
||||
In `worker/lyra_worker/main.py`:
|
||||
|
||||
(a) Update imports:
|
||||
```python
|
||||
from lyra_worker.registry import build_adapters, build_browser, build_probe, build_resolver, build_tagger
|
||||
from lyra_worker.scan import scan_library
|
||||
```
|
||||
|
||||
(b) Add the helper (module level):
|
||||
```python
|
||||
_TRUE = {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _set_config(conn, key: str, value: str) -> None:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
'INSERT INTO "Config"(key, value, secret, "updatedAt") VALUES (%s, %s, false, now()) '
|
||||
'ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value, "updatedAt" = now()',
|
||||
(key, value),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def _run_scan(conn, resolver, probe, dest_root: str = DEST_ROOT) -> None:
|
||||
result = scan_library(conn, resolver, probe, dest_root)
|
||||
_set_config(conn, "scan.result", f"imported {result.imported}, skipped {result.skipped}")
|
||||
_set_config(conn, "scan.requested", "false")
|
||||
print(f"worker: library scan done — {result.imported} imported, {result.skipped} skipped", flush=True)
|
||||
```
|
||||
|
||||
(c) Build the probe and check the flag each loop. Replace `mcfg = MonitorConfig.from_config(get_config(conn))` with a single config read, and add the scan check after the monitor tick:
|
||||
```python
|
||||
probe = build_probe()
|
||||
...
|
||||
while True:
|
||||
config = get_config(conn)
|
||||
mcfg = MonitorConfig.from_config(config)
|
||||
now = time.monotonic()
|
||||
if mcfg.enabled and now - last_tick >= MONITOR_TICK_SECONDS:
|
||||
... # unchanged monitor sweep block
|
||||
last_tick = now
|
||||
|
||||
if str(config.get("scan.requested", "")).strip().lower() in _TRUE:
|
||||
try:
|
||||
_run_scan(conn, resolver, probe)
|
||||
except Exception as e: # a scan error must never kill the worker
|
||||
print(f"worker: library scan failed: {e}", flush=True)
|
||||
conn.rollback()
|
||||
_set_config(conn, "scan.requested", "false")
|
||||
|
||||
job_id = claim_next(conn)
|
||||
... # unchanged
|
||||
```
|
||||
(`build_probe` is added to the imports from `registry`.)
|
||||
|
||||
- [ ] **Step 4: Run the trigger test to verify it passes**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/worker
|
||||
.venv/bin/python -m pytest tests/test_scan_trigger.py -v
|
||||
```
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Verify main imports + full suite**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/worker
|
||||
.venv/bin/python -c "from lyra_worker import main; print('ok')"
|
||||
.venv/bin/python -m pytest -q
|
||||
```
|
||||
Expected: `ok`; full suite green, output pristine.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra
|
||||
git add worker/lyra_worker/main.py worker/tests/test_scan_trigger.py
|
||||
git commit -m "feat: worker runs the library scan when Config scan.requested is set"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Web `/api/scan` routes
|
||||
|
||||
**Files:**
|
||||
- Create: `web/src/app/api/scan/route.ts`
|
||||
- Test: `web/src/app/api/scan/route.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `prisma`, the `Config` table.
|
||||
- Produces: `POST /api/scan` upserts `Config` `scan.requested="true"` → 202 `{requested:true}`; `GET /api/scan` returns `{ result, requested }` from `Config` (`result` = `scan.result` value or `null`; `requested` = whether `scan.requested` is `"true"`).
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
`web/src/app/api/scan/route.test.ts`:
|
||||
```ts
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { prisma } from "@/lib/db";
|
||||
import { POST, GET } from "./route";
|
||||
|
||||
describe("scan API", () => {
|
||||
it("POST sets scan.requested and GET reflects it", async () => {
|
||||
const res = await POST(new Request("http://localhost/api/scan", { method: "POST" }));
|
||||
expect(res.status).toBe(202);
|
||||
expect((await res.json()).requested).toBe(true);
|
||||
|
||||
const row = await prisma.config.findUnique({ where: { key: "scan.requested" } });
|
||||
expect(row?.value).toBe("true");
|
||||
|
||||
const g = await GET();
|
||||
const body = await g.json();
|
||||
expect(body.requested).toBe(true);
|
||||
expect(body.result).toBeNull(); // no scan.result yet
|
||||
});
|
||||
|
||||
it("GET returns the worker's scan.result when present", async () => {
|
||||
await prisma.config.create({ data: { key: "scan.result", value: "imported 3, skipped 1" } });
|
||||
const body = await (await GET()).json();
|
||||
expect(body.result).toBe("imported 3, skipped 1");
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run it to verify it fails**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/web
|
||||
DATABASE_URL=postgresql://lyra:lyra@localhost:5432/lyra_test npx vitest run api/scan 2>&1 | tail -5
|
||||
```
|
||||
Expected: FAIL — cannot resolve `./route`.
|
||||
|
||||
- [ ] **Step 3: Implement the route**
|
||||
|
||||
`web/src/app/api/scan/route.ts`:
|
||||
```ts
|
||||
import { prisma } from "@/lib/db";
|
||||
|
||||
export async function POST() {
|
||||
await prisma.config.upsert({
|
||||
where: { key: "scan.requested" },
|
||||
create: { key: "scan.requested", value: "true", secret: false },
|
||||
update: { value: "true" },
|
||||
});
|
||||
return Response.json({ requested: true }, { status: 202 });
|
||||
}
|
||||
|
||||
export async function GET() {
|
||||
const [requested, result] = await Promise.all([
|
||||
prisma.config.findUnique({ where: { key: "scan.requested" } }),
|
||||
prisma.config.findUnique({ where: { key: "scan.result" } }),
|
||||
]);
|
||||
return Response.json({
|
||||
requested: requested?.value === "true",
|
||||
result: result?.value ?? null,
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the test to verify it passes**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/web
|
||||
DATABASE_URL=postgresql://lyra:lyra@localhost:5432/lyra_test npx vitest run api/scan 2>&1 | tail -5
|
||||
```
|
||||
Expected: PASS — both tests green.
|
||||
|
||||
- [ ] **Step 5: Run the full web suite**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/web
|
||||
npm test --silent 2>&1 | tail -4
|
||||
```
|
||||
Expected: PASS — every web test green.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra
|
||||
git add web/src/app/api/scan
|
||||
git commit -m "feat: /api/scan — request a library scan + read its result"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Settings "Scan library" button
|
||||
|
||||
**Files:**
|
||||
- Modify: `web/src/app/settings/settings-form.tsx`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `POST /api/scan`, `GET /api/scan`. UI — verified by `npx tsc --noEmit`.
|
||||
|
||||
- [ ] **Step 1: Add the scan control to the settings form**
|
||||
|
||||
In `web/src/app/settings/settings-form.tsx`, add scan state + handlers inside `SettingsForm` (alongside the existing state), and render a "Scan library" section below the existing `<form>`. Add near the other `useState` hooks:
|
||||
```tsx
|
||||
const [scanResult, setScanResult] = useState<string | null>(null);
|
||||
const [scanRequested, setScanRequested] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
fetch("/api/scan")
|
||||
.then((r) => r.json())
|
||||
.then((s: { result: string | null; requested: boolean }) => {
|
||||
setScanResult(s.result);
|
||||
setScanRequested(s.requested);
|
||||
});
|
||||
}, []);
|
||||
|
||||
async function requestScan() {
|
||||
await fetch("/api/scan", { method: "POST" });
|
||||
setScanRequested(true);
|
||||
}
|
||||
```
|
||||
Then, immediately AFTER the closing `</form>` tag (still inside the component's returned fragment — wrap the form and this new block in a `<>`…`</>` if needed), add:
|
||||
```tsx
|
||||
<section>
|
||||
<h2>Library</h2>
|
||||
<p>Scan <code>/music</code> and record what's already on disk as owned + monitored.</p>
|
||||
<button type="button" onClick={requestScan} disabled={scanRequested}>
|
||||
{scanRequested ? "Scan requested…" : "Scan library"}
|
||||
</button>
|
||||
{scanResult ? <p aria-label="scan result">Last scan: {scanResult}</p> : null}
|
||||
</section>
|
||||
```
|
||||
If the component currently `return`s a single `<form>...</form>`, change it to `return (<>{<form>...</form>}<section>...</section></>);` — i.e. wrap the existing form and the new `<section>` in a fragment. Do not change any existing form fields.
|
||||
|
||||
- [ ] **Step 2: Typecheck**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/web
|
||||
npx tsc --noEmit
|
||||
```
|
||||
Expected: no type errors.
|
||||
|
||||
- [ ] **Step 3: Run the web suite (no regressions)**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra/web
|
||||
npm test --silent 2>&1 | tail -4
|
||||
```
|
||||
Expected: PASS — existing tests unaffected (no component tests for the settings form; route tests still green).
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
cd /home/jonathan/Projects/lyra
|
||||
git add web/src/app/settings/settings-form.tsx
|
||||
git commit -m "feat: Settings 'Scan library' button + last-result status"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notes: library scan complete
|
||||
|
||||
With this plan, clicking **Scan library** in Settings walks `/music`, resolves each album against
|
||||
MusicBrainz, and records matched albums as **have** (`LibraryItem`), **on monitor**
|
||||
(`MonitoredRelease`, fulfilled, with real quality), and **followed** (`WatchedArtist`) — skipping
|
||||
anything MusicBrainz can't match. Re-scans are idempotent. After it lands: re-`up --build` (applies
|
||||
the nullable-`requestId` migration to the live DB), click Scan, and confirm *Continuum*/*Sob Rock*
|
||||
appear as have + monitored under their artist. Deferred: auto-scan/filesystem-watching, multi-disc
|
||||
handling, and repairing/re-tagging existing files.
|
||||
```
|
||||
Reference in New Issue
Block a user