Jonathan
111 lines
5.5 KiB
Markdown
111 lines
5.5 KiB
Markdown
# Worker robustness: scan chunking + DB reconnect
|
||
|
||
**Date:** 2026-07-12
|
||
**Status:** approved, implementing
|
||
|
||
## Problem
|
||
|
||
Two worker-loop robustness gaps surfaced after the discovery work shipped:
|
||
|
||
1. **The library scan blocks the worker loop for its entire duration.** `scan_library`
|
||
walks the whole `/music` tree in one synchronous call, MB-resolving each album at
|
||
MusicBrainz's ~1 req/s. On a large library that is minutes-to-hours during which
|
||
`claim_next` never runs — no downloads, no monitor sweep, no discovery. It is also
|
||
not resumable: a crash mid-scan restarts from the top.
|
||
|
||
2. **A transient DB connection loss crashes the whole worker.** The loop uses one
|
||
long-lived `conn` created once by `wait_for_db()`. Any DB op on a dropped connection
|
||
(e.g. `AdminShutdown` when Postgres restarts) raises out of `run_forever` and the
|
||
process exits. `restart: unless-stopped` brings it back, but ungracefully: a scary
|
||
traceback, loss of in-memory adapter/source state, and a crash mid-pipeline can leave
|
||
a job dirty. Observed live during the 2026-07-12 deploy when `docker compose` recreated
|
||
the `db` container.
|
||
|
||
**Not in scope — discovery.** `run_discovery` is already bounded: it processes at most
|
||
`discover.maxSeeds` (default 50) due seeds per run and advances a `lastDiscoveredAt`
|
||
cursor, so it is chunked and resumable across runs. Left as-is by conscious decision.
|
||
|
||
## Design
|
||
|
||
### A. Scan chunking
|
||
|
||
Process a bounded chunk of albums per loop iteration, persisting progress in `Config`,
|
||
so the loop returns to `claim_next` / monitor between chunks. Deterministic ordering
|
||
(the existing `sorted(os.listdir)` walk) makes a path cursor stable; all inserts are
|
||
already `ON CONFLICT`, so overlap/re-processing is idempotent.
|
||
|
||
**Config state:**
|
||
- `scan.requested` (existing) — request a fresh scan.
|
||
- `scan.cursor` (new) — last fully-processed `"<artist>/<album>"` key. Non-empty ⇒ a
|
||
scan is in progress.
|
||
- `scan.progress` (new) — running totals, e.g. `"imported=12 skipped=3"`.
|
||
- `scan.result` (existing) — final summary, written once on completion.
|
||
|
||
**New function** in `scan.py`:
|
||
```
|
||
scan_chunk(conn, resolver, probe, browser, dest_root, cursor, limit)
|
||
-> (new_cursor: str, imported_delta: int, skipped_delta: int, done: bool)
|
||
```
|
||
Enumerates the same nested sorted `(artist_folder, album_folder)` walk, skips every pair
|
||
whose `"artist/album"` key is `<= cursor`, processes up to `limit` albums via the existing
|
||
`_ensure_artist` / `_populate_discography` / `_record_owned` helpers (unchanged), and
|
||
returns the last-processed key as `new_cursor` plus `done=True` when it reached the end
|
||
without hitting `limit`. `scan_library` is kept (or reimplemented as: run `scan_chunk` with
|
||
an unbounded limit) so its direct tests stay valid.
|
||
|
||
**Loop integration** (replaces the current `if scan.requested:` block in `main.py`):
|
||
- `scan.requested` truthy AND no scan in progress → **start**: clear `scan.cursor` and
|
||
`scan.progress`, clear `scan.requested`.
|
||
- scan in progress (`scan.cursor` present, or just-started) → run **exactly one**
|
||
`scan_chunk`, persist the advanced cursor and accumulated totals. On `done`: write
|
||
`scan.result`, clear `scan.cursor`/`scan.progress`.
|
||
- One chunk per iteration ⇒ job-claim and the monitor tick run between chunks.
|
||
|
||
**Config:** `scan.chunkSize`, default **25** (~25–50 s/chunk at MB ~1 req/s).
|
||
|
||
**Accepted trade-off:** an artist whose albums straddle a chunk boundary has its
|
||
discography browsed twice. Idempotent (`ON CONFLICT DO UPDATE`) and rare (albums are
|
||
contiguous per artist in sorted order), so we do not persist a cross-chunk `browsed` set.
|
||
|
||
**Library-change semantics during a scan:** additions after the cursor are picked up;
|
||
additions/removals before the cursor wait for the next scan. Acceptable.
|
||
|
||
### B. Worker DB reconnect
|
||
|
||
Wrap the loop body in `run_forever` with a connection-error boundary:
|
||
```
|
||
while True:
|
||
try:
|
||
... existing iteration ...
|
||
except psycopg.OperationalError as e:
|
||
print(f"worker: db connection lost ({e}); reconnecting...", flush=True)
|
||
try: conn.close()
|
||
except Exception: pass
|
||
conn = wait_for_db()
|
||
continue
|
||
```
|
||
`AdminShutdown` subclasses `OperationalError`, so this catches the observed case. The
|
||
inner `conn.rollback()` calls that would themselves fail on a dead connection re-raise
|
||
`OperationalError` and compose up to this boundary. Reuses `wait_for_db()`'s existing
|
||
retry/backoff. Net effect: a transient blip reconnects in-process instead of crashing.
|
||
|
||
## Testing
|
||
|
||
- **scan_chunk** (integration, `lyra_test` + a temp `/music` tree via tmp_path):
|
||
- respects `limit` and returns the correct `new_cursor`;
|
||
- resumes from a given cursor (processes only albums after it);
|
||
- `done=False` mid-walk, `done=True` at the end;
|
||
- two sequential chunks cover the same albums as one unbounded `scan_library` run
|
||
(same imported/skipped, same rows) — idempotent equivalence;
|
||
- cursor past the end is a no-op with `done=True`.
|
||
- **loop orchestration** (extract a testable `maybe_run_scan(...)` mirroring
|
||
`maybe_run_discovery`): start clears cursor+requested; one chunk per call; completion
|
||
writes `scan.result` and clears the cursor.
|
||
- **DB reconnect**: unit-test that a helper reconnects a closed connection (drive
|
||
`wait_for_db` semantics / a small boundary helper); full-loop reconnect verified by
|
||
reasoning since the infinite loop isn't directly unit-testable.
|
||
|
||
## Out of scope
|
||
Discovery per-run blocking (already bounded); the `_upsert_artist` score-accumulate
|
||
tweak; the Last.fm source.
|