Lyra/docs/superpowers/specs/2026-07-12-worker-robustness-design.md

Worker robustness: scan chunking + DB reconnect

Date: 2026-07-12 Status: approved, implementing

Problem

Two worker-loop robustness gaps surfaced after the discovery work shipped:

1. The library scan blocks the worker loop for its entire duration. scan_library walks the whole /music tree in one synchronous call, MB-resolving each album at MusicBrainz's ~1 req/s. On a large library that is minutes-to-hours during which claim_next never runs — no downloads, no monitor sweep, no discovery. It is also not resumable: a crash mid-scan restarts from the top.

2. A transient DB connection loss crashes the whole worker. The loop uses one long-lived conn created once by wait_for_db(). Any DB op on a dropped connection (e.g. AdminShutdown when Postgres restarts) raises out of run_forever and the process exits. restart: unless-stopped brings it back, but ungracefully: a scary traceback, loss of in-memory adapter/source state, and a crash mid-pipeline can leave a job dirty. Observed live during the 2026-07-12 deploy when docker compose recreated the db container.

Not in scope — discovery. run_discovery is already bounded: it processes at most discover.maxSeeds (default 50) due seeds per run and advances a lastDiscoveredAt cursor, so it is chunked and resumable across runs. Left as-is by conscious decision.

Design

A. Scan chunking

Process a bounded chunk of albums per loop iteration, persisting progress in Config, so the loop returns to claim_next / monitor between chunks. Deterministic ordering (the existing sorted(os.listdir) walk) makes a path cursor stable; all inserts are already ON CONFLICT, so overlap/re-processing is idempotent.

Config state:

• scan.requested (existing) — request a fresh scan.
• scan.cursor (new) — last fully-processed "<artist>/<album>" key. Non-empty ⇒ a scan is in progress.
• scan.progress (new) — running totals, e.g. "imported=12 skipped=3".
• scan.result (existing) — final summary, written once on completion.

New function in scan.py:

scan_chunk(conn, resolver, probe, browser, dest_root, cursor, limit)
    -> (new_cursor: str, imported_delta: int, skipped_delta: int, done: bool)

Enumerates the same nested sorted (artist_folder, album_folder) walk, skips every pair whose "artist/album" key is <= cursor, processes up to limit albums via the existing _ensure_artist / _populate_discography / _record_owned helpers (unchanged), and returns the last-processed key as new_cursor plus done=True when it reached the end without hitting limit. scan_library is kept (or reimplemented as: run scan_chunk with an unbounded limit) so its direct tests stay valid.

Loop integration (replaces the current if scan.requested: block in main.py):

• scan.requested truthy AND no scan in progress → start: clear scan.cursor and scan.progress, clear scan.requested.
• scan in progress (scan.cursor present, or just-started) → run exactly one scan_chunk, persist the advanced cursor and accumulated totals. On done: write scan.result, clear scan.cursor/scan.progress.
• One chunk per iteration ⇒ job-claim and the monitor tick run between chunks.

Config: scan.chunkSize, default 25 (~25–50 s/chunk at MB ~1 req/s).

Accepted trade-off: an artist whose albums straddle a chunk boundary has its discography browsed twice. Idempotent (ON CONFLICT DO UPDATE) and rare (albums are contiguous per artist in sorted order), so we do not persist a cross-chunk browsed set.

Library-change semantics during a scan: additions after the cursor are picked up; additions/removals before the cursor wait for the next scan. Acceptable.

B. Worker DB reconnect

Wrap the loop body in run_forever with a connection-error boundary:

while True:
    try:
        ... existing iteration ...
    except psycopg.OperationalError as e:
        print(f"worker: db connection lost ({e}); reconnecting...", flush=True)
        try: conn.close()
        except Exception: pass
        conn = wait_for_db()
        continue

AdminShutdown subclasses OperationalError, so this catches the observed case. The inner conn.rollback() calls that would themselves fail on a dead connection re-raise OperationalError and compose up to this boundary. Reuses wait_for_db()'s existing retry/backoff. Net effect: a transient blip reconnects in-process instead of crashing.

Testing

• scan_chunk (integration, lyra_test + a temp /music tree via tmp_path):
  • respects limit and returns the correct new_cursor;
  • resumes from a given cursor (processes only albums after it);
  • done=False mid-walk, done=True at the end;
  • two sequential chunks cover the same albums as one unbounded scan_library run (same imported/skipped, same rows) — idempotent equivalence;
  • cursor past the end is a no-op with done=True.
• loop orchestration (extract a testable maybe_run_scan(...) mirroring maybe_run_discovery): start clears cursor+requested; one chunk per call; completion writes scan.result and clears the cursor.
• DB reconnect: unit-test that a helper reconnects a closed connection (drive wait_for_db semantics / a small boundary helper); full-loop reconnect verified by reasoning since the infinite loop isn't directly unit-testable.

Out of scope

Discovery per-run blocking (already bounded); the _upsert_artist score-accumulate tweak; the Last.fm source.