feat(worker): reclaim in-flight jobs on startup (crash-resume)

A worker crash/restart mid-pipeline left its job stuck in matching/matched/
downloading/tagging with no auto-recovery — only the manual Retry (needs_attention
only) could rescue it. Now on startup the worker resets any such orphaned job
back to 'requested' so it gets re-claimed.

Safe at startup: this fresh worker owns none of those jobs, staging is already
cleared (clear_staging_root runs first), and the pipeline redoes intake→import
idempotently with an atomic import. Terminal jobs (imported/needs_attention) are
left untouched; claimedAt/error/downloadProgress are reset. Logged on startup.

worker 218 tests / 7-skip (in-flight→requested, terminal untouched, re-claimable).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jonathan
2026-07-14 12:37:16 +02:00
parent 4144188a28
commit 746d61b264
3 changed files with 82 additions and 1 deletions
+4 -1
View File
@@ -5,7 +5,7 @@ import uuid
import psycopg
from lyra_worker.claim import claim_next
from lyra_worker.claim import claim_next, reclaim_stuck_jobs
from lyra_worker.config import get_config
from lyra_worker.crypto import secret_key_problem
from lyra_worker.db import wait_for_db
@@ -207,6 +207,9 @@ def run_forever() -> None:
_require_secret_key()
conn = wait_for_db()
clear_staging_root(STAGING_ROOT) # sweep any staging dirs orphaned by a prior crash
reclaimed = reclaim_stuck_jobs(conn) # requeue jobs a prior crash left mid-pipeline
if reclaimed:
print(f"worker: reclaimed {reclaimed} in-flight job(s) from a prior crash", flush=True)
adapters = build_adapters(get_config(conn))
resolver = build_resolver()
tagger = build_tagger()