feat(worker): reclaim in-flight jobs on startup (crash-resume)
A worker crash/restart mid-pipeline left its job stuck in matching/matched/ downloading/tagging with no auto-recovery — only the manual Retry (needs_attention only) could rescue it. Now on startup the worker resets any such orphaned job back to 'requested' so it gets re-claimed. Safe at startup: this fresh worker owns none of those jobs, staging is already cleared (clear_staging_root runs first), and the pipeline redoes intake→import idempotently with an atomic import. Terminal jobs (imported/needs_attention) are left untouched; claimedAt/error/downloadProgress are reset. Logged on startup. worker 218 tests / 7-skip (in-flight→requested, terminal untouched, re-claimable). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -5,7 +5,7 @@ import uuid
|
||||
|
||||
import psycopg
|
||||
|
||||
from lyra_worker.claim import claim_next
|
||||
from lyra_worker.claim import claim_next, reclaim_stuck_jobs
|
||||
from lyra_worker.config import get_config
|
||||
from lyra_worker.crypto import secret_key_problem
|
||||
from lyra_worker.db import wait_for_db
|
||||
@@ -207,6 +207,9 @@ def run_forever() -> None:
|
||||
_require_secret_key()
|
||||
conn = wait_for_db()
|
||||
clear_staging_root(STAGING_ROOT) # sweep any staging dirs orphaned by a prior crash
|
||||
reclaimed = reclaim_stuck_jobs(conn) # requeue jobs a prior crash left mid-pipeline
|
||||
if reclaimed:
|
||||
print(f"worker: reclaimed {reclaimed} in-flight job(s) from a prior crash", flush=True)
|
||||
adapters = build_adapters(get_config(conn))
|
||||
resolver = build_resolver()
|
||||
tagger = build_tagger()
|
||||
|
||||
Reference in New Issue
Block a user