fix(worker): requeue jobs that fail on an unhandled pipeline exception

An unhandled exception in run_pipeline (e.g. a MusicBrainz TLS drop surfacing as
SSL: UNEXPECTED_EOF, after the MB retries are exhausted) left the job stranded in
its mid-pipeline 'matching' state: the except handler logged "pipeline failed" and
rolled back, but never reset the job. Since claim_next only picks 'requested', the
job was orphaned — stuck showing "Searching sources…" on the press until the next
worker restart's reclaim_stuck_jobs happened to sweep it up.

Add requeue_or_fail: on an unhandled failure, requeue to 'requested' for a bounded
retry (transient network blips clear on retry), then fall to 'needs_attention' at
the attempt cap (5) so a persistently-broken job stays visible instead of looping.
Mirrors reclaim_stuck_jobs (requeue) and pipeline._fail (needs_attention), and
resets Request.status to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jonathan
2026-07-20 18:40:41 +02:00
parent e8483ee22b
commit d7ea29a97a
3 changed files with 110 additions and 1 deletions
+4 -1
View File
@@ -6,7 +6,7 @@ import uuid
import psycopg
from lyra_worker.claim import claim_next, reclaim_stuck_jobs
from lyra_worker.claim import claim_next, reclaim_stuck_jobs, requeue_or_fail
from lyra_worker.config import get_config
from lyra_worker.crypto import secret_key_problem
from lyra_worker.db import wait_for_db
@@ -323,6 +323,9 @@ def run_forever() -> None:
except Exception as e: # one bad job must not take down the worker loop
print(f"worker: pipeline failed for job {job_id}: {e}", flush=True)
conn.rollback()
# Don't orphan the job in its mid-pipeline 'matching' state (claim_next only
# picks 'requested'): requeue for a bounded retry, then needs_attention.
requeue_or_fail(conn, job_id, str(e))
finish_job(conn, job_id, mcfg)
print(f"worker: finished job {job_id}", flush=True)
next_job_allowed = time.monotonic() + job_delay_seconds(config)