fix(worker): cap slskd ETA + rollback on progress-write failure (stop txn-abort job churn)
A near-stalled Soulseek peer drives the EWMA speed toward zero, so _eta_seconds returned int(remaining/speed) values far past int4 max. Writing that to Job.downloadEtaSeconds (an integer column) raised "integer out of range", which aborted the SHARED pipeline connection's transaction. The on_progress except clause logged but never rolled back, so every subsequent query cascaded "current transaction is aborted" and the whole job failed and requeued (9 such failures / 24h observed in prod). - Cap _eta_seconds at ~100h (359999s), well inside int4. - Roll back the shared conn in on_progress's except so a failed progress write can never poison the pipeline txn, honoring the existing docstring promise that "a progress write must never fail the download". - Regression test for the near-stall cap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -191,6 +191,12 @@ def _make_on_progress(conn: psycopg.Connection, job_id: str, reports: "threading
|
||||
try:
|
||||
_set_download_progress(conn, job_id, frac, eta_seconds)
|
||||
except Exception as e: # a progress write must never fail the download
|
||||
# Roll back so a failed write can't leave the shared pipeline conn in an aborted
|
||||
# txn (which would cascade "current transaction is aborted" into every later query).
|
||||
try:
|
||||
conn.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
print(f"pipeline: on_progress write failed: {e}", flush=True)
|
||||
|
||||
return on_progress
|
||||
|
||||
Reference in New Issue
Block a user