feat(worker): separate download staging volume (STAGING_DIR)

Staging is no longer forced under the library. The worker reads STAGING_ROOT
(container path, default /music/.staging) and stages per-job downloads there;
docker-compose mounts ${STAGING_DIR:-${MUSIC_DIR}/.staging} at /staging with
STAGING_ROOT=/staging. Set STAGING_DIR to a fast local disk when the library
(MUSIC_DIR) is a network share so temp download I/O stays off the share.

Safe cross-volume: import_album already assembles into a temp dir on the
library volume and swaps atomically there, so partial downloads never touch
the library and the final swap stays atomic. clear_staging_root now clears the
root's contents (mount-safe) rather than removing the root. Documented in
.env.example; new tests cover the separate-root path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jonathan
2026-07-12 23:44:27 +02:00
parent 891793953f
commit e825e3d7ad
7 changed files with 84 additions and 16 deletions
+7 -2
View File
@@ -1,3 +1,4 @@
import os
import time
import psycopg
@@ -20,6 +21,10 @@ HEARTBEAT_SECONDS = 15.0 # how often to stamp worker.heartbeat (its updatedAt =
MONITOR_TICK_SECONDS = 60.0
DISCOVER_TICK_SECONDS = 300.0 # check the discovery interval every 5 min (interval itself is hours)
DEST_ROOT = "/music" # must match run_pipeline's default library root
# Per-job download staging. Defaults inside the library (/music/.staging) to preserve prior
# behavior; set STAGING_ROOT (mounted as a separate volume) to keep temp download I/O off a
# network-share library.
STAGING_ROOT = os.environ.get("STAGING_ROOT") or f"{DEST_ROOT}/.staging"
DEFAULT_SCAN_CHUNK = 25 # albums resolved per loop iteration (~25-50s at MB ~1 req/s)
_TRUE = {"1", "true", "yes", "on"}
@@ -132,7 +137,7 @@ def maybe_run_discovery(conn, sources, browser, config, dcfg, now, last_discover
def run_forever() -> None:
conn = wait_for_db()
clear_staging_root(DEST_ROOT) # sweep any staging dirs orphaned by a prior crash
clear_staging_root(STAGING_ROOT) # sweep any staging dirs orphaned by a prior crash
adapters = build_adapters(get_config(conn))
resolver = build_resolver()
tagger = build_tagger()
@@ -184,7 +189,7 @@ def run_forever() -> None:
print(f"worker: claimed job {job_id}", flush=True)
try:
run_pipeline(conn, job_id, adapters, resolver=resolver, tagger=tagger,
upgrade_cutoff=mcfg.quality_cutoff)
staging_root=STAGING_ROOT, upgrade_cutoff=mcfg.quality_cutoff)
except Exception as e: # one bad job must not take down the worker loop
print(f"worker: pipeline failed for job {job_id}: {e}", flush=True)
conn.rollback()