feat(worker): separate download staging volume (STAGING_DIR)
Staging is no longer forced under the library. The worker reads STAGING_ROOT
(container path, default /music/.staging) and stages per-job downloads there;
docker-compose mounts ${STAGING_DIR:-${MUSIC_DIR}/.staging} at /staging with
STAGING_ROOT=/staging. Set STAGING_DIR to a fast local disk when the library
(MUSIC_DIR) is a network share so temp download I/O stays off the share.
Safe cross-volume: import_album already assembles into a temp dir on the
library volume and swaps atomically there, so partial downloads never touch
the library and the final swap stays atomic. clear_staging_root now clears the
root's contents (mount-safe) rather than removing the root. Documented in
.env.example; new tests cover the separate-root path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,3 +1,4 @@
|
||||
import os
|
||||
import time
|
||||
|
||||
import psycopg
|
||||
@@ -20,6 +21,10 @@ HEARTBEAT_SECONDS = 15.0 # how often to stamp worker.heartbeat (its updatedAt =
|
||||
MONITOR_TICK_SECONDS = 60.0
|
||||
DISCOVER_TICK_SECONDS = 300.0 # check the discovery interval every 5 min (interval itself is hours)
|
||||
DEST_ROOT = "/music" # must match run_pipeline's default library root
|
||||
# Per-job download staging. Defaults inside the library (/music/.staging) to preserve prior
|
||||
# behavior; set STAGING_ROOT (mounted as a separate volume) to keep temp download I/O off a
|
||||
# network-share library.
|
||||
STAGING_ROOT = os.environ.get("STAGING_ROOT") or f"{DEST_ROOT}/.staging"
|
||||
DEFAULT_SCAN_CHUNK = 25 # albums resolved per loop iteration (~25-50s at MB ~1 req/s)
|
||||
_TRUE = {"1", "true", "yes", "on"}
|
||||
|
||||
@@ -132,7 +137,7 @@ def maybe_run_discovery(conn, sources, browser, config, dcfg, now, last_discover
|
||||
|
||||
def run_forever() -> None:
|
||||
conn = wait_for_db()
|
||||
clear_staging_root(DEST_ROOT) # sweep any staging dirs orphaned by a prior crash
|
||||
clear_staging_root(STAGING_ROOT) # sweep any staging dirs orphaned by a prior crash
|
||||
adapters = build_adapters(get_config(conn))
|
||||
resolver = build_resolver()
|
||||
tagger = build_tagger()
|
||||
@@ -184,7 +189,7 @@ def run_forever() -> None:
|
||||
print(f"worker: claimed job {job_id}", flush=True)
|
||||
try:
|
||||
run_pipeline(conn, job_id, adapters, resolver=resolver, tagger=tagger,
|
||||
upgrade_cutoff=mcfg.quality_cutoff)
|
||||
staging_root=STAGING_ROOT, upgrade_cutoff=mcfg.quality_cutoff)
|
||||
except Exception as e: # one bad job must not take down the worker loop
|
||||
print(f"worker: pipeline failed for job {job_id}: {e}", flush=True)
|
||||
conn.rollback()
|
||||
|
||||
Reference in New Issue
Block a user