docs: design spec for Press controls + inter-job delay
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jonathan
@@ -0,0 +1,179 @@
|
||||
# Press controls + inter-job delay — design
|
||||
|
||||
Date: 2026-07-15
|
||||
|
||||
## Problem
|
||||
|
||||
The monitor can enqueue a large backlog of acquisition jobs at once. The worker drains
|
||||
them serially with **no pacing** (only a 2s idle poll when the queue is empty) and offers
|
||||
**no runtime control** over the queue — the only per-job action on the Floor is Retry (on
|
||||
`needs_attention`). The user wants:
|
||||
|
||||
1. A **configurable delay between downloads** to spread sustained Qobuz load over time.
|
||||
2. **Per-item controls** on the Floor ("the Press"): pause and remove a queued item.
|
||||
3. **Whole-press controls**: pause/resume, clear the queue, and a force-stop of the
|
||||
currently-downloading album.
|
||||
|
||||
## Constraints (from the existing code)
|
||||
|
||||
- The worker is a **single, strictly-serial process** (`worker/lyra_worker/main.py:256-298`).
|
||||
One job runs to completion in a blocking `run_pipeline` call before the loop does anything
|
||||
else.
|
||||
- **Config is read fresh every loop iteration** (`main.py:262`, `get_config`), so any Config
|
||||
change takes effect within ~2s with no restart.
|
||||
- An **in-flight download cannot be cleanly interrupted** — `run_pipeline` and the adapter
|
||||
download are synchronous/blocking with only an `on_progress` callback, no cancel token. The
|
||||
only interruption is process death, after which `reclaim_stuck_jobs` (`claim.py:8-24`)
|
||||
requeues the in-flight job at startup and `clear_staging_root` (`main.py:233`) wipes the
|
||||
partial download.
|
||||
- `claim_next` (`claim.py:27-57`) claims the oldest `state='requested'` job.
|
||||
- Deleting a `Request` cascades to its `Job` and `Candidate` rows (`schema.prisma:51,66`); the
|
||||
linked `MonitoredRelease` is `SetNull`, not cascaded.
|
||||
- `enqueue_due` (`monitor.py:115-147`) re-enqueues a monitored release once it has no live
|
||||
(non-terminal) job and its `lastSearchedAt` is older than `retryIntervalHours` (default 6).
|
||||
So a removed/cleared monitored item reappears later on the normal cadence unless `ignored`.
|
||||
- The web container runs `prisma migrate deploy` on startup (`web/docker-entrypoint.sh`), so a
|
||||
new migration auto-applies on deploy.
|
||||
|
||||
## Decisions (confirmed with the user)
|
||||
|
||||
- **Active item:** add a global **force-kill** ("Stop now") that aborts the in-flight download
|
||||
via worker process-restart. Per-item controls act only on queued items.
|
||||
- **Remove / Clear semantics:** **cancel-for-now**. Manual one-off requests are gone for good;
|
||||
monitored/wanted releases stay wanted and may reappear on the monitor's normal cadence.
|
||||
Permanent skip remains the existing per-release **Ignore**.
|
||||
- **Pause scope:** **downloads only**. Pausing halts claiming/downloading; the monitor keeps
|
||||
finding and enqueueing releases (they pile up in the queue) and Resume drains them.
|
||||
|
||||
## Data model
|
||||
|
||||
- **New column** `Job.paused Boolean @default(false)` — a Prisma migration
|
||||
(`add_job_paused`). Auto-applies on deploy.
|
||||
- **Config keys** (key/value `Config` rows, no migration):
|
||||
- `floor.paused` — `"true"`/`"false"`; global press pause.
|
||||
- `floor.jobDelaySeconds` — integer string, default `"0"`; delay between downloads.
|
||||
- `floor.killRequested` — `"true"`/`"false"`; one-shot force-kill signal, cleared by the
|
||||
worker on consumption.
|
||||
|
||||
## Worker changes
|
||||
|
||||
`worker/lyra_worker/main.py` and `worker/lyra_worker/claim.py`.
|
||||
|
||||
1. **Inter-job delay.** After `finish_job` (`main.py:297`), read `floor.jobDelaySeconds` from
|
||||
the already-fresh `config` and sleep that many seconds before the next loop iteration.
|
||||
Sleep in small (~2s) chunks so the loop stays responsive; the force-kill watcher runs on a
|
||||
separate thread and works regardless. Parse with the existing `_int`-style guard (invalid →
|
||||
0). A value of 0 preserves today's back-to-back behavior.
|
||||
|
||||
2. **Global pause.** Immediately before `claim_next` (`main.py:286`), if `floor.paused` is
|
||||
truthy (`_TRUE` idiom), skip claiming: `time.sleep(IDLE_SLEEP); continue`. Because the loop
|
||||
is serial, any in-flight job has already finished before this check — pause stops the
|
||||
*next* download. Monitor sweep / scan / discovery (above the claim in the loop body)
|
||||
continue to run, so the queue keeps filling while paused.
|
||||
|
||||
3. **Per-item pause.** `claim_next` `WHERE` gains `AND NOT "paused"`. A paused queued job is
|
||||
skipped by the claim but remains `state='requested'`, so it is not "lost" and resumes
|
||||
claiming as soon as `paused` is cleared. `reclaim_stuck_jobs` is unaffected (it only touches
|
||||
in-flight states).
|
||||
|
||||
4. **Force-kill watcher.** A daemon thread started in `run_forever` (modeled on
|
||||
`_liveness_heartbeat`, `main.py:216-226`) polls `floor.killRequested` on its own DB
|
||||
connection every ~1.5s. When set, it clears the flag (commit) and calls `os._exit(0)`. The
|
||||
container restart policy brings the worker back; startup `clear_staging_root` +
|
||||
`reclaim_stuck_jobs` wipe the partial download and requeue the aborted job. The "Stop now"
|
||||
action also sets `floor.paused=true`, so the worker returns **idle** rather than instantly
|
||||
re-downloading the aborted album; the user resumes when ready.
|
||||
|
||||
## Infrastructure (ops repo)
|
||||
|
||||
The prod worker's Docker restart policy is currently **`no`** (`docker inspect` →
|
||||
`RestartPolicy=no`), so process death — from force-kill *or any crash* — leaves it down. The
|
||||
force-kill design requires `restart: unless-stopped` on the `worker` service in the ops repo
|
||||
(`/opt/git/vm-download/docker/lyra/compose.yaml`). The in-repo `docker-compose.yml` already has
|
||||
this (`docker-compose.yml:53`); this change brings prod in line and is good hygiene
|
||||
independent of this feature. This is a one-line change in a separate repo and must be applied
|
||||
before the force-kill feature is usable in prod.
|
||||
|
||||
## API (web) — follows existing `retry` / `library DELETE` patterns
|
||||
|
||||
- **`GET /api/floor`** → `{ paused: boolean, jobDelaySeconds: number }` (reads the Config
|
||||
rows). Used by the Floor to render the paused banner and control states.
|
||||
- **`POST /api/floor`** with `{ action }`:
|
||||
- `"pause"` / `"resume"` → upsert `floor.paused`.
|
||||
- `"clear"` → delete every `Request` whose `Job.state = 'requested'` (queued, not yet
|
||||
claimed). Cascade removes the Jobs and Candidates. The in-flight and terminal jobs are
|
||||
untouched. Returns the number cleared.
|
||||
- `"stop"` → upsert `floor.paused='true'` and `floor.killRequested='true'`.
|
||||
- **`POST /api/requests/[id]/pause`** with `{ paused: boolean }` → set `Job.paused`. Guard:
|
||||
the job must be `state='requested'` (else `400`, matching the existing retry route), since
|
||||
pausing an in-flight job has no effect.
|
||||
- **`DELETE /api/requests/[id]`** → delete the `Request` (cascade Job + Candidates). Guard:
|
||||
only when `Job.state = 'requested'`, so the active download is never touched (that is the
|
||||
job of `Stop now`). Mirrors `DELETE /api/library/[id]`.
|
||||
|
||||
## Web UI
|
||||
|
||||
`web/src/app/queue.tsx` and `web/src/app/_ui/job-row.tsx`; styling via existing `.btn`
|
||||
variants in `design.css`.
|
||||
|
||||
- **Press header** (above the queue list):
|
||||
- `Pause the press` / `Resume` toggle (from `GET /api/floor`).
|
||||
- `Clear queue` — confirm dialog showing the queued count; calls `POST /api/floor
|
||||
{action:"clear"}`.
|
||||
- `Stop now ⚠` — rendered only when a job is actively downloading (derivable from the
|
||||
existing `/api/requests` data: any job in `matching`/`matched`/`downloading`/`tagging`).
|
||||
Confirm dialog ("aborts the current download and pauses the press"); calls `POST /api/floor
|
||||
{action:"stop"}`.
|
||||
- A "Press paused — N queued" banner/chip when `paused`.
|
||||
- **Per-item controls** (queued rows only — `job.state === 'requested'`), placed in the row
|
||||
`note` slot as `.btn sm ghost`:
|
||||
- `Pause` / `Resume` → `POST /api/requests/[id]/pause`.
|
||||
- `Remove` → `DELETE /api/requests/[id]`.
|
||||
- A per-item paused row shows a subtle "paused" chip and the `Resume` button.
|
||||
- Active-download and `needs_attention` rows keep their current rendering (Retry unchanged).
|
||||
- Polling cadence stays 2s; `GET /api/floor` is fetched alongside the existing three calls in
|
||||
`refresh()`.
|
||||
|
||||
## Settings
|
||||
|
||||
Add a numeric field **"Delay between downloads (seconds)"** (default 0, min 0) to the settings
|
||||
form (`web/src/app/settings/settings-form.tsx`), persisted to `floor.jobDelaySeconds` via the
|
||||
existing settings config-write path (the same mechanism used for the other numeric/monitor
|
||||
settings). Include short helptext explaining it paces sustained Qobuz load.
|
||||
|
||||
## Testing
|
||||
|
||||
- **Worker** (pytest, `worker/tests/`):
|
||||
- `claim_next` skips `paused=true` queued jobs and still claims the next unpaused one.
|
||||
- Global pause: the loop skips claiming when `floor.paused` is set (unit-test the claim-gate
|
||||
helper).
|
||||
- Delay: the delay value is read and applied (unit-test the parse/gate helper; keep the
|
||||
sleep injectable/mockable).
|
||||
- Kill watcher: when `floor.killRequested` is set, the watcher clears the flag and triggers
|
||||
exit (mock `os._exit`, assert flag cleared + exit called).
|
||||
- **Web** (route tests, mirroring `requests/[id]/retry/route.test.ts` and
|
||||
`library/bulk/route.test.ts`):
|
||||
- `POST /api/floor` for each action; `clear` deletes only `requested` jobs; `stop` sets both
|
||||
flags.
|
||||
- `POST /api/requests/[id]/pause` toggles `Job.paused`; rejects non-`requested`.
|
||||
- `DELETE /api/requests/[id]` removes a queued Request; refuses an in-flight/terminal job.
|
||||
- `GET /api/floor` returns the Config-derived state.
|
||||
|
||||
## Edge cases
|
||||
|
||||
- **Remove/Clear of monitored items** reappear later by design (cancel-for-now). No extra
|
||||
handling; the per-release Ignore is the permanent path.
|
||||
- **Kill with nothing downloading** still restarts the worker (brief blip). Minimized by only
|
||||
showing `Stop now` while a download is active.
|
||||
- **Delay during pause:** the delay sleeps only *between* jobs; pause is checked before each
|
||||
claim, so a paused press never starts a delayed job.
|
||||
- **Race:** clicking per-item Pause/Remove on a job that was just claimed — the guards
|
||||
(`state='requested'`) reject it; the UI only renders those buttons on queued rows, and the
|
||||
2s refresh reconciles.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Resuming a partially-downloaded album (no download-resume support; force-killed jobs
|
||||
re-download from scratch).
|
||||
- Reordering the queue / per-item priority.
|
||||
- Any change to the per-release Ignore or the wanted list UI.
|
||||
Reference in New Issue
Block a user