# Press controls + inter-job delay — design

Date: 2026-07-15

## Problem

The monitor can enqueue a large backlog of acquisition jobs at once. The worker drains
them serially with **no pacing** (only a 2s idle poll when the queue is empty) and offers
**no runtime control** over the queue — the only per-job action on the Floor is Retry (on
`needs_attention`). The user wants:

1. A **configurable delay between downloads** to spread sustained Qobuz load over time.
2. **Per-item controls** on the Floor ("the Press"): pause and remove a queued item.
3. **Whole-press controls**: pause/resume, clear the queue, and a force-stop of the
   currently-downloading album.

## Constraints (from the existing code)

- The worker is a **single, strictly-serial process** (`worker/lyra_worker/main.py:256-298`).
  One job runs to completion in a blocking `run_pipeline` call before the loop does anything
  else.
- **Config is read fresh every loop iteration** (`main.py:262`, `get_config`), so any Config
  change takes effect within ~2s with no restart.
- An **in-flight download cannot be cleanly interrupted** — `run_pipeline` and the adapter
  download are synchronous/blocking with only an `on_progress` callback, no cancel token. The
  only interruption is process death, after which `reclaim_stuck_jobs` (`claim.py:8-24`)
  requeues the in-flight job at startup and `clear_staging_root` (`main.py:233`) wipes the
  partial download.
- `claim_next` (`claim.py:27-57`) claims the oldest `state='requested'` job.
- Deleting a `Request` cascades to its `Job` and `Candidate` rows (`schema.prisma:51,66`); the
  linked `MonitoredRelease` is `SetNull`, not cascaded.
- `enqueue_due` (`monitor.py:115-147`) re-enqueues a monitored release once it has no live
  (non-terminal) job and its `lastSearchedAt` is older than `retryIntervalHours` (default 6).
  So a removed/cleared monitored item reappears later on the normal cadence unless `ignored`.
- The web container runs `prisma migrate deploy` on startup (`web/docker-entrypoint.sh`), so a
  new migration auto-applies on deploy.

## Decisions (confirmed with the user)

- **Active item:** add a global **force-kill** ("Stop now") that aborts the in-flight download
  via worker process-restart. Per-item controls act only on queued items.
- **Remove / Clear semantics:** **cancel-for-now**. Manual one-off requests are gone for good;
  monitored/wanted releases stay wanted and may reappear on the monitor's normal cadence.
  Permanent skip remains the existing per-release **Ignore**.
- **Pause scope:** **downloads only**. Pausing halts claiming/downloading; the monitor keeps
  finding and enqueueing releases (they pile up in the queue) and Resume drains them.

## Data model

- **New column** `Job.paused Boolean @default(false)` — a Prisma migration
  (`add_job_paused`). Auto-applies on deploy.
- **Config keys** (key/value `Config` rows, no migration):
  - `floor.paused` — `"true"`/`"false"`; global press pause.
  - `floor.jobDelaySeconds` — integer string, default `"0"`; delay between downloads.
  - `floor.killRequested` — `"true"`/`"false"`; one-shot force-kill signal, cleared by the
    worker on consumption.

## Worker changes

`worker/lyra_worker/main.py` and `worker/lyra_worker/claim.py`.

1. **Inter-job delay.** After `finish_job` (`main.py:297`), read `floor.jobDelaySeconds` from
   the already-fresh `config` and sleep that many seconds before the next loop iteration.
   Sleep in small (~2s) chunks so the loop stays responsive; the force-kill watcher runs on a
   separate thread and works regardless. Parse with the existing `_int`-style guard (invalid →
   0). A value of 0 preserves today's back-to-back behavior.

2. **Global pause.** Immediately before `claim_next` (`main.py:286`), if `floor.paused` is
   truthy (`_TRUE` idiom), skip claiming: `time.sleep(IDLE_SLEEP); continue`. Because the loop
   is serial, any in-flight job has already finished before this check — pause stops the
   *next* download. Monitor sweep / scan / discovery (above the claim in the loop body)
   continue to run, so the queue keeps filling while paused.

3. **Per-item pause.** `claim_next` `WHERE` gains `AND NOT "paused"`. A paused queued job is
   skipped by the claim but remains `state='requested'`, so it is not "lost" and resumes
   claiming as soon as `paused` is cleared. `reclaim_stuck_jobs` is unaffected (it only touches
   in-flight states).

4. **Force-kill watcher.** A daemon thread started in `run_forever` (modeled on
   `_liveness_heartbeat`, `main.py:216-226`) polls `floor.killRequested` on its own DB
   connection every ~1.5s. When set, it clears the flag (commit) and calls `os._exit(0)`. The
   container restart policy brings the worker back; startup `clear_staging_root` +
   `reclaim_stuck_jobs` wipe the partial download and requeue the aborted job. The "Stop now"
   action also sets `floor.paused=true`, so the worker returns **idle** rather than instantly
   re-downloading the aborted album; the user resumes when ready.

## Infrastructure (ops repo)

The prod worker's Docker restart policy is currently **`no`** (`docker inspect` →
`RestartPolicy=no`), so process death — from force-kill *or any crash* — leaves it down. The
force-kill design requires `restart: unless-stopped` on the `worker` service in the ops repo
(`/opt/git/vm-download/docker/lyra/compose.yaml`). The in-repo `docker-compose.yml` already has
this (`docker-compose.yml:53`); this change brings prod in line and is good hygiene
independent of this feature. This is a one-line change in a separate repo and must be applied
before the force-kill feature is usable in prod.

## API (web) — follows existing `retry` / `library DELETE` patterns

- **`GET /api/floor`** → `{ paused: boolean, jobDelaySeconds: number }` (reads the Config
  rows). Used by the Floor to render the paused banner and control states.
- **`POST /api/floor`** with `{ action }`:
  - `"pause"` / `"resume"` → upsert `floor.paused`.
  - `"clear"` → delete every `Request` whose `Job.state = 'requested'` (queued, not yet
    claimed). Cascade removes the Jobs and Candidates. The in-flight and terminal jobs are
    untouched. Returns the number cleared.
  - `"stop"` → upsert `floor.paused='true'` and `floor.killRequested='true'`.
- **`POST /api/requests/[id]/pause`** with `{ paused: boolean }` → set `Job.paused`. Guard:
  the job must be `state='requested'` (else `400`, matching the existing retry route), since
  pausing an in-flight job has no effect.
- **`DELETE /api/requests/[id]`** → delete the `Request` (cascade Job + Candidates). Guard:
  only when `Job.state = 'requested'`, so the active download is never touched (that is the
  job of `Stop now`). Mirrors `DELETE /api/library/[id]`.

## Web UI

`web/src/app/queue.tsx` and `web/src/app/_ui/job-row.tsx`; styling via existing `.btn`
variants in `design.css`.

- **Press header** (above the queue list):
  - `Pause the press` / `Resume` toggle (from `GET /api/floor`).
  - `Clear queue` — confirm dialog showing the queued count; calls `POST /api/floor
    {action:"clear"}`.
  - `Stop now ⚠` — rendered only when a job is actively downloading (derivable from the
    existing `/api/requests` data: any job in `matching`/`matched`/`downloading`/`tagging`).
    Confirm dialog ("aborts the current download and pauses the press"); calls `POST /api/floor
    {action:"stop"}`.
  - A "Press paused — N queued" banner/chip when `paused`.
- **Per-item controls** (queued rows only — `job.state === 'requested'`), placed in the row
  `note` slot as `.btn sm ghost`:
  - `Pause` / `Resume` → `POST /api/requests/[id]/pause`.
  - `Remove` → `DELETE /api/requests/[id]`.
  - A per-item paused row shows a subtle "paused" chip and the `Resume` button.
- Active-download and `needs_attention` rows keep their current rendering (Retry unchanged).
- Polling cadence stays 2s; `GET /api/floor` is fetched alongside the existing three calls in
  `refresh()`.

## Settings

Add a numeric field **"Delay between downloads (seconds)"** (default 0, min 0) to the settings
form (`web/src/app/settings/settings-form.tsx`), persisted to `floor.jobDelaySeconds` via the
existing settings config-write path (the same mechanism used for the other numeric/monitor
settings). Include short helptext explaining it paces sustained Qobuz load.

## Testing

- **Worker** (pytest, `worker/tests/`):
  - `claim_next` skips `paused=true` queued jobs and still claims the next unpaused one.
  - Global pause: the loop skips claiming when `floor.paused` is set (unit-test the claim-gate
    helper).
  - Delay: the delay value is read and applied (unit-test the parse/gate helper; keep the
    sleep injectable/mockable).
  - Kill watcher: when `floor.killRequested` is set, the watcher clears the flag and triggers
    exit (mock `os._exit`, assert flag cleared + exit called).
- **Web** (route tests, mirroring `requests/[id]/retry/route.test.ts` and
  `library/bulk/route.test.ts`):
  - `POST /api/floor` for each action; `clear` deletes only `requested` jobs; `stop` sets both
    flags.
  - `POST /api/requests/[id]/pause` toggles `Job.paused`; rejects non-`requested`.
  - `DELETE /api/requests/[id]` removes a queued Request; refuses an in-flight/terminal job.
  - `GET /api/floor` returns the Config-derived state.

## Edge cases

- **Remove/Clear of monitored items** reappear later by design (cancel-for-now). No extra
  handling; the per-release Ignore is the permanent path.
- **Kill with nothing downloading** still restarts the worker (brief blip). Minimized by only
  showing `Stop now` while a download is active.
- **Delay during pause:** the delay sleeps only *between* jobs; pause is checked before each
  claim, so a paused press never starts a delayed job.
- **Race:** clicking per-item Pause/Remove on a job that was just claimed — the guards
  (`state='requested'`) reject it; the UI only renders those buttons on queued rows, and the
  2s refresh reconciles.

## Out of scope

- Resuming a partially-downloaded album (no download-resume support; force-killed jobs
  re-download from scratch).
- Reordering the queue / per-item priority.
- Any change to the per-release Ignore or the wanted list UI.