---
name: ram-is-the-limit
description: 24 GB RAM, no swap, machine shared with other users' jobs: keep heavy jobs in the queue; scheduler has a RAM guard (--min-ram-free-gb 6)
metadata:
  type: project
---

The machine has 24 GB RAM and no swap; each training process holds 2.5–4 GB. RAM, not VRAM (32 GB), is the limit.

- 2026-10-04 (morning): I started 7 extra processes in tmux beside the 3 queued jobs -> OOM kill (rc 137) of a queued job.
- 2026-10-04 13:38: job 371 was OOM-killed again although only the 3 queued jobs were mine: **other users run jobs on this machine** (seen: `/root/miniconda3/envs/Huy/bin/python3 train_suite27.py` in /root/disk_3tb/earlywarning, `scripts/diagnose_flows.py` under /root/thao, plus VSCode / Cursor / Lingma servers, together ~6-9 GB). They are not mine: never kill them.

**Why:** a job that does not fit is killed by the kernel without warning and its fold is lost.

**How to apply:** all heavy jobs go through `experiments/decoder_lodo/runners/jobqueue.py`. Since commit after 0c47d59 the scheduler starts a job only if MemAvailable >= `--min-ram-free-gb` (running with 6; tmux window `jobq` in session `decoder_lodo`, started with `--max-jobs 3 --max-load 11 --min-gpu-free-gb 6 --min-ram-free-gb 6`; the load limit was raised from 7 to 11 because other users alone produce a load of ~8, which stalled the queue). A job killed with rc 137 can be requeued by moving its json from `runners/jobs/failed` to `pending` (cached folds are reused). Restart the scheduler only when `running/` is empty (hold pending jobs by moving them to `runners/jobs/held/`). Ctrl-C in the jobq pane closes the window: start a new one with `tmux new-window -d -t decoder_lodo -n jobq ...`. See [[no-blocking-waits]].
- **2026-11-12 incident:** with ~11 GB held by VS Code/Pylance servers of several sessions, running two 4-GB DirectNet trainings plus a 2-GB scoring job in parallel queues (sub/sub2/sub3) exhausted RAM: the OOM killer took two of my trainings and also another 6.7 GB process (`MainThread`, not mine). Rule: at most ONE heavy (>= 3 GB) job of mine at a time; check `free -g` and the largest RSS before adding a parallel queue; a RAM guard at start does not protect against growth later. See [[jobqueue-namespace-pitfall]].
