docs: add README operator guide
First operator-facing README. Covers quick start (build, configure, first-user login), the multi-drive batch workflow with concrete time estimates, the four drive-lock states with their confirm tokens, notable settings, daily report / notifications, ops cookbook (logs, user CLI, backups, /health probe, DB reset), and an honest "known gaps" list. Cross-references CLAUDE.md (architecture + rationale) and SPEC.md (per-version feature reference) for deeper docs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Brandon Walter
parent
d4c0770b9e
commit
c589e3c8e5
1 changed files with 242 additions and 0 deletions
242
README.md
Normal file
242
README.md
Normal file
|
|
@ -0,0 +1,242 @@
|
|||
# TrueNAS Burn-In Dashboard
|
||||
|
||||
Web dashboard for running disciplined burn-in tests on TrueNAS drives.
|
||||
Sits next to the NAS, not on it — orchestrates `smartctl`, `badblocks`, and
|
||||
`nvme-cli` over SSH and tracks every job in SQLite.
|
||||
|
||||
Inspired by the community `disk-burnin.sh` script (Spearfoot et al.) but
|
||||
adds: concurrent burn-ins, pool-membership safety locks, login + audit,
|
||||
live progress UI, daily email reports, and resumable state.
|
||||
|
||||
## Stack
|
||||
|
||||
FastAPI + HTMX (SSE) + asyncssh + SQLite, in one Docker container. No
|
||||
external services beyond your TrueNAS host. Templates and static assets
|
||||
are bind-mounted; Python source is baked into the image.
|
||||
|
||||
---
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# 1. Configure
|
||||
cp .env.example .env
|
||||
# edit SSH_HOST / SSH_USER / SSH_KEY (see .env.example) and, optionally,
|
||||
# INITIAL_ADMIN_USERNAME / INITIAL_ADMIN_PASSWORD for first-run setup.
|
||||
|
||||
# 2. Build + run
|
||||
docker compose up -d --build
|
||||
|
||||
# 3. Open the dashboard
|
||||
open http://localhost:8084 # or your host's IP
|
||||
|
||||
# 4. First time: the login page renders a "Create initial admin" form.
|
||||
# Pick a username + password (>= 8 chars). Done.
|
||||
```
|
||||
|
||||
If you set `INITIAL_ADMIN_*` env vars *and* the users table is empty, that
|
||||
account is created on startup automatically. After that the env vars are
|
||||
ignored — change passwords from the UI ("Change password" header link) or
|
||||
the CLI (`docker exec -it truenas-burnin python -m app.auth_cli reset
|
||||
<username>`).
|
||||
|
||||
---
|
||||
|
||||
## Burning in many drives at once
|
||||
|
||||
The dashboard runs **up to `max_parallel_burnins`** burn-ins concurrently
|
||||
(configurable in Settings, default 4) and queues the rest. Submitting 14
|
||||
drives doesn't take 14 separate clicks — you submit once and the queue
|
||||
drains automatically as slots free up.
|
||||
|
||||
### The workflow
|
||||
|
||||
1. **Select all idle drives** — click the checkbox in the table header
|
||||
(next to "DRIVE"). It auto-checks every drive that's currently
|
||||
selectable: idle, no active SMART test, not pool-locked. Pool-locked
|
||||
drives are intentionally excluded; if you really want to burn one of
|
||||
them in, unlock it individually first (see [Drive locks](#drive-locks)
|
||||
below).
|
||||
2. **Click the Burn-In button** in the batch action bar that slides up
|
||||
from the bottom — it shows the count of selected drives.
|
||||
3. **In the batch modal**: pick the stages to run (Short SMART, Long
|
||||
SMART, Surface Validate — drag to reorder), confirm your operator
|
||||
name, and click Start.
|
||||
4. **All selected drives are queued** in one POST. Up to
|
||||
`max_parallel_burnins` enter `running`; the rest sit `queued`. As each
|
||||
running job finishes, the next queued job picks up the freed slot
|
||||
automatically — no operator action between batches.
|
||||
5. The toast shows e.g. "12 burn-in(s) queued, 0 skipped, 0 pool-locked."
|
||||
|
||||
### Time estimate
|
||||
|
||||
| Drive size | Profile | Per-drive runtime (default block size) |
|
||||
|-----------|-------------|----------------------------------------|
|
||||
| 250 GB SSD | Short + Long SMART + Surface | ~1 hour |
|
||||
| 14 TB HDD | Short + Long SMART + Surface | ~24 hours |
|
||||
| 14 TB HDD | Short + Long SMART (no surface) | ~6–8 hours |
|
||||
|
||||
For 12× 14 TB drives at default 4-parallel: roughly **3–4 days** end-to-end.
|
||||
Bumping `surface_validate_block_size` from 4096 to 8192 in Settings cuts
|
||||
runtime roughly in half at ~2× RAM cost — matches the upstream
|
||||
`disk-burnin.sh` recommendation.
|
||||
|
||||
### Watch out
|
||||
|
||||
- **Stuck-job timeout** — `stuck_job_hours` (default 24) marks any job
|
||||
past that threshold as `unknown` and kills the remote process. If
|
||||
you're burning in 14 TB drives with default block size, raise this to
|
||||
**48** in Settings before starting, or you'll get false positives near
|
||||
the end of surface_validate.
|
||||
- **Thermal gate** — if drives currently under burn-in hit the
|
||||
temperature warning threshold, new jobs wait up to 3 minutes before
|
||||
acquiring a slot. Increase `temp_warn_c` if your chassis runs hot but
|
||||
is otherwise fine.
|
||||
|
||||
### Cancelling
|
||||
|
||||
Click the red ✕ next to a running job. The orchestrator:
|
||||
1. Marks the job `cancelled` in the DB.
|
||||
2. Issues `kill -9 <remote_pid>` over a fresh SSH session (the badblocks
|
||||
PID is captured at launch via `sh -c 'echo PID:$$; exec ...'`).
|
||||
3. Cancels the asyncio task, releasing the semaphore slot for the next
|
||||
queued job.
|
||||
|
||||
Cancellations are durable — restart the container and queued jobs resume,
|
||||
cancelled jobs stay cancelled.
|
||||
|
||||
---
|
||||
|
||||
## Drive locks
|
||||
|
||||
To prevent destroying live data, the dashboard refuses to start
|
||||
destructive burn-in on drives ZFS or the kernel reports as in-use.
|
||||
Detected lock states (with the typed-confirmation token required to
|
||||
override):
|
||||
|
||||
| State | Detected via | Confirm token |
|
||||
|---------------|---------------------------|------------------------------|
|
||||
| Active pool | `zpool list -vHP` | the pool name (e.g. `tank`) |
|
||||
| Boot pool | pool name = `boot-pool` | `DESTROY BOOT POOL` |
|
||||
| Exported ZFS | `lsblk` `zfs_member` partitions not in any active pool | `DESTROY EXPORTED POOL` |
|
||||
| Mounted FS | `findmnt -no SOURCE` | `DESTROY MOUNTED FILESYSTEM` |
|
||||
|
||||
Detection runs every poll cycle (~12 s). On any SSH or parser failure the
|
||||
poller fails *closed*: previously-locked drives stay locked, previously-
|
||||
unlocked drives stay unlocked, until detection recovers.
|
||||
|
||||
Unlock is in-memory only with a 10-minute TTL — bound to the
|
||||
`(pool_name, pool_role)` observed at unlock time. If a subsequent poll
|
||||
reclassifies the drive (e.g. `(exported)` → `tank` because someone
|
||||
imported the pool), the grant is invalidated automatically.
|
||||
|
||||
Every unlock writes an audit event and surfaces in the next daily report
|
||||
in a red banner.
|
||||
|
||||
---
|
||||
|
||||
## Settings highlights
|
||||
|
||||
All settings live under `/settings` (header link). Key knobs:
|
||||
|
||||
- **`max_parallel_burnins`** (default 4) — semaphore cap. Restart container
|
||||
for changes to take effect.
|
||||
- **`surface_validate_block_size` / `_block_buffer` / `_passes`** —
|
||||
badblocks `-b` / `-c` / `-p`. Defaults preserve original behaviour;
|
||||
tune for speed vs paranoia.
|
||||
- **`stuck_job_hours`** (default 24) — raise for big drives.
|
||||
- **`temp_warn_c` / `temp_crit_c`** — thermal gating thresholds.
|
||||
- **`bad_block_threshold`** (default 0) — number of bad blocks
|
||||
surface_validate tolerates before failing the stage.
|
||||
- **`retention_log_days`** (default 35) — when to NULL out
|
||||
`burnin_stages.log_text`. Nightly job at 03:00 local.
|
||||
- **`retention_backup_keep`** (default 14) — how many nightly DB
|
||||
snapshots to keep in `/data/backups/`.
|
||||
|
||||
---
|
||||
|
||||
## Notifications
|
||||
|
||||
- **Daily SMTP report** at `smtp_report_hour` (default 08:00 local) with
|
||||
drive-level summary, failed-health banner, and a red banner listing
|
||||
every pool-drive unlock from the last 24 h.
|
||||
- **Per-job email alerts** on pass/fail (configurable).
|
||||
- **Webhook URL** posts JSON on every job state change.
|
||||
|
||||
Configure SMTP in Settings → Email. Includes a "Test SMTP" button.
|
||||
|
||||
---
|
||||
|
||||
## Operations
|
||||
|
||||
### Logs
|
||||
|
||||
```bash
|
||||
docker logs -f truenas-burnin
|
||||
# JSON-structured. Filter with jq:
|
||||
docker logs truenas-burnin 2>&1 | jq -rR 'fromjson? | "\(.ts) \(.level) \(.msg)"'
|
||||
```
|
||||
|
||||
### User management
|
||||
|
||||
```bash
|
||||
docker exec -it truenas-burnin python -m app.auth_cli list
|
||||
docker exec -it truenas-burnin python -m app.auth_cli add <username>
|
||||
docker exec -it truenas-burnin python -m app.auth_cli reset <username>
|
||||
```
|
||||
|
||||
Passwords are read from a TTY prompt; never accept them on the command
|
||||
line.
|
||||
|
||||
### Backups
|
||||
|
||||
Automated nightly to `/data/backups/app-YYYY-MM-DD.db` (online
|
||||
`sqlite3.backup`, doesn't lock writers). To restore:
|
||||
|
||||
```bash
|
||||
docker compose down
|
||||
cp data/backups/app-2026-05-01.db data/app.db
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Health probe
|
||||
|
||||
`/health` is unauthenticated and returns 200 only when DB, poller, and
|
||||
SSH (when configured) all check green; 503 otherwise. Use it for
|
||||
container/orchestrator health checks.
|
||||
|
||||
```bash
|
||||
curl -sf http://localhost:8084/health | jq
|
||||
```
|
||||
|
||||
### Resetting the DB
|
||||
|
||||
If you need to start over:
|
||||
|
||||
```bash
|
||||
docker compose down
|
||||
sudo rm -f data/app.db data/session_secret
|
||||
# keep data/settings_overrides.json if you want to preserve UI settings
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- `CLAUDE.md` — full architecture, file map, deploy workflow, and the
|
||||
rationale behind every non-obvious design decision.
|
||||
- `SPEC.md` — canonical feature reference per version.
|
||||
- `tests/` — `python -m unittest discover tests/` (44 tests, stdlib-only).
|
||||
|
||||
---
|
||||
|
||||
## Known gaps / not-yet-built
|
||||
|
||||
- No multi-user RBAC — every user is effectively admin.
|
||||
- No per-drive SMART attribute trend graphs (snapshots only).
|
||||
- No scheduled burn-ins — jobs run immediately when queued.
|
||||
- No CSRF tokens on state-changing endpoints (relies on
|
||||
`SameSite=Strict` session cookie).
|
||||
|
||||
PRs welcome.
|
||||
Loading…
Add table
Reference in a new issue