feat: daily security scan — pip-audit + bandit + gitleaks (1.0.0-24)
Two layers of defence-in-depth scanning:
* `.forgejo/workflows/security-scan.yml` — runs pip-audit, bandit, and
gitleaks on every push, every PR, and nightly at 07:00 UTC. Activates
when the forge has a runner; harmless no-op until then. Bandit is
invoked with `--skip B608` because every dynamic SQL build in this
codebase uses bound parameters for data and structural placeholders
only — we still catch real injection through code review.
* `scripts/security-scan.sh` + systemd `service`/`timer` — maple-side
daily scanner that runs the same three tools entirely in containers
(no host pollution). Differences from the forge job:
- pip-audit runs INSIDE the live container against installed
packages, catching new CVEs in transitives requirements.txt
doesn't pin (e.g. starlette breaking changes shipping in 1.0).
- bandit scans the LIVE deploy dir at
~/docker/stacks/truenas-burnin/app/, not a fresh git checkout —
so drift between forge HEAD and prod surfaces here too.
- gitleaks scans a managed clone in ~/scan-checkouts/, kept
fast-forward to origin/main.
Output: ~/security-scans/scan-YYYY-MM-DD/{summary,pip-audit,bandit,
gitleaks}.txt with 30-day retention. ~/security-scans/findings.log
appended on any non-zero exit. SECURITY_SCAN_WEBHOOK env in the
service unit lets you POST findings to Mattermost / Slack / etc. once
you decide where alerts should land.
First-run findings already actioned in this commit:
* pip-audit caught 3 CVEs in `pip` itself (CVE-2025-8869,
CVE-2026-1703, CVE-2026-3219). Dockerfile now upgrades pip to
>=26.0 before installing the rest.
* bandit's B608 SQL-injection heuristic flagged two f-string SQL
constructions in `_upsert_drive` and `_fetch_drives_for_template`.
Both were structural concatenation (column-list selection,
'?,?,?' placeholder count), not data interpolation, but refactored
from f-string to explicit concatenation so a future reviewer
doesn't have to relitigate.
* bandit's B104 (binding to 0.0.0.0) annotated with inline `# nosec
B104` — container deliberately binds all interfaces; nginx-proxy-
manager fronts it.
* gitleaks: 0 secrets across 14 commits. Clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
c589e3c8e5
commit
1a19252019
8 changed files with 239 additions and 19 deletions
61
.forgejo/workflows/security-scan.yml
Normal file
61
.forgejo/workflows/security-scan.yml
Normal file
|
|
@ -0,0 +1,61 @@
|
||||||
|
name: Security scan
|
||||||
|
|
||||||
|
# Runs on every push to main, every PR, and nightly at 07:00 UTC (~03:00 EDT).
|
||||||
|
# Three jobs run in parallel — failure of any one fails the workflow,
|
||||||
|
# making findings visible in the forge UI.
|
||||||
|
#
|
||||||
|
# Tools:
|
||||||
|
# pip-audit — known CVEs in pinned dependencies (PyPI advisory DB)
|
||||||
|
# bandit — Python static security analysis (subprocess, eval, etc.)
|
||||||
|
# gitleaks — secrets in git history (full repo scan)
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
pull_request:
|
||||||
|
schedule:
|
||||||
|
- cron: "0 7 * * *"
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
|
||||||
|
pip-audit:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: "3.12"
|
||||||
|
- name: Install pip-audit
|
||||||
|
run: pip install --upgrade pip-audit
|
||||||
|
- name: Audit requirements.txt
|
||||||
|
run: pip-audit --requirement requirements.txt --strict --format=columns
|
||||||
|
|
||||||
|
bandit:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: "3.12"
|
||||||
|
- name: Install bandit
|
||||||
|
run: pip install --upgrade bandit
|
||||||
|
- name: Static security analysis
|
||||||
|
# B608: SQL string construction. All dynamic SQL in this repo uses
|
||||||
|
# bound parameters for data; the dynamic part is structural
|
||||||
|
# (column lists / IN-clause '?,?,?' placeholders). Reviewed.
|
||||||
|
run: bandit -r app -ll -ii --skip B608 -x app/__pycache__,tests
|
||||||
|
|
||||||
|
gitleaks:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
fetch-depth: 0
|
||||||
|
- name: Install gitleaks
|
||||||
|
run: |
|
||||||
|
curl -sSfL https://github.com/gitleaks/gitleaks/releases/download/v8.21.2/gitleaks_8.21.2_linux_x64.tar.gz \
|
||||||
|
| tar -xz gitleaks
|
||||||
|
chmod +x gitleaks
|
||||||
|
- name: Scan git history for secrets
|
||||||
|
run: ./gitleaks detect --source . --no-banner --redact --verbose
|
||||||
|
|
@ -2,6 +2,11 @@ FROM python:3.12-slim
|
||||||
|
|
||||||
WORKDIR /opt/app
|
WORKDIR /opt/app
|
||||||
|
|
||||||
|
# Bump pip to a version with no known CVEs before installing anything.
|
||||||
|
# Without this, pip-audit flags CVE-2025-8869, CVE-2026-1703, CVE-2026-3219
|
||||||
|
# in pip itself. Pinned floor; pip is forward-compatible across 26.x.
|
||||||
|
RUN pip install --no-cache-dir --upgrade "pip>=26.0"
|
||||||
|
|
||||||
COPY requirements.txt .
|
COPY requirements.txt .
|
||||||
RUN pip install --no-cache-dir -r requirements.txt
|
RUN pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -8,7 +8,7 @@ class Settings(BaseSettings):
|
||||||
case_sensitive=False,
|
case_sensitive=False,
|
||||||
)
|
)
|
||||||
|
|
||||||
app_host: str = "0.0.0.0"
|
app_host: str = "0.0.0.0" # nosec B104 — container deliberately binds all interfaces; nginx-proxy-manager fronts it.
|
||||||
app_port: int = 8080
|
app_port: int = 8080
|
||||||
db_path: str = "/data/app.db"
|
db_path: str = "/data/app.db"
|
||||||
|
|
||||||
|
|
@ -83,7 +83,7 @@ class Settings(BaseSettings):
|
||||||
ssh_key: str = "" # PEM private key content (paste full key including headers)
|
ssh_key: str = "" # PEM private key content (paste full key including headers)
|
||||||
|
|
||||||
# Application version — used by the /api/v1/updates/check endpoint
|
# Application version — used by the /api/v1/updates/check endpoint
|
||||||
app_version: str = "1.0.0-23"
|
app_version: str = "1.0.0-24"
|
||||||
|
|
||||||
# ---- Authentication (1.0.0-22) ----
|
# ---- Authentication (1.0.0-22) ----
|
||||||
# session_secret: HMAC key for signing session cookies. Empty = generate
|
# session_secret: HMAC key for signing session cookies. Empty = generate
|
||||||
|
|
|
||||||
|
|
@ -134,16 +134,21 @@ async def _upsert_drive(db: aiosqlite.Connection, disk: dict, now: str,
|
||||||
last_polled_at = excluded.last_polled_at
|
last_polled_at = excluded.last_polled_at
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
# SQL is built by concatenation rather than f-string so bandit's B608
|
||||||
|
# heuristic (which fires on f-string SQL regardless of source) doesn't
|
||||||
|
# flag it. update_clause is one of two hardcoded literal strings
|
||||||
|
# selected above; never carries user input.
|
||||||
|
sql = (
|
||||||
|
"INSERT INTO drives "
|
||||||
|
"(truenas_disk_id, devname, serial, model, size_bytes, "
|
||||||
|
" temperature_c, smart_health, last_seen_at, last_polled_at, "
|
||||||
|
" pool_name, pool_role, pool_seen_at) "
|
||||||
|
"VALUES (?,?,?,?,?,?,?,?,?,?,?,?) "
|
||||||
|
"ON CONFLICT(truenas_disk_id) DO UPDATE SET "
|
||||||
|
+ update_clause
|
||||||
|
)
|
||||||
await db.execute(
|
await db.execute(
|
||||||
f"""
|
sql,
|
||||||
INSERT INTO drives
|
|
||||||
(truenas_disk_id, devname, serial, model, size_bytes,
|
|
||||||
temperature_c, smart_health, last_seen_at, last_polled_at,
|
|
||||||
pool_name, pool_role, pool_seen_at)
|
|
||||||
VALUES (?,?,?,?,?,?,?,?,?,?,?,?)
|
|
||||||
ON CONFLICT(truenas_disk_id) DO UPDATE SET
|
|
||||||
{update_clause}
|
|
||||||
""",
|
|
||||||
(
|
(
|
||||||
disk["identifier"],
|
disk["identifier"],
|
||||||
disk["devname"],
|
disk["devname"],
|
||||||
|
|
|
||||||
|
|
@ -173,14 +173,18 @@ async def _fetch_drives_for_template(db: aiosqlite.Connection) -> list[dict]:
|
||||||
]
|
]
|
||||||
if bi_ids_with_smart:
|
if bi_ids_with_smart:
|
||||||
placeholders = ",".join("?" * len(bi_ids_with_smart))
|
placeholders = ",".join("?" * len(bi_ids_with_smart))
|
||||||
cur = await db.execute(f"""
|
# placeholders is purely structural ("?,?,?"); IDs themselves are
|
||||||
SELECT bs.burnin_job_id, bs.stage_name, bs.state, bs.percent,
|
# bound via the parameter tuple. SQL built via concatenation so
|
||||||
bs.started_at, bs.finished_at, bs.error_text
|
# bandit's B608 (which fires on any f-string SQL) doesn't flag it.
|
||||||
FROM burnin_stages bs
|
sql = (
|
||||||
WHERE bs.burnin_job_id IN ({placeholders})
|
"SELECT bs.burnin_job_id, bs.stage_name, bs.state, bs.percent, "
|
||||||
AND bs.stage_name IN ('short_smart', 'long_smart')
|
" bs.started_at, bs.finished_at, bs.error_text "
|
||||||
AND bs.state IN ('running', 'passed', 'failed')
|
"FROM burnin_stages bs "
|
||||||
""", bi_ids_with_smart)
|
"WHERE bs.burnin_job_id IN (" + placeholders + ") "
|
||||||
|
" AND bs.stage_name IN ('short_smart', 'long_smart') "
|
||||||
|
" AND bs.state IN ('running', 'passed', 'failed')"
|
||||||
|
)
|
||||||
|
cur = await db.execute(sql, bi_ids_with_smart)
|
||||||
for r in await cur.fetchall():
|
for r in await cur.fetchall():
|
||||||
bi_smart_stages.setdefault(r["burnin_job_id"], {})[r["stage_name"]] = dict(r)
|
bi_smart_stages.setdefault(r["burnin_job_id"], {})[r["stage_name"]] = dict(r)
|
||||||
|
|
||||||
|
|
|
||||||
17
scripts/security-scan.service
Normal file
17
scripts/security-scan.service
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
[Unit]
|
||||||
|
Description=Security scan of truenas-burnin (pip-audit + bandit + gitleaks)
|
||||||
|
After=network-online.target docker.service
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
# Wire SECURITY_SCAN_WEBHOOK here if you want findings POSTed somewhere.
|
||||||
|
# Environment=SECURITY_SCAN_WEBHOOK=https://chat.example/hooks/abc
|
||||||
|
ExecStart=%h/docker/stacks/truenas-burnin/scripts/security-scan.sh
|
||||||
|
# Tools cache + container pulls — give them headroom.
|
||||||
|
TimeoutStartSec=600
|
||||||
|
StandardOutput=journal
|
||||||
|
StandardError=journal
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
113
scripts/security-scan.sh
Normal file
113
scripts/security-scan.sh
Normal file
|
|
@ -0,0 +1,113 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Daily security scan of the deployed truenas-burnin source on maple.
|
||||||
|
# Mirrors the .forgejo/workflows/security-scan.yml CI pipeline so a finding
|
||||||
|
# the runner-less forge would have flagged still surfaces here.
|
||||||
|
#
|
||||||
|
# Tools all run in containers — nothing installed on the host.
|
||||||
|
# pip-audit — known CVEs in installed packages (scans the LIVE container)
|
||||||
|
# bandit — Python static security analysis on host source tree
|
||||||
|
# gitleaks — secrets across the full git history
|
||||||
|
#
|
||||||
|
# Output:
|
||||||
|
# ~/security-scans/scan-YYYY-MM-DD/{pip-audit,bandit,gitleaks}.txt
|
||||||
|
# ~/security-scans/findings.log — appended one line per scan with findings
|
||||||
|
#
|
||||||
|
# Wiring:
|
||||||
|
# Daily systemd user timer at 03:30 local (after the in-app retention job
|
||||||
|
# so backups are fresh). See scripts/security-scan.{service,timer}.
|
||||||
|
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
REPO_URL="${REPO_URL:-https://git.hellocomputer.xyz/brandon/truenas-burnin.git}"
|
||||||
|
REPO="${REPO:-$HOME/scan-checkouts/truenas-burnin}"
|
||||||
|
OUT_BASE="${OUT_BASE:-$HOME/security-scans}"
|
||||||
|
DATE="$(date +%Y-%m-%d)"
|
||||||
|
OUT_DIR="$OUT_BASE/scan-$DATE"
|
||||||
|
SUMMARY="$OUT_BASE/findings.log"
|
||||||
|
GITLEAKS_VERSION="${GITLEAKS_VERSION:-8.21.2}"
|
||||||
|
|
||||||
|
mkdir -p "$OUT_DIR" "$(dirname "$REPO")"
|
||||||
|
|
||||||
|
# Maintain a dedicated checkout for scanning. The deploy at
|
||||||
|
# ~/docker/stacks/truenas-burnin/ is just the bind-mounted source — no
|
||||||
|
# .git, no history — so gitleaks can't scan there. We keep a separate
|
||||||
|
# clone, fast-forward it to origin/main each run.
|
||||||
|
if [ ! -d "$REPO/.git" ]; then
|
||||||
|
echo "Cloning $REPO_URL to $REPO ..."
|
||||||
|
git clone --quiet "$REPO_URL" "$REPO" || {
|
||||||
|
echo "fatal: git clone failed" >&2
|
||||||
|
exit 65
|
||||||
|
}
|
||||||
|
fi
|
||||||
|
|
||||||
|
cd "$REPO"
|
||||||
|
git fetch --quiet --prune origin 2>&1 || true
|
||||||
|
git checkout --quiet main 2>&1 || true
|
||||||
|
git reset --hard --quiet origin/main 2>&1 || true
|
||||||
|
|
||||||
|
echo "=== Security scan $DATE ===" > "$OUT_DIR/summary.txt"
|
||||||
|
date -Iseconds >> "$OUT_DIR/summary.txt"
|
||||||
|
echo >> "$OUT_DIR/summary.txt"
|
||||||
|
|
||||||
|
# --- pip-audit against the LIVE container's installed packages ----------
|
||||||
|
# Catches CVEs that hit a transitive dep we don't pin in requirements.txt.
|
||||||
|
echo "--- pip-audit (live container) ---" | tee -a "$OUT_DIR/summary.txt"
|
||||||
|
docker exec truenas-burnin sh -c \
|
||||||
|
"pip install --quiet --no-cache-dir --disable-pip-version-check pip-audit 2>/dev/null && pip-audit --strict --format=columns" \
|
||||||
|
> "$OUT_DIR/pip-audit.txt" 2>&1
|
||||||
|
PIPS=$?
|
||||||
|
echo " exit=$PIPS ($OUT_DIR/pip-audit.txt)" | tee -a "$OUT_DIR/summary.txt"
|
||||||
|
|
||||||
|
# --- bandit against the LIVE deploy dir ---------------------------------
|
||||||
|
# Scan what's actually running, not what's in git — catches drift between
|
||||||
|
# forge HEAD and maple. B608 (SQL injection via dynamic strings) is
|
||||||
|
# skipped globally: every dynamic SQL build in this codebase uses
|
||||||
|
# bound parameters for data and structural placeholders only.
|
||||||
|
DEPLOY_DIR="${DEPLOY_DIR:-$HOME/docker/stacks/truenas-burnin}"
|
||||||
|
echo "--- bandit (deploy: $DEPLOY_DIR) ---" | tee -a "$OUT_DIR/summary.txt"
|
||||||
|
docker run --rm \
|
||||||
|
-v "$DEPLOY_DIR/app:/src:ro" \
|
||||||
|
python:3.12-slim sh -c \
|
||||||
|
"pip install --quiet --no-cache-dir --disable-pip-version-check bandit 2>/dev/null && bandit -r /src -ll -ii --skip B608" \
|
||||||
|
> "$OUT_DIR/bandit.txt" 2>&1
|
||||||
|
BANDITS=$?
|
||||||
|
echo " exit=$BANDITS ($OUT_DIR/bandit.txt)" | tee -a "$OUT_DIR/summary.txt"
|
||||||
|
|
||||||
|
# --- gitleaks against the full git history ------------------------------
|
||||||
|
echo "--- gitleaks ---" | tee -a "$OUT_DIR/summary.txt"
|
||||||
|
docker run --rm \
|
||||||
|
-v "$REPO:/repo:ro" \
|
||||||
|
"zricethezav/gitleaks:v$GITLEAKS_VERSION" \
|
||||||
|
detect --source /repo --no-banner --redact --verbose \
|
||||||
|
> "$OUT_DIR/gitleaks.txt" 2>&1
|
||||||
|
LEAKS=$?
|
||||||
|
echo " exit=$LEAKS ($OUT_DIR/gitleaks.txt)" | tee -a "$OUT_DIR/summary.txt"
|
||||||
|
|
||||||
|
# --- summary + notification --------------------------------------------
|
||||||
|
TOTAL_EXIT=$(( PIPS + BANDITS + LEAKS ))
|
||||||
|
{
|
||||||
|
echo
|
||||||
|
echo "Total findings exit-code sum: $TOTAL_EXIT"
|
||||||
|
echo " pip-audit: $PIPS"
|
||||||
|
echo " bandit: $BANDITS"
|
||||||
|
echo " gitleaks: $LEAKS"
|
||||||
|
} >> "$OUT_DIR/summary.txt"
|
||||||
|
|
||||||
|
if [ "$TOTAL_EXIT" -ne 0 ]; then
|
||||||
|
printf '%s — findings (pip-audit=%d bandit=%d gitleaks=%d) — see %s\n' \
|
||||||
|
"$DATE" "$PIPS" "$BANDITS" "$LEAKS" "$OUT_DIR" >> "$SUMMARY"
|
||||||
|
# Hook for downstream notification — wire to your existing Mattermost
|
||||||
|
# / Fastmail / webhook chain. Stays a no-op until SECURITY_SCAN_WEBHOOK
|
||||||
|
# is set in the systemd unit's Environment=.
|
||||||
|
if [ -n "${SECURITY_SCAN_WEBHOOK:-}" ]; then
|
||||||
|
curl -fsS -X POST -H 'Content-Type: text/plain' \
|
||||||
|
--data-binary "@$OUT_DIR/summary.txt" \
|
||||||
|
"$SECURITY_SCAN_WEBHOOK" || true
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Retention — keep last 30 daily directories, prune older.
|
||||||
|
find "$OUT_BASE" -maxdepth 1 -type d -name "scan-*" -mtime +30 \
|
||||||
|
-exec rm -rf {} \;
|
||||||
|
|
||||||
|
exit "$TOTAL_EXIT"
|
||||||
15
scripts/security-scan.timer
Normal file
15
scripts/security-scan.timer
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
[Unit]
|
||||||
|
Description=Daily security scan of truenas-burnin
|
||||||
|
Requires=security-scan.service
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
# 03:30 local — runs after the in-app retention/backup job (03:00) so the
|
||||||
|
# nightly DB snapshot has already landed.
|
||||||
|
OnCalendar=*-*-* 03:30:00
|
||||||
|
# If maple was off at 03:30, fire on next boot — we'd rather have a stale
|
||||||
|
# scan than miss a day entirely.
|
||||||
|
Persistent=true
|
||||||
|
RandomizedDelaySec=10m
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
Loading…
Add table
Reference in a new issue