feat(ops): a "why isn't it live?" health check (signer + deploy)

Two failures today were invisible: the ngit/Amber remote signer stalling mid-push, and the deploy timer skipping while origin/main had moved. - scripts/auto-deploy.sh records the reason for every skip (and the run time) to ~/.local/state/skatespots-deploy/{last-skip,last-run}. - scripts/ops-health.sh reads that back and prints repo/push sync, the skip reason, the throttle window and timer health; exit 1 when action is needed. --probe-push runs a bounded real push (the only true signer probe), --notify desktop-notifies (deduped), --quiet is silent when healthy. - Periodic watch: skatespots-ops-health.{service,timer} (every 15 min) plus scripts/install-ops-health-timer.sh. Docs: AGENTS.md Deploy section.
2026-10-01 10:42:01 +07:00 · 2026-10-01 10:42:01 +07:00 · 50fcb419f4
commit 50fcb419f4
parent d6713ba20e
6 changed files with 289 additions and 2 deletions
--- a/AGENTS.md
+++ b/AGENTS.md
@ -992,8 +992,13 @@ Rules:

 ### Deploy
 - **Web (auto)**: a systemd user timer (`skatespots-deploy.timer`) runs `scripts/auto-deploy.sh` every ~3 min. When `origin/main` changes it builds and runs `netlify deploy --prod --dir=dist`. It deploys **only committed code** (skips if tracked files are modified) and never mutates a dirty tree. **Doc/scripts-only commits are skipped** (matches `AGENTS.md`/`*.md`/`scripts/`/`cvm-server/`/`skills/`) — Netlify bills **15 credits per production deploy**, so don't spend them on non-site changes. One-time setup: `netlify login` + `netlify link`. Logs: `journalctl --user -u skatespots-deploy`. Manual run: `scripts/auto-deploy.sh`.
-  - ⚠️ **Deploy throttle (2026-09-28):** production deploys are capped to one per **24h** by default (after a week of 68 deploys ≈ **1,020 credits** — a third of the monthly allowance). Batching a day's commits into one 15-credit deploy is the single biggest credit lever. Override with `SKATESPOTS_DEPLOY_MIN_INTERVAL=<seconds>` (`0` disables); deploy on demand with **`scripts/auto-deploy.sh --force`**. A throttled tick deliberately records nothing, so the pending change still deploys on a later tick; a **failed** deploy doesn't set the clock, so it retries immediately.
+  - ⚠️ **Deploy throttle (2026-09-28):** production deploys are capped to one per **24h** by default (after a week of 68 deploys ≈ **1,020 credits** — a third of the monthly allowance). Batching a day's commits into one 15-credit deploy is the single biggest credit lever. Override with `SKATESPOTS_DEPLOY_MIN_INTERVAL=<seconds>` (`0` disables); deploy on demand with **`scripts/auto-deploy.sh --force`**. A throttled tick deliberately does **not** record the deployed SHA, so the pending change still deploys on a later tick; a **failed** deploy doesn't set the clock, so it retries immediately.
+  - ⚠️ **Every skip records its reason** to `~/.local/state/skatespots-deploy/last-skip` (`epoch|reason|sha|detail`) — previously a skip was silent. Read it back with `scripts/ops-health.sh` (below).
  - ⚠️ Shakespeare's "Deploy" rebuilds **its own workspace**, which is NOT updated by git pushes — so pushing to the repo does not change what Shakespeare deploys. Prefer the auto-deploy timer (or drag `dist/` to Netlify).
+- **Ops health (`scripts/ops-health.sh`, 2026-10-01):** answers *"why isn't it live?"* Two failures were previously invisible and cost a whole session: the **ngit/Amber signer stalling mid-push** (`git push` hangs at *"signing event (git state) with remote signer"*) and the **deploy timer skipping** while `origin/main` had moved. The script reports repo/push sync, the recorded skip reason, the throttle window, and timer health; **exit 1** when something needs attention.
+  - Flags: `--probe-push` runs a bounded 45s **real** `git push` — signing only happens when there *is* something to push, so this is the only true signer probe (it publishes any unpushed commits; a timeout ⇒ signer down ⇒ open Amber in the foreground, or `ngit account login --interactive`). `--notify` = desktop notification (deduped by an actionable-problem signature). `--quiet` = silent when healthy. Test against a fake state dir with `SKATESPOTS_STATE_DIR=…`.
+  - ⚠️ `git push --dry-run` does **NOT** exercise signing (prints "Everything up-to-date", never signs) — do not use it as a signer probe.
+  - **Runs automatically every 15 min:** `skatespots-ops-health.timer` → `skatespots-ops-health.service` (`--quiet --notify`). Install/refresh with `scripts/install-ops-health-timer.sh`; logs `journalctl --user -u skatespots-ops-health`.
 - **Web (manual)**: `npm run build` produces the web `dist/` (with `_headers`/`_redirects`); deploy by dragging `dist/` to the Netlify site's Deploys tab.
 - **Zapstore (Android)**, from the repo root: `SIGN_WITH='bunker://...' zsp publish zapstore.yaml`
  - `zsp publish --wizard` is **broken for this app** (the wizard can't parse a nostrhub web `repository` — it supports only GitHub/GitLab/Gitea/F-Droid/local). Always pass `zapstore.yaml`.
--- a/scripts/auto-deploy.sh
+++ b/scripts/auto-deploy.sh
@ -25,6 +25,10 @@ SHA_FILE="$STATE_DIR/last-deployed-sha"
 LOCK_FILE="$STATE_DIR/last-lock-hash"
 INTERVAL_FILE="$STATE_DIR/last-deploy-epoch"
 DIST_HASH_FILE="$STATE_DIR/last-dist-hash"
+# Diagnostics for `scripts/ops-health.sh`: what the last run did (or why it did
+# nothing). Without it a skipped deploy is invisible.
+SKIP_FILE="$STATE_DIR/last-skip"
+RUN_FILE="$STATE_DIR/last-run"
 mkdir -p "$STATE_DIR"

 # Production deploys cost 15 Netlify credits each, and this timer polls every
@ -39,18 +43,27 @@ case "$MIN_INTERVAL" in ''|*[!0-9]*) MIN_INTERVAL=0 ;; esac

 log() { echo "[$(date -Is)] $*"; }

+# Record why this run did nothing, so ops-health.sh can explain a pending deploy
+# instead of it silently sitting there. Format: epoch|reason|sha|detail
+record_skip() {
+  printf '%s|%s|%s|%s\n' "$(date +%s)" "$1" "${REMOTE_SHA:-}" "${2:-}" > "$SKIP_FILE"
+}
+
 # Serialise runs (a slow deploy must not overlap the next timer tick).
 exec 9>"$STATE_DIR/.lock"
 if ! flock -n 9; then
  log "another deploy is already running — skipping"
+  record_skip locked "another deploy was running"
  exit 0
 fi

 cd "$REPO"
+date +%s > "$RUN_FILE"

 # 1. What is published now?
 if ! git fetch --quiet origin main; then
  log "git fetch failed — skipping"
+  record_skip fetch-failed "git fetch failed"
  exit 0
 fi

@ -58,12 +71,14 @@ REMOTE_SHA="$(git rev-parse origin/main)"
 LAST_SHA="$(cat "$SHA_FILE" 2>/dev/null || true)"

 if [ "$REMOTE_SHA" = "$LAST_SHA" ]; then
-  exit 0 # nothing new since the last deploy
+  rm -f "$SKIP_FILE" # nothing pending — clear any stale skip reason
+  exit 0
 fi

 # 2. Only deploy committed code (ignore untracked build/generated files).
 if [ -n "$(git status --porcelain --untracked-files=no)" ]; then
  log "origin/main moved to ${REMOTE_SHA:0:9} but tracked files are modified — skipping"
+  record_skip dirty-tree "tracked files are modified"
  exit 0
 fi

@ -75,6 +90,7 @@ if [ -n "$LAST_SHA" ] && git cat-file -e "${LAST_SHA}^{commit}" 2>/dev/null; the
  site_affecting="$(printf '%s\n' "$changed" | grep -vE '^([^/]*\.md|scripts/|cvm-server/|skills/|\.agents/)' || true)"
  if [ -n "$changed" ] && [ -z "$site_affecting" ]; then
    log "origin/main moved to ${REMOTE_SHA:0:9} but only non-site files changed — skipping deploy (no credits spent)"
+    record_skip non-site-only "only non-site files changed"
    echo "$REMOTE_SHA" > "$SHA_FILE"
    exit 0
  fi
@ -89,6 +105,7 @@ if [ "$FORCE" -eq 0 ] && [ "$MIN_INTERVAL" -gt 0 ]; then
  if [ "$ELAPSED" -lt "$MIN_INTERVAL" ]; then
    WAIT_MIN=$(( (MIN_INTERVAL - ELAPSED + 59) / 60 ))
    log "origin/main moved to ${REMOTE_SHA:0:9} but last deploy was $((ELAPSED / 60))m ago — batching (min ${MIN_INTERVAL}s, ~${WAIT_MIN}m left). Use --force to deploy now."
+    record_skip throttled "~${WAIT_MIN}m left on the 24h deploy throttle"
    exit 0
  fi
 fi
@ -99,6 +116,7 @@ if [ "$(git rev-parse HEAD)" != "$REMOTE_SHA" ]; then
    log "fast-forwarded to ${REMOTE_SHA:0:9}"
  else
    log "local main has diverged from origin/main — skipping"
+    record_skip diverged "local main diverged from origin/main"
    exit 0
  fi
 fi
@ -123,6 +141,7 @@ cp dist/index.html dist/404.html
 log "smoke-testing the built bundle…"
 if ! node scripts/smoke-test.mjs; then
  log "SMOKE TEST FAILED — aborting deploy (the built bundle did not boot)"
+  record_skip smoke-failed "the built bundle failed the smoke test"
  exit 1
 fi

@ -141,6 +160,7 @@ snapshot_hash() {
 DIST_HASH="$(snapshot_hash)"
 if [ -f "$DIST_HASH_FILE" ] && [ "$DIST_HASH" = "$(cat "$DIST_HASH_FILE")" ]; then
  log "build output is unchanged from the live deploy — skipping Netlify (no credits spent)"
+  record_skip dist-unchanged "build output identical to the live deploy"
  echo "$REMOTE_SHA" > "$SHA_FILE"
  exit 0
 fi
@ -152,4 +172,5 @@ netlify deploy --prod --dir=dist
 echo "$REMOTE_SHA" > "$SHA_FILE"
 echo "$DIST_HASH" > "$DIST_HASH_FILE"
 date +%s > "$INTERVAL_FILE"
+rm -f "$SKIP_FILE"
 log "deployed ${REMOTE_SHA:0:9} ✓"
--- a/scripts/install-ops-health-timer.sh
+++ b/scripts/install-ops-health-timer.sh
@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+#
+# Install the periodic ops-health systemd user timer (signer + deploy watch).
+#   scripts/install-ops-health-timer.sh
+set -euo pipefail
+
+SRC="$(cd "$(dirname "$0")/systemd" && pwd)"
+DEST="$HOME/.config/systemd/user"
+mkdir -p "$DEST"
+
+cp "$SRC/skatespots-ops-health.service" "$DEST/"
+cp "$SRC/skatespots-ops-health.timer" "$DEST/"
+
+chmod +x "$(cd "$(dirname "$0")" && pwd)/ops-health.sh"
+
+systemctl --user daemon-reload
+systemctl --user enable --now skatespots-ops-health.timer
+systemctl --user list-timers skatespots-ops-health.timer --no-pager
--- a/scripts/ops-health.sh
+++ b/scripts/ops-health.sh
@ -0,0 +1,222 @@
+#!/usr/bin/env bash
+#
+# ops-health.sh — "why isn't it live?" at a glance.
+# ─────────────────────────────────────────────────────────────────────────────
+# Two silent failures cost real time: the ngit/Amber remote signer stalling
+# mid-push, and the deploy timer skipping while origin/main had moved. Both were
+# invisible. This reports everything needed to tell them apart, and can notify.
+#
+#   scripts/ops-health.sh               # read-only report
+#   scripts/ops-health.sh --probe-push  # ALSO attempt a bounded real push
+#   scripts/ops-health.sh --notify      # desktop-notify on actionable problems
+#   scripts/ops-health.sh --quiet       # print/notify only when something's wrong
+#
+# `--probe-push` publishes any unpushed commits (that IS the probe — signing only
+# happens when there's something to sign). Without it the script never writes to
+# the remote.
+#
+# Exit: 0 healthy · 1 something needs attention · 2 bad usage.
+set -uo pipefail
+
+REPO="/home/sovereignsk8/SkateSpots"
+# Overridable so the report can be tested against a fake state dir.
+STATE_DIR="${SKATESPOTS_STATE_DIR:-$HOME/.local/state/skatespots-deploy}"
+SHA_FILE="$STATE_DIR/last-deployed-sha"
+INTERVAL_FILE="$STATE_DIR/last-deploy-epoch"
+SKIP_FILE="$STATE_DIR/last-skip"
+RUN_FILE="$STATE_DIR/last-run"
+NOTIFY_FILE="$STATE_DIR/last-notified"
+PENDING_FILE="$STATE_DIR/pending-since"
+mkdir -p "$STATE_DIR"
+
+PROBE_PUSH=0
+NOTIFY=0
+QUIET=0
+for arg in "$@"; do
+  case "$arg" in
+    --probe-push) PROBE_PUSH=1 ;;
+    --notify)     NOTIFY=1 ;;
+    --quiet)      QUIET=1 ;;
+    -h|--help)    sed -n '2,20p' "$0"; exit 0 ;;
+    *) echo "ops-health: unknown option: $arg" >&2; exit 2 ;;
+  esac
+done
+
+if [ -t 1 ]; then
+  B=$'\e[1m'; DIM=$'\e[2m'; R=$'\e[31m'; G=$'\e[32m'; Y=$'\e[33m'; N=$'\e[0m'
+else
+  B=; DIM=; R=; G=; Y=; N=
+fi
+
+REPORT=""
+p() { REPORT+="$1"$'\n'; }
+
+problems=()   # short, actionable strings (for --notify and the summary)
+signature=""  # stable dedupe key for notifications
+
+human_age() {
+  local s="$1"
+  if   [ "$s" -lt 60 ];    then echo "${s}s"
+  elif [ "$s" -lt 3600 ];  then echo "$((s / 60))m"
+  elif [ "$s" -lt 86400 ]; then echo "$((s / 3600))h"
+  else echo "$((s / 86400))d"; fi
+}
+
+cd "$REPO" 2>/dev/null || { echo "ops-health: repo not found: $REPO" >&2; exit 2; }
+now=$(date +%s)
+
+# ── Repo & push state ────────────────────────────────────────────────────────
+p "${B}▸ Repo & push${N}"
+ahead=$(git rev-list --count origin/main..HEAD 2>/dev/null || echo 0)
+behind=$(git rev-list --count HEAD..origin/main 2>/dev/null || echo 0)
+head_short=$(git rev-parse --short HEAD 2>/dev/null || echo '?')
+origin_short=$(git rev-parse --short origin/main 2>/dev/null || echo '?')
+
+if [ "$ahead" = "0" ] && [ "$behind" = "0" ]; then
+  p "  ${G}✓${N} in sync with origin/main ($head_short)"
+  rm -f "$PENDING_FILE"
+elif [ "$ahead" -gt 0 ]; then
+  p "  ${Y}▲${N} ${B}${ahead}${N} unpushed commit(s) (HEAD $head_short vs origin $origin_short)"
+  [ ! -f "$PENDING_FILE" ] && date +%s > "$PENDING_FILE"
+  pend_age=$(( now - $(cat "$PENDING_FILE" 2>/dev/null || echo "$now") ))
+  p "      ${DIM}waiting $(human_age "$pend_age") — push is the missing step${N}"
+  if [ "$pend_age" -gt 1200 ]; then
+    problems+=("$ahead commit(s) unpushed for $(human_age "$pend_age") — the signer may be stuck")
+    signature+="pending-stuck;"
+  fi
+else
+  p "  ${Y}▼${N} ${behind} commit(s) behind origin/main"
+fi
+
+# Optional bounded probe: signing only happens when there IS something to push.
+if [ "$PROBE_PUSH" = "1" ] && [ "$ahead" -gt 0 ]; then
+  p "  ${DIM}probing: git push origin main (bounded 45s — this publishes the commits above)…${N}"
+  start=$(date +%s)
+  out="$(timeout 45 git push origin main 2>&1)"; rc=$?
+  el=$(( $(date +%s) - start ))
+  if [ "$rc" = "0" ]; then
+    p "  ${G}✓${N} push OK in ${el}s — signer responsive"
+  elif [ "$rc" = "124" ]; then
+    p "  ${R}✗${N} push TIMED OUT after ${el}s — the ngit/Amber signer looks unresponsive"
+    p "      ${DIM}fix: open Amber in the foreground, or: ngit account login --interactive${N}"
+    problems+=("signer unresponsive (push timed out)")
+    signature+="signer-timeout;"
+  else
+    p "  ${R}✗${N} push failed (rc=$rc):"
+    printf '%s\n' "$out" | tail -3 | while IFS= read -r l; do p "      ${DIM}$l${N}"; done
+    problems+=("push failed (rc=$rc)")
+    signature+="signer-fail;"
+  fi
+elif [ "$ahead" -gt 0 ]; then
+  p "  ${DIM}run with --probe-push to actually test signing (publishes the commits above)${N}"
+fi
+
+# ── Deploy state ─────────────────────────────────────────────────────────────
+p ""
+p "${B}▸ Deploy${N}"
+deployed="$(cat "$SHA_FILE" 2>/dev/null || echo none)"
+origin="$(git rev-parse origin/main 2>/dev/null || echo none)"
+last_epoch="$(cat "$INTERVAL_FILE" 2>/dev/null || echo 0)"
+case "$last_epoch" in ''|*[!0-9]*) last_epoch=0 ;; esac
+
+if [ "$deployed" = "$origin" ] && [ "$origin" != "none" ]; then
+  if [ "$last_epoch" -gt 0 ]; then
+    p "  ${G}✓${N} live is current (${origin:0:9}), deployed $(human_age $((now - last_epoch))) ago"
+  else
+    p "  ${G}✓${N} live is current (${origin:0:9})"
+  fi
+else
+  p "  ${Y}●${N} ${B}PENDING${N}: origin/main ${origin:0:9} but last deployed ${deployed:0:9}"
+  if [ -f "$SKIP_FILE" ]; then
+    IFS='|' read -r sk_at sk_reason sk_sha sk_detail < "$SKIP_FILE"
+    sk_age=$(( now - ${sk_at:-0} ))
+    p "      last skip: ${B}${sk_reason:-unknown}${N} ($(human_age "$sk_age") ago) ${DIM}${sk_detail}${N}"
+    case "${sk_reason:-}" in
+      throttled)
+        p "      ${DIM}→ batching to save Netlify credits; it deploys automatically when the window elapses.${N}" ;;
+      dirty-tree)
+        p "      ${R}→ tracked files are modified — COMMIT them or the deploy stays skipped.${N}"
+        problems+=("deploy blocked: working tree is dirty"); signature+="dirty;" ;;
+      smoke-failed)
+        p "      ${R}→ the built bundle failed the smoke test (real breakage — check the journal).${N}"
+        problems+=("smoke test failed during deploy"); signature+="smoke;" ;;
+      diverged)
+        p "      ${R}→ local main diverged from origin/main — a manual merge is needed.${N}"
+        problems+=("local main diverged from origin/main"); signature+="diverged;" ;;
+      fetch-failed)
+        p "      ${Y}→ couldn't fetch origin (network/relay issue).${N}" ;;
+      non-site-only|dist-unchanged)
+        p "      ${DIM}→ no deploy was needed (docs-only or byte-identical build).${N}" ;;
+      locked|"")
+        p "      ${DIM}→ reason: ${sk_reason:-unknown}${N}" ;;
+      *)
+        p "      ${DIM}→ reason: ${sk_reason}${N}" ;;
+    esac
+  else
+    p "      ${Y}no skip reason recorded — the deploy timer may not be running.${N}"
+    problems+=("deploy pending with no recorded reason"); signature+="noreason;"
+  fi
+fi
+
+min_interval="${SKATESPOTS_DEPLOY_MIN_INTERVAL:-86400}"
+case "$min_interval" in ''|*[!0-9]*) min_interval=0 ;; esac
+if [ "$min_interval" -gt 0 ] && [ "$last_epoch" -gt 0 ]; then
+  remain=$(( min_interval - (now - last_epoch) ))
+  if [ "$remain" -gt 0 ]; then
+    p "  ${DIM}throttle: next auto-deploy eligible in ~$(human_age "$remain") · force now: scripts/auto-deploy.sh --force${N}"
+  fi
+fi
+
+# ── Timer ────────────────────────────────────────────────────────────────────
+p ""
+p "${B}▸ Timer${N}"
+timer_state="$(systemctl --user is-active skatespots-deploy.timer 2>/dev/null || echo unknown)"
+if [ "$timer_state" = "active" ]; then
+  next_at="$(systemctl --user list-timers skatespots-deploy.timer --no-legend --no-pager 2>/dev/null | awk '{print $1" "$2" "$3}')"
+  p "  ${G}✓${N} skatespots-deploy.timer active${next_at:+ ${DIM}next $next_at${N}}"
+else
+  p "  ${R}✗${N} skatespots-deploy.timer: ${timer_state}"
+  problems+=("deploy timer is not active")
+  signature+="timer;"
+fi
+if [ -f "$RUN_FILE" ]; then
+  run_age=$(( now - $(cat "$RUN_FILE" 2>/dev/null || echo 0) ))
+  if [ "$run_age" -gt 600 ]; then
+    p "  ${Y}⚠${N} deploy script last ran $(human_age "$run_age") ago (it polls ~every 3m)"
+    problems+=("deploy script hasn't run in $(human_age "$run_age")")
+    signature+="stale-run;"
+  else
+    p "  ${DIM}last run $(human_age "$run_age") ago${N}"
+  fi
+fi
+
+# ── Summary + optional notification ──────────────────────────────────────────
+p ""
+if [ ${#problems[@]} -eq 0 ]; then
+  p "${G}${B}All good.${N}"
+else
+  p "${R}${B}Needs attention:${N}"
+  for prob in "${problems[@]}"; do p "  • $prob"; done
+fi
+
+if [ "$NOTIFY" = "1" ]; then
+  if [ ${#problems[@]} -gt 0 ]; then
+    prev="$(cat "$NOTIFY_FILE" 2>/dev/null || true)"
+    if [ "$signature" != "$prev" ]; then
+      body="$(printf '%s; ' "${problems[@]}")"
+      if command -v notify-send >/dev/null 2>&1; then
+        notify-send -a SkateSpots "SkateSpots ops" "$body" 2>/dev/null || true
+      fi
+      printf '%s' "$signature" > "$NOTIFY_FILE"
+    fi
+  else
+    rm -f "$NOTIFY_FILE" # healthy → re-arm notifications
+  fi
+fi
+
+if [ "$QUIET" = "1" ] && [ ${#problems[@]} -eq 0 ]; then
+  exit 0
+fi
+printf '%s' "$REPORT"
+
+[ ${#problems[@]} -eq 0 ] && exit 0 || exit 1
--- a/scripts/systemd/skatespots-ops-health.service
+++ b/scripts/systemd/skatespots-ops-health.service
@ -0,0 +1,10 @@
+[Unit]
+Description=SkateSpots ops health check (signer + deploy)
+Documentation=file:///home/sovereignsk8/SkateSpots/scripts/ops-health.sh
+
+[Service]
+Type=oneshot
+WorkingDirectory=/home/sovereignsk8/SkateSpots
+# --quiet: print nothing when healthy. --notify: desktop notification, but only
+# when the set of problems changes (no repeated spam).
+ExecStart=/home/sovereignsk8/SkateSpots/scripts/ops-health.sh --quiet --notify
--- a/scripts/systemd/skatespots-ops-health.timer
+++ b/scripts/systemd/skatespots-ops-health.timer
@ -0,0 +1,11 @@
+[Unit]
+Description=Periodic SkateSpots ops health check
+
+[Timer]
+OnBootSec=10min
+OnUnitActiveSec=15min
+AccuracySec=1min
+Persistent=true
+
+[Install]
+WantedBy=timers.target