diff --git a/AGENTS.md b/AGENTS.md index 153e21e..e56d1ef 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -992,8 +992,13 @@ Rules: ### Deploy - **Web (auto)**: a systemd user timer (`skatespots-deploy.timer`) runs `scripts/auto-deploy.sh` every ~3 min. When `origin/main` changes it builds and runs `netlify deploy --prod --dir=dist`. It deploys **only committed code** (skips if tracked files are modified) and never mutates a dirty tree. **Doc/scripts-only commits are skipped** (matches `AGENTS.md`/`*.md`/`scripts/`/`cvm-server/`/`skills/`) — Netlify bills **15 credits per production deploy**, so don't spend them on non-site changes. One-time setup: `netlify login` + `netlify link`. Logs: `journalctl --user -u skatespots-deploy`. Manual run: `scripts/auto-deploy.sh`. - - ⚠️ **Deploy throttle (2026-09-28):** production deploys are capped to one per **24h** by default (after a week of 68 deploys ≈ **1,020 credits** — a third of the monthly allowance). Batching a day's commits into one 15-credit deploy is the single biggest credit lever. Override with `SKATESPOTS_DEPLOY_MIN_INTERVAL=` (`0` disables); deploy on demand with **`scripts/auto-deploy.sh --force`**. A throttled tick deliberately records nothing, so the pending change still deploys on a later tick; a **failed** deploy doesn't set the clock, so it retries immediately. + - ⚠️ **Deploy throttle (2026-09-28):** production deploys are capped to one per **24h** by default (after a week of 68 deploys ≈ **1,020 credits** — a third of the monthly allowance). Batching a day's commits into one 15-credit deploy is the single biggest credit lever. Override with `SKATESPOTS_DEPLOY_MIN_INTERVAL=` (`0` disables); deploy on demand with **`scripts/auto-deploy.sh --force`**. A throttled tick deliberately does **not** record the deployed SHA, so the pending change still deploys on a later tick; a **failed** deploy doesn't set the clock, so it retries immediately. + - ⚠️ **Every skip records its reason** to `~/.local/state/skatespots-deploy/last-skip` (`epoch|reason|sha|detail`) — previously a skip was silent. Read it back with `scripts/ops-health.sh` (below). - ⚠️ Shakespeare's "Deploy" rebuilds **its own workspace**, which is NOT updated by git pushes — so pushing to the repo does not change what Shakespeare deploys. Prefer the auto-deploy timer (or drag `dist/` to Netlify). +- **Ops health (`scripts/ops-health.sh`, 2026-10-01):** answers *"why isn't it live?"* Two failures were previously invisible and cost a whole session: the **ngit/Amber signer stalling mid-push** (`git push` hangs at *"signing event (git state) with remote signer"*) and the **deploy timer skipping** while `origin/main` had moved. The script reports repo/push sync, the recorded skip reason, the throttle window, and timer health; **exit 1** when something needs attention. + - Flags: `--probe-push` runs a bounded 45s **real** `git push` — signing only happens when there *is* something to push, so this is the only true signer probe (it publishes any unpushed commits; a timeout ⇒ signer down ⇒ open Amber in the foreground, or `ngit account login --interactive`). `--notify` = desktop notification (deduped by an actionable-problem signature). `--quiet` = silent when healthy. Test against a fake state dir with `SKATESPOTS_STATE_DIR=…`. + - ⚠️ `git push --dry-run` does **NOT** exercise signing (prints "Everything up-to-date", never signs) — do not use it as a signer probe. + - **Runs automatically every 15 min:** `skatespots-ops-health.timer` → `skatespots-ops-health.service` (`--quiet --notify`). Install/refresh with `scripts/install-ops-health-timer.sh`; logs `journalctl --user -u skatespots-ops-health`. - **Web (manual)**: `npm run build` produces the web `dist/` (with `_headers`/`_redirects`); deploy by dragging `dist/` to the Netlify site's Deploys tab. - **Zapstore (Android)**, from the repo root: `SIGN_WITH='bunker://...' zsp publish zapstore.yaml` - `zsp publish --wizard` is **broken for this app** (the wizard can't parse a nostrhub web `repository` — it supports only GitHub/GitLab/Gitea/F-Droid/local). Always pass `zapstore.yaml`. diff --git a/scripts/auto-deploy.sh b/scripts/auto-deploy.sh index bc6a707..faaecf8 100755 --- a/scripts/auto-deploy.sh +++ b/scripts/auto-deploy.sh @@ -25,6 +25,10 @@ SHA_FILE="$STATE_DIR/last-deployed-sha" LOCK_FILE="$STATE_DIR/last-lock-hash" INTERVAL_FILE="$STATE_DIR/last-deploy-epoch" DIST_HASH_FILE="$STATE_DIR/last-dist-hash" +# Diagnostics for `scripts/ops-health.sh`: what the last run did (or why it did +# nothing). Without it a skipped deploy is invisible. +SKIP_FILE="$STATE_DIR/last-skip" +RUN_FILE="$STATE_DIR/last-run" mkdir -p "$STATE_DIR" # Production deploys cost 15 Netlify credits each, and this timer polls every @@ -39,18 +43,27 @@ case "$MIN_INTERVAL" in ''|*[!0-9]*) MIN_INTERVAL=0 ;; esac log() { echo "[$(date -Is)] $*"; } +# Record why this run did nothing, so ops-health.sh can explain a pending deploy +# instead of it silently sitting there. Format: epoch|reason|sha|detail +record_skip() { + printf '%s|%s|%s|%s\n' "$(date +%s)" "$1" "${REMOTE_SHA:-}" "${2:-}" > "$SKIP_FILE" +} + # Serialise runs (a slow deploy must not overlap the next timer tick). exec 9>"$STATE_DIR/.lock" if ! flock -n 9; then log "another deploy is already running — skipping" + record_skip locked "another deploy was running" exit 0 fi cd "$REPO" +date +%s > "$RUN_FILE" # 1. What is published now? if ! git fetch --quiet origin main; then log "git fetch failed — skipping" + record_skip fetch-failed "git fetch failed" exit 0 fi @@ -58,12 +71,14 @@ REMOTE_SHA="$(git rev-parse origin/main)" LAST_SHA="$(cat "$SHA_FILE" 2>/dev/null || true)" if [ "$REMOTE_SHA" = "$LAST_SHA" ]; then - exit 0 # nothing new since the last deploy + rm -f "$SKIP_FILE" # nothing pending — clear any stale skip reason + exit 0 fi # 2. Only deploy committed code (ignore untracked build/generated files). if [ -n "$(git status --porcelain --untracked-files=no)" ]; then log "origin/main moved to ${REMOTE_SHA:0:9} but tracked files are modified — skipping" + record_skip dirty-tree "tracked files are modified" exit 0 fi @@ -75,6 +90,7 @@ if [ -n "$LAST_SHA" ] && git cat-file -e "${LAST_SHA}^{commit}" 2>/dev/null; the site_affecting="$(printf '%s\n' "$changed" | grep -vE '^([^/]*\.md|scripts/|cvm-server/|skills/|\.agents/)' || true)" if [ -n "$changed" ] && [ -z "$site_affecting" ]; then log "origin/main moved to ${REMOTE_SHA:0:9} but only non-site files changed — skipping deploy (no credits spent)" + record_skip non-site-only "only non-site files changed" echo "$REMOTE_SHA" > "$SHA_FILE" exit 0 fi @@ -89,6 +105,7 @@ if [ "$FORCE" -eq 0 ] && [ "$MIN_INTERVAL" -gt 0 ]; then if [ "$ELAPSED" -lt "$MIN_INTERVAL" ]; then WAIT_MIN=$(( (MIN_INTERVAL - ELAPSED + 59) / 60 )) log "origin/main moved to ${REMOTE_SHA:0:9} but last deploy was $((ELAPSED / 60))m ago — batching (min ${MIN_INTERVAL}s, ~${WAIT_MIN}m left). Use --force to deploy now." + record_skip throttled "~${WAIT_MIN}m left on the 24h deploy throttle" exit 0 fi fi @@ -99,6 +116,7 @@ if [ "$(git rev-parse HEAD)" != "$REMOTE_SHA" ]; then log "fast-forwarded to ${REMOTE_SHA:0:9}" else log "local main has diverged from origin/main — skipping" + record_skip diverged "local main diverged from origin/main" exit 0 fi fi @@ -123,6 +141,7 @@ cp dist/index.html dist/404.html log "smoke-testing the built bundle…" if ! node scripts/smoke-test.mjs; then log "SMOKE TEST FAILED — aborting deploy (the built bundle did not boot)" + record_skip smoke-failed "the built bundle failed the smoke test" exit 1 fi @@ -141,6 +160,7 @@ snapshot_hash() { DIST_HASH="$(snapshot_hash)" if [ -f "$DIST_HASH_FILE" ] && [ "$DIST_HASH" = "$(cat "$DIST_HASH_FILE")" ]; then log "build output is unchanged from the live deploy — skipping Netlify (no credits spent)" + record_skip dist-unchanged "build output identical to the live deploy" echo "$REMOTE_SHA" > "$SHA_FILE" exit 0 fi @@ -152,4 +172,5 @@ netlify deploy --prod --dir=dist echo "$REMOTE_SHA" > "$SHA_FILE" echo "$DIST_HASH" > "$DIST_HASH_FILE" date +%s > "$INTERVAL_FILE" +rm -f "$SKIP_FILE" log "deployed ${REMOTE_SHA:0:9} ✓" diff --git a/scripts/install-ops-health-timer.sh b/scripts/install-ops-health-timer.sh new file mode 100755 index 0000000..21802d4 --- /dev/null +++ b/scripts/install-ops-health-timer.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# +# Install the periodic ops-health systemd user timer (signer + deploy watch). +# scripts/install-ops-health-timer.sh +set -euo pipefail + +SRC="$(cd "$(dirname "$0")/systemd" && pwd)" +DEST="$HOME/.config/systemd/user" +mkdir -p "$DEST" + +cp "$SRC/skatespots-ops-health.service" "$DEST/" +cp "$SRC/skatespots-ops-health.timer" "$DEST/" + +chmod +x "$(cd "$(dirname "$0")" && pwd)/ops-health.sh" + +systemctl --user daemon-reload +systemctl --user enable --now skatespots-ops-health.timer +systemctl --user list-timers skatespots-ops-health.timer --no-pager diff --git a/scripts/ops-health.sh b/scripts/ops-health.sh new file mode 100755 index 0000000..291567d --- /dev/null +++ b/scripts/ops-health.sh @@ -0,0 +1,222 @@ +#!/usr/bin/env bash +# +# ops-health.sh — "why isn't it live?" at a glance. +# ───────────────────────────────────────────────────────────────────────────── +# Two silent failures cost real time: the ngit/Amber remote signer stalling +# mid-push, and the deploy timer skipping while origin/main had moved. Both were +# invisible. This reports everything needed to tell them apart, and can notify. +# +# scripts/ops-health.sh # read-only report +# scripts/ops-health.sh --probe-push # ALSO attempt a bounded real push +# scripts/ops-health.sh --notify # desktop-notify on actionable problems +# scripts/ops-health.sh --quiet # print/notify only when something's wrong +# +# `--probe-push` publishes any unpushed commits (that IS the probe — signing only +# happens when there's something to sign). Without it the script never writes to +# the remote. +# +# Exit: 0 healthy · 1 something needs attention · 2 bad usage. +set -uo pipefail + +REPO="/home/sovereignsk8/SkateSpots" +# Overridable so the report can be tested against a fake state dir. +STATE_DIR="${SKATESPOTS_STATE_DIR:-$HOME/.local/state/skatespots-deploy}" +SHA_FILE="$STATE_DIR/last-deployed-sha" +INTERVAL_FILE="$STATE_DIR/last-deploy-epoch" +SKIP_FILE="$STATE_DIR/last-skip" +RUN_FILE="$STATE_DIR/last-run" +NOTIFY_FILE="$STATE_DIR/last-notified" +PENDING_FILE="$STATE_DIR/pending-since" +mkdir -p "$STATE_DIR" + +PROBE_PUSH=0 +NOTIFY=0 +QUIET=0 +for arg in "$@"; do + case "$arg" in + --probe-push) PROBE_PUSH=1 ;; + --notify) NOTIFY=1 ;; + --quiet) QUIET=1 ;; + -h|--help) sed -n '2,20p' "$0"; exit 0 ;; + *) echo "ops-health: unknown option: $arg" >&2; exit 2 ;; + esac +done + +if [ -t 1 ]; then + B=$'\e[1m'; DIM=$'\e[2m'; R=$'\e[31m'; G=$'\e[32m'; Y=$'\e[33m'; N=$'\e[0m' +else + B=; DIM=; R=; G=; Y=; N= +fi + +REPORT="" +p() { REPORT+="$1"$'\n'; } + +problems=() # short, actionable strings (for --notify and the summary) +signature="" # stable dedupe key for notifications + +human_age() { + local s="$1" + if [ "$s" -lt 60 ]; then echo "${s}s" + elif [ "$s" -lt 3600 ]; then echo "$((s / 60))m" + elif [ "$s" -lt 86400 ]; then echo "$((s / 3600))h" + else echo "$((s / 86400))d"; fi +} + +cd "$REPO" 2>/dev/null || { echo "ops-health: repo not found: $REPO" >&2; exit 2; } +now=$(date +%s) + +# ── Repo & push state ──────────────────────────────────────────────────────── +p "${B}▸ Repo & push${N}" +ahead=$(git rev-list --count origin/main..HEAD 2>/dev/null || echo 0) +behind=$(git rev-list --count HEAD..origin/main 2>/dev/null || echo 0) +head_short=$(git rev-parse --short HEAD 2>/dev/null || echo '?') +origin_short=$(git rev-parse --short origin/main 2>/dev/null || echo '?') + +if [ "$ahead" = "0" ] && [ "$behind" = "0" ]; then + p " ${G}✓${N} in sync with origin/main ($head_short)" + rm -f "$PENDING_FILE" +elif [ "$ahead" -gt 0 ]; then + p " ${Y}▲${N} ${B}${ahead}${N} unpushed commit(s) (HEAD $head_short vs origin $origin_short)" + [ ! -f "$PENDING_FILE" ] && date +%s > "$PENDING_FILE" + pend_age=$(( now - $(cat "$PENDING_FILE" 2>/dev/null || echo "$now") )) + p " ${DIM}waiting $(human_age "$pend_age") — push is the missing step${N}" + if [ "$pend_age" -gt 1200 ]; then + problems+=("$ahead commit(s) unpushed for $(human_age "$pend_age") — the signer may be stuck") + signature+="pending-stuck;" + fi +else + p " ${Y}▼${N} ${behind} commit(s) behind origin/main" +fi + +# Optional bounded probe: signing only happens when there IS something to push. +if [ "$PROBE_PUSH" = "1" ] && [ "$ahead" -gt 0 ]; then + p " ${DIM}probing: git push origin main (bounded 45s — this publishes the commits above)…${N}" + start=$(date +%s) + out="$(timeout 45 git push origin main 2>&1)"; rc=$? + el=$(( $(date +%s) - start )) + if [ "$rc" = "0" ]; then + p " ${G}✓${N} push OK in ${el}s — signer responsive" + elif [ "$rc" = "124" ]; then + p " ${R}✗${N} push TIMED OUT after ${el}s — the ngit/Amber signer looks unresponsive" + p " ${DIM}fix: open Amber in the foreground, or: ngit account login --interactive${N}" + problems+=("signer unresponsive (push timed out)") + signature+="signer-timeout;" + else + p " ${R}✗${N} push failed (rc=$rc):" + printf '%s\n' "$out" | tail -3 | while IFS= read -r l; do p " ${DIM}$l${N}"; done + problems+=("push failed (rc=$rc)") + signature+="signer-fail;" + fi +elif [ "$ahead" -gt 0 ]; then + p " ${DIM}run with --probe-push to actually test signing (publishes the commits above)${N}" +fi + +# ── Deploy state ───────────────────────────────────────────────────────────── +p "" +p "${B}▸ Deploy${N}" +deployed="$(cat "$SHA_FILE" 2>/dev/null || echo none)" +origin="$(git rev-parse origin/main 2>/dev/null || echo none)" +last_epoch="$(cat "$INTERVAL_FILE" 2>/dev/null || echo 0)" +case "$last_epoch" in ''|*[!0-9]*) last_epoch=0 ;; esac + +if [ "$deployed" = "$origin" ] && [ "$origin" != "none" ]; then + if [ "$last_epoch" -gt 0 ]; then + p " ${G}✓${N} live is current (${origin:0:9}), deployed $(human_age $((now - last_epoch))) ago" + else + p " ${G}✓${N} live is current (${origin:0:9})" + fi +else + p " ${Y}●${N} ${B}PENDING${N}: origin/main ${origin:0:9} but last deployed ${deployed:0:9}" + if [ -f "$SKIP_FILE" ]; then + IFS='|' read -r sk_at sk_reason sk_sha sk_detail < "$SKIP_FILE" + sk_age=$(( now - ${sk_at:-0} )) + p " last skip: ${B}${sk_reason:-unknown}${N} ($(human_age "$sk_age") ago) ${DIM}${sk_detail}${N}" + case "${sk_reason:-}" in + throttled) + p " ${DIM}→ batching to save Netlify credits; it deploys automatically when the window elapses.${N}" ;; + dirty-tree) + p " ${R}→ tracked files are modified — COMMIT them or the deploy stays skipped.${N}" + problems+=("deploy blocked: working tree is dirty"); signature+="dirty;" ;; + smoke-failed) + p " ${R}→ the built bundle failed the smoke test (real breakage — check the journal).${N}" + problems+=("smoke test failed during deploy"); signature+="smoke;" ;; + diverged) + p " ${R}→ local main diverged from origin/main — a manual merge is needed.${N}" + problems+=("local main diverged from origin/main"); signature+="diverged;" ;; + fetch-failed) + p " ${Y}→ couldn't fetch origin (network/relay issue).${N}" ;; + non-site-only|dist-unchanged) + p " ${DIM}→ no deploy was needed (docs-only or byte-identical build).${N}" ;; + locked|"") + p " ${DIM}→ reason: ${sk_reason:-unknown}${N}" ;; + *) + p " ${DIM}→ reason: ${sk_reason}${N}" ;; + esac + else + p " ${Y}no skip reason recorded — the deploy timer may not be running.${N}" + problems+=("deploy pending with no recorded reason"); signature+="noreason;" + fi +fi + +min_interval="${SKATESPOTS_DEPLOY_MIN_INTERVAL:-86400}" +case "$min_interval" in ''|*[!0-9]*) min_interval=0 ;; esac +if [ "$min_interval" -gt 0 ] && [ "$last_epoch" -gt 0 ]; then + remain=$(( min_interval - (now - last_epoch) )) + if [ "$remain" -gt 0 ]; then + p " ${DIM}throttle: next auto-deploy eligible in ~$(human_age "$remain") · force now: scripts/auto-deploy.sh --force${N}" + fi +fi + +# ── Timer ──────────────────────────────────────────────────────────────────── +p "" +p "${B}▸ Timer${N}" +timer_state="$(systemctl --user is-active skatespots-deploy.timer 2>/dev/null || echo unknown)" +if [ "$timer_state" = "active" ]; then + next_at="$(systemctl --user list-timers skatespots-deploy.timer --no-legend --no-pager 2>/dev/null | awk '{print $1" "$2" "$3}')" + p " ${G}✓${N} skatespots-deploy.timer active${next_at:+ ${DIM}next $next_at${N}}" +else + p " ${R}✗${N} skatespots-deploy.timer: ${timer_state}" + problems+=("deploy timer is not active") + signature+="timer;" +fi +if [ -f "$RUN_FILE" ]; then + run_age=$(( now - $(cat "$RUN_FILE" 2>/dev/null || echo 0) )) + if [ "$run_age" -gt 600 ]; then + p " ${Y}⚠${N} deploy script last ran $(human_age "$run_age") ago (it polls ~every 3m)" + problems+=("deploy script hasn't run in $(human_age "$run_age")") + signature+="stale-run;" + else + p " ${DIM}last run $(human_age "$run_age") ago${N}" + fi +fi + +# ── Summary + optional notification ────────────────────────────────────────── +p "" +if [ ${#problems[@]} -eq 0 ]; then + p "${G}${B}All good.${N}" +else + p "${R}${B}Needs attention:${N}" + for prob in "${problems[@]}"; do p " • $prob"; done +fi + +if [ "$NOTIFY" = "1" ]; then + if [ ${#problems[@]} -gt 0 ]; then + prev="$(cat "$NOTIFY_FILE" 2>/dev/null || true)" + if [ "$signature" != "$prev" ]; then + body="$(printf '%s; ' "${problems[@]}")" + if command -v notify-send >/dev/null 2>&1; then + notify-send -a SkateSpots "SkateSpots ops" "$body" 2>/dev/null || true + fi + printf '%s' "$signature" > "$NOTIFY_FILE" + fi + else + rm -f "$NOTIFY_FILE" # healthy → re-arm notifications + fi +fi + +if [ "$QUIET" = "1" ] && [ ${#problems[@]} -eq 0 ]; then + exit 0 +fi +printf '%s' "$REPORT" + +[ ${#problems[@]} -eq 0 ] && exit 0 || exit 1 diff --git a/scripts/systemd/skatespots-ops-health.service b/scripts/systemd/skatespots-ops-health.service new file mode 100644 index 0000000..edf1256 --- /dev/null +++ b/scripts/systemd/skatespots-ops-health.service @@ -0,0 +1,10 @@ +[Unit] +Description=SkateSpots ops health check (signer + deploy) +Documentation=file:///home/sovereignsk8/SkateSpots/scripts/ops-health.sh + +[Service] +Type=oneshot +WorkingDirectory=/home/sovereignsk8/SkateSpots +# --quiet: print nothing when healthy. --notify: desktop notification, but only +# when the set of problems changes (no repeated spam). +ExecStart=/home/sovereignsk8/SkateSpots/scripts/ops-health.sh --quiet --notify diff --git a/scripts/systemd/skatespots-ops-health.timer b/scripts/systemd/skatespots-ops-health.timer new file mode 100644 index 0000000..4e405e6 --- /dev/null +++ b/scripts/systemd/skatespots-ops-health.timer @@ -0,0 +1,11 @@ +[Unit] +Description=Periodic SkateSpots ops health check + +[Timer] +OnBootSec=10min +OnUnitActiveSec=15min +AccuracySec=1min +Persistent=true + +[Install] +WantedBy=timers.target