feat(backup): scoped weekly archive of the community relay dataset

scripts/archive-relay.mjs snapshots the app's own content off
goskatespots.nostr1.com to gzipped JSONL of the raw signed events, so a
restore can republish them verbatim to any relay. scripts/restore-relay.mjs
re-verifies signatures and publishes to an explicit --to (no default, to
avoid replaying a snapshot onto the live relay). scripts/backup-relay.sh
wraps it and pushes with restic when a repo is configured; a weekly systemd
user timer runs it.

Scoped on purpose: that relay is PUBLIC and carries general traffic (15.7k
kind-0 profiles, 2k kind-30078 app-data from other clients, 408 foreign
NIP-99 listings), so the archive keeps only the app's own kinds/tags plus
contributor profiles, SkateSpots labels, and reports that reference archived
content. ~114 events / 63 KiB.

Not configured yet: offsite (restic.env) and a full restore drill.
This commit is contained in:
SkateSpots 2026-09-28 12:41:50 +07:00
parent 87f7a1f86a
commit c01521ea3f
8 changed files with 456 additions and 0 deletions

View file

@ -623,6 +623,41 @@ Rules:
- `zapstore.yaml` sets `release_source` → local APK, `release_notes` → `release-notes.txt`, `icon` → `public/images/icon-1024.png`. - `zapstore.yaml` sets `release_source` → local APK, `release_notes` → `release-notes.txt`, `icon` → `public/images/icon-1024.png`.
- Dry run (no upload): `SIGN_WITH='<signer>' zsp publish --offline zapstore.yaml`. - Dry run (no upload): `SIGN_WITH='<signer>' zsp publish --offline zapstore.yaml`.
### Relay archive (dataset backup)
- `scripts/archive-relay.mjs` snapshots **the app's own content** off the
community relay to gzipped JSONL of the raw signed events (restorable to any
relay, signatures intact). Default dir `~/.backups/relay` (override with
`SKATESPOTS_ARCHIVE_DIR`), keeps the last 12, and writes `manifest-*.json` +
`latest.json` (counts by kind, author count, sha256, byte size).
- ⚠️ **`goskatespots.nostr1.com` is a PUBLIC relay with general traffic**
(verified 2026-09-28): 15.7k kind-0 profiles, mostly non-skaters; 2k kind-30078
app-data (Armada/Plebs/Amethyst); 379 kind-30023 articles (Yakihonne); 408
NIP-99 listings (Shopstr). So the archive is deliberately **scoped** — exclusive
kinds wholesale (30818/30017/30823/35xxx/…), shared kinds only with the app's
own tags (`t=skateboarding` for events + listings, `t=skatespots` collections,
`t=skatespotsnews` newsletters, `d=skatespots*` app data), `org.skatespots.*`
labels, reports/RSVPs that reference archived content, and kind-0 profiles for
archived authors only. Result: ~114 events / ~63 KiB. **Never archive the whole
relay** — it is strangers' data, and a restore would republish it.
- ⚠️ **Query ONE kind at a time, and stop on "no new events", not on a short
page.** A single filter covering many kinds comes back truncated (once kept
only 3 of 54 spots), and `querySync` can resolve early on a large page (lost
~150 of 3.3k profiles). Paginate with `until` until a page adds nothing.
- `scripts/restore-relay.mjs <archive.jsonl.gz> --dry-run` re-verifies every
signature; `--to wss://relay` republishes. There is **no default target** —
restoring is deliberate, and the mistake to avoid is publishing a snapshot back
to the live relay. No full restore drill has been run yet (needs a throwaway
relay target).
- Offsite: `scripts/backup-relay.sh` runs the archive, then `restic backup` +
`forget --prune` **if** `~/.config/skatespots/restic.env` exists (template:
`scripts/restic.env.example`). **Not configured yet — local copies only**, so
the archive currently survives relay loss but not disk loss.
- Weekly systemd user timer: `scripts/install-archive-timer.sh` copies
`scripts/systemd/skatespots-archive.{service,timer}` into `~/.config/systemd/user`
and enables them. Logs: `journalctl --user -u skatespots-archive`.
- Commits touching only `scripts/`, `*.md`, `cvm-server/`, `skills/`, `.agents/`
are **skipped by the deploy timer** → no Netlify credits spent.
### Git / ngit publishing ### Git / ngit publishing
- Push: `git push origin main` (handled by `git-remote-nostr`; publishes a signed repo-state event). - Push: `git push origin main` (handled by `git-remote-nostr`; publishes a signed repo-state event).
- Login (once): `ngit account login --interactive` → scan the QR in Amber ("Add a new application → Connect app"). Plain `ngit account login` errors; the `--interactive` flag is required. - Login (once): `ngit account login --interactive` → scan the QR in Amber ("Add a new application → Connect app"). Plain `ngit account login` errors; the `--interactive` flag is required.

235
scripts/archive-relay.mjs Executable file
View file

@ -0,0 +1,235 @@
#!/usr/bin/env node
// Archive the SkateSpots dataset off the community relay.
//
// A snapshot written as gzipped JSONL of the RAW signed events, so a restore can
// republish them verbatim (signatures intact) to any relay. The dataset is small,
// so a plain full snapshot is simpler and more trustworthy than incremental sync
// — and it is the whole disaster plan: if the relay (or its host) disappears,
// the community's content can be rebuilt anywhere.
//
// node scripts/archive-relay.mjs [--out DIR] [--relay URL] [--keep N]
//
// ⚠️ goskatespots.nostr1.com is a PUBLIC relay carrying general Nostr traffic
// (verified 2026-09-28): 15.7k kind-0 profiles (mostly non-skaters), 2k kind-30078
// app-data events (Armada/Plebs/Amethyst), 379 long-form articles (Yakihonne),
// 408 NIP-99 listings (Shopstr)… almost none of it ours. Archiving the whole
// relay would mean taking custody of strangers' data AND republishing it on a
// restore. So this stays deliberately scoped to the app's own content:
//
// 1. exclusive kinds the app owns outright (30818 spots, 30017 shops, …);
// 2. shared kinds filtered by the app's OWN tags (e.g. `t=skateboarding`
// for events/listings, `t=skatespots` for collections, `t=skatespotsnews`
// for newsletters, `d=skatespots*` for app data);
// 3. `org.skatespots.*` labels and reports that reference archived content;
// 4. kind-0 profiles for archived authors only.
// DM kinds (4, 13, 1059, 1060) are never touched.
//
// Storage defaults to ~/.backups/relay (override with SKATESPOTS_ARCHIVE_DIR).
// Offsite: scripts/backup-relay.sh wraps this and pushes with restic when
// ~/.config/skatespots/restic.env is present.
//
// Keep the kind lists in sync with src/lib/types.ts / NIP.md when a kind ships.
import { createHash } from 'node:crypto';
import { gzipSync } from 'node:zlib';
import { mkdirSync, writeFileSync, readdirSync, statSync, unlinkSync } from 'node:fs';
import { join } from 'node:path';
import { homedir } from 'node:os';
import { SimplePool, verifyEvent } from 'nostr-tools';
const DEFAULT_RELAY = 'wss://goskatespots.nostr1.com';
const PAGE = 2000; // small pages: a big page can close early and come back short
const MAX_PAGES = 400;
const AUTHOR_BATCH = 150; // kind-0 lookups batched by author
const PROFILE_KIND = 0;
// Kinds nothing else on the relay uses — keep every event of these.
const EXCLUSIVE_KINDS = [
2259, // spot status reports
7103, // event updates
9087, // crew endorsements
24899, // spot chat
30017, // skate shops
30403, // marketplace offers
30818, // skate spots
30823, // tours
31871, // trade attestations
31872, // trade confirmation requests
31873, // trusted-trader recommendations
31969, // crew join requests
32671, // Sasquatch Pro receipts
36229, // crew invites
36387, // collective join requests
...Array.from({ length: 15 }, (_, i) => 35000 + i), // 35000–35014 role/crew profiles
];
const hasTag = (v) => (ev) => ev.tags.some(([n, x]) => n === 't' && x === v);
const dPrefix = (p) => (ev) => (ev.tags.find(([n]) => n === 'd')?.[1] ?? '').startsWith(p);
const labelNamespace = (ev) =>
ev.tags.filter(([n]) => n === 'L').map(([, v]) => v ?? '').find((v) => v.startsWith('org.skatespots.'));
// Shared kinds — keep only events carrying the app's own marker.
const FILTERED_KINDS = [
{ kind: 30003, keep: hasTag('skatespots') }, // spot collections
{ kind: 30023, keep: hasTag('skatespotsnews') }, // newsletter issues
{ kind: 30078, keep: dPrefix('skatespots') }, // app data (read-state, subs, …)
{ kind: 30402, keep: hasTag('skateboarding') }, // marketplace listings
{ kind: 31923, keep: hasTag('skateboarding') }, // calendar events
{ kind: 9041, keep: hasTag('skateboarding') }, // zap goals
{ kind: 4497, keep: hasTag('skatespots.cordn.crew-chat') }, // crew chat envelopes
{ kind: 1985, keep: (ev) => !!labelNamespace(ev) }, // moderation / membership / vouch
];
// Kinds kept only when they reference archived content (matched after the above).
const REFERENCE_KINDS = [1984, 31925]; // NIP-56 reports, NIP-52 RSVPs
// ── args ──────────────────────────────────────────────────────────────────────
function arg(name, fallback) {
const i = process.argv.indexOf(`--${name}`);
return i >= 0 && process.argv[i + 1] ? process.argv[i + 1] : fallback;
}
const RELAY = arg('relay', DEFAULT_RELAY);
const OUT_DIR = arg('out', process.env.SKATESPOTS_ARCHIVE_DIR || join(homedir(), '.backups', 'relay'));
const KEEP = Math.max(1, Number(arg('keep', '12')) || 12);
const stamp = new Date().toISOString().replace(/[:.]/g, '-');
// ── fetch ─────────────────────────────────────────────────────────────────────
// Query ONE kind at a time. A single filter covering many kinds came back
// truncated (the relay caps a shared result window), which silently dropped most
// spots/listings/profiles. Stop only when a page brings nothing new: a short page
// can just mean the subscription closed early, with older events still available.
async function collect(pool, relay, filter, expectedKind) {
const seen = new Map();
let until = Math.floor(Date.now() / 1000) + 120; // tolerate future-dated events
for (let page = 0; page < MAX_PAGES; page++) {
let batch;
try {
batch = await pool.querySync([relay], { ...filter, limit: PAGE, until });
} catch {
break; // keep whatever this filter yielded rather than losing the run
}
if (!batch.length) break;
let oldest = until;
let added = 0;
for (const ev of batch) {
if (expectedKind !== undefined && ev.kind !== expectedKind) continue;
if (seen.has(ev.id)) continue;
if (!verifyEvent(ev)) continue; // never archive an invalid signature
seen.set(ev.id, ev);
added += 1;
if (ev.created_at < oldest) oldest = ev.created_at;
}
if (added === 0) break;
until = oldest - 1;
}
return [...seen.values()];
}
const coordOf = (ev) => {
const d = ev.tags.find(([n]) => n === 'd')?.[1];
return d !== undefined && ev.kind >= 30000 && ev.kind < 40000 ? `${ev.kind}:${ev.pubkey}:${d}` : null;
};
// ── run ───────────────────────────────────────────────────────────────────────
const pool = new SimplePool();
let events;
try {
const byId = new Map();
const add = (evs) => evs.forEach((e) => byId.set(e.id, e));
// 1. Kinds the app owns outright.
for (const kind of EXCLUSIVE_KINDS) add(await collect(pool, RELAY, { kinds: [kind] }, kind));
if (!byId.size) throw new Error(`no events returned by ${RELAY}`);
// 2. Shared kinds, filtered by the app's own tags.
for (const { kind, keep } of FILTERED_KINDS) {
add((await collect(pool, RELAY, { kinds: [kind] }, kind)).filter(keep));
}
// 3. Kinds kept only when they point at what we already have.
const ids = new Set(byId.keys());
const coords = new Set([...byId.values()].map(coordOf).filter(Boolean));
for (const kind of REFERENCE_KINDS) {
add(
(await collect(pool, RELAY, { kinds: [kind] }, kind)).filter((ev) =>
ev.tags.some(([n, v]) => (n === 'e' && ids.has(v)) || (n === 'a' && coords.has(v))),
),
);
}
// 4. Profiles for archived authors only (not the relay's 15k strangers).
const authors = [...new Set([...byId.values()].map((e) => e.pubkey))];
for (let i = 0; i < authors.length; i += AUTHOR_BATCH) {
const batch = authors.slice(i, i + AUTHOR_BATCH);
add(await collect(pool, RELAY, { kinds: [PROFILE_KIND], authors: batch }, PROFILE_KIND));
}
events = [...byId.values()].sort((a, b) => a.created_at - b.created_at);
} finally {
pool.close([RELAY]);
}
// ── write ─────────────────────────────────────────────────────────────────────
const jsonl = events.map((e) => JSON.stringify(e)).join('\n') + '\n';
const body = gzipSync(Buffer.from(jsonl, 'utf8'), { level: 9 });
const sha256 = createHash('sha256').update(body).digest('hex');
const byKind = {};
for (const e of events) byKind[e.kind] = (byKind[e.kind] || 0) + 1;
const file = `relay-${stamp}.jsonl.gz`;
mkdirSync(OUT_DIR, { recursive: true });
writeFileSync(join(OUT_DIR, file), body);
const previous = readdirSync(OUT_DIR)
.filter((f) => f.startsWith('relay-') && f.endsWith('.jsonl.gz') && f !== file)
.sort()
.pop() ?? null;
const manifest = {
relay: RELAY,
generatedAt: new Date().toISOString(),
file,
previous,
eventCount: events.length,
uniqueAuthors: new Set(events.map((e) => e.pubkey)).size,
byKind,
newestCreatedAt: events[events.length - 1].created_at,
oldestCreatedAt: events[0].created_at,
bytes: body.length,
sha256,
scope: {
exclusiveKinds: EXCLUSIVE_KINDS,
filtered: 'shared kinds kept only with the app tag (skateboarding / skatespots / skatespotsnews / d=skatespots*)',
referenceKinds: REFERENCE_KINDS,
profiles: 'kind 0 — authors of archived content only',
},
};
writeFileSync(join(OUT_DIR, `manifest-${stamp}.json`), JSON.stringify(manifest, null, 2) + '\n');
writeFileSync(join(OUT_DIR, 'latest.json'), JSON.stringify(manifest, null, 2) + '\n');
// ── prune ─────────────────────────────────────────────────────────────────────
const archives = readdirSync(OUT_DIR)
.filter((f) => f.startsWith('relay-') && f.endsWith('.jsonl.gz'))
.sort();
for (const old of archives.slice(0, Math.max(0, archives.length - KEEP))) {
unlinkSync(join(OUT_DIR, old));
const m = join(OUT_DIR, `manifest-${old.replace(/^relay-/, '').replace(/\.jsonl\.gz$/, '')}.json`);
try {
if (statSync(m).isFile()) unlinkSync(m);
} catch {
/* manifest already gone */
}
}
const top = Object.entries(byKind)
.sort((a, b) => b[1] - a[1])
.slice(0, 10)
.map(([k, n]) => `${k}:${n}`)
.join(' ');
console.log(
`[archive] ${events.length} events · ${manifest.uniqueAuthors} authors · ${(body.length / 1024).toFixed(0)} KiB · ${file}\n` +
`[archive] by kind — ${top}\n` +
`[archive] sha256 ${sha256.slice(0, 16)}… → ${OUT_DIR}`,
);

45
scripts/backup-relay.sh Executable file
View file

@ -0,0 +1,45 @@
#!/usr/bin/env bash
#
# Weekly SkateSpots relay archive + optional offsite copy.
#
# Driven by the systemd user timer `skatespots-archive.timer` (install with
# scripts/install-archive-timer.sh). Local archives land in
# ~/.backups/relay; if ~/.config/skatespots/restic.env exists, the whole
# directory is pushed to a restic repository (S3/R2/B2/SFTP) and old snapshots
# are pruned. Without that file the archive is local-only — still useful, but
# not a real offsite backup.
#
# Manual run / debug:
# scripts/backup-relay.sh
# journalctl --user -u skatespots-archive -n 50
set -euo pipefail
REPO="/home/sovereignsk8/SkateSpots"
# systemd runs with a minimal PATH; node lives under nvm here.
export PATH="/home/sovereignsk8/.nvm/versions/node/v22.23.1/bin:$PATH"
ARCHIVE_DIR="${SKATESPOTS_ARCHIVE_DIR:-$HOME/.backups/relay}"
RESTIC_ENV="$HOME/.config/skatespots/restic.env"
log() { echo "[$(date -Is)] $*"; }
mkdir -p "$ARCHIVE_DIR"
cd "$REPO"
log "archiving the community relay…"
node scripts/archive-relay.mjs
if [ -f "$RESTIC_ENV" ]; then
# shellcheck disable=SC1090
set -a; . "$RESTIC_ENV"; set +a
if [ -z "${RESTIC_REPOSITORY:-}" ]; then
log "restic.env present but RESTIC_REPOSITORY is empty — skipping offsite copy"
exit 0
fi
log "pushing to restic $RESTIC_REPOSITORY…"
restic backup --tag skatespots-relay "$ARCHIVE_DIR"
restic forget --tag skatespots-relay --keep-last 12 --prune
log "offsite copy done"
else
log "no $RESTIC_ENV — archive kept local-only (set one up for a real offsite backup)"
fi

View file

@ -0,0 +1,16 @@
#!/usr/bin/env bash
#
# Install the weekly relay-archive systemd user timer.
# scripts/install-archive-timer.sh
set -euo pipefail
SRC="$(cd "$(dirname "$0")/systemd" && pwd)"
DEST="$HOME/.config/systemd/user"
mkdir -p "$DEST"
cp "$SRC/skatespots-archive.service" "$DEST/"
cp "$SRC/skatespots-archive.timer" "$DEST/"
systemctl --user daemon-reload
systemctl --user enable --now skatespots-archive.timer
systemctl --user list-timers skatespots-archive.timer --no-pager

View file

@ -0,0 +1,23 @@
# Template for the offsite backup target used by scripts/backup-relay.sh.
# Copy to ~/.config/skatespots/restic.env and fill in — that file is outside the
# repo and must never be committed.
#
# Any restic backend works. Examples:
# Backblaze B2: RESTIC_REPOSITORY=b2:my-bucket:skatespots/relay
# S3 / R2: RESTIC_REPOSITORY=s3:https://<account>.r2.cloudflarestorage.com/<bucket>
# SFTP (host you already run): RESTIC_REPOSITORY=sftp:user@host:/srv/skatespots-backups
#
# One-time init, from this machine:
# set -a; . ~/.config/skatespots/restic.env; set +a
# restic init
#
# Then the weekly timer will keep it in sync (restic backup + forget --prune).
RESTIC_REPOSITORY=
RESTIC_PASSWORD=
# Backend credentials (only the ones your backend needs):
# AWS_ACCESS_KEY_ID=
# AWS_SECRET_ACCESS_KEY=
# B2_ACCOUNT_ID=
# B2_ACCOUNT_KEY=

86
scripts/restore-relay.mjs Executable file
View file

@ -0,0 +1,86 @@
#!/usr/bin/env node
// Restore a SkateSpots archive (from scripts/archive-relay.mjs) to a relay.
//
// Publishes the RAW archived events verbatim — signatures are re-verified first,
// so anything tampered with is dropped. Addressable events are safe to republish:
// the newest `created_at` wins, so replaying a snapshot converges to the same
// state. This is the "prove you can get it back" half of the backup.
//
// node scripts/restore-relay.mjs <relay-*.jsonl.gz> --dry-run
// node scripts/restore-relay.mjs <relay-*.jsonl.gz> --to wss://your.relay
//
// There is deliberately NO default target: restoring is a deliberate act, and
// the common mistake is publishing a snapshot back to the live relay. Use
// --dry-run to preview, then point --to at the relay you actually want.
import { readFileSync } from 'node:fs';
import { gunzipSync } from 'node:zlib';
import { SimplePool, verifyEvent } from 'nostr-tools';
const args = process.argv.slice(2);
const file = args.find((a) => !a.startsWith('--'));
const toIndex = args.indexOf('--to');
const TO = toIndex >= 0 ? args[toIndex + 1] : null;
const DRY = args.includes('--dry-run');
if (!file) {
console.error('usage: node scripts/restore-relay.mjs <archive.jsonl.gz> [--dry-run | --to wss://relay]');
process.exit(2);
}
if (!TO && !DRY) {
console.error('[restore] refusing to publish without an explicit --to <relay> (use --dry-run to preview)');
process.exit(2);
}
const lines = gunzipSync(readFileSync(file)).toString('utf8').split('\n').filter(Boolean);
const events = [];
let invalid = 0;
for (const line of lines) {
try {
const ev = JSON.parse(line);
if (verifyEvent(ev)) events.push(ev);
else invalid += 1;
} catch {
invalid += 1;
}
}
const byKind = {};
for (const e of events) byKind[e.kind] = (byKind[e.kind] || 0) + 1;
console.log(`[restore] ${events.length} valid events (${invalid} invalid/ignored) from ${file}`);
console.log(
'[restore] kinds:',
Object.entries(byKind)
.sort((a, b) => b[1] - a[1])
.map(([k, n]) => `${k}:${n}`)
.join(' '),
);
if (DRY) {
console.log('[restore] dry run — nothing published');
process.exit(0);
}
const pool = new SimplePool();
let ok = 0;
let fail = 0;
let index = 0;
const CONCURRENCY = 10;
async function worker() {
while (index < events.length) {
const ev = events[index++];
try {
const results = await Promise.allSettled(pool.publish([TO], ev));
if (results.some((r) => r.status === 'fulfilled')) ok += 1;
else fail += 1;
} catch {
fail += 1;
}
if ((ok + fail) % 100 === 0) process.stdout.write(`\r[restore] ${ok + fail}/${events.length}…`);
}
}
await Promise.all(Array.from({ length: Math.min(CONCURRENCY, events.length) }, worker));
pool.close([TO]);
process.stdout.write(`\r[restore] done — ${ok} accepted, ${fail} failed \n`);
if (fail) process.exit(1);

View file

@ -0,0 +1,6 @@
[Unit]
Description=Archive the SkateSpots community relay dataset
[Service]
Type=oneshot
ExecStart=%h/SkateSpots/scripts/backup-relay.sh

View file

@ -0,0 +1,10 @@
[Unit]
Description=Weekly SkateSpots relay archive
[Timer]
OnCalendar=weekly
Persistent=true
RandomizedDelaySec=1h
[Install]
WantedBy=timers.target