From 7a7b1c49501f048f451d7a2420a3ad46aa760b85 Mon Sep 17 00:00:00 2001 From: SkateSpots Date: Tue, 27 Oct 2026 16:42:47 +0700 Subject: [PATCH] docs: record Basque tokenizer fix and Cordn UX polish --- AGENTS.md | 52 ++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 44 insertions(+), 8 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e14613d..ca3d76e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1470,6 +1470,23 @@ session with **no other memory of this project**. So: (`?tab=events|chat|mls`; an explicit param wins over the auto-default). This is the cheap "pointer" integration — the two transports are NOT merged (DMs stay NIP-17; the coordinator is only the crew MLS group). + **UX polish (2026-10-27):** after "Ask to join" the panel auto-checks the + coordinator every 10s→30s (silent signers only; Amber stays manual), and stops + when the welcome lands. The empty state has 3 plain steps + "only crew members + can read this"; Cordn/group ref/key package moved under **Advanced**. The + status chip is actionable (Retry), failures show a plain message + details. + A device with no key package gets **"Set up this device"** (`requestJoin` + publishes a fresh package). The member list shows avatars + founder/admin/ + member roles; join requests load on open and show avatars. Messages show + Sending…/Sent/Failed with **Retry** (optimistic pending bubble). Invite opens + a dialog with QR + copy/share and a `?tab=mls` link; on native it uses + `https://skate-spots.org` (not `https://localhost`). The founder banner is + dismissible per crew (`skatespots:cordn-banner-dismissed:/`). + `useCordnNative` adds `retry`, `keyPackageReady`, boolean + `send`/`createChat`/`requestJoin`/`approve`, and **filters welcomes by gid** so + one crew can't adopt another crew's chat. Coordinator announce relays are now + `["wss://relay.skate-spots.org"]` only (removed purplepag.es/nostr.band/nos.lol + timeouts; logs show `local[10002=ok]`). - **Map site-links strip (2026-10-05):** the desktop footer is hidden on the map (full-bleed), so `components/MapFooterStrip.tsx` surfaces the same links (Newsletter/License/Privacy/Collaborators + "not sponsored by OpenSats") as an @@ -2244,6 +2261,15 @@ Rules: 25 codes), regional flag SVGs + English/Portuguese variants, single-world map clamp with horizontal wrap to the Pacific, menu hardening; APK sha256 `846925af…`, versionCode 10701, released with `scripts/release.sh`. + - **1.7.2 (2026-10-27):** Basque translation repaired (Marian tokenizer IDs + + EOS + one-time cache eviction), Cordn private-chat polish (auto-check, + 3-step onboarding, status/retry, new-device setup, member roles, send + states, invite QR/deep link, dismissible banner), coordinator announce + list trimmed to the community relay. APK sha256 `a25ad33c…`, versionCode + 10702. + - **1.7.3 (2026-10-27):** native invite QR uses the public + `https://skate-spots.org` URL instead of Capacitor's `https://localhost`. + APK sha256 `0448a1c7…`, versionCode 10703. ### Relay archive (dataset backup) - **Transfer content between keys (`scripts/transfer-content.mjs`, 2026-10-04):** @@ -2454,11 +2480,20 @@ Rules: - Hosted at `https://relay.skate-spots.org/models/…` (Caddy `handle_path /models/*` → `/srv/models`, CORS `*`; backup `/etc/caddy/Caddyfile.bak-…`). - - **Verified from the host:** en↔es, ca↔gl, hi↔en. The Basque packs are - converted and hosted (ONNX q8). Fixed: the ONNX **IR 9→8** mismatch and the - Marian tokenizer's **target-side** `tokenizer.json`. The decoder now emits - Basque, but it **loops** (EOS/generation config), so Basque is still **not - usable**. Next: the `eos_token_id`/generation loop. + - **Verified from the host:** en↔es, ca↔gl, hi↔en, and **eu↔en after the + 2026-10-27 fix**. The Basque packs are converted and hosted (ONNX q8). + Tested in Node against the same hosted tokenizer/ONNX bytes; outputs match + the original Python model. Two root causes: the ONNX **IR 9→8** mismatch, + and the converted `tokenizer.json` using SentencePiece IDs (special tokens + 0/1/2) instead of the Marian model's `vocab.json` IDs — so `` was never + recognized and generation looped. Rebuilt both tokenizer.json files in + model-vocab order with source-spm scores, special tokens at the model IDs, + and an EOS post-processor; also set Metaspace `add_prefix_space` so the first + word gets `▁`. Backups are on the model host. + - **Cache migration:** `src/lib/translationModelCache.ts` deletes only the old + cached `opus-mt-{eu-en,en-eu}` `tokenizer.json` entries on first Basque use + (the 390 MB ONNX stays cached). Unit-tested; the worker calls it before + loading a Basque translator. - Runtime: **`@xenova/transformers` 2.17.2** + ONNX Runtime **1.14** WASM self-hosted in `public/ort/` (37 MB, copied from `onnxruntime-web`), loaded from `../ort/`; worker `src/workers/translate.worker.ts`. M2M100 is used @@ -2481,9 +2516,10 @@ Rules: - **Conversion:** Docker Python 3.11 + torch 2.4.1 + transformers 4.46.3 + optimum 1.24; dynamic q8; `tokenizer.json` generated from the SentencePiece models (needs protobuf + `sentencepiece_model_pb2.py`). -- **Still to do:** fix the Basque EOS/generation loop, verify all 24 pairs in - the browser/APK, measure speed/memory, and ship **1.7.2** (the phone is on - 1.7.1). No SpotChat. +- **Still to do:** verify all 24 pairs in the browser/APK and measure + speed/memory on low-end phones. **1.7.2 and 1.7.3 are released** (the phone + needs the Zapstore update; the web deploy timer batches for 24h unless forced). + No SpotChat. - **Keyboard caveat:** a malicious OS keyboard or rooted device is outside the app’s control. Warn/recommend a trusted offline keyboard, and avoid the OS keyboard for **secrets** (custom secure input / QR / bunker/Amber,