Backlog: P0 mirror push side DONE (repo live, deploy key, launchd daily job); size math corrected to full-dump 2.58MB/day @180d retention (Option 2)

This commit is contained in:
FuelBoard Contributor
2026-08-15 09:24:33 +01:00
parent 0545659c1b
commit 5b83d9f510
+25 -20
View File
@@ -11,31 +11,36 @@ Status: TODO / IN PROGRESS / DONE / BLOCKED.
## P0 — Next build ## P0 — Next build
- [ ] **GitHub price-history mirror (start now — warm the data for Trends)** — the - [ ] **GitHub price-history mirror (warm the data for Trends)** — the relay does
relay does the GOV.UK heavy lifting (OAuth token stays in relay `.env`) and the GOV.UK heavy lifting (OAuth token stays in relay `.env`) and pushes a daily
pushes a daily snapshot to a PUBLIC repo; the app reads from GitHub, so this snapshot to a PUBLIC repo; the app reads from GitHub, so this doubles as the
doubles as the production data source (App Review can't reach the LAN relay) AND production data source (App Review can't reach the LAN relay) AND the history
the history archive the Trends graph (P1) reads. Shape: relay writes archive the Trends graph (P1) reads. Shape: relay writes `latest.json` (tiny
`latest.json` (tiny pointer: date, station_count, data_updated, available range) pointer: date, station_count, data_updated, available range) +
+ `history/YYYY-MM-DD.json` (full ~200 KB dump) and pushes once/day; the app `history/YYYY-MM-DD.json` (FULL dump — 2.58 MB/day, decision Option 2) and
fetches `raw.githubusercontent.com/aptonline/fuelboard-data/main/latest.json`, pushes once/day; the app fetches
then only the missing day files for starred (station, fuel) pairs (deterministic `raw.githubusercontent.com/aptonline/fuelboard-data/main/latest.json`, then only
URLs — 404 = no data that day, leave a graph gap, never error). LAN relay stays the missing day files for starred (station, fuel) pairs (deterministic URLs —
the fallback source. Telemetry implication: raw GitHub fetches are invisible to 404 = no data that day, leave a graph gap, never error). LAN relay stays the
repo analytics (page views/clones only) — keep the widget-diag beacon firing fallback source. Telemetry implication: raw GitHub fetches are invisible to repo
analytics (page views/clones only) — keep the widget-diag beacon firing
regardless of data source (build/stale-build checks survive) and add an regardless of data source (build/stale-build checks survive) and add an
opportunistic fire-and-forget X-Client app beacon on fetch (LAN = attribution + opportunistic fire-and-forget X-Client app beacon on fetch (LAN = attribution +
cadence; off-LAN skip is itself a reachability datum). `/stats` keeps the cadence; off-LAN skip is itself a reachability datum). `/stats` keeps the
intrusion canary and gains a new signal: app-hit spikes = GitHub path failing → intrusion canary and gains a new signal: app-hit spikes = GitHub path failing →
fallback active. Setup: create the PUBLIC repo `aptonline/fuelboard-data` fallback active. App Review: public repo required (reviewer fetches the data
(gh installed); GitHub PAT (aptonline) lives in relay `.env` only — never in the
repo or IPA. App Review: public repo required (reviewer fetches the data
themselves); HTTPS/ATS fine; plain JSON is not "remote code" (2.5.2-safe); bundle themselves); HTTPS/ATS fine; plain JSON is not "remote code" (2.5.2-safe); bundle
a small fallback dump so the app never demos empty. Growth ~200 KB/day (~73 MB/yr a small fallback dump so the app never demos empty. Retention: 180 days (~470 MB,
— no pruning for years); git history doubles as an audit trail. Optional pruning enforced by the push script — full dump, NOT the earlier ~200 KB
first-weeks tweak: hourly pushes to build history faster, then settle to daily. price-only estimate). *PUSH SIDE DONE 2026-08-15 — repo live
Accumulating TODAY means the Trends graph launches with weeks of real history (https://github.com/aptonline/fuelboard-data); auth = SSH deploy key
instead of an empty chart. ~/.ssh/fuelboard_deploy via alias github-fuelboard-data (no token anywhere);
clone ~/workspace/fuelboard-data; launchd com.apt.fuelboard-mirror daily 10:00
(script ~/.hermes/scripts/mirror_push.py, log
~/Library/Logs/fuelboard-mirror.log); manual trigger `launchctl start
com.apt.fuelboard-mirror`; first snapshot 2026-08-15 landed + verified raw 200.
REMAINING: app-side read (provider chain GitHub → relay → bundled dump) + the
telemetry beacon.*
## P1 — Soon ## P1 — Soon