github/codegraph
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Self-hosted telemetry on Cloudflare D1 + password-gated admin dashboard (CG-7) (#1497)

Browse Source 
* feat(telemetry): D1 schema + migrations for raw events and daily rollups

First step of replacing PostHog with self-hosted telemetry on Cloudflare D1.
Creates the codegraph-telemetry database binding and the initial migration; no
worker code paths change yet (the ingest write path and the nightly rollup cron
land next).

Schema is raw events plus daily rollups: `events` holds one row per sanitized
event with the envelope broken out into columns and event-specific props as
JSON; `daily_machines`, `daily_event_counts` and `daily_dim_counts` are the
nightly rollups the dashboard reads; `machine_first_seen` and `machine_days`
carry the retention cohorts and are never purged. One generic dimension table
covers every bar and pie, so a new breakdown is a cron change rather than a
migration.

The migration is commented as an audit surface, like the rest of this worker —
every column, and which dashboard chart each rollup table serves.

Three judgment calls worth flagging, all documented in the file:

- `events` gets `(day, event)` instead of the separate `(day)` and `(event, day)`
  indexes. D1 bills a row write per index touched, so a third index on the hot
  table costs ~97k writes/day, and `(day, event)` is a covering index for plain
  day-range scans anyway (verified with EXPLAIN QUERY PLAN).
- `daily_event_counts` and `daily_dim_counts` carry a `machines` column, and
  `machine_days` a `prod` flag. The "users by ..." panels and the production-user
  count are distinct-machine numbers, not event counts, and they are
  unrecoverable once raw events are purged.
- No CHECK constraint on `event`: the worker's allowlist is the source of truth
  and the write path is fail-silent, so a rejected INSERT would lose data
  quietly instead of erroring loudly.

Volume note in the migration footer: ~30M row writes/month against the 50M
included on Workers Paid. Storage is the tighter constraint — raw events grow
~74 MB/day, so retention should start at 90 days (~6.7 GB) rather than 180,
which would exceed D1's 10 GB per-database cap.

* feat(telemetry): admin dashboard worker — scaffold + shared-password auth

New Cloudflare Worker at telemetry-dashboard/, sibling of telemetry-worker/ and
bound read-only to the same D1 database. Serves a static frontend plus a JSON
API behind a shared password, on stats.getcodegraph.com.

Auth is the simplest thing that is actually safe for exactly two users: one
password in a secret, compared in constant time over SHA-256 digests, and an
HMAC-signed cookie (HttpOnly; Secure; SameSite=Lax; Path=/) with a one-year
expiry so you sign in once per browser. The cookie is a signed assertion, not a
lookup key — no session store. Its payload carries a fingerprint of the password
it was minted against, so rotating ADMIN_PASSWORD signs everyone out. Login
attempts are capped at 5/min per IP via a ratelimit binding.

Everything is deny-by-default: assets.run_worker_first routes every request
through the worker before the static-asset server sees it, so the dashboard
HTML, its JS, its CSS and the chart library are all behind the session check.
The login page is rendered inline by the worker rather than served from public/,
which leaves no "is this file public?" judgement calls in the asset directory.
Unauthenticated pages 302 to /login, unauthenticated /api/* gets 401. A missing
secret fails closed rather than opening the dashboard.

scripts/smoke-auth.sh is the regression net — 54 assertions against a throwaway
`wrangler dev` covering the gate, cookie flags and persistence, forged/flipped/
truncated cookies, open-redirect refusal, brute-force capping, and password
rotation invalidating live sessions.

Refs CG-11.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore(telemetry-dashboard): simplify the chart-library probe in the shell

Refs CG-11.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(telemetry): nightly rollup cron + raw-event retention purge (CG-10)

Adds a scheduled() handler to the ingest worker that recomputes
daily_event_counts / daily_dim_counts / daily_machines for the just-completed
UTC day plus a 2-day overlap (late-arriving offline buffers), then purges raw
events past the retention window. Rollup writes are idempotent upserts, so a
re-run never double-counts. Also adds an ADMIN_TOKEN-guarded
POST /admin/rollup?day=YYYY-MM-DD for backfill/repair, and drops the PostHog
forwarding path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(telemetry): dashboard charts — SQL API over D1 + the Chart.js views (CG-12, CG-13)

Replaces the scaffold page with the dashboard proper: 19 panels covering every
view of the PostHog dashboard this retires, driven by one filter row.

src/api.ts is the read API CG-12 specified: /api/{meta,summary,timeseries,
breakdown,activation,retention}, all range-scoped, all parameterized against a
closed set of dims and metrics, all shaped labels[] + datasets[] so the frontend
does no arithmetic. Rollups answer everything except the activation funnel,
which needs raw events and says where they start.

The frontend splits into a DOM-free panel registry (public/panels.js) and the
page that mounts it (public/app.js), so the render check can drive the same
registry the browser rendered from. Panels fail alone, refetch dims rather than
flashing, and every chart carries a table twin.

Two numbers are labelled rather than rounded off: range-wide "users" per
dimension is machine-days (the rollups cannot give distinct machines, and
per-day counts are taken as the largest single-event count so one machine's
install + index + usage is not counted three times), and recent activation and
retention cohorts are marked as still-converting instead of drawn as a cliff.

Both colour scales were run through the data-viz validator against the panel
surface, not picked by eye; the results are recorded in public/theme.js.

Verification, all against the committed fixture (12 machines over 10 days, every
expected number worked out by hand from the events, not recorded from a run):
  scripts/smoke-api.sh      98 assertions
  scripts/render-check.mjs  79 assertions — real Chromium over CDP, no new deps
  scripts/smoke-auth.sh     54 assertions (unchanged, still green)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(telemetry): cutover runbook + the end-to-end gate that de-risks it (CG-14)

The account-level steps of the PostHog cutover are the maintainer's to run, so
this lands the runbook they follow and the check that has to pass first.

The runbook (telemetry-worker/README.md) walks the six steps in the order that
keeps them reversible: Workers Paid → migrate → deploy → watch 24h → verify the
first rollup and the dashboard → only then delete POSTHOG_KEY and cancel the
subscription. Step 3 records the outgoing version id because `wrangler rollback`
is the escape hatch for the whole verification window, and that window is
precisely why the PostHog key is deleted last rather than first.

The new gate (scripts/smoke-cutover.sh, `npm run smoke:cutover`) covers the one
seam nothing else did. Both workers declare the same D1 database_id, so pointing
them at a single --persist-to directory runs the real chain: a client batch →
the ingest worker → D1 → the nightly rollup → the dashboard API reading the
numbers back. Every other suite stops at one link — smoke-ingest at the events
table, smoke-rollup at hand-checked SQL, smoke-api at a hand-written fixture
that the cron never touched. That left the dimension names the rollup WRITES
versus the ones the dashboard READS agreeing by convention across two branches,
where a mismatch is silent: no error, no failed request, just a panel reading
zero forever. 61 assertions, all 13 dimensions, and three deliberate traps — a
ci machine that is active but not a production user, usage_rollup counts that
must be summed rather than tallied, and an uninstall's `targets` that must not
leak into the install-scoped breakdown.

Writing it caught that the activation funnel's denominator is first-seen
machines, not install events (deliberate — a reinstall must not re-enter the
funnel), so the suite now pins that distinction rather than assuming it.

Also rewords the last PostHog reference in dashboard code: a comment justifying
the 14-day retention curve by pointing at a dashboard step 6 deletes. The
reasoning now stands on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(telemetry): tell the truth about where events are stored (CG-15)

The telemetry docs are a privacy contract, and they still described a
managed analytics store that no longer receives anything. Replace that
with what actually happens now — events land in our own D1 database on
Cloudflare, the endpoint makes no outbound requests, raw events are
purged after 90 days and only anonymous daily rollups outlive them.
This strengthens the guarantee rather than restating it: there is no
second party to share with.

- TELEMETRY.md: new "Where it is stored" section; the never-collected
  IP bullet no longer leans on a vendor-side setting to hold.
- docs/design/telemetry.md: ingest section rewritten around D1 + the
  nightly rollup/retention cron; volume math redone on Workers Paid and
  the D1 quota (storage, not writes, is what sets the 90-day window);
  new section documenting the dashboard worker and cross-linking it.
- Fixed three drifts from the worker allowlist the sweep surfaced:
  schema_version was still 1, client_name/client_version was still
  marked "plumbing to add" though session.ts passes it today, and the
  legacy sqlite_backend field the worker still accepts was undocumented.
- telemetry-worker/README.md: step 6 claimed a repo-wide grep came back
  clean, which this runbook itself falsifies. Added step 7 — deleting
  the runbook is what makes that grep true, and is the completion check.
- smoke-cutover.sh: the vendor guarantee is now asserted by class
  (no analytics-ingest endpoint referenced) rather than by one vendor's
  name, so it keeps working once the name is gone. Verified it still
  catches a planted forwarding URL. 61/61 pass.

Retention is documented as 90 days, not the 180 in the task notes: 180
days of raw events exceeds D1's 10 GB per-database cap, and the code
purges at 90.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore: untrack local Kommandr issue DB and ignore its sqlite artifacts

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby MchenryandClaude Opus 5 authored and GitHub committed 2026-08-01 16:17:10 -05:00
1 parent f6ac7b36e6
commit 49c11fc2e0
39 files changed
+7683 -88
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

telemetry-worker/.dev.vars.example
+5 -4
View File
@@ -1,4 +1,5 @@
# Copy to .dev.vars for local development (`npm run dev`) and so that
# `wrangler types` includes POSTHOG_KEY in the generated Env.
# The real key lives only in the deployed secret (`wrangler secret put POSTHOG_KEY`).
POSTHOG_KEY="phc_dev_placeholder"
# Copy to .dev.vars for local development (`npm run dev`) if you want to exercise
# POST /admin/rollup — without it that route 404s, which is also how a deploy that
# never ran `wrangler secret put ADMIN_TOKEN` behaves.
# The real token lives only in the deployed secret; nothing here is ever committed.
ADMIN_TOKEN="dev_admin_token_placeholder"
telemetry-worker/.gitignore
+2 -1
View File
@@ -1,5 +1,6 @@
node_modules/
.wrangler/
# local secrets (ADMIN_TOKEN) — see .dev.vars.example
.dev.vars
# generated by `wrangler types` (npm run types) — includes .dev.vars keys
# generated by `wrangler types` (npm run types)
worker-configuration.d.ts
telemetry-worker/README.md
+204 -10
View File
@@ -7,9 +7,12 @@ field, and everything that is never collected) is in
[`docs/design/telemetry.md`](../docs/design/telemetry.md).
What it does, in one breath: validates incoming batches against a strict allowlist (unknown
events dropped, unknown properties stripped), never reads or forwards the client IP,
rate-limits per machine ID, and forwards to PostHog off the response path. It ships nowhere
with the npm package — the engine's `files` allowlist excludes it.
events dropped, unknown properties stripped), never reads or stores the client IP,
rate-limits per machine ID, and writes the survivors to our own D1 database off the response
path. A nightly cron rolls each finished day up into anonymous daily counts and deletes the
raw rows behind it. It makes no outbound requests — nothing is forwarded to a third-party
analytics vendor. It ships nowhere with the npm package — the engine's `files` allowlist
excludes it.
## Endpoint contract
@@ -19,6 +22,81 @@ with the npm package — the engine's `files` allowlist excludes it.
for malformed/oversized/rate-limited requests. Clients treat every response as final —
no retries.
- `GET /` — plain-text pointer to the docs and the off-switches.
- `POST /admin/rollup` — manual rollup trigger, see below. `404` unless `ADMIN_TOKEN` is set.
## Storage (Cloudflare D1)
Telemetry is stored in the `codegraph-telemetry` D1 database on the same account, bound as
`env.DB` — this database is the only place accepted events go. Each request's surviving
events are written in a single `batch()` (one implicit transaction) under `ctx.waitUntil`,
so the write is off the response path. It is deliberately **fail-silent**: a D1 error is
logged to Workers Logs (counts only, never the payload) and the client still gets its `204`,
because clients never retry — losing a datapoint beats losing availability. Alongside the
raw rows, the worker upserts `machine_days` and `machine_first_seen`; when a batch is emptied
by the allowlist, nothing at all is written, so those tables only ever describe stored events.
The complete schema is [`migrations/0001_init.sql`](migrations/0001_init.sql) —
checked in for the same reason this worker's source is public: it is the entire list of what
gets kept, with a comment on every column and on which dashboard chart each rollup table
serves. Shape: raw sanitized `events`, `daily_*` rollups recomputed nightly, and
`machine_days` / `machine_first_seen` for retention cohorts. The dashboard reads rollups; raw
events exist for drill-down and are purged past the retention window.
```bash
npm run db:migrate:local # apply to the local .wrangler state (offline, no account needed)
npm run db:migrate # apply to the remote codegraph-telemetry database
npm run db:migrations # which migrations are applied remotely
npm run db:sql "select count(*) from events"
```
Both applies bootstrap from empty and are a no-op when already current. A schema change is a
new numbered file (`npx wrangler d1 migrations create codegraph-telemetry <name>`) — never an
edit to a migration that has been applied.
Volume, at ~97k accepted POSTs/day: ≈30M D1 row writes/month against the 50M included on
Workers Paid, plus roughly as much again once the purge reaches steady state — a delete bills
like an insert, and at steady state every row written is eventually deleted, so budget ≈48M.
D1 bills a row write per index touched on top of the table row, which is why `events` carries
only two indexes; dropping `events_machine_day` is the first lever if that gets tight. Storage
is the other constraint, and it is what sets the window: raw events grow ≈74 MB/day, so 90 days
lands at ≈6.7 GB against D1's 10 GB per-database cap, while 180 days would exceed it. Full
arithmetic and the remaining levers are in the migration's footer comment.
## Rollups & retention (nightly cron)
`src/rollup.ts` runs on a Cron Trigger at **00:30 UTC** and does two things.
**Rolls up** the day that just ended into `daily_machines`, `daily_event_counts` and
`daily_dim_counts`, then re-runs the two days before it — offline clients ship completed-day
rollups late, so a day keeps growing after it ends. The aggregation is one
`INSERT … SELECT … ON CONFLICT DO UPDATE` per table or dimension, so it happens inside D1 and
no event row crosses the wire. Every write overwrites the recomputed value rather than adding
to it: **re-running a day is a no-op, never a double count.** Two things the SQL is careful
about — a `usage_rollup` row is a counter the client pre-aggregated, so its `count` prop is
summed rather than the rows counted; and `index.languages` / `install.targets` are unnested
with `json_each`, one row per element. Adding a breakdown is a line in `ROLLUP_STATEMENTS`,
never a migration — that is what the generic `(dim, value)` shape buys.
**Purges** raw `events` older than `RETENTION_DAYS` (90, a var in `wrangler.jsonc`) in bounded
`DELETE` batches, and logs one line of counts. `machine_days` and `machine_first_seen` are
never purged — retention cohorts need the full history and they are two orders of magnitude
smaller. Rollups are kept forever, so shortening the window costs ad-hoc drill-back, never a
chart.
Backfill or repair without a redeploy, guarded by the `ADMIN_TOKEN` secret:
```bash
curl -X POST -H "x-admin-token: $ADMIN_TOKEN" \
'https://telemetry.getcodegraph.com/admin/rollup?day=2026-07-27' # one day
curl -X POST -H "x-admin-token: $ADMIN_TOKEN" \
'https://telemetry.getcodegraph.com/admin/rollup?day=2026-07-27&days=14' # the 14 days ending there
```
`&reset=1` drops the day's rollup rows before recomputing, for when the dimension list itself
changed and a value that no longer exists would otherwise linger. It is ignored past the
retention window, where it would delete rows and then find no events to rebuild them from —
the response says which days it refused. Keep manual ranges to a few days at production volume;
each day is a full scan of that day's events, and the request has a wall-clock budget.
## Deploy
@@ -28,20 +106,128 @@ domain route auto-provisions DNS + cert), wrangler ≥ 4.36 (the `ratelimits` bi
```bash
cd telemetry-worker
npm install
npx wrangler login # once
npx wrangler secret put POSTHOG_KEY # the phc_… project write key — never committed
npx wrangler login # once
npm run db:migrate # bring the D1 schema up to date FIRST — the worker writes on deploy
npm run deploy
npx wrangler secret put ADMIN_TOKEN # optional, see below
```
The PostHog project itself must have **"Discard client IP data"** enabled — defense in
depth on top of this worker never forwarding IPs (`$geoip_disable` is also set per event).
The worker holds no API keys — it talks to nothing but its own bound D1 database. The one
secret is `ADMIN_TOKEN`, which enables `POST /admin/rollup`; leave it unset and that route
does not exist. Generate one with `openssl rand -hex 32`, and note that rotating it takes
effect on the next request.
## Cutover from PostHog (one-time)
The replacement of PostHog by this worker's own D1 storage. It is a **hard cutover with no
backfill** — PostHog history is disposable, and the new charts start from an empty database.
**Clients are unaffected at every step:** they keep POSTing to `telemetry.getcodegraph.com`
and every response shape is unchanged, so no client can tell which storage backend is live.
The one-way door is step 6. Everything before it is reversible with `npx wrangler rollback`,
which is why the PostHog key stays put until the new path has proven itself for a day.
**Before you start:** `npm run smoke:cutover`. It runs the whole chain locally — a client
batch through the ingest worker into D1, the nightly rollup over it, then the dashboard
reading the numbers back — and is the only check that covers the seam between the two
workers. They are separate deployments that agree on a list of dimension names by
convention alone, and a mismatch there is silent: no error, no failed request, just a panel
that reads zero forever.
1. **Put the account on Workers Paid (~$5/mo).** Ingest already runs ~97k requests/day
against the free plan's 100k/day cap, so this is overdue independently of D1 — and the
included D1 quota (5 GB storage, 50M row writes/mo) comes with it. The volume arithmetic
is under [Storage](#storage-cloudflare-d1); at ~97k POSTs/day it fits, with the retention
window sized to the 10 GB per-database cap.
2. **Bring the production database up to schema.** `codegraph-telemetry`
(`5ed36dfb-d2d7-4e35-9e63-a1b99d0b1ed3`) already exists on the account and is bound in
`wrangler.jsonc`; this only applies migrations, and is a no-op if it is already current.
```bash
cd telemetry-worker
npm run db:migrate # remote; bootstraps from empty
npm run db:migrations # confirm 0001_init is listed as applied
```
3. **Deploy, and note the version you are leaving.** Print the deployment list first — the
id at the top is your rollback target for the next 24 hours.
```bash
npx wrangler deployments list # record the current version id
npm run deploy
npx wrangler secret put ADMIN_TOKEN # if not already set; enables manual rollups
```
4. **Watch for 24 hours before trusting it.** The number that matters is the daily ingest
rate: it should track the ~95–97k/day PostHog was seeing. A materially lower number means
events are being dropped somewhere between the client and the table — a schema or binding
mistake, not a real change in usage.
```bash
npm run db:sql "select count(*) as rows, max(received_at) as newest from events"
npm run db:sql "select day, count(*) from events group by day order by day desc limit 3"
```
`max(received_at)` should be seconds old at any time of day. Watch Workers Logs
(`npx wrangler tail`) alongside it for a non-zero error rate — the D1 write is deliberately
fail-silent, so a broken write shows up as a log line and a flat row count, never as a
failing request.
**If anything looks wrong, stop here and `npx wrangler rollback [version-id]`.** PostHog is
still live and still holds the key, so rolling back restores the old behaviour completely.
5. **Verify the nightly rollup and the dashboard.** After the first 00:30 UTC cron has run,
the completed day must be present in the rollup tables — the dashboard reads those, not raw
events, so an empty rollup is an empty dashboard even with ingest working perfectly.
```bash
npm run db:sql "select day, machines, prod_machines from daily_machines order by day desc limit 3"
npm run db:sql "select day, event, count from daily_event_counts order by day desc limit 10"
```
Then open the dashboard (`stats.getcodegraph.com`, see
[`../telemetry-dashboard/README.md`](../telemetry-dashboard/README.md)) and confirm the
panels render live numbers rather than empty states. If the cron did not fire, roll the day
up by hand with `POST /admin/rollup?day=…` above rather than waiting another 24 hours.
6. **Only now, retire PostHog.** Past this point the previous worker version can still be
rolled back, but it will have no key to forward with — this is the step that makes the
cutover final.
```bash
npx wrangler secret delete POSTHOG_KEY # the last vendor credential on the account
npx wrangler secret list # confirm ADMIN_TOKEN is the only secret left
```
Then cancel the subscription and delete the project.
7. **Delete this section.** The forwarding code and the `POSTHOG_HOST` var left the repo with
the D1 rewrite, and `npm run smoke:cutover` asserts on every run that the worker's source
and config reference no analytics vendor and make no outbound request at all. This runbook
is the last place the old vendor is named anywhere in the repository, so once step 6 is
done:
```bash
grep -ri posthog . --exclude-dir=node_modules --exclude-dir=.git
```
returning nothing is the check that the cutover is complete — and deleting these steps is
what makes it pass. Keep them until then: every step above is reversible, and a rollback
is useless if its instructions have already been deleted.
## Local dev & checks
```bash
cp .dev.vars.example .dev.vars # placeholder key; also feeds `wrangler types`
npm run check # wrangler types + tsc --noEmit + deploy --dry-run
npm run dev # http://localhost:8787
npm run check # wrangler types + tsc --noEmit + deploy --dry-run
npm run db:migrate:local # once, so `wrangler dev` has tables to write to
npm run dev # http://localhost:8787 (local D1 in .wrangler/)
npm run smoke # end-to-end: boots `wrangler dev`, POSTs, asserts stored rows
npm run smoke:rollup # end-to-end: seeds synthetic days, rolls them up, purges,
# asserts every number against hand-computed values
npm run smoke:cutover # the whole chain: a client batch → D1 → rollup → the dashboard
# API reads it back. Boots BOTH workers against one shared local
# D1, so it is the only check that covers the seam between them.
curl -i localhost:8787/v1/events -H 'content-type: application/json' -d '{
"machine_id": "00000000-0000-4000-8000-000000000000",
@@ -51,8 +237,16 @@ curl -i localhost:8787/v1/events -H 'content-type: application/json' -d '{
"props": { "kind": "mcp_tool", "name": "codegraph_explore",
"count": 12, "error_count": 0, "client_name": "Claude Code" } }]
}'
npx wrangler d1 execute codegraph-telemetry --local \
--command "select day, event, machine_id, props from events order by id desc limit 5"
```
To drive the cron body by hand, run `wrangler dev --test-scheduled` and hit
`localhost:8787/__scheduled?cron=30+0+*+*+*`. For `POST /admin/rollup` locally, copy
`.dev.vars.example` to `.dev.vars` — without an `ADMIN_TOKEN` the route 404s, exactly as a
deploy that never set the secret does.
## Changing the schema
The allowlist in `src/index.ts` mirrors `docs/design/telemetry.md` (and the user-facing
telemetry-worker/migrations/0001_init.sql
+205
View File
@@ -0,0 +1,205 @@
-- codegraph telemetry — initial schema (Cloudflare D1)
--
-- This file is public on purpose, like the rest of telemetry-worker/: it is the
-- complete list of everything codegraph's anonymous telemetry stores. If a column
-- is not here, it is not kept. The field-by-field contract it implements lives in
-- docs/design/telemetry.md (and, user-facing, in TELEMETRY.md).
--
-- Nothing in this database identifies a person or a codebase. No IP addresses (the
-- ingest worker never reads them), no file paths, no repo, file, or symbol names, no
-- query strings. `machine_id` is a random UUIDv4 the client mints locally and the user
-- can delete at any time (`codegraph telemetry off`, or remove ~/.codegraph/telemetry.json).
--
-- Shape: raw events + daily rollups.
-- * The ingest worker (`src/index.ts`) writes ONLY to `events`, `machine_days` and
-- `machine_first_seen`, off the response path.
-- * The nightly cron recomputes the `daily_*` rollups from `events` with idempotent
-- upserts, then purges raw `events` past the retention window.
-- * The admin dashboard reads rollups first and falls back to `events` only for the
-- activation funnel and ad-hoc drill-down (both bounded by the retention window).
--
-- Apply: npm run db:migrate:local (local .wrangler state)
-- npm run db:migrate (remote codegraph-telemetry)
-- ---------------------------------------------------------------------------
-- Raw events
-- ---------------------------------------------------------------------------
-- One row per sanitized event accepted by POST /v1/events. Everything here has
-- already passed the worker's allowlist: unknown events dropped, unknown props
-- stripped, strings length- and charset-checked, timestamps clamped.
--
-- Deliberately NO `CHECK (event IN (...))` constraint: the worker's EVENTS allowlist
-- is the single source of truth, and the write path is fail-silent by design (a
-- rejected INSERT would lose data quietly rather than error visibly). Same reasoning
-- for `json_valid(props)` — the worker constructs that JSON itself.
CREATE TABLE events (
-- rowid alias, no AUTOINCREMENT: ids are never referenced anywhere, and the
-- retention purge only ever deletes the OLDEST rows, so max(id) never drops and
-- ids stay monotonic in practice. Gives the purge a cheap keyset batch:
-- DELETE FROM events WHERE id IN (SELECT id FROM events WHERE day < ? LIMIT 5000)
id INTEGER PRIMARY KEY,
received_at TEXT NOT NULL, -- ISO 8601 UTC, worker clock, always present
ts TEXT, -- ISO 8601 UTC client timestamp, already clamped
-- by the worker (>10min future / >30d past rejected);
-- NULL when the client sent none. For usage_rollup
-- the client sets it to <rollup day>T12:00:00Z, so it
-- attributes counters to the day they happened on.
day TEXT NOT NULL, -- UTC YYYY-MM-DD from substr(ts, 1, 10), else received_at.
-- Every rollup and every chart is keyed on this.
event TEXT NOT NULL, -- install | index | usage_rollup | uninstall
machine_id TEXT NOT NULL, -- random UUIDv4, client-minted (never fingerprinted)
-- Envelope, identical for every event in a batch. All nullable: the worker's
-- sanitizer strips anything malformed rather than rejecting the batch, so an old
-- or odd client shows up as NULLs instead of vanishing.
codegraph_version TEXT,
os TEXT, -- process.platform: darwin | linux | win32 | …
arch TEXT, -- process.arch: arm64 | x64 | …
node_major INTEGER,
ci INTEGER, -- 0/1 from the client's `ci` boolean; NULL if absent.
-- "Production users" = everything except ci = 1
-- (NULL counts as production — see machine_days.prod).
schema_version INTEGER,
props TEXT NOT NULL DEFAULT '{}' -- JSON object of the sanitized event-specific props
);
-- (day, event) subsumes a plain (day) index — SQLite uses the leading-column prefix —
-- so this pair covers day-range scans, per-event day-range scans AND the retention
-- purge with one fewer index than listing them separately. That matters: D1 bills an
-- extra row write per index touched, so every index on this table costs ~97k
-- writes/day. Do not add a third without re-checking the volume note below.
CREATE INDEX events_day_event ON events (day, event);
-- Ad-hoc per-machine drill-down and the activation funnel (install → first index).
CREATE INDEX events_machine_day ON events (machine_id, day);
-- ---------------------------------------------------------------------------
-- Rollups — written by the nightly cron, read by the dashboard
-- ---------------------------------------------------------------------------
-- Rollups are kept FOREVER (they are tiny); raw `events` are purged. So any number a
-- chart needs long-term has to be recoverable from these tables alone — that is why
-- the distinct-machine columns exist alongside the event counts.
-- Daily unique machines.
-- Serves: "Daily Production Users" line; the machine denominator on daily panels.
-- `prod_machines` excludes ci = 1 (CI runners), matching the dashboard's
-- "Production Users" framing. NOTE: these are per-day distinct counts and CANNOT be
-- summed across a range — a range-wide distinct count comes from `machine_days`.
CREATE TABLE daily_machines (
day TEXT PRIMARY KEY,
machines INTEGER NOT NULL DEFAULT 0,
prod_machines INTEGER NOT NULL DEFAULT 0
);
-- Daily event volume per event type.
-- Serves: "Install" / "Uninstall" big numbers; "Installs vs uninstalls over time";
-- "New installs (daily)"; the runs series of "Daily indexing activity".
-- `count` is a row count for install/index/uninstall, but for usage_rollup it is the
-- SUM of the events' `count` prop (the client pre-aggregates locally, so one row can
-- represent hundreds of tool calls). `machines` is the distinct machines that emitted
-- that event that day — the "active users" series of "Daily indexing activity", which
-- is unrecoverable once the raw rows are purged.
CREATE TABLE daily_event_counts (
day TEXT NOT NULL,
event TEXT NOT NULL,
count INTEGER NOT NULL DEFAULT 0,
machines INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (day, event)
) WITHOUT ROWID;
-- One generic (dimension, value) table behind every bar and pie on the dashboard,
-- so a new breakdown is a cron change, never a migration.
-- Serves, by `dim`:
-- os → "Users by operating system" (pie)
-- arch → arch mix
-- codegraph_version → "Users by app version" (bar)
-- node_major → Node version mix
-- language → "Most-indexed programming languages" (bar; unnested from index.languages)
-- file_count_bucket → "Codebase size (files per project)" (bar)
-- duration_bucket → "Session run length" (pie) and "Indexing speed" (bar),
-- plus "indexing duration buckets over time" (stacked line)
-- target → "AI Agent Targets" (bar; unnested from install.targets / uninstall.targets)
-- scope → install local vs global
-- kind → install fresh / upgrade / reinstall
-- name → usage by MCP tool / CLI command (incl. prompt-hook-gate-* outcomes)
-- client_name → usage by agent (Claude Code, Cursor, …), from MCP clientInfo
-- `event` is kept in the key so the same dim can be sliced per event type (e.g. os for
-- install vs os for index). `count` is event volume (SUM of the usage_rollup `count`
-- prop where applicable); `machines` is distinct machines — the honest number for the
-- "users by …" panels, which are machine counts, not event counts.
CREATE TABLE daily_dim_counts (
day TEXT NOT NULL,
event TEXT NOT NULL,
dim TEXT NOT NULL,
value TEXT NOT NULL,
count INTEGER NOT NULL DEFAULT 0,
machines INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (day, event, dim, value)
) WITHOUT ROWID;
-- Cross-event slices of one dimension over a date range ("languages, all events, last 30d").
CREATE INDEX daily_dim_counts_dim_day ON daily_dim_counts (dim, day);
-- First day a machine was ever seen.
-- Serves: "New installs over time"; the denominator of the install → first-use
-- activation funnel; the cohort key for retention.
-- Written by the ingest worker on every batch (upsert keeps the MINIMUM day, so a
-- late-arriving offline buffer can move a machine's first day earlier but never later).
CREATE TABLE machine_first_seen (
machine_id TEXT PRIMARY KEY,
first_day TEXT NOT NULL
);
-- Cohort scans: "machines first seen between X and Y".
CREATE INDEX machine_first_seen_day ON machine_first_seen (first_day);
-- Machine × day activity matrix — the only table that can answer "distinct machines
-- over a RANGE" (daily rollups can't: summing them double-counts returning machines).
-- Serves: "Daily retention cohorts" (day 0–14 curve, joined to machine_first_seen);
-- the "Production Users" big number over the picker's range;
-- active-machine lines beyond the raw-event retention window.
-- `prod` is 0 only if EVERY event that machine sent that day carried ci = 1; a missing
-- `ci` counts as production. Kept per (machine, day) rather than as a per-machine flag
-- because the same install can run inside and outside CI on different days.
-- ~10k rows/day at current volume — WITHOUT ROWID keeps it compact (the PK is the table).
-- NOT purged by the retention job: retention cohorts need the full history.
CREATE TABLE machine_days (
machine_id TEXT NOT NULL,
day TEXT NOT NULL,
prod INTEGER NOT NULL DEFAULT 1,
PRIMARY KEY (machine_id, day)
) WITHOUT ROWID;
-- Day-keyed scans ("distinct machines active in this range").
CREATE INDEX machine_days_day ON machine_days (day);
-- ---------------------------------------------------------------------------
-- Volume & storage sanity check (Workers Paid, limits as of 2026-07)
-- ---------------------------------------------------------------------------
-- Included per month: 50M rows written, 25B rows read, 5 GB storage
-- (then $0.75/GB-mo). Hard cap: 10 GB per database.
--
-- Current ingest is ~97k accepted POSTs/day. D1 counts one row write PER INDEX
-- touched in addition to the table row, so with ~2 events per request:
--
-- events 97k × 2 × (1 table + 2 indexes) ≈ 0.6M writes/day
-- machine_days 97k × (1 table + 1 index) ≈ 0.2M writes/day
-- first_seen 97k × (1 table + 1 index) ≈ 0.2M writes/day
-- rollup cron ~1k rows/day negligible
-- ─────────────────
-- ≈ 1.0M writes/day ≈ 30M/month
--
-- Comfortably inside the 50M included, with ~1.6× headroom. (The epic's "~10M/month"
-- estimate predates counting index writes; the arithmetic above is the one to trust.)
-- Reads are trivial: the dashboard hits rollups, ~thousands of rows per page load.
--
-- STORAGE is the tighter constraint, and it decides the retention window. A raw event
-- row is ~250 B plus ~130 B of index entries, so ~74 MB/day:
--
-- 90-day retention ≈ 6.7 GB under the 10 GB cap, ~$1.30/mo over the 5 GB included
-- 180-day retention ≈ 13 GB EXCEEDS the 10 GB per-database cap
--
-- So the retention job should start at 90 days, not 180 — and the real row size must be
-- measured after cutover (`SELECT count(*), sum(length(props)) FROM events`) before
-- widening it. Rollups are kept forever regardless, so shortening the raw window costs
-- ad-hoc drill-back, never a chart. If writes or storage ever get tight, the levers, in
-- order: drop events_machine_day (drill-down only), move the machine_first_seen upsert
-- off the hot path into the nightly cron, then store timestamps as INTEGER epoch ms.
telemetry-worker/package.json
+8 -1
View File
@@ -6,7 +6,14 @@
"dev": "wrangler dev",
"deploy": "wrangler deploy",
"types": "wrangler types",
"check": "wrangler types && tsc --noEmit && wrangler deploy --dry-run"
"check": "wrangler types && tsc --noEmit && wrangler deploy --dry-run",
"smoke": "./scripts/smoke-ingest.sh",
"smoke:rollup": "./scripts/smoke-rollup.sh",
"smoke:cutover": "./scripts/smoke-cutover.sh",
"db:migrate:local": "wrangler d1 migrations apply codegraph-telemetry --local",
"db:migrate": "wrangler d1 migrations apply codegraph-telemetry --remote",
"db:migrations": "wrangler d1 migrations list codegraph-telemetry --remote",
"db:sql": "wrangler d1 execute codegraph-telemetry --remote --command"
},
"devDependencies": {
"typescript": "^5.0.0",
telemetry-worker/scripts/smoke-cutover.sh
Executable
+270
View File
@@ -0,0 +1,270 @@
#!/usr/bin/env bash
# The cutover gate (CG-14): drives the WHOLE chain the way production will run it —
# a client POSTs a batch, the ingest worker writes D1, the nightly rollup aggregates,
# and the dashboard reads the numbers back out.
#
# Every other suite tests one link. smoke-ingest.sh stops at the `events` table,
# smoke-rollup.sh hand-checks the rollup SQL, and smoke-api.sh reads a fixture that
# was written by hand rather than by the cron. That leaves exactly the seam this
# cutover turns on unverified: the dimension names the rollup WRITES versus the ones
# the dashboard READS. Those two lists live in different workers on different
# branches, and a mismatch is silent — no error, no failed request, just a panel that
# renders zero forever. Catching that after cutover means a day of lost telemetry;
# catching it here costs a minute.
#
# Both workers declare the same D1 `database_id`, so pointing them at one
# `--persist-to` directory gives them literally the same local SQLite file. The state
# is a fresh mktemp each run, so every expected number below is exact rather than a
# lower bound.
#
# npm run smoke:cutover
#
# Expected numbers are derived from THE_BATCH below and nothing else; see the table
# in that comment block.
set -uo pipefail
cd "$(dirname "$0")/.."
WORKER_DIR="$PWD"
DASH_DIR="$(cd .. && pwd)/telemetry-dashboard"
[ -d "$DASH_DIR" ] || { echo "cannot find telemetry-dashboard/ next to telemetry-worker/"; exit 1; }
INGEST_PORT="${CUTOVER_INGEST_PORT:-8795}"
DASH_PORT="${CUTOVER_DASH_PORT:-8796}"
INGEST="http://127.0.0.1:$INGEST_PORT"
DASH="http://127.0.0.1:$DASH_PORT"
# Test-only credentials. The point is to exercise the wiring, not to keep a secret.
ADMIN_TOKEN=cutover-admin-token
DASH_PASSWORD=cutover-dashboard-password
SESSION_SECRET=cutover-session-secret
STATE="$(mktemp -d -t cg-cutover-state)"
JAR="$(mktemp -t cg-cutover-jar)"
ILOG=/tmp/cg-cutover-ingest.log
DLOG=/tmp/cg-cutover-dash.log
pass=0; fail=0
ok() { pass=$((pass + 1)); printf ' ok %s\n' "$1"; }
bad() { fail=$((fail + 1)); printf ' FAIL %s — expected %s, got %s\n' "$1" "$2" "$3"; }
is() { [ "$2" = "$3" ] && ok "$1" || bad "$1" "$2" "$3"; }
DEV_PID=""
stop_dev() {
[ -n "$DEV_PID" ] || return 0
kill "$DEV_PID" 2>/dev/null
wait "$DEV_PID" 2>/dev/null
DEV_PID=""
}
cleanup() { stop_dev; rm -rf "$STATE" "$JAR"; }
trap cleanup EXIT
# Boot a worker in <dir> on <port> against the SHARED state, wait for <readyurl>.
boot() { # boot <dir> <port> <readyurl> <log> [extra wrangler args...]
local dir="$1" port="$2" ready="$3" log="$4"; shift 4
( cd "$dir" && exec npx wrangler dev --port "$port" --ip 127.0.0.1 \
--persist-to "$STATE" "$@" ) >"$log" 2>&1 &
DEV_PID=$!
for _ in $(seq 1 90); do
curl -sf -o /dev/null "$ready" && return 0
kill -0 "$DEV_PID" 2>/dev/null || break
sleep 1
done
echo "worker in $dir never came up on :$port — log follows"; cat "$log"; exit 1
}
# Resolve a dotted path through a JSON document. Numeric segments index arrays.
jget() {
node -e '
let v = JSON.parse(process.argv[1]);
for (const k of process.argv[2].split(".")) v = v?.[k];
console.log(v === undefined ? "<missing>" : typeof v === "object" && v !== null ? JSON.stringify(v) : String(v));
' "$1" "$2"
}
day_ago() { node -e 'console.log(new Date(Date.now()-process.argv[1]*864e5).toISOString().slice(0,10))' "$1"; }
# Inside the ingest clamp window (30 days) and outside the cron's 3-day lookback.
DAY="$(day_ago 5)"
RANGE="from=$DAY&to=$DAY"
# ---------------------------------------------------------------------------
# THE_BATCH — three machines, one day. Everything asserted below follows from here.
#
# machine os arch node version ci events
# m1 darwin arm64 22 1.5.0 false install(local/fresh, [claude,cursor])
# index([typescript,python], 100-1k, 10-60s)
# usage_rollup(codegraph_explore x12, Claude Code)
# m2 linux x64 20 1.5.0 false install(global/upgrade, [codex])
# index([typescript], 1k-10k, 1-5m)
# usage_rollup(codegraph_explore x8, Codex CLI)
# m3 linux arm64 22 1.4.1 TRUE index([go], <100, <10s)
# uninstall([claude])
#
# The three deliberate traps:
# * m3 is ci=true, so it counts as active but NOT as a production user.
# * tool_calls must SUM the `count` prop (12 + 8 = 20), not count the 2 rows.
# * m3's uninstall carries targets=[claude], so a `target` breakdown that forgets
# to scope by event would report claude twice.
# ---------------------------------------------------------------------------
M1=11111111-1111-4111-8111-111111111111
M2=22222222-2222-4222-8222-222222222222
M3=33333333-3333-4333-8333-333333333333
post_batch() { # post_batch <json>
curl -s -o /dev/null -w '%{http_code}' -X POST "$INGEST/v1/events" \
-H 'content-type: application/json' --data-binary "$1"
}
batch() { # batch <machine> <os> <arch> <node> <version> <ci> <events-json>
node -e '
const [m, os, arch, node_major, v, ci, events, day] = process.argv.slice(1);
process.stdout.write(JSON.stringify({
machine_id: m, codegraph_version: v, os, arch,
node_major: Number(node_major), ci: ci === "true", schema_version: 1,
events: JSON.parse(events).map((e) => ({ ...e, ts: `${day}T12:00:00Z` })),
}));
' "$@" "$DAY"
}
# ---------------------------------------------------------------------------
echo "cutover chain: client → ingest worker → D1 → rollup → dashboard"
echo
echo "migrating the shared local D1 state"
( cd "$WORKER_DIR" && npx wrangler d1 migrations apply codegraph-telemetry \
--local --persist-to "$STATE" ) >/tmp/cg-cutover-migrate.log 2>&1 ||
{ echo "migration failed:"; cat /tmp/cg-cutover-migrate.log; exit 1; }
echo "booting the ingest worker on :$INGEST_PORT"
boot "$WORKER_DIR" "$INGEST_PORT" "$INGEST/" "$ILOG" --var "ADMIN_TOKEN:$ADMIN_TOKEN"
echo
echo "ingest accepts the batch"
is "m1 batch → 204" 204 "$(post_batch "$(batch "$M1" darwin arm64 22 1.5.0 false '[
{"event":"install","props":{"scope":"local","kind":"fresh","targets":["claude","cursor"]}},
{"event":"index","props":{"languages":["typescript","python"],"file_count_bucket":"100-1k","duration_bucket":"10-60s"}},
{"event":"usage_rollup","props":{"kind":"mcp_tool","name":"codegraph_explore","count":12,"client_name":"Claude Code"}}
]')")"
is "m2 batch → 204" 204 "$(post_batch "$(batch "$M2" linux x64 20 1.5.0 false '[
{"event":"install","props":{"scope":"global","kind":"upgrade","targets":["codex"]}},
{"event":"index","props":{"languages":["typescript"],"file_count_bucket":"1k-10k","duration_bucket":"1-5m"}},
{"event":"usage_rollup","props":{"kind":"mcp_tool","name":"codegraph_explore","count":8,"client_name":"Codex CLI"}}
]')")"
is "m3 (ci) batch → 204" 204 "$(post_batch "$(batch "$M3" linux arm64 22 1.4.1 true '[
{"event":"index","props":{"languages":["go"],"file_count_bucket":"<100","duration_bucket":"<10s"}},
{"event":"uninstall","props":{"targets":["claude"]}}
]')")"
sleep 2 # let the ctx.waitUntil writes drain before rolling up
echo
echo "the nightly rollup aggregates the day"
ROLL=$(curl -s -X POST -H "x-admin-token: $ADMIN_TOKEN" "$INGEST/admin/rollup?day=$DAY")
is "POST /admin/rollup → ok" true "$(jget "$ROLL" ok)"
is "rollup wrote rows" true "$(node -e 'process.stdout.write(String((JSON.parse(process.argv[1]).rows ?? 0) > 0))' "$ROLL")"
stop_dev # free the D1 lock before the dashboard opens the same file
echo
echo "booting the dashboard on :$DASH_PORT against the same D1"
( cd "$DASH_DIR" && npm run --silent vendor ) >/dev/null 2>&1
boot "$DASH_DIR" "$DASH_PORT" "$DASH/robots.txt" "$DLOG" \
--var "ADMIN_PASSWORD:$DASH_PASSWORD" --var "SESSION_SECRET:$SESSION_SECRET"
curl -s -o /dev/null -c "$JAR" -X POST "$DASH/login" --data-urlencode "password=$DASH_PASSWORD"
api() { curl -s -b "$JAR" "$DASH/api/$1"; }
is "dashboard session established" 200 "$(curl -s -o /dev/null -w '%{http_code}' -b "$JAR" "$DASH/api/health")"
# --- the big numbers -------------------------------------------------------
echo
echo "summary panel reads back what was ingested"
S=$(api "summary?$RANGE")
is "production users (ci machine excluded)" 2 "$(jget "$S" production_users)"
is "active machines" 3 "$(jget "$S" active_machines)"
is "new machines" 3 "$(jget "$S" new_machines)"
is "installs" 2 "$(jget "$S" installs)"
is "uninstalls" 1 "$(jget "$S" uninstalls)"
is "indexing runs" 3 "$(jget "$S" index_runs)"
is "tool calls SUM the count prop (12+8)" 20 "$(jget "$S" tool_calls)"
# --- every dimension the dashboard offers ----------------------------------
# This is the actual point of the suite: each of these is a distinct string that
# must match between rollup.ts and api.ts's DIMS registry. An empty `labels` means
# the dashboard is asking for a dimension the cron never writes.
echo
echo "every breakdown dimension resolves against the cron's output"
bd() { # bd <desc> <query> <expected-labels-json> <expected-data-json>
local body; body=$(api "breakdown?$RANGE&$2")
is "$1 — labels" "$3" "$(jget "$body" labels)"
is "$1 — data" "$4" "$(jget "$body" datasets.0.data)"
}
bd "os" "dim=os" '["linux","darwin"]' '[2,1]'
bd "arch" "dim=arch" '["arm64","x64"]' '[2,1]'
bd "version" "dim=codegraph_version" '["1.5.0","1.4.1"]' '[2,1]'
bd "node major" "dim=node_major" '["22","20"]' '[2,1]'
bd "language" "dim=language" '["typescript","go","python"]' '[2,1,1]'
bd "files in project" "dim=file_count_bucket" '["<100","100-1k","1k-10k","10k+"]' '[1,1,1,0]'
bd "run length" "dim=duration_bucket" '["<10s","10-60s","1-5m","5m+"]' '[1,1,1,0]'
bd "install scope" "dim=scope" '["global","local"]' '[1,1]'
bd "install kind" "dim=kind" '["fresh","upgrade"]' '[1,1]'
bd "tool name" "dim=name" '["codegraph_explore"]' '[20]'
bd "agent" "dim=client_name" '["Claude Code","Codex CLI"]' '[12,8]'
# The trap: `target` defaults to event=install, so the uninstall's own claude target
# must NOT be folded in — and must still be reachable by asking for it explicitly.
bd "agent target (install-scoped)" "dim=target" '["claude","codex","cursor"]' '[1,1,1]'
bd "agent target (uninstall)" "dim=target&event=uninstall" '["claude"]' '[1]'
# --- the remaining panels --------------------------------------------------
echo
echo "the timeseries and funnel panels see the day"
# Every entry in api.ts's SERIES registry — each one reads a different rollup table,
# so this is the second half of the write-vs-read seam the breakdowns cover above.
ts() { # ts <desc> <metric> <series-0> [series-1]
local body; body=$(api "timeseries?$RANGE&metric=$2")
is "$1 — day" "[\"$DAY\"]" "$(jget "$body" labels)"
is "$1 — series" "$3" "$(jget "$body" datasets.0.data)"
[ $# -ge 4 ] && is "$1 — second series" "$4" "$(jget "$body" datasets.1.data)"
}
ts "installs and uninstalls" installs_uninstalls '[2]' '[1]'
ts "new installs" new_installs '[3]'
ts "production users" production_users '[2]'
ts "indexing activity" indexing_activity '[3]' '[3]'
ts "tool calls (sums the prop)" tool_calls '[20]' '[2]'
MET=$(api "meta")
is "meta anchors on the rolled-up day" "$DAY" "$(jget "$MET" latest_day)"
is "meta reports the rollup ran" "$DAY" "$(jget "$MET" latest_rollup_day)"
# The funnel is the one panel that reads RAW events rather than a rollup, so it is
# also the one the retention purge can blind — worth pinning that it works today.
#
# Its denominator is FIRST-SEEN MACHINES, not `install` events (api.ts: "a machine
# that reinstalls does not re-enter the funnel"). m3 is the discriminator: it never
# sent an install event, but it is new and it indexed, so it belongs in both legs.
# Reading 2 here would mean the funnel had quietly become an install-event ratio.
ACT=$(api "activation?$RANGE&window=1")
is "funnel counts new machines, not install events" 3 "$(jget "$ACT" installs)"
is "all three indexed within the window" 3 "$(jget "$ACT" activated)"
is "nobody dropped out" 0 "$(jget "$ACT" dropped)"
is "raw-event floor is reported to the caller" "$DAY" "$(jget "$ACT" raw_events_from)"
is "retention endpoint answers" 200 \
"$(curl -s -o /dev/null -w '%{http_code}' -b "$JAR" "$DASH/api/retention?$RANGE")"
# --- the guarantee the cutover is selling ----------------------------------
echo
echo "the no-third-party guarantee still holds"
is "ingest worker makes no outbound fetch" 0 \
"$(grep -E 'fetch\(' "$WORKER_DIR"/src/*.ts | grep -vc 'async fetch(request' || true)"
is "ingest worker names no third-party analytics endpoint" 0 \
"$(grep -rEil 'https?://[a-z0-9.-]+/(batch|capture|collect|track|ingest)' \
"$WORKER_DIR"/src "$WORKER_DIR"/wrangler.jsonc 2>/dev/null | wc -l | tr -d ' ')"
echo
if [ "$fail" -eq 0 ]; then
echo "$pass passed, 0 failed — the chain is whole; safe to cut over"
else
echo "$pass passed, $fail failed"
fi
[ "$fail" -eq 0 ]
telemetry-worker/scripts/smoke-ingest.sh
Executable
+187
View File
@@ -0,0 +1,187 @@
#!/usr/bin/env bash
# End-to-end check of the ingest contract against a real `wrangler dev` + local D1.
#
# Boots the worker, POSTs a spread of good and bad batches, then shuts the worker
# down and inspects the rows that actually landed. Every request uses a fresh
# machine_id, so the script is re-runnable against a dirty local database and never
# trips the per-machine rate limit.
#
# npm run db:migrate:local # once
# npm run smoke # or: INGEST_PORT=8791 ./scripts/smoke-ingest.sh
set -euo pipefail
cd "$(dirname "$0")/.."
PORT="${INGEST_PORT:-8787}"
BASE="http://127.0.0.1:$PORT"
DB=codegraph-telemetry
pass=0; fail=0
ok() { pass=$((pass + 1)); printf ' ok %s\n' "$1"; }
bad() { fail=$((fail + 1)); printf ' FAIL %s — expected %s, got %s\n' "$1" "$2" "$3"; }
is() { [ "$2" = "$3" ] && ok "$1" || bad "$1" "$2" "$3"; }
uuid() { node -e 'console.log(crypto.randomUUID())'; }
# HTTP status of a POST /v1/events with the given body.
post() { curl -s -o /dev/null -w '%{http_code}' -X POST "$BASE/v1/events" \
-H 'content-type: application/json' --data-binary "$1"; }
# First column of the first row of a query against the LOCAL D1 state.
q() {
npx wrangler d1 execute "$DB" --local --json --command "$1" 2>/dev/null |
node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{
const r=JSON.parse(s.slice(s.indexOf("[")))[0]?.results?.[0];
console.log(r===undefined?"":String(Object.values(r)[0]));})'
}
# ---------------------------------------------------------------------------
# Boot
# ---------------------------------------------------------------------------
echo "booting wrangler dev on :$PORT"
npx wrangler dev --port "$PORT" >/tmp/cg-smoke-ingest.log 2>&1 &
DEV_PID=$!
cleanup() { kill "$DEV_PID" 2>/dev/null || true; wait "$DEV_PID" 2>/dev/null || true; }
trap cleanup EXIT
for _ in $(seq 1 60); do
curl -sf -o /dev/null "$BASE/" && break
kill -0 "$DEV_PID" 2>/dev/null || { echo "wrangler dev died:"; cat /tmp/cg-smoke-ingest.log; exit 1; }
sleep 1
done
curl -sf -o /dev/null "$BASE/" || { echo "worker never came up:"; cat /tmp/cg-smoke-ingest.log; exit 1; }
# ---------------------------------------------------------------------------
# Request contract
# ---------------------------------------------------------------------------
echo
echo "request contract"
INFO=$(curl -s "$BASE/")
case "$INFO" in *"codegraph anonymous-telemetry ingest"*) ok "GET / serves the info text";;
*) bad "GET / serves the info text" "info text" "$INFO";; esac
case "$INFO" in *"never forwarded to any third-party analytics"*) ok "info text states the storage guarantee";;
*) bad "info text states the storage guarantee" "the no-third-party sentence" "missing";; esac
# The guarantee above holds only while the worker makes no outbound request at all,
# so the only `fetch(` anywhere in the source may be the handler's own declaration.
is "worker source makes no outbound fetch" 0 \
"$(grep -E 'fetch\(' src/*.ts | grep -vc 'async fetch(request' || true)"
is "unknown path → 404" 404 "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/nope")"
is "GET /v1/events → 405" 405 "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/v1/events")"
is "non-JSON body → 400" 400 "$(post 'not json')"
is "JSON array body → 400" 400 "$(post '[]')"
is "missing machine_id → 400" 400 "$(post '{"events":[]}')"
is "malformed machine_id → 400" 400 "$(post '{"machine_id":"nope","events":[]}')"
is "chunked (no length) → 411" 411 "$(curl -s -o /dev/null -w '%{http_code}' -X POST "$BASE/v1/events" \
-H 'content-type: application/json' -H 'transfer-encoding: chunked' --data-binary '{"machine_id":"x"}')"
BIG=$(node -e 'process.stdout.write(JSON.stringify({machine_id:"00000000-0000-4000-8000-000000000000",pad:"x".repeat(70000),events:[]}))')
is "oversized body → 413" 413 "$(post "$BIG")"
# ---------------------------------------------------------------------------
# Accepted batches
# ---------------------------------------------------------------------------
echo
echo "ingest"
M_OK=$(uuid); M_DROP=$(uuid); M_CI=$(uuid); M_BACK=$(uuid)
TODAY=$(date -u +%F)
# Three valid events + one unknown event + unknown/malformed props that must be stripped.
is "valid batch → 204" 204 "$(post "$(node -e '
const [m] = process.argv.slice(1);
process.stdout.write(JSON.stringify({
machine_id: m, codegraph_version: "1.5.0", os: "darwin", arch: "arm64",
node_major: 22, ci: false, schema_version: 1, secret_field: "must not be stored",
events: [
{ event: "install", ts: "2026-07-27T10:00:00Z",
props: { scope: "local", kind: "fresh", targets: ["claude", "cursor"], nope: "strip me" } },
{ event: "index", ts: "2026-07-27T10:01:00Z",
props: { languages: ["typescript"], file_count_bucket: "100-1k",
duration_bucket: "bogus-bucket", repo_path: "/Users/someone/secret" } },
{ event: "usage_rollup",
props: { kind: "mcp_tool", name: "codegraph_explore", count: 12, client_name: "Claude Code" } },
{ event: "not_an_event", props: { count: 1 } },
],
}));' "$M_OK")")"
# Nothing survives the allowlist: unknown event + usage_rollup missing required props.
is "all-dropped batch → 204" 204 "$(post "$(node -e '
const [m] = process.argv.slice(1);
process.stdout.write(JSON.stringify({ machine_id: m, os: "linux", events: [
{ event: "made_up" },
{ event: "usage_rollup", props: { kind: "mcp_tool" } },
{ event: "install", props: { scope: "local" } },
]}));' "$M_DROP")")"
# NOTE: build every body into a variable first. Escaped quotes nested inside
# "$(post "…\"…\"…")" break out of the quoting context and get brace-expanded.
index_batch() { # <machine_id> [ci] [ts]
node -e 'const [m, ci, ts] = process.argv.slice(1);
const e = { event: "index", props: {} };
if (ts) e.ts = ts;
const b = { machine_id: m, os: "linux", events: [e] };
if (ci) b.ci = ci === "true";
process.stdout.write(JSON.stringify(b));' "$@"
}
# ci = true, then a non-CI batch for the same machine/day: prod must flip 0 → 1.
CI_ON=$(index_batch "$M_CI" true); CI_OFF=$(index_batch "$M_CI" false)
is "ci batch → 204" 204 "$(post "$CI_ON")"
is "same machine, non-ci → 204" 204 "$(post "$CI_OFF")"
# A late offline buffer arriving second must move first_day EARLIER, never later.
RECENT=$(index_batch "$M_BACK" "" 2026-07-27T09:00:00Z)
BACKDATED=$(index_batch "$M_BACK" "" 2026-07-20T09:00:00Z)
is "recent batch → 204" 204 "$(post "$RECENT")"
is "backdated batch → 204" 204 "$(post "$BACKDATED")"
sleep 2 # let the ctx.waitUntil writes drain
cleanup; trap - EXIT
sleep 1 # and let miniflare release the local sqlite file
# ---------------------------------------------------------------------------
# What actually got stored
# ---------------------------------------------------------------------------
echo
echo "stored rows"
is "3 of 4 events stored (unknown dropped)" 3 "$(q "select count(*) from events where machine_id='$M_OK'")"
is "all-dropped batch stored nothing" 0 "$(q "select count(*) from events where machine_id='$M_DROP'")"
is "…and no machine_days row for it" 0 "$(q "select count(*) from machine_days where machine_id='$M_DROP'")"
is "envelope columns land in their own columns" "darwin|arm64|22|0|1.5.0" \
"$(q "select os||'|'||arch||'|'||node_major||'|'||ci||'|'||codegraph_version from events where machine_id='$M_OK' limit 1")"
is "day derived from the client ts" "2026-07-27" \
"$(q "select day from events where machine_id='$M_OK' and event='install'")"
is "day falls back to received_at when ts is absent" "$TODAY" \
"$(q "select day from events where machine_id='$M_OK' and event='usage_rollup'")"
is "ts is NULL when the client sent none" 1 \
"$(q "select ts is null from events where machine_id='$M_OK' and event='usage_rollup'")"
is "allowlisted props stored" "local|fresh|2" \
"$(q "select json_extract(props,'\$.scope')||'|'||json_extract(props,'\$.kind')||'|'||json_array_length(props,'\$.targets') from events where machine_id='$M_OK' and event='install'")"
is "unknown prop stripped" 0 \
"$(q "select count(*) from events where machine_id='$M_OK' and props like '%strip me%'")"
is "malformed enum prop stripped" 0 \
"$(q "select count(*) from events where machine_id='$M_OK' and props like '%bogus-bucket%'")"
is "path-shaped prop stripped" 0 \
"$(q "select count(*) from events where machine_id='$M_OK' and props like '%/Users/%'")"
is "unknown envelope field stored nowhere" 0 \
"$(q "select count(*) from events where props like '%must not be stored%'")"
# The valid batch mixes ts-dated events (2026-07-27) with an undated rollup (today),
# so it legitimately spans two days and must produce a machine_days row for each.
is "machine_days: one row per distinct day in the batch" 2 \
"$(q "select count(*) from machine_days where machine_id='$M_OK'")"
is "machine_days: non-ci machine is production" 1 \
"$(q "select min(prod) from machine_days where machine_id='$M_OK'")"
is "machine_days: a later non-ci batch flips the day to production" 1 \
"$(q "select prod from machine_days where machine_id='$M_CI'")"
is "machine_days: each backdated batch gets its own day" "2026-07-20,2026-07-27" \
"$(q "select group_concat(day) from (select day from machine_days where machine_id='$M_BACK' order by day)")"
is "machine_first_seen recorded" "2026-07-27" "$(q "select first_day from machine_first_seen where machine_id='$M_OK'")"
is "machine_first_seen only moves earlier" "2026-07-20" \
"$(q "select first_day from machine_first_seen where machine_id='$M_BACK'")"
echo
echo "$pass passed, $fail failed"
[ "$fail" -eq 0 ]
telemetry-worker/scripts/smoke-rollup.sh
Executable
+276
View File
@@ -0,0 +1,276 @@
#!/usr/bin/env bash
# End-to-end check of the nightly rollup + retention purge against a real
# `wrangler dev` and the local D1 state.
#
# Seeds three synthetic days of events straight into local D1 (the ingest path clamps
# client timestamps to the last 30 days, so backdating far enough to exercise the purge
# has to bypass it), drives the rollup through the admin endpoint and the cron handler,
# then inspects what actually landed against hand-computed numbers.
#
# What it pins:
# * rollup numbers match the events they came from, including the two that are easy
# to get wrong — usage_rollup SUMs its `count` prop, and array props unnest
# * running a day twice changes nothing (idempotent upserts, no double counting)
# * ?reset=1 drops stale rollup rows on a live day and REFUSES to blank a day whose
# raw events are already purged
# * the purge deletes only rows past the window, and leaves machine_days /
# machine_first_seen alone
# * /admin/rollup does not exist without ADMIN_TOKEN, and rejects a wrong one
#
# Re-runnable: it wipes its own synthetic days first, and they are chosen to sit
# outside the cron's 3-day lookback so the nightly run never rewrites them.
#
# npm run smoke:rollup # or: ROLLUP_PORT=8792 ./scripts/smoke-rollup.sh
set -euo pipefail
cd "$(dirname "$0")/.."
PORT="${ROLLUP_PORT:-8788}"
BASE="http://127.0.0.1:$PORT"
DB=codegraph-telemetry
TOKEN=smoke-admin-token
SEED_SQL=/tmp/cg-smoke-rollup-seed.sql
LOG=/tmp/cg-smoke-rollup.log
pass=0; fail=0
ok() { pass=$((pass + 1)); printf ' ok %s\n' "$1"; }
bad() { fail=$((fail + 1)); printf ' FAIL %s — expected %s, got %s\n' "$1" "$2" "$3"; }
is() { [ "$2" = "$3" ] && ok "$1" || bad "$1" "$2" "$3"; }
day_ago() { node -e 'console.log(new Date(Date.now()-process.argv[1]*864e5).toISOString().slice(0,10))' "$1"; }
# Synthetic days. MAIN/RESET sit inside the 90-day retention window but outside the
# cron's 3-day lookback; OLD sits past the window so the purge takes it.
DAY_MAIN=$(day_ago 40)
DAY_RESET=$(day_ago 41)
DAY_OLD=$(day_ago 200)
CUTOFF=$(day_ago 90)
# First column of the first row of a query against the LOCAL D1 state.
q() {
npx wrangler d1 execute "$DB" --local --json --command "$1" 2>/dev/null |
node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{
const r=JSON.parse(s.slice(s.indexOf("[")))[0]?.results?.[0];
console.log(r===undefined?"":String(Object.values(r)[0]));})'
}
# A daily_dim_counts cell as "count/machines" — "" when the row does not exist.
dim() { q "select count||'/'||machines from daily_dim_counts
where day='$1' and event='$2' and dim='$3' and value='$4'"; }
# POST /admin/rollup, printing the HTTP status.
roll() { curl -s -o /dev/null -w '%{http_code}' -X POST -H "x-admin-token: $TOKEN" "$BASE/admin/rollup?$1"; }
boot() { # extra wrangler dev args
npx wrangler dev --port "$PORT" "$@" >"$LOG" 2>&1 &
DEV_PID=$!
trap 'kill "$DEV_PID" 2>/dev/null || true; wait "$DEV_PID" 2>/dev/null || true' EXIT
local up=
for _ in $(seq 1 60); do
curl -sf -o /dev/null "$BASE/" && { up=1; break; }
kill -0 "$DEV_PID" 2>/dev/null || { echo "wrangler dev died:"; cat "$LOG"; exit 1; }
sleep 1
done
[ -n "$up" ] || { echo "worker never came up:"; cat "$LOG"; exit 1; }
# If wrangler could not bind the port, something else answers every probe and the
# whole run silently grades a different server. Check who picked up.
case "$(curl -s "$BASE/")" in
*'codegraph anonymous-telemetry ingest'*) : ;;
*) echo "port $PORT is serving something else — set ROLLUP_PORT to a free one"; exit 1 ;;
esac
}
shutdown() {
kill "$DEV_PID" 2>/dev/null || true
wait "$DEV_PID" 2>/dev/null || true
trap - EXIT
sleep 1 # let miniflare release the local sqlite file
}
# ---------------------------------------------------------------------------
# Seed
# ---------------------------------------------------------------------------
echo "applying migrations to local D1"
npx wrangler d1 migrations apply "$DB" --local >/dev/null 2>&1
echo "seeding $DAY_MAIN / $DAY_RESET / $DAY_OLD"
node -e '
const [main, reset, old, seedFile] = process.argv.slice(1);
const sq = (v) => `'"'"'${String(v).replace(/'"'"'/g, "'"'"''"'"'")}'"'"'`;
const M = ["11111111-1111-4111-8111-111111111111", "22222222-2222-4222-8222-222222222222",
"33333333-3333-4333-8333-333333333333", "44444444-4444-4444-8444-444444444444",
"99999999-9999-4999-8999-999999999999"];
const out = [];
// Re-runnable: every table this script touches, scoped to its own synthetic days.
for (const t of ["events", "daily_event_counts", "daily_dim_counts", "daily_machines", "machine_days"]) {
out.push(`DELETE FROM ${t} WHERE day IN (${[main, reset, old].map(sq).join(", ")});`);
}
out.push(`DELETE FROM machine_first_seen WHERE machine_id IN (${M.map(sq).join(", ")});`);
// day, machine, event, os, arch, version, node_major, ci, props
const rows = [
[main, M[0], "install", "darwin", "arm64", "1.5.0", 22, 0, {targets:["claude","cursor"], scope:"local", kind:"fresh"}],
[main, M[0], "index", "darwin", "arm64", "1.5.0", 22, 0, {languages:["typescript","go"], file_count_bucket:"100-1k", duration_bucket:"10-60s"}],
[main, M[0], "usage_rollup", "darwin", "arm64", "1.5.0", 22, 0, {kind:"mcp_tool", name:"codegraph_explore", count:10, error_count:2, client_name:"Claude Code"}],
[main, M[1], "index", "darwin", "x64", "1.5.0", 20, 0, {languages:["typescript"], file_count_bucket:"1k-10k", duration_bucket:"10-60s"}],
[main, M[1], "usage_rollup", "darwin", "x64", "1.5.0", 20, 0, {kind:"mcp_tool", name:"codegraph_explore", count:5, error_count:0, client_name:"Cursor"}],
[main, M[2], "install", "linux", "x64", "1.4.1", 22, 1, {targets:["claude"], scope:"global", kind:"upgrade"}],
[main, M[2], "uninstall", "linux", "x64", "1.4.1", 22, 1, {targets:["claude"]}],
[reset, M[3], "index", "darwin", "arm64", "1.5.0", 22, 0, {languages:["python"], file_count_bucket:"<100", duration_bucket:"<10s"}],
[old, M[4], "install", "linux", "x64", "1.0.0", 20, 0, {targets:["codex"], scope:"local", kind:"fresh"}],
[old, M[4], "index", "linux", "x64", "1.0.0", 20, 0, {languages:["rust"], file_count_bucket:"<100", duration_bucket:"<10s"}],
];
for (const [day, m, event, os, arch, version, node, ci, props] of rows) {
out.push(`INSERT INTO events (received_at, ts, day, event, machine_id, codegraph_version, os, arch, node_major, ci, schema_version, props)
VALUES (${sq(day + "T12:00:00.000Z")}, ${sq(day + "T12:00:00.000Z")}, ${sq(day)}, ${sq(event)}, ${sq(m)},
${sq(version)}, ${sq(os)}, ${sq(arch)}, ${node}, ${ci}, 1, ${sq(JSON.stringify(props))});`);
}
// What the ingest path would have written alongside those events.
for (const [m, day, prod] of [[M[0], main, 1], [M[1], main, 1], [M[2], main, 0], [M[3], reset, 1], [M[4], old, 1]]) {
out.push(`INSERT INTO machine_days (machine_id, day, prod) VALUES (${sq(m)}, ${sq(day)}, ${prod});`);
out.push(`INSERT INTO machine_first_seen (machine_id, first_day) VALUES (${sq(m)}, ${sq(day)})
ON CONFLICT (machine_id) DO UPDATE SET first_day = min(machine_first_seen.first_day, excluded.first_day);`);
}
// A rollup row from a dimension that no longer exists — only ?reset=1 should clear it.
out.push(`INSERT INTO daily_dim_counts (day, event, dim, value, count, machines)
VALUES (${sq(reset)}, ${sq("index")}, ${sq("obsolete_dim")}, ${sq("stale")}, 99, 99);`);
require("fs").writeFileSync(seedFile, out.join("\n"));
' "$DAY_MAIN" "$DAY_RESET" "$DAY_OLD" "$SEED_SQL"
npx wrangler d1 execute "$DB" --local --file "$SEED_SQL" >/dev/null
# ---------------------------------------------------------------------------
# The admin route does not exist without a token
# ---------------------------------------------------------------------------
echo
echo "admin route, no ADMIN_TOKEN configured"
boot
is "POST /admin/rollup → 404" 404 "$(curl -s -o /dev/null -w '%{http_code}' -X POST "$BASE/admin/rollup")"
is "…even with a token header" 404 \
"$(curl -s -o /dev/null -w '%{http_code}' -X POST -H "x-admin-token: $TOKEN" "$BASE/admin/rollup")"
shutdown
# ---------------------------------------------------------------------------
# Drive the rollup
# ---------------------------------------------------------------------------
echo
echo "admin route, ADMIN_TOKEN configured"
boot --test-scheduled --var "ADMIN_TOKEN:$TOKEN"
is "no token → 401" 401 "$(curl -s -o /dev/null -w '%{http_code}' -X POST "$BASE/admin/rollup")"
is "wrong token → 401" 401 \
"$(curl -s -o /dev/null -w '%{http_code}' -X POST -H 'x-admin-token: nope' "$BASE/admin/rollup")"
is "GET → 405" 405 "$(curl -s -o /dev/null -w '%{http_code}' -H "x-admin-token: $TOKEN" "$BASE/admin/rollup")"
is "impossible day → 400" 400 "$(roll 'day=2026-02-31')"
is "malformed day → 400" 400 "$(roll 'day=yesterday')"
is "days out of range → 400" 400 "$(roll "day=$DAY_MAIN&days=99")"
echo
echo "rollup"
is "rollup $DAY_MAIN → 200" 200 "$(roll "day=$DAY_MAIN")"
is "rollup $DAY_MAIN again → 200" 200 "$(roll "day=$DAY_MAIN")"
is "rollup $DAY_OLD, whose events are still there → 200" 200 "$(roll "day=$DAY_OLD")"
is "rollup $DAY_RESET with reset → 200" 200 "$(roll "day=$DAY_RESET&reset=1")"
# The cron body: rolls up the last three days and purges everything past the window.
is "cron trigger → 200" 200 "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/__scheduled?cron=30+0+*+*+*")"
sleep 2
# Rolling a purged day with reset=1 must NOT blank the rollups it already has: past
# the window the reset is ignored, so the delete-then-rebuild can't find zero events.
is "rollup $DAY_OLD after the purge, with reset → 200" 200 "$(roll "day=$DAY_OLD&reset=1")"
is "…and reports the reset it refused to run" "[\"$DAY_OLD\"]" \
"$(curl -s -X POST -H "x-admin-token: $TOKEN" "$BASE/admin/rollup?day=$DAY_OLD&reset=1" |
node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>console.log(JSON.stringify(JSON.parse(s).reset_ignored)))')"
shutdown
# ---------------------------------------------------------------------------
# What actually landed — every number below is hand-computed from the seed above
# ---------------------------------------------------------------------------
echo
echo "daily_machines"
is "3 machines, 2 of them production (one is ci)" "3/2" \
"$(q "select machines||'/'||prod_machines from daily_machines where day='$DAY_MAIN'")"
echo
echo "daily_event_counts"
is "install: 2 events from 2 machines" "2/2" \
"$(q "select count||'/'||machines from daily_event_counts where day='$DAY_MAIN' and event='install'")"
is "index: 2 events from 2 machines" "2/2" \
"$(q "select count||'/'||machines from daily_event_counts where day='$DAY_MAIN' and event='index'")"
is "uninstall: 1 event from 1 machine" "1/1" \
"$(q "select count||'/'||machines from daily_event_counts where day='$DAY_MAIN' and event='uninstall'")"
# The one that is easy to get wrong: 2 rows carrying count 10 and 5 is 15 tool calls.
is "usage_rollup: SUMs the count prop (10+5), not the rows" "15/2" \
"$(q "select count||'/'||machines from daily_event_counts where day='$DAY_MAIN' and event='usage_rollup'")"
is "one row per event type" 4 "$(q "select count(*) from daily_event_counts where day='$DAY_MAIN'")"
echo
echo "daily_dim_counts"
is "os / index / darwin" "2/2" "$(dim "$DAY_MAIN" index os darwin)"
is "os / usage_rollup / darwin sums counts" "15/2" "$(dim "$DAY_MAIN" usage_rollup os darwin)"
is "arch / uninstall / x64" "1/1" "$(dim "$DAY_MAIN" uninstall arch x64)"
is "codegraph_version / index / 1.5.0" "2/2" "$(dim "$DAY_MAIN" index codegraph_version 1.5.0)"
is "node_major / install / 22 (stored as text)" "2/2" "$(dim "$DAY_MAIN" install node_major 22)"
is "file_count_bucket / index / 100-1k" "1/1" "$(dim "$DAY_MAIN" index file_count_bucket 100-1k)"
is "duration_bucket / index / 10-60s" "2/2" "$(dim "$DAY_MAIN" index duration_bucket 10-60s)"
is "scope / install / global" "1/1" "$(dim "$DAY_MAIN" install scope global)"
is "kind / install / fresh" "1/1" "$(dim "$DAY_MAIN" install kind fresh)"
is "kind / usage_rollup / mcp_tool (same dim, other event)" "15/2" "$(dim "$DAY_MAIN" usage_rollup kind mcp_tool)"
is "name / usage_rollup / codegraph_explore" "15/2" "$(dim "$DAY_MAIN" usage_rollup name codegraph_explore)"
is "client_name / usage_rollup / Claude Code" "10/1" "$(dim "$DAY_MAIN" usage_rollup client_name 'Claude Code')"
# languages and targets are JSON arrays: one row per element, counted once per event.
is "language / index / typescript (unnested, 2 events)" "2/2" "$(dim "$DAY_MAIN" index language typescript)"
is "language / index / go (unnested, 1 event)" "1/1" "$(dim "$DAY_MAIN" index language go)"
is "target / install / claude (unnested, 2 events)" "2/2" "$(dim "$DAY_MAIN" install target claude)"
is "target / install / cursor" "1/1" "$(dim "$DAY_MAIN" install target cursor)"
is "target / uninstall / claude" "1/1" "$(dim "$DAY_MAIN" uninstall target claude)"
# Only groups with at least one error are stored, so machines = machines that saw one.
is "name_error / usage_rollup / codegraph_explore" "2/1" "$(dim "$DAY_MAIN" usage_rollup name_error codegraph_explore)"
is "no dimension row for a machine with no errors" "" "$(dim "$DAY_MAIN" usage_rollup name_error nothing)"
is "40 dimension rows in total (no strays, no doubles)" 40 \
"$(q "select count(*) from daily_dim_counts where day='$DAY_MAIN'")"
# Independent of the hand-computed numbers: recompute two of them straight off `events`.
echo
echo "cross-check against the raw events"
is "machines matches count(distinct machine_id)" \
"$(q "select count(distinct machine_id) from events where day='$DAY_MAIN' and event='index'")" \
"$(q "select machines from daily_event_counts where day='$DAY_MAIN' and event='index'")"
is "usage count matches sum(props.count)" \
"$(q "select sum(json_extract(props,'\$.count')) from events where day='$DAY_MAIN' and event='usage_rollup'")" \
"$(q "select count from daily_event_counts where day='$DAY_MAIN' and event='usage_rollup'")"
echo
echo "reset"
is "?reset=1 drops a rollup row whose dimension no longer exists" 0 \
"$(q "select count(*) from daily_dim_counts where day='$DAY_RESET' and dim='obsolete_dim'")"
is "…and recomputes the day correctly" "1/1" "$(dim "$DAY_RESET" index language python)"
is "…leaving exactly the 7 dimensions that day has" 7 \
"$(q "select count(*) from daily_dim_counts where day='$DAY_RESET'")"
echo
echo "retention purge"
is "raw events past the window are gone" 0 "$(q "select count(*) from events where day='$DAY_OLD'")"
is "nothing older than the cutoff survives" 0 "$(q "select count(*) from events where day<'$CUTOFF'")"
is "events inside the window are untouched" 7 "$(q "select count(*) from events where day='$DAY_MAIN'")"
is "machine_days is NOT purged (retention cohorts need it)" 1 \
"$(q "select count(*) from machine_days where day='$DAY_OLD'")"
is "machine_first_seen is NOT purged" "$DAY_OLD" \
"$(q "select first_day from machine_first_seen where machine_id='99999999-9999-4999-8999-999999999999'")"
echo
echo "rollups outlive the events they came from"
is "daily_event_counts survives the purge" "1/1" \
"$(q "select count||'/'||machines from daily_event_counts where day='$DAY_OLD' and event='index'")"
is "daily_dim_counts survives the purge" "1/1" "$(dim "$DAY_OLD" index language rust)"
is "…all 14 rows of it, even after a reset run over the purged day" 14 \
"$(q "select count(*) from daily_dim_counts where day='$DAY_OLD'")"
is "daily_machines is still rebuilt for a purged day (machine_days survives)" "1/1" \
"$(q "select machines||'/'||prod_machines from daily_machines where day='$DAY_OLD'")"
echo
echo "$pass passed, $fail failed"
[ "$fail" -eq 0 ]
telemetry-worker/src/env.d.ts
Vendored
+10
View File
@@ -0,0 +1,10 @@
/**
* Secrets are set with `wrangler secret put`, so they are deliberately absent from
* wrangler.jsonc (this repo is public) and `wrangler types` cannot see them. Declared
* here by interface merging so the worker type-checks with or without a local
* `.dev.vars`. Anything listed here may be missing at runtime — check before use.
*/
interface Env {
/** Shared secret for `POST /admin/rollup`. Unset ⇒ the route does not exist (404). */
ADMIN_TOKEN: string;
}
telemetry-worker/src/index.ts
+137 -36
View File
@@ -3,15 +3,22 @@
*
* This file is public on purpose: it is the exact code that receives codegraph's
* anonymous usage telemetry, so anyone can audit what is (and is not) stored.
* The schema contract lives in docs/design/telemetry.md.
* The schema contract lives in docs/design/telemetry.md; the storage schema — the
* complete list of what is kept — is migrations/0001_init.sql.
*
* Guarantees enforced here:
* - strict allowlist: unknown events are dropped, unknown properties are stripped
* - the client IP is never read, logged, or forwarded
* - the client IP is never read, logged, or stored
* - accepted events land in our own Cloudflare D1 database and are never forwarded
* to a third-party analytics vendor — this worker makes no outbound requests
* - per-machine rate limiting, bounded body/batch sizes
* - forwarding happens off the response path (ctx.waitUntil); bodies are never logged
* - the write happens off the response path (ctx.waitUntil); bodies are never logged
* - raw events expire: a nightly cron rolls each day up into anonymous daily counts
* and then deletes the rows behind it (rollup.ts)
*/
import { handleAdminRollup, retentionDays, runNightly } from './rollup';
const MAX_BODY_BYTES = 64 * 1024;
const MAX_EVENTS_PER_BATCH = 100;
const UUID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
@@ -20,13 +27,25 @@ const TOKEN_RE = /^[A-Za-z0-9_.:+-]+$/;
// Human-ish labels: MCP clientInfo names like "Claude Code", "cursor-vscode/1.2".
const LABEL_RE = /^[A-Za-z0-9_.:+/ @()-]+$/;
const INFO_TEXT = `codegraph anonymous-telemetry ingest.
const infoText = (keepDays: number): string => `codegraph anonymous-telemetry ingest.
What gets collected (and what never does) is documented field-by-field:
https://github.com/colbymchenry/codegraph/blob/main/docs/design/telemetry.md
This endpoint's full source:
https://github.com/colbymchenry/codegraph/tree/main/telemetry-worker
Guarantees: no code, file paths, repo/file/symbol names, or query strings are ever
sent; the client IP is never read or stored; the machine ID is a random UUID the
client mints locally and can delete at any time. Accepted events are stored in our
own database on Cloudflare (D1) and are never forwarded to any third-party analytics
vendor. The stored schema is the complete list of what is kept:
https://github.com/colbymchenry/codegraph/blob/main/telemetry-worker/migrations/0001_init.sql
Individual events are deleted after ${keepDays} days. What outlives them: anonymous
daily totals (counts per day of things like operating system, version and language),
and which days each machine ID was active, so returning-user numbers survive. No event
details, and still nothing that identifies a person or a codebase.
Disable any time: codegraph telemetry off | CODEGRAPH_TELEMETRY=0 | DO_NOT_TRACK=1
`;
@@ -117,11 +136,17 @@ const ENVELOPE_PROPS: Record<string, Sanitize> = {
schema_version: nonNegInt(99),
};
interface PostHogEvent {
/**
* One sanitized event, ready to become one `events` row. The envelope is NOT
* folded in here: it is identical for every event in a batch and lands in its own
* columns, so it is carried alongside (`common`) and bound at write time.
*/
interface StoredEvent {
event: string;
distinct_id: string;
timestamp?: string;
properties: JsonObject;
/** Clamped ISO 8601 UTC; absent when the client sent none or sent nonsense. */
ts?: string;
/** Event-specific props only — stored as the `props` JSON column. */
props: JsonObject;
}
function clampTimestamp(v: unknown): string | undefined {
@@ -134,7 +159,7 @@ function clampTimestamp(v: unknown): string | undefined {
return new Date(t).toISOString();
}
function sanitizeEvent(raw: unknown, machineId: string, common: JsonObject): PostHogEvent | null {
function sanitizeEvent(raw: unknown): StoredEvent | null {
if (typeof raw !== 'object' || raw === null) return null;
const e = raw as JsonObject;
if (typeof e.event !== 'string') return null;
@@ -151,36 +176,94 @@ function sanitizeEvent(raw: unknown, machineId: string, common: JsonObject): Pos
if (!(req in props)) return null;
}
const out: PostHogEvent = {
event: e.event,
distinct_id: machineId,
properties: {
...props,
...common,
// Anonymous events: no person profiles, no geo enrichment.
$process_person_profile: false,
$geoip_disable: true,
$lib: 'codegraph-telemetry-worker',
},
};
const out: StoredEvent = { event: e.event, props };
const ts = clampTimestamp(e.ts);
if (ts !== undefined) out.timestamp = ts;
if (ts !== undefined) out.ts = ts;
return out;
}
async function forwardToPostHog(env: Env, batch: PostHogEvent[]): Promise<void> {
/**
* Re-narrow a sanitized envelope value for binding. The ENVELOPE_PROPS sanitizers
* already guarantee these types; these just turn "absent" into a NULL bind.
*/
const asText = (v: unknown): string | null => (typeof v === 'string' ? v : null);
const asInt = (v: unknown): number | null => (typeof v === 'number' ? v : null);
const asFlag = (v: unknown): number | null => (typeof v === 'boolean' ? (v ? 1 : 0) : null);
const INSERT_EVENT = `INSERT INTO events (
received_at, ts, day, event, machine_id,
codegraph_version, os, arch, node_major, ci, schema_version, props
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`;
// prod = 0 only if EVERY event this machine sent that day carried ci = 1, so a later
// non-CI batch flips the day to production and never back (max, not overwrite).
const UPSERT_MACHINE_DAY = `INSERT INTO machine_days (machine_id, day, prod) VALUES (?, ?, ?)
ON CONFLICT (machine_id, day) DO UPDATE SET prod = max(machine_days.prod, excluded.prod)`;
// A late-arriving offline buffer can move a machine's first day earlier, never later.
const UPSERT_FIRST_SEEN = `INSERT INTO machine_first_seen (machine_id, first_day) VALUES (?, ?)
ON CONFLICT (machine_id) DO UPDATE SET first_day = min(machine_first_seen.first_day, excluded.first_day)`;
/**
* Persist a sanitized batch: one `events` row per event, plus the machine×day and
* first-seen bookkeeping the dashboard's retention/activation panels need. One D1
* `batch()` = one implicit transaction = one round trip.
*
* Fail-silent by design: the client treats every response as final and never retries,
* so a failed write loses a datapoint rather than costing availability. The error is
* logged (Workers Logs) with counts only — never the payload.
*/
async function writeToD1(
env: Env,
machineId: string,
common: JsonObject,
batch: StoredEvent[],
): Promise<void> {
try {
const res = await fetch(`${env.POSTHOG_HOST}/batch/`, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ api_key: env.POSTHOG_KEY, batch }),
signal: AbortSignal.timeout(5000),
});
if (!res.ok) {
console.error(JSON.stringify({ msg: 'posthog forward failed', status: res.status, events: batch.length }));
const receivedAt = new Date().toISOString();
const insertEvent = env.DB.prepare(INSERT_EVENT);
const stmts: D1PreparedStatement[] = [];
// Envelope columns are identical for every row in the batch.
const envelopeCols = [
asText(common.codegraph_version),
asText(common.os),
asText(common.arch),
asInt(common.node_major),
asFlag(common.ci),
asInt(common.schema_version),
] as const;
// A batch can span days (offline buffers hold completed-day rollups), so
// machine_days gets one row per distinct day rather than one per batch.
const days = new Set<string>();
for (const e of batch) {
const day = (e.ts ?? receivedAt).slice(0, 10);
days.add(day);
stmts.push(
insertEvent.bind(
receivedAt,
e.ts ?? null,
day,
e.event,
machineId,
...envelopeCols,
JSON.stringify(e.props),
),
);
}
const prod = common.ci === true ? 0 : 1;
const upsertDay = env.DB.prepare(UPSERT_MACHINE_DAY);
for (const day of days) stmts.push(upsertDay.bind(machineId, day, prod));
const firstDay = [...days].sort()[0];
if (firstDay !== undefined) {
stmts.push(env.DB.prepare(UPSERT_FIRST_SEEN).bind(machineId, firstDay));
}
await env.DB.batch(stmts);
} catch (err) {
console.error(JSON.stringify({ msg: 'posthog forward error', err: String(err), events: batch.length }));
console.error(JSON.stringify({ msg: 'd1 write failed', err: String(err), events: batch.length }));
}
}
@@ -190,7 +273,13 @@ export default {
const url = new URL(request.url);
if (request.method === 'GET' && url.pathname === '/') {
return new Response(INFO_TEXT, { headers: { 'content-type': 'text/plain; charset=utf-8' } });
return new Response(infoText(retentionDays(env)), {
headers: { 'content-type': 'text/plain; charset=utf-8' },
});
}
// Backfill/repair for the nightly rollup. 404s unless ADMIN_TOKEN is configured.
if (url.pathname === '/admin/rollup') {
return await handleAdminRollup(request, env, url);
}
if (url.pathname !== '/v1/events') {
return new Response('not found\n', { status: 404 });
@@ -240,14 +329,16 @@ export default {
}
const rawEvents = Array.isArray(body.events) ? body.events.slice(0, MAX_EVENTS_PER_BATCH) : [];
const batch: PostHogEvent[] = [];
const batch: StoredEvent[] = [];
for (const raw of rawEvents) {
const sanitized = sanitizeEvent(raw, machineId, common);
const sanitized = sanitizeEvent(raw);
if (sanitized) batch.push(sanitized);
}
// Nothing survived the allowlist ⇒ nothing is written at all, not even the
// machine×day bookkeeping: those tables must only ever describe stored events.
if (batch.length > 0) {
ctx.waitUntil(forwardToPostHog(env, batch));
ctx.waitUntil(writeToD1(env, machineId, common, batch));
}
// Accepted (including "everything was dropped by the allowlist") — the
// client treats every response as final and never retries.
@@ -257,4 +348,14 @@ export default {
return new Response('internal error\n', { status: 500 });
}
},
/**
* Nightly (00:30 UTC, see wrangler.jsonc): roll the completed day up into the
* daily_* tables and purge raw events past the retention window. Awaited rather
* than backgrounded so a failure marks the cron run failed — everything it does is
* an idempotent upsert or a bounded delete, so the retry is safe.
*/
async scheduled(event, env): Promise<void> {
await runNightly(env, event.scheduledTime);
},
} satisfies ExportedHandler<Env>;
telemetry-worker/src/rollup.ts
+397
View File
@@ -0,0 +1,397 @@
/**
* codegraph telemetry — nightly rollup + raw-event retention purge.
*
* Public for the same reason the ingest path is: this is every read and every write
* we make over the stored events, including the one that deletes them.
*
* Two jobs, both driven by the cron trigger in wrangler.jsonc (00:30 UTC daily):
*
* 1. ROLL UP the just-completed UTC day into `daily_machines`, `daily_event_counts`
* and `daily_dim_counts` — plus the two days before it, because clients buffer
* offline and ship completed-day rollups late, so a day keeps growing after it
* ends. Every write is an upsert that OVERWRITES the recomputed value rather than
* adding to it, so re-running a day is a no-op and never double-counts.
*
* 2. PURGE raw `events` past the retention window, in bounded batches. Rollups are
* kept forever, so only ad-hoc drill-down has a horizon; `machine_days` and
* `machine_first_seen` are never purged, because retention cohorts need the full
* history and they are two orders of magnitude smaller than the raw rows.
*
* `POST /admin/rollup` re-runs a day (or a short range) on demand for backfill and
* repair, guarded by the ADMIN_TOKEN secret. Like everything else here it makes no
* outbound requests — the only thing this worker talks to is its own D1 database.
*/
/** Raw-event retention when RETENTION_DAYS is unset or nonsense. Storage-bound — see README. */
export const DEFAULT_RETENTION_DAYS = 90;
/** The just-completed day, plus the two before it (late offline buffers). */
export const ROLLUP_LOOKBACK_DAYS = 3;
/** Widest range one manual /admin/rollup call will attempt. */
export const MAX_MANUAL_DAYS = 31;
/** Rows per purge DELETE — bounded so one statement stays well inside D1's limits. */
const PURGE_BATCH_ROWS = 5_000;
/** Ceiling on one night's deletions (≈1.5 days of ingest at current volume). */
const PURGE_MAX_BATCHES = 60;
const DAY_MS = 86_400_000;
/** UTC YYYY-MM-DD — the key every event, rollup and chart is bucketed on. */
export function utcDay(atMs: number): string {
return new Date(atMs).toISOString().slice(0, 10);
}
/** Rejects both the wrong shape and impossible dates (`2026-02-31` round-trips as `2026-03-03`). */
export function isValidDay(day: string): boolean {
if (!/^\d{4}-\d{2}-\d{2}$/.test(day)) return false;
const t = Date.parse(`${day}T00:00:00Z`);
return Number.isFinite(t) && utcDay(t) === day;
}
/** Configured retention, clamped to something sane; falls back to the default. */
export function retentionDays(env: Env): number {
const raw = Number(env.RETENTION_DAYS);
return Number.isInteger(raw) && raw >= 1 && raw <= 3650 ? raw : DEFAULT_RETENTION_DAYS;
}
/** Oldest day kept: everything strictly before this is purged. */
export function retentionCutoff(atMs: number, keepDays: number): string {
return utcDay(atMs - keepDays * DAY_MS);
}
// ---------------------------------------------------------------------------
// The rollup statements
// ---------------------------------------------------------------------------
// One `INSERT … SELECT … ON CONFLICT DO UPDATE` per table or dimension: the whole
// aggregation happens inside D1, so a day rolls up in one round trip and no event
// row ever crosses the wire. Each takes exactly one bound parameter — the day.
//
// Adding a breakdown is a line in ROLLUP_STATEMENTS, never a migration — that is
// what the generic (dim, value) shape of daily_dim_counts buys.
/**
* A group's event volume. For install/index/uninstall one row is one event, but a
* usage_rollup row is a counter the client pre-aggregated (one per machine × day ×
* tool), so its `count` prop is what has to be summed — counting rows there would
* silently report "machines that used the tool" and undercount by an order of magnitude.
*/
const COUNT = `CASE WHEN e.event = 'usage_rollup'
THEN sum(coalesce(json_extract(e.props, '$.count'), 0))
ELSE count(*) END`;
const DIM_CONFLICT = `ON CONFLICT (day, event, dim, value) DO UPDATE
SET count = excluded.count, machines = excluded.machines`;
const prop = (name: string): string => `json_extract(e.props, '$.${name}')`;
const quoted = (values: readonly string[]): string => values.map((v) => `'${v}'`).join(', ');
const onlyEvents = (...events: readonly string[]): string => ` AND e.event IN (${quoted(events)})`;
/** One dimension whose value is a scalar column or a scalar prop. */
function dimStatement(dim: string, value: string, where = ''): string {
return `INSERT INTO daily_dim_counts (day, event, dim, value, count, machines)
SELECT e.day, e.event, '${dim}', CAST(${value} AS TEXT), ${COUNT}, count(DISTINCT e.machine_id)
FROM events e
WHERE e.day = ? AND ${value} IS NOT NULL AND ${value} <> ''${where}
GROUP BY e.day, e.event, ${value}
${DIM_CONFLICT}`;
}
/**
* One dimension unnested from a JSON array prop — one row per element, so an index
* of a TypeScript+Go repo counts once under each language. `json_each` over a path
* the props do not have yields no rows, which is exactly the wanted behaviour for
* events that omit the array.
*/
function arrayDimStatement(dim: string, path: string, events: readonly string[]): string {
return `INSERT INTO daily_dim_counts (day, event, dim, value, count, machines)
SELECT e.day, e.event, '${dim}', CAST(j.value AS TEXT), count(*), count(DISTINCT e.machine_id)
FROM events e, json_each(e.props, '${path}') j
WHERE e.day = ? AND e.event IN (${quoted(events)}) AND j.value <> ''
GROUP BY e.day, e.event, j.value
${DIM_CONFLICT}`;
}
/**
* Rebuilt from `machine_days`, not from `events`: that table is never purged, so this
* number stays right for days whose raw rows are long gone. `prod` is already the
* per-machine-day maximum the ingest path maintains (0 only if every event that
* machine sent that day carried ci = 1).
*
* So this is also the one rollup that can still be rebuilt for a day whose raw events
* are long gone.
*/
const DAILY_MACHINES = `INSERT INTO daily_machines (day, machines, prod_machines)
SELECT day, count(*), coalesce(sum(prod), 0) FROM machine_days WHERE day = ? GROUP BY day
ON CONFLICT (day) DO UPDATE
SET machines = excluded.machines, prod_machines = excluded.prod_machines`;
const ROLLUP_STATEMENTS: readonly string[] = [
DAILY_MACHINES,
`INSERT INTO daily_event_counts (day, event, count, machines)
SELECT e.day, e.event, ${COUNT}, count(DISTINCT e.machine_id)
FROM events e
WHERE e.day = ?
GROUP BY e.day, e.event
ON CONFLICT (day, event) DO UPDATE
SET count = excluded.count, machines = excluded.machines`,
// Envelope dimensions — every event type carries them.
dimStatement('os', 'e.os'),
dimStatement('arch', 'e.arch'),
dimStatement('codegraph_version', 'e.codegraph_version'),
dimStatement('node_major', 'e.node_major'),
// Event-specific scalar props.
dimStatement('file_count_bucket', prop('file_count_bucket'), onlyEvents('index')),
dimStatement('duration_bucket', prop('duration_bucket'), onlyEvents('index')),
dimStatement('scope', prop('scope'), onlyEvents('install')),
// `kind` is fresh/upgrade/reinstall on install and mcp_tool/cli_command on
// usage_rollup; `event` is part of the primary key, so both live here without colliding.
dimStatement('kind', prop('kind'), onlyEvents('install', 'usage_rollup')),
dimStatement('name', prop('name'), onlyEvents('usage_rollup')),
dimStatement('client_name', prop('client_name'), onlyEvents('usage_rollup')),
// Array props.
arrayDimStatement('language', '$.languages', ['index']),
arrayDimStatement('target', '$.targets', ['install', 'uninstall']),
// Errors per tool/command. Not in the migration's documented dim list because dims
// are a cron concern rather than a schema one, but rolled up because it is the one
// usage number that is gone for good after the purge. Only groups with at least one
// error are stored, so `count` is errors and `machines` is the machines that saw one
// — NOT the machines that ran the tool (that is the `name` dim).
`INSERT INTO daily_dim_counts (day, event, dim, value, count, machines)
SELECT e.day, e.event, 'name_error', CAST(${prop('name')} AS TEXT),
sum(${prop('error_count')}), count(DISTINCT e.machine_id)
FROM events e
WHERE e.day = ? AND e.event = 'usage_rollup'
AND ${prop('name')} IS NOT NULL AND coalesce(${prop('error_count')}, 0) > 0
GROUP BY e.day, e.event, ${prop('name')}
${DIM_CONFLICT}`,
];
/** Rollup tables derived from raw `events` — the ones `reset` wipes before recomputing. */
const EVENT_DERIVED_TABLES = ['daily_event_counts', 'daily_dim_counts'] as const;
// ---------------------------------------------------------------------------
// Running it
// ---------------------------------------------------------------------------
export interface DayResult {
day: string;
/** Rollup rows written for the day. */
rows: number;
/** Day is past the retention window — a `reset` on it is ignored (see below). */
pastRetention: boolean;
}
/**
* Recompute every rollup for one UTC day. One D1 `batch()` = one implicit
* transaction, so a day is either fully recomputed or not touched at all.
*
* Plain (upsert-only) runs are safe on any day: a day whose raw events are already
* purged selects nothing, so nothing is written and the rollups it earned while the
* events were still around survive untouched. That is what keeps rollups permanent.
*
* `reset` drops the day's event-derived rollup rows first instead of upserting over
* them — repair for when the dimension list itself changes and a value that no longer
* exists would otherwise linger. It is IGNORED past the retention window, where it
* would delete rows and then find no events to rebuild them from: silently blanking a
* real day is the one irreversible thing this file could do.
*/
export async function rollupDay(
env: Env,
day: string,
opts: { cutoff: string; reset?: boolean },
): Promise<DayResult> {
const pastRetention = day < opts.cutoff;
const statements: D1PreparedStatement[] = [];
if (opts.reset && !pastRetention) {
for (const table of EVENT_DERIVED_TABLES) {
statements.push(env.DB.prepare(`DELETE FROM ${table} WHERE day = ?`).bind(day));
}
}
for (const sql of ROLLUP_STATEMENTS) {
statements.push(env.DB.prepare(sql).bind(day));
}
const results = await env.DB.batch(statements);
const rows = results.reduce((total, r) => total + (r.meta?.changes ?? 0), 0);
return { day, rows, pastRetention };
}
export interface PurgeResult {
/** Everything strictly before this day was deleted. */
cutoff: string;
deleted: number;
batches: number;
/** Hit the per-run batch ceiling — more rows are still due, next run takes them. */
capped: boolean;
}
/**
* Delete raw events older than the window, oldest first, in bounded batches.
* `id` is a rowid alias and the purge only ever removes the oldest rows, so the
* keyset subquery stays a cheap index range scan on (day, event).
*/
export async function purgeOldEvents(env: Env, cutoff: string): Promise<PurgeResult> {
const del = env.DB.prepare(
`DELETE FROM events WHERE id IN (SELECT id FROM events WHERE day < ? LIMIT ${PURGE_BATCH_ROWS})`,
);
let deleted = 0;
for (let batch = 1; batch <= PURGE_MAX_BATCHES; batch++) {
const { meta } = await del.bind(cutoff).run();
const removed = meta?.changes ?? 0;
deleted += removed;
if (removed < PURGE_BATCH_ROWS) return { cutoff, deleted, batches: batch, capped: false };
}
return { cutoff, deleted, batches: PURGE_MAX_BATCHES, capped: true };
}
/**
* The cron body: roll up the completed day and the two before it, then purge.
*
* Logs one line of counts — never a day's contents, never a machine id. Throws if
* anything failed so the invocation is marked failed (and retried) rather than
* quietly skipping a day; every write here is idempotent, so a retry is safe.
*/
export async function runNightly(env: Env, atMs: number): Promise<void> {
const started = Date.now();
const keepDays = retentionDays(env);
const cutoff = retentionCutoff(atMs, keepDays);
const rolled: string[] = [];
const failed: string[] = [];
let rows = 0;
for (let back = 1; back <= ROLLUP_LOOKBACK_DAYS; back++) {
const day = utcDay(atMs - back * DAY_MS);
try {
rows += (await rollupDay(env, day, { cutoff })).rows;
rolled.push(day);
} catch (err) {
failed.push(day);
console.error(JSON.stringify({ msg: 'rollup day failed', day, err: String(err) }));
}
}
let purge: PurgeResult | null = null;
try {
purge = await purgeOldEvents(env, cutoff);
} catch (err) {
console.error(JSON.stringify({ msg: 'purge failed', cutoff, err: String(err) }));
}
console.log(
JSON.stringify({
msg: 'nightly rollup',
days: rolled,
rows,
failed: failed.length,
retention_days: keepDays,
purged_before: cutoff,
purged: purge?.deleted ?? null,
purge_batches: purge?.batches ?? null,
purge_capped: purge?.capped ?? null,
ms: Date.now() - started,
}),
);
if (failed.length > 0 || purge === null) {
throw new Error(`nightly rollup incomplete: ${failed.length} day(s) failed, purge ${purge ? 'ok' : 'failed'}`);
}
}
// ---------------------------------------------------------------------------
// POST /admin/rollup — manual backfill / repair
// ---------------------------------------------------------------------------
const json = (body: unknown, status = 200): Response =>
new Response(JSON.stringify(body), {
status,
headers: { 'content-type': 'application/json; charset=utf-8' },
});
/** Constant-time over digests, so neither the length nor a prefix of the token leaks. */
async function tokenMatches(provided: string, expected: string): Promise<boolean> {
const encoder = new TextEncoder();
const [a, b] = await Promise.all([
crypto.subtle.digest('SHA-256', encoder.encode(provided)),
crypto.subtle.digest('SHA-256', encoder.encode(expected)),
]);
return crypto.subtle.timingSafeEqual(a, b);
}
/**
* `POST /admin/rollup?day=YYYY-MM-DD[&days=N][&reset=1]`, header `x-admin-token`.
*
* Re-runs the rollup for `day` (default: yesterday), or for the `N` days ending on it.
* Exists so a backfill or a repair never needs a redeploy. It only ever recomputes
* aggregates from stored rows — there is no path here that deletes raw events; the
* purge runs on the cron and nowhere else.
*/
export async function handleAdminRollup(request: Request, env: Env, url: URL): Promise<Response> {
// No secret configured ⇒ no admin surface at all, and nothing that hints there is one.
const expected = env.ADMIN_TOKEN;
if (typeof expected !== 'string' || expected.length === 0) {
return new Response('not found\n', { status: 404 });
}
if (request.method !== 'POST') {
return new Response('method not allowed\n', { status: 405, headers: { allow: 'POST' } });
}
if (!(await tokenMatches(request.headers.get('x-admin-token') ?? '', expected))) {
// Cap how fast the token can be guessed at. Only failures spend the budget, so a
// chunked backfill loop is never throttled. Best-effort and fails open like the
// ingest limiter — the token itself is the guard, this only slows a guesser down.
try {
const { success } = await env.ADMIN_RATE_LIMITER.limit({ key: 'admin' });
if (!success) return new Response('rate limited\n', { status: 429 });
} catch (err) {
console.error(JSON.stringify({ msg: 'rate limiter unavailable', err: String(err) }));
}
return new Response('unauthorized\n', { status: 401 });
}
const now = Date.now();
const day = url.searchParams.get('day') ?? utcDay(now - DAY_MS);
if (!isValidDay(day)) return json({ error: 'day must be YYYY-MM-DD' }, 400);
const requested = url.searchParams.get('days');
const span = requested === null ? 1 : Number(requested);
if (!Number.isInteger(span) || span < 1 || span > MAX_MANUAL_DAYS) {
return json({ error: `days must be an integer between 1 and ${MAX_MANUAL_DAYS}` }, 400);
}
const reset = url.searchParams.get('reset') === '1';
const cutoff = retentionCutoff(now, retentionDays(env));
const endMs = Date.parse(`${day}T00:00:00Z`);
const days: DayResult[] = [];
try {
for (let back = span - 1; back >= 0; back--) {
days.push(await rollupDay(env, utcDay(endMs - back * DAY_MS), { cutoff, reset }));
}
} catch (err) {
console.error(JSON.stringify({ msg: 'manual rollup failed', through: day, err: String(err) }));
return json({ error: 'rollup failed', through: day, completed: days }, 500);
}
const rows = days.reduce((total, d) => total + d.rows, 0);
// A day past the window kept its rollups but ignored the reset — say so rather than
// reporting a repair that did not happen.
const resetIgnored = reset ? days.filter((d) => d.pastRetention).map((d) => d.day) : [];
console.log(
JSON.stringify({
msg: 'manual rollup',
through: day,
days: span,
reset,
reset_ignored: resetIgnored.length,
rows,
ms: Date.now() - now,
}),
);
return json({ ok: true, through: day, retention_cutoff: cutoff, rows, reset_ignored: resetIgnored, days });
}
telemetry-worker/wrangler.jsonc
+34 -3
View File
@@ -1,5 +1,7 @@
// codegraph telemetry ingest — see README.md and docs/design/telemetry.md.
// Secrets are NOT configured here: POSTHOG_KEY is set via `wrangler secret put POSTHOG_KEY`.
// Accepted events go straight into the bound D1 database and the worker makes no
// outbound requests. The only secret is ADMIN_TOKEN, which guards the manual rollup
// trigger (`wrangler secret put ADMIN_TOKEN`); leave it unset and that route 404s.
{
"$schema": "node_modules/wrangler/config-schema.json",
"name": "codegraph-telemetry",
@@ -15,8 +17,30 @@
"observability": { "enabled": true, "head_sampling_rate": 1 },
// Non-secret config. Swap host here if the backend ever moves (EU, self-hosted…).
"vars": { "POSTHOG_HOST": "https://us.i.posthog.com" },
// Nightly rollup + retention purge (src/rollup.ts). 00:30 UTC — half an hour after
// the day it rolls up closed, so straggling writes for it have landed. It also
// re-runs the two days before that, because offline clients ship completed-day
// rollups late; the writes are idempotent upserts, so re-running is free.
"triggers": { "crons": ["30 0 * * *"] },
// How many days of RAW events are kept. Rollups are kept forever, so shortening
// this costs ad-hoc drill-back, never a chart. 90 is a storage limit, not a policy
// one: raw events grow ≈74 MB/day, so 90 days ≈ 6.7 GB against D1's 10 GB
// per-database cap — the arithmetic is in migrations/0001_init.sql's footer.
// Measure real row size after cutover before widening it.
"vars": { "RETENTION_DAYS": 90 },
// Telemetry storage. Schema + the chart each table serves: migrations/0001_init.sql.
// Apply with `npm run db:migrate:local` (local state) / `npm run db:migrate` (remote).
// The admin dashboard worker binds this same database read-mostly, also as `DB`.
"d1_databases": [
{
"binding": "DB",
"database_name": "codegraph-telemetry",
"database_id": "5ed36dfb-d2d7-4e35-9e63-a1b99d0b1ed3",
"migrations_dir": "migrations"
}
],
// Per-machine_id rate limit. Legit clients flush a handful of times per day;
// 6/min absorbs install+index bursts while capping abuse.
@@ -25,6 +49,13 @@
"name": "MACHINE_RATE_LIMITER",
"namespace_id": "1001",
"simple": { "limit": 6, "period": 60 }
},
// POST /admin/rollup. The ADMIN_TOKEN secret is the real guard; this only caps
// how fast it can be guessed at, with enough room for a chunked backfill loop.
{
"name": "ADMIN_RATE_LIMITER",
"namespace_id": "1002",
"simple": { "limit": 10, "period": 60 }
}
]
}