Merge main into fix/union-declarations-not-indexed
Resolves the CHANGELOG conflict — main and this branch each prepended a bullet to [Unreleased] > Fixes; both are kept. Everything else auto-merged, including src/mcp/tools.ts, which main reworked heavily for the explore allocation/displacement work (CG-28/31/36/38) while this branch added the `union` kind to its container sets. Verified on the merged tree with the native kernel built: 3070 passed, 9 skipped, 0 failed.
This commit is contained in:
Colby McHenry
@@ -0,0 +1,164 @@
|
||||
# Deterministic measurement — cluster starvation inside one file (task CG-36)
|
||||
|
||||
**Date:** 2026-08-06 · **Baseline:** `feature/CG-24` @ `76ab1fe` (the CG-24 epic tip) ·
|
||||
**Harness:** `scripts/agent-eval/probe-file-spend.mjs` and `probe-suite-envelope.mjs` over the
|
||||
deterministic 6-repo corpus at `/tmp/codegraph-corpus`, clean full-rebuilt indexes (CG-33), plus
|
||||
two hermetic fixtures through `probe-allocation.mjs`. No agent A/B: the claim is which bytes go
|
||||
to which file, and the agent runs are far too noisy to see a 2K shift.
|
||||
|
||||
**Verdict: the defect is real, the diagnosis in the issue was half wrong, and the fix holds.**
|
||||
All 8 starvation flags across the suite clear; net **+1,012 source chars**. One repo (okhttp)
|
||||
trades its weakest file for +7,196 chars in the two that answer the question — stated in full
|
||||
below rather than smoothed over.
|
||||
|
||||
---
|
||||
|
||||
## The defect
|
||||
|
||||
A file's ranked clusters were all-or-nothing past the first one. The top-ranked cluster was
|
||||
always taken — shrunk to the highest-importance whole symbol ranges that fit when it overran —
|
||||
and every cluster below it was rendered **whole** and then either fit the remainder or was
|
||||
dropped entirely. On a file whose top-ranked cluster is trivial, that discards the answer.
|
||||
|
||||
What made it invisible: the response stays FULL. The unspent reservation carries forward exactly
|
||||
as CG-31 designed, so a lower-scoring file takes the bytes and every envelope-share measure still
|
||||
reads healthy. Measured on the epic tip:
|
||||
|
||||
| repo | file | score | reserved | spent | share |
|
||||
|---|---|---|---|---|---|
|
||||
| django | `db/models/sql/query.py` | 83 | 7,947 | 1,923 | **24%** |
|
||||
| django | `contrib/admin/filters.py` | 18 | 2,271 | 8,057 | **355%** |
|
||||
| okhttp | `.../RealInterceptorChain.kt` | 86 | 6,058 | 1,474 | **24%** |
|
||||
| okhttp | `.../CallServerInterceptor.kt` | 20 | 1,974 | 5,832 | **295%** |
|
||||
|
||||
A score-83 file spends a quarter of its reservation while a score-18 file takes 3.5× its own.
|
||||
|
||||
## What the issue got wrong
|
||||
|
||||
The issue named two candidate fix points and suspected the first: cluster ranking breaks ties on
|
||||
**density** (`score / span`) after `hasSpine` → `maxImportance`, which structurally favours a
|
||||
small trivial cluster over a large answer-bearing one. Dumping the cluster set says otherwise —
|
||||
in **both** real cases the loser lost on `maxImportance`, not on density:
|
||||
|
||||
```
|
||||
django/db/models/sql/query.py budget 10,135 spent 1,923
|
||||
KEPT 1379–1400 span 22 score 14 maxImp 6 (check_related_objects — a glue symbol)
|
||||
DROPPED 306– 929 span 624 score 290 maxImp 3 (133 members: the whole Query class)
|
||||
|
||||
okhttp .../RealInterceptorChain.kt budget 6,058 spent 1,474
|
||||
KEPT 16– 44 span 29 score 44 maxImp 6 (package decl + import block)
|
||||
DROPPED 113– 373 span 261 score 171 maxImp 3 (73 members: the chain itself)
|
||||
```
|
||||
|
||||
`maxImportance` first is deliberate and protective — it is what stops Alamofire's `Session.swift`
|
||||
from losing its budget to the top-of-file property list — so **ranking was not touched**. The
|
||||
lever is the second fix point: stop dropping the loser whole.
|
||||
|
||||
## The change
|
||||
|
||||
Two sites, one rule — *hold the remainder while it is still worth a section*, which is CG-26's
|
||||
between-FILES lesson applied between CLUSTERS.
|
||||
|
||||
1. **Selection.** A later cluster is now shrunk into what is left of the file's budget, by the
|
||||
same whole-member rule the first cluster already used. Below `MIN_CHARS` (700) the remainder
|
||||
cannot hold a readable block, so it stays a drop rather than a stutter of fragments the next
|
||||
call's dedup has to shred around. The never-empty windowing floors may overrun that room; the
|
||||
first cluster is allowed that overshoot, a later one is not.
|
||||
2. **The ceiling trim.** When the exact section cost overruns `renderCeiling`, the weakest chosen
|
||||
cluster is re-rendered into the room that remains before being dropped. This one is worth
|
||||
naming on its own: on excalidraw's `typeChecks.ts` the section-cost estimate missed by
|
||||
**13 chars** and a 1,512-char cluster — the file's highest-*scoring* one, last in rank order
|
||||
only because rank breaks ties on density — was thrown away to pay for it. Recovered 1,501 of
|
||||
excalidraw's 1,449-char loss.
|
||||
|
||||
## Suite result
|
||||
|
||||
`node scripts/agent-eval/probe-file-spend.mjs`, 6 repos, clean rebuilds. Only files whose spend
|
||||
moved are listed; score is the candidate's ranking score, reserved its allocation.
|
||||
|
||||
| repo | file | score | reserved | before | after |
|
||||
|---|---|---|---|---|---|
|
||||
| django | `db/models/sql/query.py` | 83 | 7,947 | 1,923 | **10,082** |
|
||||
| django | `contrib/admin/filters.py` | 18 | 2,271 | 8,057 | 2,198 |
|
||||
| django | `utils/autoreload.py` | 12 | 1,747 | 3,145 | 1,709 |
|
||||
| django | `db/models/fields/related_descriptors.py` | 11 | 1,660 | 2,516 | 1,493 |
|
||||
| excalidraw | `element/src/typeChecks.ts` | 23 | 2,740 | 3,102 | 2,573 |
|
||||
| excalidraw | `excalidraw/types.ts` | 14 | 1,942 | 819 | 1,372 |
|
||||
| okhttp | `.../RealInterceptorChain.kt` | 86 | 6,058 | 1,474 | **6,038** |
|
||||
| okhttp | `.../Interceptor.kt` | 64 | 4,697 | 2,027 | **4,659** |
|
||||
| okhttp | `.../RealCall.kt` | 52 | 3,972 | 3,628 | 3,922 |
|
||||
| okhttp | `.../Call.kt` | 54 | 4,097 | 4,097 | 2,073 |
|
||||
| okhttp | `.../CallServerInterceptor.kt` | 20 | 1,974 | 5,832 | 1,959 |
|
||||
| okhttp | `androidMain/.../AndroidDns.kt` | 21 | 1,999 | 1,812 | **0** |
|
||||
| tokio | `task/local.rs` | 40 | 4,361 | 4,599 | 4,798 |
|
||||
| tokio | `runtime/task/harness.rs` | 14 | 1,981 | 2,565 | 2,341 |
|
||||
| gin | `routergroup.go` | 87 | 5,782 | 3,273 | **5,632** |
|
||||
| gin | `tree.go` | 17 | 1,693 | 892 | 1,969 |
|
||||
| gin | `ginS/gins.go` | 26 | 2,213 | 4,431 | 2,171 |
|
||||
| alamofire | `Source/Core/Session.swift` | 34 | 2,792 | 2,797 | 3,396 |
|
||||
| alamofire | `Source/Core/Request.swift` | 148 | 9,100 | 8,865 | 8,453 |
|
||||
|
||||
Bytes move up the score order in every repo. Envelope totals:
|
||||
|
||||
| repo | source before | after | Δ | files | ceiling |
|
||||
|---|---|---|---|---|---|
|
||||
| django | 20,878 | 20,719 | −159 | 6 → 6 | 24,963 ≤ 25,000 |
|
||||
| excalidraw | 19,652 | 19,704 | +52 | 8 → 8 | 24,813 ≤ 25,000 |
|
||||
| okhttp | 18,870 | 18,651 | −219 | 6 → **5** | 24,985 ≤ 25,000 |
|
||||
| tokio | 21,607 | 21,582 | −25 | 5 → 5 | 24,777 ≤ 25,000 |
|
||||
| gin | 10,776 | 11,952 | **+1,176** | 4 → 4 | 14,655 ≤ 19,500 |
|
||||
| alamofire | 11,662 | 11,849 | +187 | 2 → 2 | 12,862 ≤ 19,500 |
|
||||
| **total** | **103,445** | **104,457** | **+1,012** | | |
|
||||
|
||||
Starvation flags: **8 → 0**.
|
||||
|
||||
## The one cost, stated plainly
|
||||
|
||||
okhttp drops its rank-6 file, `androidMain/.../AndroidDns.kt` (score 21, a platform DNS helper on
|
||||
a question about the interceptor chain), and 219 source chars, in exchange for +4,564 to
|
||||
`RealInterceptorChain.kt` and +2,632 to `Interceptor.kt` — the two files that answer the question.
|
||||
|
||||
This is not a new defect and it is not the fix over-reaching. okhttp's reservations are
|
||||
**structurally over-subscribed**: the allocator splits `maxOutputChars` charging a flat
|
||||
`FILE_OVERHEAD` of 200 per file while a real header runs 300–500, so the sum of promises
|
||||
(~22,800 source + ~2,100 of real headers) exceeds what the ~24,760-char render ceiling can hold.
|
||||
`owedPayableBelow` already refuses to hold bytes back for a file it can see will be dropped, and
|
||||
AndroidDns.kt is the file past that line. On the epic tip it survived only because the files above
|
||||
it under-spent — by luck, not by design. Closing the over-subscription means charging the
|
||||
allocator per-file header estimates rather than the flat 200; that is a wider change than this
|
||||
issue, and CG-26 deliberately kept `FILE_OVERHEAD` as the allocator's own constant.
|
||||
|
||||
## What did NOT change
|
||||
|
||||
- **Cluster ranking.** `hasSpine` → `maxImportance` → density → score → span, untouched.
|
||||
- **Alamofire `Session.swift`.** The shape density-first exists for; it *gains* 599 chars.
|
||||
- **The factory-closure outcome (CG-27).** `probe-factory-closure.mjs`: 7 of 11 inner closure
|
||||
definitions delivered, identical to the epic tip.
|
||||
- **The reservation invariant (CG-31/CG-26).** `explore-reservation-invariant.test.ts` green;
|
||||
every repo stays at or under its hard ceiling.
|
||||
- **All four allocation fixtures pass** — `payroll-go`, `self-query`, and the two added here.
|
||||
|
||||
## What ships so this stays measurable
|
||||
|
||||
- `scripts/agent-eval/probe-file-spend.mjs` — the standing per-file reservation-vs-delivered
|
||||
sweep. It flags a **pair**, never a single file: a large share unspent *while* a materially
|
||||
lower-scoring file overspends. Either alone is legitimate (a small file has less to say;
|
||||
carry-forward is the mechanism that hands its slack down), which is why the envelope probe
|
||||
could never see this. Exit code 1 on any flag, so it gates.
|
||||
- `__tests__/fixtures/starved-cluster-ts/` — django's and okhttp's shape reduced to a fixture.
|
||||
Fails on the epic tip (28.8% of reservation, neither `proceed` nor `writeAndRead` delivered),
|
||||
passes with the fix.
|
||||
- `__tests__/fixtures/dense-header-ts/` — the `Session.swift` shape, byte-identical on both
|
||||
builds. The counterweight: it fails if a future change lets density outrank importance again.
|
||||
- `spendShareAtLeast` in `probe-allocation.mjs`, and `__tests__/explore-cluster-starvation.test.ts`
|
||||
pinning both fixtures in `npm test`.
|
||||
|
||||
## Method note
|
||||
|
||||
None of this is visible in the rendered markdown. To see it you must dump the cluster set —
|
||||
patch `dist/mcp/tools.js` just before `let assembled = assembleSection(chosenIndices);` and log
|
||||
`fileBudget` / `projectedChars` / each ranked cluster's span, score, `maxImportance`, chosen flag
|
||||
and members. Reading only the response makes member-selection effects look like budget effects.
|
||||
Equally, a source-chars diff between builds is not automatically a regression: excalidraw's
|
||||
−1,449 on the first cut was the elastic epilogue expanding into room a 13-char accounting error
|
||||
had released, not source lost to allocation.
|
||||
@@ -0,0 +1,149 @@
|
||||
# Deterministic measurement — declaration-only files in the explore envelope (task CG-28)
|
||||
|
||||
**Date:** 2026-08-06 · **Baseline:** `feature/CG-24` @ `463f6e7` ·
|
||||
**Harness:** `scripts/agent-eval/probe-decl-only.mjs` against a hermetic fixture
|
||||
(`__tests__/fixtures/ambient-decls-ts/`, copied to a temp dir and indexed per run, so two runs on
|
||||
one build give identical numbers), plus `probe-suite-envelope.mjs` and a corpus-wide flag-rate
|
||||
survey for the regression side. No agent A/B: the claim under test is which FILES get selected and
|
||||
in what order, and the agent runs are far too noisy to see that.
|
||||
|
||||
**Verdict, both halves:**
|
||||
|
||||
- **The motivating file is already handled — CG-25 credited.** The Wrangler `worker-configuration.d.ts`
|
||||
that opened this issue is demoted by the generated penalty alone, worth 15–46 points of envelope
|
||||
share on the four flow queries measured. No new mechanism needed for it.
|
||||
- **The narrower gap is real and was fixed.** A declaration file with NO banner carried `pen 1.00`,
|
||||
took **rank #1 and 51% of delivered source** on a prose flow query, and displaced the flow's own
|
||||
entry file out of the response entirely. It is now damped — but only when nothing in the index
|
||||
depends on it, which is the condition that makes the rule safe.
|
||||
|
||||
---
|
||||
|
||||
## The fixture
|
||||
|
||||
`__tests__/fixtures/ambient-decls-ts/` — an upload path (route → stream → metadata → queue) with
|
||||
four declaration-shaped files competing against it for one envelope. All four declare nothing but
|
||||
`interface`/`type_alias` and have no bodies; they differ only in the two properties under test.
|
||||
|
||||
| file | banner | depended on | lines |
|
||||
|---|---|---|---|
|
||||
| `types/worker-configuration.d.ts` | Wrangler | no | 271 |
|
||||
| `types/platform-shims.d.ts` | none | no | 212 |
|
||||
| `src/storage/types.ts` | none | **yes** (2 imports, 3 references) | 18 |
|
||||
| implementation (`routes/`, `storage/`, `lib/`) | — | — | 37–71 each |
|
||||
|
||||
The declaration files carry the same generic identifiers the prose queries use — `Body`, `Message`,
|
||||
`ImageMetadata`, `ReadableStream`, `Upload*` — which is the whole mechanism of the original report.
|
||||
|
||||
## Result 1 — what CG-25 is worth (the obsolescence leg)
|
||||
|
||||
Same fixture, same queries, one variable: `--variant strip-banner` deletes the two banner COMMENT
|
||||
lines from `worker-configuration.d.ts` and changes nothing else, so the two declaration files become
|
||||
indistinguishable to the ranker. Delivered share of the envelope for that file:
|
||||
|
||||
| query | with banner | banner stripped |
|
||||
|---|---|---|
|
||||
| flow-upload | not a candidate | 15.1% (2,271 chars) |
|
||||
| flow-pipe | not a candidate | 38.5% (4,398 chars) |
|
||||
| flow-generic | cliffed to a pointer, 0 chars | 35.1% (3,076 chars) |
|
||||
| flow-queue | 9.4% via clusters (1,264 chars) | **46.1%, rank #1, whole file** (7,390 chars) |
|
||||
|
||||
The generated penalty alone is the difference between "rank #1 and nearly half the answer" and
|
||||
"named in the not-shown list". **The file this issue was filed about needs nothing further.**
|
||||
|
||||
## Result 2 — the gap that survived
|
||||
|
||||
`platform-shims.d.ts` — hand-written, no banner — on the `feature/CG-24` tip:
|
||||
|
||||
| query | rank | score | pen | delivered |
|
||||
|---|---|---|---|---|
|
||||
| flow-upload | **#1** | 53.0 | 1.00 | 6,044 chars (**50.7%**) |
|
||||
| flow-queue | **#1** | 21.0 | 1.00 | 6,044 chars (44.9%) |
|
||||
|
||||
On `flow-upload` the response carried three files and `src/routes/upload.ts` — the handler the
|
||||
question is *about* — was not one of them. That is the CG-24 epic symptom, reproduced with no
|
||||
generated banner anywhere in it.
|
||||
|
||||
Note also: `.pyi` is **not an indexed extension**, so Python stubs never enter the graph and cannot
|
||||
take an envelope. That third of the issue's premise does not occur today.
|
||||
|
||||
## The mechanism, and why it is drawn this tight
|
||||
|
||||
`AMBIENT_DECLARATION_RANK_PENALTY` (0.5) multiplies score and graph mass in `rankPenalty`, for files
|
||||
`QueryBuilder.getAmbientDeclarationPathsAmong` flags. Four conditions, all required — the first
|
||||
three were the obvious rule, the fourth is the one that makes it safe:
|
||||
|
||||
1. declares ≥1 symbol;
|
||||
2. **every** declared symbol is type-level (`interface`, `type_alias`, `enum`, `enum_member`,
|
||||
`namespace`);
|
||||
3. originates no `calls`/`instantiates` edge;
|
||||
4. **nothing outside the file points at it.**
|
||||
|
||||
Conditions 2 and 4 were both forced by measurement, not taste:
|
||||
|
||||
**Why not just "no callables" (condition 2).** Surveyed across the corpus, a rule of "declares no
|
||||
callable and calls nothing" flags **1.1%–18.0%** of files, and what it catches is real source:
|
||||
okhttp's `SocketPolicy.kt` (19 declarations, a Kotlin sealed hierarchy), `BrotliInterceptor.kt`,
|
||||
`tokio/src/runtime/mod.rs`, Alamofire's umbrella `Alamofire.swift`, and all 500+ of django's
|
||||
`conf/locale/*/formats.py` constant tables. Requiring every symbol to be type-level drops that to
|
||||
**0%–4%**.
|
||||
|
||||
**Why "nothing depends on it" (condition 4).** Without it the rule also flags
|
||||
`__tests__/fixtures/displacement-ts/src/pipeline/types.ts` — pure interfaces, no bodies, structurally
|
||||
identical to an ambient shim — and demoting it **broke the CG-31 displacement gate**, which is a
|
||||
different invariant entirely. That file carries 13 inbound imports and 21 references: the pipeline
|
||||
stages that answer a query about the pipeline are typed *by* it, so it is part of that answer's
|
||||
structure. The ambient shims carry **zero** inbound edges — reachable by name, attached to nothing.
|
||||
That is the real distinction, and the graph already holds it.
|
||||
|
||||
**The counter-case guard.** A query that NAMES a declared type is a question about the declaration,
|
||||
so its file is exempt and ranks at full weight. Only shape-precise tokens count (the same
|
||||
NL-stopword reasoning as named-seed selection) — "…the file **body**…" must not exempt a `Body`
|
||||
interface it never meant to name. This needed its own set: `namedSeedIds` is callable-only by
|
||||
construction, so a type can never become a named seed.
|
||||
|
||||
**No double-charging.** Generated and ambient-declaration are combined with `Math.min`, not
|
||||
multiplied. A generated `.d.ts` has one property that two signals happen to see; charging it twice
|
||||
(0.3 × 0.5 = 0.15) is how a file gets cliffed out of answers where it is genuinely relevant. The
|
||||
low-value multiplier is orthogonal and still compounds.
|
||||
|
||||
## Result 3 — after the fix
|
||||
|
||||
| query | before | after |
|
||||
|---|---|---|
|
||||
| flow-upload | rank **#1**, 50.7% | rank **#2**, 38.6% — and `src/routes/upload.ts` now delivered (2,417 chars) |
|
||||
| flow-queue | rank **#1**, 44.9% | rank **#3**, 41.3% |
|
||||
| flow-pipe / flow-generic | not a candidate | unchanged |
|
||||
| type-shim (`UploadStorage StoredUploadObject ImageMetadataShim`) | rank #1, `pen 1.00` | **unchanged** — exempt |
|
||||
| type-prose (*what does the UploadStorage interface declare…*) | rank #1, `pen 1.00` | **unchanged** — exempt |
|
||||
|
||||
The byte share falls less than the rank does, and that is the correct outcome rather than a weak
|
||||
fix: on this fixture every implementation file already delivers its entire contents, so the
|
||||
declaration file is filling envelope nobody else needs. What it was actually taking was a **file
|
||||
slot** — which is why the entry file came back. The issue explicitly forbids suppression, and a
|
||||
damped file is still a candidate, still named in the response, and one follow-up explore away.
|
||||
|
||||
## Regression evidence
|
||||
|
||||
- **`probe-suite-envelope.mjs`, 6 repos, new build vs a clean `feature/CG-24` baseline build:
|
||||
byte-identical.** django 20,878 · excalidraw 19,652 · okhttp 18,870 · tokio 21,607 · gin 10,776 ·
|
||||
alamofire 11,662 source chars, same file counts, on both builds.
|
||||
- **VS Code** — the repo the issue names for `.d.ts` surface — across five flow and type queries:
|
||||
**zero** ambient-declaration files reach the ranked candidate set, so the output cannot differ.
|
||||
- **Corpus-wide flag rate:** django 0.00% · okhttp 0.00% · gin 0.00% · alamofire 0.00% ·
|
||||
tokio 0.12% · vscode 0.53% · excalidraw 0.74%. What it catches is `global.d.ts`, `vite-env.d.ts`,
|
||||
`css.d.ts`, unreferenced vendored headers and test fixtures — exactly the intended shape.
|
||||
- `probe-allocation.mjs`: `payroll-go` PASS, `self-query` PASS.
|
||||
- Full suite: **178 files, 2,978 passed**, 6 skipped.
|
||||
|
||||
The change is inert everywhere the shape does not occur, which is most places. That is the point:
|
||||
the defect is real but rare, and the mechanism costs nothing where it does not apply.
|
||||
|
||||
## Reproducing
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
node scripts/agent-eval/probe-decl-only.mjs # as committed
|
||||
node scripts/agent-eval/probe-decl-only.mjs --variant strip-banner # what CG-25 is worth
|
||||
npx vitest run __tests__/explore-declaration-only.test.ts # the standing gate
|
||||
```
|
||||
@@ -0,0 +1,140 @@
|
||||
# Agent A/B — cluster-path displacement guard (task CG-31)
|
||||
|
||||
**Date:** 2026-08-06 · **New:** `bugfix/CG-31` · **Baseline:** `bugfix/CG-30` @ `0d014a6` ·
|
||||
**Harness:** `scripts/agent-eval/ab-new-vs-baseline.sh`, `--model sonnet --effort high`,
|
||||
**both arms codegraph-on**, CLI blocked (0 contamination in every run),
|
||||
`CODEGRAPH_NO_PROMPT_HOOK=1`. Every index measured on was **fully rebuilt**, never
|
||||
incrementally synced (CG-33).
|
||||
|
||||
Baseline is the CG-30 tip, not `main`, so every number here isolates CG-31. CG-30's own A/B
|
||||
against `main` is `explore-oversize-member-ab-cg30.md`; read them in sequence for the combined
|
||||
picture the two issues asked for.
|
||||
|
||||
CG-31 stops a clustered render from spending a reservation still owed to a file the render loop
|
||||
has not reached. The whole-file BUY arm has always refused that trade (`owedBelow`); the cluster
|
||||
path read what was left before the hard ceiling instead of what was still promised.
|
||||
|
||||
**Verdict: no regression, and this one is a straight win on both halves.** Four of six suite
|
||||
repos deliver MORE source and one more file each; the other two are byte-identical. The agent
|
||||
runs are faster in all three repos measured, with Read at or below baseline.
|
||||
|
||||
---
|
||||
|
||||
## The two corrections the measurement forced
|
||||
|
||||
Worth recording, because the first version of the guard was **wrong in the direction the guard
|
||||
itself is about**, and only a suite measurement showed it.
|
||||
|
||||
**1. Holding back the full owed sum was too much.** The allocator splits the envelope; the render
|
||||
loop spends against a ceiling that also has to hold the response's own prose, so on a saturated
|
||||
response the promises are over-subscribed and the tail is going to be dropped whatever happens
|
||||
above it. Bytes held for a file that is then dropped are bytes nobody receives. Measured: django
|
||||
−2,319 source, tokio −1,298, both handed to a section the ceiling threw away.
|
||||
`owedPayableBelow` now holds back only the prefix of what is owed below that the response can
|
||||
still pay, in rank order.
|
||||
|
||||
**2. The final truncation was eating the guard's work.** It cut at the last file-section header,
|
||||
which drops that whole section *and* the trailing notes. Dropping the notes alone is almost
|
||||
always enough. The epilogue is a pointer list and two reminders; a section is source the agent
|
||||
otherwise has to Read. Cutting the epilogue first is what turned the remaining deficits into
|
||||
gains — and it is the same starvation CG-31 is about, arriving one layer below the guard.
|
||||
|
||||
A third, smaller fix: `flow.text` is prepended to `lines` to make the final output but was never
|
||||
counted in `totalChars`, so the render loop spent against a ceiling it was ~2K under on
|
||||
symbol-bag queries.
|
||||
|
||||
## Deterministic measurement — the primary evidence
|
||||
|
||||
Same clean-rebuilt index, same query, both builds. One `codegraph_explore` per repo.
|
||||
|
||||
| repo | base source | new source | Δ | base files | new files |
|
||||
|---|---|---|---|---|---|
|
||||
| django | 20,033 | **20,791** | +758 | 5 (truncated) | **6** |
|
||||
| excalidraw | 18,776 | **20,204** | +1,428 | 7 (truncated) | **8** |
|
||||
| okhttp | 15,628 | **19,034** | +3,406 | 4 (truncated) | **5** |
|
||||
| tokio | 20,340 | **21,521** | +1,181 | 4 (truncated) | **5** |
|
||||
| gin | 10,776 | 10,776 | 0 | 4 | 4 |
|
||||
| alamofire | 11,662 | 11,662 | 0 | 2 | 2 |
|
||||
|
||||
Queries: django "How does a QuerySet turn into SQL and fetch rows from the database?";
|
||||
excalidraw "How does updating an element re-render the canvas on screen?"; gin "How does a
|
||||
registered route handler get invoked for an incoming HTTP request?"; alamofire "How does a
|
||||
request get built and sent through the session?"; okhttp "How does a call go through the
|
||||
interceptor chain to the network?"; tokio "How does a spawned task get scheduled and run by a
|
||||
worker?".
|
||||
|
||||
No repo delivers less. **Four of six stopped truncating**, which is where the extra file comes
|
||||
from: each of those responses had been throwing a fully-rendered section away.
|
||||
|
||||
**gin and alamofire are byte-identical between the builds** — nothing in them is oversize enough
|
||||
for the guard to engage and neither response was truncated. That is what a control should show,
|
||||
and it means every gin number in the agent table below is run-to-run variance.
|
||||
|
||||
**Fixture** — `__tests__/fixtures/displacement-ts`, four pipeline stages competing for one
|
||||
envelope, the first a single ~20K function. Padded past 500 indexed files on purpose: the
|
||||
displacement only exists on the 24K tier, where the reservations plus the preamble genuinely
|
||||
saturate the render ceiling.
|
||||
|
||||
| | baseline | new |
|
||||
|---|---|---|
|
||||
| `ingest.ts` | 9,301 chars on a 6,289 spendable, then **dropped whole** by the ceiling — 0 delivered | 4,851, bounded |
|
||||
| `types.ts` / `sink.ts` | skipped `budget-whole-file` | delivered |
|
||||
| admitted files delivered | **3 of 6** | **6 of 6** |
|
||||
| envelope | 14,908 | 22,066 |
|
||||
|
||||
Pinned by `__tests__/explore-displacement-guard.test.ts` (11 tests; 3 fail on the baseline).
|
||||
|
||||
**Allocation fixtures** — `scripts/agent-eval/allocation-fixtures.json` flips back to
|
||||
**BOTH PASS**. Its `afterCG30` verdict blamed an over-RESERVED incidental file; the reservation
|
||||
is identical in both arms — the file was over-SPENDING, which is exactly this defect. Recorded
|
||||
honestly in `afterCG31`.
|
||||
|
||||
## Agent runs
|
||||
|
||||
| | django new | django base | okhttp new | okhttp base | gin new | gin base |
|
||||
|---|---|---|---|---|---|---|
|
||||
| runs | 3 | 3 | 2 | 2 | 2 | 2 |
|
||||
| duration (s) | **36** [33–51] | 46 [35–49] | **42** [40–44] | 52 [50–53] | **33** [31–35] | 39 |
|
||||
| tool calls | 3 [3–4] | 3 [3–4] | **4** [3–4] | 5 [4–5] | **4** [3–4] | 5 [4–5] |
|
||||
| Read | 0 [0–1] | 0 | **0** | 1 [0–2] | 1 [0–1] | 1 [0–2] |
|
||||
| Grep/Glob | 0 | 0 | 0 | 0 | 0 | 0 |
|
||||
| codegraph calls | 2 | 2 [2–3] | 3 [2–3] | 3 [2–3] | 2 | 3 [2–3] |
|
||||
| occupancy share | **32.1%** [31.1%–36.1%] | 33.8% [28.7%–42.1%] | **40.0%** [36.6%–43.3%] | 40.8% [40.3%–41.4%] | **29.7%** [28.3%–31.1%] | 33.5% [29.8%–37.2%] |
|
||||
| allocation efficiency | 96.8% | 98.9% | 88.2% | 97.2% | **91.2%** | 85.0% |
|
||||
|
||||
Prompts are the deterministic queries above with "Trace the flow end to end." appended.
|
||||
|
||||
**Sufficiency, pooled per call.** okhttp: **0** "Read a file we returned" in 5 against the
|
||||
baseline's 1 in 5 — the arm that returns 3,406 more chars needs fewer follow-up Reads, which is
|
||||
the mechanism working. django: 1 in 6 against 0 in 7. gin: 1 in 4 against 1 in 5, on a repo where
|
||||
the builds emit identical bytes. Neither arm produced a single recall miss (a Read of a file we
|
||||
did NOT return, or a Grep) on any repo.
|
||||
|
||||
**Where the new arm looks worse, and why it is not read as a regression:**
|
||||
|
||||
- *okhttp allocation efficiency, 88.2% vs 97.2%.* The new arm's envelope is 85,197 chars against
|
||||
the baseline's 66,014 — it returns substantially more source, and the metric is the share of
|
||||
returned bytes the answer *cited*. A larger, more complete response with a smaller cited share
|
||||
and Read driven to 0 is the trade this tool exists to make. The metric's own documentation says
|
||||
it is relative and must not be read as waste.
|
||||
- *django, 1 allocation miss in 6 answered calls against 0 in 7.* One run, n=3, and django is the
|
||||
repo whose duration range overlaps most (33–51 vs 35–49).
|
||||
|
||||
## Residual carried forward — for CG-26
|
||||
|
||||
Four repos stopped truncating; **okhttp, django, excalidraw and tokio now land at 24,758–24,998
|
||||
chars against a 25,000 hard ceiling.** That is deliberate (the ceiling exists so the host never
|
||||
externalizes the result) but it means the render loop's 600-char margin for the epilogue is still
|
||||
wrong — the epilogue measures 1,064 (gin), 1,788 (django), 2,231 (excalidraw). The response now
|
||||
survives that by dropping the epilogue rather than a section, which is strictly better, but the
|
||||
honest fix is for the loop to budget for the epilogue in the first place.
|
||||
|
||||
A margin sweep was run and deliberately **not** shipped: at 1,200 django stops truncating on its
|
||||
own but tokio loses 286 chars; at 2,400 django loses 1,895. Tuning one constant against the suite
|
||||
is the trap CG-30's own record warns about. Sizing the margin from the epilogue the response is
|
||||
actually going to emit is the real fix and belongs with the end-to-end reservation invariant.
|
||||
|
||||
Second residual: the whole-file BUY arm's fit test (`totalChars + fileContent.length +
|
||||
FILE_OVERHEAD <= renderCeiling`) has no `owedBelow` term of its own — its displacement guard is
|
||||
source-space only. It was left alone here to keep this change attributable; the epilogue-first cut
|
||||
removes the failure mode it would have caused.
|
||||
@@ -0,0 +1,133 @@
|
||||
# Deterministic measurement — the factory-closure envelope (task CG-27)
|
||||
|
||||
**Date:** 2026-08-06 · **Baseline:** `feature/CG-24` @ `dc4fd75` ·
|
||||
**Harness:** `scripts/agent-eval/probe-factory-closure.mjs` against a hermetic fixture
|
||||
(`__tests__/fixtures/factory-closure-ts/`, copied to a temp dir and indexed per run, so two runs
|
||||
on one build give identical numbers). No agent A/B: the claim under test is which SYMBOLS get
|
||||
selected inside one file, and the agent runs are far too noisy to see that.
|
||||
|
||||
**Verdict: the premise does not survive measurement. CG-27 is closed as obsolete, CG-30 credited.**
|
||||
The literal change the issue proposes is a large REGRESSION, and a more careful mechanism reaching
|
||||
the same intent is noise (69 vs 68 inner definitions delivered across nine query shapes).
|
||||
|
||||
---
|
||||
|
||||
## The claim
|
||||
|
||||
`ENVELOPE_KINDS` in `src/mcp/tools.ts` drops a node covering >50% of its file from the cluster
|
||||
ranges, so the granular symbols inside form their own clusters instead of merging into one blob.
|
||||
It lists container kinds — `class`, `struct`, `interface`, `enum`, … — and **not `function` or
|
||||
`method`**. A factory closure (`createFoo()` returning an object of closures) therefore survives
|
||||
as a file-spanning range. That shape is common, not a one-repo quirk: Svelte 5 `.svelte.ts` rune
|
||||
stores, React custom-hook modules, IIFE/module-pattern JS, and Zustand's
|
||||
`create((set, get) => ({ … }))`.
|
||||
|
||||
CG-30 already bounds the BYTES such a member may spend, so what remained was a ranking claim:
|
||||
a file-spanning range merges every inner symbol into one cluster, so selection cannot rank and
|
||||
pick the relevant closures independently. The issue required that claim be measured before any fix.
|
||||
|
||||
## The fixture
|
||||
|
||||
`__tests__/fixtures/factory-closure-ts/` — a dashboard app with three stores written as factory
|
||||
closures, two stateless services and a UI consumer competing for one envelope.
|
||||
|
||||
| file | lines | shape |
|
||||
|---|---|---|
|
||||
| `src/stores/dashboard-store.ts` | 385 | `createDashboardStore` spans 15–376 (**94%**), 11 closures inside; a tail type alias + helper at file scope |
|
||||
| `src/stores/alerts-store.ts` | 141 | `createAlertsStore` spans 19–138 (**85%**), 9 closures inside |
|
||||
| `src/stores/session-store.ts` | 148 | a factory and NOTHING else at file scope — no companion type, no tail helper |
|
||||
| `src/services/metric-service.ts`, `src/services/filter-parser.ts` | 105, 62 | ordinary top-level functions — the control |
|
||||
|
||||
Both factory files are past `WHOLE_FILE_MAX_LINES` where it matters, so they render through the
|
||||
cluster path and the envelope actually bites.
|
||||
|
||||
## Result 1 — the envelope is almost never selected in the first place
|
||||
|
||||
`shrinkCluster` orders a cluster's members by **(importance desc, size ASC)** and refuses any
|
||||
member that overruns the cap once something is kept. A file-spanning member is therefore only ever
|
||||
selected when it is the FIRST candidate — which requires it to be the *sole* member of the top
|
||||
importance tier. In eight of the nine query shapes measured, some smaller member shared that tier
|
||||
(a one-line type alias, a tail helper, another closure), so the factory sorted last and was never
|
||||
kept. The envelope was inert.
|
||||
|
||||
## Result 2 — the proposed change is a large regression
|
||||
|
||||
Making the >50% drop kind-independent, measured on the primary query
|
||||
(*"how does the dashboard store refresh its metrics and apply a filter"*):
|
||||
|
||||
| | baseline | drop the range |
|
||||
|---|---|---|
|
||||
| `dashboard-store.ts` (rank #1) delivered | 7,539 chars | **397** |
|
||||
| inner closure definitions delivered | 7 of 11 | **0 of 11** |
|
||||
| its own reservation left unspent | 0 | ~5,200 of 5,601 |
|
||||
|
||||
The mechanism, from the cluster dump: dropping the range **splits** the file into two clusters —
|
||||
`378-384` (a one-line type alias plus a four-line helper, score 15, span 7) and `4-362` (every
|
||||
closure, score 116, span 359). Cluster ranking breaks the `maxImportance` tie on **density**, so
|
||||
the trivial cluster wins, is taken first, and is the only one that may be shrunk. The
|
||||
answer-bearing cluster then does not fit the remainder and is **dropped whole** — later clusters
|
||||
are never shrunk, by design.
|
||||
|
||||
The enclosing range is what was holding the file together as one cluster, inside which
|
||||
`shrinkCluster` was already doing exactly the per-symbol ranking the issue asked for.
|
||||
|
||||
## Result 3 — the careful version of the same intent is noise
|
||||
|
||||
Deferring the envelope MEMBER inside `shrinkCluster` (leaving clustering granularity untouched, so
|
||||
Result 2's split never happens) reaches the issue's intent by a better mechanism. Nine query
|
||||
shapes, same fixture, same indexes — inner closure definitions delivered:
|
||||
|
||||
| query | target | baseline | deferred |
|
||||
|---|---|---|---|
|
||||
| how does the dashboard store refresh its metrics and apply a filter | dashboard | 7/11 | **8/11** |
|
||||
| createDashboardStore | dashboard | 8/11 | 8/11 |
|
||||
| how is the dashboard store created and wired up | dashboard | **9/11** | 8/11 |
|
||||
| createDashboardStore exportCsv summarize | dashboard | 9/11 | 9/11 |
|
||||
| where is the dashboard store constructed | dashboard | 7/11 | 7/11 |
|
||||
| how are widgets loaded and the layout reconciled | dashboard | 4/11 | 4/11 |
|
||||
| createSessionStore (adverse: the factory IS the sole top-tier member) | alerts | 6/9 | **7/9** |
|
||||
| how are alerts refreshed and acknowledged | alerts | 9/9 | 9/9 |
|
||||
| createAlertsStore | alerts | 9/9 | 9/9 |
|
||||
| **total** | | **68** | **69** |
|
||||
|
||||
One better, one worse, seven unchanged — on a fixture built specifically to make this pattern
|
||||
maximally visible. That is not a measurable selection improvement, so nothing shipped.
|
||||
|
||||
## Where the envelope DOES get selected, and why CG-30 already covers it
|
||||
|
||||
The adverse row above is the one configuration the ordering cannot neutralise: `createAlertsStore`
|
||||
was the sole importance-10 member, so it was kept first at 3,939 chars against a 2,468 cap and
|
||||
every closure was skipped. CG-30 then **windowed it on whole lines** rather than emitting it whole
|
||||
or dropping the file — the response carried lines 16–108, a contiguous, readable head of the
|
||||
factory carrying 6 of its 9 closure definitions. Bounded, sufficient, never empty. That is the
|
||||
symptom this issue was filed against, already absorbed.
|
||||
|
||||
---
|
||||
|
||||
## Byproduct — a real defect this measurement exposed (filed separately)
|
||||
|
||||
Result 2's mechanism is not confined to the hypothetical change. Instrumenting the **epic tip**
|
||||
across the deterministic 6-repo suite for files that drop a cluster while leaving most of their
|
||||
reservation unspent:
|
||||
|
||||
| file | budget | spent | unspent | kept cluster | dropped cluster |
|
||||
|---|---|---|---|---|---|
|
||||
| `django/db/models/sql/query.py` | 10,135 | 1,923 | **8,212 (81%)** | 1379–1400, score 14 | 306–929, **score 290** |
|
||||
| `okhttp .../RealInterceptorChain.kt` | 6,058 | 1,474 | **4,584 (76%)** | 16–44, score 44 | 113–373, score 171 |
|
||||
| `okhttp .../Interceptor.kt` | 4,697 | 2,027 | 2,670 (57%) | 85–138, score 21 | 154–257, score 10 |
|
||||
| `gin/routergroup.go` | 5,782 | 3,273 | 2,509 (43%) | 33–91, score 116 | 103–188, score 128 |
|
||||
|
||||
A file whose top cluster by density is trivial keeps that one, drops the cluster carrying 20x the
|
||||
score, and leaves most of its own reservation unspent — because only the first-chosen cluster may
|
||||
be shrunk. `query.py` is the file CLAUDE.md already names as the `_fetch_all` case.
|
||||
|
||||
## Reproducing
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
node scripts/agent-eval/probe-factory-closure.mjs # primary query
|
||||
node scripts/agent-eval/probe-factory-closure.mjs \
|
||||
--target src/stores/alerts-store.ts --factory createAlertsStore \
|
||||
--query "createSessionStore" # the adverse configuration
|
||||
npx vitest run __tests__/explore-factory-closure.test.ts # the standing gate
|
||||
```
|
||||
@@ -0,0 +1,124 @@
|
||||
# Epic resolution — explore response noise (CG-24)
|
||||
|
||||
Worked 2026-08-05 → 2026-08-06. Started from one bad `codegraph_explore` response
|
||||
in a real session and ended with four shipped fixes, one open defect, and five
|
||||
issues closed because measurement contradicted them.
|
||||
|
||||
**The headline: the reported symptom was not an explore bug.** It was a degraded
|
||||
index. The explore defects the investigation found are real and were fixed, but
|
||||
none of them caused the report.
|
||||
|
||||
## The report
|
||||
|
||||
A prose flow query returned an unusable response: the symbol the agent had named
|
||||
never rendered, and a 12k-line generated Cloudflare ambient-types file took 60.7%
|
||||
of the output envelope.
|
||||
|
||||
```
|
||||
# deliv% bytes reserved score pen flags file
|
||||
1 1.0% 251 10,970 87.0 1.00 named entry central <the named file>
|
||||
2 60.7% 15,043 6,484 49.0 1.00 entry central worker-configuration.d.ts
|
||||
4 — — — 19.9 1.00 dropped: budget <a third file>
|
||||
```
|
||||
|
||||
## Root cause
|
||||
|
||||
**Index drift ([CG-33](index-drift-cg33.md)).** The live incrementally-synced index
|
||||
diverged from a clean rebuild by 4.3% of distinct edges, bidirectionally,
|
||||
overwhelmingly `calls`. RWR graph mass is relative and normalized, so call edges
|
||||
missing elsewhere inflate an unaffected file's share — the `.d.ts` carried mass
|
||||
0.24750 drifted vs 0.13119 rebuilt (~1.9×), score 49.0 vs 27.0.
|
||||
|
||||
Two causes, both fixed: incremental sync re-resolved only references *in* changed
|
||||
files, and `getNodesByName` had no `ORDER BY`, so ties broke by rowid — i.e. by
|
||||
the order files happened to be **written**. The second is why scope alone could
|
||||
never converge. Stale edges dropped 671 → 2 across an 80-commit replay.
|
||||
|
||||
On a freshly rebuilt index the reported query answers correctly **with no explore
|
||||
change at all**.
|
||||
|
||||
## Shipped
|
||||
|
||||
| | what |
|
||||
|---|---|
|
||||
| **CG-30** | Bounded how far an oversize cluster member may overshoot; windows on whole lines past 1.5× instead of emitting whole — or, when larger than the response ceiling, dropping the file silently. |
|
||||
| **CG-31** | Gave the cluster path the `owedBelow` displacement guard the whole-file BUY arm always had, holding back only the prefix of what is owed below that the response can actually pay. |
|
||||
| **CG-26** | Closed the remaining holes: whole-file arms had no displacement guard at all, section overhead was charged at a flat 200 against a real 300–500, and `owedPayableBelow` held all-or-nothing. |
|
||||
| **CG-25** | Recognize `Generated by <tool> by running <command>` banners. Precision held by requiring two `by` clauses, so ordinary prose does not match. |
|
||||
| **CG-28** | Damp declaration-only files that nothing in the index depends on. Does not stack with the generated penalty (`Math.min`), and naming a declaration symbol exempts its file. |
|
||||
| **CG-33 / CG-35** | Incremental sync converges with a rebuild, plus a regression suite that fails when the fix is disabled. |
|
||||
| **CG-36** | A later cluster is shrunk into the remainder rather than dropped whole — at selection, and again in the ceiling trim. All 8 starvation flags across the suite clear; +1,012 source chars net. `explore-cluster-starvation-cg36.md`. |
|
||||
|
||||
Deterministic across the 6-repo suite: no repo truncates, none loses a file,
|
||||
okhttp gains one, every repo lands at or under the 25,000 hard ceiling.
|
||||
|
||||
## Follow-up — CG-38 (closed)
|
||||
|
||||
**Agent-named symbols in the tail of a large file never rendered.** On the
|
||||
motivating repo, `queueMessage` (line 1087) and `flushQueuedMessages` (1102) in a
|
||||
1,414-line file were absent from the response on both prose and symbol-bag
|
||||
queries, even when that file won rank #1 with 67% of the envelope. The response
|
||||
returned the `QueuedMessage` *interface* at line 70 — a fuzzy near-match on the
|
||||
query token — instead of the function.
|
||||
|
||||
**Pre-existing, not caused by this epic.** A controlled bisect (index held fixed,
|
||||
engine varied across every merge point) shows the pre-epic engine rendering 12
|
||||
lines here and CG-36 rendering 463; the symbols render at neither. The epic
|
||||
strictly improves the case. An earlier claim that the epic regressed it was
|
||||
wrong — it compared runs across two different indexes.
|
||||
|
||||
Two independent causes, both longstanding: `buildFlowFromNamedSymbols` discarded
|
||||
the named-symbol IDENTITY along with the narrative whenever the named symbols did
|
||||
not form a call chain, so the importance-9 injection never ran; and the ceiling
|
||||
trim cut in SOURCE order, so a named def at the end of a large file was always the
|
||||
first thing dropped. Full account, plus the ranker-penalty lead (real, and
|
||||
orthogonal — the defs are absent at both `generated` flag states on the old build
|
||||
and present at both on the new one): `explore-tail-render-cg38.md`.
|
||||
|
||||
This epic's probes measure envelope share, starvation, source totals and file
|
||||
counts. **None measured "did the agent-named symbol render"** — which is why this
|
||||
survived the whole epic. `scripts/agent-eval/probe-named-symbol.mjs`,
|
||||
`__tests__/fixtures/tail-render-ts` and
|
||||
`__tests__/explore-named-symbol-render.test.ts` close that gap: per-symbol and
|
||||
binary, checking the definition LINE against the response's rendered lines.
|
||||
|
||||
CG-36's own measurement is worth carrying forward, because the issue named the wrong
|
||||
fix point: both real cases (`query.py`, `RealInterceptorChain.kt`) lost on
|
||||
`maxImportance`, **not** on the density tiebreak the issue suspected. Ranking was
|
||||
left alone; the never-shrink rule was the lever. Full numbers and the one cost
|
||||
(okhttp trades its rank-6 file for +7,196 chars in the two that answer the
|
||||
question) in `explore-cluster-starvation-cg36.md`.
|
||||
|
||||
## Closed because measurement contradicted them
|
||||
|
||||
Five, which is the story of this epic as much as the fixes are.
|
||||
|
||||
| | why |
|
||||
|---|---|
|
||||
| **CG-32** | Named file "didn't render first." Drift artifact; on a clean index it renders first and takes 89%. |
|
||||
| **CG-34** | "Allocator over-reserves for low-scoring files." Filed on a runner's diagnosis without checking the numbers. The file was never over-reserved (4,314 in both arms) — it was over-*spending*, which is CG-31. |
|
||||
| **CG-27** | Adding `function`/`method` to `ENVELOPE_KINDS` measured as a **large regression** — rank #1 fell from 7,539 delivered chars to 397, 7 of 11 inner closures to 0. The enclosing range was holding the file together as one cluster, inside which `shrinkCluster` already did the per-symbol ranking the issue wanted. A careful version was noise (69 vs 68 across nine queries). |
|
||||
| **CG-29** | Prose-vs-symbol query gap. Inverted on measurement: prose matches symbol on django and delivers 63% more source on okhttp. The founding observation was drift. |
|
||||
| **CG-37** | Duplicate of CG-36, filed without seeing it. |
|
||||
|
||||
## What this epic is actually a lesson in
|
||||
|
||||
**A confident diagnosis is worth less than a cheap measurement.** Every issue
|
||||
above was filed by someone — human or agent — who had read the code and had a
|
||||
plausible mechanism. Five were wrong. The ones that survived did so because a
|
||||
deterministic probe disagreed with them and the probe won.
|
||||
|
||||
Two specific traps this cost real time on, both now guarded in tooling:
|
||||
|
||||
- **`.codegraph/graph.db` does not exist** — the index is `codegraph.db`, and
|
||||
`sqlite3` against a mistyped path *creates* an empty database rather than
|
||||
failing. An empty schema reads exactly like a stale pre-migration index. This
|
||||
produced a wrong root cause. `diff-index-drift.mjs` refuses a missing path.
|
||||
- **`ab-new-vs-baseline.sh` checks the engine out at the baseline ref mid-run.**
|
||||
A commit made while it runs captures baseline sources and silently reverts the
|
||||
fix under test. This happened during CG-30. Check the `changed:` line before
|
||||
believing any A/B result.
|
||||
|
||||
And one measurement discipline worth keeping: **compare sets, not totals.** The
|
||||
drift that started all of this shows up as +0.7% on raw edge counts, because it
|
||||
is bidirectional and nets out. On distinct edge triples it is 4.3%.
|
||||
@@ -0,0 +1,108 @@
|
||||
# Agent A/B — bounded oversize cluster member (task CG-30)
|
||||
|
||||
**Date:** 2026-08-06 · **New:** `bugfix/CG-30` · **Baseline:** `main` @ `d6d1728` ·
|
||||
**Harness:** `scripts/agent-eval/ab-new-vs-baseline.sh`, `--model sonnet --effort high`,
|
||||
**both arms codegraph-on**, CLI blocked (0 contamination in every run),
|
||||
`CODEGRAPH_NO_PROMPT_HOOK=1`.
|
||||
|
||||
CG-30 bounds how far a cluster's top member may overshoot what its file may spend: past 1.5x it
|
||||
is windowed on whole lines instead of emitted whole. The risk the A/B exists to price is the one
|
||||
CLAUDE.md names — a section that is no longer sufficient sends the agent to Read, and one or two
|
||||
of those teach it to stop calling codegraph at all.
|
||||
|
||||
**Verdict: no regression, and the deterministic win is unambiguous.** The behavioural bar holds
|
||||
(Read/Grep ~0, no abandonment, allocation efficiency 100% on the repo where the bound engages),
|
||||
the cost is a ~10% median duration increase on django inside overlapping ranges, and the one
|
||||
allocation-miss call the new arm produced is matched by two recall-miss calls in the baseline.
|
||||
|
||||
> **Harness note, recorded because it cost a re-run:** `ab-new-vs-baseline.sh` checks the engine
|
||||
> out at the BASELINE ref while its baseline arm runs and restores it on exit. **Do not commit
|
||||
> while it is running** — a commit made mid-run captures baseline sources. The first django/gin
|
||||
> batches were void for exactly this reason (their `changed:` line listed only
|
||||
> `explore-diagnostics.ts`, i.e. both arms ran identical retrieval code) and were re-run. Check
|
||||
> that line before believing any A/B in this harness.
|
||||
|
||||
---
|
||||
|
||||
## Deterministic measurement — where the bound actually engages
|
||||
|
||||
Same index, same query, both builds. This is the primary evidence; the agent runs below only
|
||||
price the risk.
|
||||
|
||||
**django** — `codegraph explore "How does a QuerySet turn into SQL and fetch rows from the
|
||||
database?"`
|
||||
|
||||
| | baseline | new |
|
||||
|---|---|---|
|
||||
| `django/db/models/query.py` | 7,784 chars on a 3,669 budget — **2.12x** | 5,464 — **1.49x**, windowed |
|
||||
| `django/contrib/admin/filters.py` | 3,633 (inherited 2,271 spendable) | **8,057** (inherited 9,160) |
|
||||
| source delivered | 17,929 chars, 5 files | **20,033** chars, 5 files |
|
||||
|
||||
The reported CG-30 signature, reproduced on a public repo and then closed: the rank-#1 file took
|
||||
2.12x its budget, and the files below it inherited the shortfall. Bounding it hands those bytes
|
||||
straight down the rank order — the response carries the same five files and 2,104 more chars of
|
||||
actual source.
|
||||
|
||||
**gin (control)** — the two builds produce **byte-identical** explore output (13,457 chars) for
|
||||
the route-dispatch query. Nothing in gin is oversize enough for the bound to engage (max
|
||||
observed 0.94x of spendable), which is exactly what a control should show — and it means every
|
||||
gin number in the agent table below is run-to-run variance, not the change.
|
||||
|
||||
**Fixture** — `__tests__/fixtures/oversize-member-ts`, three report builders competing for one
|
||||
envelope, each a single long function:
|
||||
|
||||
| File | baseline | new |
|
||||
|---|---|---|
|
||||
| `monthly.ts` (24.5K, one ~490-line function) | 12,391 chars on a 3,334 budget — **3.7x** | 4,941 — **1.48x**, windowed |
|
||||
| `quarterly.ts` (11.4K, one ~200-line function) | **dropped** — `budget-clusters`, no headroom left | 4,004 delivered |
|
||||
| response | 19,223 chars, 3 files | 15,852 chars, 4 files |
|
||||
|
||||
The two rows are the same defect from both sides: a member bigger than the file's share eats the
|
||||
envelope, and a member bigger than the whole response ceiling makes the file vanish. Pinned by
|
||||
`__tests__/explore-oversize-member.test.ts` (9 tests; 4 fail on `main`).
|
||||
|
||||
## Agent runs
|
||||
|
||||
| | django new | django base | gin new | gin base | excalidraw new | excalidraw base |
|
||||
|---|---|---|---|---|---|---|
|
||||
| runs | 5 | 5 | 3 | 3 | 2 | 2 |
|
||||
| duration (s) | 39 [36–71] | 35 [35–60] | 39 [37–51] | 34 [28–46] | 52 [43–60] | 41 [40–42] |
|
||||
| tool calls | 3 [3–10] | 4 [3–23] | 4 [3–4] | 3 | 4 [3–5] | 4 [3–4] |
|
||||
| codegraph calls | 2 [2–3] | 2 [0–3] | 2 [2–3] | 2 | 3 [2–4] | 3 [2–3] |
|
||||
| Read | 0 [0–5] | 0 [0–13] | 0 [0–1] | 0 | 0 | 0 |
|
||||
| Grep/Glob | 0 | 0 | 0 | 0 | 0 | 0 |
|
||||
| occupancy share | 33.1% [30.7%–49.5%] | 34.4% [29.3%–47.3%] | 28.9% [26.4%–37.3%] | 30.1% [28.6%–31.9%] | 43.0% | 39.3% |
|
||||
| allocation efficiency | 100.0% | 98.6% | 88.5% | 98.3% | 85.8% | 90.5% |
|
||||
|
||||
django is pooled over two batches (n=2 + n=3). Questions: django "How does a QuerySet turn into
|
||||
SQL and fetch rows from the database? Trace the flow end to end."; gin "How does a registered
|
||||
route handler get invoked for an incoming HTTP request?…"; excalidraw "How does updating an
|
||||
element re-render the canvas on screen?…".
|
||||
|
||||
**Sufficiency, pooled per call — the bar that matters.** django: new 1 "Read a file we returned"
|
||||
in 10 answered calls (the allocation-miss signal a window would trip first) against the
|
||||
baseline's 1 "Read a file we did not return" + 1 Grep in 10 — a shift in miss type, not an
|
||||
increase. gin: 1 allocation miss in 7 against 0 in 6, on a repo where the two builds emit
|
||||
identical bytes, so it is variance by construction. excalidraw: 0 misses in either arm.
|
||||
|
||||
**Where the new arm looks worse, and why it is not read as a regression:**
|
||||
|
||||
- *django duration, ~10% slower median.* Ranges overlap (36–71 vs 35–60) at n=5, and one
|
||||
baseline run lost its codegraph attach entirely (0 codegraph calls, 13 Reads, 23 tool calls),
|
||||
which distorts that arm's spread in both directions.
|
||||
- *gin allocation efficiency 88.5% vs 98.3%.* The builds are byte-identical on gin. This is the
|
||||
metric's documented relativity — attribution is by citation and the agent's follow-up queries
|
||||
differ per run — not an effect of the change.
|
||||
- *excalidraw occupancy/duration.* Call-count noise: one of the two new-arm runs made a 4th
|
||||
explore call where the baseline made 2–3, and duration, envelope and occupancy all follow it.
|
||||
Per-call envelope is flat (20,015 vs 19,446 chars/call), Read/Grep stay 0, tool calls match.
|
||||
CLAUDE.md's own worked example records 3–10 codegraph calls on this prompt.
|
||||
|
||||
## Caveat carried forward
|
||||
|
||||
The `self-query` probe fixture in `scripts/agent-eval/allocation-fixtures.json` flips to FAIL
|
||||
under this change. Allocation is unchanged between arms and `tools.ts` delivers the identical
|
||||
8,282 chars in both — what changed is that an over-reserved incidental file now *delivers*
|
||||
instead of being cut by the hard-ceiling truncation, which is what its previous PASS depended
|
||||
on. Recorded as that fixture's `afterCG30` block. The over-reservation itself is epic CG-24's
|
||||
subject; it should not be answered by loosening this bound.
|
||||
@@ -0,0 +1,156 @@
|
||||
# Agent A/B — the end-to-end reservation invariant (task CG-26)
|
||||
|
||||
**Date:** 2026-08-06 · **New:** `bugfix/CG-26` @ `7cbde95` · **Baseline:** `bugfix/CG-31` @ `c54e008` ·
|
||||
**Harness:** `scripts/agent-eval/ab-new-vs-baseline.sh`, `--model sonnet --effort high`,
|
||||
**both arms codegraph-on**, CLI blocked (0 contamination in every run),
|
||||
`CODEGRAPH_NO_PROMPT_HOOK=1`. Every index measured on was **fully rebuilt**, never
|
||||
incrementally synced (CG-33).
|
||||
|
||||
Baseline is the CG-31 tip, not `main`, so every number here isolates CG-26. Read the three
|
||||
in sequence: `explore-oversize-member-ab-cg30.md` → `explore-displacement-guard-ab-cg31.md` →
|
||||
this one.
|
||||
|
||||
**The invariant:** every admitted file receives at least its reservation before any file draws
|
||||
on carry-forward slack. CG-30 bounded an oversize cluster member; CG-31 gave the cluster path a
|
||||
displacement guard. This closes the three holes left over — and each one was starving a file that
|
||||
had been admitted, reserved, and in the worst case *rendered*.
|
||||
|
||||
**Verdict: no behavioural regression on three repos, and the response is honest about its own
|
||||
budget for the first time.** No repo truncates. No repo loses a file; okhttp gains one. Two repos
|
||||
trade a few hundred source chars on their LAST-ranked file for the pointer list that names what
|
||||
the response could not cover — bytes the CG-31 tip only had because it over-filled a ceiling it
|
||||
mis-measured and then discarded the whole epilogue.
|
||||
|
||||
---
|
||||
|
||||
## The three holes
|
||||
|
||||
**1. The whole-file arms had no displacement guard.** The BUY arm's fit test read
|
||||
`totalChars + fileContent.length + FILE_OVERHEAD <= renderCeiling` — room before the ceiling,
|
||||
which belongs to every file the loop has not reached — while its own source-space sibling
|
||||
(`owedBelow`) refused exactly that trade. GRACE was not fit-tested at all. Measured on okhttp:
|
||||
`CallServerInterceptor.kt` shipped **8,499 chars against a 5,964 funded ceiling**, and the rank-6
|
||||
file below it delivered nothing. Both arms now test the render they actually produce against
|
||||
`fundedHeadroom`, and a whole render that does not fit **falls through to clustering** instead of
|
||||
skipping the file — a clustered section traded for no section is the trade the funding pool exists
|
||||
to refuse.
|
||||
|
||||
**2. Section overhead was charged at a flat 200 chars.** A real header — path plus up to
|
||||
`maxSymbolsInFileHeader` symbol names — runs 300–500. Everything downstream is expressed in those
|
||||
units (`headroom`, `fundedHeadroom`, every fit test), so the under-count was not a rounding error:
|
||||
it funded promises out of bytes that did not exist. okhttp allocated **26,601 chars against a
|
||||
24,400 ceiling** and the final truncation threw a fully-rendered section away. Sections are
|
||||
charged their real cost now; `owedPayableBelow` holds back each pending file's reservation *plus a
|
||||
per-file overhead estimated from that file's own symbols*; and a marginal overrun **trims the
|
||||
weakest cluster** — or windows the last one into the room that is left — rather than skipping a
|
||||
file over a ~300-char accounting difference.
|
||||
|
||||
**3. `owedPayableBelow` held all-or-nothing.** CG-31 was right that a promise the ceiling cannot
|
||||
reach is not a claim on this file's bytes — but it dropped the *partial* case. When the last
|
||||
admitted file's FULL reservation no longer fit, nothing at all was held for it. On the
|
||||
precise-query fixture the rank-5 file took 4,134 chars against a 2,948 reservation while rank 6 —
|
||||
admitted, reserved 2,539 — was left **4 chars** and skipped. It now holds the remainder, while
|
||||
that remainder is still worth a section (`MIN_CHARS`).
|
||||
|
||||
## The epilogue, budgeted instead of discarded
|
||||
|
||||
CG-31 handed this forward: the loop reserved a flat **600** chars for an epilogue that measures
|
||||
1,064 (gin), 1,788 (django), 2,231 (excalidraw), and four of six suite repos survived by
|
||||
discarding the epilogue **whole** — shipping with no pointer list and no reminders at all. A
|
||||
margin sweep was run and deliberately not shipped, because tuning one constant against the suite
|
||||
is the trap CG-30's record warns about.
|
||||
|
||||
The fix is not a bigger constant. The epilogue is **two things**:
|
||||
|
||||
- a **floor** the render loop reserves, sized from the real strings: the one line that says an
|
||||
uncovered area exists and that another explore — not a Read — reaches it, plus a pointer for
|
||||
every file whose bytes were deliberately WITHHELD (a cliffed file's bytes were traded away on
|
||||
the promise that the agent can still name it — CG-12; if the ceiling eats that name the trade
|
||||
was a silent drop);
|
||||
- an **elastic tail** — the rest of the pointer list and the reminders — fitted, in priority
|
||||
order and entry by entry, to the room that is actually left once the loop is done.
|
||||
|
||||
So a saturated response now lands with as much of its epilogue as it can pay for, instead of none
|
||||
of it, and `renderCeiling` is `hardCeiling − floor` rather than `hardCeiling − 600`.
|
||||
|
||||
## Deterministic measurement — the primary evidence
|
||||
|
||||
Same clean-rebuilt index, same query, both builds. One `codegraph_explore` per repo.
|
||||
Reproduce with `node scripts/agent-eval/probe-suite-envelope.mjs` (added by this task).
|
||||
|
||||
| repo | base source | new source | Δ | base files | new files | ceiling behaviour |
|
||||
|---|---|---|---|---|---|---|
|
||||
| django | 20,791 | **20,878** | +87 | 6 | 6 | was discarding its epilogue |
|
||||
| tokio | 21,521 | **21,607** | +86 | 5 | 5 | was discarding its epilogue |
|
||||
| okhttp | 19,034 | 18,870 | −164 | 5 | **6** | +1 file delivered; keeps its pointer list |
|
||||
| excalidraw | 20,204 | 19,652 | −552 | 8 | 8 | keeps its pointer list |
|
||||
| gin | 10,776 | 10,776 | 0 | 4 | 4 | byte-identical |
|
||||
| alamofire | 11,662 | 11,662 | 0 | 2 | 2 | byte-identical |
|
||||
|
||||
Queries are the CG-30/CG-31 ones, unchanged.
|
||||
|
||||
**Read the two negatives honestly.** They are not starvation — they are the reverse. At the CG-31
|
||||
tip both responses were *over-filled*: the loop under-counted its own section overhead, spent past
|
||||
the render ceiling, and the hard-ceiling cut then took the epilogue away to pay for it. okhttp
|
||||
also had a file rendered and dropped. Now the accounting is exact, so the loop stops where it
|
||||
said it would, and the ~500 chars go to the pointer list naming the files the response could not
|
||||
cover (2 on excalidraw, both `max-files` skips). No admitted file is starved in either.
|
||||
|
||||
**gin and alamofire are byte-identical between the builds** — neither saturates, so neither the
|
||||
guard nor the epilogue fit engages. That is what a control should show.
|
||||
|
||||
**Fixtures.** `__tests__/explore-reservation-invariant.test.ts` (14 tests; **3 fail on the CG-31
|
||||
tip**) pins the invariant on all three render paths and in both directions — the rank-#1 file when
|
||||
files below it overspend, and an admitted lower-ranked file when the top one does — plus the two
|
||||
things the ceiling must no longer do (allocate past itself; drop a rendered section) and the
|
||||
concentration it must not flatten. `__tests__/explore-displacement-guard.test.ts` (CG-31, 11
|
||||
tests) still passes unchanged.
|
||||
|
||||
**Allocation fixtures** — `scripts/agent-eval/allocation-fixtures.json`: **both PASS**. The
|
||||
self-query gate changed shape and the reason is recorded in `afterCG26`: the envelope-denominated
|
||||
`answerShareAtLeast` reads 47.5% here against 51.0% at the CG-31 tip while `tools.ts` delivers
|
||||
**byte-identical** source in both arms. What moved is the denominator — the response now delivers
|
||||
a fifth admitted file (`memory-budget.ts`, rank 4, paid its full 3,123-char reservation; the CG-31
|
||||
tip rendered it and let the ceiling drop the section) and keeps epilogue prose it used to discard.
|
||||
Both are the improvements this epic exists to make. The gate is now denominated in delivered
|
||||
SOURCE, where the answer group reads 55.5%, and it passes on both arms.
|
||||
|
||||
## Agent runs
|
||||
|
||||
| | django new | django base | excalidraw new | excalidraw base | okhttp new | okhttp base |
|
||||
|---|---|---|---|---|---|---|
|
||||
| runs | 2 | 2 | 2 | 2 | 2 | 2 |
|
||||
| duration (s) | **42** | 43 [36–50] | 53 [45–62] | 45 [37–54] | 49 [39–59] | 42 [32–52] |
|
||||
| tool calls | 4 [3–4] | 4 [3–5] | **3** [2–4] | 5 [4–5] | 4 | 4 [3–5] |
|
||||
| Read | **0** | 0 | **0** | 0 | **0** | 0 |
|
||||
| Grep/Glob | 0 | 0 | 0 | 0 | 0 | 0 |
|
||||
| codegraph calls | 3 [2–3] | 3 [2–3] | **3** [2–3] | 4 [3–4] | 3 | 3 [2–4] |
|
||||
| occupancy share | **36.6%** | 37.6% | **38.2%** | 47.2% | 44.8% | 40.8% |
|
||||
| allocation efficiency | **99.2%** | 94.7% | 81.9% | 82.5% | 75.8% | 87.6% |
|
||||
|
||||
Prompts are the deterministic queries with "Trace the flow end to end." appended.
|
||||
|
||||
**Read is 0 in all 12 runs, in both arms.** Sufficiency, pooled per call: **0 "Read a file we
|
||||
returned" and 0 recall misses on every repo in both arms** — the responses that deliver a few
|
||||
hundred fewer chars do not send the agent back to the file.
|
||||
|
||||
**Where the new arm looks worse, and why it is not read as a regression:**
|
||||
|
||||
- *okhttp allocation efficiency, 75.8% vs 87.6%, and occupancy 44.8% vs 40.8%.* The new arm's
|
||||
envelope is 99,411 chars against 89,297 — it returns one more file and more source overall, and
|
||||
the metric is the share of returned bytes the answer *cited*. Same trade the CG-31 record noted
|
||||
on this repo; the metric's own documentation says it is relative and must not be read as waste.
|
||||
- *Duration on excalidraw and okhttp.* n=2 with fully overlapping ranges (45–62 vs 37–54;
|
||||
39–59 vs 32–52), on a machine also running the other arm's build. excalidraw's new arm does the
|
||||
same work in **3 tool calls against 5** and holds **9 points less context**.
|
||||
|
||||
## Residuals
|
||||
|
||||
None from this task. The render-loop budget is now exact end to end: `totalChars` counts
|
||||
`flow.text`, the real per-section cost, and the epilogue floor; `allocatedChars ≤ hardCeiling` on
|
||||
every suite repo; and the final section-boundary truncation is now unreachable in normal
|
||||
operation (it stays as the backstop).
|
||||
|
||||
One thing deliberately NOT changed: the pointer list still caps at 10 files. Trimming happens
|
||||
from the bottom of the rank order and the "+N more files" tail is rewritten to confess every entry
|
||||
dropped, so the count is never silently wrong.
|
||||
@@ -0,0 +1,185 @@
|
||||
# CG-38 — an agent-named symbol in the tail of a large file never rendered
|
||||
|
||||
**Status: fixed.** Two independent causes, both longstanding. Not a CG-24 regression —
|
||||
the controlled bisect (index held fixed, engine varied across every epic merge point)
|
||||
found the symptom at every build including pre-epic.
|
||||
|
||||
## The report
|
||||
|
||||
On a 1,414-line Svelte store, `codegraph_explore` never returned `queueMessage`
|
||||
(L1087) or `flushQueuedMessages` (L1102) — on a bare symbol bag *or* a prose
|
||||
question — even though their file won rank #1 with score 127 and 67.3% of the
|
||||
envelope. What came back instead was the same-stem `QueuedMessage` **interface** at
|
||||
L70. The agent had to Read the file to find the two functions it had asked for by
|
||||
name, which is the one outcome explore exists to prevent.
|
||||
|
||||
CLAUDE.md's *"guarantee named symbols render"* — the importance-9 named-def
|
||||
injection — was not holding.
|
||||
|
||||
## What it actually was
|
||||
|
||||
### 1. The named-symbol IDENTITY was discarded with the narrative
|
||||
|
||||
`buildFlowFromNamedSymbols` returns two unrelated things: the Flow prose, and the
|
||||
SET of node ids the agent named. Downstream, that set is what injects a named def
|
||||
into its file's cluster ranges and ranks it **importance 9** — the entire mechanism
|
||||
behind the guarantee.
|
||||
|
||||
Its last gate was:
|
||||
|
||||
```ts
|
||||
if (!hasMain && synthLines.length === 0 && !boundaryText && !polyText) return EMPTY;
|
||||
```
|
||||
|
||||
`EMPTY` zeroes `namedNodeIds` too. So whenever the named symbols happened not to
|
||||
produce anything to *print*, the guarantee silently switched off. Two sibling
|
||||
closures in one factory are exactly that case: `queueMessage` and
|
||||
`flushQueuedMessages` never call each other, so there is no chain, no synthesized
|
||||
hop and no dispatch boundary — and both defs lost importance 9. The file then
|
||||
rendered from its head, which is how a 6-line interface displaced two functions
|
||||
1,000 lines below it.
|
||||
|
||||
Measured: `flow.namedNodeIds` was **empty** on the reported query, while
|
||||
`findAllSymbols` resolved both tokens to exactly 1 node each.
|
||||
|
||||
The fix separates the two outputs (`identityOnly()`), restricted to **shape-precise
|
||||
tokens** (camelCase / PascalCase / snake_case / qualified — the same test the gather
|
||||
path uses). With a narrative present, the prose is itself corroboration and that path
|
||||
is unchanged; with nothing corroborating it, only an unambiguous symbol reference may
|
||||
promote, so an English word in a prose question that happens to exact-match a
|
||||
callable cannot earn importance 9.
|
||||
|
||||
### 2. The ceiling trim cut in SOURCE ORDER
|
||||
|
||||
Restoring importance 9 was not enough — the symbols still did not render.
|
||||
|
||||
`shrinkCluster` *had* kept them: on the reported query it emitted a block spanning
|
||||
`1022-1121`, which covers both. But the shrink's output measured **26,297 chars
|
||||
against a 16,532 cap**, so `windowToCeiling` fired, and it fills parts in source
|
||||
order and drops everything after the first overrun:
|
||||
|
||||
```
|
||||
shrunk: 101-107, 197-226, ..., 648-989, 1022-1121 (26,297)
|
||||
windowed: 101-107, 197-226, ..., 648-839 (16,532) ← tail gone
|
||||
```
|
||||
|
||||
A trim that cuts in source order will always take the END of a large file first —
|
||||
which is precisely where an agent-named symbol is most likely to be, and least
|
||||
likely to be reachable any other way. `windowToCeiling` already had the concept it
|
||||
needed (`focusLine`, for the spine's next-hop call site, CG-30); it just wasn't told
|
||||
about named defs. It now takes a `focusLines` list — spine call site plus every
|
||||
member at importance ≥ 9, capped at 6 — and:
|
||||
|
||||
- tries the **full-ceiling** fill FIRST, holding back 40% only when a focus line is
|
||||
actually left uncovered (so a cluster whose head already reaches its focus keeps
|
||||
the whole ceiling for source — an improvement on the old unconditional hold-back);
|
||||
- **splits** the reserve evenly between the uncovered focus lines with carry-forward,
|
||||
rather than handing it out greedily in source order. Greedy reproduced the bug one
|
||||
level down: on the prose query, four focus lines resolved and the two earliest took
|
||||
the entire reserve, dropping `flushQueuedMessages` again.
|
||||
|
||||
## The accounting gap — found, measured, deliberately NOT shipped
|
||||
|
||||
`shrinkCluster`'s fit test uses the raw source span
|
||||
(`slice().join('\n').length`) while the render adds `contextPadding` around every
|
||||
block and a line-number prefix to every line. On the reported file that estimate ran
|
||||
**~60% under** (16.5K accounted, 26.3K rendered).
|
||||
|
||||
An exact projection (prefix-summed line costs, mirroring the merge + padding
|
||||
`buildSection` performs) was built and measured. **It is worse, and it is not
|
||||
shipped:**
|
||||
|
||||
| | main | exact accounting | exact + spend-the-remainder |
|
||||
|---|---|---|---|
|
||||
| django | 20,719 | 20,747 | 20,747 |
|
||||
| excalidraw | 19,704 | **19,606** | **19,606** |
|
||||
| okhttp | 18,651 | 18,766 | 18,766 |
|
||||
| tokio | 21,582 | **21,424** | 21,555 |
|
||||
| gin | 11,952 | 12,082 | 12,082 |
|
||||
| alamofire | 11,849 | 11,849 | 11,849 |
|
||||
| `probe-allocation` | 4 PASS | **payroll-go FAIL** | **payroll-go FAIL** |
|
||||
|
||||
The mechanism: exact accounting stops at the last member that fits **whole**, and the
|
||||
released bytes carry forward to lower-ranked files. On `payroll-go` that moved 1,296
|
||||
chars out of the rank-#2 answer file `cycle.go` and into the rank-#5
|
||||
`payslipstore/store.go`, taking `runPayrollCycleAll`'s `s.store.Upsert(ctx, slip)`
|
||||
call — the "create" half of the query — with it.
|
||||
|
||||
So the slack is doing no harm where it is: `bound()` clamps the render to the ceiling
|
||||
exactly, so the over-keep costs no bytes. What the slack must **not** do is decide
|
||||
*which* members survive — and that is the ceiling trim's job, which is what this task
|
||||
fixed. The comment on `shrinkCluster` now says so, so the next reader does not
|
||||
"fix" it.
|
||||
|
||||
## The index-dependence lead — explained, and orthogonal
|
||||
|
||||
The issue's sharpest lead was that flagging the ambient `.d.ts` as `generated` seemed
|
||||
to make an unrelated file's render *worse*. Flipping `files.generated` on that one row
|
||||
(the CG-25 method — holds the index constant, attributes the delta to the ranker
|
||||
alone) confirms the mechanism is real:
|
||||
|
||||
| | `generated=1` | `generated=0` |
|
||||
|---|---|---|
|
||||
| `.d.ts` graphScore | 0.1875 | 0.75 |
|
||||
| `maxGraph` | 0.3297 | 0.75 |
|
||||
| gate (6% of max) | 0.0198 | 0.0450 |
|
||||
| files ranked | 3 | 2 |
|
||||
| rank-#1 allowance | 9,100 | 8,166 |
|
||||
|
||||
`rankPenalty` scales `fileGraphScore`, `fileGraphScore` sets `maxGraph`, and the
|
||||
relevance gate is 6% of `maxGraph` — so a penalty on one file does move the admitted
|
||||
set and every other file's allowance. Confirmed.
|
||||
|
||||
But it is **not** what hid the symbols. On main they are absent at *both* flag states
|
||||
(render stops at L316 / L381); with the fix they are present at *both*. The
|
||||
allocation moves; the guarantee does not depend on it. Pinned by the last case in
|
||||
`__tests__/explore-named-symbol-render.test.ts`.
|
||||
|
||||
## Results
|
||||
|
||||
Real repro (`queueMessage` L1087 / `flushQueuedMessages` L1102), all shapes:
|
||||
|
||||
| query shape | main | fixed |
|
||||
|---|---|---|
|
||||
| symbol bag | absent | **both render** |
|
||||
| prose, symbols named | absent | **both render** |
|
||||
| symbols + decoy interface | absent | **both render** |
|
||||
| prose, no symbols named | absent | **both render** |
|
||||
|
||||
Fixture (`__tests__/fixtures/tail-render-ts`, 7 symbol checks over 3 query shapes):
|
||||
**7/7 fail on main, 7/7 pass** — deterministic over 4 consecutive runs per arm.
|
||||
|
||||
Standing bars, all held:
|
||||
|
||||
- `probe-allocation.mjs` — payroll-go / starved-cluster / dense-header / self-query all PASS
|
||||
- `probe-file-spend.mjs` — no starvation flags
|
||||
- `probe-suite-envelope.mjs` — **byte-identical to main on all six repos** (20,719 /
|
||||
19,704 / 18,651 / 21,582 / 11,952 / 11,849), same file counts
|
||||
- full suite green
|
||||
|
||||
The suite being byte-identical is the point: the focus windows only change what a
|
||||
render does once it has *already* overrun its ceiling, which none of the six suite
|
||||
queries does.
|
||||
|
||||
## Instruments
|
||||
|
||||
- `scripts/agent-eval/probe-named-symbol.mjs` — the measurement the epic lacked.
|
||||
Per-SYMBOL and binary: is the symbol's **definition line** among the response's
|
||||
rendered lines? The name alone proves nothing — it appears in the section header's
|
||||
symbol list and at call sites whether or not the body was sent, which is exactly how
|
||||
this hid through a whole epic of aggregate probes.
|
||||
- `__tests__/fixtures/tail-render-ts` — mirrors the reported file's geometry: decoy
|
||||
same-stem interface at L70, factory closure at L104 spanning ~92% of the file (so
|
||||
every symbol merges into ONE cluster), targets at L1088/L1096/L1102, plus a
|
||||
2,500-line generated `.d.ts` for the ranker to penalise. Generated by script; edit
|
||||
the geometry, not individual lines.
|
||||
- `__tests__/explore-named-symbol-render.test.ts` — the standing gate, including the
|
||||
fixture-shape assertions (if the fixture rots, the gate means nothing).
|
||||
|
||||
## Method note
|
||||
|
||||
A `git stash -- <path>` "baseline" reverts to **HEAD**, not to `main`. With a WIP
|
||||
commit on the branch that silently measures your own change against itself — it
|
||||
produced a clean "passes on main" here that was pure fiction. Use the file swap
|
||||
(`git show main:<path> > <path>`), as `.kommandr/memory/baseline-builds-use-fresh-file-swap`
|
||||
already says for builds.
|
||||
@@ -0,0 +1,189 @@
|
||||
# Index drift: incremental sync vs. full rebuild (CG-33)
|
||||
|
||||
Measured 2026-08-06. A live, auto-sync-maintained index **does not converge** to
|
||||
a clean full rebuild of the identical working tree. On codegraph's own repo,
|
||||
**4.3% of distinct edges were wrong**, in both directions, overwhelmingly
|
||||
`calls` edges.
|
||||
|
||||
This matters because it is silent: nothing warns, nothing surfaces it, and the
|
||||
README tells users the index is never stale and there is nothing to re-run.
|
||||
Retrieval quality decays invisibly, and the user-visible symptom — an agent
|
||||
falling back to Read — reads as "codegraph isn't very good" rather than "this
|
||||
index needs rebuilding."
|
||||
|
||||
## Result
|
||||
|
||||
Subject: codegraph's own `.codegraph/codegraph.db`, long-lived and
|
||||
incrementally synced, against a full rebuild of the same tree with the same
|
||||
build. Edges compared as distinct `(source, target, kind)` triples.
|
||||
|
||||
| | count |
|
||||
|---|---|
|
||||
| distinct edge triples (rebuild) | 28,809 |
|
||||
| in rebuild but **missing** from live | **751** |
|
||||
| in live but **absent** from rebuild (stale) | **476** |
|
||||
| **total divergent** | **1,227 — 4.3%** |
|
||||
|
||||
Missing edges by kind: `calls=635`, `contains=38`, `references=34`,
|
||||
`instantiates=21`, `imports=13`, `extends=10`.
|
||||
|
||||
### Raw counts hide it
|
||||
|
||||
Raw edge **rows** were 39,845 live vs 40,122 rebuilt — a benign-looking +0.7%.
|
||||
The divergence is bidirectional, so a net-count check nets it out and reports
|
||||
almost nothing wrong. **Any drift detector must compare edge sets, not totals.**
|
||||
|
||||
### The indexer is deterministic
|
||||
|
||||
Control, rebuild vs rebuild on the same tree and build: **0 differing edges**
|
||||
(28,809 both runs). So the live-vs-rebuild delta is not run-to-run noise.
|
||||
|
||||
### It is resolution, not residue
|
||||
|
||||
Node sets are identical — `files` 501 = 501, `nodes` 10,110 = 10,110,
|
||||
heuristic edges 36 = 36 — and every integrity check is 0 on *both* indexes:
|
||||
no duplicate nodes, no orphan edges, no nodes referencing a missing file row.
|
||||
|
||||
Nothing accumulates. Cross-file **resolution** goes stale.
|
||||
|
||||
## Mechanism — two causes, both confirmed
|
||||
|
||||
`ReferenceResolver` binds a reference to one of the same-named definitions
|
||||
**project-wide**. Two things follow, and the drift needed both to be fixed.
|
||||
|
||||
**1. Scope.** Incremental sync re-resolves only the references *in* the changed
|
||||
files. Adding or removing a definition of `pct` changes the correct answer for
|
||||
every `pct(...)` reference in the repo, including references in files the sync
|
||||
never touches — and those references resolved successfully once, which *deletes*
|
||||
their `unresolved_refs` row, so nothing existed to revisit them with. (The #1240
|
||||
retry only revisits refs parked as `status='failed'`.) The index kept an answer
|
||||
that was correct against an older graph.
|
||||
|
||||
**2. Tie-break.** When nothing disambiguated the candidates, `findBestMatch`
|
||||
kept the first one, and `getNodesByName` had no `ORDER BY` — so the winner was
|
||||
decided by rowid, i.e. by the order files happened to be **written**. A full
|
||||
index writes in scan order; a sync appends each file as it changes. The same
|
||||
tree therefore resolved to different edges depending on how the index was built,
|
||||
and no amount of re-resolution could converge, because re-resolving against the
|
||||
identical graph still picked a different candidate.
|
||||
|
||||
### The fix
|
||||
|
||||
- `getNodesByName` orders by `(file_path, start_line)` — a property of the code,
|
||||
not of the write order (`src/db/queries.ts`).
|
||||
- `sync` returns a `definitionDelta`: the names whose set of definitions the sync
|
||||
changed, computed as the symmetric difference of `file\0name` pairs sampled
|
||||
before and after the store phase (`ExtractionOrchestrator.sync`).
|
||||
- For each delta name, `resurrectStaleResolutionEdges` deletes the resolution
|
||||
edges targeting a symbol of that name whose source is in an *unchanged* file,
|
||||
and re-inserts each as the reference that created it (the `metadata.refName`
|
||||
stamp). The existing orphan sweep then resolves them against the post-sync
|
||||
graph — the same input a rebuild resolves from. Kill switch:
|
||||
`CODEGRAPH_NO_REBIND=1`.
|
||||
|
||||
The delta is compared **per file**, not as one name set over the whole batch: a
|
||||
commit that adds `collect` to a new file while an unrelated changed file already
|
||||
defines `collect` cancels out of a batch-wide name set, and that miss was the
|
||||
largest residual class in the first measurement of this fix.
|
||||
|
||||
Conservative by construction, because a wrong deletion is a permanent edge loss
|
||||
while a missed rebind is only residual drift: an edge with no `refName` stamp
|
||||
(synthesized, or built by an older engine) is never touched, edges whose source
|
||||
file the sync already re-extracted are skipped, and a per-name ceiling of 500
|
||||
edges declines the generic names.
|
||||
|
||||
### Result
|
||||
|
||||
Replaying real commits of this repo through `sync` one at a time, then diffing
|
||||
against a clean rebuild of the final tree:
|
||||
|
||||
| replay | baseline (`main`) | + ORDER BY only | + rebind pass (shipped) |
|
||||
|---|---|---|---|
|
||||
| 16 commits | 48 (24 missing / 24 stale) | 20 | **0 — converged** |
|
||||
| 80 commits | 1,634 (963 / 671) | 890 | **361 (359 / 2)** |
|
||||
|
||||
The direction that actively misleads — **stale** edges the index keeps asserting
|
||||
— drops from 671 to **2** over 80 commits, a 99.7% reduction.
|
||||
|
||||
Index and sync wall-clock are unchanged (392-file repo: index 1.88–2.02s in both
|
||||
arms, single-file sync 0.183s in both). The `ORDER BY` costs 18% per *uncached*
|
||||
name lookup in a tight loop (237ms → 280ms over 10,127 lookups), which does not
|
||||
reach wall-clock because `ReferenceResolver` memoizes the lookup per name. A
|
||||
composite `(name, file_path, start_line)` index would make the sort free, but it
|
||||
would widen every node index entry with a full path string on the write-heavy
|
||||
indexing path — not worth 43ms.
|
||||
|
||||
### The residual, and why it is not chased
|
||||
|
||||
At 80 commits, 357 of the 361 remaining edges are a single pre-existing class:
|
||||
references to very generic names (`push` 260, `join` 97) that failed at index
|
||||
time and stay parked because `getRetryableFailedReferences` declines any name
|
||||
with more than 500 failed refs (1,412 for `push`, 2,346 for `join`). That
|
||||
ceiling is #1240/#999 policy, it is present on `main`, and what it declines to
|
||||
create is cross-language garbage: a TypeScript test file "calling" an R method
|
||||
named `push`, or a Rust method named `join`. **The full rebuild is the wrong one
|
||||
here** — converging would mean teaching sync to manufacture thousands of wrong
|
||||
edges. Left as is, deliberately.
|
||||
|
||||
### `codegraph status` — decided: no drift metric
|
||||
|
||||
The issue asked whether `status` should surface divergence. Decision: **no**.
|
||||
|
||||
A drift number cannot be computed without the full rebuild it would be
|
||||
recommending, so anything cheap enough to run on `status` would be an estimate —
|
||||
and an honest estimate is not available. Shipping a proxy would violate the
|
||||
product rule that a screen must not overclaim, and post-fix it would fire on the
|
||||
generic-name residual above, training users to ignore it. (`status` already
|
||||
refuses to warn on parked failed refs for the same reason: every repo with
|
||||
external-library imports has them, so the warning would be permanent noise.)
|
||||
|
||||
The check that *is* exact stays available and is documented below.
|
||||
|
||||
## Why it degrades retrieval
|
||||
|
||||
Graph mass (RWR) is **relative and normalized**, so call edges missing elsewhere
|
||||
inflate an unaffected file's share of the mass. Explore ranks files by that mass
|
||||
(`allocateExploreBudget` weights on it), so drift silently promotes files that
|
||||
should rank low.
|
||||
|
||||
Observed on a private application repo under heavy development: a generated
|
||||
ambient-types file carried graph mass **0.24750** on the drifted index vs
|
||||
**0.13119** on a clean rebuild (~1.9×), and score **49.0** vs **27.0**. On the
|
||||
drifted index it took **60.7%** of an explore envelope and starved the file the
|
||||
agent had actually named by symbol, which rendered **251 chars of a 10,970
|
||||
reservation**. After a full re-index — no code change — the same query answers
|
||||
correctly. That incident is what prompted this measurement; see CG-24.
|
||||
|
||||
Severity scales with churn and index age. codegraph's own repo shows 4.3%;
|
||||
a repo under heavier active development plausibly drifts further.
|
||||
|
||||
## Reproducing
|
||||
|
||||
`scripts/agent-eval/diff-index-drift.mjs` is read-only and diffs two indexes.
|
||||
Snapshot the live index **before** rebuilding — the original artifact for this
|
||||
investigation was destroyed by re-indexing over it:
|
||||
|
||||
```bash
|
||||
cp .codegraph/codegraph.db /tmp/live.db # snapshot FIRST
|
||||
node dist/bin/codegraph.js index . # full rebuild
|
||||
node scripts/agent-eval/diff-index-drift.mjs /tmp/live.db .codegraph/codegraph.db
|
||||
```
|
||||
|
||||
Exit code is 0 when converged, 1 when drifted. To re-confirm determinism, diff
|
||||
two consecutive rebuilds — that must report 0.
|
||||
|
||||
To reproduce the *regression* rather than measure a live index, replay real
|
||||
commits through `sync`: clone the repo, check out `HEAD~N`, index, then
|
||||
`git checkout <sha> && codegraph sync` for each commit in order, snapshot the
|
||||
database, and diff it against a rebuild of the final tree. That is what produced
|
||||
the table above, and the unit-scale version of it is
|
||||
`__tests__/sync-rebuild-convergence.test.ts`.
|
||||
|
||||
## Note on probing an index
|
||||
|
||||
The index file is `.codegraph/codegraph.db`. There is no `graph.db`. `sqlite3`
|
||||
against a mistyped path **creates an empty database** rather than failing, and
|
||||
every subsequent query then answers from an empty schema — which reads exactly
|
||||
like a stale pre-migration index. That produced a wrong root cause during this
|
||||
investigation. `diff-index-drift.mjs` checks `existsSync` before opening for
|
||||
exactly this reason.
|
||||
Reference in New Issue
Block a user