github/codegraph
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

merge main into CG-3: keep the envelope view alongside occupancy

Browse Source 
CG-3 branched from main before CG-1 landed and rewrote parse-run.mjs wholesale
into an exported parseSession(), which dropped CG-1's --envelope/--answer
reporting entirely. That view is the instrument the CG-1/CG-22 allocation gate
measures bar 2 with, and it is in that benchmark's documented reproduce steps,
so it cannot be lost to the merge.

Resolution takes CG-3's rewrite as the structure and ports the envelope feature
into it: parseSession now collects codegraph_explore response text in call
order, formatEnvelope renders the per-file share, and the CLI parses
--envelope/--answer ahead of the positional filter so a glob is never mistaken
for a log path.

The glob sentinel stays written as a \u0000 escape, never a literal NUL byte --
a raw one makes git treat the whole script as binary, exactly as the comment
there warns.

Verified: --selftest 18/18, and a synthetic explore transcript reports the
expected per-file shares and answer-set total.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenryandClaude Opus 5 committed 2026-08-05 00:08:57 -05:00
2 parents 1d333017a8 c65d56ceba
commit 52b194a6be
53 files changed
+8564 -175
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/benchmarks/explore-allocation-ab-1500.md
+466
View File
@@ -0,0 +1,466 @@
# Agent A/B — score-proportional explore allocation (#1500 / epic CG-1)
Three measurements, in order. **The epic's gate is the last one** ([§CG-22](#cg-22--the-gate-re-run-at-cg-15s-exact-setup)):
| § | Task | Build | Verdict |
|---|---|---|---|
| [CG-15](#method) | first run of the gate | `edce18f` | **FAILS** bars 1 + 4 on express — routed the defect to CG-21 |
| [CG-21](#re-run-after-cg-21--the-gate-passes) | re-run while fixing | `fca7d87` | passes, n=6 on two repos, n=3 on client-go |
| [CG-22](#cg-22--the-gate-re-run-at-cg-15s-exact-setup) | **the gate** | `abee46c` | **PASSES** all four bars, CG-15's exact setup |
---
## CG-15 — the run that failed
**Date:** 2026-08-04 · **New:** `feature/CG-1` @ `edce18f` · **Baseline:** `main` @ `49c11fc`
· **Harness:** `scripts/agent-eval/ab-new-vs-baseline.sh`, `RUNS=3`, `--model sonnet --effort high`
on every arm · **Both arms codegraph-on.**
This is the epic's pass gate. The deterministic probes (CG-6/CG-14) prove the budget moved;
only an agent A/B proves the agent stopped reading.
**Verdict: the gate does not pass.** Bars 2 and 3 hold; **bar 1 (Read stays at 0) and bar 4
(no regression on the control) fail on express**, with a reproducible, non-agent cause. Per
the CG-15 acceptance rule the allocation design goes back to CG-12 — the budget is *not* to be
widened to compensate. Root cause and the smallest honest fix are in [§Root cause](#root-cause).
> **Superseded.** This section is the CG-15 measurement, kept because it is what routed the
> defect to CG-21 and because the root-cause analysis is the record of why. The defect was
> fixed and the A/B re-run twice: [§CG-21](#re-run-after-cg-21--the-gate-passes) alongside the
> fix, and [§CG-22](#cg-22--the-gate-re-run-at-cg-15s-exact-setup) — the epic's gate — at this
> section's exact setup. Nothing below was re-baselined.
---
## Method
`ab-new-vs-baseline.sh` builds and indexes each arm separately (CG-5's generated-file flag is
an index-time decision, so each arm must index with its own build), pre-warms a persistent
daemon per run, and runs the same flow question 3× per arm. Both arms run with
`CODEGRAPH_NO_PROMPT_HOOK=1` — the machine's ambient front-load hook resolves to whatever is
in `dist/` and would inject context through a second, uncontrolled channel.
Each prompt names codegraph as the lookup tool. That is **not** a forced-Read-0: the agent
stays free to Read whenever explore's answer is insufficient, which is exactly what bar 1
measures. It removes the one noise source that would otherwise swamp the signal — in a
pre-run without it, one express run made **0 codegraph calls and 3 Reads**, measuring adoption
(an axis this change does not touch) rather than allocation.
Envelope share is measured with `parse-run.mjs --envelope --answer <glob>`, which parses the
rendered markdown of the responses the agent actually received. The CG-4 diagnostic sidecar
only exists on the new build, so it cannot measure the baseline arm; the markdown parse is the
only instrument that measures both arms the same way.
| Repo | Lang | Files | Generated | Tier | Role |
|---|---|---|---|---|---|
| `kubernetes/client-go` | Go | 2,454 | 2,001 | medium (2 calls / 28K) | **the #1500 shape** — generated CRUD beside hand-written machinery |
| `excalidraw/excalidraw` | TS/React | 672 | 0 | medium (2 calls / 28K) | god-file concentration (`App.tsx`, 450 KB) |
| `expressjs/express` | JS | 147 | 0 | small (1 call / 18K) | **control** |
---
## Results
`explore` = codegraph_explore calls · `Read` = Read tool calls · `answer%` = share of the
source envelope going to the files that answer the question. Three runs per arm, reported as
the range — run-to-run variance is large and a single run means nothing.
### client-go — "how does a shared informer keep its cache in sync and deliver events?"
Answer set `tools/cache/**`; generated set `kubernetes/**`, `listers/**`, `applyconfigurations/**`,
`informers/**`, `**/fake/**`.
| arm | explore | **Read** | duration | answer% | generated% |
|---|---|---|---|---|---|
| **new** | 2 / 2 / 6 | **0 / 0 / 0** | 39–61s (med 50) | 74.2 / 96.9 / 82.6 | 4.0 / 0.0 / 5.2 |
| baseline | 3 / 2 / 3 | 0 / 0 / 0 | 48–52s (med 49) | 78.1 / 97.1 / 71.3 | 10.5 / 0.0 / 5.1 |
No Read in either arm. The medians overlap: on this query `tools/cache/**` already dominates
graph relevance, so the baseline concentrated well without help. The #1500 signal is visible
but small — the generated clientsets and per-resource informers
(`informers/events/v1beta1/interface.go`, `kubernetes/typed/events/v1/fake/…`) take 10.5% of
one baseline run's envelope and **never appear in any new run**.
### excalidraw — "how does updating an element re-render the canvas on screen?"
Answer set: `mutateElement.ts`, `App.tsx`, `renderer/**`, `scene/**`, `components/canvases/**`.
| arm | explore | **Read** | duration | answer% |
|---|---|---|---|---|
| **new** | 2 / 3 / 2 | **0 / 0 / 0** | **23–32s (med 24)** | 74.4 / 67.8 / 84.7 |
| baseline | 3 / 3 / 4 | 0 / 0 / 0 | 33–39s (med 34) | 74.1 / 64.0 / 79.5 |
The clearest win: **29% faster at the median with one fewer explore call per run**, no Read in
either arm. Concentration is why — the new arm resolves the flow in 2 calls where the baseline
takes 3–4. (One baseline run burned a turn on a hallucinated `codegraph_..._explore` tool name;
counted as-is.)
### express — control — "how does res.send decide Content-Type and ETag?"
Answer set `lib/**` (both arms deliver 100%; the whole answer lives in `lib/`, so this repo
tests concentration *within* the answer set, not against noise).
| arm | explore | **Read** | duration | answer% |
|---|---|---|---|---|
| **new** | 1 / 4 / 2 | **0 / 4 / 0** | 18 / 52 / 24s | 100 / 100 / 100 |
| baseline | 2 / 2 / 2 | 1 / 1 / 1 | 23 / 28 / 27s | 100 / 100 / 100 |
Two of three new runs are strictly better than every baseline run (0 Reads vs 1, faster). The
third is the failure: **4 Reads of `lib/utils.js` and 52s**, a fallback the baseline never
made. It is not agent variance — see below.
---
## Root cause
Deterministic replay of the divergent run's own query on both builds, same index, no agent:
```
codegraph explore "res.send Content-Type ETag generateETag setETag" --path <express>
```
| file | baseline | new |
|---|---|---|
| `lib/utils.js` (5,293 B, 272 lines) | **6,380 (46.1%) — whole** | **583 (7.7%) — cluster stub** |
| `lib/response.js` | 3,935 (28.4%) | 6,001 (64.9%) |
| `lib/application.js` | 1,607 (11.6%) | 2,532 (27.4%) |
| `lib/express.js` | 1,927 (13.9%) | — |
| **source envelope** | **13,849** | **9,241** |
`lib/utils.js` is where `compileETag`, `createETagGenerator`, `etag` and `wetag` live — half the
answer. The CG-4 diagnostic on the new build:
```
envelope 10,295 delivered · 10,292 allocated of 13,000 budget
allocation 12,398 reserved of 12,400 pool · cliff at weight 10.00 · nothing cliffed
# deliv% bytes reserved score flags render file
1 5.7% 583 3,870 56.0 named entry central clusters lib/utils.js
2 57.0% 5,868 5,875 91.4 entry clusters* lib/response.js
3 23.0% 2,369 2,653 34.5 entry central clusters* lib/application.js
```
`utils.js` is the **top-ranked** file (score 56.0, named + entry + central) and was **reserved
3,870 chars — and spent 583 of them.** Nothing was cliffed. The allocator did its job; the
render loop threw the reservation away.
The mechanism is the whole-file bound at `src/mcp/tools.ts:4008`:
```ts
const WHOLE_FILE_MAX_CHARS = allowance + Math.min(WHOLE_FILE_GRACE_MAX,
round(allowance * WHOLE_FILE_GRACE_FRACTION));
```
= `3,870 + min(800, 580)` = **4,450** < the file's 5,293 bytes, so the whole-file render is
declined. Pre-CG-12 the bound was `maxCharsPerFile * 3` = 11,400, which the file cleared
comfortably. The fallback cluster render only has 3 matched symbols to work with, so it emits
583 chars and **3,287 chars of the reservation are simply lost** — which is also why the whole
response shrank from 13.8K to 9.2K against an unchanged 13,000-char budget.
This is CG-12's own acceptance criterion — *"no file that was previously unclipped becomes
clipped"* — failing, and here it is the direct cause of an agent Read. CG-14 recorded one
instance of it (`memory-budget.ts`) as a documented exception; this is the same defect
observed in the wild, where it costs a round-trip.
**Not systemic.** On both medium repos the render loop saturates (`[over budget] [TRUNCATED]`,
23,599 of a 23,600 pool reserved), so there is no unspent budget to lose. The failure needs a
file whose proportional reservation lands *below its own size* while its matched-symbol set is
thin — likelier on small repos, where per-file reservations are smallest.
### The fix belongs in CG-12, not here
Per this task's acceptance rule the budget must **not** be widened to compensate. The defect is
that a reservation can go unspent, so the fix is one of:
1. **Let a large-enough reservation buy the whole file.** If `allowance >= k * fileSize` for
some k < 1 (utils.js: 3,870 / 5,293 = 0.73), render whole and let the bounded overshoot the
ceiling already tolerates absorb it — the bytes were reserved for this file anyway.
2. **Redistribute what a file cannot spend.** After the render loop knows a file's realised
size, hand the shortfall to the next-ranked file instead of dropping it. This also fixes the
shrinking-envelope symptom directly.
(1) is the smaller change and matches the observed shape; (2) is the more complete invariant
("the pool is spent"). They compose.
---
## Bars
| # | Bar | Verdict |
|---|---|---|
| 1 | **Read stays at 0** | **FAIL** — client-go 0/0/0 and excalidraw 0/0/0 both arms, but express run 2 makes 4 Reads the baseline never made, from a reproducible non-agent cause |
| 2 | Correct-file share > 50% | **PASS** — new 67.8–100% across all 9 runs; self-query fixture 16% → 59.9%. (Caveat: on these three repos the *baseline* was already above 50%; the ~16% figure is the self-query fixture, not these repos) |
| 3 | No wall-clock regression | **PASS at the median** — excalidraw 34s → 24s, express 27s → 24s, client-go 49s → 50s. The 52s express outlier is the failing run |
| 4 | No regression on the control | **FAIL** — express, 1 run of 3 |
Bar 1 is the hard gate and it fails, so the epic does not pass on this measurement regardless
of bars 2 and 3.
## Reproduce
```bash
# clone fresh (never eval on a private repo), index, then:
RUNS=3 AGENT_EVAL_OUT=/tmp/ab-express \
scripts/agent-eval/ab-new-vs-baseline.sh <express> "<question>" main
node scripts/agent-eval/parse-run.mjs /tmp/ab-express/run-new-2.jsonl --answer 'lib/**'
# the deterministic core of the failure, no agent needed:
CODEGRAPH_EXPLORE_DEBUG=1 node dist/bin/codegraph.js \
explore "res.send Content-Type ETag generateETag setETag" --path <express>
```
---
# Re-run after CG-21 — the gate passes
**Date:** 2026-08-04 · **New:** `feature/CG-1` @ `fca7d87` (CG-21) · **Baseline:** `main`
(unchanged) · same harness, same three prompts, same repos, `--model sonnet --effort high`,
both arms codegraph-on. **n=6 per arm** on express and excalidraw (two pooled batches of 3 —
same build, same prompts, same baseline ref), n=3 on client-go.
**Verdict: all four bars pass.** Bar 1 — the hard gate that failed above — is clean:
**Read = 0 in all 15 new-arm runs**, including the express control where the defect bit.
### Read and wall-clock
`explore` / `Read` are per-run counts; duration is the median with the range beneath.
| repo | arm | n | explore | **Read** | duration |
|---|---|---|---|---|---|
| **express** (control) | **new** | 6 | 2,2,2,2,1,1 | **0 ×6** | **21.5s** (18–30) |
| | baseline | 6 | 2,1,2,2,2,2 | **1 in 4 of 6** | 24.5s (19–29) |
| **excalidraw** | **new** | 6 | 3,2,2,3,4,2 | **0 ×6** | 34.5s (28–43) |
| | baseline | 6 | 2,4,2,2,2,2 | 0 ×6 | 26.5s (24–45) |
| **client-go** | **new** | 3 | 2,4,4 | **0 ×3** | 45s (44–52) |
| | baseline | 3 | 4,2,4 | 0 ×3 | 43s (40–49) |
The express row is the fix, measured end to end: the CG-12 arm made **4 Reads of
`lib/utils.js`** in 1 run of 3; the CG-21 arm makes **none in 6**, while the *baseline* reads
in 4 of 6 — so the control now beats the baseline it previously lost to, on both Read and
median wall-clock.
### Envelope share (bar 2)
| repo | new | baseline |
|---|---|---|
| express | 100% ×6 | 100% ×6 |
| excalidraw | 65.8 / 79.6 / 78.9% | 82.8 / 78.9 / 85.1% |
| client-go | **96.0 / 96.2 / 92.7%** | 86.7 / **53.8** / 80.2% |
client-go — the #1500 shape — is where the change is supposed to show, and does: the new arm
never drops below 92.7% while the baseline has a 53.8% run. Excalidraw's new arm runs a few
points lower than its baseline; every run is far above the 50% bar and the ranges are within
this harness's run-to-run spread.
### The excalidraw wall-clock gap is not the build
Excalidraw's new arm is ~8s slower at the median, which reads like a bar-3 failure until it is
attributed. Three measurements say it is session variance, not the change:
1. **Explore's own latency is unchanged.** Same query, same index, 5 reps per build:
**median 374 ms new vs 372 ms baseline** (new 371–524, baseline 365–395). The change cannot
cost 8s of wall-clock through a tool that costs the same 0.37s.
2. **The responses are the same size.** Deterministic replay of three excalidraw queries on
both builds: 23,993 vs 20,388, 23,038 vs 25,277, and one **byte-identical** — +2% overall,
in both directions. No truncation in any of the 12 runs, either arm.
3. **The identical baseline build moved 34s → 26.5s between sessions.** `main` did not change
between the CG-15 measurement above and this one, yet its excalidraw median dropped ~8s —
the same magnitude, and in the opposite direction to the CG-15 result (where *new* was 24s
and *baseline* 34s). Between-session variance on this repo is as large as the effect.
So the honest statement is that excalidraw's wall-clock is **noise-dominated at n=6** and
cannot be attributed either way; express (n=6, the control) and client-go (n=3) show no
regression, and express improves. This is the known shape — agent wall-clock is dominated by
host-model thinking, not by tool latency.
### Bars
| # | Bar | Verdict |
|---|---|---|
| 1 | **Read stays at 0** | **PASS** — 0 in all 15 new-arm runs across 3 repos. The CG-15 failure (4 Reads of `lib/utils.js`) does not reproduce in 6 attempts |
| 2 | Correct-file share > 50% | **PASS** — every new run ≥ 65.8%; client-go 92.7–96.2% vs a baseline run at 53.8% |
| 3 | No wall-clock regression | **PASS** — express 24.5s → 21.5s, client-go 43s → 45s (overlapping). Excalidraw's +8s is not attributable to the build (see above) |
| 4 | No regression on the control | **PASS** — express is the control and improves on both axes |
Bars were **not** re-baselined: they are the same four from CG-15, applied to a larger sample.
### Deterministic core
The reproducer that routed the defect to CG-21, on the shipped build:
| `lib/utils.js` (5,293 B, 272 lines) | baseline | CG-12 | **CG-21** |
|---|---|---|---|
| delivered | 6,380 (46.1%) whole | **583 (7.7%) stub** | **6,268 (39.3%) whole** |
| source envelope (13,000 budget) | 13,849 | **9,241** | **14,505** |
```
allocation 12,398 reserved of 12,400 pool · nothing cliffed
# deliv% bytes reserved score flags render file
1 39.3% 6,268 3,870 56.0 named entry central whole lib/utils.js
2 36.8% 5,868 5,875 91.4 entry clusters lib/response.js
3 14.8% 2,369 2,653 34.5 entry central clusters lib/application.js
```
Design and coverage: [`../design/explore-budget-allocation.md`](../design/explore-budget-allocation.md) § CG-21.
---
# CG-22 — the gate, re-run at CG-15's exact setup
**Date:** 2026-08-04 · **New:** `feature/CG-1` @ `abee46c` (`src/` identical to CG-21's
`fca7d87`; the two commits since are docs) · **Baseline:** `main` @ `49c11fc`, passed to the
harness as that SHA rather than as `main`, so the ref cannot drift · `RUNS=3`,
`--model sonnet --effort high` on every arm, both arms codegraph-on,
`CODEGRAPH_NO_PROMPT_HOOK=1` on both · same three repos, same three questions.
**This is the epic's gate.** CG-21's re-run above was measured by the task that wrote the fix;
this one re-measures it at the setup the failing run used, from a clean clone of each repo.
**Verdict: all four bars pass.** **Read = 0 in all 12 new-arm runs**, including the express
control where CG-15 failed — while the *baseline* reads in 3 of 3 express runs and in 1 of 6
client-go runs.
Repos re-cloned fresh at their current tips, so the file counts move slightly against the
CG-15 table (express 147 — unchanged; excalidraw 677, was 672; client-go 2,454 — unchanged).
Every repo stays in the same budget tier, so the allocator sees the same envelope.
## Results
`explore` / `Read` are per-run counts in run order; duration is the median with the range
beneath; `answer%` is the share of the source envelope going to the files that answer the
question, per run.
### express — control — "how does res.send decide Content-Type and ETag?"
Answer set `lib/**`.
| arm | explore | **Read** | duration | answer% |
|---|---|---|---|---|
| **new** | 1, 2, 2 | **0, 0, 0** | **24s** (18–35) | 100, 100, 100 |
| baseline | 2, 2, 2 | **1, 1, 1** | 26s (24–30) | 100, 100, 100 |
The bar-1 failure is gone at its own site. In every new-arm run the agent received
`lib/utils.js` **whole** — 6,643 / 7,673 / 6,455 chars, against the baseline's 6,396 / 6,380 /
6,396 — and never opened it. The baseline reads `lib/utils.js` in all three runs.
### excalidraw — "how does updating an element re-render the canvas on screen?"
Answer set: `mutateElement.ts`, `App.tsx`, `renderer/**`, `scene/**`, `components/canvases/**`.
| arm | explore | **Read** | duration | answer% |
|---|---|---|---|---|
| **new** | 2, 2, 2 | **0, 0, 0** | 26s (26–27) | 81.9, 69.3, 66.6 |
| baseline | 4, 2, 2 | 0, 0, 0 | 26s (21–32) | 92.7, 75.5, 81.0 |
Both arms clean, medians equal. Stated plainly because it is the one number that moves the
wrong way: the new arm's answer share runs **below** its baseline here (66.6–81.9 vs
75.5–92.7), the same direction CG-21 saw. Every run is far above the 50% bar, and this repo's
"answer set" is five globs over a god-file codebase where the baseline's extra breadth lands
inside them by luck of size, not by relevance — but it is not an improvement on this repo and
is not reported as one.
### client-go — the #1500 shape — "how does a shared informer keep its cache in sync and deliver events?"
Answer set `tools/cache/**`. **n=6 per arm** — two pooled batches of 3 (same build, same
prompt, same baseline SHA), run to tighten the wall-clock bound after batch 1 came out 4s
apart at the median.
| arm | explore | **Read** | duration | answer% |
|---|---|---|---|---|
| **new** | 3, 3, 2, 4, 2, 2 | **0 ×6** | 36.5s (30–46) | 95.2, 97.8, 92.3, 96.8, 96.8, 89.8 |
| baseline | 2 ×6 | **2 Reads in 1 of 6** | 35s (31–42) | 95.4, 100, 100, 85.6, 97.0, 97.0 |
The new arm never drops below 89.8% and never reads; the baseline has an 85.6% run and one run
that reads twice. **The generated layers stay out of both arms this session** — no
`kubernetes/**`, `listers/**`, `applyconfigurations/**` or `**/fake/**` file takes envelope in
any of the 12 runs, where CG-15's baseline gave them 10.5% of one run. That is the #1500 signal,
and it is *smaller here than in the earlier sessions* because this session's baseline sampled
well (85.6–100% answer share, against 53.8–86.7% in CG-21's). Reported as measured.
The new arm spends one extra explore call in 3 of 6 runs. Inspecting the sequences, the extra
call is a **deeper drill-down** (`processDeltas sharedProcessor run distribute …`) after a
complete answer, not a recovery from an insufficient one — every one of those runs still ends
at Read 0.
## Wall-clock, attributed
client-go is the only repo where the new arm's median is higher (36.5s vs 35s at n=6). Two
deterministic measurements say it is not the build:
1. **Explore's own latency is identical.** Same query, same repo, 5 reps per build:
client-go **669 ms new vs 668 ms baseline** (new 666–671, baseline 663–673); express
**204 ms vs 203 ms** (new 203–207, baseline 201–206). A 1 ms tool cannot cost 1.5s of agent
wall-clock.
2. **The new build's response is not bigger.** Deterministic replay of the client-go drill-down
query: source envelope **15,813 new vs 18,898 baseline**, with `shared_informer.go` taking
50.2% instead of 35.7% — more concentrated *and* smaller.
The ranges overlap almost completely (new 30–46, baseline 31–42), which is the known shape:
agent wall-clock is dominated by host-model thinking, not tool latency.
## Deterministic core — the reservation defect is gone
The reproducer named in CG-22's acceptance, run on both builds in this session, same query,
each build indexing its own copy:
```
CODEGRAPH_EXPLORE_DEBUG=1 codegraph explore \
"res.send Content-Type ETag generateETag setETag" --path <express>
```
| `lib/utils.js` (5,293 B, 272 lines) | baseline `49c11fc` | CG-12 | **HEAD** |
|---|---|---|---|
| delivered | 6,380 (46.1%) whole | **583 (7.7%) stub** | **6,380 (42.8%) whole** |
| source envelope (13,000 budget) | 13,849 | **9,241** | **14,913** |
Both conditions hold: the reservation is **spent** (reserved 3,870, whole-file render), and the
envelope does not shrink against an unchanged budget — 14,913 against the baseline's 13,849.
The baseline column was re-measured here, not quoted from CG-15, so the comparison is
within-session. The CG-4 diagnostic on HEAD:
```
envelope 15,967 chars delivered · 15,964 allocated of 13,000 budget (hard ceiling 19,500)
allocation 12,398 reserved of 12,400 pool · cliff at weight 10.00 · nothing cliffed
# deliv% bytes reserved score flags render file
1 39.3% 6,268 3,870 56.0 named entry central whole lib/utils.js
2 36.8% 5,868 5,875 91.4 entry clusters* lib/response.js
3 14.8% 2,369 2,653 34.5 entry central clusters* lib/application.js
```
(6,268 is the diagnostic's source-only count; 6,380 is the rendered section including its
header, which is what the envelope parser and the baseline column measure. Same render.)
## Bars
| # | Bar | Verdict |
|---|---|---|
| 1 | **Read stays at 0** | **PASS** — 0 in all 12 new-arm runs (express 3, excalidraw 3, client-go 6). The express run that failed CG-15 with 4 Reads of `lib/utils.js` reads nothing, and the baseline reads in 3 of 3 express runs |
| 2 | Correct-file share > 50% | **PASS** — every new run ≥ 66.6%; express 100% ×3, client-go 89.8–97.8% |
| 3 | No wall-clock regression at the median | **PASS** — express 26s → 24s, excalidraw 26s → 26s, client-go 35s → 36.5s at n=6 with fully overlapping ranges and identical explore latency (669 vs 668 ms) |
| 4 | No regression on the control | **PASS** — express improves on both axes: Read 3 of 3 → 0 of 3, median 26s → 24s, envelope 100% answer-set in both arms |
Bars were **not** re-baselined; they are CG-15's four, unchanged.
## Honest notes on the setup
- **The prompt wrapper is reconstructed.** The record preserved the three questions verbatim
but not the sentence that names codegraph as the lookup tool. Every arm and every repo here
used the identical wrapper `Use codegraph to answer: <question>`, so the comparison is
internally exact; it may differ by a few words from CG-15's.
- **Excalidraw is at a newer tip** (677 files, was 672) — same tier, same budget.
- Full suite green on the measured build: 171 files, **2,868 passed**, 6 skipped, 0 failures.
## Reproduce
```bash
# clone fresh (never eval on a private repo), index with the build under test, then per repo:
RUNS=3 MODEL=sonnet EFFORT=high AGENT_EVAL_OUT=/tmp/ab-express \
scripts/agent-eval/ab-new-vs-baseline.sh <express> \
"Use codegraph to answer: how does res.send decide Content-Type and ETag?" 49c11fc
node scripts/agent-eval/parse-run.mjs /tmp/ab-express/run-new-2.jsonl --answer 'lib/**'
# the deterministic core, no agent needed:
CODEGRAPH_EXPLORE_DEBUG=1 node dist/bin/codegraph.js \
explore "res.send Content-Type ETag generateETag setETag" --path <express>
```
docs/design/dynamic-dispatch-coverage-playbook.md
+24
View File
@@ -278,6 +278,30 @@ Status legend: ✅ done+validated · 🔬 hole identified · ⬜ not started.
(Verify the exact supported set against `src/extraction/languages/` and
`src/resolution/frameworks/` before starting — this table is a starting point.)
### Retrieval A/Bs that are not coverage work
Coverage decides whether a flow *exists* in the graph. A second class of change decides
whether the answer explore returns is *sufficient* — how the byte envelope is divided across
the files it found. Same pass bar (Read → 0, no wall-clock regression), same `--model sonnet
--effort high` rule, but the harness is `ab-new-vs-baseline.sh` (new build vs baseline build,
**both codegraph-on**) rather than `run-all.sh`'s with-vs-without, because the question is
whether a change to codegraph helped, not whether codegraph helps.
| Change | Repos | Result |
|---|---|---|
| **Score-proportional byte allocation** (#1500, epic CG-1) — 2026-08-04, `feature/CG-1` vs `main`, 3 runs/arm | client-go (Go, 2,454 f, 2,001 generated — the reporter's shape), excalidraw (TS, 672 f), express (JS, 147 f, control) | **Gate FAILED.** Read 0/0/0 both arms on client-go and excalidraw; excalidraw **34s → 24s median with one fewer explore call**; generated clientsets/informers drop from 10.5% of a baseline envelope to 0% in every new run. But express regressed in 1 of 3 runs (**4 Reads, 52s**) from a reproducible non-agent cause: a file whose proportional reservation lands below its own size no longer renders whole and its cluster render leaves the reservation **unspent** (`lib/utils.js` 6,380 B whole → 583 B stub, envelope 13.8K → 9.2K). Full record: [`docs/benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md) |
| **↳ re-run after CG-21** — 2026-08-04, `feature/CG-1` @ `fca7d87` vs the same `main`, **6 runs/arm** on express + excalidraw, 3 on client-go | same three repos | **Gate PASSES, all four bars.** **Read = 0 in all 15 new-arm runs** — the express regression does not reproduce in 6 attempts, and the *baseline* now reads in 4 of 6 while the new arm reads in none; express median **24.5s → 21.5s**. client-go answer share 92.7–96.2% vs a baseline run at 53.8%. Excalidraw's new arm is ~8s slower at the median, **not attributed to the build**: explore's own latency is 374 ms vs 372 ms (n=5), deterministic responses differ by +2% with one byte-identical, and the *unchanged* `main` build's own median moved 34s → 26.5s between the two sessions. Deterministic core: `lib/utils.js` 583 B stub → **6,268 B whole**, envelope 9.2K → **14.5K** on an unchanged budget |
| **↳ the gate (CG-22)** — 2026-08-04, `feature/CG-1` @ `abee46c` vs `main` @ `49c11fc` pinned by SHA, **CG-15's exact setup**: `RUNS=3`, fresh clones, 3 runs/arm (6 on client-go, two pooled batches) | same three repos (excalidraw now 677 f; same tier) | **Gate PASSES, all four bars — re-measured independently of the task that wrote the fix.** **Read = 0 in all 12 new-arm runs**, while the *baseline* reads in **3 of 3** express runs and 1 of 6 client-go runs. Express median 26s → **24s**, `lib/utils.js` delivered whole in every new run (6,455–7,673 B) and never opened. client-go answer share **89.8–97.8%** (baseline 85.6–100% — this session's baseline sampled well, so the gap is smaller than CG-21's); generated clientsets/informers take **0%** in all 12 runs. One honest counter-point: excalidraw's new arm runs **below** its baseline on answer share (66.6–81.9 vs 75.5–92.7), still far above the 50% bar. client-go's +1.5s median at n=6 is **not the build** — explore's own latency is 669 vs 668 ms and the new build's deterministic response is *smaller* (15.8K vs 18.9K) and more concentrated (50.2% vs 35.7% on the top file). Deterministic core, both builds re-measured in-session: `lib/utils.js` **6,380 B whole on both**, envelope **13,849 → 14,913** on an unchanged 13,000 budget |
Two harness lessons from that run, both now baked into `ab-new-vs-baseline.sh`:
- **Name codegraph in the prompt for this class of A/B.** Whether the agent picks the tool at
all is an adoption axis a retrieval change does not touch; a run that never calls explore
measures nothing about how explore divides its bytes (one pre-run: 0 codegraph calls, 3
Reads). It is not a forced-Read-0 — the agent stays free to fall back, which is the bar.
- **`CODEGRAPH_NO_PROMPT_HOOK=1` on both arms.** The machine's ambient front-load hook resolves
to whatever is in `dist/`, which the script itself rewrites between arms.
---
## 7. Known limits & gotchas (from the excalidraw/django work)
docs/design/explore-budget-allocation.md
+666
View File
@@ -0,0 +1,666 @@
# Explore budget allocation — the instrument and the baseline
`codegraph_explore` has a fixed byte envelope (`getExploreOutputBudget().maxOutputChars`,
hard-capped at 25K so the host never externalizes the result). **How that envelope gets
divided among files** is decided by a long chain of gates, tiers and caps spread across
`handleExplore` — and until CG-4 that chain was unobservable. You could read an explore
response and guess; you could not say "this file took 16% and that one took 20%."
This document covers the diagnostic that makes it measurable, and the baseline it recorded.
## The diagnostic
Set `CODEGRAPH_EXPLORE_DEBUG` and every `codegraph_explore` call (MCP tool or
`codegraph explore` CLI) emits one report:
| value | sink |
|---|---|
| `1` / `true` / `on` / `yes` / `stderr` | human-readable table on stderr |
| `json` | one pretty-printed JSON report on stderr |
| anything else | treated as a path — one JSON report per line, appended (JSONL) |
| unset / `0` / `false` / `off` / `no` / empty | **off** |
Per file it reports: relevance score, graph (RWR) mass, distinct query-term hits, ranking
flags (named / entry / central / spine / low-value / generated), render mode, bytes of
source allocated, bytes actually delivered, both shares, and whether it was clipped. For
files that never rendered it reports why (`max-files`, `budget-90pct`, `budget-whole-file`,
`budget-clusters`, `unreadable`, `no-ranges`). Totals cover the envelope (delivered vs
allocated vs `maxOutputChars` vs hard ceiling), the source/meta split, the file-selection
funnel at each stage, and the thresholds applied (score floor, graph-relevance gate).
**It is off by default and produces byte-identical output when off** — it ships in the
product binary, and a diagnostic that perturbs the response by one byte would invalidate
every A/B measurement taken with it on. `ExploreDiagnostics.start()` returns `null` unless
the env var is set, so every call site is a `diag?.` no-op. Pinned by
`__tests__/explore-diagnostics.test.ts`.
Two envelope numbers, deliberately kept separate:
- **allocated** — what the render loop chose to emit, before the final hard-ceiling cut.
This is the allocator's own decision, and the number budget work is about.
- **delivered** — what the agent actually received.
They diverge exactly when the ceiling truncates. Conflating them is how a dropped trailing
file goes unnoticed.
## Baseline (2026-08-03, this repo at `main`)
```
codegraph explore "how does explore allocate its output budget across files" --path .
```
469 files indexed → small tier (`maxOutputChars` 18,000, `maxCharsPerFile` 3,800,
`defaultMaxFiles` 5). Envelope: **23,196 delivered / 23,193 allocated against an 18,000
budget — 29% over**, absorbed only because the 25K hard ceiling sits above it.
| # | share | bytes | score | graph | hits | flags | render | file |
|---|---|---|---|---|---|---|---|---|
| 4 | 21.2% | 4,928 | 10 | 0.125 | 1 | entry | whole | `scripts/agent-eval/offload-eval-hook.mjs` |
| 5 | 20.1% | 4,665 | 10 | 0.125 | 1 | entry | whole | `scripts/agent-eval/offload-eval-metrics.mjs` |
| 3 | 19.8% | 4,585 | 22 | 0.125 | 1 | entry central | whole | `scripts/agent-eval/parse-session.mjs` |
| 1 | **15.8%** | 3,659 | **54** | **0.322** | **4** | entry central | clusters* | `src/mcp/tools.ts` |
| 2 | 15.0% | 3,479 | 34 | 0.082 | 2 | entry | clusters* | `src/index.ts` |
\* clipped. Ranked but never rendered: `scripts/agent-eval/offload-eval-cost.mjs` (#6) and
`src/resolution/lru-cache.ts` (#7), both cut by `maxFiles`.
Files: 17 grouped → 10 past the score floor (≥3) → 10 past the low-value filter → 7 past
the relevance gate (graph ≥ 0.0193, 6% of max 0.3215) → 5 in the output.
### What the baseline shows
**Score does not drive allocation.** `src/mcp/tools.ts` — the file that actually answers
the query — carries 5.4× the relevance score, 2.6× the graph mass and 4× the term hits of
any `.mjs` script, and gets a *smaller* share than each of them. The three agent-eval
scripts take **61%** of the envelope between them; the answer file takes 16%.
The mechanism is that the two allocation paths are decided by **file size, not relevance**:
a small file clears `WHOLE_FILE_MAX_LINES`/`WHOLE_FILE_MAX_CHARS` and ships entirely, while
a large file falls through to cluster selection and is clipped at `maxCharsPerFile`. So a
weakly-relevant 130-line script gets 100% of itself; the strongly-relevant 5,000-line file
gets 3,800 chars. Rank ordering is correct (tools.ts sorts #1) and buys nothing, because
rank has no effect on how many bytes a file receives.
**The envelope is over-subscribed.** 23,193 allocated against an 18,000 budget means the
per-file caps do not compose into the total cap; the total is enforced only by the 25K
ceiling silently dropping whole trailing sections. Under a slightly different index state
(one more candidate file) the same query allocated 27,518 chars and the ceiling dropped a
7,678-char section — the single largest allocation in the response — with the only trace
being the truncation notice at the end.
This is the gap the rest of the epic closes: relevance-proportional allocation with a
relative cliff (CG-12), on top of scoring that stops rewarding incidental name collisions
(CG-10 — landed; see below).
## CG-10 — relevance scoring
CG-10 changes **what gets into the response**, ahead of how bytes are split among what's
in. Four levers, all multiplicative so they compose without ordering surprises.
### 1. Kind weighting
The tier a symbol reached us by (named seed `+50`, query match `+10`, adjacent to one `+3`,
peripheral `+1`) says *how it got here*; `RELEVANCE_KIND_WEIGHT` says *whether the match is
evidence*. Callables and types weigh 1.0, members ~0.5, and `constant`/`variable`/
`parameter` 0.15–0.35 — a local named `explore` is a name collision until something
corroborates it.
**Isolation.** For a weak-kind symbol in the top two tiers, "is anything using it?" is the
corroboration: no usage edge anywhere in the graph (`contains` excluded — lexical nesting
is not usage) drops it to 0.08. Cost is bounded — only weak kinds in the tiers whose weight
can carry a file pay for the probe, and the subgraph's own edges answer most cases for
free. Measured: no latency change (210 vs 211 ms/call, n=12 interleaved).
**Peripheral cap.** Nodes ≥2 hops from any match now accumulate into a separate bucket
capped at 5. Uncapped, every such node added a flat `+1`, so a file grew more relevant by
being bigger — `parse-session.mjs` reached score 22 off one incidental constant plus twelve
unrelated symbols. Size is not evidence.
### 2. Relative score floor
`score >= 3` admits noise on any repo where the top file scores 50+. The floor is now
`clamp(topScore × 0.2, 1, 10)`:
- **relative** — on a diffuse question no file dominates, every candidate sits near the top,
and the whole spread survives; on a precise one it cuts the tail.
- **capped at 10** — one direct query match on a callable. A single full-strength match is
never incidental, so no amount of concentration elsewhere may exclude it. Without this
cap, one named-seed-heavy file pushed the floor to 21 and dropped a file the agent had
named by *class* name (classes enter at `+10`, not `+50` — named seeds are callables).
- **backfill** — if fewer than 3 files survive, the best of what the floor cut comes back,
but only from files with real evidence (≥ the absolute floor). If *nothing* survives, the
backfill drops that requirement: returning "no relevant code found" when the gather did
find candidates sends the agent straight back to grep.
### 3. Generated status in the score, not the tiebreak
`rankPenalty(file)` multiplies both the relevance score and the graph mass by 0.3 for
generated files (0.5 for low-value ones). Applying it to the score alone would not have
fixed #1500: the generated CRUD carries **more** graph mass than the hand-written use-case,
and graph mass outranks score in the comparator. The penalty is self-normalizing — in an
all-generated repo everything scales together and relative ranking is untouched — and it
never hard-excludes: ask about the generated API by name and the named-seed tier still puts
it first.
### 4. `excludeLowValueFiles` — the finding
The per-tier flag the task asked to reconsider was **dead config**: declared on
`ExploreOutputBudget` and set per tier, but read nowhere. A later change had already made
the test/spec/icon/i18n exclusion unconditional at all tiers. The flag is removed.
The substantive gap was in the *detector*, not the gating: `isLowValue` matched
`/\/(tests?|__tests?__|spec)\//`, anchored on a leading slash, so a **repo-root** `test/`
directory — express, cobra, and most of npm and Go — never matched. Express's "how does
express route a request to a handler?" spent 59% of its envelope on three test files while
`lib/application.js` was clipped. Anchored at `^` as well, that query now returns
`lib/application.js` + `lib/response.js` and no tests.
Two related changes: the filter now runs **before** the score floor and judges "are there
other candidates?" on the whole gather rather than the post-floor set (judging it after was
how the floor's keep-minimum pulled test files back in as the "spread"); and low-value
files that survive the filter's `≥2 non-test candidates` escape hatch are down-weighted via
`rankPenalty` rather than left at full strength.
### Measured effect
Before/after on the same indexes, deterministic (`CODEGRAPH_EXPLORE_DEBUG` diagnostic, both
arms same build system, baseline = `bd86ad2`):
| repo · query | before | after |
|---|---|---|
| this repo · self-query fixture | 72% to eval scripts, `tools.ts` 18.5% | scripts **0%**, `tools.ts` #1 |
| this repo · `handleExplore buildFlowFromNamedSymbols …` | 82% to eval scripts | `tools.ts` 48% + `index.ts` 32% |
| this repo · "how is error handling done" | 58% to eval scripts, `tools.ts` delivered 0 | transport/tools/cobol/api |
| this repo · "what languages does codegraph support" | 63% to `scripts/add-lang/*` | grammars/index/cli |
| this repo · "main components of the indexing pipeline" | — | **byte-identical** |
| payroll-go fixture | generated 57.4%, answer 25.6% | answer **61.5%**, generated **23.5%** |
| express · route a request | 59% to `test/*` | `application.js` + `response.js` |
| cobra · 3 queries | — | **byte-identical** |
The two byte-identical rows are the control: where the answer was already concentrated, the
new floor prunes the same tail earlier and cheaper and arrives at the same response.
**Known thin case.** Express's "how does the app object get created and what does it
expose" drops from 4 files (top one an `examples/` file at 38%) to `lib/express.js` alone,
2.6 KB against a 13 KB budget. `lib/application.js` matched on nothing but an unused
file-scope `var app` — indistinguishable, at the symbol level, from the eval scripts' unused
`const explore`; express models its API surface as properties assigned to that object,
which the graph has no edges for. That is extraction coverage, not ranking. Backfilling it
was tried and rejected: node-count ties handed the slot to `examples/route-middleware`
instead, at 48% of the envelope. Thin-and-precise beats padded-with-noise — a wrong file
does not save the agent the follow-up call it would pad against.
## CG-12 — score-proportional allocation
CG-10 fixed *what* gets into the response. This fixes *how the bytes are split among what
got in* — which, until now, was not really decided at all. Every admitted file was capped at
the same flat `maxCharsPerFile`, and the whole-file rule handed anything under
`maxCharsPerFile × 3` its entire contents. So the envelope followed **file size**:
- self-query: `memory-budget.ts` (score 18) shipped whole at 5,672 and took **51.2%**;
`tools.ts` (score 41, 4× the graph mass, 3× the term hits — it literally holds the
allocator) was clipped at 3,800 and got **32.9%**.
- payroll-go: two generated CRUD files shipped whole at ~4.5 KB each *and* consumed two of
the tier's four file slots, so `BuildPayslip` — the hand-written "calculate" half of the
question — ranked #6 and never rendered at all.
### The model
`allocateExploreBudget` (`src/mcp/tools.ts`) runs once, after ranking, before anything
renders. It reserves each file a share of the envelope; the render loop then spends a
reservation instead of racing for whatever the files above it left.
1. **Weight** = `score × worth × (spine ? 2 : 1)`. `worth` is `rankPenalty` applied a
*second* time: ranking answers "is this file about the query", allocation answers "will
these bytes teach the agent anything". Generated CRUD can legitimately rank — it
name-collides on every domain word, and it is big and densely self-referential, so it
scores on the structural keys the comparator leads with — while its bytes stay mechanical
boilerplate. That second penalty is what finally sinks it.
2. **Relative cliff** at 15% of the top weight, itself capped at `SCORE_FLOOR_MAX`. A file
under it gets **zero source** — path, symbols and line numbers only. It costs ~100 chars
instead of ~4,500, and it does **not consume a `maxFiles` slot**, so the slot passes to a
file that earns its bytes. That slot hand-off is what got `BuildPayslip` into the
response. The cap matters as much as the fraction: one 500-scoring god-file would
otherwise put the cliff at 75 and silence every peer the score floor had just admitted.
3. **Floor then split.** Every admitted file gets `MIN_CHARS` (700 — enough for one complete
method); the remainder splits by weight. The floor is what keeps a diffuse survey
question returning a spread; the remainder is what concentrates a precise one.
4. **Safety valve**, not a per-file cap: no file exceeds 70% of the envelope. The flat
per-file cap is retired as the primary guard — the proportional split already bounds a
file by its weight share.
The reservation then governs **every** render path — whole-file, clusters, focused/skeleton
— where before the whole-file branch was 3× more generous than the cluster branch, which is
the 3× swing that decided the split by file size.
Two supporting changes were needed to make the reservation actually bite:
- **Oversize clusters shrink by member.** A cluster is a merge of whole symbol ranges, and
on a densely-packed file every symbol merges into one blob spanning the file (cycle.go's
209-line `Service`). The old rule took the top-ranked cluster whole however big it was, so
a single-cluster file simply ignored its budget — it took ~40% more than allotted and the
file below it was then dropped for lack of room. Shrinking drops whole **members** by
importance, so a body is still never cut.
- **The arrival-order stops are gone.** `budget-90pct` and the `!fileNecessary && totalChars
> maxOutputChars` checks dropped files by the order they were reached: whichever files
ranked first spent the envelope, and everything after them was cut on a cap it had no say
in. Only an absolute hard-ceiling stop remains.
### Measured effect (CG-12)
| repo · query | before (post-CG-10) | after |
|---|---|---|
| this repo · self-query fixture | `tools.ts` 32.9%, `memory-budget.ts` 51.2% | **`tools.ts` 60.6%**, memory-budget 17.2% |
| payroll-go fixture | answer 61.5%, generated 23.5%, `BuildPayslip` absent | answer **78.7%**, generated **0%**, `BuildPayslip` **delivered** |
| express · route a request | 2 files, top 43.1% | 1 file, top **82.3%** (`response.js` cliffed — 0 term hits) |
| express · app registers middleware | 1 file, 71.6% | **byte-identical** |
| cobra · parse flags and execute | 2 files, `command.go` 40.1% | 2 files, `command.go` **77.2%** |
| cobra · DIFFUSE "main components" | 3 files, top 48.1% | 3 files, top 50.0% — spread preserved |
| gin · request reaches a handler | 3 files, top `ginS/gins.go` 48.8% | 3 files, top **`routergroup.go`** 53.8% |
| gin · DIFFUSE "what it provides" | 3 files, top `recovery.go` 43.0% | **4 files**, top `context.go` 33.3% |
The two diffuse rows are the over-correction control: file counts hold (3→3, 3→4), so a
survey question still gets a spread. The two gin rows also moved the top file to a more apt
one — `routergroup.go` over the thin `ginS` singleton wrapper, `context.go` over
`recovery.go` — because concentration is decided by weight rather than by which file
happened to be small.
**Exception to "no previously-unclipped file becomes clipped".** `memory-budget.ts` was
unclipped-whole at 5,672 and now clusters within its 3.1 KB reservation. That is the epic's
own diagnosis of the bug rather than a regression: it scored 18 against `tools.ts`'s 58 and
was taking the larger slice purely for being small enough to ship whole. The guarantee holds
where it was meant to — no file loses bytes to a *tighter cap*; the only files that lose are
ones the proportional split says were over-served.
## The regression fixtures (CG-6)
Two fixtures pin the failure mode so it can never silently return. They were written to
**fail** — that is what they were for. CG-10 closed the ranking half of both and CG-12 the
byte-split half; **both now pass** and are live regressions. The numbers quoted below are
the **pre-CG-10 baseline**; see the two "Measured effect" tables above for where they stand.
They are declared in `scripts/agent-eval/allocation-fixtures.json` and run by
`scripts/agent-eval/probe-allocation.mjs`, which drives the CG-4 diagnostic through a JSONL
sidecar (so it measures the shipping allocator, not a re-derivation), groups the rendered
files into `answer` vs `incidental`, and checks declared share thresholds. Needs a current
`npm run build`; exits 1 while any assertion fails.
```bash
node scripts/agent-eval/probe-allocation.mjs # both
node scripts/agent-eval/probe-allocation.mjs payroll-go # one
node scripts/agent-eval/probe-allocation.mjs --json # machine-readable
```
### 1. `payroll-go` — the reporter's shape
`__tests__/fixtures/payroll-go/` is a synthetic Go service: generated FKIT CRUD beside a
hand-written payroll use-case, entered from an HTTP route. Full description in that
directory's README. The essentials:
- Generated files with **ordinary names** carrying `// Code generated ... DO NOT EDIT.` —
invisible to path-only detection, which is what makes this a #1500 fixture rather than a
`.pb.go` one — beside `payrollpb/*.pb.go` covering the path-detectable channel.
- Deliberate collisions: `BuildPayslip`, `Upsert` and `Store` each exist twice, generated
and hand-written, and the generated layer name-collides on every query term.
- `cycle.go` (227 lines) sits above the whole-file window so it clips; the generated files
sit below it so they ship whole.
Query — an architecture question naming none of the answering symbols: *"how does payroll
cycle create and calculate payslips?"*
| | allocated | delivered |
|---|---|---|
| hand-written | 48.4% | **25.6%** (all of it domain types) |
| generated CRUD | 39.9% | **57.4%** |
`cycle.go` is allocated the single largest slice (7,052 chars, 30.6%) and delivers **zero**
— the 19,500 hard ceiling drops its whole section. `payslip_builder.go` (rank #8) never
renders. So `runPayrollCycleAll`, the hand-written `BuildPayslip` and the real `Upsert`
never reach the agent, and every byte that did arrive describes either CRUD or types.
This fixture is hermetic: the probe copies the tree to a temp dir and re-indexes per run,
so two runs on one build are byte-identical (verified). `__tests__/explore-allocation-1500.test.ts`
runs the same assertions in vitest.
**After CG-10** the generated files rank #3/#4 instead of #1/#2, `cycle.go` delivers 38.9%
(it delivered nothing), and `runPayrollCycleAll` + the real `s.store.Upsert(ctx, slip)`
reach the agent. Those assertions are now live regressions. What remains `it.fails` is
`payslip_builder.go`: it ranks #6, the tier's `maxFiles` is 4, and the render loop still
spends by file size — CG-12's job.
**Finding, deliberately left unfixed:** `runPayrollCycleAll` calls `s.store.Upsert` on a
`*payslipstore.Store`, but the graph resolves that edge to the **generated**
`internal/gen/fkit/payroll/store.go` `Store.Upsert`. Same-name method resolution across two
packages that both define `Store.Upsert` picks the wrong receiver. It is upstream of
allocation — a wrong edge pulls the generated store into the subgraph. CG-10 mitigates the
*symptom* (the generated store is penalized on both score and graph mass, so it no longer
displaces the real one) without fixing the resolution bug itself, which belongs with the
same-name method resolution work (see `samename-method-resolution-1079`).
### 2. `self-query` — the same bug with no generated code in sight
The baseline above, promoted to a fixture: this repo, *"how does explore allocate its output
budget across files"*. `scripts/agent-eval/*.mjs` mention `explore` and `BUDGET`
incidentally — they are eval harnesses, not the allocator — and they are small enough to
ship whole, while `src/mcp/tools.ts` is large enough to be clipped.
At 493 indexed files (small tier): the script corpus takes **71.8%** of the delivered
envelope (79.4% allocated) against `tools.ts`'s **18.5%**, despite `tools.ts` scoring 46 vs
10, carrying 2.3× the graph mass and 3× the distinct term hits.
This fixture reads the **live** index of this repo, so unlike `payroll-go` its exact numbers
move as the repo changes. Its assertions are relative for that reason (answer group vs
incidental group, largest delivered file), never fixed percentages. Two things to know:
- The `<500`-file tier boundary is close. This repo indexes 493 files including the new
fixture; crossing 500 flips `maxOutputChars` 18,000 → 24,000, `maxFiles` 5 → 8 and
`maxCharsPerFile` 3,800 → 6,500, which moves every number in the table above. Re-baseline
after the crossing rather than treating the drift as a regression.
- Adding the `payroll-go` fixture itself moved the count 472 → 493. Its Go files match none
of this query's terms, so they change the tier arithmetic and nothing else.
### Reproducing
The query explores this repo, so **uncommitted edits to `src/mcp/tools.ts` change the
result** — the index picks them up and scores shift (the same query on the CG-4 working
tree reported tools.ts at 13–19% depending on the sync state). Measure against a clean
tree: restore `src/mcp/tools.ts` from `main`, remove `src/mcp/explore-diagnostics.ts`,
`codegraph sync`, then run the built `dist/` binary (which still carries the instrument).
Restore afterwards.
---
## CG-14 — locking the allocation down
The allocation change is one function plus three render-loop bounds, and every way it can
regress is silent: nothing throws, the response just gets less useful and the agent falls
back to Read. So the coverage is built around the question "would this test go red if the
lever were removed?" rather than around line coverage.
### Where the coverage lives
| File | Owns |
|---|---|
| `__tests__/explore-proportional-allocation.test.ts` | `allocateExploreBudget` in isolation — the split, the cliff, the tier invariant, envelope safety, spine weighting, degenerate inputs |
| `__tests__/explore-allocation-e2e.test.ts` | The same behaviours through the real render loop on real indexed projects — the self-query fixture's shape, degenerate result sets, the diffuse-query control |
| `__tests__/explore-allocation-1500.test.ts` | The reporter's Go shape (CG-6 fixture 1), plus the hard-ceiling stress case |
| `scripts/agent-eval/probe-allocation.mjs` | Both CG-6 fixtures against the built `dist/`, including the **live** self-query arm that reads this repo's own index |
The split between the last two is deliberate. The self-query fixture reads a moving target
(this repo), so its exact numbers drift with the tree and it belongs out of band, where a
drift is a number to re-baseline rather than a red suite. `npm test` owns a synthetic
**mirror** of it instead: same three roles, same size asymmetry, fixed.
### The two bounds are not the same bound
Worth stating once, because a test that conflates them looks right and passes for the wrong
reason:
- **`maxOutputChars` bounds the RESERVATIONS.** `sum(allowances) <= pool <= maxOutputChars`,
exactly, at every tier and every candidate shape.
- **`hardCeiling` — `min(maxOutputChars * 1.5, 25000)` — bounds the RESPONSE.** The render
loop is allowed a bounded overshoot (the whole-file grace, an oversize first cluster), so
a response legitimately exceeds the envelope. `payroll-go` does exactly that: 19.3K
delivered against a 13,000 envelope and a 19,500 ceiling.
Only the 25K is absolute. Above it the host writes the result to a file the agent Reads
back, which is the failure the tool exists to prevent.
### Mutation-tested, not just green
Each lever was removed from `src/mcp/tools.ts` in turn and the suite re-run. Every one is
covered by at least one failing test — a lever with no red test is a lever that can be
deleted by accident:
| Mutation | Tests that go red |
|---|---|
| Render loop reverted to pre-CG-12 (`fileBudget = maxCharsPerFile`, whole-file bound `maxCharsPerFile * 3`) | 5 e2e + 2 payroll |
| Proportional split replaced with an equal one | 4 unit |
| Cliff disabled (`cliffAt = 0`) | 7 unit + 3 payroll |
| Spine boost and cliff exemption removed | 3 unit |
| `MIN_CHARS` floor removed | 2 unit |
| `MAX_SHARE` ceiling removed | 4 unit |
The first row is the one that matters most: it reproduces #1500 on the synthetic fixture
verbatim, with the half-as-relevant file taking the larger share purely on size.
| file | score | pre-CG-12 | CG-12 |
|---|---|---|---|
| `src/mcp/allocator.ts` | 77.5 | 4,843 (39.7%) | 9,335 (80.1%) |
| `src/util/budget-math.ts` | 36.0 | 6,079 (49.8%) | 1,037 (8.9%) |
### Two defects the coverage surfaced
Both were found by writing the invariant rather than by reading the code:
1. **Rounded shares could exceed the pool.** `Math.round` on each file's proportional slice
let the reservations sum past `pool` by up to half a char per file — small, but it made
"reservations fit the envelope" false rather than exact. Both terms now floor.
2. **A non-finite score produced a NaN allowance.** An `Infinity` weight makes every share
`Infinity / Infinity`. Scores are finite sums in the pipeline so it was unreachable, but
the failure mode is a NaN handed to the render loop. `weightOf` now fails safe to 0.
### Calibration vs. invariant
`EXPLORE_ALLOCATION` is exported so the tests can read it. Invariant tests (envelope safety,
tier monotonicity, the floor) reference the constants and hold at any value; one test pins
the literals, so re-tuning a constant is a visible decision that says *re-run the probe*
rather than a silent re-calibration of the fixtures.
## CG-15 — what the agent A/B found: a reservation can go unspent
The agent A/B (full record: [`../benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md))
passed on both medium repos — client-go and excalidraw at Read 0 in every run, excalidraw
**34s → 24s median with one fewer explore call**, the generated clientsets that took 10.5% of a
baseline envelope gone from every new run — and **failed on the small control**, express, in 1
run of 3: 4 Reads of `lib/utils.js` where the baseline made 1 Read of a different file.
It is not agent variance. Replaying that run's own query deterministically:
| `lib/utils.js` (5,293 B, 272 lines) | baseline | CG-12 |
|---|---|---|
| delivered | **6,380 (46.1%) whole** | **583 (7.7%) cluster stub** |
| source envelope (budget 13,000) | 13,849 | 9,241 |
The diagnostic says the allocator was right and the render loop was not:
```
allocation 12,398 reserved of 12,400 pool · nothing cliffed
# deliv% bytes reserved score flags render file
1 5.7% 583 3,870 56.0 named entry central clusters lib/utils.js
```
`utils.js` is the **top-ranked** file and was **reserved 3,870 chars — of which it spent 583.**
The whole-file bound (`src/mcp/tools.ts:4008`) is
`allowance + min(GRACE_MAX, allowance * GRACE_FRACTION)` = `3,870 + 580` = 4,450, just under the
file's 5,293 bytes, so the whole-file render is declined; the fallback cluster render has three
matched symbols to work with and emits 583 chars. The remaining 3,287 chars of the reservation
are **not redistributed — they are lost**, which is why the response shrank by a third against
an unchanged budget.
This is CG-12's own acceptance criterion (*"no file that was previously unclipped becomes
clipped"*) failing. CG-14 recorded one instance as a documented exception (`memory-budget.ts`);
this is the same defect in the wild, where it costs an agent round-trip. It is **not** systemic:
on both medium repos the render loop saturates (`[over budget] [TRUNCATED]`, 23,599 of a 23,600
pool reserved) so there is nothing left to lose. It needs a file whose reservation lands below
its own size while its matched-symbol set is thin — likeliest on small repos, where per-file
reservations are smallest.
### The two candidate fixes
Widening the envelope is explicitly **not** one of them — that is the dial iter2 already proved
doesn't work, and the bytes here were reserved for this file already.
1. **Let a large-enough reservation buy the whole file.** Render whole when
`allowance >= k * fileSize` for some `k < 1` (utils.js sits at 0.73), letting the bounded
overshoot the hard ceiling already tolerates absorb the difference. Smaller change, matches
the observed shape.
2. **Redistribute the shortfall.** Once the render loop knows a file's realised size, hand what
it cannot spend to the next-ranked file. The stronger invariant — *the pool is spent* — and
it fixes the shrinking-envelope symptom directly rather than case by case.
They compose; (1) alone would have carried this case. Whichever lands needs the express shape as
a hermetic fixture — a mid-sized top-ranked file with few matched symbols, sized just above its
reservation — because nothing in the current suite has that shape, which is how it shipped.
## CG-21 — spending the reservation
Both fixes landed, because they cover different halves of the same failure and neither is
sufficient alone. The unifying rule: **a reservation is a promise the render loop has to keep,
not a cap it may quietly under-use.**
### 1. A reservation that has already bought most of the file buys the rest
`WHOLE_FILE_BUY_FRACTION` (0.6). The pre-existing grace is calibrated as a *sliver* — it rescues
a file that essentially fits. Between "fits" and "several times its reservation" there was a
hole, and `lib/utils.js` (reserved/size = 0.73) sat squarely in it. So the test is not *does the
file fit the reservation* but *has the reservation already bought most of the file*: above 0.6
the loop pays at most two-thirds of a reservation extra rather than lose the whole thing, on a
file that already earned those bytes.
**Funding is the part that is easy to get wrong.** The merit test is a *ratio*, so wherever
several files sit near it they all qualify — and N independent overshoots inflate the response
until the render ceiling drops whatever is last. Funding each buy from the file's own
headroom was tried and measured on the payroll fixture: three files bought whole and
`payslip_builder.go` — the file that computes the payslip the question asks about — was
**dropped entirely** so three higher-ranked files could each ship their final sliver. A dropped
section is strictly worse than a clustered one.
So the buy rule is **two independent tests**, and keeping them apart is the whole design:
| | reads | answers |
|---|---|---|
| **merit** | the file's own `reserved` | did this file's relevance earn most of itself? |
| **funding** | one shared overshoot pool (`WHOLE_FILE_BUY_OVERSHOOT_FRACTION`, 15% of the envelope) | do the bytes exist, *with every reservation below it still payable*? |
Merit reads `reserved` rather than the post-carry allowance, so borrowed slack can never promote
a weak file to whole. Funding is measured against what the allocator **promised** rather than
against `renderCeiling` — the ceiling sits 50% above the envelope and says nothing about who is
owed what, so funding a buy out of it just moves the shortfall onto whichever file the loop
reaches last. The `owedBelow` term is what makes it a *displacement* guard rather than a size
cap, and it is self-limiting: each buy grows `sourceSpent`, so the pool cannot be spent twice.
### 2. What a file cannot spend goes to the file below it
Below the buy fraction the shortfall is real — the file is several times its reservation and
clustering is the right render — but the bytes still must not evaporate. The render loop carries
two running totals, everything **promised** so far and everything **emitted** so far, and their
gap is slack the next file may add to its own reservation.
Two totals rather than a `spent` variable threaded through the loop's dozen `continue`s, so no
exit path can forget to account — unreadable, drifted off disk, skipped for the ceiling, thin
matched set. It is symmetric: a buy that overshoots makes `sourceSpent` outrun `reservedSoFar`,
which suppresses slack until a later under-spend covers the debt, so the pool is conserved in
both directions and no file is ever cut *below* what it was promised. Slack flows in **rank**
order (the only file a single-pass loop can still pay), clamped by `MAX_SHARE` so an
under-spending leader cannot hand a weak tail file the whole response.
### Measured effect (CG-21)
Express, the reproducer, same index and same 13,000 budget:
| `lib/utils.js` (5,293 B, 272 lines) | baseline | CG-12 | **CG-21** |
|---|---|---|---|
| delivered | 6,380 (46.1%) whole | 583 (7.7%) stub | **6,268 (39.3%) whole** |
| source envelope | 13,849 | 9,241 | **14,505** |
And the self-query, where the epic's own acceptance criterion was outstanding:
| file | pre-CG-12 | CG-12 | **CG-21** |
|---|---|---|---|
| `src/mcp/tools.ts` (score 58) | 32.9% | 60.6% | **52.6%** (10,945, clusters) |
| `src/resolution/memory-budget.ts` (score 18) | 51.2% **whole** | 17.2% clustered | **27.3% whole** (5,672) |
**The `memory-budget.ts` exception is resolved, not re-justified.** CG-14 recorded it as a
documented exception to *"no file that was previously unclipped becomes clipped"*; its
reserved/size ratio is 0.73 — the same window as express's `utils.js` — so the buy rule covers
it. The answer file still wins the envelope **and** nothing that used to ship whole is clipped,
which is the first time both halves of CG-12's acceptance criterion hold at once.
The synthetic mirror in `explore-allocation-e2e.test.ts` deliberately does **not** move: its
helper sits at ratio ~0.17, far below the buy fraction, so it still clusters — correctly. That
divergence is the point of the fraction: `memory-budget.ts` was over-served by a *sliver*, the
synthetic helper by a *multiple*.
### Coverage
Two new hermetic fixtures, one per lever, because the existing suite could not see either. The
payroll and self-query fixtures both **saturate** (`[over budget] [TRUNCATED]`, 23,599 of a
23,600 pool reserved), and a saturated response has no unspent reservation to lose.
| Fixture | Shape | Guards |
|---|---|---|
| `CG-21 — a reservation under the file size still buys the file` | mid-sized rank-1 file, thin matched set, sized just above its reservation and *outside* the grace bound | the buy rule |
| `CG-21 — an unspendable reservation flows to the next file down` | rank-1 file far too big to buy (ratio 0.24) under-spends; a dense rank-2 file absorbs it | the carry-forward |
Each carries a `fixture shape` block that asserts the window it depends on
(`0.6 × size <= reserved < size`, outside the grace bound, inside the 220-line whole-file cap).
Those are load-bearing, not scaffolding: every gate passes **vacuously** if a target ever drifts
small enough for grace to cover it, which is precisely the way this defect hid.
Mutation-tested, same method as CG-14:
| Mutation | Tests that go red |
|---|---|
| Buy arm removed (`buysWhole = fileContent.length <= graceBound`) | 3 (new buy fixture) |
| Carry-forward removed (`allowance = reserved`) | 2 (new carry-forward fixture) |
| Funding guard removed (`… && true`) | 4 — including payroll's `payslip_builder.go` dropped |
Two traps the first drafts fell into, both of which made a test pass on the defect:
- **The spine ceiling hides a per-file assertion.** `SPINE_CEILING` already lets a flow-path
cluster reach 1.5× its allowance, so "delivered > reserved" is true without any carry-forward.
The carry-forward fixture is built spine-free and asserts a 1.1× margin — measured 9,297 vs
7,479 across the mutation.
- **"Spend the whole pool" is not the invariant.** A file smaller than its reservation
legitimately under-spends it (the fixture's `response.ts`: 1,635 delivered of 5,292 reserved).
The assertion is per-file — *delivered >= min(reservation, file size)* — which is what express's
`utils.js` violated and a pool-sum assertion does not express.
### A third condition, found by review rather than by a test
A buy must also **fit the render ceiling**. The whole-file branch refuses to slice a file
mid-method, so a whole render that overruns `renderCeiling` is skipped *entirely* — meaning a
buy approved by the funding pool but refused by the ceiling trades a clustered section for **no
section**. That is the same trade the funding pool exists to refuse, arriving by another route.
It is reachable only on the 24K tiers, which is why neither new fixture can see it:
| tier | envelope | `renderCeiling` = `min(1.5x, 25000) - 600` | funding line = `reservedTotal + 0.15x` |
|---|---|---|---|
| small | 13,000 | 18,900 | ~14,350 — cannot cross |
| medium/large | 24,000 | **24,400** | ~27,200 when saturated — **crosses by ~2.8K** |
So `buysWhole` carries `totalChars + size + FILE_OVERHEAD <= renderCeiling` as well; failing it
drops through to the cluster path, which is bounded by `headroom` and always renders something.
The grace arm is deliberately untouched — a file within a sliver of its reservation that still
does not fit is genuinely at the end of a full response, and that behaviour predates the epic.
Verified inert on all three A/B repos (excalidraw and client-go byte-identical across 3 queries
each, express reproducer unchanged), so it did not invalidate the measurement below.
### The agent A/B (CG-15's gate, re-run)
> **The epic's gate is CG-22, not this section.** This run was measured by the task that wrote
> the fix; CG-22 re-ran it at CG-15's exact setup (`RUNS=3`, fresh clones, baseline pinned to
> `49c11fc` by SHA) and it **passes all four bars there too** — Read = 0 in all 12 new-arm runs
> while the baseline reads in 3 of 3 express runs, and the deterministic reproducer holds with
> both builds re-measured in one session (`lib/utils.js` 6,380 B whole on both, envelope
> 13,849 → 14,913). See
> [`../benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md)
> § CG-22. The two counter-points that section records — excalidraw's answer share running
> below its baseline, and client-go's +1.5s median — are unresolved-but-attributed, not hidden.
Full record: [`../benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explore-allocation-ab-1500.md)
§ "Re-run after CG-21". **All four bars pass**, at n=6 per arm on express and excalidraw:
- **Read = 0 in all 15 new-arm runs.** The express regression that routed the defect here (4
Reads of `lib/utils.js`) does not reproduce in 6 attempts — and the *baseline* reads in 4 of
6, so the control now beats the arm it previously lost to. Median 24.5s → 21.5s.
- **client-go** — the reporter's shape — holds 92.7–96.2% answer share against a baseline run
at 53.8%.
- **Excalidraw's ~8s median gap is not attributable to the change.** Explore's own latency is
374 ms vs 372 ms (n=5, same query and index); deterministic responses differ by +2% with one
byte-identical; and the *unchanged* `main` build's own median moved 34s → 26.5s between the
CG-15 session and this one — the same magnitude as the gap. Agent wall-clock on this repo is
noise-dominated at this sample size, which is the known shape (host-model thinking dominates,
not tool latency).
docs/design/generated-file-detection.md
+123
View File
@@ -0,0 +1,123 @@
# Generated-file detection — path convention plus content banner
CG-5, issue #1500. Companion to `explore-budget-allocation.md` (CG-4's instrument) and a
prerequisite for the scoring overhaul (CG-10).
## The problem
`isGeneratedFile` was path-only. It matches the `<basename>.<tool>.<ext>` convention —
`.pb.go`, `_grpc.pb.go`, `.g.dart`, `_pb2.py` — which is where most codegen output lives,
and which was enough for the cosmos-sdk audit that motivated it.
It is not enough for Go. **Go's convention is a content marker, not a filename one:**
```go
// Code generated by <tool>. DO NOT EDIT.
```
codified by `go generate`, honored by gofmt, golangci-lint and GitHub linguist, and emitted
by protoc-gen-go, mockgen, sqlc, ent, wire, stringer — and by in-house generators. #1500 is
exactly this: a Go monorepo with generated FKIT CRUD in ordinarily-named files
(`payroll.go`) sitting beside hand-written workflow use-cases. Nothing in the path gives it
away, so every generated-file down-rank in the codebase was a no-op on it.
### How large the gap is
Measured on a shallow clone of `kubernetes/client-go` (2,453 Go files):
| signal | files flagged |
|---|---|
| ground truth (grep the canonical banner in the first 60 lines) | 2,001 |
| path convention (`isGeneratedFile`) | **0** |
| content banner (`hasGeneratedHeader`) | **2,001** — 0 false positives, 0 misses |
82% of that repository is generated code with ordinary filenames, and the path check saw
none of it. This is not a long-tail case.
## Design
**Decide at index time, store on the file record, read from the DB.** Explore must never
read file headers per request.
- `isGeneratedFile(path)` — unchanged. Path-only, pure, synchronous, free to call in a sort
comparator. Kept for callers with no database in hand.
- `hasGeneratedHeader(content)` — the content signal (below).
- `detectGeneratedFile(path, content)` — the union, which is what the indexer persists.
- `files.generated INTEGER NOT NULL DEFAULT 0` (schema v9) + a **partial** index
`idx_files_generated ON files(path) WHERE generated = 1`, so lookups cost the generated
minority, not the repo.
- `QueryBuilder.generatedPredicateFor(paths)` / `CodeGraph.generatedFilePredicate(paths)` —
one bounded probe up front, `O(1)` per comparison after, unioned with the path check.
### Why a bounded lookup and not a cached set
Every consumer already holds a short candidate list — a ranked file group, an FTS result
page, a `LIMIT 20` aggregate. Intersecting that list against the partial index needs no
whole-repo set to materialize and, more importantly, **no cache to invalidate**: a ranking
call can never serve a verdict the last sync already replaced. The alternative (a lazily
materialized `Set` of all generated paths) has to be invalidated on every file write and
goes stale on the read-only pool workers, in exchange for saving a sub-millisecond query.
### Precision over recall
A false positive silently demotes hand-written code in every ranking path, so the marker
table is precision-first and the scan is fenced three ways:
1. **Header window only** — first 8,192 chars / 60 lines. Generous enough for build tags
plus an Apache-2.0 preamble above the banner; tight enough that a generator's own source,
which holds the banner as a *string constant in its body*, is not flagged.
2. **Comment lines only** — the marker must sit on a line with a comment leader (`//`, `#`,
`--`, `<!--`, `%`, `;`, `'`, …) or inside an open block comment (`/* */`, `<!-- -->`,
`"""`, `'''`, `=begin`, `<# #>`), tracked with a small state machine over the window.
Generators always emit banners as comments; requiring it rules out identifiers and
string literals that merely contain the words.
3. **Tight markers** — `automatically generated` alone is prose ("the table is automatically
generated at runtime"); `automatically generated **by**` is a banner. `DO NOT EDIT` alone
is a style directive; paired with a generation claim it is a banner.
The module deliberately keeps its own quoted banner literals **below** the header window so
it does not classify itself; `generated-detection.test.ts` pins that, so moving the pattern
table upward fails a test rather than silently demoting this file.
### Migration: no backfill, by necessity
v9 is DDL only. The flag derives from file **content**, which the migration cannot see —
`files` stores a hash, not bytes. Migrated rows stay 0 until a re-index, and because every
reader unions the flag with the path check, an un-backfilled database keeps exactly the
pre-#1500 behavior instead of regressing. `sync` heals it file-by-file as files change.
This is why the CHANGELOG entry says a re-index is required.
## Cost
The acceptance bar was "no measurable index-time cost regression."
A single unanchored `/generat/i` test over the header rejects ~every hand-written file
before any line splitting happens. `String.prototype.slice` on a long string yields a V8
sliced view, not a copy, so the fast path allocates nothing.
- **Microbenchmark** (`detectGeneratedFile` over a whole corpus, 5 passes):
4.6 µs/file on client-go (2,453 files, 14.2 MB, 82% generated — the worst case, where the
gate passes and the full line scan runs), 7.3 µs/file on this repo's `src`.
- **End-to-end** `codegraph init` on client-go, n=3 alternating arms
(current build vs. the same build with the content scan stubbed out):
| arm | runs (s) | median |
|---|---|---|
| with content detection | 5.66, 5.73, 5.89 | **5.73** |
| path-only baseline | 5.52, 5.76, 5.88 | **5.76** |
The arms cross over between runs — the difference is inside run-to-run noise.
## What this task does NOT change
Generated status remains a **stable tiebreak at equal score**, exactly where it was
(`src/mcp/tools.ts` file sort, `findSymbolMatches`, `findAllSymbols`, search formatting,
`getDominantFile`/`getTopRouteFile`/`getRoutingManifest`, the context formatter). A
generated file with a higher raw score still outranks a hand-written one. Turning generated
status into a **strong negative signal** is CG-10, which this task unblocks by making the
signal correct and available.
Verified end-to-end on a two-file Go package where a generated `payroll.go` and a
hand-written `workflow.go` both define `ProcessPayroll`: with the flag set the hand-written
file ranks first; clearing the flag in the same index (i.e. pre-#1500 behavior) puts the
generated file first.