Parity: 0 diffs on redis/git/fmt/protobuf/ALS sweeps; full-init dumps
byte-identical on all five + linux at kernel scale (10.4M dump lines,
same sha256 both arms). Linux 2c/6GB envelope: kernel-arm 19.1min vs
wasm-arm 22.9min (parse 356s vs 435s) on a much richer graph (the new
blanks recover error-swallowed code: git 2x nodes, linux kernel/+mm/ 3x).
Deferral guard corrected by measurement (C/C++ error incidence 9-42%;
--max-deferral flag); defer-reuse memo kills the 3x re-blank/re-parse
cost deferred files paid.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
tree-sitter-wasms shipped 2023-era C/C++ grammars; the kernel walker must
compile the crates.io versions, so production wasm upgrades FIRST and in
isolation (the R2 pattern). Built with tree-sitter-cli 0.25.10 from each
tag's CHECKED-IN parser.c: tree-sitter-c v0.24.2 (b780e47, parser.c
f2883ff9), tree-sitter-cpp v0.23.4 (f41e1a0, parser.c 2a35a43b, scanner.c
cf60387d) — sha-matched against the crates.io tarballs. .metal/.cu map to
language 'cpp' so the dialects ride the same coherent grammar.
Full suite green with the upgraded grammars (2,490; one unrelated daemon
idle-timeout flake passed solo 9/9) — incl. the UE-macro, misparse-guard,
Metal, and CUDA coverage. Checklist updated with the vendored revs and the
metal/cuda routing correction.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Four §7a.1 instrumented-run findings, each measured:
1. File-size trigger + truncate-at-barrier: a fully-backfilled WAL still
grows the FILE without bound — the writer only restarts at frame 0 when a
commit finds zero reader marks, which the instrumented run showed never
happens (file marched 361→721MB through two COMPLETE backfills; 22GB by
phase end). backpressure() now also trips at 4× the soft cap on raw file
size and TRUNCATEs at the parked barrier; the timer path truncates
opportunistically after complete backfills. Dubbo peak: 251MB → 69MB at
the same 16MB valve; dumps byte-identical under aggressive folding.
2. cgroup memory credit: memory.current counts reclaimable page cache — a
post-parse container read 57MB of headroom on a 6GB box and silently
disabled the pool. inactive_file is credited back (the docker-stats
working-set convention); the same run now reads a sane 4.4GB budget.
3. Pool at 2 cores reversed: sequential resolution measured FASTER than
pooled-6-on-2 at kernel scale (853s vs 1,150s), and synthesis is
Amdahl-bound by cFnPtrEdges (306s of 358s) so pooling it bought nothing.
cpuCap = min(ap−1, 6), no floor: ap=2 → sequential is the fast path.
4. Parse floor of 2: one parse worker at a 2-cpuset measured 34% slower
(493s vs 369s) — main + store-worker don't fill the second core. Floor
restores the baseline (373.5s measured).
Plus the observability §7a.1 burned three 25-minute cycles for: valve
armed/fire/timer-pass/heartbeat lines, checkpoint-worker error capture,
pool sizing decisions (incl. the disabled path), backpressure-hook
presence — all behind CODEGRAPH_SYNTH_TIMINGS / CODEGRAPH_WAL_VALVE_DEBUG.
Suite: 2,490 passed / 4 skipped (kernel required). Kernel-scale record
runs with this build follow in the migration plan §7a.1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Pool sizing used os.cpus().length, which enumerates the HOST's CPUs: inside
a 2-CPU cpuset it sized 6 resolver workers (the §7a.1 false-'sequential'
premise) and 8 parse workers, and at true 8-core concurrency six ~1GB
workers OOM-killed a 7GB container (oom_kill=5) mid-synthesis — sizing had
no memory term and no override knob.
resolvePoolSize (pure, matrix-tested): explicit CODEGRAPH_RESOLVE_WORKERS
override (0 disables, cap 16); CPU term max(2, min(availableParallelism-1,
6)) — cpuset-honest, floored at 2 so true 2-core boxes keep pooled
synthesis's ~2×; memory term floor(budget*0.7 / clamp(0.2*dbSize, 256MB,
1.5GB)) with budget = min(freemem, cgroup v2/v1 headroom). Parse pool's
core input switches to availableParallelism. Dev machines are unchanged
(still 6 workers); the 8c/7GB kernel-scale container now sizes 4.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Python (codegraph-kernel/src/python.rs) and Go (src/go.rs) join the
native kernel, mirroring the wasm extractors bug-for-bug. Python:
decorated_definition docstrings/decorators (decorates refs only for
bare-identifier decorators — the call-kind quirk), function-in-class →
method, module-level assignments always extract as variable, from-import
per-name binding refs, self.x fn-ref candidates as bare names. Go:
receiver methods with Recv::name qualified names + contains edges to the
first earlier struct of that name, type_spec struct/interface
classification with embedding→extends and interface method nodes,
composite-literal instantiates keeping the package qualifier, top-level
var/const initializer walks attributed to the declared symbol (#693),
2-hop field chains (#1276), New().Method() re-encode (#645/#608), and
the GO_SPEC fn-ref layers.
Grammars: tree-sitter-python 0.23.6 + tree-sitter-go 0.23.4 crates, with
wasm vendored from the same tags (parser.c sha-matched) — both were
2023-era in tree-sitter-wasms.
Gates: extraction sweeps 100% (flask 83/83, django 3,035/3,038 +3
error-file deferrals, gin 99/99, prometheus 978/979 +1); full-init
dump-diffs byte-identical on flask (10,833 rows), gin (17,540), django
(360,794), and prometheus (213,758); torture fixtures enforced in npm
test. DEFAULT_ROUTED now covers typescript/tsx/javascript/jsx/java/
python/go. Full suite: 2,471 tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kernel-routed files ship their flat tables from the parse worker to the
store worker as buffers (tryKernelExtractRaw → kernelBuffers on the
result → KernelStoreBundle); the store worker decodes and finalizes
(finalizeStoreBundle shared with the object path so filter semantics
can never drift). Files with applicable framework extract() hooks keep
the decoded path; the no-writer fallback materializes via
materializeKernelResult. Byte-identical dumps re-verified on dubbo,
excalidraw, express, gson; full suite green (2,467).
Measurement (plan §4d): dubbo's parse-loop wall is 94% store-writer
busy time — the many-core fresh-index wall is single-writer SQLite
ingest, not extraction or main-thread work. d2s improves the writer
lane ~11% (structured-clone deserialization avoided on the writer) and
frees the main thread; the remaining many-core gap is a
store-architecture arc (deferred index builds, multi-file
transactions, buffer→bind), out of the kernel project's scope.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Java joins the native kernel (codegraph-kernel/src/java.rs), mirroring
the wasm extractor's Java paths bug-for-bug: package namespaces,
imports, javadoc, annotations→decorates, type_list inheritance,
static-final constants, enum constants, anonymous classes (including
the TS side's 0-based-line quirk on the extends ref), method_invocation
calls with the this.field unwrap and the Foo.getInstance().bar() chain
encoding (#645/#608), static-member value reads, method-reference
fn-refs (#756), value-reference edges, and the full Lombok member
synthesizer (#912: Getter/Setter/Data/Value/Builder/ToString/
EqualsAndHashCode/Slf4j-family with taken-member dedup). The shared
docstring/textutil modules moved to crate level. Grammar:
tree-sitter-java 0.23.5, with the wasm grammar vendored from the same
tag (parser.c sha-matched) replacing tree-sitter-wasms' 2023-era build.
Gate (plan §4c): extraction sweeps 100% — gson 262/262, retrofit
341/341, dubbo 4,048/4,048 — plus a Java torture fixture in npm test;
full-init dump-diffs byte-identical on gson (49,766 rows), retrofit
(62,735), and dubbo (441,266 rows); all R2/R3 repos re-verified; Linux
container runs all 23 kernel tests green under CODEGRAPH_KERNEL_EXPECT=1.
The gate caught a real cross-language bug: fn-ref dedupe and value-ref
self-target checks must compare node ID STRINGS, not node-table rows —
ids collide for same-(kind, name, line) nodes, which minified one-line
bundles hit routinely (retrofit's website JS exposed it; latent in the
TS/JS walker since R2, never released). Fixed in both walkers.
Benchmark honesty: dubbo fresh-init on an 11-core Mac is ~flat
(parse-loop wall 5,020→4,394ms; total ~11.3s both arms) because that
wall is main-thread-bound (reads + store), not worker-CPU-bound — the
§6 expectation assumed otherwise. Where worker CPU binds the kernel
delivers: dubbo on a 2-CPU/6GB container drops 27.8-28.6s → 22.3-22.8s
(~1.25×). The identified lever for the many-core headline is decoding
kernel buffers directly into store rows (skipping per-node JS object
materialization); the buffer contract already carries everything.
DEFAULT_ROUTED now includes java. Full suite: 2,467 tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate evidence (docs/design/rust-kernel-migration-plan.md §4b):
- Graph parity, byte-identical (stronger than the §5 ≤0.5% bar): full
codegraph-init dump-diffs kernel-vs-wasm on express (13,712 rows),
excalidraw (89,898), and vscode (2,378,238 rows) — identical bytes.
Python control repo (flask) identical + timing unchanged. The parity
harness is now ORDER-sensitive (emission order drives rowids, which
drive resolution order) and dumps come from the new
scripts/dump-graph.mjs (natural keys, no rowids/timestamps).
- The one real find, caught by the vscode tier: tree-sitter error
RECOVERY is encoding-dependent — byte-identical grammar sources and
the same core (0.25.10) recover erroring files differently under
UTF-8 (native) vs UTF-16 (web-tree-sitter) parsing; proven by
reproducing the wasm tree with a native UTF-16 parse. Policy: the
kernel defers any file whose tree has_error() to the wasm extractor
(silent 'defer:' signal, per file) — parity by construction on
erroring files (incidence 0-0.42% across the gate repos), and the
harness fails if deferrals exceed 10% so a broken kernel can't hide
behind the fallback.
- Retrieval invariants: canonical excalidraw flow (mutateElement →
renderStaticScene) connects end-to-end on the kernel-indexed graph;
synthesized-edge families present. Agent A/B is vacuous under
byte-identical DBs (same justification as #1320-#1322).
- Perf: vscode init 105.4s → 82.1s (1.28×) on an 11-core Mac;
excalidraw on a 2-CPU/6GB Linux container (the CI-runner envelope)
6.2-7.1s → 4.3-4.8s (~1.5×). Linux arm64 in-container build: all 22
kernel tests green under CODEGRAPH_KERNEL_EXPECT=1. Windows VM leg
deferred (VM stopped; prlctl start needs Parallels Pro) — benign: a
missing .node falls back to wasm, and the release matrix builds and
gates the win32 prebuilds.
- Full suite: 2,465 tests pass WITH default-on routing, so the entire
extraction corpus now exercises the kernel for TS/JS wherever a
.node is staged.
DEFAULT_ROUTED = {typescript, tsx, javascript, jsx}. Override:
CODEGRAPH_KERNEL_LANGS (replaces the set) / CODEGRAPH_KERNEL=0 (kill).
Changelog entry added under [Unreleased].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the R1 seed .scm emitter with a bespoke Rust walker
(codegraph-kernel/src/tsjs/) that mirrors TreeSitterExtractor's TS/JS
paths function-for-function: declarations (incl. #808 field/property
classification), qualified names, docstrings (#780 wrapper climbs),
signatures, imports/re-exports + per-binding refs, calls with
receiver-qualified callees (#1230 literal-receiver skip), instantiations,
decorators, inheritance, type annotations (#381), type-alias members +
tuple contracts (#359/#634), React component recognition (#841
forwardRef/memo/styled), object-of-functions / zustand-through-middleware
/ RTK Query endpoints + generated hooks / vuex + pinia store shapes,
function-as-value capture with the flush gate (#756), and value-reference
edges with the shadow prune (#895/#897). The generic query emitter is
deleted — extraction parity needs logic .scm can't express; future
languages get walkers too (migration plan §4a).
Positions and JS string-slice semantics are emitted in UTF-16 code units
natively, so kernel output is byte-identical to web-tree-sitter's — no
column diff class exists.
Parity evidence (macOS): scripts/kernel-parity.mjs (full-object multiset
diff per file) — this repo 353/353 files, excalidraw 643/643 (10,650
nodes / 10,726 edges / 68,307 refs), plus torture fixtures checked into
__tests__/fixtures/kernel-parity/ and enforced in npm test by
kernel-tsjs-parity.test.ts. The strict compare caught one real decoder
bug the loose harness missed: refs must NOT carry denormalized
filePath/language at the extractFromSource seam (the store fills them).
Perf: extraction 2.6× single-thread on excalidraw (487ms vs 1,255ms,
identical outputs). Routing stays opt-in (CODEGRAPH_KERNEL_LANGS) until
the R3 equivalence gate (large repo, DB dump-diff, retrieval invariants,
agent A/B, Linux/Windows) passes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 0 of the Rust extraction-kernel migration (docs/design/
rust-kernel-migration-plan.md, now checked in with §3a recording the
shipped state):
- codegraph-kernel/ napi-rs crate: extractFile(path, content, language)
→ five flat buffers (meta/nodes/edges/refs/arena), one JS boundary
crossing per file. Node ids computed Rust-side, byte-identical to
generateNodeId (pinned by test vector). Reserved per-node metrics slot
for the Arc 3.2 code-metrics work.
- Generic .scm-driven emitter (@def.<kind>/@name/@ref.<kind> captures,
byte-range scope stack → ::-joined qualified names, contains edges,
refs attributed to the innermost enclosing symbol). Seed TS/JS queries
are smoke-level; R2 replaces them with the full port.
- Routing seam in extractFromSource with per-file wasm fallback.
DEFAULT_ROUTED is empty — no behavior change until a language passes
its equivalence gate (R3). Dev opt-in: CODEGRAPH_KERNEL_LANGS. Kill
switch: CODEGRAPH_KERNEL=0. Loader verifies ABI + kind tables before
routing; EDGE_KINDS became a runtime array because kind order is now
wire contract.
- Grammar-source parity: vendored TS/TSX/JS wasm grammars built from the
exact crate revisions (tree-sitter-typescript v0.23.2,
tree-sitter-javascript v0.25.0, checked-in parser.c, ts-cli 0.25.10) —
the tree-sitter-wasms builds were 2023-era, which the new
kernel-grammar-parity test caught on day one. Production TS/JS parsing
gets 2.5 years of grammar fixes; full suite green (2456 tests).
- Build/release wiring: scripts/build-kernel.sh + npm run build:kernel;
release.yml kernel prebuild matrix (continue-on-error — the kernel is
optional everywhere, bundles fall back to the wasm path); bundles stage
lib/kernel/codegraph-kernel.node; release job runs the kernel suites
with CODEGRAPH_KERNEL_EXPECT=1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh init on a 4,402-file Java repo (dubbo): 18.2s -> 13.5s (-26%), with
the resolution phase going 12.6s -> 7.9s (-38%). Graphs verified
byte-identical on both the pool path (dubbo) and the sequential path
(excalidraw).
Two changes:
1. The fastInit+pool path restored WAL for the resolver workers but left
wal_autocheckpoint at its default, so the persist loop inline-checkpointed
hot pages all phase long (#1231's pathology inside resolution — measured
at 58% of resolution wall). Checkpointing is now deferred behind the
bounded valve and folded once at maintenance, mirroring the deferWal path.
2. The resolution loop is double-buffered: batch k+1 is prefetched (OFFSET
past batch k's still-pending rows, under an explicit ORDER BY rowid) and
fanned out across the pool while batch k's ref cleanup runs on the main
thread. Batch settle-waits dropped 2572ms -> 117ms.
Correctness invariant found by the byte-identical gate and now documented in
the loop: batch k+1's resolution READS batch k's edges (resolveMethodOnType
walks supertype chains over extends/implements edges that resolution itself
inserts), so edges must persist BEFORE the next batch fans out; only the ref
cleanup overlaps.
Also extends the CODEGRAPH_SYNTH_TIMINGS instrumentation with phase labels
(grammar-init, parse-loop, fts-rebuild, resolver-reinit, resolution,
callback-synthesis) and pool timings (worker open/resolve, per-batch mode,
persist), so the next profile is one env var away.
Suite green (2444). Sync path timings unchanged. Pool floor re-validated:
forced-on at 40k refs is still net-slower, so the 150k threshold stands.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`.gitignore: /repos/` lists `repos/` as ONE ignored entry, while the
CLI hint (#1156) suggests `includeIgnored: ["repos/a/", "repos/b/"]` —
the child spelling. findIgnoredEmbeddedRepos tested the opt-in matcher
against the PARENT path only, which a child pattern never matches, so
the documented opt-in silently indexed nothing and init looped the
byte-identical suggestion back at the user (#1295).
Ignored dirs that don't match as a whole are now descended (the walk
was already bounded: depth 4 / 2000 entries, and only runs when
includeIgnored is configured) and each nested repo root is matched
individually — parent spelling opts in everything under the dir, child
spelling exactly the named repos. findUnindexedIgnoredRepos gets the
same per-repo check so the hint stops nagging about repos that are
already configured while still naming unopted siblings.
Fixes#1295
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
", ".join(sorted(x)) resolved by bare name to a project function named
join — one nested inside a DIFFERENT function, so scope alone rules the
edge out. Both defects from #1230, fixed independently:
1. Extraction: a member call on a LITERAL receiver (string, number,
collection, regex — across grammars) emits no call ref at all. A
literal's methods are the language's builtins, never project
symbols; the bare-name fallback let them exact-match any same-named
project function. Silent miss, never a wrong edge.
2. Resolution: matchByExactName filters out candidates nested inside a
same-file FUNCTION container unless the ref originates within that
container's line range. Class members (parent is a class-like node),
top-level symbols, and C++ namespace prefixes (no parent node) are
untouched.
requests re-index: byte-identical (813 calls edges). excalidraw: -27
edges, all literal-receiver refs by construction. The issue's repro is
pinned: join has exactly one caller (format_fields), report_missing
has zero project callees.
Fixes#1230
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
target.conn.Exec("insert") with `conn *sql.DB` emitted a BARE `Exec`
ref (the receiver chain was dropped for non-identifier receivers), and
exact-match then bound it to the only local `Exec` — an unrelated
interface's method — fabricating an internal dependency (#1276).
Extraction now keeps Go 2-hop selector chains (`base.field.Method`),
and a dedicated matcher resolves them EXCLUSIVELY via two inference
hops: base's type from the enclosing scope (#1108 machinery), field's
declared type from the struct's own declaration lines (comment-
stripped, per-line — chi's "the tree router" doc comment otherwise
donates a phantom type). resolveMethodOnType validates the target.
Package-qualified field types are followed only when the package is
in-module — `handler http.Handler` must not bind a same-named local
decoy. Failure at any hop leaves the ref unresolved: chained Go
receivers never fall through to the bare-name strategies (they were
never emitted before, so no prior recall depends on that path).
chi before/after: node count stable (1,181); 8 correct field-chain
edges gained (mx.tree.FindRoute/InsertRoute/routes, validated,
including the unexported `node` type); the removed edges are the
prior bare-name guesses on external receivers.
Fixes#1276
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
SEC_ATTR UINT32 LostName(VOID) — an unknown attribute macro before a
typedef'd return type — misparses in tree-sitter's C grammar: the macro
becomes the type, the return type the declarator, and the PARAMETER
LIST is stored as the function name ("(VOID)"). The C++ grammar
recovers this shape via recoverMangledCppName, but in C the real name
never reaches the mangled string, so only a pre-parse blank can help.
Attribute macros are project-specific, so the blank keys on structure:
line-leading ALL-CAPS token followed by TWO identifiers then `(` — the
`MACRO Ret name(` definition shape. Plain typedef'd returns, ALL-CAPS
calls, #define lines, multi-word builtin returns, and mid-line uses are
all rejected by construction. Offset-preserving like the C++ blanks.
curl re-index: 5,531 C functions before and after, zero name changes;
7 nodes in memdebug.c improve start-line accuracy by 1 (the macro line
no longer counts as part of the definition).
Fixes#1211
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
An out-of-line member definition inside a namespace block takes its
qualifiedName from the declarator's receiver, which is spelled RELATIVE
to the enclosing namespace — so `namespace simulator {
ManifestStartup::Output ManifestStartup::Apply(...) {} }` indexed as
ManifestStartup::Apply while the class node carried
simulator::ManifestStartup. Fully-qualified call sites
(simulator::ManifestStartup::Apply(...)) never resolved; callers and
file impact came up empty (#1291).
The receiver-based qualifiedName now composes the active namespace
prefix, anchored at the first prefix segment the receiver re-spells
(so `namespace sim { void sim::M::f() {} }` doesn't double-prefix).
namespacePrefix is only ever non-empty for C++ — Go/Rust/Kotlin/Lua
receivers pass through unchanged.
leveldb re-index: node count byte-stable (3,044), calls edges +6,
namespace-qualified method names 947 -> 1,252.
Fixes#1291
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
template<typename T> T Box<T>::get() stored qualified_name Box<T>::get —
the <T> qualifier never matched the class node indexed as Box, so the
method didn't link to its class, while the inline form of the same
method produced Box::get. ICU-shaped multi-line template parameter
lists leaked whole <…> blocks (newlines included) into qualified_name,
exceeding NAME_MAX for downstream consumers.
extractCppReceiverType now applies stripCppTemplateArgs (the #1043
normalization for base-class refs) to the receiver qualifier.
fmt re-index: template-arg-in-qualifier names 25 -> 4 (remaining are a
FMT_BEGIN_EXPORT misparse artifact and gmock conversion-operator names,
both distinct pre-existing shapes), node count byte-stable at 7,536.
Fixes#1286
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* perf(index): ~34% faster fresh indexing, byte-identical graphs
Profiling a fresh init on a medium TS repo (excalidraw, 657 files) showed
the main thread as the critical path: per-row SQLite statement calls,
repeated import-resolution walks, and per-row FTS trigger firings, with
the parse workers ~75% idle behind it. This lands the semantics-preserving
tranche of fixes:
- Multi-row batched INSERTs (nodes/edges/unresolved refs/name segments)
behind cached per-batch-size prepared statements; row order preserved,
so rowid-based resolution determinism (#1015) is unchanged.
- storeFileBundle: one transaction per file instead of four; nested
transaction() calls now flatten (BEGIN-in-BEGIN previously threw, so no
caller depended on nested rollback).
- Dedicated store-writer thread for the fresh-DB bulk path (bundles
applied in file order on a single writer connection; main thread does
no DB work during the parse loop). Kill switch: CODEGRAPH_NO_STORE_WORKER=1.
- Bulk FTS mode: drop the nodes_fts sync triggers during the bulk load,
rebuild once at the end; crash inside the window self-heals on the
next open.
- Per-context memos for resolveImportPath/findExportedSymbol + a per-file
exported-symbol index, invalidated exactly where clearCaches() already
resets the resolver's own caches.
- Fast-init on completely fresh DBs (journal in memory, no fsync until
the index completes; interrupted init re-runs from scratch). Kill
switch: CODEGRAPH_NO_FAST_INIT=1.
- MaybeYield returns undefined on the not-due path so per-ref yield
checks stop paying a promise + microtask hop each.
- Parse pool prewarm for bulk indexing; compile-cache enabled at CLI and
worker entry points.
Excalidraw fresh init: 5.11s -> 3.36s median (n=5, warm cache, M-series).
Graph dumps byte-identical across init, re-index, and sync paths; full
suite green (2403 passed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* perf(resolution): parallel reference resolution with canonical admission
Fan resolution batches across a pool of read-only worker threads, each
hosting a full ReferenceResolver over its own SQLite connection; results
are admitted on the main thread in chunk order, so edge insertion order,
row cleanup, failure parking, and deferred post-pass queues are exactly
the sequence the single-threaded loop produces. Per-ref inputs match the
baseline because the sequential path already resolves each batch against
the state committed BEFORE that batch.
Validated byte-identical on excalidraw (pool forced on) and apache/dubbo
(4,048 Java files): dubbo full index 39s -> 19s (2.05x) with identical
graph dumps (91,495 nodes / 223,953 edges).
The pool only engages when total pending refs clear a threshold (default
150k, CODEGRAPH_PARALLEL_RESOLVE_MIN to tune, CODEGRAPH_NO_PARALLEL_RESOLVE=1
to disable): measured on a ~58k-ref repo the workers' boot CPU contends
with resolution on the same cores and makes indexing slower, so small
repos keep the sequential path. When fast-init left the DB in
memory-journal mode, WAL is restored before resolution only when the pool
will run (readers + rollback-journal writers don't mix).
Also: sqlite adapter readOnly open support.
TreeCursor spine rewrite of the body walker was built, measured neutral
on real repos and equal in a 20k-child microbench (web-tree-sitter's
namedChild(i) is not quadratic in this binding), and rejected — per-node
JS<->WASM marshaling is the floor, which a traversal swap cannot remove.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Two first-run UX bugs surfaced by indexing a real 1,342-file C repo:
1. After 'Resolving refs' hit 100%, the ~40 dynamic-dispatch synthesis
passes ran with no progress surface, so the bar sat frozen at 100%
long enough to read as a hang (the C fn-pointer pass alone can hold
for a while on C-heavy repos). Synthesis now reports per-pass
progress through a new 'linking' IndexProgress phase, rendered as
'Linking dynamic dispatch'. The step total is pinned by a test to
the synthesizer's actual __mark() count so adding a pass without
bumping it fails loudly.
2. node:sqlite's ExperimentalWarning is emitted once per THREAD, so the
main process plus every parse worker printed it mid-index,
interleaved with the progress UI. All launch paths now pass
--disable-warning=ExperimentalWarning: both bundle launchers, the
Windows npm-shim invocation, and the CLI self-relaunch
(NODE_RUNTIME_FLAGS, deliberately excluded from the re-exec gate so
an older installed launcher never triggers a pointless re-exec, and
version-gated off nodes older than the flag).
Verified end-to-end on the same repo: zero warnings, live linking bar,
byte-identical graph (50,520 nodes / 148,232 edges). Full suite green.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(cpp): resolve explicit operator calls (a.operator+(b)) to the operator method (#1247)
tree-sitter-cpp can't parse an operator_name in field position: the
call_expression carries `function: <receiver>` plus an ERROR child
wrapping the operator_name instead of a field_expression callee, so the
extractor emitted a calls ref named just the receiver (`a`) and the edge
never resolved — while the operator method itself indexed fine.
Two-part fix, scoped to the explicit call form (infix `a + b` / `a[i]`
need receiver type inference and are tracked in #1258):
- extraction: recover the operator_name from the ERROR child and emit
`<receiver>.operator+` (`->` receivers normalized, `this->` emits the
bare name), like any other member call
- resolution: matchMethodCall's dot pattern now admits an operator
method part (cpp-gated; symbol chars failed the \w match), so
receiver-type inference + resolveMethodOnType validate the target —
a same-named operator on an unrelated class can't capture the edge
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cpp): harden explicit operator-call recovery against real-world shapes (#1247)
Validated on nlohmann/json (dozens of explicit operator[] / operator* /
operator< call sites). Two refinements the synthetic fixtures missed:
- normalize spaced call-site operator names (`it.operator * ()`,
`other.operator < (*this)`) to the compact form definitions index as
- drop the ref for a complex receiver (`obj()->operator+`, member chains
ending in a call) instead of emitting a bare operator name: exact-name
fallback GUESSED among unrelated same-named operators (linked a
std::map operator[] call to an in-repo operator[]) — silent miss,
never a wrong edge
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* chore: ignore .kommandr/ directory
* fix(sync): resolve cross-file refs when an edit adds or removes the satisfying symbol (#1240)
Incremental sync scoped reference resolution to the changed files' own
refs, and a completed pass deleted every ref it failed to resolve — so
a symbol change in one file could never repair references in UNCHANGED
files, in either direction, until a full re-index:
- New-export case: a.ts imports/calls `greet` before b.ts defines it.
The failed refs were deleted at index time; when b.ts later gained
`greet`, nothing revisited a.ts — the calls/imports edges stayed
missing while status reported a clean index.
- Removal case: when a re-index (or file deletion) dropped a symbol,
the incoming edges cascade-deleted and the callers — whose resolved
refs had been consumed — never got a chance to rebind to an
alternative definition or reconnect when the symbol returned.
Fix, sharing one lifecycle:
- Schema v8: unresolved_refs gains status ('pending'/'failed') and
name_tail (last dotted segment, so `h.greet` is findable by `greet`).
Both resolver persist paths now park unresolvable refs as failed
instead of deleting them. All pending-work readers (batched drain,
non-progress guard, #1187 orphan sweep, status pendingRefs) filter to
pending, preserving their invariants and keeping status honest.
- Sync retry: after scoped resolution, failed refs whose name tail
matches a symbol name now present in the changed files are re-resolved
through a per-ref-yielding path (watchdog-safe, #1091 class). Names
matching >500 failed refs are skipped as external/builtin noise (#999
rationale).
- Removal side: createEdges stamps each resolution edge with its
originating reference (metadata.refName, + refKind when kind promotion
rewrote it). When the #899 restore misses a target or sync deletes a
file, the dropped edge is resurrected as exactly that ref — re-resolved
in the same sync (rebinding to an alternative definition) or parked
failed until the symbol reappears. Edges without the stamp (pre-upgrade,
synthesized) still drop silently: reconstructing from the target's plain
name would strip receiver context and risk a rebind a full re-index
would never make.
- Pure-removal syncs clear resolver caches so a long-lived daemon can't
resolve resurrected refs against the pre-removal graph.
Validated: issue repro now yields a graph byte-identical to a full
re-index; move/remove-readd/file-deletion scenarios all rebind or heal;
baseline-vs-new A/B on express and gin shows identical node/edge counts
and no timing regression (DB grows ~25% from the parked ref rows — pure
cache, reset by any full re-index). 8 regression tests added.
Fixes#1240
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Parse timeouts are now judged by the worker's own clock: the base timer
only marks a job late (after a long synchronous store stall, Node runs the
timers phase before the poll phase, so the timer fired before an
already-delivered result was processed — killing workers over parses that
took milliseconds, even on 0-byte files); a result arriving before a 3×
hard-kill backstop is accepted, timed-out files are retried, and
CODEGRAPH_PARSE_TIMEOUT_MS overrides the budget. Grammar WASM bytes are
pre-read once on the main thread and handed to every worker, so
spawns/respawns load grammars from memory instead of re-reading a
saturated disk.
Bulk indexing defers WAL auto-checkpointing for the whole run: the default
1000-page interval re-writes hot B-tree/FTS pages into the main DB file
over and over — ~95% of all disk I/O under throttled measurement. A
WalCheckpointValve bounds WAL growth with off-thread PASSIVE backfill
passes (never blocking the writer or the #850 watchdog heartbeat), pauses
the writer for a full backfill if the disk truly can't keep up, and folds
the WAL at the parse→resolution boundary so post-parse reads never page a
bulk-write-sized WAL. Opt out with CODEGRAPH_NO_WAL_DEFER=1; tune with
CODEGRAPH_WAL_VALVE_MB.
Measured at 150 IOPS (HDD class): commons-lang 1526s → 59s with 0 dropped
files (was 8); guava-scale completes in 7.6 min with a full graph where
v1.3.1 needed 25 min for a repo 5× smaller. Unthrottled: no change.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Large-codebase indexing died at the end of "Resolving refs" two ways:
watchdog kills of healthy work (24k-file Java on Windows, #1212 — third
iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale,
where v1.3.0 could not complete at any watchdog setting). Root causes:
~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no
yield points, several materialized whole-graph snapshots (kotlin
expect/actual opened with getAllNodes() — 2M nodes in one array; the
C fn-pointer pass retained every C file's contents twice plus every
function node), and the post-index WAL checkpoint ran minutes of
synchronous IO on the main thread, killing even a successful index at
the finish line.
The pipeline tail now follows the same discipline as the rest: never
hold O(graph) in the heap, yield everywhere.
- All synthesis passes stream node-kind scans (cursors, not arrays) and
yield on time-budgeted checkpoints; language gates skip passes whose
filters a project's file languages provably can't satisfy.
- kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded,
units stream one file at a time, and the all-functions array +
write-only id map are gone; spring reads each .java once, not twice.
- runMaintenance moved to a worker thread (own SQLite connection);
per-file store commits chunk with yields behind a serialized flush
chain (preserving #1015 file-order determinism); resolver warm-up
streams the DISTINCT name set; resolution batch-tail and merged-edge
inserts run in bounded sub-transactions.
- Daemon: fixed a socket-handoff race that could leave a fresh MCP
session permanently silent (client-hello tail unshifted into a
flowing stream with zero listeners — the long-standing #662 test
flake was this real bug); first tool call no longer queues behind
the query pool's cold start (pool.ready gate).
Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully
indexes in 27m8s on a 2-core/6GB container at default heap + default
watchdog; llvm-project (180k files) completes under 1GB RSS including
kill-and-sync recovery; synthesized-edge and full-graph parity are
byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky
daemon test passed 25/25 under load. Env-gated diagnostics kept:
CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop
tracing. Design record: docs/design/main-thread-stall-followup.md.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A Git super-repo whose `.gitignore` excludes its child repositories indexed
~nothing at the parent: CodeGraph respects `.gitignore` by default (#970,
#1065), so the excluded children were skipped and `codegraph init` printed
"Done" with 0 nodes — even though `init` inside each child worked fine. The
empty index was silent and unexplained.
`init`/`index` now detect the gitignored child repos they skipped when an
index comes up empty of symbols, name them, and — in an interactive terminal
— offer to index them (writing an `includeIgnored` entry to codegraph.json and
re-indexing on the spot); non-interactive runs print the exact codegraph.json
snippet to add. Gated on nodesCreated === 0, so a project that deliberately
keeps gitignored reference clones out of a working index is never nagged.
- extraction: findUnindexedIgnoredRepos — the inverse of discoverEmbeddedRepoRoots
(bounded, skips default-ignored dirs, respects existing includeIgnored)
- project-config: addIncludeIgnoredPatterns — create/merge codegraph.json,
idempotent, refuses to clobber malformed JSON
- cli: wire the detect-name-offer flow into both `init` and `index`
- tests: +13 covering detection, config writing, and the no-nag gate
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Carries #1133 forward onto current main (rebased for conflicts). A lean Unreal-Engine-style `.h` whose only C++ signal is `class ENGINE_API Foo : public Bar` (no public:/virtual/namespace/template) was misdetected as C and its class + inheritance edge silently dropped; looksLikeCpp now recognizes the export-macro-annotated class/struct shape, matching what blankCppExportMacros already recovers.
Fixes#1159.
Co-Authored-By: robertyluo <robertyluo@tencent.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Heavily-reflected UE C++ classes (`UCharacterMovementComponent`, `AActor`, `UWorld`, …) were dropped from the index: in-body no-semicolon reflection macros (`UPROPERTY`/`UFUNCTION`/`GENERATED_BODY`/`UE_DEPRECATED_*`), member-level `*_API` prefixes, and mid-line `UMETA`/`UPARAM`/`UE_DEPRECATED` accumulate tree-sitter parse errors until the enclosing class_specifier collapses into an ERROR node. Three offset-preserving, C++-only pre-parse passes (`blankCppAnnotationMacroCalls`, `blankCppApiPrefixMacros`, `blankCppInlineAnnotationMacros`) blank the markup before parsing so the class survives.
Validated on the real Epic engine headers (CharacterMovementComponent.h / Actor.h / World.h): each main class + its base clause goes from DROPPED on main to recovered with the fix.
Closes#1160.
Thanks @luoyxy for the report and root-cause analysis.
Adds an `include` list to the root `codegraph.json` that forces gitignored first-party source (second-VCS / SVN / Perforce dual-tracked repos) into the index — discovered directly off disk on the full index, incremental sync, and file-watching, on both git and non-git projects. Gitignore-style patterns, root-relative; explicit `exclude` still wins and built-in skips (node_modules, dist, .git) are never re-included. Complements `exclude` and `includeIgnored`.
Closes#1163.
Thanks @luoyxy for the contribution.
Four gaps in the MyBatis mapper extractor, all reported and reproduced by
@ESPINS in #1182 and verified against main:
1. Single-quoted attribute values (namespace/id/refid/resultType/parameterType)
were dropped — the regexes hardcoded double quotes. Now accept either quote
via a backreference.
2. Tags inside <!-- ... --> produced phantom statement/include symbols. A
length-preserving, CDATA-aware pre-pass blanks comments before scanning,
keeping offsets/line numbers intact.
3. Legacy iBatis 2 <sqlMap> files had zero statement coverage (the root finder
gated on a <mapper namespace> root). It now also recognizes <sqlMap>
(namespaced and namespace-less DAO.method ids) and iBatis's extra
<statement>/<procedure> verbs — closing the gap with no new dependency
(option (c) from the issue; the batis-xml parser route is declined).
4. Two statements sharing a qualifiedName AND a start line (a vendor-split
databaseId pair on one line) collided on the node id, so INSERT OR REPLACE
silently dropped one. The id-hash now folds in the statement's byte offset;
the stored qualifiedName/startLine are unchanged so the Java<->XML bridge is
untouched.
Gaps 1 and 2 follow @ESPINS's fix-mybatis-quotes-comments branch. Tests add
extractor-level coverage for all four gaps plus a DB-level e2e that proves
iBatis statements land and both vendor-split nodes survive a real indexAll.
Co-authored-by: Jimin Lee <dlwlalsggg@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds ArkTS (.ets, HarmonyOS/OpenHarmony) as a first-class language:
full TypeScript-grade extraction via the harmony-contrib tree-sitter
grammar (MIT, vendored byte-identical from the tree-sitter-arkts 0.2.0
npm tarball), plus the ArkUI constructs that make HarmonyOS apps
traceable:
- @Component/@ComponentV2 structs with decorators from both grammar
positions; members extract as class members with qualified names.
- build() component trees: child instantiation edges via
arkui_component_expression, no synthesizer needed.
- Attribute chains emitted dot-prefixed and resolved ONLY against
@Extend/@Styles/@AnimatableExtend/@Builder helpers (unique-or-drop) —
bare-name fallthrough produced 36,840 wrong edges (17% of calls) on
the OpenHarmony samples monorepo. All four grammar chain shapes
handled, including the detached-chain forms.
- .onClick(this.handler) method-reference bindings.
- ohpm workspace modules: bare imports follow oh-package.json5 file:
deps (ambiguous names dropped), honoring each module's main entry —
which also lets .ts consumers resolve .ets modules.
- ArkUI dynamic-dispatch bridges, all provenance:'heuristic' with
wiring-site metadata: assignment-gated state->build() re-render
(V1 @State family + V2 @Local/@Provider/@Consumer),
@ohos.events.emitter emit->subscriber pairing on static event keys
(numeric ids same-file, named constants same-module, fan-out capped),
and router.pushUrl literal urls -> the target page's @Entry struct.
- $r/$rawfile resource intrinsics treated as built-ins; arkts joins the
web language family, value-reference edges, re-export chase, and the
other TS-applicable gates.
Also ships a language-agnostic index-completeness guard: indexAll
stamps index_state (indexing -> complete/partial/failed), reconciles
discovered vs accounted files (a loaded run silently dropped 37 files),
and codegraph status surfaces truncated/partial indexes in human and
--json output.
Validated on HarmoneyOpenEye (82 files), CoolMallArkTS (528, modular
ohpm + ArkUI V2), and openharmony/applications_app_samples (11,693
files, 202,890 nodes stable across re-index, attribute false-positive
audit 36,840 -> 588 residual all-plausible). Supersedes PRs #656 and
#988 with credit — both informed this implementation.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Follow-ups noted in #1173:
- cloudposse/atmos remote-state: module.M.outputs.X emits a scoped
module.M:remote-output.X candidate; the resolver bridges it to the
target COMPONENT's own output when every gate holds — the module
source is the stack-config remote-state module, the component name is
static (a literal, or component = var.X whose variable declares a
literal default in the same directory), and exactly one directory in
the repo matches the component name and declares that output. Dynamic
(each.value) or ambiguous wiring stays unlinked. On
cloudposse/terraform-aws-components: 254 remote-state bridge edges,
every one re-derived from a matching source declaration (789/789
cross-directory output edges explained: 528 local-module + 254
remote-state + 7 checker-artifact false alarms under deprecated/);
coverage 66.4% -> 69.1%.
- provider aliases: provider "aws" { alias = "east" } is addressed as
provider.aws.east so aliased and default configurations stop
colliding; provider = aws.east on a resource/data block (and the
values of a module's providers map) reference the selected
configuration, resolved same-directory first then up the module tree
— the one construct Terraform genuinely inherits from parents. The
selection is no longer misread as a resource reference (aws.east).
- moved/import/removed blocks reference the resource addresses they
name (anchored to the file node — no phantom symbols), so a
refactor's paper trail joins the graph; check-assert conditions
contribute their references while check-scoped data blocks keep
indexing as before. Scoped module candidates are suppressed there:
module.a.aws_x.b names a resource inside a module instance, not an
output. +91 edges on cloud-foundation-fabric's moved-heavy stages.
Also fixes a latent test bug from #1173: cg.getNodeById is not public
API (cg.getNode is) — it only passed because the asserted edge list was
empty.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(extraction): add Terraform and OpenTofu language support
Index .tf, .tfvars, and .tofu files via the tree-sitter-terraform dialect
of HCL (vendored from @tree-sitter-grammars/tree-sitter-hcl, Apache-2.0).
Symbols extracted:
- resource / data → class (qualified "type.name" / "data.type.name")
- module → module (qualified "module.name")
- variable → variable (qualified "var.name")
- output → variable (qualified "output.name")
- provider → namespace
- locals → constant per attribute (qualified "local.key")
References resolved cross-file:
- var.X, local.X, module.M[.out], data.T.N[.attr], <type>.<name>[.attr]
- built-ins skipped: each.*, count.*, self.*, path.*, terraform.workspace
The Terraform framework resolver disambiguates same-named candidates
across modules by preferring the one in the same directory as the
reference site, then by closest common-ancestor path, falling back to
the generic name matcher only when neither applies.
Validated on two Terraform monorepos (277 and 470 .tf files): indexing
runs in 1.3s and 2.4s respectively, query latency stays under 200ms,
and cross-module references resolve to the correct module 100% of the
time on inspected samples.
18 new extraction tests; full suite 1146/1148 green (2 pre-existing
flaky skips, 0 regressions).
* feat(terraform): bridge the module boundary and enforce directory scoping
Builds on #706. The module declaration was a dead end: module.M.out
resolved to the declaration and stopped, module inputs never reached the
child module's variables, and impact could not cross the boundary — on
real multi-module repos that breaks the core blast-radius question
("what breaks upstream if I change this module's variable/output").
- module blocks now wire across the boundary through :-scoped refs only
the Terraform resolver understands: module.M:var.<input> → the child's
variable node, module.M:output.<o> → the child's output node (emitted
alongside the module.M declaration ref), and module.M:file → the local
source directory's entry file (imports). Registry/git sources emit no
file ref and resolve nothing — an out-of-repo module stays a visible
boundary instead of a guess.
- .tfvars top-level assignments reference the variable they set, walking
up to the nearest ancestor directory (envs/prod.tfvars → root vars).
- Resolution now enforces Terraform's real scoping: same-directory only
(no cross-module fallback by common path prefix, no single-candidate
anywhere-in-tree binding), and terraform refs never fall through to
the generic name matcher — var.X can never legally bind outside its
module directory, so the fallback could only add wrong edges.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(terraform): README language table + changelog entry + agent-eval corpus
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Javier Rodríguez Fernández <jfernandez@freepik.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
CUDA rides the C++ grammar via the Metal (#1121) dialect pattern:
blankCudaConstructs (offset-preserving) blanks execution-space specifiers
(__global__ family), __launch_bounds__(...), and <<<grid, block>>> launch
configs — which otherwise lex as shift operators and destroy the
host→kernel call edge entirely. Gated by .cu/.cuh extension OR by content
(looksLikeCudaSource), because much real CUDA lives in .h/.hpp headers:
cutlass launches most kernels from headers and flash-attention's launch
templates are .h. Safe by construction — no CUDA marker is valid C++
anywhere, and the launch blank is bounded + brace-balance-checked so a
stray <<< (committed merge-conflict markers) can never blank real code.
All real-world launch styles connect: plain, templated
(k<T, 256><<<...>>>), function-pointer (auto kernel = &fn<...>; with
branch reassignments each linked), dim3{...} brace-init configs, and
kernels defined through name-in-first-argument macros
(DEFINE_FLASH_FORWARD_KERNEL style — gtest TEST_F / PYBIND11_MODULE
shapes deliberately excluded by the two-lone-identifiers rule).
Two general C++ resolution wins the flow validation forced out:
- namespace blocks now prefix contained symbols' qualifiedNames
(prefix-only — no namespace nodes, avoiding #1093-style crowd-out), so
ns::fn(...) calls resolve; previously every namespace-qualified C++
call was a permanently dead edge. cutlass: +30,864 edges (~10%), node
count byte-identical.
- templated callees (fn<T, 256>(args)) strip template args at extraction
(mirroring #1043 for base classes), so they match their definitions.
Validated on llm.c (165 host→kernel launch edges, was 0),
flash-attention (run_flash_fwd → flash_fwd_kernel → compute_attn traces
in one codegraph_explore call), and NVIDIA CUTLASS; fmt as the plain-C++
control (unchanged). A/B n=2/arm: Read/Grep displacement decisive on all
three repos (flash-attention Reads 29,13 → 5,2).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
escripts (.escript) index like any module — the ELP grammar has a
first-class shebang node, so no source transform is needed; main/1 and its
helpers get full function/call extraction.
OTP application resource files (<app>.app.src and compiled <app>.app) join
the graph as Erlang terms the grammar parses natively. They route by full
suffix (their last-dot extension, .src, is far too generic for the
extension map). The application tuple yields structure: {mod, {Mod, _}}
links the app to its callback module — the app's entry point — and
{applications, [...]} / {included_applications, [...]} connect umbrella
sibling apps, resolving through the OTP app-name == module-name convention;
kernel/stdlib and other out-of-repo apps stay unresolved.
App-file refs resolve only ever to MODULES: validation on emqx caught the
ssl OTP-app dependency resolving to a test helper FUNCTION named ssl (the
same defect class as the earlier -behaviour gate), so the matchReference
module-only gate now covers every ref an .app/.app.src file emits.
Validated on emqx: 2 app.src + 6 escripts indexed, entry-module and
umbrella-dependency edges all namespace-targeted post-gate, escript
functions extracted; a stray legacy/module.src stays unknown.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Calls hidden inside -define bodies were invisible: the extractor consumed
pp_define without walking the replacement, and macro use sites produced no
edges, so a call path routed through a macro (ejabberd's SQL upsert macros,
logging wrappers) was completely dark.
The macro's constant node now participates in the graph. The -define body's
calls are attributed to the MACRO — true exactly once, instead of a per-use
duplicate that would explode on logging macros — and each use site links
in: ?MACRO(...) with arguments emits a `calls` ref (inlined code joins the
call chain), a bare ?CONSTANT read emits `references` (answering "where is
this macro used" without polluting call paths). Compiler-predefined macros
(?MODULE, ?LINE, ?FUNCTION_NAME, ...) are excluded, macro-use arguments
keep walking so a call nested in ?assertEqual(ok, do_thing()) still
attributes to the enclosing function, and macro-to-macro chains connect.
Validated: node counts unchanged on cowboy/ejabberd/emqx; edges +26/+7.3K/
+42K with honest hub shapes (?T i18n, ?SLOG logging, ?QOS_1 protocol
constants); 40/40 sampled edges precise; +1.3s index cost on emqx's 2,273
files. The payoff chain on ejabberd: set_password_scram_t → ?SQL_UPSERT_T →
ejabberd_sql:sql_query_t — database writes through SQL macros now trace
end-to-end.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
gen_server:call/cast/send_request now connects to the TARGET module's
handle_call/handle_cast for every statically-named target, not just self:
a bare atom reaches the module of that name (OTP's {local, ?MODULE}
convention names a server after its module), and a macro defined as a bare
atom (-define(STORE, kv_store)) resolves the same way, alongside the
existing ?MODULE / -define(SERVER, ?MODULE) self paths. A registered name
that matches no module emits a qualified ref that never resolves — silent,
never guessed. Pid, var, and tuple targets ({global, Name}, {Name, Node})
stay unlinked.
Validated on emqx: 53 new edges, 53/53 precise (each source line is a real
registered-name gen_server request; each target module self-registers under
that name, macro-indirected registrations included). Nearly all are
test-suite → handler links — production code goes through API wrappers the
self path already covers — which is exactly the tests-exercising-this-
handler linkage blast-radius and test-gap reporting consume. ejabberd
yields zero (it always wraps): no false positives invented.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Vendored WhatsApp/tree-sitter-erlang 0.19 (the ELP grammar, ABI 14) with an
Erlang-shaped extractor: multi-clause/multi-arity functions merged into one
symbol, -spec signatures, records with fields, -type/-opaque aliases, -define
macros, -include/-include_lib file edges, and -export-driven visibility.
Modules wrap in a namespace so remote mod:fn(...) calls resolve through the
existing qualified-name matcher as mod::fn with zero resolver changes.
-behaviour declarations link to the behaviour module — gated to namespace
targets only (bare-name fallthrough linked -behaviour(supervisor) to an
unrelated macro constant on emqx). OTP indirection with static targets is
followed: spawn/apply/proc_lib/timer/rpc MFA-argument callees, and
gen_server:call/cast(?MODULE | ?SERVER) to the module's own
handle_call/handle_cast. Var-module dispatch and message sends stay
deliberately unlinked. codegraph_explore also normalizes Erlang-native query
spelling (mod:fn/3, init/2) so named symbols resolve as typed.
Benchmarked on cowboy (189 files), ejabberd (414), emqx (2,447): extraction
PASS on all three; with-codegraph arms reached 2/2/0 file Reads vs 10/5+/19
without, fastest on the largest repo.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Vendored patched govindbanura/tree-sitter-vbnet grammar (MIT, ~20-fix patch
+ new external scanner for XML literals and multi-line LINQ continuation;
provenance + rebuild instructions in docs/grammars/tree-sitter-vbnet.md),
vbnet extractor with VB-specific call/index disambiguation, Inherits/
Implements heritage, As New instantiation, events, Declare P/Invoke, and
MustOverride abstract members.
Parse health on five real repos: PolicyPlus 100%, CompactGUI 100%,
staxrip 95.2%, SCrawler 87.2%, PCL 87.5% (upstream grammar: 3-18%).
Retrieval A/B (sonnet): 26-43% faster with 0-5 file reads vs 7-20 without.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Programs, sections/paragraphs (reconstructed extents over the grammar's
flat header stream), PERFORM/THRU/GO TO/CALL call edges, COPY copybook
imports incl. standalone .cpy fragments, DATA DIVISION records/fields/
88-levels with write-site impact references, and CICS flows: EXEC
LINK/XCTL program targets (literal + same-file VALUE deref), EXEC SQL
INCLUDE, and pseudo-conversational RETURN/START TRANSID hops resolved
to the owning program via a CICS framework resolver. Fixed and free
source format (free format via a scanner wide-mode sentinel).
Grammar: vendored wasm built from a patched yutaro-sakamoto/
tree-sitter-cobol (EXEC blocks as an external-scanner token, copybook
fragment entry point, single-quote continuation, COPY REPLACING
pseudo-text, NOT=, CALL GIVING, ENTRY, FREE, bitwise ops, abbreviated
relations, COBOL-2002 usages, and more). Patch + provenance + upstream
PR draft in docs/grammars/. Parse health: AWS CardDemo 43/44 native
(upstream: 9/31), 44/44 through preParse; copybooks 28/29; CobolCraft
free-format 17/17 (upstream: 0); NIST COBOL85 unchanged at 373/382.
Copybook members resolve to files like C includes (basename index,
name-matcher short-circuit so compiler-supplied members stay honestly
unresolved): CardDemo imports 5 -> 285. Impact proof: ACCT-CURR-BAL
(CVACT01Y copybook) surfaces its 4 writer programs cross-file.
Also: run-all.sh now neutralizes the ambient prompt-hook in both A/B
arms (CODEGRAPH_NO_PROMPT_HOOK=1); COBOL corpus entries for agent-eval.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
CFML joins the #1108 receiver-inference family: new/createObject/typed-arg/property(inject) declarations type the receiver, variables./this. fields scan whole-file, method QNs re-scoped to Class::member in all three extraction paths. 1,649 typed edges on fw1/ColdBox/CFWheels, 1,649/1,649 audit-consistent, inherited methods resolve via #1152 extends edges.
Co-authored-by: ghedwards <125586+ghedwards@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
.metal was absent from EXTENSION_MAP, so Metal Shading Language files were
silently skipped. MSL ≈ C++14, and the C++ grammar extracts its functions,
structs, type aliases, and call edges at parity with plain C++ — except MSL's
post-declarator [[attribute]] annotations, which misparse struct fields into
spurious extends refs from the struct to the field's own type (a wrong
inheritance edge whenever the repo typedefs float3/float4x4 itself, common in
shared ShaderTypes.h). blankMetalAttributes blanks them pre-parse,
offset-preserving, following the blankCppExportMacros pattern (#1061), gated
to .metal files only — in regular C++ the attribute position is legal syntax
the grammar parses natively. The preParse hook gains an optional filePath
param to support the gate.
Validated on llama.cpp's ggml-metal.metal (10.7k lines: 130 kernels vs 113
`kernel void` ground-truth lines, rope_yarn resolves its 4 kernel callers)
and SDL's shaders (PQtoLinear ← GetOutputColor), 0 bogus extends edges.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The Ruby extractor dropped the method name from a `receiver.method` call:
`lg.log()` was recorded as a call to `lg` (the bare receiver), which
matches no symbol, so the reference resolved to nothing and no method
edge was ever produced. A Ruby method invoked through a receiver had no
recorded callers and was invisible to impact/blast-radius and explore
flow traces. This is the Ruby-specific blocker noted in #1108 — that
local-variable type-inference fix couldn't help Ruby because the call
reference itself was missing.
extractCall recognized receiver-bearing calls by the `object`/`name`/
`function` fields other grammars use; tree-sitter-ruby's `call` node uses
`receiver` + `method`, so it fell through to the generic fallback that
takes the first named child (the receiver) as the callee. Handle Ruby
`call`/`method_call` explicitly: build `receiver.method`, keep bare
`foo(...)` as the method name, emit `Foo.new` as an `instantiates` ref,
and give a capitalized (constant) receiver a `references` edge so a class
used only via its class methods still records a dependent.
With this plus #1108, `lg = Logger.new; lg.log` resolves `lg.log` to
`Logger#log`, and the two-file same-name case is same-file-correct
(#1079). Adds Ruby to the local-variable inference test matrix plus a
focused test asserting `Foo.new` stays an instantiation.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(extraction): broaden the curated C++ inline-macro library list
Since #1102 the post-parse salvage already recovers the NAME for any macro, so
adding a library now buys full return-type recovery for it. Extend the curated
list across the major C++ ecosystem: Mozilla/SpiderMonkey, Protobuf, {fmt},
Hedley + nlohmann/json, GLM, Bullet (SIMD_FORCE_INLINE), Skia, OpenCV, EASTL,
Cocos2d-x, Chromium/WebKit (NEVER_INLINE), GLib, SQLite, and the unambiguous
Windows calling conventions (WINAPI / APIENTRY / STDMETHODCALLTYPE / WINAPIV —
which sit between the return type and the name, so blanking them recovers the
return type, e.g. `HRESULT WINAPI Foo()` -> Foo : HRESULT).
Every entry is an exact, curated token matched only in specifier position, so a
real all-caps return type is never touched. Anything still missed keeps its name
via the universal salvage. CARLA control unchanged (440->6 mangles, 0
regressions — none of these libs appear there, confirming no collateral). Eleven
representative full-recovery tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(changelog): note broadened C++ inline-macro library coverage (#1103)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(extraction): universal recovery of macro-mangled C/C++ function names
The curated inline-macro blank list (#1100/#1101) can't enumerate every
library's macro. Add a universal post-parse net so a function is findable by
name regardless of which macro decorates it, plus a batch of common libraries
to the curated list for full name+return-type recovery.
- recoverMangledCppName: after extraction, recover the real identifier from a
name still mangled by an un-blanked macro (`MACRO Ret name(…)` misparses to
"Ret name"). It's a new `recoverMangledName` extractor hook wired only onto
C/C++, applied to every name they produce. Safe by construction: it only
touches an already-mangled name (an internal space that isn't a legit
`operator …`/destructor), so a clean name is returned unchanged; guarded
against the `Ret (name)` parenthesized-name idiom and bare primitives. Scoped
to C/C++ so Kotlin/Scala backtick identifiers (which legitimately contain
spaces) are never touched.
- Curated list extended past UE/pugixml/Godot/Boost to Qt (Q_INVOKABLE, …),
Folly, Abseil, LLVM, V8, Eigen, and rapidjson.
Validated on CARLA (large UE project, 1131 C++/h files) vs the pre-fix baseline:
function-name mangles 440 -> 6, 431 fixed, and — critically — 0 regressions
(the salvage also recovers names that the pre-parse's own non-local error-recovery
shifts would otherwise re-mangle, erasing the 7 shifts seen in #1101). The 6
residual are all the moodycamel `Ret (name)` idiom, correctly left alone. On a
made-up macro with no list entry (`WEBKIT_EXPORT WTFString compute()`), the name
`compute` is still recovered. Full suite green; eleven regression/safety tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(changelog): note universal C++ macro-mangled name recovery (#1102)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(extraction): recognize common third-party C++ inline macros, not just UE
Extend blankCppInlineMacros beyond Unreal Engine's FORCEINLINE family to the
inline/linkage macros that vendored third-party libraries define and that
mangle function names the same way:
- pugixml: PUGI__FN / PUGI__FN_NO_INLINE (before the return type) and
PUGIXML_FUNCTION (linkage macro, between return type and name — the blank
mechanism handles both positions).
- Godot: _FORCE_INLINE_ / _ALWAYS_INLINE_.
- Boost: BOOST_FORCEINLINE / BOOST_NOINLINE.
- Generic cross-ecosystem hints: ALWAYS_INLINE / FORCE_INLINE / NOINLINE.
The list now drives a single generated alternation (longest-token-first), so
adding a codebase's macro is a one-line change. Still curated exact tokens in
specifier position only — a real all-caps return type like `HRESULT DoIt()` is
never touched (verified by controls).
Validated on CARLA (large UE project, 1131 C++/h files): function-name mangles
440 -> 16 (428 fixed). The 16 residual and 7 clean->mangled shifts are all in
third-party vendored files — chiefly pugixml.cpp, a 12k-line macro amalgamation
where error recovery is non-local, so blanking one of several *stacked* macros
(PUGI__FN + PUGI__UNSIGNED_OVERFLOW …) shifts an already-imperfect extraction.
Normal C++/UE code (ActionRoguelike, ALS) sees zero regressions — blanking a
macro there only helps. Chasing pugixml's internal attribute macros is left out
of scope. Seven regression tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(changelog): note third-party C++ inline macro recognition (#1101)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(extraction): recover C++ function names prefixed by an inline-specifier macro
An unknown inline-specifier macro before a function's return type
(`FORCEINLINE FString GetName(…)`) threw tree-sitter into error recovery: the
macro was read as the return type and — for a non-primitive return — the return
type was glued onto the name, so the function was indexed as
`"FString GetName"` instead of `GetName`, unfindable by name and with no caller
links. This is pervasive in Unreal Engine, where inline helpers are written
`FORCEINLINE <ret> <name>(…)` (e.g. ALS's `FORCEINLINE FString GetEnumerationToString`).
Add `blankCppInlineMacros`, a preParse that blanks the known UE inline macros
(`FORCEINLINE`, `FORCENOINLINE`, `FORCEINLINE_DEBUGGABLE`) with equal-length
spaces so byte offsets stay exact and the declaration parses as an ordinary
function — recovering both the real name AND the return type. This is the same
recover-don't-drop approach as blankCppExportMacros (#946/#1061), and the two
are composed into the cppExtractor preParse.
Matched tightly (exact known tokens, only in specifier position — followed by
the identifier that starts the return type/name), so ordinary identifiers, real
all-caps return types (`HRESULT DoIt()`), string literals, expression uses, and
longer words (`FORCEINLINE_COUNT`) are untouched — verified by controls. C++-only;
Kotlin/Scala re-index byte-for-byte identical. Five regression tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(changelog): note C++ inline-specifier-macro function name fix (#1100)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(extraction): correct C++ reference-return and conversion-operator method names
Two pre-existing C++ name-extraction bugs surfaced while validating the #1093
forward-declaration fix against real Unreal Engine repos (ActionRoguelike, ALS):
1. Inline methods/functions returning a reference were named after the whole
declarator. `const int& getRef() const {…}` parses with a reference_declarator
wrapping the function_declarator; extractName unwrapped pointer_declarator but
not reference_declarator, so the method was named "& getRef() const" instead
of "getRef" — polluting search and breaking caller linkage. Ubiquitous in UE
headers (`const FGameplayTagContainer& GetActiveTags() const`). Now the
reference wrapper is unwrapped alongside the pointer wrapper.
2. User-defined conversion operators were named with their full declarator —
`operator EALSMovementState() const` — instead of `operator EALSMovementState`,
so they didn't match the symbolic-overload style (`operator+`) and carried
`() const` noise. The operator_cast declarator is now named `operator <type>`.
Both are additive and C++-scoped (reference_declarator / operator_cast are C++
grammar nodes). Pointer, value, and out-of-line reference returns, and symbolic
operator overloads, are unchanged. Six regression tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(changelog): note C++ reference-return and conversion-operator name fixes (#1096)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A `class Foo;` forward declaration parses as a bodiless class_specifier.
extractStruct (#831) and extractEnum already skip their bodiless forms,
but extractClass did not — so every forward decl across dozens of headers
minted a phantom bodiless `class` node that competed with, and could be
picked as the blast-radius representative over, the single real definition.
Add an opt-in `skipBodilessClass` extractor flag (set only on cppExtractor)
and skip a bodiless class node when it's set, mirroring the struct/enum
skip. The flag keeps this C/C++-scoped: languages where a bodiless class is
a complete definition (Kotlin `class Empty`, Scala `case object`/`trait`)
leave it unset and are unaffected. The body is now resolved once at the top
of extractClass and reused for the member walk.
Regression tests cover the collapse to a single definition, elaborated-type
references creating no phantom, and Kotlin/Scala staying indexed.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
