github/codegraph
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(indexing): bounded-memory yielding pipeline tail + daemon session fixes (#1212) (#1226)

Browse Source 
Large-codebase indexing died at the end of "Resolving refs" two ways:
watchdog kills of healthy work (24k-file Java on Windows, #1212 — third
iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale,
where v1.3.0 could not complete at any watchdog setting). Root causes:
~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no
yield points, several materialized whole-graph snapshots (kotlin
expect/actual opened with getAllNodes() — 2M nodes in one array; the
C fn-pointer pass retained every C file's contents twice plus every
function node), and the post-index WAL checkpoint ran minutes of
synchronous IO on the main thread, killing even a successful index at
the finish line.

The pipeline tail now follows the same discipline as the rest: never
hold O(graph) in the heap, yield everywhere.

- All synthesis passes stream node-kind scans (cursors, not arrays) and
  yield on time-budgeted checkpoints; language gates skip passes whose
  filters a project's file languages provably can't satisfy.
- kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded,
  units stream one file at a time, and the all-functions array +
  write-only id map are gone; spring reads each .java once, not twice.
- runMaintenance moved to a worker thread (own SQLite connection);
  per-file store commits chunk with yields behind a serialized flush
  chain (preserving #1015 file-order determinism); resolver warm-up
  streams the DISTINCT name set; resolution batch-tail and merged-edge
  inserts run in bounded sub-transactions.
- Daemon: fixed a socket-handoff race that could leave a fresh MCP
  session permanently silent (client-hello tail unshifted into a
  flowing stream with zero listeners — the long-standing #662 test
  flake was this real bug); first tool call no longer queues behind
  the query pool's cold start (pool.ready gate).

Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully
indexes in 27m8s on a 2-core/6GB container at default heap + default
watchdog; llvm-project (180k files) completes under 1GB RSS including
kill-and-sync recovery; synthesized-edge and full-graph parity are
byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky
daemon test passed 25/25 under load. Env-gated diagnostics kept:
CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop
tracing. Design record: docs/design/main-thread-stall-followup.md.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby MchenryandClaude Fable 5 authored and GitHub committed 2026-07-08 23:18:23 -05:00
1 parent 58b6bf5c60
commit a3f90089e8
21 files changed
+1036 -264
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

src/resolution/c-fnptr-synthesizer.ts
+95 -76
View File
@@ -52,6 +52,8 @@ import * as path from 'node:path';
import type { Edge, Node } from '../types';
import type { QueryBuilder } from '../db/queries';
import type { ResolutionContext } from './types';
import type { MaybeYield } from './cooperative-yield';
import { LRUCache } from './lru-cache';
import { stripCommentsForRegex } from './strip-comments';
const C_CPP_EXT = /\.(c|h|cc|cpp|cxx|hpp|hh|hxx|cppm|ipp|inl|tcc)$/i;
@@ -306,22 +308,31 @@ const INCLUDE_RE = /#[ \t]*include[ \t]+"([^"\n]+)"/g;
/** Included files worth scanning for registration tables (e.g. a generated `.def`). */
const INCLUDABLE_EXT = /\.(def|inc|h|hh|hpp|hxx|c|cc|cpp|cxx|ipp|tcc|tbl)$/i;
export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
export async function cFnPointerDispatchEdges(_queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
const files = ctx.getAllFiles().filter((f) => C_CPP_EXT.test(f));
if (files.length === 0) return [];
// Cache raw + stripped source per file (read once, reused across passes).
// Raw is needed for `#include "…"` directives — strip blanks string contents.
const rawCache = new Map<string, string | null>();
// Cache raw + stripped source per file, LRU-BOUNDED. The old unbounded Maps
// retained every C/C++ file's raw AND stripped text for the whole pass —
// multiple GB on the Linux kernel, one of the two OOM culprits in #1212.
// Every sweep below iterates in `files` order, and node-kind scans return
// rows in file-commit order, so access is near-sequential and a small LRU
// hits; a miss just re-reads + re-strips.
const rawCache = new LRUCache<string, string | null>(128);
const raw = (file: string): string | null => {
if (rawCache.has(file)) return rawCache.get(file)!;
const r = ctx.readFile(file);
rawCache.set(file, r);
return r;
};
const srcCache = new Map<string, string>();
const srcCache = new LRUCache<string, string>(128);
const src = (file: string): string | null => {
if (srcCache.has(file)) return srcCache.get(file)!;
// A cached '' (empty or unreadable file) returns '' where the miss path
// returns null for unreadable — every caller falsy-checks, so the two are
// interchangeable.
const hit = srcCache.get(file);
if (hit !== undefined) return hit;
const r = raw(file);
const s = r == null ? '' : stripCommentsForRegex(r, 'c');
srcCache.set(file, s);
@@ -347,6 +358,7 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
const fnPtrTypedefs = new Set<string>();
const fnTypeTypedefs = new Set<string>();
for (const file of files) {
if ((++scannedFiles & 15) === 0) await onYield();
const s = src(file);
if (!s || !s.includes('typedef')) continue;
FNPTR_TYPEDEF_RE.lastIndex = 0;
@@ -426,9 +438,10 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
if (fields.some((f) => f.isFnPtr)) structLayout.set(name, fields);
};
for (const st of ctx.getNodesByKind('struct')) {
for (const st of (ctx.iterateNodesByKind?.('struct') ?? ctx.getNodesByKind('struct'))) {
if ((++scannedFiles & 255) === 0) await onYield();
if (!C_CPP_EXT.test(st.filePath)) continue;
const s = srcCache.get(st.filePath) ?? src(st.filePath);
const s = src(st.filePath);
if (!s) continue;
const body = sliceLines(s, st.startLine, st.endLine);
const open = body.indexOf('{');
@@ -444,11 +457,9 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
const fnPtrFieldOf = (struct: string, field: string): boolean =>
!!structLayout.get(struct)?.some((f) => f.name === field && f.isFnPtr);
// C/C++ function + method nodes, materialized once (bounded by C/C++ files).
const cFns: Node[] = [];
for (const fn of iterateFns(queries)) {
if (C_CPP_EXT.test(fn.filePath)) cFns.push(fn);
}
// C/C++ function + method nodes are STREAMED per sweep (see passes D/E) —
// the old materialized `cFns` array held every function node on the repo
// (O(nodes) memory, part of the #1212 kernel OOM).
// ---- function-name → node resolution (prefer a function in the same file) ----
const resolveFn = (name: string, preferFile?: string): Node | null => {
@@ -463,13 +474,13 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
};
// ---- Pass C: registrations — Map<"struct.field", Set<funcNodeId>> ----
// Ids only — retaining the full Node per registration (the old `idToNode`)
// was write-only dead weight at O(registrations) memory.
const reg = new Map<string, Set<string>>();
const idToNode = new Map<string, Node>();
const addReg = (struct: string, field: string, fn: Node): void => {
const key = `${struct}.${field}`;
if (!reg.has(key)) reg.set(key, new Set());
reg.get(key)!.add(fn.id);
idToNode.set(fn.id, fn);
};
// Bare arrays-of-fn-pointers (no struct): array VARIABLE name → per-file sets
@@ -483,7 +494,6 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
let e = entries.find((x) => x.file === file);
if (!e) { e = { file, ids: new Set() }; entries.push(e); }
e.ids.add(fn.id);
idToNode.set(fn.id, fn);
};
// A struct value `{ … }` (one element) — register its function entries to the
@@ -560,25 +570,26 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
};
// Per-file macro + include parsing (any file, indexed or not), cached.
const fnMacroCache = new Map<string, Map<string, MacroDef>>();
// Derived per-file caches, LRU-bounded like the content caches (#1212).
const fnMacroCache = new LRUCache<string, Map<string, MacroDef>>(256);
const fileFnMacros = (file: string): Map<string, MacroDef> => {
let m = fnMacroCache.get(file);
if (!m) { m = parseFunctionMacros(src(file) ?? ''); fnMacroCache.set(file, m); }
return m;
};
const objMacroCache = new Map<string, Map<string, string>>();
const objMacroCache = new LRUCache<string, Map<string, string>>(256);
const fileObjMacros = (file: string): Map<string, string> => {
let m = objMacroCache.get(file);
if (!m) { m = parseObjectMacros(src(file) ?? ''); objMacroCache.set(file, m); }
return m;
};
const definedCache = new Map<string, Set<string>>();
const definedCache = new LRUCache<string, Set<string>>(256);
const fileDefinedNames = (file: string): Set<string> => {
let d = definedCache.get(file);
if (!d) { d = parseDefinedNames(src(file) ?? ''); definedCache.set(file, d); }
return d;
};
const includeCache = new Map<string, string[]>();
const includeCache = new LRUCache<string, string[]>(1024);
const localIncludesOf = (file: string): string[] => {
let out = includeCache.get(file);
if (out) return out;
@@ -635,39 +646,7 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
objEnv: Map<string, string>;
}
const indexedSet = new Set(files);
const units: Unit[] = [];
const seenInclude = new Set<string>();
for (const file of files) {
const env = new Map<string, MacroDef>();
const objEnv = new Map<string, string>();
const defined = new Set<string>();
buildEnv(file, 2, new Set(), env, objEnv, defined);
const s = src(file);
if (s) units.push({ text: s, file, env, objEnv });
for (const target of localIncludesOf(file)) {
if (seenInclude.has(`${file}>${target}`)) continue;
const incSrc = src(target);
if (!incSrc) continue;
if (indexedSet.has(target)) {
// Re-scan an indexed header only when this includer unlocks guarded code.
const ownDef = fileDefinedNames(target);
const adds = [...defined].some((n) => !ownDef.has(n));
if (!adds || !/#\s*if/.test(incSrc)) continue;
}
seenInclude.add(`${file}>${target}`);
// The include is pasted into the includer — evaluate its conditionals in
// the includer's defined set (a no-op when it has none). Re-parse the
// included file's OWN macros from that resolved text so a macro it defines
// conditionally (vim's `EXCMD`, whose plain last-wins parse picks the enum
// arm) overrides with the ARM THAT IS ACTUALLY ACTIVE here.
const text = evalConditionals(incSrc, defined);
const incEnv = new Map(env);
for (const [k, v] of parseFunctionMacros(text)) incEnv.set(k, v);
const incObjEnv = new Map(objEnv);
for (const [k, v] of parseObjectMacros(text)) incObjEnv.set(k, v);
units.push({ text, file: target, env: incEnv, objEnv: incObjEnv });
}
}
// Global variable → struct type, for resolving a dispatch through a file-scope
// table by subscript (`cmdnames[i].cmd_func(…)`).
@@ -716,9 +695,12 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
// (below) is what separates this from a plain data/struct array.
const ARRAY_TABLE_RE =
/(?:^|[;{}])\s*(?:(?:static|const|extern|register|volatile)\s+)*(\w+)\s+(\*\s*)?(\w+)\s*\[[^\]]*\]\s*=\s*\{/g;
for (const unit of units) {
// Process ONE unit's text and discard it. The old shape built every unit up
// front (`const units: Unit[]`) — the full text of every C file plus its
// expanded includes held simultaneously, gigabytes on the kernel (#1212).
const processUnit = (unit: Unit): void => {
const s = unit.text;
if (!s || !s.includes('{')) continue;
if (!s || !s.includes('{')) return;
INLINE_STRUCT_RE.lastIndex = 0;
let im: RegExpExecArray | null;
@@ -745,7 +727,7 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
}
}
if (!s.includes('=')) continue;
if (!s.includes('=')) return;
INIT_RE.lastIndex = 0;
let m: RegExpExecArray | null;
while ((m = INIT_RE.exec(s))) {
@@ -777,6 +759,41 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
registerArrayValue(am[3]!, s.slice(open + 1, close), unit.file, unit.env);
ARRAY_TABLE_RE.lastIndex = close;
}
};
// ---- Pass C: registrations — stream each file (and its qualifying local
// includes) through processUnit, one at a time.
for (const file of files) {
if ((++scannedFiles & 15) === 0) await onYield();
const env = new Map<string, MacroDef>();
const objEnv = new Map<string, string>();
const defined = new Set<string>();
buildEnv(file, 2, new Set(), env, objEnv, defined);
const s = src(file);
if (s) processUnit({ text: s, file, env, objEnv });
for (const target of localIncludesOf(file)) {
if (seenInclude.has(`${file}>${target}`)) continue;
const incSrc = src(target);
if (!incSrc) continue;
if (indexedSet.has(target)) {
// Re-scan an indexed header only when this includer unlocks guarded code.
const ownDef = fileDefinedNames(target);
const adds = [...defined].some((n) => !ownDef.has(n));
if (!adds || !/#\s*if/.test(incSrc)) continue;
}
seenInclude.add(`${file}>${target}`);
// The include is pasted into the includer — evaluate its conditionals in
// the includer's defined set (a no-op when it has none). Re-parse the
// included file's OWN macros from that resolved text so a macro it defines
// conditionally (vim's `EXCMD`, whose plain last-wins parse picks the enum
// arm) overrides with the ARM THAT IS ACTUALLY ACTIVE here.
const text = evalConditionals(incSrc, defined);
const incEnv = new Map(env);
for (const [k, v] of parseFunctionMacros(text)) incEnv.set(k, v);
const incObjEnv = new Map(objEnv);
for (const [k, v] of parseObjectMacros(text)) incObjEnv.set(k, v);
processUnit({ text, file: target, env: incEnv, objEnv: incObjEnv });
}
}
// ---- receiver-type resolution within a function's source ----
@@ -828,19 +845,23 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
// a fixpoint so a hook slot inherits a registry field's handlers.
const FIELD_ASSIGN_RE = /(\w+)\s*(?:->|\.)\s*(\w+)\s*=\s*(\w+)\s*(?:->|\.)\s*(\w+)/g;
const propagations: { to: string; from: string }[] = [];
for (const fn of cFns) {
const s = srcCache.get(fn.filePath);
if (!s) continue;
const body = sliceLines(s, fn.startLine, fn.endLine);
if (!body.includes('=')) continue;
FIELD_ASSIGN_RE.lastIndex = 0;
let m: RegExpExecArray | null;
while ((m = FIELD_ASSIGN_RE.exec(body))) {
const [, lrecv, lfield, rrecv, rfield] = m;
const lt = recvTypeIn(body, lrecv!);
const rt = recvTypeIn(body, rrecv!);
if (lt && rt && fnPtrFieldOf(lt, lfield!) && fnPtrFieldOf(rt, rfield!)) {
propagations.push({ to: `${lt}.${lfield}`, from: `${rt}.${rfield}` });
for (const file of files) {
if ((++scannedFiles & 15) === 0) await onYield();
const s = src(file);
if (!s || !s.includes('=')) continue;
for (const fn of ctx.getNodesInFile(file)) {
if (!FN_KINDS.has(fn.kind)) continue;
const body = sliceLines(s, fn.startLine, fn.endLine);
if (!body.includes('=')) continue;
FIELD_ASSIGN_RE.lastIndex = 0;
let m: RegExpExecArray | null;
while ((m = FIELD_ASSIGN_RE.exec(body))) {
const [, lrecv, lfield, rrecv, rfield] = m;
const lt = recvTypeIn(body, lrecv!);
const rt = recvTypeIn(body, rrecv!);
if (lt && rt && fnPtrFieldOf(lt, lfield!) && fnPtrFieldOf(rt, rfield!)) {
propagations.push({ to: `${lt}.${lfield}`, from: `${rt}.${rfield}` });
}
}
}
}
@@ -875,9 +896,12 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
const ARRAY_DISPATCH_RE = /(?:\(\s*\*\s*)?\b(\w+)\s*\[[^\][]*\]\s*\)?\s*\(/g;
const edges: Edge[] = [];
const seen = new Set<string>();
for (const fn of cFns) {
const s = srcCache.get(fn.filePath);
for (const file of files) {
if ((++scannedFiles & 15) === 0) await onYield();
const s = src(file);
if (!s) continue;
for (const fn of ctx.getNodesInFile(file)) {
if (!FN_KINDS.has(fn.kind)) continue;
const body = sliceLines(s, fn.startLine, fn.endLine);
DISPATCH_RE.lastIndex = 0;
let m: RegExpExecArray | null;
@@ -956,12 +980,7 @@ export function cFnPointerDispatchEdges(queries: QueryBuilder, ctx: ResolutionCo
}
}
}
}
}
return edges;
}
/** C/C++ function + method nodes, streamed (memory-safe on symbol-dense repos). */
function* iterateFns(queries: QueryBuilder): IterableIterator<Node> {
yield* queries.iterateNodesByKind('function');
yield* queries.iterateNodesByKind('method');
}
src/resolution/callback-synthesizer.ts
+243 -96
View File
@@ -280,11 +280,13 @@ async function closureCollectionEdges(queries: QueryBuilder, ctx: ResolutionCont
/** Phase 2: string-keyed EventEmitter channels (on('e', fn) ↔ emit('e')). */
async function eventEmitterEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
const emitsByEvent = new Map<string, Set<string>>(); // event → dispatcher node ids
const handlersByEvent = new Map<string, Map<string, string>>(); // event → handler id → registration site (file:line)
let scanned = 0;
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if ((++scanned & 255) === 0) await onYield(); // #1091: yield mid-scan on huge graphs
const content = ctx.readFile(file);
if (!content) continue;
@@ -348,10 +350,12 @@ async function eventEmitterEdges(ctx: ResolutionContext, onYield: MaybeYield): P
* `this.setState`). Over-approximation (all setState methods reach render) is
* accepted — it's reachability-correct, like the callback channels.
*/
function reactRenderEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
async function reactRenderEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
for (const cls of queries.getNodesByKind('class')) {
for (const cls of queries.iterateNodesByKind('class')) {
if ((++scanned255 & 63) === 0) await onYield();
const children = queries.getOutgoingEdges(cls.id, ['contains'])
.map((e) => queries.getNodeById(e.target))
.filter((n): n is Node => !!n && n.kind === 'method');
@@ -387,10 +391,12 @@ function reactRenderEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[]
* body calls `setState(` → `build`. The setState gate + `.dart` file keep this to
* Flutter State classes. Over-approximation accepted (reachability-correct).
*/
function flutterBuildEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
async function flutterBuildEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
for (const cls of queries.getNodesByKind('class')) {
for (const cls of queries.iterateNodesByKind('class')) {
if ((++scanned255 & 63) === 0) await onYield();
const children = queries.getOutgoingEdges(cls.id, ['contains'])
.map((e) => queries.getNodeById(e.target))
.filter((n): n is Node => !!n && n.kind === 'method');
@@ -447,10 +453,12 @@ const ARKUI_ARRAY_MUTATORS = 'push|pop|shift|unshift|splice|sort|reverse|fill';
* with no reactive properties, gets nothing (this is the precision line the
* all-sibling-methods design would erase).
*/
function arkuiStateBuildEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
async function arkuiStateBuildEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
for (const struct of queries.getNodesByKind('struct')) {
for (const struct of queries.iterateNodesByKind('struct')) {
if ((++scanned255 & 63) === 0) await onYield();
if (struct.language !== 'arkts') continue;
const children = queries.getOutgoingEdges(struct.id, ['contains'])
.map((e) => queries.getNodeById(e.target))
@@ -515,7 +523,8 @@ const ARKUI_EMITTER_FANOUT_CAP = 8;
* handling — their bodies' calls already attribute to the registering method,
* so targeting that method keeps the chain connected.
*/
function arkuiEmitterEdges(ctx: ResolutionContext): Edge[] {
async function arkuiEmitterEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
interface Site { nodeId: string; file: string; line: number }
// bucket key -> emit sites / handler sites
const emits = new Map<string, Site[]>();
@@ -533,6 +542,7 @@ function arkuiEmitterEdges(ctx: ResolutionContext): Edge[] {
};
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!file.endsWith('.ets')) continue;
const content = ctx.readFile(file);
if (!content || !content.includes('emitter.')) continue;
@@ -619,7 +629,7 @@ const ARKUI_ROUTER_RE = /\brouter\s*\.\s*(?:pushUrl|replaceUrl)\s*\(\s*\{[^)]{0,
* anything still ambiguous is dropped rather than guessed. Only `@Entry`
* structs qualify as targets — the decorator is what makes a file a page.
*/
function arkuiRouterEdges(ctx: ResolutionContext): Edge[] {
async function arkuiRouterEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
const edges: Edge[] = [];
const seen = new Set<string>();
@@ -635,7 +645,9 @@ function arkuiRouterEdges(ctx: ResolutionContext): Edge[] {
return '';
};
let scannedFiles = 0;
for (const file of allFiles) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!file.endsWith('.ets')) continue;
const content = ctx.readFile(file);
if (!content || !content.includes('router.')) continue;
@@ -690,7 +702,8 @@ function arkuiRouterEdges(ctx: ResolutionContext): Edge[] {
* implementation(s). Over-approximation accepted (reachability-correct); capped
* per class and gated to C++ to avoid touching other languages' dispatch.
*/
function cppOverrideEdges(queries: QueryBuilder): Edge[] {
async function cppOverrideEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
const methodsOf = (classId: string): Node[] =>
@@ -698,7 +711,8 @@ function cppOverrideEdges(queries: QueryBuilder): Edge[] {
.getOutgoingEdges(classId, ['contains'])
.map((e) => queries.getNodeById(e.target))
.filter((n): n is Node => !!n && n.kind === 'method');
for (const cls of queries.getNodesByKind('class')) {
for (const cls of queries.iterateNodesByKind('class')) {
if ((++scanned255 & 63) === 0) await onYield();
const subMethods = methodsOf(cls.id).filter((n) => n.language === 'cpp');
if (subMethods.length === 0) continue;
for (const ext of queries.getOutgoingEdges(cls.id, ['extends'])) {
@@ -762,7 +776,8 @@ const IFACE_OVERRIDE_LANGS = new Set([
* with the other dispatch synthesizers; capped per interface. Empty interfaces
* (`any`) are skipped so they don't match every struct.
*/
function goImplementsEdges(queries: QueryBuilder): Edge[] {
async function goImplementsEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
@@ -775,11 +790,21 @@ function goImplementsEdges(queries: QueryBuilder): Edge[] {
.map((n) => n.name),
);
const goStructs = queries.getNodesByKind('struct').filter((s) => s.language === 'go');
// Materializes GO structs only (the pass is language-gated by the caller),
// never the whole struct kind — that array is O(nodes) on struct-heavy
// repos like the Linux kernel (#1212).
const goStructs: Node[] = [];
for (const s of queries.iterateNodesByKind('struct')) {
if ((++scanned255 & 63) === 0) await onYield();
if (s.language === 'go') goStructs.push(s);
}
const structMethods = new Map<string, Set<string>>();
for (const s of goStructs) structMethods.set(s.id, methodNameSet(s.id));
for (const iface of queries.getNodesByKind('interface')) {
for (const iface of queries.iterateNodesByKind('interface')) {
if ((++scanned255 & 63) === 0) await onYield();
if ((++scanned255 & 63) === 0) await onYield();
if (iface.language !== 'go') continue;
const want = methodNameSet(iface.id);
if (want.size === 0) continue; // empty interface (`any`) — would match everything
@@ -831,7 +856,8 @@ function goImplementsEdges(queries: QueryBuilder): Edge[] {
* matching the same-file edges extraction already emits). Skips methods that
* already have a type parent (the same-file case). (#583, cross-file half)
*/
function goCrossFileMethodContainsEdges(queries: QueryBuilder): Edge[] {
async function goCrossFileMethodContainsEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
const TYPE_KINDS = new Set<NodeKind>(['struct', 'class', 'interface', 'enum', 'type_alias']);
@@ -840,7 +866,10 @@ function goCrossFileMethodContainsEdges(queries: QueryBuilder): Edge[] {
return i >= 0 ? p.slice(0, i) : '';
};
for (const method of queries.getNodesByKind('method')) {
for (const method of queries.iterateNodesByKind('method')) {
if ((++scanned255 & 63) === 0) await onYield();
if ((++scanned255 & 63) === 0) await onYield();
if (method.language !== 'go') continue;
// The receiver type is encoded in the method's qualifiedName as `Recv::name`
// (extraction sets `${receiverType}::${name}` for receiver methods).
@@ -908,13 +937,18 @@ function kmpKindsCompatible(a: string, b: string): boolean {
return a === b || (KMP_TYPE_KINDS.has(a) && KMP_TYPE_KINDS.has(b));
}
function kotlinExpectActualEdges(queries: QueryBuilder): Edge[] {
async function kotlinExpectActualEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
const actuals = queries
.getAllNodes()
.filter((n) => n.language === 'kotlin' && !!n.decorators?.includes('actual'));
for (const act of actuals) {
// SQL-side language+decorator pre-filter, streamed. The old
// `getAllNodes().filter(...)` hydrated the ENTIRE node table into one array
// just to find kotlin `actual` declarations — on a 2M-node graph that alone
// exceeded Node's default heap and killed the index (#1212). The LIKE
// pre-filter can over-match (substring), so the exact decorator check stays.
for (const act of queries.iterateNodesByLanguageWithDecorator('kotlin', 'actual')) {
if ((++scanned255 & 63) === 0) await onYield();
if (!act.decorators?.includes('actual')) continue;
let added = 0;
for (const cand of queries.getNodesByQualifiedNameExact(act.qualifiedName)) {
if (added >= MAX_CALLBACKS_PER_CHANNEL) break;
@@ -944,7 +978,8 @@ function kotlinExpectActualEdges(queries: QueryBuilder): Edge[] {
return edges;
}
function interfaceOverrideEdges(queries: QueryBuilder): Edge[] {
async function interfaceOverrideEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
const methodsOf = (classId: string): Node[] =>
@@ -957,7 +992,8 @@ function interfaceOverrideEdges(queries: QueryBuilder): Edge[] {
// types that conform to protocols. Iterate both.
const concreteKinds = ['class', 'struct'] as const;
for (const kind of concreteKinds) {
for (const cls of queries.getNodesByKind(kind)) {
for (const cls of queries.iterateNodesByKind(kind)) {
if ((++scanned255 & 63) === 0) await onYield();
const implMethods = methodsOf(cls.id).filter((n) => IFACE_OVERRIDE_LANGS.has(n.language));
if (implMethods.length === 0) continue;
for (const sup of queries.getOutgoingEdges(cls.id, ['implements', 'extends'])) {
@@ -1023,7 +1059,8 @@ function interfaceOverrideEdges(queries: QueryBuilder): Edge[] {
* Provenance: `heuristic`, `synthesizedBy: 'go-grpc-stub-impl'`. The
* stub's source line is the wiring site shown in the trace trail.
*/
function goGrpcStubImplEdges(queries: QueryBuilder): Edge[] {
async function goGrpcStubImplEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
@@ -1037,7 +1074,8 @@ function goGrpcStubImplEdges(queries: QueryBuilder): Edge[] {
const methodNamesByStruct = new Map<string, Set<string>>();
const methodNodesByStruct = new Map<string, Node[]>();
const goStructs: Node[] = [];
for (const s of queries.getNodesByKind('struct')) {
for (const s of queries.iterateNodesByKind('struct')) {
if ((++scanned255 & 63) === 0) await onYield();
if (s.language !== 'go') continue;
goStructs.push(s);
const ms = queries
@@ -1117,11 +1155,13 @@ function goGrpcStubImplEdges(queries: QueryBuilder): Edge[] {
* (or nothing) and are dropped.
*/
async function reactJsxChildEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
const PARENT_KINDS = new Set(['method', 'function', 'component']);
let scanned = 0;
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if ((++scanned & 255) === 0) await onYield(); // #1091: yield mid-scan on huge graphs
const content = ctx.readFile(file);
if (!content || (!content.includes('</') && !content.includes('/>'))) continue; // JSX-file gate
@@ -1167,7 +1207,9 @@ async function reactJsxChildEdges(ctx: ResolutionContext, onYield: MaybeYield):
* component, handler→function/method) keeps precision; inline arrows / `$emit`
* skipped.
*/
function vueTemplateEdges(ctx: ResolutionContext): Edge[] {
async function vueTemplateEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
const COMPONENT_KINDS = new Set(['component', 'function', 'class']);
@@ -1181,11 +1223,13 @@ function vueTemplateEdges(ctx: ResolutionContext): Edge[] {
// misses it (flat components match by basename and don't need this). Map each
// nested component's Nuxt name → node so those template usages resolve.
const nuxtComponents = new Map<string, Node>();
for (const c of ctx.getNodesByKind('component')) {
for (const c of (ctx.iterateNodesByKind?.('component') ?? ctx.getNodesByKind('component'))) {
if ((++scanned255 & 63) === 0) await onYield();
const nn = nuxtComponentName(c.filePath);
if (nn && !nuxtComponents.has(nn)) nuxtComponents.set(nn, c);
}
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!file.endsWith('.vue')) continue;
const content = ctx.readFile(file);
const tpl = content && content.match(/<template[^>]*>([\s\S]*)<\/template>/i)?.[1];
@@ -1310,7 +1354,8 @@ const RN_JVM_EMIT_RE = /\.emit\s*\(\s*"([^"]+)"\s*,/g;
// is followed by `… ) {`) never matches. Multi-line tolerant. (java/kotlin/swift)
const RN_NATIVE_SENDEVENT_RE = /\bsendEvent\s*\([^;{}]*?"([^"]+)"/g;
function rnEventEdges(ctx: ResolutionContext): Edge[] {
async function rnEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
// Native dispatchers (source = the native method whose body sends the
// event) and JS handlers (target = the function/method registered as
// the listener) keyed by event name.
@@ -1318,6 +1363,7 @@ function rnEventEdges(ctx: ResolutionContext): Edge[] {
const jsHandlersByEvent = new Map<string, Map<string, string>>();
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
const content = ctx.readFile(file);
if (!content) continue;
@@ -1500,11 +1546,13 @@ const FABRIC_NATIVE_SUFFIXES = ['', 'View', 'ViewManager', 'ComponentView', 'Man
* caller. The Expo method nodes are id-prefixed `expo-module:` and qualified
* `<file>::<module>.<method>` by the framework extractor.
*/
function expoCrossPlatformEdges(queries: QueryBuilder): Edge[] {
async function expoCrossPlatformEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
const byKey = new Map<string, Node[]>();
for (const m of queries.getNodesByKind('method')) {
for (const m of queries.iterateNodesByKind('method')) {
if ((++scanned255 & 63) === 0) await onYield();
if (!m.id.startsWith('expo-module:')) continue;
const key = m.qualifiedName.split('::').pop(); // `<module>.<method>`
if (!key) continue;
@@ -1547,7 +1595,8 @@ function expoCrossPlatformEdges(queries: QueryBuilder): Edge[] {
* `getFreeDiskStorage`) — that's the JS-visible name, and how the iOS selector
* lines up with the bare Android method name.
*/
function rnCrossPlatformEdges(queries: QueryBuilder): Edge[] {
async function rnCrossPlatformEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
const NATIVE = new Set(['java', 'kotlin', 'objc', 'cpp']);
@@ -1570,6 +1619,7 @@ function rnCrossPlatformEdges(queries: QueryBuilder): Edge[] {
// below only runs for genuine cross-platform candidates.
const byName = new Map<string, Node[]>();
for (const m of queries.iterateNodesByKind('method')) {
if ((++scanned255 & 63) === 0) await onYield();
if (!NATIVE.has(m.language)) continue;
const key = norm(m.name);
const arr = byName.get(key);
@@ -1613,18 +1663,25 @@ function rnCrossPlatformEdges(queries: QueryBuilder): Edge[] {
return edges;
}
function fabricNativeImplEdges(ctx: ResolutionContext): Edge[] {
async function fabricNativeImplEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
// The Fabric extractor IDs are prefixed `fabric-component:` so we can
// filter to just those without iterating all `component` nodes.
const components = ctx.getNodesByKind('component').filter((n) => n.id.startsWith('fabric-component:'));
// filter to just those while streaming — never materializing the whole
// `component` kind (#1212).
const components: Node[] = [];
for (const n of (ctx.iterateNodesByKind?.('component') ?? ctx.getNodesByKind('component'))) {
if ((++scanned255 & 63) === 0) await onYield();
if (n.id.startsWith('fabric-component:')) components.push(n);
}
if (components.length === 0) return edges;
// Pre-index native classes by name for O(1) lookup.
const nativeClassesByName = new Map<string, Node[]>();
for (const n of ctx.getNodesByKind('class')) {
for (const n of (ctx.iterateNodesByKind?.('class') ?? ctx.getNodesByKind('class'))) {
if ((++scanned255 & 63) === 0) await onYield();
if (n.language !== 'objc' && n.language !== 'kotlin' && n.language !== 'java' && n.language !== 'cpp') continue;
const arr = nativeClassesByName.get(n.name);
if (arr) arr.push(n);
@@ -1675,12 +1732,14 @@ function fabricNativeImplEdges(ctx: ResolutionContext): Edge[] {
* same simple name) are dropped. We need-not bridge by package because Java
* mapper interfaces are typically uniquely named within a project.
*/
function mybatisJavaXmlEdges(queries: QueryBuilder): Edge[] {
async function mybatisJavaXmlEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
// Index Java methods by `<ClassName>::<methodName>` for O(1) lookup.
const javaIndex = new Map<string, Node[]>();
for (const m of queries.iterateNodesByKind('method')) {
if ((++scanned255 & 63) === 0) await onYield();
if (m.language !== 'java' && m.language !== 'kotlin') continue;
const parts = m.qualifiedName.split('::');
const last = parts[parts.length - 1];
@@ -1692,6 +1751,7 @@ function mybatisJavaXmlEdges(queries: QueryBuilder): Edge[] {
}
for (const xml of queries.iterateNodesByKind('method')) {
if ((++scanned255 & 63) === 0) await onYield();
if (xml.language !== 'xml') continue;
// Qualified name: `<namespace>::<id>`. Extract the simple class name.
const colonIdx = xml.qualifiedName.lastIndexOf('::');
@@ -1785,10 +1845,13 @@ function goHandlerIdent(expr: string): string | null {
return m ? m[1]! : null;
}
function ginMiddlewareChainEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
async function ginMiddlewareChainEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
let scannedFiles = 0;
// 1. Find the chain dispatcher(s): a Go method that invokes a `handlers` slice by index.
const dispatchers: Node[] = [];
for (const n of queries.iterateNodesByKind('method')) {
if ((++scanned255 & 63) === 0) await onYield();
if (n.language !== 'go') continue;
const content = ctx.readFile(n.filePath);
const src = content && sliceLines(content, n.startLine, n.endLine);
@@ -1801,6 +1864,7 @@ function ginMiddlewareChainEdges(queries: QueryBuilder, ctx: ResolutionContext):
// closures are dropped by goHandlerIdent; the rest are HandlerFuncs.
const registered = new Map<string, string>(); // name → registeredAt (file:line)
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!file.endsWith('.go')) continue;
const content = ctx.readFile(file);
if (!content || (!content.includes('.Use(') && !/\.(?:GET|POST|PUT|PATCH|DELETE|OPTIONS|HEAD|Any|Handle)\(/.test(content))) continue;
@@ -1852,10 +1916,12 @@ function ginMiddlewareChainEdges(queries: QueryBuilder, ctx: ResolutionContext):
* clause. Link the unit → its form so a `.dfm`/`.fmx` used only as a form
* definition isn't orphaned, and editing the form surfaces its code-behind unit.
*/
function pascalFormEdges(ctx: ResolutionContext): Edge[] {
async function pascalFormEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
const edges: Edge[] = [];
const allFiles = new Set(ctx.getAllFiles());
for (const file of allFiles) {
if ((++scannedFiles & 255) === 0) await onYield();
if (!/\.(dfm|fmx)$/i.test(file)) continue;
const pasFile = file.replace(/\.(dfm|fmx)$/i, '.pas');
if (!allFiles.has(pasFile)) continue;
@@ -1889,12 +1955,14 @@ function pascalFormEdges(ctx: ResolutionContext): Edge[] {
* a loader's data shows the page it feeds) and the page's dependencies include
* its loader.
*/
function svelteKitLoadEdges(ctx: ResolutionContext): Edge[] {
async function svelteKitLoadEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
const edges: Edge[] = [];
const allFiles = new Set(ctx.getAllFiles());
const HOOKS = new Set(['load', 'actions']);
const HOOK_KINDS = new Set(['function', 'method', 'constant', 'variable']);
for (const file of allFiles) {
if ((++scannedFiles & 255) === 0) await onYield();
const m = file.match(/(.*\/)(\+(?:page|layout))\.svelte$/);
if (!m) continue;
const dir = m[1]!;
@@ -1941,10 +2009,12 @@ const THUNK_DECL_RE = /create(?:Async)?Thunk/;
const THUNK_DISPATCH_RE = /\bdispatch\s*\(\s*([A-Za-z_]\w*)\s*[(),]/g;
const THUNK_FANOUT_CAP = 24;
function reduxThunkEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
async function reduxThunkEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
for (const node of queries.iterateNodesByKind('constant')) {
if ((++scanned255 & 63) === 0) await onYield();
// Cheap gate: the initializer (captured in `signature`) must be a create(Async)Thunk call —
// avoids reading every constant's body on a large repo.
if (!node.signature || !THUNK_DECL_RE.test(node.signature)) continue;
@@ -2069,10 +2139,12 @@ function resolveRegistryHandler(ctx: ResolutionContext, name: string, chained: s
}
async function objectRegistryEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
let scanned = 0;
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if ((++scanned & 255) === 0) await onYield(); // #1091: yield mid-scan on huge graphs
if (!REGISTRY_JS_EXT.test(file)) continue;
const content = ctx.readFile(file);
@@ -2174,10 +2246,12 @@ function rtkEndpointNameFromHook(hook: string): string | null {
return mid.charAt(0).toLowerCase() + mid.slice(1);
}
function rtkQueryEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
async function rtkQueryEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
for (const hook of queries.iterateNodesByKind('function')) {
if ((++scanned255 & 63) === 0) await onYield();
// Only our extracted generated-hook bindings (sentinel) — not a real hook fn.
if (hook.signature !== RTK_GENERATED_HOOK_SIGNATURE) continue;
const endpointName = rtkEndpointNameFromHook(hook.name);
@@ -2219,10 +2293,12 @@ const PINIA_BIND_RE = /\bconst\s+(\w+)\s*=\s*(?:await\s+)?(\w+)\s*\(/g;
const PINIA_CALL_RE = /(\w+)\s*\.\s*(\w+)\s*\(/g;
const PINIA_FANOUT_CAP = 80;
function piniaStoreEdges(ctx: ResolutionContext): Edge[] {
async function piniaStoreEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
// 1. Map each `const useXStore = defineStore(...)` factory → its store file.
const factoryFile = new Map<string, string>();
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!PINIA_CONSUMER_EXT.test(file)) continue;
const content = ctx.readFile(file);
if (!content || !content.includes('defineStore')) continue;
@@ -2235,6 +2311,7 @@ function piniaStoreEdges(ctx: ResolutionContext): Edge[] {
const edges: Edge[] = [];
const seen = new Set<string>();
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!PINIA_CONSUMER_EXT.test(file)) continue;
const content = ctx.readFile(file);
if (!content || !content.includes('Store')) continue;
@@ -2304,7 +2381,8 @@ function pathHasSegment(filePath: string, seg: string): boolean {
return new RegExp('[\\\\/]' + seg.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') + '[\\\\/.]').test(filePath);
}
function vuexDispatchEdges(ctx: ResolutionContext): Edge[] {
async function vuexDispatchEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
const storeFileCache = new Map<string, boolean>();
const isStoreFile = (file: string): boolean => {
let v = storeFileCache.get(file);
@@ -2339,6 +2417,7 @@ function vuexDispatchEdges(ctx: ResolutionContext): Edge[] {
const edges: Edge[] = [];
const seen = new Set<string>();
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!PINIA_CONSUMER_EXT.test(file)) continue;
const content = ctx.readFile(file);
if (!content || (!content.includes('dispatch(') && !content.includes('commit('))) continue;
@@ -2395,7 +2474,8 @@ const CELERY_PY_EXT = /\.py$/;
const CELERY_FANOUT_CAP = 80;
const CELERY_DECORATOR_LOOKBACK = 12; // max lines above a `def` to scan for its decorators
function celeryDispatchEdges(ctx: ResolutionContext): Edge[] {
async function celeryDispatchEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
// Memoize the decorator check per task-candidate node: it reads the file and scans a few
// lines above the def. Only called on names that are actually `.delay`/`.apply_async`
// receivers, so the candidate set stays small.
@@ -2434,6 +2514,7 @@ function celeryDispatchEdges(ctx: ResolutionContext): Edge[] {
const edges: Edge[] = [];
const seen = new Set<string>();
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!CELERY_PY_EXT.test(file)) continue;
const content = ctx.readFile(file);
if (!content || (!content.includes('.delay(') && !content.includes('.apply_async('))) continue;
@@ -2506,13 +2587,20 @@ function springFirstParamType(sig: string | undefined): string | null {
return /^[A-Z][A-Za-z0-9_]*$/.test(type) ? type : null;
}
function springEventEdges(ctx: ResolutionContext): Edge[] {
async function springEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
// Pass 1 — event-type → listener methods, scanning only event-relevant files.
// This is the ONLY full read sweep: publisher files are recorded here so
// pass 2 re-reads just those instead of every .java file again (#1212 —
// the double full-repo read was one of the tail's longest unyielded spans).
const listeners = new Map<string, Node[]>();
const publisherFiles: string[] = [];
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!SPRING_JAVA_EXT.test(file)) continue;
const content = ctx.readFile(file);
if (!content) continue;
if (content.includes('.publishEvent(')) publisherFiles.push(file);
const hasAnno = content.includes('@EventListener') || content.includes('@TransactionalEventListener');
const hasAppListener = SPRING_APP_LISTENER_RE.test(content);
if (!hasAnno && !hasAppListener) continue;
@@ -2543,11 +2631,12 @@ function springEventEdges(ctx: ResolutionContext): Edge[] {
}
if (!listeners.size) return [];
// Pass 2 — link each publishEvent(new XEvent(...)) site → every listener of XEvent.
// Pass 2 — link each publishEvent(new XEvent(...)) site → every listener of
// XEvent. Only the publisher files recorded in pass 1 are (re-)read.
const edges: Edge[] = [];
const seen = new Set<string>();
for (const file of ctx.getAllFiles()) {
if (!SPRING_JAVA_EXT.test(file)) continue;
for (const file of publisherFiles) {
if ((++scannedFiles & 15) === 0) await onYield();
const content = ctx.readFile(file);
if (!content || !content.includes('.publishEvent(')) continue;
const safe = stripCommentsForRegex(content, 'java');
@@ -2626,10 +2715,12 @@ function resolveMediatrArgType(arg: string, lines: string[], methodStart: number
return declType;
}
function mediatrDispatchEdges(ctx: ResolutionContext): Edge[] {
async function mediatrDispatchEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
// Pass 1 — request/notification type → the Handle method of each handler class.
const handlers = new Map<string, Node[]>();
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!MEDIATR_CS_EXT.test(file)) continue;
const content = ctx.readFile(file);
if (!content || (!content.includes('IRequestHandler<') && !content.includes('INotificationHandler<'))) continue;
@@ -2657,6 +2748,7 @@ function mediatrDispatchEdges(ctx: ResolutionContext): Edge[] {
const edges: Edge[] = [];
const seen = new Set<string>();
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!MEDIATR_CS_EXT.test(file)) continue;
const content = ctx.readFile(file);
if (!content || (!content.includes('.Send(') && !content.includes('.Publish('))) continue;
@@ -2714,7 +2806,8 @@ const SIDEKIQ_WORKER_RE = /\binclude\s+Sidekiq::(?:Job|Worker)\b/;
const SIDEKIQ_RB_EXT = /\.rb$/;
const SIDEKIQ_FANOUT_CAP = 80;
function sidekiqDispatchEdges(ctx: ResolutionContext): Edge[] {
async function sidekiqDispatchEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
// class node id → its instance `perform` method (null if the class isn't a Sidekiq worker),
// memoized. Reads the class body for the mixin; only consulted for actual dispatch receivers.
const performCache = new Map<string, Node | null>();
@@ -2753,6 +2846,7 @@ function sidekiqDispatchEdges(ctx: ResolutionContext): Edge[] {
const edges: Edge[] = [];
const seen = new Set<string>();
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!SIDEKIQ_RB_EXT.test(file)) continue;
const content = ctx.readFile(file);
if (!content || !/\.perform_(?:async|in|at)\b/.test(content)) continue;
@@ -2917,6 +3011,7 @@ function nixLeadingPlainSegments(name: string): string[] {
}
async function nixOptionPathEdges(queries: QueryBuilder, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
type Rec = { id: string; filePath: string; startLine: number; endLine: number; segs: string[] };
// One streaming pass over nix bindings (variables + the odd function-valued
@@ -2925,6 +3020,7 @@ async function nixOptionPathEdges(queries: QueryBuilder, onYield: MaybeYield): P
let scanned = 0;
for (const kind of ['variable', 'function'] as NodeKind[]) {
for (const node of queries.iterateNodesByKind(kind)) {
if ((++scanned255 & 63) === 0) await onYield();
if ((++scanned & 0x3fff) === 0 && onYield) await onYield();
if (node.language !== 'nix') continue;
const segs = nixLeadingPlainSegments(node.name);
@@ -3030,9 +3126,16 @@ async function nixOptionPathEdges(queries: QueryBuilder, onYield: MaybeYield): P
return edges;
}
function erlangBehaviourDispatchEdges(queries: QueryBuilder, ctx: ResolutionContext): Edge[] {
// Cheap language gate: no Erlang modules → no cost beyond one kind query.
const erlangModules = queries.getNodesByKind('namespace').filter((n) => n.language === 'erlang');
async function erlangBehaviourDispatchEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
let scanned255 = 0;
// Cheap language gate: no Erlang modules → no cost beyond one streamed
// kind scan (never a materialized array of every namespace — #1212).
const erlangModules: Node[] = [];
for (const n of queries.iterateNodesByKind('namespace')) {
if ((++scanned255 & 63) === 0) await onYield();
if (n.language === 'erlang') erlangModules.push(n);
}
if (erlangModules.length === 0) return [];
// Pass 1 — scan every Erlang file with `-callback` decls: behaviour module →
@@ -3045,6 +3148,7 @@ function erlangBehaviourDispatchEdges(queries: QueryBuilder, ctx: ResolutionCont
const declaringBehaviours = new Map<string, Node[]>(); // `fn/arity` → behaviour namespaces
const callbackNames = new Set<string>();
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!ERLANG_EXT.test(file)) continue;
const behaviour = moduleByFile.get(file);
if (!behaviour) continue; // a .hrl or module-less file can't be a behaviour
@@ -3095,6 +3199,7 @@ function erlangBehaviourDispatchEdges(queries: QueryBuilder, ctx: ResolutionCont
const edges: Edge[] = [];
const seen = new Set<string>();
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!ERLANG_EXT.test(file)) continue;
const content = ctx.readFile(file);
if (!content || !/[A-Z][A-Za-z0-9_@]*:[a-z]/.test(content)) continue;
@@ -3186,7 +3291,8 @@ function phpArrayBody(src: string, openIdx: number): string | null {
return null;
}
function laravelEventEdges(ctx: ResolutionContext): Edge[] {
async function laravelEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scannedFiles = 0;
// event short name → its listener `handle` methods (deduped by node id).
const listeners = new Map<string, Map<string, Node>>();
const add = (event: string, handle: Node) => {
@@ -3204,6 +3310,7 @@ function laravelEventEdges(ctx: ResolutionContext): Edge[] {
// Pass 1 — build the event→handle map from both registration mechanisms.
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!LARAVEL_PHP_EXT.test(file)) continue;
const content = ctx.readFile(file);
if (!content) continue;
@@ -3246,6 +3353,7 @@ function laravelEventEdges(ctx: ResolutionContext): Edge[] {
const edges: Edge[] = [];
const seen = new Set<string>();
for (const file of ctx.getAllFiles()) {
if ((++scannedFiles & 15) === 0) await onYield();
if (!LARAVEL_PHP_EXT.test(file)) continue;
const content = ctx.readFile(file);
if (!content || !content.includes('event(')) continue;
@@ -3301,59 +3409,90 @@ export async function synthesizeCallbackEdges(queries: QueryBuilder, ctx: Resolu
// watchdog still catches that. See ./cooperative-yield.
const yieldToLoop = createYielder();
// Per-pass wall-clock timing to stderr, opt-in via CODEGRAPH_SYNTH_TIMINGS
// (=1: passes over 250ms; =all: every pass). This is the diagnostic that
// located both the #1091/#1122 watchdog stalls and the #1212 OOM — keep it.
const markT = { t: Date.now() };
const __mark = (label: string): void => {
const now = Date.now();
const dt = now - markT.t;
markT.t = now;
if (process.env.CODEGRAPH_SYNTH_TIMINGS && (dt > 250 || process.env.CODEGRAPH_SYNTH_TIMINGS === 'all')) {
console.error(`[synth-timing] ${label}: ${dt}ms`);
}
};
// Language gating: one indexed DISTINCT over the files table lets a pass
// whose own filters reference a specific language/extension be skipped
// outright when the project has no such files — its result is provably
// empty, so skipping is behavior-identical and the cost drops to zero
// (the Kotlin pass was the OOM culprit on the pure-C Linux kernel, #1212).
// Passes without an explicit language filter always run.
const langs = queries.getDistinctFileLanguages();
const has = (...ls: string[]): boolean => ls.some((l) => langs.has(l));
const JS_FAMILY = ['typescript', 'javascript', 'tsx', 'jsx'];
const NONE: Edge[] = [];
// Cross-file Go method→type `contains` edges must be synthesized AND persisted
// FIRST: a method declared in a different file from its receiver type is
// otherwise orphaned from the struct, and goImplementsEdges (next) derives a
// struct's method set from its `contains` edges — so without this it would
// under-count the interfaces a cross-file struct satisfies. (#583)
const goMethodContains = goCrossFileMethodContainsEdges(queries);
if (goMethodContains.length > 0) queries.insertEdges(goMethodContains);
await yieldToLoop();
const goMethodContains = has('go') ? await goCrossFileMethodContainsEdges(queries, yieldToLoop) : NONE;
for (let i = 0; i < goMethodContains.length; i += 2000) {
queries.insertEdges(goMethodContains.slice(i, i + 2000));
await yieldToLoop();
}
await yieldToLoop(); __mark('goMethodContains');
// Go implicit `implements` edges must be synthesized AND persisted next: the
// interface-dispatch bridge below reads `implements` edges from the DB, and
// Go has none statically. (Other languages already have static implements
// edges from extraction, so they don't need this pre-pass.)
const goImpl = goImplementsEdges(queries);
if (goImpl.length > 0) queries.insertEdges(goImpl);
await yieldToLoop();
const goImpl = has('go') ? await goImplementsEdges(queries, yieldToLoop) : NONE;
for (let i = 0; i < goImpl.length; i += 2000) {
queries.insertEdges(goImpl.slice(i, i + 2000));
await yieldToLoop();
}
await yieldToLoop(); __mark('goImplements');
const fieldEdges = await fieldChannelEdges(queries, ctx, yieldToLoop); await yieldToLoop();
const closureCollEdges = await closureCollectionEdges(queries, ctx, yieldToLoop); await yieldToLoop();
const emitterEdges = await eventEmitterEdges(ctx, yieldToLoop); await yieldToLoop();
const renderEdges = reactRenderEdges(queries, ctx); await yieldToLoop();
const jsxEdges = await reactJsxChildEdges(ctx, yieldToLoop); await yieldToLoop();
const vueEdges = vueTemplateEdges(ctx); await yieldToLoop();
const svelteKitEdges = svelteKitLoadEdges(ctx); await yieldToLoop();
const pascalEdges = pascalFormEdges(ctx); await yieldToLoop();
const flutterEdges = flutterBuildEdges(queries, ctx); await yieldToLoop();
const arkuiStateEdges = arkuiStateBuildEdges(queries, ctx); await yieldToLoop();
const arkuiEmitter = arkuiEmitterEdges(ctx); await yieldToLoop();
const arkuiRoutes = arkuiRouterEdges(ctx); await yieldToLoop();
const cppEdges = cppOverrideEdges(queries); await yieldToLoop();
const ifaceEdges = interfaceOverrideEdges(queries); await yieldToLoop();
const kotlinExpectActual = kotlinExpectActualEdges(queries); await yieldToLoop();
const goGrpcEdges = goGrpcStubImplEdges(queries); await yieldToLoop();
const rnEventEdgesList = rnEventEdges(ctx); await yieldToLoop();
const fabricNativeEdges = fabricNativeImplEdges(ctx); await yieldToLoop();
const expoXPlatEdges = expoCrossPlatformEdges(queries); await yieldToLoop();
const rnXPlatEdges = rnCrossPlatformEdges(queries); await yieldToLoop();
const mybatisEdges = mybatisJavaXmlEdges(queries); await yieldToLoop();
const ginEdges = ginMiddlewareChainEdges(queries, ctx); await yieldToLoop();
const thunkEdges = reduxThunkEdges(queries, ctx); await yieldToLoop();
const registryEdges = await objectRegistryEdges(ctx, yieldToLoop); await yieldToLoop();
const rtkEdges = rtkQueryEdges(queries, ctx); await yieldToLoop();
const piniaEdges = piniaStoreEdges(ctx); await yieldToLoop();
const vuexEdges = vuexDispatchEdges(ctx); await yieldToLoop();
const celeryEdges = celeryDispatchEdges(ctx); await yieldToLoop();
const springEdges = springEventEdges(ctx); await yieldToLoop();
const mediatrEdges = mediatrDispatchEdges(ctx); await yieldToLoop();
const sidekiqEdges = sidekiqDispatchEdges(ctx); await yieldToLoop();
const erlangBehaviourEdges = erlangBehaviourDispatchEdges(queries, ctx); await yieldToLoop();
const laravelEdges = laravelEventEdges(ctx); await yieldToLoop();
const cFnPtrEdges = cFnPointerDispatchEdges(queries, ctx); await yieldToLoop();
const goframeEdges = goframeRouteEdges(ctx); await yieldToLoop();
const nixOptionEdges = await nixOptionPathEdges(queries, yieldToLoop); await yieldToLoop();
const fieldEdges = await fieldChannelEdges(queries, ctx, yieldToLoop); await yieldToLoop(); __mark('fieldEdges');
const closureCollEdges = await closureCollectionEdges(queries, ctx, yieldToLoop); await yieldToLoop(); __mark('closureCollEdges');
const emitterEdges = await eventEmitterEdges(ctx, yieldToLoop); await yieldToLoop(); __mark('emitterEdges');
const renderEdges = await reactRenderEdges(queries, ctx, yieldToLoop); await yieldToLoop(); __mark('renderEdges');
const jsxEdges = await reactJsxChildEdges(ctx, yieldToLoop); await yieldToLoop(); __mark('jsxEdges');
const vueEdges = has('vue') ? await vueTemplateEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('vueEdges');
const svelteKitEdges = has('svelte') ? await svelteKitLoadEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('svelteKitEdges');
const pascalEdges = await pascalFormEdges(ctx, yieldToLoop); await yieldToLoop(); __mark('pascalEdges');
const flutterEdges = has('dart') ? await flutterBuildEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('flutterEdges');
const arkuiStateEdges = has('arkts') ? await arkuiStateBuildEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('arkuiStateEdges');
const arkuiEmitter = has('arkts') ? await arkuiEmitterEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('arkuiEmitter');
const arkuiRoutes = has('arkts') ? await arkuiRouterEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('arkuiRoutes');
const cppEdges = has('cpp') ? await cppOverrideEdges(queries, yieldToLoop) : NONE; await yieldToLoop(); __mark('cppEdges');
const ifaceEdges = has('java', 'kotlin', 'csharp', 'swift', 'scala', 'go', 'rust', 'arkts', ...JS_FAMILY)
? await interfaceOverrideEdges(queries, yieldToLoop) : NONE; await yieldToLoop(); __mark('ifaceEdges');
const kotlinExpectActual = has('kotlin') ? await kotlinExpectActualEdges(queries, yieldToLoop) : NONE; await yieldToLoop(); __mark('kotlinExpectActual');
const goGrpcEdges = has('go') ? await goGrpcStubImplEdges(queries, yieldToLoop) : NONE; await yieldToLoop(); __mark('goGrpcEdges');
const rnEventEdgesList = has(...JS_FAMILY) ? await rnEventEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('rnEventEdgesList');
const fabricNativeEdges = await fabricNativeImplEdges(ctx, yieldToLoop); await yieldToLoop(); __mark('fabricNativeEdges');
const expoXPlatEdges = await expoCrossPlatformEdges(queries, yieldToLoop); await yieldToLoop(); __mark('expoXPlatEdges');
const rnXPlatEdges = await rnCrossPlatformEdges(queries, yieldToLoop); await yieldToLoop(); __mark('rnXPlatEdges');
const mybatisEdges = has('java', 'kotlin') && has('xml') ? await mybatisJavaXmlEdges(queries, yieldToLoop) : NONE; await yieldToLoop(); __mark('mybatisEdges');
const ginEdges = has('go') ? await ginMiddlewareChainEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('ginEdges');
const thunkEdges = has(...JS_FAMILY) ? await reduxThunkEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('thunkEdges');
const registryEdges = await objectRegistryEdges(ctx, yieldToLoop); await yieldToLoop(); __mark('registryEdges');
const rtkEdges = has(...JS_FAMILY) ? await rtkQueryEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('rtkEdges');
const piniaEdges = has('vue', ...JS_FAMILY) ? await piniaStoreEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('piniaEdges');
const vuexEdges = has('vue', ...JS_FAMILY) ? await vuexDispatchEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('vuexEdges');
const celeryEdges = has('python') ? await celeryDispatchEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('celeryEdges');
const springEdges = has('java') ? await springEventEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('springEdges');
const mediatrEdges = has('csharp') ? await mediatrDispatchEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('mediatrEdges');
const sidekiqEdges = has('ruby') ? await sidekiqDispatchEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('sidekiqEdges');
const erlangBehaviourEdges = has('erlang') ? await erlangBehaviourDispatchEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('erlangBehaviourEdges');
const laravelEdges = has('php') ? await laravelEventEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('laravelEdges');
const cFnPtrEdges = has('c', 'cpp') ? await cFnPointerDispatchEdges(queries, ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('cFnPtrEdges');
const goframeEdges = has('go') ? await goframeRouteEdges(ctx, yieldToLoop) : NONE; await yieldToLoop(); __mark('goframeEdges');
const nixOptionEdges = has('nix') ? await nixOptionPathEdges(queries, yieldToLoop) : NONE; await yieldToLoop(); __mark('nixOptionEdges');
const merged: Edge[] = [];
const seen = new Set<string>();
@@ -3400,6 +3539,14 @@ export async function synthesizeCallbackEdges(queries: QueryBuilder, ctx: Resolu
seen.add(key);
merged.push(e);
}
if (merged.length > 0) queries.insertEdges(merged);
__mark('dedupe-merge');
// Chunked insert with yields: on the Linux kernel the merged synthesized
// edge set is ~275k rows, and one transaction for all of them was a 20s
// unyielded main-thread span (#1212 follow-up) — the last one in the tail.
for (let i = 0; i < merged.length; i += 2000) {
queries.insertEdges(merged.slice(i, i + 2000));
await yieldToLoop();
}
__mark('insertMergedEdges');
return merged.length + goImpl.length + goMethodContains.length;
}
src/resolution/goframe-synthesizer.ts
+7 -3
View File
@@ -25,6 +25,7 @@
import type { Edge, Node } from '../types';
import type { ResolutionContext } from './types';
import type { MaybeYield } from './cooperative-yield';
import { GOFRAME_ROUTE_MARKER } from './frameworks/goframe';
const FANOUT_CAP = 2000; // backstop only; real apps are 1 route → 1 method.
@@ -73,13 +74,15 @@ function selectHandler(candidates: Node[], routeFile: string): Node | null {
return sameModule.length === 1 ? sameModule[0]! : null;
}
export function goframeRouteEdges(ctx: ResolutionContext): Edge[] {
export async function goframeRouteEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
// Route nodes the goframe extractor created, keyed by their package-qualified
// request type (`cash.ListReq`). `wanted` holds every key a handler signature
// could match — the qualified form plus its bare type fallback.
const routesByReqType = new Map<string, Node[]>();
const wanted = new Set<string>();
for (const route of ctx.getNodesByKind('route')) {
for (const route of (ctx.iterateNodesByKind?.('route') ?? ctx.getNodesByKind('route'))) {
if ((++scanned255 & 63) === 0) await onYield();
if (route.language !== 'go') continue;
const marker = route.qualifiedName.indexOf(GOFRAME_ROUTE_MARKER);
if (marker < 0) continue;
@@ -98,7 +101,8 @@ export function goframeRouteEdges(ctx: ResolutionContext): Edge[] {
// pointer, indexed by every matching (qualified + bare) form so a route can
// match precisely on `pkg.Type` and fall back to the bare `Type`.
const handlersByKey = new Map<string, Node[]>();
for (const method of ctx.getNodesByKind('method')) {
for (const method of (ctx.iterateNodesByKind?.('method') ?? ctx.getNodesByKind('method'))) {
if ((++scanned255 & 63) === 0) await onYield();
if (method.language !== 'go' || !method.signature) continue;
for (const t of pointerParamTypes(method.signature)) {
if (!wanted.has(t)) continue;
src/resolution/index.ts
+60 -20
View File
@@ -328,6 +328,30 @@ export class ReferenceResolver {
this.cachesWarmed = true;
}
/**
* warmCaches for the async resolution entry points: streams the distinct
* name set with periodic yields instead of one synchronous `.all()`. On a
* multi-million-node index the DISTINCT scan is a solid multi-second block
* (measured up to 28s inside `codegraph sync` on the Linux kernel index),
* long enough to matter to the #850 watchdog on slower hardware. Same
* result, same memory — only the event loop keeps turning.
*/
async warmCachesYielding(onYield: MaybeYield): Promise<void> {
if (this.cachesWarmed) return;
this.knownFiles = new Set(this.queries.getAllFilePaths());
const names = new Set<string>();
let scanned = 0;
for (const name of this.queries.iterateNodeNames()) {
names.add(name);
if ((++scanned & 8191) === 0) await onYield();
}
this.knownNames = names;
this.cachesWarmed = true;
}
/**
* Clear internal caches
*/
@@ -421,6 +445,12 @@ export class ReferenceResolver {
return result;
},
// Streamed, uncached — synthesizers scan-and-filter whole kinds, and
// both the materialized array AND the per-kind cache retention are
// O(nodes) memory (#1212). Per-ref resolvers keep the cached array
// variant above.
iterateNodesByKind: (kind: Node['kind']) => this.queries.iterateNodesByKind(kind),
fileExists: (filePath: string) => {
// Check pre-built known files set first (O(1))
if (this.knownFiles) {
@@ -1113,8 +1143,6 @@ export class ReferenceResolver {
onProgress?: (current: number, total: number) => void,
batchSize: number = 5000
): Promise<ResolutionResult> {
this.warmCaches();
// Resolution runs on the indexer's MAIN thread, and the #850 liveness
// watchdog SIGKILLs a process whose event loop stalls past its window (60s
// by default). A single dense batch's resolveAll — or the synthesis pass
@@ -1123,6 +1151,8 @@ export class ReferenceResolver {
// window to fire; see ./cooperative-yield.
const maybeYield = createYielder();
await this.warmCachesYielding(maybeYield);
const total = this.queries.getUnresolvedReferencesCount();
let processed = 0;
const aggregateStats = {
@@ -1141,32 +1171,42 @@ export class ReferenceResolver {
const result = await this.resolveBatchYielding(batch, maybeYield);
// Persist in bounded sub-transactions with yields between: a whole
// batch's edge insert / keyed deletes are otherwise one solid
// synchronous span each on a multi-GB index, sitting BETWEEN the
// per-ref yields — the last unyielded stretch of the resolution loop.
// Crash semantics are unchanged (already several transactions): edges
// land before their refs are deleted, so a kill mid-way re-resolves
// the remainder idempotently on the next run/sweep (#1187).
const PERSIST_CHUNK = 1000;
// Persist edges immediately
const edges = this.createEdges(result.resolved);
if (edges.length > 0) {
this.queries.insertEdges(edges);
for (let i = 0; i < edges.length; i += PERSIST_CHUNK) {
this.queries.insertEdges(edges.slice(i, i + PERSIST_CHUNK));
await maybeYield();
}
// Clean up resolved refs so they don't appear in the next batch
if (result.resolved.length > 0) {
this.queries.deleteSpecificResolvedReferences(
result.resolved.map((r) => ({
fromNodeId: r.original.fromNodeId,
referenceName: r.original.referenceName,
referenceKind: r.original.referenceKind,
}))
);
const resolvedKeys = result.resolved.map((r) => ({
fromNodeId: r.original.fromNodeId,
referenceName: r.original.referenceName,
referenceKind: r.original.referenceKind,
}));
for (let i = 0; i < resolvedKeys.length; i += PERSIST_CHUNK) {
this.queries.deleteSpecificResolvedReferences(resolvedKeys.slice(i, i + PERSIST_CHUNK));
await maybeYield();
}
// Delete unresolvable refs from this batch to avoid re-processing them
if (result.unresolved.length > 0) {
this.queries.deleteSpecificResolvedReferences(
result.unresolved.map((r) => ({
fromNodeId: r.fromNodeId,
referenceName: r.referenceName,
referenceKind: r.referenceKind,
}))
);
const unresolvedKeys = result.unresolved.map((r) => ({
fromNodeId: r.fromNodeId,
referenceName: r.referenceName,
referenceKind: r.referenceKind,
}));
for (let i = 0; i < unresolvedKeys.length; i += PERSIST_CHUNK) {
this.queries.deleteSpecificResolvedReferences(unresolvedKeys.slice(i, i + PERSIST_CHUNK));
await maybeYield();
}
// Aggregate stats
src/resolution/types.ts
+9
View File
@@ -71,6 +71,15 @@ export interface ResolutionContext {
getNodesByQualifiedName(qualifiedName: string): Node[];
/** Get all nodes of a kind */
getNodesByKind(kind: Node['kind']): Node[];
/**
* Stream nodes of a kind one at a time instead of materializing (and, unlike
* `getNodesByKind`, without populating the resolver's per-kind array cache).
* For unbounded kinds (`function`, `method`, `struct`) on a symbol-dense
* project the full array is gigabytes — the dynamic-edge synthesizers must
* use this so their memory stays O(1) in node count (#610, #1212). Optional
* so minimal test contexts compile; callers fall back to getNodesByKind.
*/
iterateNodesByKind?(kind: Node['kind']): IterableIterator<Node>;
/** Check if a file exists */
fileExists(filePath: string): boolean;
/** Read file content */