Two first-run UX bugs surfaced by indexing a real 1,342-file C repo:
1. After 'Resolving refs' hit 100%, the ~40 dynamic-dispatch synthesis
passes ran with no progress surface, so the bar sat frozen at 100%
long enough to read as a hang (the C fn-pointer pass alone can hold
for a while on C-heavy repos). Synthesis now reports per-pass
progress through a new 'linking' IndexProgress phase, rendered as
'Linking dynamic dispatch'. The step total is pinned by a test to
the synthesizer's actual __mark() count so adding a pass without
bumping it fails loudly.
2. node:sqlite's ExperimentalWarning is emitted once per THREAD, so the
main process plus every parse worker printed it mid-index,
interleaved with the progress UI. All launch paths now pass
--disable-warning=ExperimentalWarning: both bundle launchers, the
Windows npm-shim invocation, and the CLI self-relaunch
(NODE_RUNTIME_FLAGS, deliberately excluded from the re-exec gate so
an older installed launcher never triggers a pointless re-exec, and
version-gated off nodes older than the flag).
Verified end-to-end on the same repo: zero warnings, live linking bar,
byte-identical graph (50,520 nodes / 148,232 edges). Full suite green.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* chore: ignore .kommandr/ directory
* fix(sync): resolve cross-file refs when an edit adds or removes the satisfying symbol (#1240)
Incremental sync scoped reference resolution to the changed files' own
refs, and a completed pass deleted every ref it failed to resolve — so
a symbol change in one file could never repair references in UNCHANGED
files, in either direction, until a full re-index:
- New-export case: a.ts imports/calls `greet` before b.ts defines it.
The failed refs were deleted at index time; when b.ts later gained
`greet`, nothing revisited a.ts — the calls/imports edges stayed
missing while status reported a clean index.
- Removal case: when a re-index (or file deletion) dropped a symbol,
the incoming edges cascade-deleted and the callers — whose resolved
refs had been consumed — never got a chance to rebind to an
alternative definition or reconnect when the symbol returned.
Fix, sharing one lifecycle:
- Schema v8: unresolved_refs gains status ('pending'/'failed') and
name_tail (last dotted segment, so `h.greet` is findable by `greet`).
Both resolver persist paths now park unresolvable refs as failed
instead of deleting them. All pending-work readers (batched drain,
non-progress guard, #1187 orphan sweep, status pendingRefs) filter to
pending, preserving their invariants and keeping status honest.
- Sync retry: after scoped resolution, failed refs whose name tail
matches a symbol name now present in the changed files are re-resolved
through a per-ref-yielding path (watchdog-safe, #1091 class). Names
matching >500 failed refs are skipped as external/builtin noise (#999
rationale).
- Removal side: createEdges stamps each resolution edge with its
originating reference (metadata.refName, + refKind when kind promotion
rewrote it). When the #899 restore misses a target or sync deletes a
file, the dropped edge is resurrected as exactly that ref — re-resolved
in the same sync (rebinding to an alternative definition) or parked
failed until the symbol reappears. Edges without the stamp (pre-upgrade,
synthesized) still drop silently: reconstructing from the target's plain
name would strip receiver context and risk a rebind a full re-index
would never make.
- Pure-removal syncs clear resolver caches so a long-lived daemon can't
resolve resurrected refs against the pre-removal graph.
Validated: issue repro now yields a graph byte-identical to a full
re-index; move/remove-readd/file-deletion scenarios all rebind or heal;
baseline-vs-new A/B on express and gin shows identical node/edge counts
and no timing regression (DB grows ~25% from the parked ref rows — pure
cache, reset by any full re-index). 8 regression tests added.
Fixes#1240
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Parse timeouts are now judged by the worker's own clock: the base timer
only marks a job late (after a long synchronous store stall, Node runs the
timers phase before the poll phase, so the timer fired before an
already-delivered result was processed — killing workers over parses that
took milliseconds, even on 0-byte files); a result arriving before a 3×
hard-kill backstop is accepted, timed-out files are retried, and
CODEGRAPH_PARSE_TIMEOUT_MS overrides the budget. Grammar WASM bytes are
pre-read once on the main thread and handed to every worker, so
spawns/respawns load grammars from memory instead of re-reading a
saturated disk.
Bulk indexing defers WAL auto-checkpointing for the whole run: the default
1000-page interval re-writes hot B-tree/FTS pages into the main DB file
over and over — ~95% of all disk I/O under throttled measurement. A
WalCheckpointValve bounds WAL growth with off-thread PASSIVE backfill
passes (never blocking the writer or the #850 watchdog heartbeat), pauses
the writer for a full backfill if the disk truly can't keep up, and folds
the WAL at the parse→resolution boundary so post-parse reads never page a
bulk-write-sized WAL. Opt out with CODEGRAPH_NO_WAL_DEFER=1; tune with
CODEGRAPH_WAL_VALVE_MB.
Measured at 150 IOPS (HDD class): commons-lang 1526s → 59s with 0 dropped
files (was 8); guava-scale completes in 7.6 min with a full graph where
v1.3.1 needed 25 min for a repo 5× smaller. Unthrottled: no change.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Large-codebase indexing died at the end of "Resolving refs" two ways:
watchdog kills of healthy work (24k-file Java on Windows, #1212 — third
iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale,
where v1.3.0 could not complete at any watchdog setting). Root causes:
~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no
yield points, several materialized whole-graph snapshots (kotlin
expect/actual opened with getAllNodes() — 2M nodes in one array; the
C fn-pointer pass retained every C file's contents twice plus every
function node), and the post-index WAL checkpoint ran minutes of
synchronous IO on the main thread, killing even a successful index at
the finish line.
The pipeline tail now follows the same discipline as the rest: never
hold O(graph) in the heap, yield everywhere.
- All synthesis passes stream node-kind scans (cursors, not arrays) and
yield on time-budgeted checkpoints; language gates skip passes whose
filters a project's file languages provably can't satisfy.
- kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded,
units stream one file at a time, and the all-functions array +
write-only id map are gone; spring reads each .java once, not twice.
- runMaintenance moved to a worker thread (own SQLite connection);
per-file store commits chunk with yields behind a serialized flush
chain (preserving #1015 file-order determinism); resolver warm-up
streams the DISTINCT name set; resolution batch-tail and merged-edge
inserts run in bounded sub-transactions.
- Daemon: fixed a socket-handoff race that could leave a fresh MCP
session permanently silent (client-hello tail unshifted into a
flowing stream with zero listeners — the long-standing #662 test
flake was this real bug); first tool call no longer queues behind
the query pool's cold start (pool.ready gate).
Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully
indexes in 27m8s on a 2-core/6GB container at default heap + default
watchdog; llvm-project (180k files) completes under 1GB RSS including
kill-and-sync recovery; synthesized-edge and full-graph parity are
byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky
daemon test passed 25/25 under load. Env-gated diagnostics kept:
CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop
tracing. Design record: docs/design/main-thread-stall-followup.md.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A Git super-repo whose `.gitignore` excludes its child repositories indexed
~nothing at the parent: CodeGraph respects `.gitignore` by default (#970,
#1065), so the excluded children were skipped and `codegraph init` printed
"Done" with 0 nodes — even though `init` inside each child worked fine. The
empty index was silent and unexplained.
`init`/`index` now detect the gitignored child repos they skipped when an
index comes up empty of symbols, name them, and — in an interactive terminal
— offer to index them (writing an `includeIgnored` entry to codegraph.json and
re-indexing on the spot); non-interactive runs print the exact codegraph.json
snippet to add. Gated on nodesCreated === 0, so a project that deliberately
keeps gitignored reference clones out of a working index is never nagged.
- extraction: findUnindexedIgnoredRepos — the inverse of discoverEmbeddedRepoRoots
(bounded, skips default-ignored dirs, respects existing includeIgnored)
- project-config: addIncludeIgnoredPatterns — create/merge codegraph.json,
idempotent, refuses to clobber malformed JSON
- cli: wire the detect-name-offer flow into both `init` and `index`
- tests: +13 covering detection, config writing, and the no-nag gate
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an `include` list to the root `codegraph.json` that forces gitignored first-party source (second-VCS / SVN / Perforce dual-tracked repos) into the index — discovered directly off disk on the full index, incremental sync, and file-watching, on both git and non-git projects. Gitignore-style patterns, root-relative; explicit `exclude` still wins and built-in skips (node_modules, dist, .git) are never re-included. Complements `exclude` and `includeIgnored`.
Closes#1163.
Thanks @luoyxy for the contribution.
Adds ArkTS (.ets, HarmonyOS/OpenHarmony) as a first-class language:
full TypeScript-grade extraction via the harmony-contrib tree-sitter
grammar (MIT, vendored byte-identical from the tree-sitter-arkts 0.2.0
npm tarball), plus the ArkUI constructs that make HarmonyOS apps
traceable:
- @Component/@ComponentV2 structs with decorators from both grammar
positions; members extract as class members with qualified names.
- build() component trees: child instantiation edges via
arkui_component_expression, no synthesizer needed.
- Attribute chains emitted dot-prefixed and resolved ONLY against
@Extend/@Styles/@AnimatableExtend/@Builder helpers (unique-or-drop) —
bare-name fallthrough produced 36,840 wrong edges (17% of calls) on
the OpenHarmony samples monorepo. All four grammar chain shapes
handled, including the detached-chain forms.
- .onClick(this.handler) method-reference bindings.
- ohpm workspace modules: bare imports follow oh-package.json5 file:
deps (ambiguous names dropped), honoring each module's main entry —
which also lets .ts consumers resolve .ets modules.
- ArkUI dynamic-dispatch bridges, all provenance:'heuristic' with
wiring-site metadata: assignment-gated state->build() re-render
(V1 @State family + V2 @Local/@Provider/@Consumer),
@ohos.events.emitter emit->subscriber pairing on static event keys
(numeric ids same-file, named constants same-module, fan-out capped),
and router.pushUrl literal urls -> the target page's @Entry struct.
- $r/$rawfile resource intrinsics treated as built-ins; arkts joins the
web language family, value-reference edges, re-export chase, and the
other TS-applicable gates.
Also ships a language-agnostic index-completeness guard: indexAll
stamps index_state (indexing -> complete/partial/failed), reconciles
discovered vs accounted files (a loaded run silently dropped 37 files),
and codegraph status surfaces truncated/partial indexes in human and
--json output.
Validated on HarmoneyOpenEye (82 files), CoolMallArkTS (528, modular
ohpm + ArkUI V2), and openharmony/applications_app_samples (11,693
files, 202,890 nodes stable across re-index, attribute false-positive
audit 36,840 -> 588 residual all-plausible). Supersedes PRs #656 and
#988 with credit — both informed this implementation.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The gitlink discovery added in #1031/#1033 indexed a tracked 160000 gitlink even when the parent .gitignore excludes its directory, pulling a gitignored reference/benchmark corpus of git add'ed clones into the index (one report: ~138k files, 4.8 GiB, wedged "Resolving refs" watchdog).
Gate both gitlink-discovery sites on the same rule the untracked-embedded path already uses: skip a gitignored gitlink unless codegraph.json includeIgnored opts it in; index non-ignored gitlinks as before. Validated real-world on macOS, Linux, and Windows.
Closes#1065
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An Android `res/` tree (layouts, value bags, drawables, menus, navigation
graphs) holds only non-code resources that yield zero symbols, yet on an
Android app it dominates the file count (one report: 26k XML = 97% of
files, 0 symbols) — bloating the DB, slowing indexing, and padding
explore/search results and file counts with entries that have nothing to
find.
Default-ignore the Android resource type directories (`res/layout/`,
`res/values/`, `res/drawable/`, … and their `-<qualifier>` variants) at
discovery, via DEFAULT_IGNORE_PATTERNS so it applies uniformly to the git
index, the non-git walk, and change detection. The `res/<type>/`
structure is self-identifying, so non-Android projects are untouched, and
the only XML that carries symbols — MyBatis mappers under
`src/main/resources/` — never lives under `res/`, so nothing useful is
dropped. `res/raw/` is deliberately kept (arbitrary bundled assets), and
a `.gitignore` negation re-includes anything.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A nested git repo tracked as a gitlink (mode 160000) — a clone `git add`ed
into the super-repo without a `.gitmodules` entry, or a submodule that
isn't active/initialized in this checkout — fell through both file-collection
passes: it's tracked, so the untracked `-o` listing skips it, but it's not
an active submodule, so `--recurse-submodules` won't expand it. Indexing the
top level therefore pulled in only the outer repo's own files and stopped at
the nested repo's boundary (one report: ~10 files at the root).
Switch the tracked scan to `ls-files -s` to expose file modes, collect the
unexpanded 160000 entries, and recurse into each that has a real working tree
on disk as its own embedded repo. Mirror the same discovery in
discoverEmbeddedRepoRoots so the watcher's scope stays equal to the indexer's.
Active submodules (#147) and untracked nested clones (#193) are unchanged;
gitlinks under default-ignored dirs (vendor/, node_modules/) stay excluded
(#407); an uninitialized submodule with no checkout on disk is left alone.
Adds four-shape coverage in extraction.test.ts.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
indexAll parsed every file through a single worker thread, so a full `codegraph index` used one core no matter the machine. Add ParseWorkerPool (src/extraction/parse-pool.ts), modeled on the shipped QueryPool: indexAll now parses across clamp(cores-1,1,8) workers. CODEGRAPH_PARSE_WORKERS overrides the count; 1 reproduces the previous single-worker path exactly (the rollback).
Parses run concurrently but results commit to SQLite in file order. This matters: the post-index resolution phase selects among ambiguous same-named candidates by node DB-insertion order, so a stable commit order keeps the graph deterministic — byte-identical to the serial path — instead of drifting with parse-completion timing. A bounded reorder buffer (backpressure on dispatched-but-uncommitted count) keeps memory flat even if a file is slow at the commit cursor.
Crash/timeout of a worker rejects only that file's parse (feeding the existing retry pass) and respawns; per-worker recycle every 250 parses reclaims WASM heap. In-process fallback unchanged when the compiled worker is absent (tests).
Validated on real OSS (django +9%, redis +17%; modest and parse-fraction-dependent), graph byte-identical across worker counts, peak memory flat-to-lower since workers recycle independently — so the #320 OOM concern doesn't materialize. Adds 11 pool unit tests.
Closes#1015. Refs #320.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three fixes for a repo that commits a large JS/TS theme/SDK (Metronic under
static/, ~1,600 tracked files):
1. A SECOND "Resolving refs" quadratic that #915 didn't cover. #915 capped
import-name collisions; this caps method-name collisions (init/update/render
re-declared on every widget), which flow through matchMethodCall Strategy 3
and findBestMatch instead. New AMBIGUOUS_NAME_CEILING (default 500, env
CODEGRAPH_AMBIGUOUS_NAME_CEILING): above it the fuzzy strategies decline
rather than score K candidates — no proximity score can pick the one true
target among thousands anyway. Resolving drops from O(K^2) to linear in refs
(e.g. 900-file synthetic: 28.7s -> 3.4s), edge counts unchanged, and the cap
never fires on normal repos (max real method-collision ~40).
2. A new `exclude` array in codegraph.json keeps git-TRACKED paths out of the
index, which .gitignore can't do (enumeration is `git ls-files`). Mirrors the
existing includeIgnored plumbing across the git, sync, and non-git-walk
paths.
3. `index`/`init` now install the #850 liveness + #277 ppid watchdogs (which
were serve-only), so a wedged or orphaned indexer self-terminates instead of
pinning a core. The --liftoff-only relaunch's spawnSync can't forward
signals, so killing the parent shim used to orphan the worker.
Tests: ubiquitous-name ceiling, exclude (incl. tracked-file exclusion on git +
non-git), orphan self-termination (POSIX), and ppid-parser units. Shared the
ppid parsers out of mcp/index.ts into mcp/ppid-watchdog.ts.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#514 (v1.0.0) began walking into gitignored directories to discover and
index the git repos nested inside them. That broke users who rely on
.gitignore to exclude a directory: a gitignored folder of cloned
reference repos blew graphs up (one report went 10k to 500k edges, #976)
and stalled indexing on multi-gigabyte trees of clones (#970).
Respect .gitignore by default again. Discovering embedded repos inside a
gitignored directory is now opt-in via codegraph.json:
{ "includeIgnored": ["packages/", "services/"] }
The single choke point findIgnoredEmbeddedRepos now returns nothing
unless a gitignored dir matches the project's includeIgnored patterns,
and the matcher is threaded from the scan root through the full-index,
incremental-sync, and watcher-scope paths. Downstream ScopeIgnore and the
watcher are unchanged: they key off the discovered embedded roots, so
gating discovery fixes the indexer, sync, and watcher together. Untracked
embedded repos (#193) stay indexed by default.
This restores the super-repo-of-clones behavior (#622, #699) for the
people who want it, while making the default match what every other tool
(and CodeGraph's own git ls-files foundation) does: .gitignore excludes.
project-config.ts now parses codegraph.json once (loadParsedConfig) and
exposes loadIncludeIgnoredPatterns alongside the existing extension map.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The directory walk deliberately follows an in-root symlink whose target
lives outside the repo root (the standard Dota custom-game layout, where
`game/` and `content/` link into the SDK tree) and enumerates the files
under it. But the read path then rejected every one of them via the
strict symlink-escape guard, logging `Path traversal blocked in batch
reader` and indexing nothing — discovery and the reader disagreed.
Add an opt-in `allowSymlinkEscape` to validatePathWithinRoot that waives
only the realpath-escape rejection (the lexical `../` guard still
applies) and pass it at the three indexing read sites (batch reader,
indexFile, indexFileWithContent). The content-serving sinks
(ContextBuilder, MCP tools) keep the strict guard, so this stays inside
the #527 model: indexing now follows the symlink, getCode still refuses
to serve out-of-root contents.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The extension → language table was hardcoded, so a codebase using a
non-standard extension for a supported language (e.g. `.dota_lua` for Lua)
had those files silently skipped — no way to opt them in short of patching
the source.
Add an opt-in, project-scoped `codegraph.json` at the repo root:
{ "extensions": { ".dota_lua": "lua", ".tpl": "php" } }
Mappings merge on top of the built-in defaults and take precedence (so a
built-in can be re-pointed, e.g. `.h` → `cpp`). Absent or malformed config
is the zero-config default — byte-identical to prior behavior; an invalid
target language or unparseable file is warned-and-skipped, never fatal.
Implementation:
- New `src/project-config.ts` — `loadExtensionOverrides(rootDir)`, validated
against `isLanguageSupported`, mtime-cached per root.
- `detectLanguage` / `isSourceFile` gain an optional `overrides` arg
(omitting it is the existing behavior).
- Overrides threaded per-operation through every extraction call site
(scan/walk gates, git change-detection, grammar selection, extraction,
the file watcher), resolved from the project root — no process-global
state, so the multi-project daemon stays isolated. The parse worker
receives the resolved language in its message.
Tests: 13 new cases (unit, loader validation/normalization/caching, and a
full-index integration proving a custom-extension file is extracted while
the zero-config path indexes nothing). Worker path smoke-tested via the
built CLI.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a very large repo (the report is a ~93k-file / 5.7GB-DB Java monorepo) the
first MCP `tools/call` after a fresh `serve --mcp` could hang for 10+ minutes
with zero output, and with the liveness watchdog on, the daemon was SIGKILLed
mid-query instead. Root cause: the post-open catch-up reconcile that the first
tool call is gated on does ~2*N synchronous `fs.existsSync`/`fs.statSync` calls
plus a load-all-files query in two non-yielding loops. On a huge repo that wedges
the event loop for minutes, which (a) trips the 60s watchdog (it SIGKILLs a
process whose loop stops turning) and (b) blocks the first call the whole time.
Two complementary fixes:
- Make the reconcile yield. `ExtractionOrchestrator.sync()` now uses the
yielding `scanDirectoryAsync`, and both O(files) reconcile loops
`await setImmediate` every SYNC_RECONCILE_YIELD_INTERVAL (1000) files. The loop
can no longer wedge the main thread, so the watchdog stays fed and the socket /
any concurrent read stays responsive while a big reconcile runs. Results are
unchanged — only yield points are added.
- Time-box the catch-up gate. The first `tools/call` now waits on the reconcile
for at most CODEGRAPH_CATCHUP_GATE_TIMEOUT_MS (default 3000ms), then serves and
lets the reconcile finish in the background (which now yields, so the served
call runs concurrently). `=0` restores the old unbounded wait. On a normal repo
the reconcile finishes well under the budget, so behavior is unchanged.
Tests: adds two time-box cases to mcp-catchup-gate (serves promptly when the
reconcile runs long; `=0` restores the unbounded wait). Full suite green
(1655 passed). Validated end-to-end through the real daemon: first call returns
at the ~3s time-box instead of waiting an injected 8s reconcile; no-delay control
unchanged; `=0` opt-out waits the full reconcile.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Change detection's git fast path (collectGitStatus) consumed `git status`
output with only an isSourceFile filter, on the assumption that git already
omits ignored paths. It doesn't: gitignore is a no-op for *tracked* files, and
the built-in default excludes (vendor/, node_modules/) aren't gitignore at all.
So a tracked file inside a committed dependency dir, or under a .gitignored
dir, surfaced as a change the full index never tracks — `codegraph status`
reported phantom pending changes that `sync` (a filtered filesystem reconcile)
never cleared, and the public getChangedFiles() API returned the same wrong
list.
Apply buildDefaultIgnore(repoDir) per recursion level, matching repo-relative
paths — structurally equivalent to the full-index path's ScopeIgnore (each
embedded repo judged by its own rules) with no extra git subprocess calls.
Deletions stay unfiltered: getChangedFiles acts on one only when the path is
already tracked in the DB, where removal is always correct, and that lets a
newly-excluded dir's stale rows clean themselves up.
Unblocks #699 (an .ignore overlay inherits this leak unless change detection
consults the same matcher as enumeration).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A worktree of a submodule points its `.git` into
`.git/modules/<module>/worktrees/<name>`, but `classifyGitDir` only matched
the top-level `.git/worktrees/` shape — so submodule worktrees fell through
to "embedded" and every symbol they shared with the real submodule checkout
got indexed twice (one report: ~28% of the index was duplicates, inflating
both query results and the DB). Broaden the worktree detector to allow the
optional `modules/<module>` segment. The submodule's own checkout
(`.git/modules/<module>`, no `worktrees/`) is unaffected and stays indexed as
distinct code.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the indexed root is a directory an enclosing git repo ignores,
`git ls-files --directory` collapses the whole cwd to a single literal
`./` entry. That sentinel reached the `ignore` matcher, which rejects it
("path should be a `path.relative()`d string, but got "./""), aborting
buildScopeIgnore — the one ignore-building call in FileWatcher.start().
So the MCP daemon's startWatching() threw, was caught as "Failed to open
project", and auto-sync never started: the index silently went stale
until a manual `codegraph sync` (CODEGRAPH_NO_DAEMON=1 was the only
workaround).
Filter the `./`/`.` self-entry wherever we consume `--directory` output
(listIgnoredDirs + the untracked-dir loop in discoverEmbeddedRepoRoots).
Semantically correct, not just a crash guard: `./` means "the whole cwd",
never a nested repo to recurse into.
Not platform-specific (reported on Codex/Windows, reproduced on macOS):
the trigger is git state, not the OS.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`storeExtractionResult` deletes a re-indexed file's nodes via `deleteFile`,
which cascades through `edges.FK ... ON DELETE CASCADE` to delete every edge
whose source OR target is one of those nodes. Edges whose source is in the
re-indexed file are re-emitted by the extractor, but edges whose source is in
a *different* (unchanged) file are not — they are silently dropped. This is
issue #899: re-indexing a callee file severs `calls`/`references` edges from
callers that import it via module-attribute access (`pkg.mod.fn(...)`), so
`codegraph callers fn` reports 0 callers for functions that have real call
sites. A docstring-only edit on the callee is sufficient to trigger it.
The bug affects every incremental path that routes through `sync()` /
`indexFile()`: `codegraph sync`, the file-watcher auto-sync (which calls
`sync()`), and the git sync hooks. `codegraph index` was already fixed by
#894 (it now clears-then-rebuilds, so it's a full re-extraction, not
incremental). `sync` remains the fast incremental path and still has the bug.
Fix: before the delete, snapshot incoming cross-file edges paired with the
target node's (name, kind). After re-inserting the file's nodes + same-file
edges, re-insert the snapshot — re-resolving each edge's target to the
re-indexed node's NEW id by (filePath, kind, name). Node ids are
`sha256(filePath:kind:name:line)`, so any line shift in the callee file (e.g.
a docstring-only edit above the symbol) changes every target id and a naive
re-insert by old id would drop them all. Matching by (kind, name) is stable
across line shifts; if the symbol was renamed/removed, no match is found and
the edge stays dropped (correct). `insertEdges` still filters to endpoints
that exist, so edges whose caller (source) was deleted are also dropped.
Regression tests in `__tests__/sync.test.ts` model the RAGFlow production
case: a `pkg/mod.py` with two callees, both called from `test/test_callers.py`
via `mod.<fn>(...)`. The first test confirms a docstring-only edit that shifts
the second callee's line preserves both incoming edges. The second test
confirms renaming a callee correctly drops its old incoming edge (no phantom
preservation against a non-existent symbol).
A git worktree nested in a project (e.g. Claude Code's gitignored
`.claude/worktrees/<name>/`) was swept into the index as an embedded repo: its
`.git` is a FILE pointing into the host repo's `.git/worktrees/`, and embedded-
repo discovery treated any `.git` (file or directory) as a distinct repo to
index. Each worktree then duplicated the entire graph — one report went from
~1,850 files to 24,533, with search/explore flooded by stale copies.
classifyGitDir() now distinguishes:
- `.git` directory -> embedded clone, index (#193/#514/#622, unchanged)
- `.git` file → worktrees/ -> worktree, skip (#848)
- `.git` file → modules/ -> submodule, index (unchanged)
Applied at both embedded-repo entry points: findNestedGitRepos discovery (which
also covers the sync/change-detection path) and the untracked-subdir recursion
in collectGitFiles.
Verified: the reproduction drops from 6 files / betaHelper×3 to 3 files / ×1,
with a genuine embedded clone and submodules still indexed. Regression test added.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
A workspace holding several independent git repos failed two ways:
- Enumeration: a super-repo whose .gitignore hides its child repos
(/packages/) indexed 0 of their files — git never lists ignored dirs,
and the #193 embedded-repo recursion only fired for UNTRACKED dirs.
Gitignored embedded repos are now discovered (ignored-dirs listing +
bounded .git search) and enumerated by their own git ls-files.
- Change detection: git status in the parent says nothing about embedded
repos (untracked OR ignored), so codegraph sync missed every child
change. Status now recurses per embedded repo.
ScopeIgnore is the new single source of truth for indexer + watcher
scope: parent rules for ordinary paths, the child repo's own rules for
paths inside it, built-in defaults uniformly on full paths (a git repo
inside node_modules is an npm git-dependency, not project code), and
ancestors of embedded roots are never pruned (the Linux per-directory
watcher must descend to reach them).
Non-git workspace roots already worked via the per-directory gitignore
walk — locked in by test.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
A .gitignore transparently encrypted in place by corporate DLP / endpoint
software (UTF-16 header + ciphertext), or one containing a pattern the
`ignore` library can't compile to a regex (`\[` -> "Unterminated character
class"), crashed the entire sync/index. The throw is LAZY — it surfaces at
match time (`ig.ignores()`), not `.add()` — so the existing add-time
try/catch never caught it, and the error never named the offending file.
Read .gitignore defensively: skip a file that isn't valid UTF-8 text whole
(NUL byte or fatal UTF-8 decode), drop only the individual uncompilable
patterns from a text one (probe-compile, then per-line fallback), and warn
with the file path. Indexing continues either way. The watcher inherits the
fix via buildDefaultIgnore.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two environments that share one working tree — most concretely Windows
and WSL — can't safely share a single `.codegraph/`: the daemon lockfile
records a platform-specific pid + socket (named pipe vs Unix socket), and
SQLite locking across the WSL2/Windows filesystem boundary is unreliable,
so two daemons over one index risks corruption.
Add a `CODEGRAPH_DIR` env var (default `.codegraph`) that overrides the
per-project data directory name, so each environment keeps its own index
in the same tree (e.g. `CODEGRAPH_DIR=.codegraph-win` on Windows). The
name is resolved live and validated (rejects separators / `..` / absolute,
falling back to the default with a one-time stderr warning). Indexing and
file-watching now skip ANY `.codegraph-*` sibling so neither side trips
over the other's data.
Routes the previously-hardcoded `.codegraph` literals (db path, lockfile,
error log, watcher ignore, file-scan skip, installer) through the
resolver. No extraction-version bump — index content is unchanged.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Batch of small, localized fixes from an open-issue triage:
- .codegraph/.gitignore now ignores everything but itself, so the database,
daemon.pid, sockets, and logs stop showing up in git status (#492, #484)
- MCP server answers resources/list and prompts/list with empty lists instead
of -32601, clearing scary log lines in opencode/Codex (#621)
- index SAP HANA .xsjs/.xsjslib as JavaScript (#556) and TS .mts/.cts (#366)
- visit anonymous AMD/CommonJS/IIFE wrapper bodies so their inner functions and
calls are indexed instead of coming up empty (#528)
- batch the changed-file lookup so a huge first sync no longer hits
"too many SQL variables" (#540)
- list files with `git ls-files -z` so non-ASCII/CJK paths survive
core.quotepath and are no longer silently skipped (#541)
- attach Go methods on generic receivers (*T[P]) to their type (#583, RC1)
- impact no longer climbs the structural `contains` edge, so a leaf symbol
stops dragging in its sibling methods (#536)
- README: explicit `codegraph install` step, run in a new shell (#631)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes #357. The no-symbol file-level class is yaml/twig/properties, but the
count fix only covered yaml/twig — so a .properties-only project still printed
"No files found to index" even though the files were stored. Introduce a single
isFileLevelOnlyLanguage predicate (the canonical set behind the tree-sitter
no-symbol branch, xml excluded since its MyBatis extractor emits a file node)
and use it at both count sites and the extraction dispatch so the list can't
drift. Adds .properties regression coverage for indexAll() and indexFiles().
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On Windows, v0.9.5's detached shared daemon (#411) has no inherited console,
so any console-subsystem child it spawns gets a fresh visible console window
unless the spawn passes `windowsHide: true`. The fix adds the flag to all
ten `spawnSync` / `execFileSync` / `execSync` call sites across extraction,
sync, installer, and the WASM-flags relaunch. macOS/Linux ignore the option,
so this is a no-op elsewhere.
Fixes#485, #510, #530.
Co-authored work:
- #498 (csw-chen) — full sweep across extraction, sync, installer, and wasm-runtime. **This is the change being merged.**
- #505 (yushengruohui) — independently identified and fixed the 7 git execFileSync sites. Superseded by #498's broader sweep; same diagnosis.
- #521 (JirA44) — independently identified and fixed the WASM-runtime spawnSync re-exec. Superseded by #498's broader sweep; same diagnosis.
Validated on Windows 11 ARM64 (Parallels): a detached parent's 15 git spawns produce 15 visible black flash-windows without the fix and 0 with it.
The file watcher registered a recursive watch over the entire project (node_modules, build output, caches included) and filtered only in the callback — exhausting the Linux inotify budget on large repos (#276). It now uses chokidar and excludes the same directories the indexer ignores (built-in default-ignore set + the project .gitignore) BEFORE registering a watch, so the watch count on a 900-dir node_modules drops from ~1200 to ~14 even with no .gitignore. Stacks with the shared daemon (#411): one watcher across agents, now small.
Also hardens the #411 daemon lockfile against a concurrent-startup race the new watcher timing made reproducible — the lock is now created atomically with its content (temp-write + hard-link), so racing daemons can never both win. Validated on macOS, Linux (Docker), and Windows (chokidar + fs.linkSync on NTFS).
Co-Authored-By: Colby McHenry <me@colbymchenry.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
node_modules and other dependency/build/cache dirs were indexed whenever a project lacked a .gitignore excluding them (common in non-git projects), flooding context/search with third-party symbols. Added a built-in default-ignore set spanning every supported language/framework (node_modules, vendor, dist, build, target, .venv, __pycache__, Pods, .next, etc.), applied UNIFORMLY in both the git and non-git enumeration paths — including to tracked files, since committing/vendoring a dependency dir doesn't make it project code. The only opt-in is an explicit .gitignore negation (e.g. !vendor/). First-party-prone names (packages, lib, app, bin, src, deps, env) are deliberately excluded from the list.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sync): detect changes via filesystem, not git status
Incremental sync detected changes with `git status --porcelain`, which only sees uncommitted working-tree changes — so committed changes from git pull/checkout/merge/rebase (clean tree afterward) were never reconciled, and non-git projects leaned on a slow full rescan. Change detection is now filesystem-based and git-independent: a (size, mtime) stat pre-filter skips unchanged files, then a content hash confirms the rest; removals are checked against the filesystem (git ls-files still lists deleted-but-unstaged files). Also adds a non-blocking catch-up sync on MCP connect so changes made while the server was down (e.g. a terminal git pull) are reconciled on connect.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(changelog): add 0.9.5 entry for filesystem-based sync fix
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove .codegraph/config.json and the entire config surface. CodeGraph now
indexes every file whose extension maps to a supported language and respects
.gitignore everywhere — git repos via git itself, non-git projects via the
`ignore` library (root + nested .gitignore files, the same way git does).
- Remove CodeGraphConfig/DEFAULT_CONFIG, src/config.ts, and the public config
API (the `config` option on init, getConfig/updateConfig/getConfigPath).
- Derive the source-file allowlist from EXTENSION_MAP (isSourceFile); maxFileSize
is now a constant. Drop the .codegraphignore marker.
- Behavior change: committed, non-gitignored dirs (vendor/, a committed dist/)
are now indexed — .gitignore is the single source of truth.
Earlier inert fields (languages, frameworks, extractDocstrings, trackCallSites,
customPatterns) and their dead helpers are removed as part of this.
Resolves#283.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both git fast-paths in ExtractionOrchestrator (sync and getChangedFiles)
classified every untracked (`??`) file as "added" without checking the
index. Indexing a file doesn't make git track it, so the file stayed `??`
and was re-reported as pending and re-indexed on every run: `codegraph
status` listed it under Pending Changes forever and each `sync` re-added
it, even though its symbols were already queryable.
Merge the modified + added handling into a single hash-compared loop so
untracked files get the same treatment as tracked ones: "added" only if
missing from the index, "modified" if contents changed, skipped otherwise.
The non-git fallback path already did this and is unchanged.
Closes#206. Reported by @15290391025.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`codegraph init -i` from a git super-repo containing independent nested
git repositories (not submodules) reported "No files found to index":
git ls-files reports an embedded repo only as an opaque `subdir/` entry
and never lists its files. Detect embedded repos via that trailing-slash
signal and recurse `git ls-files` into each, indexing tracked + untracked
source and honoring each repo's own .gitignore.
Reported by @timxx.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`git ls-files -co --exclude-standard` only sees the submodule pointer in
the main repo's index, so projects using submodules indexed 0 files. Now
the tracked list runs with `-c --recurse-submodules` so submodule
contents are included; untracked files are gathered with a separate
`-o --exclude-standard` call (the two flags can't be combined — git only
supports --recurse-submodules with --cached/--stage).
Fixes#147.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: add framework extract wiring plan
* feat(resolution): replace extractNodes with extract() returning nodes and references
* feat(resolution): add getApplicableFrameworks helper for per-language dispatch
* feat(django): emit route nodes and route->view references in extract()
* feat(flask,fastapi): emit route nodes and route->handler references
* feat(express): emit route nodes and route->handler references
* feat(laravel): emit route nodes and route->handler references
* feat(rails): emit route nodes and route->handler references
* feat(spring): emit route nodes and route->handler references
* feat(go): emit route nodes and route->handler references
* feat(rust): emit route nodes and route->handler references
* feat(aspnet): emit route nodes and route->handler references
* feat(swift,vapor): emit route nodes and route->handler references
* chore(react,svelte): migrate resolvers to extract() interface
* feat(extraction): run framework extractors after tree-sitter parse
* docs: document framework route extraction
* feat(strip-comments): add per-language comment stripper for framework extractors
Replaces comment characters and string-literal contents with spaces (not
removal) so source offsets stay valid for downstream regex match index ->
line number conversion. Handles Python triple-quoted docstrings, Ruby
=begin/=end, Rust nested block comments, and the standard //, #, /* */
forms across the supported languages.
This is consumed by framework extract() methods in a follow-up commit so
that commented-out / docstring routing examples don't surface as phantom
route nodes in the graph.
* feat(frameworks): strip comments before regex extraction (prevents phantom routes)
Pipes the per-language stripCommentsForRegex helper into every framework
extract() that scans raw source: django/flask/fastapi (python.ts),
express, laravel, rails, spring, go, rust, aspnet, vapor, plus
swiftui/uikit struct extraction in swift.ts.
Without this, examples like:
# path('/admin/', AdminPanel.as_view())
""" path('/users/', UserListView.as_view()) """
urlpatterns = [path('/real/', RealView.as_view())]
produced 3 phantom route nodes. Now only the real one is extracted.
Each framework gets a regression test in __tests__/frameworks.test.ts
asserting that line-, block-, docstring- and (where relevant)
heredoc-style commented-out routes do not surface as nodes.
---------
Co-authored-by: Colby McHenry <me@colbymchenry.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two correctness bugs in the core extraction pipeline, surfaced by an
adversarial stress corpus (5k synthetic export-const declarations
plus a deliberate 8MB single-line file):
1) Every `export const X = ...` produced TWO nodes for the same
symbol — one kind:'variable' from extractExportedVariables, plus
one kind:'constant' from extractVariable (called when the walker
descended into the export_statement child). Stress test showed
100% duplication across 5,003 export-const declarations. The
dedicated extractVariable dispatch is the correct one — it picks
kind from isConst, captures the initializer signature, and walks
type annotations; the export-statement helper was redundant
because the language extractors' isExported predicate already
walks parent chains. Remove the export_statement branch from the
dispatch (children are descended into normally) and drop the
private helper.
2) The bulk indexAll path read each file's stats but never compared
stats.size against config.maxFileSize. Vendored generated files
(multi-MB headers, minified bundles, etc.) were indexed regardless
of the user's size cap. The single-file extractFile path enforced
it; only the bulk path was missing the check. Mirror the
single-file behaviour: emit a 'size_exceeded' warning, count the
file as skipped, advance progress, and continue.
On the stress workspace (5,005 synthetic files; 50,000 fns in one
3MB file; 8MB single-line file; 5,000 export-const declarations):
before: 65,014 nodes (100% var/const duplication, every >1MB file
indexed despite maxFileSize=1MB)
after: 10,008 nodes (0 duplicates, large files correctly skipped
with size_exceeded warnings)
Tests calibrated to the duplicate behavior were updated to look for
kind:'constant' on `export const`, which is the correct kind. Full
suite: 380 passed (was 374 passed, 6 failed before this fix).
Addresses C++ classes missing from .h files where extension-based detection defaults to 'c' language which has no class extraction support. Adds looksLikeCpp() heuristic that scans first 8KB for C++-specific patterns (namespace, class, template, access specifiers) to promote .h files to 'cpp' language when C++ constructs are detected. Ensures cpp grammar is loaded alongside c to handle potential .h promotion during parsing.
Eliminates the intermediate 'finalizing' phase that was added as a progress bar transition state but served no functional purpose. Simplifies the progress flow by going directly from 'storing' to 'resolving' phases, removing associated UI labels and progress callbacks.
Adds strategic yield points and direct stdout writes to prevent progress animation from freezing when the main thread is blocked by synchronous operations. Introduces 'finalizing' phase to smooth transition between parsing and resolving steps, ensuring progress reaches 100% completion.
Recycles workers before each retry attempt instead of once per batch to maximize WASM memory headroom. Adds final fallback that strips comment-only lines from files that still crash on clean workers, reducing memory pressure from compiler test files with extensive CHECK directives while preserving line numbers for accurate node positions.
Implements file-size-based timeouts (base 10s + 10s per 100KB), more frequent worker recycling (250 files), and automatic retry logic for WASM memory corruption failures. Workers now crash immediately on memory errors to prevent cascading failures, with failed files automatically retried on fresh workers with clean heaps.
Adds --verbose flag to init and index commands that shows timestamped progress output instead of animated progress bars. Implements worker timeout protection (10s per file) and periodic worker recycling (every 500 files) to prevent WASM memory crashes from hanging the entire indexing process. Includes detailed logging of worker lifecycle events and memory usage for debugging large repository indexing issues.
Offloads tree-sitter parsing to a dedicated worker thread, keeping the main thread unblocked so shimmer progress animations render smoothly during indexing. Refactors shimmer progress renderer into separate worker for consistent 50ms animation updates. Falls back to in-process parsing when worker compilation unavailable (e.g., tests).
Replaces fixed-width padding with terminal escape sequences for proper progress line clearing across different terminal widths. Adds periodic parser reset every 5000 parses per language to prevent WASM heap fragmentation that causes "memory access out of bounds" crashes in large repositories. Includes filename truncation to fit available terminal width.
Adds expression index on lower(name) for memory-efficient case-insensitive searches, replacing in-memory caches that caused OOM on large codebases. Includes batched reference resolution, enhanced error reporting with detailed breakdown by error type, and improved CLI progress display for scanning phases.
When a project directory is gitignored by a parent git repository, `git ls-files` returns no results even though files exist. Added detection for this scenario using `git rev-parse` and `git check-ignore` to fall back to filesystem walking when the project directory is ignored by an ancestor repo.
Eliminates anonymous error reporting functionality that was collecting stack traces and error context via Sentry. Removes all telemetry-related code, configuration options, and documentation references.
xyyxrandGitHub
JoshandGitHub
luo jiyinandGitHub
csw-chenandGitHub
andreinknvandGitHub