mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-04 10:48:20 +00:00
Martin Vogel
8c1a9d61c2
Indexing the same C# corpus twice on the same machine produced 5,794,336 edges
and then 5,794,328. Every other edge type matched exactly, every semantic
checksum matched (447,064 functions, 81,534 tokens, identical vec/idf/toks),
and the difference was ten WRITES edges out of one generated file.
Three places in extraction decided graph content from a clock:
* the unified AST walk stopped on a thread-CPU deadline;
* a file whose parse spent more than half its budget was refused the
per-file and cross-file LSP walks;
* so was a file whose parse plus walk exceeded the budget.
The first is the one that bit. Any file large enough to reach the deadline is
cut at whatever node the clock lands on, and five generated .NET JIT tests
reached it on every run: hugeSimpleExpr1.cs stopped at node 2,660,352 in one
run and 2,574,336 in the next, out of 2,774,794. That file's tail is field
assignments, so the two runs disagreed by exactly ten WRITES edges. The graph
was a function of how fast the machine happened to be running, and nothing said
so: result->walk_truncated was set and no code anywhere read it.
The deadline existed because those files cost 18-48 us per visited node, and
that turns out to be one fixable thing. csharp_callable_value_site() was the
only language branch in call_reference_candidate_site() still climbing with
ts_node_parent, which tree-sitter answers by descending from the ROOT, scanning
children at every level; every other language already climbs on the walk
cursor. On expressions megabytes deep that was ~85% of the entire unified walk
(sampled: stamp_usage_site -> ts_node_parent -> ts_node_child_with_descendant).
It now takes the cursor and the parent the caller already resolved, and
is_direct_argument_value_cursor() is split so a caller that has stepped the
cursor onto the parent can continue from there rather than paying for it twice.
With that fixed, the budgets are not needed to make indexing finish, so they
are gone rather than made deterministic. A cutoff in nodes would at least be
reproducible, but it still answers "what is in this repo?" with "as much as we
felt like reading". CBM_WALK_MAX_NODES reinstates one for an operator who wants
it; the default is no budget.
Removing the two LSP-skip rules was checked against the file and the path they
were added for (a 23 MB single-expression JIT test that once cost 354 s in the
walk and then crashed the cross-file resolve): the whole 12k-file C# corpus
twice with cross-file resolve, that file alone, and synthetic C# nested 10,000
levels deep — no crash, and past roughly that depth tree-sitter stops producing
a usable tree, so a tree deep enough to endanger a recursive walk never reaches
one.
C# corpus, before -> after:
edges 5,794,318 / 5,794,328 / 5,794,336 -> 5,794,873 every run
truncated 5 files, unreported -> 0
time ~115 s -> 97-101 s
Deterministic, about 550 edges more complete (the tails nobody was walking),
and faster than the build that was cutting them off. All ten bench corpora now
return identical nodes and edges across two runs.
A truncated walk and a skipped LSP walk are now reported in the coverage report
with how far the walk got ("1673216/4133946 nodes walked"), like a partial
parse. A file the walk abandoned halfway used to be reported as fully indexed,
which is a graph quietly lying about its coverage.
Tests: extract_csharp_argument_values_use_the_walk_cursor asserts no occurrence
falls back to the root-descending parent lookup — 128 fallbacks without the fix
and 0 with it. Its fixture parenthesises the argument on purpose: a bare
Sink(Target) needs no climb at all, so it cannot tell the two walks apart and
passed either way while the bug was in place.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>