fix(extract): no clock decides what ends up in the graph

Indexing the same C# corpus twice on the same machine produced 5,794,336 edges
and then 5,794,328. Every other edge type matched exactly, every semantic
checksum matched (447,064 functions, 81,534 tokens, identical vec/idf/toks),
and the difference was ten WRITES edges out of one generated file.

Three places in extraction decided graph content from a clock:

  * the unified AST walk stopped on a thread-CPU deadline;
  * a file whose parse spent more than half its budget was refused the
    per-file and cross-file LSP walks;
  * so was a file whose parse plus walk exceeded the budget.

The first is the one that bit. Any file large enough to reach the deadline is
cut at whatever node the clock lands on, and five generated .NET JIT tests
reached it on every run: hugeSimpleExpr1.cs stopped at node 2,660,352 in one
run and 2,574,336 in the next, out of 2,774,794. That file's tail is field
assignments, so the two runs disagreed by exactly ten WRITES edges. The graph
was a function of how fast the machine happened to be running, and nothing said
so: result->walk_truncated was set and no code anywhere read it.

The deadline existed because those files cost 18-48 us per visited node, and
that turns out to be one fixable thing. csharp_callable_value_site() was the
only language branch in call_reference_candidate_site() still climbing with
ts_node_parent, which tree-sitter answers by descending from the ROOT, scanning
children at every level; every other language already climbs on the walk
cursor. On expressions megabytes deep that was ~85% of the entire unified walk
(sampled: stamp_usage_site -> ts_node_parent -> ts_node_child_with_descendant).
It now takes the cursor and the parent the caller already resolved, and
is_direct_argument_value_cursor() is split so a caller that has stepped the
cursor onto the parent can continue from there rather than paying for it twice.

With that fixed, the budgets are not needed to make indexing finish, so they
are gone rather than made deterministic. A cutoff in nodes would at least be
reproducible, but it still answers "what is in this repo?" with "as much as we
felt like reading". CBM_WALK_MAX_NODES reinstates one for an operator who wants
it; the default is no budget.

Removing the two LSP-skip rules was checked against the file and the path they
were added for (a 23 MB single-expression JIT test that once cost 354 s in the
walk and then crashed the cross-file resolve): the whole 12k-file C# corpus
twice with cross-file resolve, that file alone, and synthetic C# nested 10,000
levels deep — no crash, and past roughly that depth tree-sitter stops producing
a usable tree, so a tree deep enough to endanger a recursive walk never reaches
one.

C# corpus, before -> after:

  edges      5,794,318 / 5,794,328 / 5,794,336   ->  5,794,873 every run
  truncated  5 files, unreported                 ->  0
  time       ~115 s                              ->  97-101 s

Deterministic, about 550 edges more complete (the tails nobody was walking),
and faster than the build that was cutting them off. All ten bench corpora now
return identical nodes and edges across two runs.

A truncated walk and a skipped LSP walk are now reported in the coverage report
with how far the walk got ("1673216/4133946 nodes walked"), like a partial
parse. A file the walk abandoned halfway used to be reported as fully indexed,
which is a graph quietly lying about its coverage.

Tests: extract_csharp_argument_values_use_the_walk_cursor asserts no occurrence
falls back to the root-descending parent lookup — 128 fallbacks without the fix
and 0 with it. Its fixture parenthesises the argument on purpose: a bare
Sink(Target) needs no climb at all, so it cannot tell the two walks apart and
passed either way while the bug was in place.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
Martin Vogel committed 2026-09-19 15:50:41 +02:00
1 parent c7ccb479b4
commit 8c1a9d61c2
6 files changed
+308 -96

No files matched your search

+85 -43
View File
@@ -225,17 +225,56 @@ static const char *cbm_string_read(void *payload, uint32_t byte, TSPoint point,
* edge). A generous WALL ceiling stays as a backstop so a genuinely
* stuck/spinning parse still terminates in bounded time. */
#define CBM_PARSE_WALL_CEILING_FACTOR 12ULL /* ~60 s ceiling for the 5 s CPU budget */
/* A parse that used more than 1/N of the per-file budget disqualifies the file
* from the unbudgeted LSP walks (see cbm_extract_file_ex). */
#define CBM_LSP_BUDGET_SHARE_DIV 2ULL
/* The unified walk may spend this many parse budgets of thread CPU time: wide
* enough for a 7,873-definition reference file (~10 s), tight enough to stop the
* generated JIT tests (65-350 s). */
#define CBM_WALK_BUDGET_FACTOR 6ULL
/* The unified walk has NO budget: it walks every node of every file.
*
* It used to stop on a thread-CPU deadline, which made the GRAPH a function of
* how fast the machine happened to be running. Two indexes of the same C#
* corpus, same machine, minutes apart, stopped the walk of
* src/tests/JIT/jit64/opt/cse/hugeSimpleExpr1.cs at node 2,660,352 and at node
* 2,574,336 (measured 2026-09-19). That file's tail is all field assignments,
* so the two runs disagreed by 10 WRITES edges: a graph that differed run to
* run for a reason no user could see or reproduce. A budget in NODES would at
* least have been reproducible, but it still answers "what is in this repo?"
* with "depends how much we felt like reading" — so there is no budget at all.
* Everything the walk does is a pure function of the tree, and now so is when
* it stops: at the end.
*
* What the deadline was protecting against is real and is handled where it
* belongs — in the cost per node, not in a clock. Generated blobs (2.7-8.2 M
* node trees) used to cost 18-48 us per node because usage stamping called
* ts_node_parent, which descends from the root; the walk now carries its own
* parent chain, so a deep tree costs what a shallow one does.
*
* CBM_WALK_MAX_NODES reinstates a budget for an operator who needs one
* (0 = the default, no budget). */
#define CBM_WALK_MAX_NODES_DEFAULT 0u
/* Read fresh each call, like cbm_max_file_bytes: cheap next to a file walk, and
* no memoized copy to go stale between runs in the same process. */
static uint32_t cbm_walk_max_nodes(void) {
const char *raw = getenv("CBM_WALK_MAX_NODES");
if (raw && raw[0]) {
errno = 0;
char *end = NULL;
unsigned long v = strtoul(raw, &end, 10);
if (errno == 0 && end != raw && *end == '\0' && v <= UINT32_MAX) {
return (uint32_t)v; /* 0 is a legitimate choice: no budget */
}
/* Unparseable / out-of-range → the default, never an accidental 0. */
}
return CBM_WALK_MAX_NODES_DEFAULT;
}
typedef struct {
uint64_t cpu_deadline_ns; // trip once this thread's CPU time passes it
uint64_t wall_ceiling_ns; // hard wall backstop for a spinning/stuck parse
/* Set by the callback when IT ended the parse. ts_parser_parse_with_options
* returns NULL for several reasons and the budget is only one of them, but
* the failure used to be reported as "parse timeout" whenever a budget was
* configured at all. A 7-line .properties file was accused of timing out
* (2026-09-19); whatever really happened to it, saying "timeout" sent the
* next reader looking at the clock. */
bool tripped;
} CBMParseBudget;
#ifdef CBM_ENABLE_TEST_SEAMS
@@ -253,12 +292,17 @@ static CBM_TLS uint64_t tl_parse_wall_seam_offset_ns = 0;
* parameter here would not match the opts.progress_callback assignment below. */
// cppcheck-suppress constParameterCallback
static bool cbm_timeout_cb(TSParseState *state) {
const CBMParseBudget *budget = (const CBMParseBudget *)state->payload;
CBMParseBudget *budget = (CBMParseBudget *)state->payload;
uint64_t wall = now_ns();
#ifdef CBM_ENABLE_TEST_SEAMS
wall += tl_parse_wall_seam_offset_ns;
#endif
return cbm_thread_cpu_time_ns() > budget->cpu_deadline_ns || wall > budget->wall_ceiling_ns;
bool over =
cbm_thread_cpu_time_ns() > budget->cpu_deadline_ns || wall > budget->wall_ceiling_ns;
if (over) {
budget->tripped = true; /* so a NULL tree can name its real cause */
}
return over;
}
// --- Thread-local parser pool ---
@@ -1882,7 +1926,6 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
ts_parser_reset(parser);
uint64_t t0 = now_ns();
uint64_t cpu_start_ns = cbm_thread_cpu_time_ns();
// Build string input + timeout options for parse_with_options
CBMStringInput str_input = {source, (uint32_t)source_len};
@@ -1922,40 +1965,47 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
if (!tree) {
result->has_error = true;
result->error_msg =
cbm_arena_strdup(a, timeout_micros > 0 ? "parse timeout" : "parse failed");
/* Only the budget's own callback may call this a timeout. Any other
* NULL is a parse failure, and saying so is the difference between a
* reader chasing the clock and a reader chasing the real cause. */
result->error_msg = cbm_arena_strdup(a, budget.tripped ? "parse timeout" : "parse failed");
cbm_index_mark_done(rel_path);
return result;
}
TSNode root = ts_tree_root_node(tree);
/* Parse-budget share. A file whose parse alone consumed more than
* 1/CBM_LSP_BUDGET_SHARE_DIV of its budget is too large for the per-file
* LSP walk that follows: the walk is superlinear in expression size and
* has no budget of its own (C#, a 23 MB single-expression JIT test: 354 s
* in the walk, then a crash in the cross-file resolve on the same tree,
* 2026-09-14 -- the parse used to time out at 5 s and hide both). The
* unified extractor's defs stay; the LSP refinement here and the
* cross-file resolve (cbm_pxc_dispatch_file) skip the file, logged. The
* budget is the same for every parser, so the rule is too. */
bool lsp_skipped = timeout_micros > 0 && (t1 - t0) * CBM_LSP_BUDGET_SHARE_DIV > budget_ns;
/* No file is disqualified from the LSP walks by how long its parse took.
*
* The rule used to be "a parse that spent more than half its budget means
* this tree is too heavy for the walks" — a clock deciding which files get
* type-aware refinement, so the same repo could come back with different
* graphs. It was added on 2026-09-14 against a 23 MB single-expression C#
* JIT test that cost 354 s in the per-file walk and then crashed the
* cross-file resolve.
*
* Removing it was checked against exactly that file and that path
* (2026-09-19): the whole 12k-file C# corpus twice, cross-file resolve
* included, and hugeexpr1.cs on its own — no crash, same 1,224,981 nodes /
* 5,794,873 edges as with the rule in place, same ~100 s. Synthetic C# with
* valid nesting 10,000 levels deep indexes fine; past roughly that depth
* tree-sitter itself stops producing a usable tree, so a tree deep enough
* to endanger a recursive walk never reaches one. The 354 s is gone for a
* different reason: the walk that spent it was C#'s own, climbing with
* ts_node_parent, and it now uses the cursor (extract_usages.c).
*
* CBM_TEST_LSP_SKIP_ON still names a file for the tests that pin the
* skipped-file behaviour itself. */
#ifdef CBM_ENABLE_TEST_SEAMS
{
const char *skip_on = getenv("CBM_TEST_LSP_SKIP_ON");
if (skip_on && skip_on[0] && rel_path && strstr(rel_path, skip_on)) {
lsp_skipped = true; /* the test names the file; no real timing involved */
result->lsp_skipped = true; /* the test names the file; no timing involved */
cbm_log_warn("extract.lsp.skipped", "reason", "test_seam", "path",
rel_path ? rel_path : "");
}
}
#endif
if (lsp_skipped) {
char parse_ms[CBM_SZ_32];
snprintf(parse_ms, sizeof(parse_ms), "%llu",
(unsigned long long)((t1 - t0) / CBM_NSEC_PER_MSEC));
cbm_log_warn("extract.lsp.skipped", "reason", "parse_budget", "parse_ms", parse_ms, "path",
rel_path ? rel_path : "");
result->lsp_skipped = true;
}
// Compute module QN. Java/Go derive the module from the CONTAINING
// DIRECTORY (package semantics) rather than baking the filename stem in,
@@ -1979,8 +2029,7 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
.root = root,
.macro_table = macro_table,
.return_type_table = return_type_table,
.walk_deadline_cpu_ns =
timeout_micros > 0 ? cbm_thread_cpu_time_ns() + budget_ns * CBM_WALK_BUDGET_FACTOR : 0,
.walk_budget_nodes = cbm_walk_max_nodes(),
};
// Run extractors: defs + imports use separate walks (unique recursion patterns),
@@ -1988,19 +2037,12 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
cbm_extract_definitions(&ctx);
cbm_extract_imports(&ctx);
cbm_extract_unified(&ctx);
result->tree_nodes = ts_node_descendant_count(root);
result->walk_nodes_visited = ctx.walk_nodes_visited;
if (ctx.walk_budget_exhausted) {
result->walk_truncated = true;
result->lsp_skipped = true;
cbm_log_warn("extract.walk.truncated", "reason", "cpu_budget", "path",
rel_path ? rel_path : "");
}
/* A file that spent the budget on parse plus walk is too heavy for the
* unbudgeted LSP walks as well (the C# JIT test files: 65-73 s each in
* the per-file walk after a parse under the share rule). */
if (!result->lsp_skipped && timeout_micros > 0 &&
cbm_thread_cpu_time_ns() - cpu_start_ns > budget_ns) {
result->lsp_skipped = true;
cbm_log_warn("extract.lsp.skipped", "reason", "file_budget", "path",
cbm_log_warn("extract.walk.truncated", "reason", "node_budget", "path",
rel_path ? rel_path : "");
}
+18 -7
View File
@@ -569,6 +569,12 @@ typedef struct CBMFileResult {
* to that point are kept, the rest of the file is not walked. Implies
* lsp_skipped. */
bool walk_truncated;
/* Size of this file's parse tree, and how much of it the unified walk got
* through. Reported for a truncated or LSP-skipped file so the coverage
* report says how much of it is missing, instead of leaving the gap
* silent. */
uint32_t tree_nodes;
uint32_t walk_nodes_visited;
CBMLanguage cached_lang; // language of cached tree (for parser selection)
// Retained source bytes — copied into `arena` by the parallel
@@ -659,14 +665,19 @@ typedef struct {
* class-body variable def records which class declares it (parent_class)
* without changing its module-level qualified name. NULL elsewhere. */
const char *var_parent_class;
/* Per-file walk budget (thread CPU time, ns; 0 = unbounded). The unified
* cursor walk checks it every 1024 nodes and stops when it is spent, so
* no single file can hold a worker for minutes: a 23 MB single-expression
* C# test file cost 346 s in usage stamping alone (tree-sitter's
* ts_node_parent descends from the root, quadratic on a deep tree;
* 2026-09-14). What was extracted before the stop is kept. */
uint64_t walk_deadline_cpu_ns;
/* Per-file walk budget in VISITED NODES (0 = unbounded). The unified cursor
* walk stops once it is spent, so no single file can hold a worker for
* minutes: a 23 MB single-expression C# test file cost 346 s in usage
* stamping alone (tree-sitter's ts_node_parent descends from the root,
* quadratic on a deep tree; 2026-09-14). What was extracted before the stop
* is kept, and the file is named in the coverage report. Counted in nodes
* rather than CPU time so that the same file always stops at the same node
* — see CBM_WALK_MAX_NODES_DEFAULT for what a clock did here. */
uint32_t walk_budget_nodes;
bool walk_budget_exhausted;
/* How many nodes the unified walk actually visited (whether or not it ran
* out of budget) — the measurement the budget has to be expressed in. */
uint32_t walk_nodes_visited;
} CBMExtractCtx;
// --- Public API ---
+13 -12
View File
@@ -2600,31 +2600,31 @@ void cbm_extract_unified(CBMExtractCtx *ctx) {
uint32_t depth = 0;
uint32_t visited = 0;
#ifdef CBM_ENABLE_TEST_SEAMS
/* CBM_TEST_WALK_BUDGET_NODES=<n>: the budget is "spent" after n nodes,
* no real timing involved. */
uint32_t seam_budget_nodes = 0;
/* CBM_TEST_WALK_BUDGET_NODES=<n>: spend the budget after n nodes. Production
* now counts nodes too, so this seam only shortens the budget; it no longer
* has to stand in for a mechanism of its own. */
uint32_t budget_nodes = ctx->walk_budget_nodes;
{
const char *seam = getenv("CBM_TEST_WALK_BUDGET_NODES");
if (seam && seam[0]) {
seam_budget_nodes = (uint32_t)strtoul(seam, NULL, 10);
budget_nodes = (uint32_t)strtoul(seam, NULL, 10);
}
}
#else
const uint32_t budget_nodes = ctx->walk_budget_nodes;
#endif
for (;;) {
TSNode node = ts_tree_cursor_current_node(&cursor);
visited++;
if (ctx->walk_deadline_cpu_ns != 0 && (visited & 1023u) == 0 &&
cbm_thread_cpu_time_ns() > ctx->walk_deadline_cpu_ns) {
/* The budget is spent in nodes, so the walk of a given file always ends
* on the same node — on any machine, under any load. A CPU deadline
* here made the graph differ between two runs on one machine; see
* CBM_WALK_MAX_NODES_DEFAULT (cbm.c). */
if (budget_nodes != 0 && visited > budget_nodes) {
ctx->walk_budget_exhausted = true;
break;
}
#ifdef CBM_ENABLE_TEST_SEAMS
if (seam_budget_nodes != 0 && visited > seam_budget_nodes) {
ctx->walk_budget_exhausted = true;
break;
}
#endif
bool trivia = is_unified_trivia_node(node);
if (!trivia) {
/* Trivia consumes no semantic state. Scope expiry may be deferred
@@ -2678,6 +2678,7 @@ void cbm_extract_unified(CBMExtractCtx *ctx) {
}
cbm_finalize_lexical_usages(ctx, &state);
ctx->walk_nodes_visited = visited;
ts_tree_cursor_delete(&occurrence_cursor);
ts_tree_cursor_delete(&cursor);
}
+63 -22
View File
@@ -1355,20 +1355,20 @@ static bool is_direct_argument_value(TSNode node) {
return !ts_node_is_null(grandparent) && is_argument_container_kind(ts_node_type(grandparent));
}
/* Cursor-backed counterpart for the unified walker. `cursor` must currently
* point at `node`; it is consumed while walking toward the argument owner. */
static bool is_direct_argument_value_cursor(TSNode node, TSTreeCursor *cursor) {
TSNode current = node;
while (!ts_node_is_null(current)) {
TSNode parent;
const char *field;
if (!occurrence_parent(cursor, current, &parent, &field))
return false;
/* The body of the walk above, entered one level in: for a caller that has
* ALREADY stepped the cursor onto `parent` and knows which `field` of it the
* node below occupies. A caller that climbed to find its site has that pair in
* hand, and re-deriving it would mean either stepping the cursor twice or
* paying ts_node_parent for what the cursor just told us. */
static bool is_direct_argument_value_from(TSNode parent, const char *field, TSTreeCursor *cursor) {
for (;;) {
const char *parent_kind = ts_node_type(parent);
if (is_labeled_argument_kind(parent_kind)) {
if (!field || strcmp(field, "value") != 0)
return false;
current = parent;
/* The labeled argument now has to be the argument value itself. */
if (!occurrence_parent(cursor, parent, &parent, &field))
return false;
continue;
}
if (field && strcmp(field, "arguments") == 0)
@@ -1389,7 +1389,16 @@ static bool is_direct_argument_value_cursor(TSNode node, TSTreeCursor *cursor) {
return occurrence_parent(cursor, parent, &grandparent, &argument_field) &&
is_argument_container_kind(ts_node_type(grandparent));
}
return false;
}
/* Cursor-backed counterpart for the unified walker. `cursor` must currently
* point at `node`; it is consumed while walking toward the argument owner. */
static bool is_direct_argument_value_cursor(TSNode node, TSTreeCursor *cursor) {
TSNode parent;
const char *field;
if (!occurrence_parent(cursor, node, &parent, &field))
return false;
return is_direct_argument_value_from(parent, field, cursor);
}
static bool is_direct_argument_value_walk(TSNode node, WalkState *state) {
@@ -1443,15 +1452,36 @@ static bool language_may_stamp_exact_callable_value_candidate(CBMLanguage langua
* arguments are narrow syntactic candidates only. A language LSP must still
* prove one target at this exact occurrence before the graph upgrades USAGE to
* CALL_REFERENCE; unresolved, reassigned, and composite expressions stay USAGE. */
static TSNode csharp_callable_value_site(TSNode node) {
usage_slow_parent_fallback_test_note();
/* One step up from `site`, on the walk cursor when the caller has one. Every
* other language branch in call_reference_candidate_site climbs this way; C#
* used ts_node_parent, which tree-sitter answers by descending from the ROOT,
* so each step costs O(depth) with a child scan at every level. On the
* generated .NET JIT tests — single expressions megabytes deep — that was
* 18-48 us per visited node and ~85% of the whole unified walk (sampled
* 2026-09-19: stamp_usage_site -> ts_node_parent ->
* ts_node_child_with_descendant). Those files are also the ones the old CPU
* deadline used to cut off mid-walk, so the slow path and the nondeterminism
* it forced were the same defect seen from two ends. */
static bool csharp_site_parent(TSTreeCursor *cursor, TSNode site, TSNode *parent,
const char **field) {
if (!cursor) {
usage_slow_parent_fallback_test_note();
}
return occurrence_parent(cursor, site, parent, field);
}
static TSNode csharp_callable_value_site(TSNode node, TSNode parent, const char *parent_field,
TSTreeCursor *cursor) {
const char *kind = ts_node_type(node);
if (strcmp(kind, "identifier") != 0 && strcmp(kind, "simple_identifier") != 0) {
return (TSNode){0};
}
/* The caller has already resolved node's parent (and the field it occupies)
* with the cursor, so the climb starts from that pair instead of asking for
* it again. Invariant below: `parent`/`field` always describe `site`. */
TSNode site = node;
TSNode parent = ts_node_parent(site);
const char *field = parent_field;
if (!ts_node_is_null(parent) && strcmp(ts_node_type(parent), "generic_name") == 0) {
TSNode name = ts_node_child_by_field_name(parent, TS_FIELD("name"));
if (ts_node_is_null(name) && ts_node_named_child_count(parent) > 0) {
@@ -1461,7 +1491,9 @@ static TSNode csharp_callable_value_site(TSNode node) {
return (TSNode){0};
}
site = parent;
parent = ts_node_parent(site);
if (!csharp_site_parent(cursor, site, &parent, &field)) {
return (TSNode){0};
}
}
if (!ts_node_is_null(parent) && strcmp(ts_node_type(parent), "member_access_expression") == 0) {
@@ -1470,7 +1502,9 @@ static TSNode csharp_callable_value_site(TSNode node) {
return (TSNode){0};
}
site = parent;
parent = ts_node_parent(site);
if (!csharp_site_parent(cursor, site, &parent, &field)) {
return (TSNode){0};
}
}
while (!ts_node_is_null(parent) &&
@@ -1483,12 +1517,17 @@ static TSNode csharp_callable_value_site(TSNode node) {
return (TSNode){0};
}
site = parent;
parent = ts_node_parent(site);
if (!csharp_site_parent(cursor, site, &parent, &field)) {
return (TSNode){0};
}
}
/* The wrapper chain proves admission, but the semantic row is keyed to the
* terminal method-name identifier. Keep the raw carrier on that leaf. */
return is_direct_argument_value(site) ? node : (TSNode){0};
if (ts_node_is_null(parent)) {
return (TSNode){0};
}
return is_direct_argument_value_from(parent, field, cursor) ? node : (TSNode){0};
}
/* Climb out of the parentheses wrapping a direct argument and return the
@@ -1545,10 +1584,12 @@ static TSNode call_reference_candidate_site(CBMExtractCtx *ctx, TSNode node, con
const char *parent_field = NULL;
if (!cursor) {
usage_slow_parent_fallback_test_note();
parent = ts_node_parent(node);
} else {
(void)occurrence_parent(cursor, node, &parent, &parent_field);
}
/* Resolve the parent AND the field node occupies in it, on both paths. The
* C# site walk below carries that field into the argument test, and
* ts_node_parent on its own would leave it NULL there — silently losing the
* "this node IS the arguments" case whenever no cursor is available. */
(void)occurrence_parent(cursor, node, &parent, &parent_field);
bool ts_family = ctx->language == CBM_LANG_JAVASCRIPT || ctx->language == CBM_LANG_TYPESCRIPT ||
ctx->language == CBM_LANG_TSX || ctx->language == CBM_LANG_ARKTS;
if (ts_family && strcmp(kind, "property_identifier") == 0 && !ts_node_is_null(parent) &&
@@ -1588,7 +1629,7 @@ static TSNode call_reference_candidate_site(CBMExtractCtx *ctx, TSNode node, con
return is_direct_argument_value_walk(node, state) ? node : (TSNode){0};
}
if (ctx->language == CBM_LANG_CSHARP) {
return csharp_callable_value_site(node);
return csharp_callable_value_site(node, parent, parent_field, cursor);
}
if (strcmp(kind, "identifier") != 0 && strcmp(kind, "simple_identifier") != 0) {
return (TSNode){0};
+22
View File
@@ -1188,6 +1188,28 @@ static void extract_worker(int worker_id, void *ctx_ptr) {
pp_err_add(errs, fi->rel_path, result->error_ranges ? result->error_ranges : "unknown",
result->parse_unusable ? "parse_unusable" : "parse_partial");
}
/* A truncated walk is a coverage gap like a partial parse, and until now
* it was the only one we kept to ourselves: result->walk_truncated was
* set and never read by anything, so a file the walk abandoned halfway
* was reported as fully indexed. Say how far it got — "walked 812k of
* 3.4M nodes" is the difference between a graph with a known hole and a
* graph that quietly lies about its coverage. Independent of the
* branches above: a truncated walk is not a parse error. */
if (result->walk_truncated) {
char how_far[CBM_SZ_64];
snprintf(how_far, sizeof(how_far), "%u/%u nodes walked", result->walk_nodes_visited,
result->tree_nodes);
pp_err_add(errs, fi->rel_path, how_far, "walk_truncated");
} else if (result->lsp_skipped) {
/* Indexed, but without the per-file and cross-file LSP refinement:
* the same kind of hole from the other direction. Nothing in
* production sets this any more except a truncated walk (handled
* above) and the test seam — it is reported anyway, so that if
* something sets it again the gap arrives named, not silent. */
char size_text[CBM_SZ_64];
snprintf(size_text, sizeof(size_text), "%u nodes", result->tree_nodes);
pp_err_add(errs, fi->rel_path, size_text, "lsp_skipped");
}
/* Create definition nodes in local gbuf */
for (int d = 0; d < result->defs.count; d++) {
+107 -12
View File
@@ -6193,6 +6193,96 @@ TEST(extract_wide_flat_reference_fields_are_linear) {
}
PASS();
}
/* The same guard for C#, whose callable-value site walk was the last one still
* climbing with ts_node_parent instead of the walk cursor. tree-sitter answers
* ts_node_parent by descending from the ROOT, so every step costs O(depth) and
* a deep file pays it per identifier. The .NET JIT tests are exactly that —
* single expressions megabytes deep — and they cost 18-48 us per node, which is
* why they were the only files a CPU-time budget ever cut off, and why the C#
* graph differed between two runs on one machine (2026-09-19). The budget is
* gone; this keeps the reason it was needed from coming back. */
static uint64_t extract_csharp_argument_value_work(int statement_count, int *out_usages,
uint64_t *out_slow_parent_fallbacks) {
static const char prefix[] = "class C {\n"
" static void Sink(object o) {}\n"
" static void Target() {}\n"
" static void Wide() {\n";
/* Parenthesised on purpose: a bare `Sink(Target)` needs no climb at all, so
* it cannot tell the cursor walk from the root-descending one and the test
* would pass either way. The wrapper makes the site walk take a step. */
static const char statement[] = " Sink((Target));\n";
static const char suffix[] = " }\n}\n";
size_t capacity = sizeof(prefix) + (size_t)statement_count * sizeof(statement) + sizeof(suffix);
char *source = malloc(capacity);
if (!source) {
return UINT64_MAX;
}
size_t offset = 0;
memcpy(source + offset, prefix, sizeof(prefix) - 1U);
offset += sizeof(prefix) - 1U;
for (int i = 0; i < statement_count; i++) {
memcpy(source + offset, statement, sizeof(statement) - 1U);
offset += sizeof(statement) - 1U;
}
memcpy(source + offset, suffix, sizeof(suffix));
offset += sizeof(suffix) - 1U;
cbm_usage_field_lookup_test_reset();
CBMFileResult *result =
cbm_extract_file(source, (int)offset, CBM_LANG_CSHARP, "proj", "Wide.cs", 0, NULL, NULL);
free(source);
if (!result) {
return UINT64_MAX;
}
int usages = 0;
for (int i = 0; i < result->usages.count; i++) {
if (result->usages.items[i].ref_name &&
strcmp(result->usages.items[i].ref_name, "Target") == 0) {
usages++;
}
}
uint64_t work = cbm_usage_field_lookup_test_work();
*out_slow_parent_fallbacks = cbm_usage_slow_parent_fallback_test_count();
cbm_free_result(result);
*out_usages = usages;
return work;
}
TEST(extract_csharp_argument_values_use_the_walk_cursor) {
enum { SMALL = 128, BIG = 1024, INPUT_GROWTH = 8, WORK_RATIO_MAX = 12 };
int small_usages = 0;
int big_usages = 0;
uint64_t small_slow_parent_fallbacks = 0;
uint64_t big_slow_parent_fallbacks = 0;
uint64_t small_work =
extract_csharp_argument_value_work(SMALL, &small_usages, &small_slow_parent_fallbacks);
uint64_t big_work =
extract_csharp_argument_value_work(BIG, &big_usages, &big_slow_parent_fallbacks);
ASSERT_TRUE(small_work != UINT64_MAX);
ASSERT_TRUE(big_work != UINT64_MAX);
/* Anti-vacuous: the occurrences really were classified, so a zero fallback
* count means "took the cursor", not "never looked". */
ASSERT_EQ(small_usages, SMALL);
ASSERT_EQ(big_usages, BIG);
/* The assertion this test exists for: not one occurrence fell back to the
* root-descending parent lookup. */
ASSERT_EQ(small_slow_parent_fallbacks, 0);
ASSERT_EQ(big_slow_parent_fallbacks, 0);
fprintf(stderr, " [csharp-argument-values] work(%d)=%llu work(%d)=%llu input_growth=%dx\n",
SMALL, (unsigned long long)small_work, BIG, (unsigned long long)big_work, INPUT_GROWTH);
uint64_t maximum = small_work * WORK_RATIO_MAX + 256U;
if (big_work > maximum) {
char message[192];
snprintf(message, sizeof(message),
"csharp argument-value lookup grew from %llu to %llu for %dx input "
"(maximum %dx + 256) -- the site walk is not linear",
(unsigned long long)small_work, (unsigned long long)big_work, INPUT_GROWTH,
WORK_RATIO_MAX);
FAIL(message);
}
PASS();
}
#endif
/* ===================================================================
@@ -7438,13 +7528,15 @@ TEST(extract_spill_round_trip_keeps_every_field) {
PASS();
}
/* ── LSP budget share: an oversized parse disqualifies the file from the walks ── */
/* ── A skipped file gets no LSP walk, per-file or cross-file ── */
/* CBM_TEST_LSP_SKIP_ON names the file (no real timing): the result carries
* lsp_skipped, the per-file LSP walk did not run (no LSP-resolved calls),
* and the shared cross-file dispatcher returns without touching it. The
* unified extractor definitions are still there. */
TEST(extract_lsp_skipped_when_parse_used_its_budget_share) {
/* CBM_TEST_LSP_SKIP_ON names the file: the result carries lsp_skipped, the
* per-file LSP walk did not run (no LSP-resolved calls), and the shared
* cross-file dispatcher returns without touching it. The unified extractor
* definitions are still there. Nothing in production sets lsp_skipped from a
* clock any more — this pins what the flag DOES, which is what the cross-file
* dispatcher and the per-file walks both have to honour. */
TEST(extract_lsp_skipped_file_gets_no_walk) {
cbm_setenv("CBM_TEST_LSP_SKIP_ON", "budget_share.py", 1);
CBMFileResult *skipped = extract(COMPACT_PY_SRC, CBM_LANG_PYTHON, "t", "budget_share.py");
cbm_unsetenv("CBM_TEST_LSP_SKIP_ON");
@@ -7469,10 +7561,12 @@ TEST(extract_lsp_skipped_when_parse_used_its_budget_share) {
PASS();
}
/* CBM_TEST_WALK_BUDGET_NODES stops the unified walk after that many nodes
* (no real timing): the result is walk_truncated, therefore lsp_skipped, and
* the definitions the walk had not reached are the only loss. */
TEST(extract_walk_truncated_at_its_cpu_budget) {
/* CBM_TEST_WALK_BUDGET_NODES stops the unified walk after that many nodes: the
* result is walk_truncated, therefore lsp_skipped, and the definitions the walk
* had not reached are the only loss. There is no budget by default (see
* CBM_WALK_MAX_NODES_DEFAULT), so this drives the same node counter an operator
* would set with CBM_WALK_MAX_NODES rather than a mechanism of its own. */
TEST(extract_walk_truncated_when_a_node_budget_is_set) {
cbm_setenv("CBM_TEST_WALK_BUDGET_NODES", "8", 1);
CBMFileResult *cut = extract(COMPACT_PY_SRC, CBM_LANG_PYTHON, "t", "walk_budget.py");
cbm_unsetenv("CBM_TEST_WALK_BUDGET_NODES");
@@ -7493,8 +7587,8 @@ SUITE(extraction) {
RUN_TEST(extract_compact_keeps_every_field_and_shrinks_the_arena);
RUN_TEST(extract_compact_is_idempotent_and_survives_empty_results);
RUN_TEST(extract_spill_round_trip_keeps_every_field);
RUN_TEST(extract_lsp_skipped_when_parse_used_its_budget_share);
RUN_TEST(extract_walk_truncated_at_its_cpu_budget);
RUN_TEST(extract_lsp_skipped_file_gets_no_walk);
RUN_TEST(extract_walk_truncated_when_a_node_budget_is_set);
/* Initialize extraction library */
cbm_init();
@@ -7502,6 +7596,7 @@ SUITE(extraction) {
RUN_TEST(extract_wide_flat_file_is_linear);
#if defined(CBM_CALL_REFERENCE_LOOKUP_TEST_API) && CBM_CALL_REFERENCE_LOOKUP_TEST_API
RUN_TEST(extract_wide_flat_reference_fields_are_linear);
RUN_TEST(extract_csharp_argument_values_use_the_walk_cursor);
#endif
/* Perl call-graph noise (#459 follow-up) */