mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-04 10:48:20 +00:00
main
28
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
6e784c38a2 |
fix(lsp): name Go init() callers per file in the Go LSP
#1910 gives every Go init() its own QN, pkg.init#<file>:<N>, on the def side and in compute_func_qn. The Go LSP computes its enclosing-function QN on its own and still produced the flat pkg.init. The pipeline joins LSP rows to textual calls by exact caller QN, so every LSP resolution inside an init body missed the join: typed calls fell back to the registry resolver and callable references (Register(fn)) were dropped. On kubernetes (8c078bc3), without this commit all 499 lsp_* CALLS from init bodies fell back (import_map, same_module, unique_name) and all 562 CALL_REFERENCE edges from them were lost; with it, the init edges match main's strategy mix. - helpers.c: cbm_go_init_qn is the one formula for all three sides (def extractor, compute_func_qn, Go LSP process_function). The per-file ordinal now walks the top-level declarations with a cursor: ts_node_named_child(root, i) is O(i) per call, so the indexed loop was quadratic on wide generated files. - go_lsp: GoLSPContext carries the file's rel_path. cbm_run_go_lsp, cbm_run_go_lsp_cross and cbm_run_go_lsp_cross_with_registry take it, so the per-file walk, the shared Tier-2 registry walk and the per-file fallback all name init callers like the def side. Tier 3 copies the per-file rows' caller QN, so it follows. - pass_lsp_cross: the dispatch passes rel to the Tier-2 walk and to the per-file fallback (cbm_pxc_run_one_with_php_bindings gains rel_path; the incremental route can run without shared registries). - compute_func_qn: a NULL base QN keeps the flat QN, as the receiver branch above it does. Tests: golsp_init_caller_qn_matches_def (per-file rows and the caller join), golsp_crossfile_init_caller_qn_per_file (both cross entry points), pipeline_go_init_lsp_edges_per_file and its _parallel twin (three inits in two files: lsp_ CALLS and a CALL_REFERENCE on the right init, no cross-wiring, nothing onto a decoy), and pipeline_go_init_dispatch_fallback_names_init_caller. Each fails with its piece reverted; the pipeline tests also fail with the compute_func_qn mirror disabled, which no test covered before. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b87d067637 |
fix(cpp): one spelling per template specialization in the preprocessed pass
The preprocessed second pass rebuilt the expanded source with simplecpp's TokenList::stringify(), which puts one space between every pair of tokens: `template<> struct bitonic<int64_t, AVX2>` became `bitonic < int64_t , AVX2 >`. Every QN builder takes the class segment from node text, so a definition the #961 rescue re-adopted from that buffer (vxsort's INLINE/NOINLINE macros break the raw parse on every method line) got a second, token-spaced QN, and the QN-equality dedup kept both. CALLS split across the two spellings (dotnet/runtime: 902 compact + 996 spaced into the bitonic sorters). The expanded buffer now keeps the source spacing: tokens that touched in the source (same file and line, no macro expansion between them) stay touching, every other gap is still one space, and the line structure and #line directives are unchanged, so every line remap holds. One fix point instead of a normaliser at each QN builder; no vendored code changes. Proof (macOS arm64, dotnet/runtime 60108ba6, two identical runs each): - duplicate definition groups 706 -> 217 (v0.11.0: 545); nodes with a token-spaced QN 495 -> 0; CALLS into bitonic 902 + 996 -> 998, one spelling. The remaining groups are coreclr gc.cpp's SVR/WKS builds (184) and 33 scope disagreements between the two parses. - Side effect: rapidjson document.h's GenericValue members now exist once, with the raw parse's scope (the token-spaced copy that carried the nested scope is gone); `operator[]` / `operator()` in the expanded pass now parse as single tokens. - New test (RED without the change): extraction extract_cpp_template_specialization_preprocessed_qn_single_spelling. Three c_lsp collision fixtures recalibrated to the new byte offsets (comment padding and offsets only, assertions unchanged). - extraction, c_lsp, pipeline, edge_types_probe, parse_coverage, conditional_variants, lsp_resolution_probe: 1852 passed. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
244067b0a5 |
test(lsp): count only invocation rows in the issue-1527 C resolver tests
Main's C/C++ resolver now also publishes one member-access row (CBM_RESOLVED_FIELD_REFERENCE) per enclosing function, object type and member. count_real_resolved counted every row whose callee QN matched, so the fluent-chain, scale and memo-recovery tests saw those rows on top of the call resolutions they count (49 instead of 45, 3000 instead of 2000). The helper now counts invocation rows only; the expected numbers are unchanged. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e9e72d093c |
fix(lsp): evaluate operator and receiver chains iteratively
The depth guard of the C/C++ and Python expression evaluators (C_EVAL_DEPTH_LIMIT, PY_LSP_MAX_EVAL_DEPTH = 256) now reports a sticky per-file failure that stops the walk and drops every LSP resolution of the file. Both evaluators recursed once per chain link, so a 300-term sum or a C++ fluent chain of ~130 calls counted as 256 levels of nesting: one long expression cost its whole file every LSP call edge. A chain's length is not nesting, and the guard must not decide graph content for it. c_eval_expr_type and py_eval_expr_type now first walk down the chain operands below the node (the operand the evaluator evaluates first and derives the link's type from), then evaluate them innermost first at the caller's depth, keeping pending links on a small arena-owned stack in the context that nested evaluations share. When a link is evaluated, its chain operand is already memoized, so the evaluator's own recursive call into it is a memo hit one frame deep. Every node is still evaluated by the unchanged inner evaluator, evaluation does not mutate scope within one outermost call, and only operands the recursion itself evaluates are pre-evaluated, so the types are exactly the recursive ones. No longer counted as depth (any length): - C/C++: binary operators other than == != < > <= >= && || (those yield bool without evaluating an operand), member access (. and ->), calls on a non-name callee, subscripts, conditional consequences, parentheses, comma lists. - Python: binary operators, attribute access, calls on obj.m(...), f()() and (expr)(), subscripts, the else branch of a conditional, parentheses. Still counted, unchanged guard and reporting (genuine nesting): - C/C++: call arguments used for template argument deduction, std::move/std::forward operands, unary, pointer and update operands, assignment left sides, lambda return expressions, co_await, fold and _Generic operands. - Python: tuple/list/set elements, dictionary keys and values, the true branch of a conditional, await operands, next()/iter()/assert_type() arguments, called lambdas. Proof: - New c_lsp and py_lsp tests build chains of 300, 2,000 and 20,000 terms (a binary + chain and a .a() receiver chain) next to an unrelated s.m(). On the previous evaluator, and with this change reverted, all four fail at 300 terms: the depth error is recorded and the file resolves 0 calls (C++ per-file; Python per-file and through the cross-file dispatch). With it, every size records no error, resolves run -> S.m, and the chain's result type resolves x.done() / the final .done(). - Real input, isolated cache, before and after: redis 4f20cb48 (C): 36,876 CALLS, 10,699 LSP-resolved; django fb113765 (Python): 60,123 CALLS, 25,153 LSP-resolved. Identical CALLS edge sets on both (0 lost, 0 gained, 0 strategy changes), 0 files with an LSP error before and after, wall time unchanged (redis 5.5 s, django 9.1-10.2 s both ways). Refs #1527 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c3c97a99d7 |
fix(lsp): bound repeated resolver work without truncating large files
Memoize C expression evaluation, index repeated scope and registry queries, and report incomplete walks explicitly on memo or depth failure. Refs #1527 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
2ef6ba7101 |
fix(extract): name macro-wrapped C/C++ definitions after the macro argument (#946, #1957)
Why: a function-like macro in a definition's declarator,
`void CBasePlayer::__API_HOOK(ReloadWeapons)(int slot)` or
`static void FNAME(first_caller)(void)`, parses as a function_declarator
nested directly inside the definition's own function_declarator. The
declarator walk descended to the terminal identifier, which is the MACRO,
so every definition wrapped by the same macro got one qualified name.
ReGameDLL_CS's hook definitions landed as 20 nodes literally named
__API_HOOK, and the Linux kernel's arch/x86/kvm/mmu/paging_tmpl.h (19
FNAME(...) definitions) as a single Function FNAME. The last definition
won, calls in every other body were sourced to it with a call line
outside its range, and callers never reached the real functions. The C
LSP had its own copy of the walk: it named the caller after the macro and
bound the macro's argument list as the function's parameters, so member
calls through a real parameter went unresolved.
Fix: cbm_c_macro_wrapped_name_node() recognises the shape. Valid C/C++
never nests two function_declarators directly (a function cannot return a
function; a function-pointer return has a parenthesized_declarator in
between), so the nesting always means a macro call. When that macro is a
plain or class-qualified name taking exactly one bare-name argument, the
definition is named after the argument; any other shape (multi-argument
wrappers, non-name arguments) keeps today's walk. The shared C-family
declarator resolver uses it, so the def name, the call-scope QN and the
unified extractor agree; the C LSP uses it too and keeps the outer
(real) parameter list. The class scope of `Cls::MACRO(x)` was already
derived from the qualified name and is unchanged.
Proof: dev builds of origin/main
|
||
|
Martin Vogel
|
bf2daa3603 |
fix(lsp): detach field arrays before overlay refinement
Copy inherited field-type pointer arrays into the writable registry arena before Python refines an existing field. Keep shared type objects intact and avoid retaining per-file scratch types in the sealed fallback. Add deterministic shared-state and later-overlay coverage, retaining the sequential Python field regression and adding a forced-parallel variant. Refs #1277 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a8c4040e8a |
fix(lsp): bound structured-binding field reads and TS method heritage walk (#1743)
Two deterministic indexing-worker crashes reported on #1743. C/C++ (the original LLVM report, "segfault at 9"): structured-binding decomposition read rt->field_types[binding_idx] guarded only by `rt->field_types[binding_idx] != NULL`. field_types[] is parallel to field_names[] and NULL-terminated, so a binding list longer than the struct's field list (ill-formed, but present in compiler test suites, e.g. llvm clang/test/CXX/dcl.decl/dcl.decomp/p3.cpp) read past the terminator into the next arena object. That object is a CBMType whose first word is its kind (9), so the name was bound to the pointer 0x9 and decltype(name) dereferenced it in c_resolve_name_to_type. All three decomposition sites now go through c_struct_field_type_at(), which bounds the index by field_names[] and yields NULL (unknown) past the end. TypeScript (kilocode report on v0.11.0): lookup_method walked the extends/implements chain recursively with no depth bound. The Effect idiom `class Service extends Context.Service<Service, I>()(...)` gets its base resolved to the class itself, so `Service.of(...)` (or a sibling class whose base resolves to that Service) recursed until the resolver thread's stack overflowed. lookup_method now has the same depth-guarded entry as lookup_member_type (TS_LSP_MAX_MEMBER_DEPTH); past the cap the method is simply not found. Proof on real input (macOS arm64, fresh cache each run): - llvm/llvm-project llvmorg-15.0.7: before, 5 worker SIGSEGVs at 0x9 in c_resolve_name_to_type and 3 files quarantined; after, 0 crashes, 0 quarantined, 1,375,933 nodes / 4,257,770 edges. - Kilo-Org/kilocode main: before, the resolve worker died every attempt (SIGBUS, lookup_method recursion depth ~65k); after, indexed cleanly, 90,745 nodes / 440,654 edges. Tests: c_lsp::clsp_structured_binding_more_names_than_fields and ts_lsp::tslsp_nocrash_cyclic_extends_method_call (both crash RED without the fix). Fixes #1743 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7b77fd4834 |
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (
|
||
 Martin Vogelandastandrik
|
da61c81af2 |
fix(lsp): unify the two type short-name indexes that broke the main build
main does not compile. internal/cbm/lsp/type_registry.h declares type_short_buckets, type_short_entries and type_short_bucket_count twice, and cbm_registry_types_by_short_name twice with different parameter types. Two changes implemented the same feature independently: |
||
|
astandrik
|
32ae20d0ea |
fix(lsp): preserve short-name index fallback
Signed-off-by: astandrik <astandrik@yandex-team.ru> |
||
|
astandrik
|
793716dcb4 |
perf(lsp): index C++ short-name type lookups
Avoid repeated full type-registry scans in the namespaced-type fallback. Build the optional index only for C/C++ cross registries while preserving candidate order, post-finalize tails, and allocation fallback. Refs #1677 Signed-off-by: astandrik <astandrik@yandex-team.ru> |
||
|
Martin Vogel
|
8018561cfe |
fix(release): externalize runtime assets and harden VT verification
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
679871aedb |
fix(c_lsp): remove the whole-function ASan suppression; assert the resolution it hid
c_lsp_process_file carried __attribute__((no_sanitize("address"))) over the
entire C LSP orchestration, excluding every direct memory access in it from
ASan -- placed for a stack-pointer-to-registry lifetime bug whose witness test
also had its assertion discarded ((void)find_resolved), so nothing could ever
prove the bug gone. Flagged by the memory-diagnostics standards report
(private/C_MEMORY_DIAGNOSTICS_STANDARDS_REPORT.md, priority 2) as the single
highest-value ASan coverage gap in the repository.
This branch's C LSP registry rework resolved the underlying lifetime: with the
suppression removed, the full c_lsp suite passes under ASan+UBSan (760 tests),
including the formerly-failing template field type resolution, whose test now
asserts the resolved call instead of discarding it.
Honest limit: the original bug predates the branch and its fix is the branch's
broad registry rework, not an isolatable commit -- so the revert-proof is
main-vs-branch (main: suppression + discarded assertion; branch: neither),
not a single-change revert-check.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
87a0e3f74f |
fix: preserve semantic graph relationships
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
2abcebc821 |
perf(registry,rust-lsp): embedded-type + free-func short-name indexes
The Rust cross-file resolver's trait-method, sole-trait-impl, and free-func fallback loops scanned the ENTIRE type/func registry per call site on a miss. On kernel-scale registries (~10^6 entries, macro-inflated) four tiny trait-heavy rust files burned ~96 s EACH (~200k eval-capped call sites x full-registry strcmp scans). Fix: two hash indexes built at registry finalize, mirroring the existing bucket+entries idiom -- fnv1a(bare(embedded_type)) -> chain of declaring type indices (trait -> implementers), and fnv1a(short_name) -> chain of free-func indices. Chains preserve ascending registry order (first-match tie-break unchanged); allocation-free iterators fall back to the post-finalize tail and to a full scan on unfinalized registries, so the candidate set is identical in every mode. The four rust loops now walk the iterator and re-apply their exact original predicates -- byte-identical results, loop bodies untouched. Verified: graph equivalence on the kernel rust/ subtree (nodes 9654, edges 44798 identical with and without the change); registry_short_name_indexes unit test pins iterator contents/order/dedup; suites c_lsp 750, rust_lsp 505, registry 54, pipeline 216 -- zero failures. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
bd005b4b09 |
fix(py-lsp): route instance-field discovery through a per-file overlay
py_register_instance_field mutated the shared Tier-2 registry through direct struct writes that bypass the read_only seal: it overwrote field-type pointers in place and reassigned the sealed type entry's field arrays to memory owned by the per-file arena. In the fused parallel resolve path (one sealed registry shared by all resolve workers) that is a cross-thread data race, and it leaves the shared registry pointing into arenas that die with their file -- a cross-file use-after-free. The count-based seal guard could not see it because type/func counts never change. Python extraction emits no instance fields, so resolve-time discovery is the ONLY source of self.x / PEP-526 fields; a plain read_only gate would silently drop real attribute-chain CALLS edges. Fix instead routes discovery into a per-file field overlay on PyLSPContext (arena-backed, dies with the file) and has py_lookup_field_depth consult shared-base + overlay at the exact precedence point of the direct field scan. Mutable per-file registries keep the original in-place write, byte-identical. Reproduce-first, both halves pinned: - seal_py_shared_registry_readonly_fields: RED on the bug (field count 0->1 on a sealed registry), GREEN with the overlay. - pylsp_fused_self_attr_chain_via_overlay: GREEN with the overlay, RED under a simulated plain gate (self.b.work() CALLS edge lost) -- proven both ways. Suites: c_lsp 749, py_lsp 63+45 stress/scale/bench, pipeline 216, registry 54, scope 10, type_rep 19 -- zero failures. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
d10701dced |
fix(lsp): seal shared Tier-2 cross-registry against O(n^2) resolve hang
The per-file C cross-LSP resolver mutated the finalized, project-wide cross
registry that is shared read-only across the parallel resolve workers. Each
post-finalize add landed in a tail the hash index does not cover, so every
lookup linear-scanned an ever-growing tail -> O(files*defs) on large C
codebases. The Linux-kernel full index hung at "[4/9] Resolving" on 11 cores
for >6 min and never finished, plus a heap data race across workers.
Seal every shared cross-registry: CBMTypeRegistry gains a read_only flag, set
by all five {c,py,cs,ts,go}_build_cross_registry builders right after
finalize, and cbm_registry_add_func/_type no-op on a sealed registry. One
chokepoint guards every language, robust to any resolver mutation site. The
plain-C function-registration site is also skipped directly when the registry
is shared.
Add six cross-language invariant tests (c, c++, py, c#, ts, go): resolving
against a finalized shared registry must leave func_count/type_count
unchanged -- RED before the fix (C +1, C++ +2 post-finalize adds), GREEN
after.
Verified: full suite 5720 passed / 0 failed; Linux-kernel full index now
completes all 9 phases in 643s (4.88M nodes, 11.9M edges) instead of hanging
at phase 4.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
 Martin Vogel
|
eed87fd372 |
Forbid test skips and convert existing skips to hard failures
Add scripts/check-no-test-skips.sh (run from lint) which fails the lint phase on any plain SKIP() or direct tf_skip_count manipulation; only SKIP_PLATFORM() (for genuinely platform-specific tests) is tolerated. Add FAIL() and SKIP_PLATFORM() helpers to the test framework and convert the remaining SKIP()/perf-gated skips across the suite into pass-or-fail assertions, so a suite that cannot meet its preconditions reports a red failure instead of a silent skip. |
||
|
Martin Vogel
|
48a9b2c2a9 |
test(c-lsp): regression for #220 (C++ CALLS attributed to function)
#220: C++ CALLS edges originated from the Module node instead of the enclosing Function, breaking function-level trace_path. The cross-LSP resolver now sets caller_qn = enclosing_func_qn; the test asserts a call (doWork→helper) is attributed to doWork, not the module. Closes #220. |
||
|
Martin Vogel
|
ba49137165 |
test(c-lsp): xxhash-header extraction guard (issue #355 proxy)
Issue #355 reported a segfault extracting a macro-heavy xxhash C header (dhw/xx_hash.h, not available to us). Add a guard that runs the vendored xxhash.h (~7.5k lines, same macro-dense family) through C extraction under ASan. It passes on current main — the C-LSP crash hardening (#322/#323/ #360) appears to cover this class. Kept as a regression guard; #355 stays open pending the reporter's confirmation/file. |
||
|
Martin Vogel
|
7dafde46dc |
test(c-lsp): regression for issue #312 (defaulted template param + auto)
A C++ function template with a defaulted type param and an `auto` parameter, called with matching arg count, previously segfaulted in the definitions pass (unbound `U` flowed through type substitution as NULL). Fixed by the cbm_type_substitute NULL-preservation guard; this locks in a reproduction so it can't regress. Closes #312. |
||
 Smiie-2
|
d895c16553 |
fix(c-lsp): prevent segfault on nested C++ template calls with default args
Root cause: unbound template parameters flowed through cbm_type_substitute as NULL, and call-expression return handling then dereferenced the NULL substituted type. Reproduced on MangoHud's src/hud_elements.cpp. - type_rep: cbm_type_substitute now returns the original param `t` (not NULL) when a type arg is unbound, in both the TYPE_PARAM and named-param paths — unbound params stay intact instead of poisoning the type. - c_lsp: guard `if (!ret) return cbm_type_unknown();` before unwrapping the substituted return type; guard the substitute result before use; add template_function call-node resolution (qualified/scoped/plain names). - extract_defs: factor template-inner lookup into find_cpp_template_inner_node, which recurses into nested template_declarations and also handles field_declaration; reuse it from unwrap_template_inner. - tests: clsp_nocrash_template_function_multi_param_nested_call (the nested reproducer) and typerep_substitute_unbound_param_preserved. Distilled from #360 onto current main. The formal-parameter-count clamp that overlapped this PR already landed via #322, so only the additive crash fixes are taken here. Contributes to the C++ stability cluster (#390; relates to #215/#312). |
||
 Romano
|
05f0a26b2f |
fix(c-lsp): guard pending template call resolution (#322)
Fixes the OOB read in c_resolve_pending_template_calls when a call site has more arguments than the parsed function signature (issue #215). Verified locally: build clean, all 3,617 tests pass including the new clsp_nocrash_template_extra_call_args regression. |
||
|
Martin Vogel
|
aa2b60be7d |
Fix 29 CodeQL alerts: command injection, snprintf overflow, TOCTOU races
- Command injection (CRITICAL): validate shell args before system() in update command's unzip and version-check calls - TOCTOU cli.c: use open(O_CREAT, 0755) + fdopen() to set permissions atomically instead of fopen() + chmod() after close - TOCTOU pass_envscan.c: open file first, then fstat() on fd to check size, eliminating stat-then-open race window - Overflowing snprintf (11 locations): clamp offset after each append to prevent unsigned underflow on truncation in cypher.c, store.c, http_server.c, test_c_lsp.c - Add CBM_SNPRINTF_APPEND macro in str_util.h for future safe appends - CodeQL: remove pull_request trigger (only scan push to main) - CodeQL gate: increase timeout from 30 to 45 minutes - Add fuzz testing script (random JSON-RPC + Cypher mutations) - 12 Scorecard governance alerts dismissed (not code vulnerabilities) |
||
|
Martin Vogel
|
2e21a0133b |
Release security hardening: VirusTotal + attestations + cosign + SBOM + OpenSSF + SECURITY.md
Release pipeline (step 5 + 6) now includes: - SLSA build provenance attestations for all binaries - Sigstore cosign keyless signing (.bundle files) - SBOM (CycloneDX) with all vendored dependencies - VirusTotal scan of all binaries (70+ engines) - OpenSSF Scorecard (repo security health score) - All results appended to release notes automatically Also: - SECURITY.md: vulnerability reporting + security measures - Removed DLL API strings from test binary (issue #89) |
||
|
Martin Vogel
|
0d216c7936 |
Resolve all clang-tidy, cppcheck, and clang-format warnings across 94 files
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories. All non-FP warnings fixed at source; true false positives get per-instance NOLINT with reasoning. Fixes include: braces around statements (1679), isolate declarations (75), explicit casts for multilevel pointer conversions, missing includes, function parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting. |
||
|
Martin Vogel
|
94a9a92f83 |
Add C-native pipeline with blocking clang-tidy, cppcheck, clang-format linting
Complete C implementation of the indexing pipeline (src/), parallel worker pool, graph buffer with merge support, SQLite writer, and 1893-test suite. Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT suppressions), cppcheck, and clang-format. All issues fixed at source — proper headers for external linkage, GROW_ARRAY macro restructured to eliminate type parameter, null-deref paths guarded, named intermediates for suspicious-argument checks. |