eval_expr_partial, the three-valued evaluator that execute_single runs on
every seed row before relationship expansion, recursed into the right
child of every AND / OR / XOR node. With operator chains linked along the
right child, that is one stack frame per operand: the 50,000-operand OR
and AND chain tests overflowed the 256 KiB query worker stack in this
early pass, before the already iterative eval_expr ever ran.
Walk a chain the way eval_expr does: recurse into NOT and into each
operand, and follow the links in a loop. The operands are read in the
same order and with the same short-circuits as the nested form (AND stops
at the first definite false, OR at the first definite true, XOR reads
every operand), and the running value is combined three-valued, so an
UNKNOWN operand anywhere in the chain keeps the seed for the full
evaluation after expansion.
A new test runs three-operand OR / XOR / AND chains whose UNKNOWN operand
sits on a chain link and not only on the chain's head. A loop that lets a
later operand overwrite the UNKNOWN prunes the seed and fails it.
eval_expr_partial moves into the whitelist's chain-walker group with
eval_expr and cypher_expr_selectivity: it still recurses, but only as
deep as the parse-depth cap allows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Use a thread-local test clock for the large UNION structural regression
so sanitizer overhead cannot consume its execution deadline. Preserve
all row, ordering, and constrained-stack checks. Add an exact positive
budget expiry case alongside the existing deadline tests.
Request the fixture's exact 20000-row budget to avoid reserving the
default 100000 binding slots for every branch, and assert no truncation.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A WHERE chain such as `a OR b OR c ...` was folded into a left-deep tree,
so every walker over the expression tree recursed once per operand: the
evaluator, the seed-selectivity pass of the planner and, every 128 nodes,
the free routine. UNION branches were parsed by a nested cbm_parse call
per branch, so the parser recursed once per branch as well. Stack use
therefore followed the length of a query, not its nesting, and the
daemon evaluates queries on a worker thread with a 256 KiB stack. A long
but flat query, such as a few thousand `OR name = ...` terms, needs to
work there.
Chains of one operator are now linked along the right child at parse
time, OP(a, OP(b, OP(c, ...))), and eval_expr and cypher_expr_selectivity
follow the links in a loop, recursing only into the operands, whose
nesting the existing parse-depth cap bounds. Operand order and the
left-to-right short-circuit of AND and OR are the same as in the nested
form, XOR keeps its running parity, and a missing value takes part in a
chain exactly as before. expr_free already walks with an explicit stack
and needs no change for a right-linked chain.
UNION branches are read by one parser in a loop in cbm_parse and linked
as they come; parse_post_where no longer parses the branch after the
keyword, and the ALL flag still sits on the branch before it.
Tests: 50,000-operand OR and AND chains and 20,000 UNION ALL branches run
on a thread with the worker stack size inside a forked child, with the
expected rows computed by the test; a chain with operands over a missing
property is compared with its parenthesised forms in both associations;
50 UNION and UNION ALL branches return the right rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Four syntax-tree walkers recursed once per nesting level of the indexed
file, so their stack use followed the depth of the parse tree rather than
anything bounded. They walk iteratively now, with identical results.
- terminal_callee_leaf (extract_calls.c) and its PHP LSP mirror
php_reference_callee_leaf descended one wrapper node per frame on a
callee such as ((((f))))(...). Both are loops now: each step either
returns or moves to the child the search continues in, so the nodes
visited and the leaf chosen are unchanged.
- cbm_collect_error_regions and cbm_mark_pp_error_rows (cbm.c) recursed
into every child that contains an error. One cursor walk now drives
both. It stops on exactly the visible children ts_node_child
enumerated, in the same order, and never enters an ERROR subtree.
- is_direct_argument_value (extract_usages.c) moved from a labeled
argument's value to the argument itself by recursing. The grammar bounds
that step, but it is a loop now, taking the slow-parent test note once
per level as before.
calls_append_args (pass_calls.c) formatted each argument into a 512-byte
local and assumed the result fit. An argument that does not fit now ends
the array like one the props buffer cannot take. The parallel twin formats in
place and already stops on truncation.
Indexing this repository's src/ and internal/cbm/ with and without the
change gives byte-identical sorted dumps of nodes, edges per type, CALLS
edges and parse-coverage rows.
CBM_WORKER_STACK_SIZE moves to worker_pool.h so the new extraction tests
run on a thread of exactly the pipeline's size, in a forked child on POSIX
so an overflow is a signal instead of a dead runner. They nest 200,000
parentheses around a first-class callable and around an invalid token,
pin the labeled-argument classification across the loop conversion, and
hand the formatter an argument longer than its local buffer.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Delphi code that calls 1C's OLE API through late-bound Variants uses
Cyrillic member names (v8_Base.<Cyrillic>.<Cyrillic>('1', Now),
NM.<Cyrillic>()). Every such name produced a tree-sitter ERROR node, so the
unit was reported as parse_partial and a procedure whose own name is
Cyrillic was dropped from the graph.
Root cause is upstream tree-sitter-pascal (Isopod, the pin we vendor):
its identifier token is ASCII-only, /[&]?[a-zA-Z_]+[0-9_a-zA-Z]*/, and
there is no other token a non-ASCII letter can lex as.
Fix: pascal becomes a self-maintained fork of the pin we already vendored
(Isopod/tree-sitter-pascal @ 042119eca2e1, v0.10.2 = upstream master),
following the arkts model. One patch
(tools/tree-sitter-pascal/patches/unicode-identifiers.patch) changes only
that token to /[&]?[\p{L}_][\p{L}\p{M}\p{Nd}_]*/. Keywords are still
extracted from this `word` token, so begin/end/procedure lex as before
(`begin` glued to Cyrillic letters is one identifier; `&begin` stays an
escaped identifier). Node types are unchanged. The \p{} classes compile
to numeric code-point ranges: the vendored parser.c has 0 non-ASCII bytes.
Regenerated with tree-sitter-cli 0.25.10 --abi 14 (ABI unchanged, no
external scanner; tree_sitter/parser.h is byte-identical to chialisp's).
Fidelity: the UNPATCHED grammar regenerated with 0.25.10 gives the same
symbol/token/field/production counts as upstream's shipped parser and
byte-identical parse trees on upstream's 88 corpus cases and its three
examples/*.pas (37,773 tree lines for codetoolmanager.pas); the patched
parser gives the same identical trees on all of those ASCII inputs and
passes all 88 corpus cases.
Proof (7 Pascal files: the report's snippet, a Cyrillic procedure with
keyword-boundary cases, an ASCII control, a broken unit, upstream's three
examples; fresh caches): parse_partial 3 -> 1 (only the genuinely broken
unit remains), nodes 581 -> 582 (the Cyrillic-named procedure), edges
1480 -> 1482 (+1 DEFINES, +1 SEMANTICALLY_RELATED); per-file function
counts for the ASCII files are unchanged.
Registered like arkts: MANIFEST.md fork row + note, audit FORKS map,
THIRD_PARTY.md, vendored checksums. LICENSE stays upstream's MIT,
byte-identical.
Tests: parse_coverage gains the report's snippet and a Cyrillic-named
procedure (not partial, definitions extracted), an ASCII control, and a
guard that broken Pascal is still flagged; the fork carries
test/corpus/unicode_identifiers.txt.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Why: a function-like macro in a definition's declarator,
`void CBasePlayer::__API_HOOK(ReloadWeapons)(int slot)` or
`static void FNAME(first_caller)(void)`, parses as a function_declarator
nested directly inside the definition's own function_declarator. The
declarator walk descended to the terminal identifier, which is the MACRO,
so every definition wrapped by the same macro got one qualified name.
ReGameDLL_CS's hook definitions landed as 20 nodes literally named
__API_HOOK, and the Linux kernel's arch/x86/kvm/mmu/paging_tmpl.h (19
FNAME(...) definitions) as a single Function FNAME. The last definition
won, calls in every other body were sourced to it with a call line
outside its range, and callers never reached the real functions. The C
LSP had its own copy of the walk: it named the caller after the macro and
bound the macro's argument list as the function's parameters, so member
calls through a real parameter went unresolved.
Fix: cbm_c_macro_wrapped_name_node() recognises the shape. Valid C/C++
never nests two function_declarators directly (a function cannot return a
function; a function-pointer return has a parenthesized_declarator in
between), so the nesting always means a macro call. When that macro is a
plain or class-qualified name taking exactly one bare-name argument, the
definition is named after the argument; any other shape (multi-argument
wrappers, non-name arguments) keeps today's walk. The shared C-family
declarator resolver uses it, so the def name, the call-scope QN and the
unified extractor agree; the C LSP uses it too and keeps the outer
(real) parameter list. The class scope of `Cls::MACRO(x)` was already
derived from the qualified name and is unchanged.
Proof: dev builds of origin/main 1f9b4db1 (before) and this change
(after), same machine, fresh indexes.
rehlds/ReGameDLL_CS @ 4a50c42 (#946):
- nodes named __API_HOOK: 20 -> 0
- Function 986 -> 1003, Method 3768 -> 3859: 129 hook definitions now
have their own node, and every added definition starts on a
`__API_HOOK(<name>)(` line (0 false definitions). Removed: the 20
__API_HOOK nodes, plus CGrenade::Explode(Vector, Vector), which now
shares CGrenade.Explode with its hooked overload, as any C++ overload
pair does today (one node per QN).
- CALLS 9436 -> 9916; CALLS sourced from __API_HOOK 410 -> 0; callable-
sourced CALLS whose call line lies outside the caller's range 565 -> 198.
- parse_partial files 131 -> 131.
- 5 samples checked against source (CBasePlayer::Spawn 5771-6142 with 32
in-range out-calls incl. ReloadWeapons via lsp_implicit_this,
CBasePlayer::TakeDamage, PM_Move, CGrenade::ShootSmokeGrenade,
CBotManager::OnEvent).
- 18 of the 147 hook definitions are still not extracted, before and
after: the parser loses them to raw ERROR regions (a `T *EXT_FUNC`
pointer return misparsed as a multiplication, a preceding
semicolon-less LINK_HOOK_*(...) invocation, an #ifdef-split
signature). That is parser recovery, not naming, and is left for a
follow-up.
Linux arch/x86/kvm/mmu @ 2d2338c93 (#1957):
- paging_tmpl.h: 1 Function FNAME -> 19 distinct functions (all 19
FNAME(...) definitions; 0 false definitions in the directory).
- CALLS from paging_tmpl.h functions with the call line outside the
caller's range: 32 -> 0 (directory: 35 -> 3, the 3 pre-existing in
mmu.c); e.g. fetch 651-786 now owns its 13 calls, sync_spte 934-1005
its 8.
- 5 samples checked against source (prefetch_gpte, walk_addr_generic,
fetch, sync_spte, page_fault).
- #1957's own reproduction: first_caller and later_function are distinct
Functions, sink() at line 9 is sourced to first_caller (7-10), and
main -> first_caller resolves.
Tests: extraction::extract_cpp_macro_wrapped_declarator_names_def_issue946
(qualified + unqualified + EXT_FUNC hooks, macro in a header: def names,
QNs, range, real parameter count, per-body call scope),
extraction::extract_c_fname_macro_definitions_stay_distinct_issue1957
(the reported C reproduction incl. the LSP caller QN, and the kernel
template shape with a token-pasting FNAME defined only under #if),
extraction::extract_c_macro_wrapped_declarator_controls_issue946
(precision controls: file-scope and in-body macro call statements, a call
through a returned function pointer, a real function returning a function
pointer, a macro-wrapped prototype and a two-argument wrapper mint no
definition named after a macro argument), and
c_lsp::clsp_macro_wrapped_declarator_caller_qn_issue946 (LSP caller QN,
lsp_implicit_this, typed parameter). RED 3x before, GREEN after, RED on
revert; the LSP half alone binds the c_lsp test and the #1957 test.
Fixes#946Fixes#1957
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Filter non-PHP candidates before selection and distinguish an empty namespace map from missing namespace information.
Refs #1186.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The job log of a red test job carries a summary of each failing suite,
not its log, and the only artifact was shard-manifest.txt. A failure
whose summary was not enough could only be understood by reproducing it.
The four jobs in .github/workflows/_test.yml that run the parallel
harness (test-unix, test-windows, test-diag, test-lsan-macos) get one
more step, "Failing suite logs". It runs only when the job failed and
uploads build/c/test-logs/failed/: the logs of the suites that did not
end green, plus results.txt, as the harness leaves them there. One red
suite is typically a few kilobytes; a whole run's logs are about 25 MB.
Retention is 7 days. The artifact names follow the shard-manifest ones
(suite-logs-<os>-<compiler or msystem>-<job index>, suite-logs-diag,
suite-logs-lsan-macos), and the step uses the upload-artifact reference
the file already pins.
A job that failed outside the suite waves has no such directory; the
step then warns and uploads nothing. The step cannot change a job's
result: it is skipped on a green job, and a failed job stays failed
whatever the upload does. Triggers, permissions, job dependencies,
timeouts, the matrix and the shard-completeness download pattern are
unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Windows sanitizer job went red on one suite that aborted under
AddressSanitizer, and the job log showed an empty "every failure site"
section followed by the shadow-byte legend. The summary in
scripts/run-tests-parallel.sh searched the suite log for "FAIL" only. A
sanitizer report never contains that word, and its useful part (error
kind, access size, top frames) is at the start of the report, far above
the last 15 lines.
The summary now lives in scripts/suite-failure-report.sh, which the
harness sources. Between the unchanged failure-site and last-15-lines
sections it prints, for a failing suite, the test that was running and
the first sanitizer report. The framework flushes each test name before
the test body runs, so the last test line ahead of a report is the test
that was running; a report that comes after the last test finished is
labelled as such instead of blaming a test. The report is shown from its
header (AddressSanitizer, LeakSanitizer, ThreadSanitizer,
MemorySanitizer, UndefinedBehaviorSanitizer) to its own SUMMARY line, at
most 40 lines, without the shadow dump and the legend; a recoverable
"runtime error:" line is shown with its stack only. Further sanitizer
headers, "runtime error:" lines and SUMMARY lines follow (at most 10),
then signal and timeout lines (at most 5).
A suite that dies with no report and no completion summary gets the
running test named. A trap-mode UBSan build stops on an illegal
instruction and prints nothing, so that name is the only evidence its
log holds. A log with an assertion failure and no report prints exactly
what it printed before.
The same file keeps the logs a red run needs. An EXIT trap in the
harness copies the log of every suite that did not record rc=0, plus
results.txt, into test-logs/failed/. The trap covers the scheduler
giving up and the union guard as well as the normal end, and it never
changes the exit status.
scripts/run-test-wave.py reported pass=0 for a suite that died without
its summary line. In that case, and only then, it now counts the PASS
markers in the log. A suite with a summary line is never recounted, so
the totals of a green run are unchanged.
tests/test_failure_report_contract.sh drives the real functions with
fabricated suite logs and runs as Step 0i2 of scripts/test.sh. It fails
on the previous behaviour in every sanitizer, running-test and
result-line check and passes on it in the unchanged-output checks. The
awk code uses no interval expressions and no character classes; the test
passes with BSD awk and grep on macOS and with mawk and GNU grep on
Ubuntu. scripts/README.md names the new internal file.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The doc<->code candidate scoring (#2574, doc_section_worker in
semantic_edges) scores every documentation section against the functions
that share a key term with it, key terms being those held by at most
funcs/20 functions. Its cost therefore grows with sections x functions.
Measured on 11 corpora (10-09 main vs 4d9592b5, semantic_edges): under a
second up to ~100k functions, +12.6 s on java (325k), +14.6 s on csharp
(451k), +174 s on the Linux kernel (784k functions, 28k sections), almost
all of the kernel's regression against v0.11.0.
A new boolean setting, doc_candidates (default true), and an optional
index_repository argument of the same name switch the scoring off: no
candidate rows are stored and nodes and edges are unchanged. The argument
wins over the setting. The index handler resolves the value once and
passes it to a daemon job or supervised worker as an argument, the same way
as the resource policy, so every route sees one decision; the session
auto-index reads the setting. With the switch off, a delta-route reindex
replaces the candidate rows it would otherwise keep, so turning it off
leaves no stale candidates. The default output is unchanged.
Tests (each fails on its assertion when the switch has no effect):
- pipeline_doc_candidates_off_stores_none_and_keeps_graph: no rows, same
node and edge counts as the default run, which has rows;
- pipeline_doc_candidates_off_clears_rows_on_delta: a closure-repair
reindex with the switch off removes the earlier rows;
- tool_index_repository_doc_candidates_switch: argument false -> none;
setting default -> rows; setting false -> none; argument true over
setting false -> rows.
Proof, release-built binaries on macOS arm64 (M5 Pro 18c/48 GB):
- django, main 4d9592b5 vs this change with defaults: nodes (63,441),
edges (307,584) and doc_link_candidates rows (11,304) hash-identical;
- django with doc_candidates false: nodes and edges hash-identical, no
candidate rows;
- Linux kernel with doc_candidates false: semantic_edges 17.9 s (191.9 s on
main, 17.6 s before #2574), whole index 210.9 s (v0.11.0: ~234 s), no
candidate rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Discovery reads the tracked-file set through the supervised subprocess
API instead of cbm_popen, so no shell parses the command:
- git is resolved to an absolute path from absolute PATH entries only
(cbm_test_impact_find_git), never the current directory;
- the repository path is a single argv entry;
- the child environment drops repository-local variables, so an
inherited GIT_DIR cannot redirect -C;
- core.fsmonitor is off for the call;
- stdout goes to a private mkstemp file that is read and removed.
discover.c needs no entry on the security allow-list.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The 2026-10-12 dry run (run 38186491924, windows-11-arm 2/2) killed
doc_mentions_go at the 900 s wall clock. The test framework prints a test's
name before running it, so the last line shows the hang was in
doc_mentions_go_local_rules, before the pipeline's first info line
(pipeline.discover). Nothing in the log says which step. The hang has not
come back: 0 of 32 runs on a local Windows arm64 VM, and the next CI
sample passed (run 38204190338, 80 s).
The steps before discovery have no deadline: the fixture's temp
directory, the fixture writes, three `git` children per index (git context
in cbm_pipeline_new and again in the run), staging, the coverage check and
the test config. This change makes each of them name itself at debug level,
and the Go suite's fixture index runs at debug level, so a recurrence ends
on the step it is stuck in:
- git_capture: spawn / read / close / done, with the git arguments;
- cbm_pipeline_run and its staged run: stage, staged, coverage,
git_context, test_config, discover;
- go_index_lookup: mkdtemp, fixture, index; it raises the level to debug
and restores the caller's level afterwards.
Debug level is otherwise unused on the indexing path, so the default
output does not change and only this suite's fixture indexes print the
markers.
Checked: doc_mentions_go passes 23 of 23 on macOS and on a Windows arm64
VM (win.sh test), each printing the same markers in order: 30 from the
helper, 60 pipeline setup steps and 120 git stages (10 indexes, 3 git
children each). clang-format is clean. CI's cppcheck 2.20 reports one style
finding in git_context.c (buf_size > 0 always true), which main has too, in
a function this change does not touch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Smoke's binary string audit blocked on every platform in the dry run on
70543ba8 (run 38204190338, all 12 smoke jobs): "Credential patterns
found in binary". #2068 rebuilt the graph-ui bundle. The two reviewed
Monaco 0.56.0 quick-input lines are still there but minified to different
bytes, so their SHA-256 no longer matched the pair recorded on 09-09.
Re-reviewed: the CI-built bundle has exactly two flagged lines (15,644 and
72,472 bytes). Every match is the input box's password-mode flag
(_password=!1, password=!!e.password, inputBox.password=this.password,
inputBox.password=!1), not a credential. All 12 smoke logs show the same
two lines. The new pair is computed as filter_reviewed_lines does
(printf '%s' line | sha256) on CI's binaries: GNU strings on the Linux
amd64 artifact and Apple strings on the darwin arm64 artifact give the same
two hashes, and both are exempted. The comment now says how to redo this
when the bundle changes again.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings in #2596 (MSan: hidden uninitialized reads) and #2068 (CodeAtlas
web).
README: main's CBM Atlas line and this branch's infrastructure-as-code
line are both kept.
tools/list metadata: main and this branch each stayed under the 20 KiB
discovery budget, but together they reached 20,809 bytes. The branch's
wording is shorter now (search_graph kind/namespace, one hint per trace
mode: 136 bytes less), and the reviewed budget is 21 KiB so the
topology/deploy/run modes keep their usage hints.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
git applies ignore rules to untracked paths only, but discovery applied
them to every path. A nested .gitignore line `manager`, meant for a
built binary, hid cluster-api's tracked config/manager/ kustomization,
and with it the Deployment its overlays point at.
When the indexed directory is in a git repository, discovery now reads
`git ls-files -z` once. A directory that git's rules ignore (.gitignore
files, info/exclude, the global excludes file) is still entered when it
holds tracked files. Inside it, untracked files and directories stay
ignored, as in git. A tracked file is kept although a rule matches it.
The built-in skip list, .cbmignore and the size cap apply as before. If
git cannot run, nothing changes.
On the 17 infra corpora git lists 34 tracked files that an ignore rule
matches. The fix indexes 26 of them, all from that list. The other 8 are
never indexed for other reasons: no language (.tfvars, .gitkeep) or a
built-in skip directory (vendor/). No file is lost and no untracked
file appears. cluster-api gains the manager kustomization (4 Resources,
its overlay IMPORTS), helm-charts a Chart, and otel-demo a Route.
Test: discover_gitignore_never_hides_tracked_files builds a real git
repository. It fails without the fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
After a parallel extract or resolve, the main buffer's counter was set to
the shared allocator's value. Merged worker nodes keep their shared ids,
and merged edges take fresh ids from the main counter after the highest
of them, so the main counter can end above the shared one. Setting it
down handed the merged edges' ids out a second time.
All four sync points (full route and closure repair, after extract and
after resolve) now go through cbm_pipeline_sync_next_id, which only
raises the counter. The delta preseed still pins it with
cbm_gbuf_set_next_id, as before.
On the 17 infra corpora the old sync left 94,239 duplicate edge ids in
the buffer (11 corpora; deckhouse 44,959), and the fix leaves none. The
persisted graph does not change, because the dump numbers edges itself.
The duplicates matter to anything that finds buffer edges by id:
cascade_delete_edges unindexes by id and then frees the edge, so a
duplicate can leave a freed edge in an index.
Tests: a parallel full index with the counters pinned by a test seam
fails with a duplicate edge id without the fix. Both full-route and both
closure-repair syncs are checked to run through the helper.
gbuf_raise_next_id_never_lowers covers the new buffer call.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
parse_match_pattern allocated the pattern's node and relationship arrays
and returned on a parse error without freeing them. Its callers count a
pattern only after it parsed, so cbm_query_free never saw it: every query
whose pattern failed partway leaked, including every named path
(`MATCH p = (...)`, not supported, fails at the first node). The element
whose parse failed leaked too (its variable, label or properties), and
parse_props freed only the array, not the entries it had already parsed.
The fuzz job found it on its first run, from a checked-in seed:
`MATCH p = (a:Class)-[:INHERITS*1..3]->(b:Class) RETURN a.name, length(p)`
(LeakSanitizer: 288 bytes in 2 allocations).
parse_match_pattern now frees and zeroes the pattern on every failure,
counting the failed element (parse_node and parse_rel zero it first, so
only what they allocated is freed). parse_props frees its parsed entries
on every failure through one helper.
New test cypher_parse_error_frees_partial_pattern drives five failure
points: a named path, a property list after one entry, a cut-off
relationship, a missing label after a relationship, and a second MATCH
clause's properties.
Checked with the fuzz binary (clang 18 ASan/LSan, arm64 container) on the
five queries: before this change the first four leak (288, 306, 339 and
348 bytes); after it all five are clean, and scripts/fuzz.sh all --runs
20000 --seed 1 passes for extract, cypher and config. The fifth goes
through the same parse_props failure as the second. macOS: the cypher
suite passes, 217 tests.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The cbm-fuzz rule links through $(CC), the C driver. On Linux, clang's C
driver links UBSan's C runtime but leaves out its C++ part
(ubsan_standalone_cxx), while the C++ objects (preprocessor.cpp and the
vendored simplecpp) carry -fsanitize=vptr checks. The link failed with
undefined references to __ubsan_vptr_type_cache and
__ubsan_handle_dynamic_type_cache_miss, so the fuzz job has never built in
CI since it was added (dry runs 37212102899 and 38186491924).
-fsanitize-link-c++-runtime on the link line pulls that runtime in.
Checked in an ubuntu:24.04 arm64 container with CI's packages (clang 18,
llvm, libclang-rt-dev):
- a minimal reproduction (a C++ virtual call under -fsanitize=undefined,
linked through clang) fails with exactly those two symbols without the
flag, and links and runs with it;
- `make fuzz` links cbm-fuzz (lsp_all.o was compiled with -g0 added: the
12.5 GB container ran out of memory on it with full debug info, which
CI's runner does not);
- macOS Homebrew clang accepts the flag under -Werror.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/memwaste.sh set no memory budget, so the daemon derived one from
the machine's RAM: 1792 MB on the 7 GB macos-14 runner, about 4 GB on the
16 GB Linux and Windows runners. The event layer itself holds 0.5-1 GB, and
on macOS the index stopped with over_memory_budget before it measured
anything, in both full-mode runs since the lane was added (dry runs
37212102899 and 38186491924, peaks 3449 and 3111 MB at the stop).
The script now passes CBM_MEM_BUDGET_MB=4096, what the larger runners
already derived, unless the caller sets one.
Checked on macOS arm64 with CBM_WORKERS=3, the macos-14 runner's core
count:
- budget 1792: over_memory_budget in parallel_extract (rss 3904 MB), the
CI failure;
- budget 4096: the index completes and the report is written;
- the edited script with no budget in the environment: 4096 is applied,
rc 0.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
GCC 11's arm_neon.h declares the SHA2 intrinsics under
`#pragma GCC target ("+nothing+crypto")`. sha256_blocks_arm was marked
`target("+sha2")`, a narrower target, so GCC 11 refused to inline
vsha256hq_u32 and friends ("target specific option mismatch") and the
build stopped. GCC 13 accepts "+sha2", which is why PR CI (ubuntu-24.04-arm)
stayed green; the dry run's ubuntu-22.04-arm leg (GCC 11) failed on all
three shards (run 38186491924).
GCC now gets "+crypto", which implies "+sha2" on every GCC. The function
still uses only SHA2 instructions, and the runtime HWCAP check gates it.
clang keeps "sha2".
Checked in arm64 containers:
- sha256.c with CI's ASan/UBSan flags compiles on GCC 11, 12, 13 and 14;
GCC 11 with "+sha2" fails with the CI errors;
- a known-answer harness ("abc", "", a 1 MiB pattern) matches hashlib on
the arm-sha2 backend and the forced portable backend, GCC 11 and 14,
-O0 and -O2.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Normalize Windows cache paths when reporting runtime settings and resolve
Git executables before spawning history queries. Keep executable lookup
results per thread and cover the ownership contract deterministically.
Support Git 2.43 while denying history-query network transports, and close
the activity database on daemon startup failure so Windows can clean up.
Validation: 463 native macOS tests, 421 Windows tests, 44 MSan UI tests,
and focused canonical lint passed. Production-only revert checks confirm
the repaired failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Dockerfile-variant check read the character after "Dockerfile" /
"Containerfile" before it knew the name held that base, so every name
shorter than the base was read past its end. GCC's AddressSanitizer
caught it once the doc-link layer began asking the language of short
code file names (a.go): 27 to 51 suites aborted per Linux shard. The
base is compared first now.
The test passes each short name in a heap block of exactly its size;
under GCC ASan it fails without the fix (heap-buffer-overflow in
lang_dockerfile_variant).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
winbase.h defines IGNORE and minwindef.h defines IN, so the Windows
build failed on the Dockerfile-variant suffix in language.c and the
Image reference edge types in pass_docker.c. Renamed to DOCKERIGNORE
and REF_TYPES.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
trace_path mode=run walks what a Service or workload runs: traffic to
the workload (INFRA_MAPS), the Image its container runs (USAGE), the
Dockerfile that builds it and what that builds on (DEPENDS_ON), the
program its ENTRYPOINT starts (SPAWNS) and the environment each
container is given (CONFIGURES) -- Service -> Deployment -> Image ->
Dockerfile -> main in one call.
The README lists the Image label, the container edges (a Dockerfile's
bases and entrypoint, image references, the build vias, compose
services, container environment) and the new mode.
The complexity guard gets the container layer: a fixture with
multi-stage Dockerfiles, an ENTRYPOINT script and a compose project
building two components of one repository, its nodes and edges gated
linear per copy, and the Image edges the component split examines
gated linear too -- every copy's builds join one repository Image by
design, so that join must stay linear.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An image is its repository, tag and digest on the edges -- until one
repository is built from several Dockerfiles, one per component tag:
opentelemetry-demo builds every service as
ghcr.io/open-telemetry/demo:<version>-<service>, so its one Image was
built from 25 Dockerfiles and a walk from any service through the image
reached all of them.
Such a repository is now split by component: its build tags without
their common prefix / suffix (cut at separators -- latest-frontend,
latest-cart -> frontend, cart). A component built from exactly one
Dockerfile, with no unknown part, becomes an Image of its own,
`<repo>:*-<component>` {repository, component, tag_pattern}; its build
edges leave from it, and every reference whose tag names a component (a
K8s container, a compose service, a FROM) uses it -- whatever its
version, the longest component a tag holds as whole parts, none on a
tie. Everything else stays on the repository Image, which goes when no
edge is left (now on the full route too). Recomputed whole each run.
A closure repair cannot re-point the references of files it does not
repair, so when the run's component Images differ from the previous
ones it hands over to a full rebuild.
Edges are deleted by pointer (cbm_gbuf_delete_edges): after the
parallel resolve merge, cbm_gbuf_set_next_id can hand an edge id out
twice, and a delete by id took unrelated edges with it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Five suites (grammar_regression grammar_labels pipeline lang_contract
grammar_probe_e) were excluded from the MSan lane for "stack-overflow
under instrumentation". That was never deep recursion. MSan aborted while
REPORTING real uninitialized reads:
- every abort was on the main thread. MSan's "T<n>" is the OS thread id
and equalled the pid in all five x86-64 aborts (CI job 91868259558);
the old block read it as a worker-thread ordinal, so the worker-stack
knob, MSAN_ORIGINS and CBM_WORKERS=1 could never change anything;
- with the default slow unwinder (the image's MSan-instrumented
libunwind) a report exhausted the main thread's 256 MiB stack, and the
overflow handler then failed with "nested bug in the same thread";
- with fast_unwind_on_fatal=1 the same test prints the
use-of-uninitialized-value report instead (measured on one tree and
test: slow unwind aborts at the stack limit, fast unwind reports
wit/scanner.c:13).
The two reads are fixed in the previous commits of this series, so:
- MSAN_OPTIONS defaults to fast_unwind_on_fatal=1 (the build already
keeps frame pointers);
- MSAN_EXCLUDE is down to cli, whose install-path failure is a separate,
still-undiagnosed cause;
- the known-red block, the header (it still claimed CI runs without
exclusions) and the MSAN_ORIGINS and stack comments say what is true;
- the partial-lane warning and the exclude-name typo check lived in a
branch behind `elif true` that never ran. They now run in the
per-suite path, the check as a case match instead of a grep -q pipe
that pipefail can misreport;
- the comments in scripts/ci/msan-lane.sh and above the test-msan job
in _test.yml no longer say the five suites overflow.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The test filled 3675 commits' files and count but never their timestamp,
and cbm_compute_change_coupling reads each commit's timestamp to track a
pair's last co-change (pass_githistory.c:304). MemorySanitizer reports
the read; the stack array is now zero-initialized like every other
githistory test's commits. Production is unaffected: its commits come
from calloc and a zeroed parse state.
This was the read behind the pipeline suite's MSan "stack overflow"
(the x86-64 CI log of 2026-08-04 aborted in this test).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
tree_sitter_wit_external_scanner_create() is declared to return void *
but had an empty body. tree-sitter's runtime stores its result as the
scanner payload (parser.c ts_parser__external_scanner_create), so every
WIT parse used an indeterminate value, which C11 6.9.1p12 makes
undefined. The scanner never reads its payload, so ordinary builds were
unaffected, and the vendored build's -w hid -Wreturn-type.
MemorySanitizer reports it at wit/scanner.c:13 from grammar_labels,
grammar_probe_e, grammar_regression and lang_contract. Those four suites
were excluded from the MSan lane because the report itself crashed and
posed as a stack overflow (fixed in the msan.sh commit of this series).
Recorded in MANIFEST.md's local-patch table. Upstream
bytecodealliance/tree-sitter-wit still has the empty body at
059bfdd1af72, the last change to src/scanner.c.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel