- Separate loss from transient failure. Touch helpers now report OK /
lost / transient (version_cohort: OK/UNSAFE/IO; ipc: 1/0/-1). Only a
proven unlink or replacement stops the daemon. A refresh failure on a
still-valid file (EROFS, EPERM, EMFILE on the marker reopen) logs
daemon.coordination_touch_failed once, retries every tick, and logs
daemon.coordination_touch_recovered when it clears.
- Cohort and claim locks now prove the path still names the held inode
(private_file_revalidate against the stored directory + name), not
just st_nlink, so rename-away, hard-link+unlink and a replaced
directory are caught like unlink.
- The runtime service's self-owned participant guard is refreshed by
the listener heartbeat too.
- Tests: rename-away detection; cohort lease/claim touch (lease with
only lifetime held is OK, each unlinked file reported independently);
IPC listener touch now checks timestamps are actually refreshed and
covers the participant guard and identity-marker replacement.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
macOS com.apple.tmp_cleaner deletes /tmp files whose atime, mtime and
ctime are all older than three days. The daemon's long-held lock files
are empty and never written, so under a daemon that lives past three
days the cleaner unlinks them while they are held. The next participant
then creates and locks a fresh inode: forked index workers see the
cohort daemon marker as free, read the live daemon as uncoordinated and
exit 1 on every reindex, and a lost IPC lifetime lock or cohort lifetime
file stops protecting the generation at all. A lost socket identity
marker additionally makes listener close leave the socket pair behind.
The host lifetime loop now runs a heartbeat every 60 s over every file
the generation holds (cohort daemon claim and lease, participant guard,
IPC lifetime reservation, identity marker). Each file is refreshed with
futimens through its held descriptor, only after proving the path still
names that descriptor, so the cleaner never selects it. If any file is
already lost, the daemon logs daemon.lifetime_end
reason=coordination_file_lost and stops cleanly; the next client starts
a coordinated generation. Re-claiming in place would race that client.
Windows needs no refresh (no age-based cleaner; held lock handles deny
deletion), so its entry points only validate the handles.
Closes#2178
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
main fails scripts/test.sh at Step 5f on every platform:
=== Step 5f: worker request-scope regression ===
worker did not index the admitted repository
{"content":[{"type":"text","text":"missing or incomplete trusted
worker policy"}], ... "isError":true}
Second collision from the #1723 merge (92abefa3), and the same kind as
the first. #1723 made an index worker refuse any request that does not
carry `_cbm_index_policy`: a worker never resolves the resource policy
from config or environment, the supervisor does and sends it along.
tests/test_worker_session_scope.sh, added on 2026-09-06 by a08b9ecb,
plays the supervisor by hand -- it runs `cli --index-worker
index_repository '{"repo_path":...,"mode":"fast"}'` directly -- and
#1723's branch never contained that script, so its CI never ran the two
together. The merge was textually clean and the result was wrong. It
stayed hidden behind the lint failure the previous commit fixes, because
the test stage only runs once lint is green.
The worker is right and the script was incomplete: it now sends what the
supervisor sends, both limits off, which is the default
(`"index_max_files":"off","index_max_source_mb":"off"`, the same object
tests/test_index_policy.c uses). The fail-closed case carries it too, so
that a request without repo_path is still refused for its missing scope
("request workspace scope invalid") and not, by accident, for a missing
policy. No assertion changed.
RED, production binary built from this branch, script unmodified:
worker did not index the admitted repository
... "missing or incomplete trusted worker policy" ... rc=1
GREEN:
ok: index worker is scoped to the admitted request, not the daemon
environment rc=0
On the failed CI run of this PR, 12 of the 13 red test jobs carry exactly
that signature and nothing else. The 13th, test-windows-guards, is
test_daemon_stability.py section_cold_storm ("secure CLI coordination
could not be created (endpoint)", a racing cold start) -- a known
nondeterministic guard, unrelated to either commit here. Step 6, which
Step 5f's failure kept CI from reaching, passes locally:
security-strings allow-list 4 passed, destructive-ordering contract PASS.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
main fails the memory-core linter:
src/mcp/mcp.c: grew by 3 (814 -> 817)
PR #1723 (merged as 92abefa3) added a new early-return path to
handle_index_repository and, following the function's convention, freed
its three argument strings there with raw free(). Its CI lint was green
on 2026-09-09. On 2026-09-19 the memwaste merge cut this file's raw sites
and ratcheted the baseline down to 814, so by the time #1723 landed its
green was measured against a rule that no longer existed. The linter is
right and the merge-result check that let it through was incomplete: it
built and ran the suites and did not run the linter. It does now.
The fix is not cbm_free. The three strings come from
cbm_mcp_get_string_arg and resolved_repo_path_from_project_arg and are
raw heap memory, released raw by every other path in the function;
routing three of them through the core would free raw memory through the
accounting layer. The actual defect is that the function had 41 raw
free() calls of the same three arguments across a dozen early returns,
and #1723 added one more block in the same shape.
One helper, index_args_free(repo_path, mode_str, name_override), now
replaces twelve of those blocks. free(NULL) is a no-op, so the one path
where repo_path is still NULL passes NULL. On the cross-repo branch
mode_str used to be freed before handle_cross_repo_mode and the other
two after it; mode_str is neither passed to nor read by that call, so
releasing all three after it is not observable. Five sites on the
normal-completion path are left as they were: they free the strings at
different points as ownership is handed to the pipeline, and collapsing
them would change that order for no gain.
Raw sites in this file: 817 -> 785. The baseline follows the improvement,
814 -> 785, as the linter asks; it can only tighten.
Build: clean, -Werror, ASan/UBSan. mcp cli daemon_application:
692 passed, 4 skipped, 0 failed -- index_repository_honors_allowed_root,
which runs through a replaced early return, included.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#1787 landed correct production code and a test that cannot pass on today's
main. The test is the only thing wrong, and only in how it asks.
check_index_coverage answers in the compact table by default, where a field
name never appears:
main.go main.go coverage_unavailable metadata_match read_source_and_reindex []
The test asserts on the JSON field spelling:
strstr(response, "\"freshness\":\"metadata_match\"")
Both halves are right; they were simply written against different response
formats. The branch was cut on 2026-09-02, before that default changed, and
was never rebased -- its CI green is from a main that no longer exists. The
neighbouring coverage tests already pass "format":"json" for exactly this
reason; this one predates them.
So the request gains format=json and the assertions are untouched. Asserting
the field rather than a bare strstr for "metadata_match" is what keeps the
check exact: the value also appears in the table's recommended_action column
and would be matched by any other path's row.
Verified: build/c/test-runner mcp -> 318 passed, 0 failed, 4 skipped. Before
this change the same suite reported 317 passed, 1 failed on identical
production code.
What this is NOT: a production defect. coverage_path_freshness on main uses
cbm_path_info_utf8, exactly as #1787 intended, and the Windows second-vs-
nanosecond truncation it fixes is gone. I checked that by reading origin/main
rather than the local checkout, which was stale at 59a05eb1 and still carried
the old helper -- that misreading briefly convinced me the merge had dropped
the fix, and it had not.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dependabot split one upgrade across three PRs -- init (#2185), analyze
(#2187) and upload-sarif (#2186) -- but init and analyze share a
configuration file and CodeQL refuses to read one written by a different
version:
Loaded a configuration file for version '4.38.0',
but running version '4.37.9'
So #2185 and #2187 each go red on their own and neither can reach a green
required check, which means they cannot be merged one after the other
either: whichever lands first leaves the pair mismatched. The action also
warns about it directly -- "Not all workflow steps that use
github/codeql-action actions use the same version".
Both refs in codeql.yml therefore move in one commit, to the same pinned
SHA b96794f015dfd88f77b49b1c93e0fa7110f94c63 that both bot PRs target.
scorecard.yml's upload-sarif is deliberately not touched here: it runs in
a separate workflow that shares no configuration with init/analyze, which
is why #2186 is green on its own and can land as its own PR.
Supersedes #2185 and #2187.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
TSan failed on all three platforms in PR CI. Both races are in code this branch
added, and both were invisible to the local ladder because it has no TSan leg —
that lane is CI-only, so "macOS and Linux green" said nothing about either.
1. system_info.c: the cached system info
cbm_system_info() published its cache with a plain flag:
if (!info_cached) { cached_info = detect_...(); info_cached = 1; }
Fine while only the main thread asked. The memory-relief work started calling
cbm_mem_system_under_pressure() from the per-file extract loop, so several
workers now race on the flag AND on the struct — and a struct assignment is
not atomic, so a reader could see half of one. Reported at
system_info.c:284 against :274, from extract_worker, on macOS, ubuntu-arm and
ubuntu-latest alike.
Now a three-state atomic: whichever thread wins the compare-exchange is the
only writer of cached_info, a loser returns the copy it detected itself (the
same answer — detection only reads the OS), and later readers see the
published struct through the acquire/release pair. No mutex, so nothing has
to be initialised before first use.
2. pass_parallel.c: the spill pointer
pp_spill_active() tested ec->pctx->spill BEFORE its atomic spill_mode gate,
so it read an 8-byte pointer with no synchronisation while pp_spill_enter was
publishing that pointer under spill_mu. pp_spill_enter assigns the pointer
and only then release-stores spill_mode, so gating on the atomic first makes
the pointer read ordered by that release. Short-circuit order is load bearing
in that expression now, and the comment says so.
Found by running the TSan lane locally with halt_on_error=0 rather than CI's
halt_on_error=1: the first setting stops at the first race, so each fix would
have revealed the next one a CI round trip later. Enumerating instead showed
the second race was the last one in that suite.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The verdict guard added a few commits ago demanded `=== All tests passed ===`
from every run. Only the full leg prints it. `scripts/test.sh --suites` stops
after the suites it was named and says nothing more, because it has no end to
reach — so every `win.sh test <suites>` run came back red while passing:
=== test.sh: ITERATION mode — suites: artifact (incremental build) ===
=== artifact ===
23 passed, 1 skipped
GUARD: the log has a summary but no completion marker (runner rc=0)
That is the same defect as the one the guard exists to prevent, pointing the
other way. A guard against false greens is not allowed to invent false reds,
and this one broke the documented iteration workflow on its first real use.
vm_verdict now takes a mode. The full leg is unchanged: failures, then the
marker, then the note about a lost exit status. A named subset is judged on its
summaries and, since it has no marker to lean on, on its exit status — it is a
developer tool rather than a gate, so there it obeys rc instead of overriding
it.
The contract test grew four cases, and they are self-proving rather than
inert: the SAME markerless log must be green in iteration mode and red for the
full leg, so a build that ignored the mode would fail them. That gap is why the
original eight passed while the shipped path was broken — they only ever fed it
full-leg logs, which is the case I was thinking about rather than the case I
shipped.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
PR CI's lint job failed on src/foundation/mem_override_libc.c:143-144 while the
local lint lane passed on the same file. The two run different formatters: CI
installs LLVM 20, the local lane uses the Homebrew build, which is now 22. A
wrapped attribute is one of the few things they lay out differently, so the
wrapped form was stable locally and a violation in CI.
__attribute__((no_builtin("memcpy", "memmove",
"memset"))) static void *slow_move(...)
Written on one line the declaration is 117 columns, over the 100-column limit,
so it has to wrap and the two versions disagree about how. Spelling the
attribute as a macro brings the declaration to 73 columns, which leaves no
wrapping decision for either version to have an opinion about. The three
siblings below it already fit inline and are left alone.
Also repins scripts/vendored-checksums.txt for MANIFEST.md itself: the security
job's vendored-integrity layer checksums the manifest too, so editing it to
record the properties scanner patch invalidated its own entry. Produced by
scripts/security-vendored.sh --update; the one changed line is the manifest
hash, which is also the check that the earlier manual scanner.c repin was right.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Indexing the same C# corpus twice on the same machine produced 5,794,336 edges
and then 5,794,328. Every other edge type matched exactly, every semantic
checksum matched (447,064 functions, 81,534 tokens, identical vec/idf/toks),
and the difference was ten WRITES edges out of one generated file.
Three places in extraction decided graph content from a clock:
* the unified AST walk stopped on a thread-CPU deadline;
* a file whose parse spent more than half its budget was refused the
per-file and cross-file LSP walks;
* so was a file whose parse plus walk exceeded the budget.
The first is the one that bit. Any file large enough to reach the deadline is
cut at whatever node the clock lands on, and five generated .NET JIT tests
reached it on every run: hugeSimpleExpr1.cs stopped at node 2,660,352 in one
run and 2,574,336 in the next, out of 2,774,794. That file's tail is field
assignments, so the two runs disagreed by exactly ten WRITES edges. The graph
was a function of how fast the machine happened to be running, and nothing said
so: result->walk_truncated was set and no code anywhere read it.
The deadline existed because those files cost 18-48 us per visited node, and
that turns out to be one fixable thing. csharp_callable_value_site() was the
only language branch in call_reference_candidate_site() still climbing with
ts_node_parent, which tree-sitter answers by descending from the ROOT, scanning
children at every level; every other language already climbs on the walk
cursor. On expressions megabytes deep that was ~85% of the entire unified walk
(sampled: stamp_usage_site -> ts_node_parent -> ts_node_child_with_descendant).
It now takes the cursor and the parent the caller already resolved, and
is_direct_argument_value_cursor() is split so a caller that has stepped the
cursor onto the parent can continue from there rather than paying for it twice.
With that fixed, the budgets are not needed to make indexing finish, so they
are gone rather than made deterministic. A cutoff in nodes would at least be
reproducible, but it still answers "what is in this repo?" with "as much as we
felt like reading". CBM_WALK_MAX_NODES reinstates one for an operator who wants
it; the default is no budget.
Removing the two LSP-skip rules was checked against the file and the path they
were added for (a 23 MB single-expression JIT test that once cost 354 s in the
walk and then crashed the cross-file resolve): the whole 12k-file C# corpus
twice with cross-file resolve, that file alone, and synthetic C# nested 10,000
levels deep — no crash, and past roughly that depth tree-sitter stops producing
a usable tree, so a tree deep enough to endanger a recursive walk never reaches
one.
C# corpus, before -> after:
edges 5,794,318 / 5,794,328 / 5,794,336 -> 5,794,873 every run
truncated 5 files, unreported -> 0
time ~115 s -> 97-101 s
Deterministic, about 550 edges more complete (the tails nobody was walking),
and faster than the build that was cutting them off. All ten bench corpora now
return identical nodes and edges across two runs.
A truncated walk and a skipped LSP walk are now reported in the coverage report
with how far the walk got ("1673216/4133946 nodes walked"), like a partial
parse. A file the walk abandoned halfway used to be reported as fully indexed,
which is a graph quietly lying about its coverage.
Tests: extract_csharp_argument_values_use_the_walk_cursor asserts no occurrence
falls back to the root-descending parent lookup — 128 fallbacks without the fix
and 0 with it. Its fixture parenthesises the argument on purpose: a bare
Sink(Target) needs no climb at all, so it cannot tell the two walks apart and
passed either way while the bug was in place.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Indexing a repo with .properties files in it produced a different graph from
one run to the next. Every few runs one small file - 253 bytes, valid, the same
file that had indexed fine a moment earlier - vanished from the graph, and the
coverage tree put raw-path File/Folder nodes where it had been. On the kotlin
corpus that showed up as node counts wandering between 33,486 and 33,494.
The vendored tree-sitter properties grammar keeps its external scanner's entire
state in a file-scope static:
static bool reached_eof = false;
bool ..._external_scanner_scan(void *payload, TSLexer *lexer, const bool *valid) {
lexer->result_symbol = FAKE_EOL;
return reached_eof = !reached_eof && valid[FAKE_EOL] && lexer->eof(lexer);
}
That flag is how the scanner refuses to emit a second end-of-input FAKE_EOL,
and the grammar needs exactly one to close a file. One static is one object for
the whole process, so with several index workers parsing .properties files at
once, whichever thread reached EOF first set the flag and the next thread's
`!reached_eof` was false: its parse never received the FAKE_EOL it was waiting
for and sat at end-of-input asking for a token that would never arrive. The
per-file parse budget stopped it five seconds later and the file was dropped.
Measured on a 253-byte file: 1,060,630 progress callbacks, ~106 M parse
operations, byte offset pinned at EOF, has_error false - the parser was not
recovering from an error, it was waiting.
The state now lives in the payload from ..._external_scanner_create(), which is
what tree-sitter's create/destroy contract is for. The serialize/deserialize
wire format is unchanged: the state is still the returned length.
Proof, interleaved under identical load so both binaries meet the same machine,
40 runs each: unpatched 6 stalls, patched 0. Under load the kotlin corpus now
returns 33494/203579 ten times out of ten and java 693129/5523595 three times
out of three; both used to wander.
All 103 vendored scanners were audited for the same shape. This was the only
one that WRITES a process-wide static. haskell's debug_parse_metadata and
purescript's res_cont/res_fail are read-only - constants missing a const - and
are left alone.
Upstream 6310671b24d4 still carries the static, so MANIFEST.md records this in
the local-patch table and scripts/vendored-checksums.txt is repinned.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A grammar unity object, grammar_<lang>.c, #includes its own vendored parser.c
and, where the grammar has one, scanner.c. Those files were prerequisites of
nothing. Editing a vendored scanner therefore left the object untouched and the
edit silently absent from the binary.
This is not hypothetical: the properties scanner fix in the previous commit
appeared not to work at all. An A/B said 3 stalls against 2 - no effect - and
the reason was that prod_grammar_properties.o predated the edit by five minutes
and `nm` still showed the old `_reached_eof` symbol sitting in it. The measured
"no effect" was a stale object, not a bad fix.
The build already treats its other two unity objects this way and says why:
LSP_UNITY_DEPS and TS_RUNTIME_DEPS both list the sources and headers they pull
in, because this Makefile deliberately does not use compiler auto-dependency
(-MMD) generation. Grammars were the gap, and MANIFEST.md's "Local source
patches" table means patching a vendored scanner is an expected thing to do -
so the trap was reachable by design.
Secondary expansion is what lets one pattern rule name each grammar's own
directory ($$* is the stem, grammar_properties -> properties). Applied to all
three grammar object rules: production, test and tsan. Non-grammar objects
match no directory and gain no prerequisite.
Verified by touching internal/cbm/vendored/grammars/properties/scanner.c and
watching exactly one object rebuild.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On 2026-09-18 a Windows leg printed "7925 passed, 0 failed" and
"=== All tests passed ===" and still exited 1, while the identical work run
through win.sh's other entry exited 0. The status travels
ssh -> cmd.exe -> msys2_shell.cmd and that chain loses it.
A channel that can turn 0 into 1 can turn 1 into 0, and that direction is a
false green: a red Windows leg reported as passing, which is the one thing the
local ladder exists to prevent. So the two channels are now split by what each
can actually know. The log decides the test outcome — it is written by the
runner on the VM and cannot be mangled in transit — and rc only decides what
the log cannot see. Green needs zero reported failures AND the completion
marker scripts/test.sh prints last, so a leg that stopped early has a summary
but no marker and is red. A lost exit status is reported, not obeyed.
The logic is a function, vm_verdict(), reachable as `--verdict <log> <rc>`, so
tests/test_vm_verdict_contract.sh drives the REAL code with synthetic logs
instead of a copy that could drift from it: lost exit status, failures under
rc=0, a marker that must not launder failures, failures in a later suite,
partial runs, and no summary at all (which keeps its own exit code 90, so
"never ran" stays distinguishable from "ran and failed"). Eight cases, wired
into scripts/test.sh as step 0y.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
inject_calls_pattern_tokens read a node's CALLS edges straight out of the
graph buffer, which holds them in insertion order — and insertion order is the
order parallel workers happened to merge. It then injected the callee names in
that order and truncated at max_tokens, so which callees survived depended on
worker scheduling. Co-occurrence enrichment reads a window around each token
position, so the surviving vectors differed between runs while the document
frequencies (and therefore idf) stayed identical, which is why this showed up
as a handful of SEMANTICALLY_RELATED edges flickering and nothing else.
It now collects the neighbours through collect_sorted_call_neighbors(), the
same canonical ordering the rest of the pass already uses.
Three runs of the Go corpus now agree exactly — 334535 nodes / 1795409 edges
with vec=0e45744ff1042b51 idf=7e994a1d864a7b07 toks=2518a08a0b6af0c7 — and so
do php, kotlin and rust. The edge counts move slightly against the previous
build (php 57182 -> 57198, kotlin +17, rust +3): sorting changes which tokens
survive the truncation, so the semantic edges change with it. That is a
deliberate output change, not a regression.
Also adds the checksums that found it, all off unless their env var is set:
CBM_SEM_STATE_DUMP writes one hash per scoring input (codes, tf-idf, struct
profile, minhash) and one per corpus (ri vectors, idf, and the token SEQUENCE,
which is what differed here), and CBM_SEM_PAIR_DUMP writes every candidate pair
that passed the threshold with its score, in canonical order. This subsystem
has now produced three order-dependence bugs; a checksum at the seam is what
turns "the graph differs" into a localised answer in minutes, and it costs
nothing when the variable is unset.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The memory-core linter is a ratchet: raw malloc/calloc/realloc/free/strdup
sites may not grow, because allocations belong on one accounted path
(src/foundation/mem_core.h) rather than spread across hundreds of call
sites. This branch's own work had added 35 raw sites across nine files --
the parallel k8s prep, the configlink key matching, the worker pool's
counted path, the Go shared stdlib registry, the spill namespace copy,
the route service array and the namespace-name arrays.
All of them now use cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/
cbm_mem_strdup. One was a latent bug rather than a style point:
pass_k8s freed a buffer that k8s_read_file had allocated with raw malloc,
so converting only the free would have mismatched the allocator -- the
read function and every free of its result are converted together.
The Windows free-space path widens its argument through the memory core
instead of cbm_utf8_to_wide(), which allocates with raw malloc.
Two files improved enough to tighten their baselines, which the ratchet
requires so the gain cannot be quietly given back: sqlite_writer.c 92->74
(the index-cell arena), pass_k8s.c 13->6 and registry.c 19->17.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two indexes of the same corpus with the same binary produced different
graphs. On the go corpus the difference was 9 nodes and 22 edges, all of
it Route nodes and their HANDLES edges.
Two order-dependent choices, both reading arrays that are in graph-buffer
insertion order -- i.e. in whatever order parallel workers happened to
merge:
- create_grpc_routes() collected proto services into a fixed
services[64] and stopped there, so WHICH services got Routes depended
on scheduling. It was also a silent quality cut: every service past
the 64th got no Routes at all, and kubernetes has far more than 64.
All services are collected now, which is why the go corpus gains ~48
Route nodes that were previously dropped.
- match_one_infra_route() linked an infra Route to the FIRST matching
handler in array order. It now picks the best candidate under a total
order on qualified name, so the choice cannot depend on scheduling.
Two runs of the go corpus after this: node sets identical, edge sets
differing by a single SEMANTICALLY_RELATED edge -- down from 9 nodes and
22 edges. Node counts are identical across four consecutive runs.
The remaining flicker is one near-threshold semantic edge. Ruled out by
reading rather than guessing: funcs[] is already canonicalised by
qualified name, the CALLS inputs are byte-identical between runs, LSH
bucket fill is sequential, candidate truncation follows from that,
admission is applied sequentially in canonical order, corpus batches are
sequential and co-occurrence accumulates per target in fixed order. It
needs instrumented tracing of the pair that flips, and is recorded rather
than claimed fixed.
The new test indexes 70 proto services -- more than the old cap -- and
asserts every one gets its Route, so the cap cannot come back whatever
order the workers produce.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The same repository indexed with results parked on disk produced a
DIFFERENT graph than one indexed in memory. Measured on the php corpus,
same binary, reproducible 3/3 each way: 16,561 nodes / 57,182 edges in
memory against 16,562 / 59,379 while spilling. Not jitter, and not
one-directional -- 1,458 CALLS pairs appeared only under spill and 840
only without it. Present on main, so it ships in v0.11.0 today, and it
bites exactly the large-repo users who need spilling.
Cause: pass_parallel built the namespace map -- what `use` / `using` /
package imports resolve against -- from the in-memory result cache, where
a PARKED result is NULL. Every spilled file therefore contributed no
namespace, and a file missing from that map does not fail to resolve, it
resolves through the looser fallback. That is why edge counts moved in
both directions instead of simply dropping.
The fix keeps each parked file's namespace IN MEMORY while its result is
on disk (one short string per file, nothing against the result it
replaces), and builds the map from names so the memory and disk paths
agree. Non-spill runs take the identical code path and are unchanged.
Across ten languages and four memory settings (default, 2 GB, 700 MB,
forced spill), php/java/csharp/kotlin now produce identical graphs where
they differed by up to 40,341 edges before.
The existing spill-parity contract in test_parallel passed throughout,
because its fixture is Go+Java and the compact parity harness omits the
pass that builds the map -- neither reaches the namespace path. The new
test runs the real pipeline over more than MIN_FILES_FOR_PARALLEL
namespaced PHP files, with and without spilling, and fails without this
fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Indexing the linux kernel with NO ceiling set took the machine down: on a
48 GB host the default budget is 50% of TOTAL ram (24 GB), the run used
essentially all of it (24.5 GB peak RSS), and with a 12 GiB VM already
running there was nothing left. The budget never fired -- it read 22 GB
charged against 24 GB -- while the host was out of memory.
Total RAM is the CEILING; what is free right now is the CONSTRAINT.
- cbm_mem_clamp_to_available() reduces a fraction-derived budget to fit
what is actually free, reserving a quarter of it (held between 1 and
8 GB) for the OS and everything else, never below a 512 MB floor. An
explicit CBM_MEM_BUDGET_MB still wins -- it is the user's deliberate
choice -- but is warned about when it exceeds free memory, because
the failure it buys reads like a product bug rather than a setting.
- cbm_mem_should_relieve() = our budget exceeded OR the machine short.
Deliberately separate from cbm_mem_over_budget(): system pressure
must RELIEVE (spill, reclaim), never abort, since the spike may not
even be ours. The extraction loop now enters spill on real scarcity,
and the dump phase -- where the kernel run actually died, with our
charge comfortably under budget -- keys its free-heavy decision on it
too. cbm_mem_system_under_pressure() existed and had no callers.
Same scenario after: budget clamped to 18,947 MB, peak RSS 19.5 GB,
status indexed in 185.7 s, machine fine.
Spill also gets limits and cleanup, because relief that fills the disk or
leaves gigabytes behind is not relief:
- a free-space preflight (never start spilling below 10 GB free) and a
ceiling at half of what is free; past it results stay in memory and
back-pressure takes over. cbm_fs_free_bytes() is new (statvfs
f_bavail / GetDiskFreeSpaceExW quota).
- POSIX unlinks each spill file while holding it open, so the space
returns on ANY exit including SIGKILL -- a killed run cannot leave
spill behind because there is no name left to leave it under.
Windows keeps close-path removal plus a sweep where an open file
refuses deletion, which is exactly the liveness test.
- the staging-database sweep now reclaims orphans of EVERY project in
the cache directory, not just the one being indexed. A kernel index
killed by the OOM killer used to park ~15 GB until someone re-indexed
the kernel, and indefinitely if nobody did. The per-stage lock probe
still keeps a live writer's stage, whichever project owns it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Indexing broke on x86-64 Linux: the index worker refused to start and
SIGKILLed its own process group, so every venue reported nothing but
"index worker ended with killed (exit=-1, signal=9)".
glibc places a thread's static TLS block inside the stack allocation it
makes for that thread. This branch's new thread-local caches -- a
512-slot field cache (~24 KB), per-depth tree cursors (~4 KB) and two
parked CBMArenas (~8 KB; an arena carries blocks[256] + block_sizes[256])
-- took static TLS from 47,905 to 93,513 bytes. The 64 KB parent-death
watchdog thread then could not be created at all, and a worker without
process-tree containment correctly refuses to index.
Four layers, because there were four things wrong:
- The big thread-locals move to the heap behind a thread-local POINTER,
allocated once per thread and reused, each with a correct fallback if
allocation fails (the field cache and cursor pool are caches; a miss
is slower, never wrong). Static TLS is back to 52,233 bytes.
- PARENT_WATCHDOG_STACK_SIZE 64 KB -> 256 KB. "The watchdog only polls,
a tiny stack suffices" was true when written and false once TLS
reached ~50 KB, because the TLS block comes out of that same
allocation.
- cbm_thread_create retries with the default stack on EINVAL. A stack
size is a hint about what a thread needs; the platform refusing the
hint is not a reason to fail to create the thread.
- tests/test_thread_stack_tls_contract.sh (Step 5a of scripts/test.sh)
keeps static TLS a quarter clear of the smallest requested stack,
reading that smallest size from the sources so a new small stack
tightens the gate automatically.
Two diagnostics that made this findable at all, and that stay:
smoke-test.sh prints the worker log its failure message names (CI deletes
that sandbox with the job, so the one artifact naming the cause was
unreadable), and worker_containment_unavailable() now reports WHICH of
its conditions failed, with pid/pgrp/ppid.
Verified in the amd64 container, the venue nothing on the local ladder
covers: baseline main indexes 9039 nodes/15067 edges, this branch before
the fix refuses to start the worker, and after it indexes 9039/15067.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/fuzz.sh drives the libFuzzer targets in two shapes from one entry:
a fixed --runs from a fixed --seed, so a CI smoke is a function of the code
and not a lottery (O9), and a time-boxed --max-total-time for exploration.
Checked-in seeds are copied into a working corpus under build/fuzz/, so a
run never rewrites the repository; a crash leaves its reproducer behind and
the script exits non-zero. Wired as a smoke into dry-run.yml and as
exploration into nightly-soak.yml.
The venue-parity contract rightly refused both workflow steps: a venue may
provision or plumb, but only a CANONICAL leg entry may exercise the
product, and neither script was registered. Registering them — rather than
working around the contract — is the fix, since both own their leg end to
end (build, corpus or binary, artifact paths, exit code) exactly as
soak-legs.sh owns the soak. _memwaste.yml joins the policed venue walk for
the same reason, and both scripts join the layer-5 interface probes.
Those probes immediately earned their keep, finding two real defects:
`fuzz.sh --help` exited 2 because the positional target consumed the flag,
and memwaste.sh read a typo'd flag as the repository path — which would
have indexed nothing and reported an empty lane as a clean one.
test_shell_line_endings.sh now degrades where the checkout has no
reachable repository metadata: the Linux container bind-mounts the working
tree, and a git worktree's .git is a pointer to the host repository, so
`git ls-files` cannot run there and the contract failed with "matched no
files — the glob set is broken", killing the leg before it compiled. It is
a repository property, so it stays gating in every venue that has the
metadata (macOS host, the Windows VM's real checkout at C:\cbm with 117
files, every hosted-CI checkout). test_version_metadata_contract.sh already
guards the same way for the same reason; the alternatives considered and
rejected are recorded at the case.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every item here was measured on a real corpus before and after, not
reasoned about. Go corpus, event lane, first run -> now: 245.5 M
allocations -> 134.8 M, 190 GB cumulative -> 29.3 GB, 158 GB never
written -> 9.2 GB, 132 M churn pairs -> 35.7 M, 93 GB of allocator byte
work -> 0.84 GB. Wall clock: Go 40.7 s -> 19.3 s, linux kernel 239.6 s ->
194.6 s, and all ten LSP languages faster with none slower.
Graph output is unchanged, proven rather than assumed: per-class multiset
digests over the full kernel index (8,563,471 nodes identical, 18/20 node
classes and 18/24 edge classes byte-identical, the rest differing only in
the classes a same-binary re-run also perturbs through worker merge order).
The changes, each with its own reason:
arena an exact arena that is appended to later restarts growth at
8 KB instead of the 64 KB default — the cross-file pass adds a
few resolved calls to a compacted result, so the default block
was ~60 KB untouched per result, 0.5 GB of the Go worker peak.
cbm_arena_init_lazy defers the first block for arenas that are
usually never used at all.
scratch per-thread reuse instead of allocate-and-free per unit of
work: result_compact (8 MB kept), the cross-file pass (4 MB),
extract_defs' walk stack, and the sqlite index cells (a 1 MB
arena replacing 12.1 M malloc/free pairs on Go). Walk frames
moving into scratch is why pass_k8s now opens the keep window
explicitly: without it every YAML file opened a 512 KB scratch
on the main thread, +3 GB.
sqlite index cells are written in place — building the record and
copying it into the cell was a second allocation per entry,
16 M record buffers of pure churn on Go.
registry is_test_qn and common_prefix_len in one pass, per-bucket test
flags computed once instead of per candidate comparison, and
the reachability keys arena allocated lazily.
minhash the unique-trigram set is sized from the trigram count and
skips repeats, and LSH queries reuse one seen-set per worker
instead of allocating per query.
parallel lsp_surface rows and the k8s manifest prep now run across
workers (graph writes stay sequential and ordered), the file
node is hoisted out of register_and_link_def, and configlink
prepares dependency imports once with a length prefilter.
go_lsp one sealed stdlib registry shared by all workers, published
by CAS, instead of one per worker.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_mem_charged() trusted mimalloc's committed counter. In a release build
that counter never decreases on purge: prim.c only tracks decommit under
`#if !MI_DEBUG && MI_SECURE<=2`, so after a large spill the budget still
believed memory it had already returned was held.
On the linux kernel corpus with a 15 GB budget that read 19,787 MB charged
against 15,000 allowed — the run spilled and swept repeatedly while the OS
had the memory back. Charging max(OS charge, tracked live bytes) gives
14,379 MB peak charged, sweep down from 14,990 MB to 5,067 MB, and no
budget-pressure warnings.
The reasoning and the measured numbers are at the call site, because the
next reader will otherwise re-derive "the allocator says X" from scratch.
Also adds cbm_mem_allocator_stats_log() behind CBM_MEM_ALLOCATOR_STATS:
mimalloc's own MIMALLOC_SHOW_STATS never prints for the worker, which
exits through _Exit, and mi_stats_merge does not exist in the vendored v3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Answers a question the existing tooling could not: of everything we
allocate, how much is never used?
Lanes, all driven by scripts/memwaste.sh:
event every allocation observed through the memory core — size, age,
churn, and whether the bytes were ever written
access SanitizerCoverage load/store instrumentation, so "allocated and
never touched" is measured rather than inferred
scaling k vs 2k replicas through clang profile counters, to catch
super-linear work per file
padding allocator slack per size class
ratchet the numbers recorded per venue so a regression is visible
The event lane needed a foreign-free counter: on Windows the override sees
frees for blocks it never allocated (they come from the CRT), and counting
those as untracked tripped the soundness gate. mem_events now separates
them (14 daemon / 5 worker on the Go corpus).
Two portability fixes the lanes surfaced, both real:
- the enable decision must not be cached before `environ` exists. Under
-fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our
wrapper before glibc sets up environ, so the lane silently disabled
itself and produced no dump at all on Linux.
- the access map needs 48 bits: arm64 Linux hands out addresses above
2^47, which a 47-bit map dropped on the floor.
mem_profile is retired into this: it measured a strict subset and had its
own sampling story.
Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on
all three platforms report-only, and pr.yml/release.yml carry the wiring.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The XML documents that make up a .NET repository project system were
absent from EXT_TABLE, so .csproj, .props, .targets, .resx, .plist,
.xaml and the app manifests were never indexed and their content was
unreachable through search_code. tree_sitter_xml is already wired up,
so these only needed rows in the table.
Fixes#2229
Signed-off-by: yasmoradi <msynk.dev@gmail.com>
Badge 6768 -> 8050 passing, the macOS full-suite count on v0.11.0's main
(8,050 passed, 10 platform-skipped, 141 suites, run 35003197455); the
build-from-source section states the suite total and suite count.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every packaging surface states the newest v-prefixed tag, as the
version-metadata contract requires: the Chocolatey nuspec and installer
(checksum re-pinned to the released windows-amd64 archive), the Go
installer, the Nix flake, server.json, the npm and PyPI wrappers.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The v0.10.9 release run (34948714902) failed in both Linux portable
builds: the static link found malloc, free, realloc, calloc, memalign,
valloc, pvalloc and posix_memalign defined twice, once by mimalloc's
override and once by libc.a(malloc.o). Nothing in the build flags had
changed since v0.10.8; what changed was one call. cbm_mem_release_to_os
called glibc's malloc_trim(0), and that symbol lives only in malloc.o, so
the reference pulled the whole object into the link next to mimalloc's
definitions. Dynamic builds never noticed: interposition resolves the
duplicates at load time, which is why the whole test matrix was green.
The call was pointless where it broke things: on Linux and Windows the
override routes every malloc to mimalloc, so glibc's heap holds nothing
to trim and mi_collect(true) is the release. The branch and its include
are gone; macOS keeps its pressure-relief call, since the system heap
still serves everything outside the core there.
Verification: the static build (scripts/build.sh CC=gcc CXX=g++
STATIC=1) in the local arm64 container reproduced the CI failure exactly
before the change and links after it, the binary reporting its version;
mem and parallel suites green on the host.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cppcheck cannot evaluate `#if __has_feature(memory_sanitizer)` even
behind `#if defined(__has_feature)` ("undefined function-like macro
invocation", the lint lane on d89ccabe), and foundation/sanitized.h
already exists for exactly that: it defines __has_feature away where the
preprocessor lacks it. The spill store's image annotation now includes
that header and tests the feature directly; the annotation itself is
unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The MSan lane on 9724d903 stopped in the extraction suite's spill
round-trip test: uninitialized bytes in fwrite at offset 4087 of a
6,952-byte region, the parked result's arena block (the stack overflow
that followed is MemorySanitizer's own report path failing on the nested
bug). The park writes a result as an opaque image: the record header, a
struct copy, and the compacted arena block. Both carry padding bytes no
code ever wrote, inside structs and between objects, and MSan tracks that
mark through the compaction's memcpy, so zeroing the destination block
only moved the report (offset 6714 in the local MSan container). The
image is read back whole and never interpreted byte by byte; the padding
content is irrelevant on disk.
Under MSan the header and the block are declared defined right before
the write (__msan_unpoison, compiled in only when the build is
instrumented); every other build compiles the annotation to nothing. The
exact block is also zeroed once when it is created, so the gaps between
objects are deterministic in the spill file.
Verification: extraction, arena and parallel suites under ASan on the
fixed tree; the extraction suite in the local MSan container image
passes 348 / 0 with no sanitizer report (it reproduced the CI report exactly before the fix).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Shane McCarronandClaude Opus 5
Martin Vogel
lorenzozanee
DavidHLP
yasmoradi