mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-06 17:49:47 +00:00
a35671e962f1f2fb4e94deaf55e71cf05f6b3828
544
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
0bed0d372d |
Merge pull request #2327 from DeusData/fix/issue-1736
fix(grammar): accept a lone '&' in JSX strings and text (#1736) |
||
|
Martin Vogel
|
11867dfa74 |
feat(identity): callable identity plumbing, no language enabled (#2061)
Overloads that share a base QN (Java/C#/C++/Kotlin/Swift/Scala/ObjC) merge
into one graph node today, and that node collects the CALLS of every
overload. The agreed cure is one signature-qualified callable identity for
all overloading languages:
callable_qn := base_qn [<tparams>] (params) [cvref]
This change lands the plumbing with every language still at
CBM_CALLABLE_ID_NONE, so the graph is byte-identical; each language is
switched on later in its own change (with its index-format bump).
- internal/cbm/callable_sig.{c,h}: the per-language mode side table
(NONE|TYPED|LABELED_TYPED|LABELED|ARITY; kept out of CBMLangSpec like
cbm_string_dispatch_suffixes, since ~160 positional rows would otherwise
all need the member under -Wmissing-field-initializers), the shared
builder cbm_callable_sig[_mode] for the def and the call-scope QN, and
the inverse cbm_qn_callable_base_len[_named]. Suffix contract: no '.',
'::' or '->'; whitespace-normalized; comments/annotations/attributes
dropped; qualified types keep their last segment; '->' spelled '=>';
variadic '[]' (Java) / '~'; C++ top-level cv dropped, arrays decay,
cvref kept; <= 200 bytes, longer ones keep whole leading entries and end
in '#<fnv1a64 of the full suffix>)'. The builder is iterative (cursor
walk, bounded declarator loop) and self-checks against the inverse.
- The inverse is anchored to the node's bare name wherever the name is
known: real QNs already end in ')' without being callables (a Java field
whose name is its declarator `X = f()`, make `$(VAR)` targets), and only
the anchor makes the inverse exact on them.
- Leaf splitters strip a suffix first (no-op today): registry by-name key
(indexed by the bare name for a suffixed callable), cbm_lsp_bare_segment,
cbm_pipeline_qn_class_method_tail, pxc_last_component, pxc_qn_leaf,
go_index_visit_node, collect_prod_defs, sg_qn_prefix_len.
- New unused fields: CBMDefinition.qn_sig_off, CBMResolvedCall.callee_sig
(relocated by result compaction), CBMRegisteredFunc.sig, CBMLSPDef.sig.
The lsp_surface codec is unchanged (no format bump in this change).
- get_code_snippet and trace_path gain base-QN tiers
(cbm_store_find_nodes_by_qn_base) that find nothing until suffixes exist.
- store: cbm_store_find_nodes_by_qn_suffix built its LIKE pattern in a
512-byte buffer (a long suffix was truncated into a pattern that matched
nothing) and did not escape '_'/'%' (`my_func` also matched `myXfunc`).
The pattern is now heap-built, escaped, and matched with ESCAPE '\'.
Proof: full sorted node+edge+lsp_surface dumps are byte-identical before
and after on 7 corpora (this repo, elasticsearch libs, Exposed, dotnet
System.Linq, System.Collections, coreclr/gc, a C corpus).
Refs #2061
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
cf72f1a657 |
fix(tests): ignore an inherited git repository environment (#2003)
Git exports repository-local variables (GIT_DIR, GIT_INDEX_FILE,
GIT_WORK_TREE, ...) into hooks, and they take precedence over
`git -C <dir>`. Every fixture git command in the test runner therefore
acted on the caller's real repository whenever the suite ran with those
variables set: with GIT_DIR alone, fixture assertions failed and commits,
branches and linked worktrees were written into the caller's clone; with
the full hook set, fixture commits failed and git-dependent tests degraded
to "git not available" SKIPs. The pre-commit hook already clears the
variables (
|
||
|
Martin Vogel
|
16e2732819 |
ci: non-product PRs (T0b) run the contract steps too
Under R1, a T0b push (test-infrastructure/, lint config, graph-ui, memwaste tooling) selected lint and no test leg, so the static contract steps that police exactly those files -- the VM scripts, the compose and run.sh wiring, the venue-parity rules -- ran nowhere. Replayed over September, 13 of the 18 T0b pushes ran no contract step at all. The R1 table now gives T0b lint + contracts, so those PRs get the contracts job (scripts/test.sh --contracts-only) as T0a PRs do. The fixture table's six T0b cases expect it, and the history replay asserts that every push runs the contract steps (the contracts lane or any test leg). Comments that described the job as docs-only are updated. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
d3efe3baec |
ci: a truncated PR file list selects every lane
The pulls/files API lists at most 3000 files and says nothing when it stops, and the lane decision is only as complete as that list: a cut list can drop exactly the file that needed a lane. select-lanes.sh takes --expect-files N (the PR's changed_files; batch rows take "expect_files") and decides full with a file_list_truncated reason when the list has fewer distinct paths than N (a complete list never does: renames only add their old path), when N is at or over the 3000-file cap, or when the list itself reaches the cap. The tier still classifies what was listed. An empty or non-numeric count is a usage error, so a workflow that loses the count fails loudly instead of skipping the check. pr.yml passes github.event.pull_request.changed_files. The selector contract covers short, complete, rename-inflated, at-cap and just-under-cap lists and the github-output and batch channels; the wiring contract requires the count to be passed. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a8d5a8154f |
ci: report-only test-impact shadow job on PRs
A new pr.yml job, test-impact-shadow, predicts from the code graph which C test suites a change can affect and uploads the prediction, so graph-based selection can be judged against what actually failed before anything gates on it. It is continue-on-error and not in ci-ok's needs: it can never block or pass a merge. scripts/ci/test-impact-shadow.sh --fetch-release downloads the latest published release archive for the host plus checksums.txt with the job's read-only token and verifies the SHA-256 (seconds instead of a build, and a PR that breaks the indexer cannot corrupt its own prediction); pr.yml builds the checkout only if that fails. --binary/--out then indexes the checkout into a throwaway HOME/cache with a private CBM_RUNTIME_DIR (short, under /tmp, so the daemon socket fits sun_path), runs detect_changes against the merge commit's base at depth 15 and hands it to predict.py. The graph is only consulted when C sources changed. scripts/test-impact/predict.py ports the replay's mapping: impacted test functions -> registered suites (perf excluded; nested RUN_SUITE children resolve to the registered suite that runs them), changed test files -> whole suite, fixtures -> suites that name them, grammars -> grammar suites, and run-all on the selector's full, a test header, test infrastructure, or a saturated/truncated impact set. It writes prediction.json and a short step-summary table. tests/test_test_impact_predict.sh (Step 0z5) pins the mapping on a synthetic tree; the wiring contract requires the job to stay continue-on-error; the script joins the venue-parity interface probes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
adec281d1a |
ci: run only the selected lanes on PRs; dry run and release unchanged
pr.yml's `changes` job now lists the changed files (renames with their old path too), runs scripts/ci/select-lanes.sh in a credential-less step and outputs tier, full, lanes and the pr-smoke matrix. security, lint and test pass the lanes to their reusable workflows, which take a `lanes` input defaulting to "all": dry-run.yml and release.yml pass none and run exactly as before (setup-matrix output for every broad/shard combination is byte-identical; the TSan matrix moved into setup-matrix with the same legs). Under a selection, setup-matrix keeps only the tagged legs of the selected lanes (an untagged leg is an error, never a silent drop) and each lane job has an `if:` on its lane. pr-smoke and memwaste run on their lanes instead of the old product regex. Docs-only PRs (T0a) get a `contracts` job running the new `scripts/test.sh --contracts-only` (Step 0 contracts, no build). ci-ok needs contracts, lists the run's jobs (actions: read) and passes LANES/JOBS to require-all-green.sh; shard-completeness passes --lanes. tests/test_lane_wiring_contract.sh (Step 0z4) pins it: dry run and release pass no lanes and the inputs default to all, pr.yml wires the selection through, and every lane is served by a tagged leg or an `if:` naming it. The smoke-fixture contract now checks the selector's decisions for the smoke-harness paths instead of the removed regex; select-lanes.sh joins the venue-parity interface probes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
fb558a6b13 |
ci: lane-aware ci-ok and shard-union gates
Under lane selection a PR runs a subset of its jobs, and "skipped" stops meaning "not needed": a selected lane that was skipped, cancelled or never created is a silent gate loss that the needs-level ci-ok waves through. require-all-green.sh keeps the needs-level check and, when given the selector's LANES and the run's JOBS (Actions API job list), requires every selected lane to have at least one job and all of them to have succeeded. Unselected lanes may be skipped or absent but may not fail; memwaste stays report-only. A selection it cannot trust (not a JSON list, unknown lanes, or neither lint nor contracts) is red. Matrix jobs are mapped by the lane value in their name, single jobs by their job id. verify-shard-union.sh takes an optional --lanes: no manifests at all is green only when the selection holds no test leg; without --lanes (dry run, release) or with "all" it fails exactly as before. tests/test_lane_gate_contract.sh drives both with synthetic inputs (selected-skipped/cancelled/missing -> red, unselected-skipped -> green, no legs -> green only for an empty leg selection); Step 0z3. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e78eb87111 |
ci: select-lanes/replay --help stop at the header
Both printed the first code line after the comment header. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e4da6643ff |
ci: lane selector for PR CI (R1 rule set) with contract and history replay
scripts/ci/select-lanes.sh maps a PR's changed-file list to a tier, a lane set, a full flag and the reasons: each path gets a category, the categories give a tier (T0a docs .. T5 core), the tier gives the R1 lanes, and a few categories add lanes of their own (pkg -> wrappers + 3-OS smoke, graph-ui -> Windows guards, memory core -> memwaste, ...). It fails safe: tier T5, an unmatched path, an empty list, the selector itself, any workflow, the test harness, Makefile.cbm and vendored code select every lane. The output is a pure function of the file set. tests/test_select_lanes.sh pins the decision table (43 cases), determinism (order, duplicates, CRLF), the input channels and the lane universe. scripts/test-impact/replay-selector.sh replays all 767 September 2026 PR pushes and the 22 real failures from a compact fixture: every push gets its expected tier (729 equal to the analysis, 38 documented deviations where the analysis split Makefile.cbm diffs by content and the selector, seeing names only, is conservatively full), and every real failure's lane is selected. Both run as Step 0z/0z2 of scripts/test.sh. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
  Glen KoandClaude
|
2fbf3fb977 |
fix(cli): fold per-user /home entry into the alias loop, add test seam
Address review on #2309: - Fold the /home/<name> handling into the existing #2175 alias loop via a shared activation_alias_map() that builds the mapped path in a stack buffer and returns activation_string_copy(); raw allocator sites in activation_transaction.c drop from 80 to 78 (baseline lowered). - Comment the deliberate difference: the per-user entry may resolve to a directory owned by the current account, the whole-alias case may not. - Add cbm_activation_transaction_set_home_root_for_testing() (test seams only) and a Linux test that fails without the per-user branch. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Glen Ko <gleko@amazon.com> |
||
|
Martin Vogel
|
878379a8af |
Merge pull request #2411 from DeusData/fix/harness-timeout-race-deterministic
test(harness): hold the scheduler contract's fixtures by construction |
||
|
Martin Vogel
|
5910c47a97 |
Merge pull request #2332 from DeusData/fix/issue-1916
fix(pipeline): compose axios.create baseURL into HTTP_CALLS (#1916) |
||
|
Martin Vogel
|
c5c13a878d |
fix(security): refresh the vendored checksum of grammars/MANIFEST.md
The fork notes for javascript + tsx changed MANIFEST.md, which is covered by scripts/vendored-checksums.txt; its entry still held the pre-change hash, so Layer 8 (security-vendored.sh) failed. Regenerated with --update; the only line that changed is MANIFEST.md. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
1b79558975 |
fix(grammar): accept a lone '&' in JSX strings and text (#1736)
A plain '&' inside JSX -- a URL query in an attribute (href="https://fonts.googleapis.com/css2?family=Inter&display=swap"), or text such as <p>Tom & Jerry</p> -- produced a tree-sitter ERROR node in .js, .jsx and .tsx files. The file was flagged parse_partial in the coverage view, and the error recovery turned words from the URL (create-next-app -> create/next/app) into identifiers that resolved to unrelated symbols as false USAGE edges. Root cause is upstream tree-sitter-javascript, which the tsx dialect builds on: unescaped_{double,single}_jsx_string_fragment is /([^"&]|&[^#A-Za-z])+/, so '&' is only accepted before a space/digit or as a complete html_character_reference (& / &). jsx_text is scanned by the external scanner, which stops at every '&', and nothing in _jsx_child accepted a '&' that is not a reference. So '&' + letter in a string, and any lone '&' in JSX text, had no valid token. Fix: javascript and tsx become self-maintained forks of the pins we already vendored (javascript @ 58404d8cf191; typescript tsx @ 75b3874edb2d on its javascript base @ 3a837b6f3658), following the arkts model. One patch (tools/tree-sitter-javascript/patches/ jsx-lone-ampersand.patch) is applied to both javascript bases: - a one-character '&' token at lexical precedence -1, aliased to the existing string_fragment (in strings) and jsx_text (in text); the lower precedence keeps & / & lexing as html_character_reference - the string-fragment regexes exclude a '&' directly before the closing quote, so "a&" no longer runs past the end of the string Node types are unchanged (same type/named set). The external scanners stay upstream bytes. Parsers regenerated with tree-sitter-cli 0.25.10 (ABI 15; tsx moves from ABI 14, and its tree_sitter/ headers now match arkts). Without the patch both fork dirs regenerate upstream exactly. The upstream corpus passes unchanged: javascript 116/116; tsx gives the same results as unpatched upstream. Proof on vercel/next.js examples/ (canary 725705bd, 1,722 js/jsx/tsx files): parse_partial 92 -> 47 (tsx 33 -> 0, jsx 2 -> 0, js 12 -> 2; the two left are Babel do-expressions and Flow types), no newly flagged file, code-graph nodes 14,739 -> 14,739, edges 24,773 -> 24,657: the 115 USAGE edges removed were all false cross-example edges from URL fragments, plus a 4 SIMILAR_TO / +3 SEMANTICALLY_RELATED reshuffle from the corrected ASTs. Registered like arkts: MANIFEST.md fork rows + notes (ABI tally recounted), audit FORKS map, THIRD_PARTY.md, vendored checksums. LICENSEs stay upstream's MIT byte-identical. Tests: parse_coverage gains a .js/.jsx/.tsx matrix (double/single-quoted attribute, '&' before the quote, '&' + letter/space/digit in text, plus &/& controls) and a guard that broken JSX is still flagged. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0d783bcca4 |
test(harness): hold the scheduler contract's fixtures by construction
The parallel-scheduler contract failed twice on test-windows CLANG64 while running fake suites only: - PR #2394 (run 36364479687): "Windows timeout race refused without a surviving descendant to refuse over". The refusal came 38s after the forced leader exit (a cold powershell/CIM descendant probe, once in the wave loop and again in the cleanup pass), while the fixture's descendant was a `time.sleep(30)` that had already exited on its own. - PR #2345 (run 36199749518): "suite 'hang_after_summary' taskkill could not prove process-tree cleanup". The scheduler bounded taskkill.exe with --kill-grace, which the contract passes as 1s, so a slow cold start of the tool was reported as a failed cleanup. Both verdicts were a race between a clock and runner latency. An audit of the fixture found more of the same shape: - Every hanging fixture ended on its own after 30s. A starved scheduler could therefore see a "hung" leader exit 0, and a descendant the scheduler failed to kill could vanish before the leak check looked. - The 1s --timeout could kill a slow leader before it had printed its summary (asserted pass=1), spawned its descendant or written the descendant pid. That pid was then read back with int() from a file written create-then-write, so a reader could see no file or an empty one. The empty-file read recorded earlier was the scheduler's ready file, which |
||
|
Martin Vogel
|
2278498371 |
Merge pull request #2393 from DeusData/fix/issue-2264
fix(install): migrate v0.10.8 OpenCode agent profiles on upgrade (#2264) |
||
|
Martin Vogel
|
d17f97f431 |
Merge pull request #2352 from DeusData/fix/issue-2176
fix(rescript): stop binary .res resources hanging the indexer (#2176) |
||
|
Martin Vogel
|
2b329fe75d |
fix(cli): client detection for OpenCode, Hermes, Claude (#1167, #1180)
Three detection defects kept `codebase-memory-mcp install` from configuring agents people run, and made it configure one they never installed. OpenCode without a config file or a CLI on PATH (#1167, #1180). OpenCode was detected only from its global config file or directory, OPENCODE_CONFIG_DIR, or an `opencode` found on PATH. It now also counts as installed when its data directory exists: ~/.local/share/opencode on every OS, %USERPROFILE%\.local\share\opencode on Windows (opencode.ai/docs/troubleshooting, "Storage"; the same page names it as the global config location of older installs). The OpenCode CLI and the Desktop app both create it -- the Desktop's bundled opencode-cli server uses the same XDG paths (anomalyco/opencode packages/core/src/global.ts creates the config and data dirs at startup; packages/desktop/src/main/sidecar.ts only redirects XDG_STATE_HOME) -- so detection no longer depends on PATH. The install target ~/.config/opencode/opencode.json(c) -- %USERPROFILE%\.config\opencode on Windows per the same docs page -- is unchanged and right for CLI and Desktop alike. A PATH longer than 4 KB, which hides every CLI from find_in_path(), is handled separately by PR #2381. Hermes home on Windows (#1180). Native Windows Hermes -- the install.ps1 CLI, the MSIX package and the Desktop app -- defaults HERMES_HOME to %LOCALAPPDATA%\hermes and falls back to a legacy ~/.hermes only while that directory is absent (NousResearch/hermes-agent website/docs/user-guide/windows-native.md, "Data layout"; apps/desktop/electron/data-paths.mjs, resolveDesktopHermesHome). cbm probed and wrote ~/.hermes only, so a Desktop install was never detected and a Hermes found on PATH got its config.yaml and skill in a home it never reads. On Windows the home now resolves to <home>/AppData/Local/hermes unless only ~/.hermes exists, using the home-derived AppData convention of the existing Zed, goose, Kilo and VS Code probes; detection, install and uninstall all follow it. HERMES_HOME still wins, and macOS/Linux keep ~/.hermes. Claude Code from a bare ~/.claude (#1180). Any ~/.claude directory counted as a Claude Code install, so a user who only had the folder got hooks, skills and MCP entries for a client they never installed. Detection now needs what Claude Code itself leaves behind: settings.json in the config dir, .claude.json in the home (in CLAUDE_CONFIG_DIR when set), or the claude CLI. Proof on macOS against synthetic HOME/cache/runtime dirs with a minimal PATH (`install --dry-run --skip-binary`, then a config-only install): bare ~/.claude main: Claude-Code fix: none ~/.local/share/opencode only main: none fix: OpenCode In the OpenCode case the fixed install writes the codebase-memory-mcp entry to ~/.config/opencode/opencode.json; main writes nothing. Controls (a launched Desktop layout, a real Claude user) are detected by both. The Windows Hermes home is covered by a platform-parameterised resolver test that runs on every host plus a _WIN32-bound end-to-end detection and plan test. Tests (cli suite, 335 passed): seven new tests; five fail on main three times out of three (the _WIN32-bound Hermes test and the look-alike control are green on macOS by design). Reverting the Claude+Hermes half and the OpenCode data-dir probe separately turns exactly their own tests red. Nine existing Claude fixtures that used a bare ~/.claude as the install marker now write .claude.json; two Hermes tests and smoke-test 8x expect %LOCALAPPDATA%\hermes on Windows. Refs #1167 Fixes #1180 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ea6fa49fc2 |
chore(security): refresh vendored checksums for the #2176 ReScript scanner patch
security-static (scripts/security-vendored.sh) failed on this branch: the #2176 fix patches internal/cbm/vendored/grammars/rescript/scanner.c (skip the template-string branch in the error-recovery state) and records that patch in vendored/grammars/MANIFEST.md, but the checked-in digests in scripts/vendored-checksums.txt still described the unpatched files. Both changes are intentional and reviewed in the parent commit; this records their SHA-256 via `scripts/security-vendored.sh --update`. Only the two affected lines change, and the integrity check passes again. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
dc735c01fb |
fix(install): migrate v0.10.8 OpenCode agent profiles on upgrade (#2264)
Upgrading from v0.10.8 to v0.11.0 on a machine with OpenCode reported all
three OpenCode subagent profiles as "preserved modified profile" plus an
op=agent_install error each, and the activation stopped ("one or more agent
configurations failed; the published/current executable was kept"), although
nobody had edited those files.
Root cause: #1933 (v0.11.0) added the tool_search / tool_search_regex
permissions to the rendered OpenCode profile. Install and uninstall recognise
a profile as ours only when it matches the current rendering or one of the
known released renderings (the other access mode, the v0.9.1-rc.1 Codex
shape, the pre-tier Verify file). The v0.10.8 OpenCode shape was not in that
set, so cbm_text_migrate_owned_document classified the untouched v0.10.8
files as user-modified. Reproduced end to end: install with the v0.10.8
release binary, then install with v0.11.0/main, into an isolated HOME; the
three profiles are 1729/1880/1910 bytes, the same sizes as in the report.
cbm_render_graph_profile_opencode_v0108() renders the pre-#1933 shape
(both access modes), and install/uninstall now list it as a released
rendering, so those files are upgraded in place and uninstall removes them
as owned. A profile the user actually edited is still preserved. The
released-list assembly that install and uninstall duplicated moves into one
helper, and the two legacy renderers share one implementation (cli.c drops
three raw free sites; the memory-core baseline is tightened 173 -> 170).
The Codex and OpenCode op=mcp_install errors in the same report do not
reproduce from a pristine v0.10.8 config and depend on the reporter's own
config.toml / opencode.json; they are not addressed here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
f4654a7c77 |
Merge pull request #2373 from DeusData/fix/defs-walk-no-cap
fix(extract): definitions walk stack grows without a ceiling |
||
|
Martin Vogel
|
77ab4bea87 |
fix(extract): definitions walk stack grows without a ceiling
walk_defs keeps its pending frames in a growable stack that was bounded by an 8M-frame ceiling (env CBM_WALK_DEFS_MAX). At the ceiling wd_push logged "extract.walk_defs_capped" once and then stopped pushing. Children are pushed last-to-first, so a file wider than the ceiling lost its FIRST definitions: a work cap decided graph content. Fix: no ceiling. The stack doubles on demand, heap frames now come from the memory core (cbm_realloc/cbm_free, class EXTRACT) instead of safe_realloc and raw free, and the doubling is overflow-guarded (int capacity and size_t byte count). The only thing that can stop the walk is an allocation failure, and that is loud: logged at error level as "extract.walk_stack_alloc_failed" (walker=walk_defs, pending, path) -- the same event the complexity walkers use -- and the file's result is marked has_error with "definitions walk: stack allocation failed", which the definitions passes already report as a per-file extract error. The arena-backed path (ctx->scratch) gets the same failure handling. Traversal order is unchanged. CBM_WALK_DEFS_MAX is removed. It was read only in wd_stack_max(); it was not documented anywhere (docs, --help, config, tests), so nothing else changes. extract_defs.c loses its raw free() call, so the memory-core baseline for the file drops from 5 to 4. Test (extraction suite): walk_defs_wide_file_extracts_every_definition extracts a C file with 1000 top-level functions with CBM_WALK_DEFS_MAX=256 set, and asserts every wd0..wd999 definition is present. RED on main (extract.walk_defs_capped limit=256; wd0 missing), GREEN with the fix, RED again with the production change reverted. The extraction (379), complexity (5), ast_profile (10) and semantic (35) suites pass. Real input: Linux kernel, index-only, fresh cache and runtime dirs, same machine. main and fix both produce 8,530,336 nodes and 15,764,128 edges, SEMANTICALLY_RELATED 607 each, and the full node set (label, QN, lines) and edge set (type, source QN, target QN) hash identically -- no kernel file approaches the old ceiling. Index wall time 210.0 s (main) vs 194.4 s (fix), within run-to-run noise. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
1d5ff68e50 |
chore(lint): one helper for pass_calls' transient-QN lookups
pc_module_var added a third compute-QN / find_by_qn / free sequence to pass_calls.c, growing its memory-core ratchet (23 -> 24). Route all three through pc_find_by_computed_qn, which leaves 22 raw sites, and ratchet the pass_calls.c baseline down to match. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ff127ea44c | Merge branch 'main' into codex/fix-1648-sanitized-build-config | ||
|
Martin Vogel
|
8ab13c94f6 |
Merge pull request #2263 from tamerbak/fix/smoke-tool-inventory
Fix/smoke tool inventory |
||
|
Martin Vogel
|
fcdc6bedb7 | Merge branch 'main' into fix/smoke-tool-inventory | ||
|
Martin Vogel
|
f07c18702a | Merge branch 'main' into fix/1696-smoke-runtime-isolation | ||
|
Martin Vogel
|
a6d3d5d820 | Merge branch 'main' into codex/fix-1648-sanitized-build-config | ||
|
Martin Vogel
|
c0bfd073ff | Merge branch 'main' into codex/fix-precommit-git-environment | ||
|
Martin Vogel
|
24afb562dc |
fix(graph): route the new scratch through the memory core
The lint chain stops at its first red target, so a clang-format failure hid the memory-core linter until PR CI: the coverage line-offset table, the Cypher direction swap and the search label each added a raw allocator site. The table now comes from cbm_alloc/cbm_free (extract class), the swap frees through safe_str_free, and the BM25 row array moves to cbm_calloc/cbm_free so mcp.c's raw-site count goes down (baseline 814 -> 813) rather than up. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
 Anton StandrikandClaude Fable 5.1
|
fa138acf44 |
fix(test): arm smoke runtime cleanup at init, retire the daemon first
Review on #2199: cbm_test_runtime_init ran more than a hundred lines before the only EXIT trap, so under `set -e` a failure in the fixture mktemp or its cygpath conversion left the private root behind, and the fixture trap ran smoke_rmtree ahead of the runtime cleanup. Arm `trap 'cbm_test_runtime_cleanup "$BINARY"' EXIT` immediately after a successful init, as soak-test.sh and memlab.sh do. The fixture trap that replaces it now retires the private daemon first and removes the fixtures second, so no earlier cleanup step stands between the exit and the runtime cleanup; on Windows the retirement is also what unblocks the fixture rm. tests/test_smoke_runtime_isolation_contract.sh pins the early trap to the window between the init call and the first fixture; the pin fails against the previous head. Part of #1696. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Anton Standrik <astandrik@yandex-team.ru> |
||
|
Anton Standrik
|
183cd727be |
fix(test): isolate smoke harness daemon runtime
scripts/smoke-test.sh retires the account daemon from seven call sites, but its wrappers sandbox only HOME/TMPDIR/CBM_CACHE_DIR and only CBM_RUNTIME_DIR moves the daemon rendezvous, so every retirement reached the operator's live daemon. Source scripts/test-runtime.sh so every product process runs under a harness-owned runtime and cache, and clean that root up from the EXIT trap. Add tests/test_smoke_runtime_isolation_contract.sh, which fails before this change, and wire it into scripts/test.sh. Part of #1696. Signed-off-by: Anton Standrik <astandrik@yandex-team.ru> |
||
|
astandrik
|
eccc615398 |
test: verify sanitizer build configuration
Signed-off-by: astandrik <astandrik@yandex-team.ru> |
||
 Tamer
|
dadc57dbfa |
fix(smoke): keep MCP tool inventory exact
Signed-off-by: Tamer <tamer.elbakkali@gmail.com> |
||
|
Martin Vogel
|
af3e003ad8 |
fix(mcp): one release point for handle_index_repository's argument strings
main fails the memory-core linter:
src/mcp/mcp.c: grew by 3 (814 -> 817)
PR #1723 (merged as
|
||
|
Martin Vogel
|
92abefa3f5 |
Merge pull request #1723 from liuchong/feat/index-resource-discovery
feat(index): add opt-in discovery resource limits |
||
|
Martin Vogel
|
5aeebbac68 |
Merge pull request #2197 from astandrik/fix/1696-memlab-runtime-isolation
fix(test): isolate memlab harness daemon runtime |
||
|
Martin Vogel
|
6cd1900643 |
Merge pull request #2196 from astandrik/fix/1696-soak-runtime-isolation
fix(test): isolate soak harness daemon runtime |
||
|
Martin Vogel
|
dae1e7dff2 |
style: format slow_move's attribute so both clang-format versions agree
PR CI's lint job failed on src/foundation/mem_override_libc.c:143-144 while the
local lint lane passed on the same file. The two run different formatters: CI
installs LLVM 20, the local lane uses the Homebrew build, which is now 22. A
wrapped attribute is one of the few things they lay out differently, so the
wrapped form was stable locally and a violation in CI.
__attribute__((no_builtin("memcpy", "memmove",
"memset"))) static void *slow_move(...)
Written on one line the declaration is 117 columns, over the 100-column limit,
so it has to wrap and the two versions disagree about how. Spelling the
attribute as a macro brings the declaration to 73 columns, which leaves no
wrapping decision for either version to have an opinion about. The three
siblings below it already fit inline and are left alone.
Also repins scripts/vendored-checksums.txt for MANIFEST.md itself: the security
job's vendored-integrity layer checksums the manifest too, so editing it to
record the properties scanner patch invalidated its own entry. Produced by
scripts/security-vendored.sh --update; the one changed line is the manifest
hash, which is also the check that the earlier manual scanner.c repin was right.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
c7ccb479b4 |
fix(grammar): give the properties scanner per-parser state
Indexing a repo with .properties files in it produced a different graph from
one run to the next. Every few runs one small file - 253 bytes, valid, the same
file that had indexed fine a moment earlier - vanished from the graph, and the
coverage tree put raw-path File/Folder nodes where it had been. On the kotlin
corpus that showed up as node counts wandering between 33,486 and 33,494.
The vendored tree-sitter properties grammar keeps its external scanner's entire
state in a file-scope static:
static bool reached_eof = false;
bool ..._external_scanner_scan(void *payload, TSLexer *lexer, const bool *valid) {
lexer->result_symbol = FAKE_EOL;
return reached_eof = !reached_eof && valid[FAKE_EOL] && lexer->eof(lexer);
}
That flag is how the scanner refuses to emit a second end-of-input FAKE_EOL,
and the grammar needs exactly one to close a file. One static is one object for
the whole process, so with several index workers parsing .properties files at
once, whichever thread reached EOF first set the flag and the next thread's
`!reached_eof` was false: its parse never received the FAKE_EOL it was waiting
for and sat at end-of-input asking for a token that would never arrive. The
per-file parse budget stopped it five seconds later and the file was dropped.
Measured on a 253-byte file: 1,060,630 progress callbacks, ~106 M parse
operations, byte offset pinned at EOF, has_error false - the parser was not
recovering from an error, it was waiting.
The state now lives in the payload from ..._external_scanner_create(), which is
what tree-sitter's create/destroy contract is for. The serialize/deserialize
wire format is unchanged: the state is still the returned length.
Proof, interleaved under identical load so both binaries meet the same machine,
40 runs each: unpatched 6 stalls, patched 0. Under load the kotlin corpus now
returns 33494/203579 ten times out of ten and java 693129/5523595 three times
out of three; both used to wander.
All 103 vendored scanners were audited for the same shape. This was the only
one that WRITES a process-wide static. haskell's debug_parse_metadata and
purescript's res_cont/res_fail are read-only - constants missing a const - and
are left alone.
Upstream 6310671b24d4 still carries the static, so MANIFEST.md records this in
the local-patch table and scripts/vendored-checksums.txt is repinned.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
2f33999044 |
test(vm): let the log decide the Windows leg, not the exit status
On 2026-09-18 a Windows leg printed "7925 passed, 0 failed" and "=== All tests passed ===" and still exited 1, while the identical work run through win.sh's other entry exited 0. The status travels ssh -> cmd.exe -> msys2_shell.cmd and that chain loses it. A channel that can turn 0 into 1 can turn 1 into 0, and that direction is a false green: a red Windows leg reported as passing, which is the one thing the local ladder exists to prevent. So the two channels are now split by what each can actually know. The log decides the test outcome — it is written by the runner on the VM and cannot be mangled in transit — and rc only decides what the log cannot see. Green needs zero reported failures AND the completion marker scripts/test.sh prints last, so a leg that stopped early has a summary but no marker and is red. A lost exit status is reported, not obeyed. The logic is a function, vm_verdict(), reachable as `--verdict <log> <rc>`, so tests/test_vm_verdict_contract.sh drives the REAL code with synthetic logs instead of a copy that could drift from it: lost exit status, failures under rc=0, a marker that must not launder failures, failures in a later suite, partial runs, and no summary at all (which keeps its own exit code 90, so "never ran" stays distinguishable from "ran and failed"). Eight cases, wired into scripts/test.sh as step 0y. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
1d58385b51 |
refactor(mem): route this branch's allocations through the memory core
The memory-core linter is a ratchet: raw malloc/calloc/realloc/free/strdup sites may not grow, because allocations belong on one accounted path (src/foundation/mem_core.h) rather than spread across hundreds of call sites. This branch's own work had added 35 raw sites across nine files -- the parallel k8s prep, the configlink key matching, the worker pool's counted path, the Go shared stdlib registry, the spill namespace copy, the route service array and the namespace-name arrays. All of them now use cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/ cbm_mem_strdup. One was a latent bug rather than a style point: pass_k8s freed a buffer that k8s_read_file had allocated with raw malloc, so converting only the free would have mismatched the allocator -- the read function and every free of its result are converted together. The Windows free-space path widens its argument through the memory core instead of cbm_utf8_to_wide(), which allocates with raw malloc. Two files improved enough to tighten their baselines, which the ratchet requires so the gain cannot be quietly given back: sqlite_writer.c 92->74 (the index-cell arena), pass_k8s.c 13->6 and registry.c 19->17. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
92e7e6a913 |
fix: keep large thread-locals off the static TLS block
Indexing broke on x86-64 Linux: the index worker refused to start and
SIGKILLed its own process group, so every venue reported nothing but
"index worker ended with killed (exit=-1, signal=9)".
glibc places a thread's static TLS block inside the stack allocation it
makes for that thread. This branch's new thread-local caches -- a
512-slot field cache (~24 KB), per-depth tree cursors (~4 KB) and two
parked CBMArenas (~8 KB; an arena carries blocks[256] + block_sizes[256])
-- took static TLS from 47,905 to 93,513 bytes. The 64 KB parent-death
watchdog thread then could not be created at all, and a worker without
process-tree containment correctly refuses to index.
Four layers, because there were four things wrong:
- The big thread-locals move to the heap behind a thread-local POINTER,
allocated once per thread and reused, each with a correct fallback if
allocation fails (the field cache and cursor pool are caches; a miss
is slower, never wrong). Static TLS is back to 52,233 bytes.
- PARENT_WATCHDOG_STACK_SIZE 64 KB -> 256 KB. "The watchdog only polls,
a tiny stack suffices" was true when written and false once TLS
reached ~50 KB, because the TLS block comes out of that same
allocation.
- cbm_thread_create retries with the default stack on EINVAL. A stack
size is a hint about what a thread needs; the platform refusing the
hint is not a reason to fail to create the thread.
- tests/test_thread_stack_tls_contract.sh (Step 5a of scripts/test.sh)
keeps static TLS a quarter clear of the smallest requested stack,
reading that smallest size from the sources so a new small stack
tightens the gate automatically.
Two diagnostics that made this findable at all, and that stay:
smoke-test.sh prints the worker log its failure message names (CI deletes
that sandbox with the job, so the one artifact naming the cause was
unreadable), and worker_containment_unavailable() now reports WHICH of
its conditions failed, with pid/pgrp/ppid.
Verified in the amd64 container, the venue nothing on the local ladder
covers: baseline main indexes 9039 nodes/15067 edges, this branch before
the fix refuses to start the worker, and after it indexes 9039/15067.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
f0a86ef2c2 |
feat(ci): seeded fuzz leg, and register both new legs as canonical entries
scripts/fuzz.sh drives the libFuzzer targets in two shapes from one entry: a fixed --runs from a fixed --seed, so a CI smoke is a function of the code and not a lottery (O9), and a time-boxed --max-total-time for exploration. Checked-in seeds are copied into a working corpus under build/fuzz/, so a run never rewrites the repository; a crash leaves its reproducer behind and the script exits non-zero. Wired as a smoke into dry-run.yml and as exploration into nightly-soak.yml. The venue-parity contract rightly refused both workflow steps: a venue may provision or plumb, but only a CANONICAL leg entry may exercise the product, and neither script was registered. Registering them — rather than working around the contract — is the fix, since both own their leg end to end (build, corpus or binary, artifact paths, exit code) exactly as soak-legs.sh owns the soak. _memwaste.yml joins the policed venue walk for the same reason, and both scripts join the layer-5 interface probes. Those probes immediately earned their keep, finding two real defects: `fuzz.sh --help` exited 2 because the positional target consumed the flag, and memwaste.sh read a typo'd flag as the repository path — which would have indexed nothing and reported an empty lane as a clean one. test_shell_line_endings.sh now degrades where the checkout has no reachable repository metadata: the Linux container bind-mounts the working tree, and a git worktree's .git is a pointer to the host repository, so `git ls-files` cannot run there and the contract failed with "matched no files — the glob set is broken", killing the leg before it compiled. It is a repository property, so it stays gating in every venue that has the metadata (macOS host, the Windows VM's real checkout at C:\cbm with 117 files, every hosted-CI checkout). test_version_metadata_contract.sh already guards the same way for the same reason; the alternatives considered and rejected are recorded at the case. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
81264814f1 |
feat(qa): memory + CPU waste sanitizer over the memory core
Answers a question the existing tooling could not: of everything we
allocate, how much is never used?
Lanes, all driven by scripts/memwaste.sh:
event every allocation observed through the memory core — size, age,
churn, and whether the bytes were ever written
access SanitizerCoverage load/store instrumentation, so "allocated and
never touched" is measured rather than inferred
scaling k vs 2k replicas through clang profile counters, to catch
super-linear work per file
padding allocator slack per size class
ratchet the numbers recorded per venue so a regression is visible
The event lane needed a foreign-free counter: on Windows the override sees
frees for blocks it never allocated (they come from the CRT), and counting
those as untracked tripped the soundness gate. mem_events now separates
them (14 daemon / 5 worker on the Go corpus).
Two portability fixes the lanes surfaced, both real:
- the enable decision must not be cached before `environ` exists. Under
-fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our
wrapper before glibc sets up environ, so the lane silently disabled
itself and produced no dump at all on Linux.
- the access map needs 48 bits: arm64 Linux hands out addresses above
2^47, which a 47-bit map dropped on the floor.
mem_profile is retired into this: it measured a strict subset and had its
own sampling story.
Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on
all three platforms report-only, and pr.yml/release.yml carry the wiring.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
7b77fd4834 |
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (
|
||
|
Martin Vogel
|
7fedfe85c6 |
lint(mem): scan all of src/ and internal/, exempt vendored/ by path segment
The memory-core linter scanned src/ and internal/cbm/. Widen it to src/ and every subtree under internal/, so a new tree added there is covered without anyone remembering to list it, and the count is for the whole project -- cli, mcp, daemon, store, pipeline, the extraction engine. Vendored code is now excluded wherever a vendored/ segment appears in the path rather than by one hardcoded prefix, so that exemption cannot drift either. internal/ holds only cbm today, so the regenerated baseline is byte-identical: 92 files, 4086 raw sites. Widening lost no coverage and vendored stays out (0 sites listed). The red/green proof still holds under the new scope, and the failure message now names the NEW site -- one appended malloc reports grew by 2 (83 -> 85) at the appended line, instead of listing six pre-existing ones. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
da3258f3c7 |
feat(mem): a central allocation core, a linter that enforces it, and phase attribution in the index pipeline
Memory in this project is now allocated through ONE core. This lands the
core, the gate that keeps new code on it, and the instrumentation that lets
an index run say which class of memory grew in which pass -- the three things
the 2026-09-13 audit found missing.
WHAT THE AUDIT FOUND
There was no centralized allocation layer. foundation/mem.h was policy and
measurement only (budget, RSS, over_budget, memory map, phase marks) with no
alloc/free wrapper, so the budget could be observed after the fact and never
enforced. Raw allocator calls in the tree: 4086 across 92 files. The
extraction engine had already adopted arenas (1301 call sites); the graph
buffer -- which holds the 35 GB peak of a kernel index -- used none,
allocating every 64 B node, 48 B edge and every name/qualified_name/
properties_json individually.
Two further facts made a self-accounting core the only workable design:
1. cbm_mem_map_collect() walks THIS thread's mimalloc heap only. Walking
the process-wide heap is a data race TSan caught on macOS, so it is
deliberately not done, and an 18-worker index attributes almost
nothing -- the rest lands in residual.
2. The mimalloc global override is ON for Linux/MinGW and permanently OFF
for macOS (two-level namespace: this binary's free becomes mi_free while
system libraries keep allocating from the system zone, and a pointer
crossing that boundary aborts). On macOS ordinary malloc is served by
the system allocator; the startup audit reports owned_classes=0/6. Any
accounting that assumes mimalloc owns the pointer is blind on an entire
platform.
THE CORE (src/foundation/mem_core.{h,c})
cbm_alloc / cbm_calloc / cbm_realloc / cbm_mem_strdup / cbm_free, each tagged
with an allocation class (gbuf_node, gbuf_edge, gbuf_string, gbuf_index,
extract, semantic, dump, store, other). Per-class live bytes, live blocks and
peak, kept in atomics -- thread-safe by construction, identical on every
platform, independent of which allocator serves malloc. libc semantics are
preserved exactly (alloc(0) is freeable, free(NULL) is a no-op, realloc(NULL)
allocates, a failed realloc leaves the block intact) so adoption is a
mechanical rename and never a behaviour change.
No per-allocation header: a {class,size} prefix costs 16 bytes on ~100M blocks
at kernel scale, 1.6 GB of overhead to measure a memory problem. Sizes come
from the platform usable-size query (malloc_size / malloc_usable_size /
_msize), which is correct under either allocator and free. Where no query
exists (BSD) the request size is charged, which understates -- the safe
direction for a diagnostic. A mismatched class on free clamps at zero rather
than wrapping, so a small drift can never masquerade as a colossal leak.
Arena-backed allocators report in bulk through cbm_mem_class_add_external /
remove_external so their memory appears in the same table; the extraction
engine must not be rewritten to per-object allocation, that would undo the
batching that keeps its allocation count low.
THE LINTER (scripts/lint-memory-core.py, wired into make lint-ci)
Raw malloc/calloc/realloc/free/strdup/strndup outside the core is a defect.
4086 sites cannot migrate at once, so the gate is a ratchet: a checked-in
baseline (scripts/memory-core-baseline.txt) records each file's count and
the build goes red when any file grows, or a new file appears with any.
Files only ever go down; the baseline line is lowered in the change that
migrates the file, and --strict fails a stale baseline. Comments and string
literals are stripped first, so prose that mentions malloc( cannot trip it --
the security audit already bit on exactly that with fork(. Proven both ways:
one appended malloc turns it red naming the file (83 -> 85), restoring turns
it green. Scans src/ and internal/cbm/: cli, mcp, daemon, store, pipeline.
PHASE ATTRIBUTION (src/pipeline/pipeline.c)
cbm_mem_phase_mark and the new cbm_mem_class_log now fire at pipeline.begin
and at every pass.timing site: parallel_extract, the six sequential passes,
the eight predump passes, tests and dump_and_persist, with peaks reset per
index. Both instruments existed in foundation/ and were wired only into MCP
request handling, never into the index pipeline -- which is where the memory
is. Locating the kernel peak this week required an external RSS sampler
because nothing in-process could say which phase it was in.
PRESSURE PRIMITIVES (system_info.c, mem.{c,h}, platform.h)
cbm_system_available_ram() -- macOS host_statistics64 (free + inactive +
purgeable), Linux MemAvailable, Windows GlobalMemoryStatusEx; 0 when unknown,
never cached -- and cbm_mem_system_under_pressure(), true below 12.5% of RAM
available and false when unknown so nothing ever aborts on a guess. Landed
UNWIRED: an earlier draft used them to let the over-budget latch press on
whenever the system was not under pressure, and that permitted the very
overshoot this work exists to end (Decision A's tests at
test_pipeline.c:13481 and :13583 went red under it, correctly). They are the
backstop for the next step, admission control, which keeps a run under budget
before an allocation rather than measuring it after.
mem suite 10 new tests green; pipeline + mem 339 passed 0 failed; lint-ci
clean including the new gate.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Anton Standrik
|
823e720982 |
fix(test): isolate memlab harness daemon runtime
scripts/memlab.sh gave the profiled run a private CBM_CACHE_DIR only, so the process joined the operator's account daemon and cleanup removed the cache without stopping any daemon. Source scripts/test-runtime.sh for a harness-owned runtime and cache, and stop the private daemon before the work directory is removed. Add tests/test_memlab_runtime_isolation_contract.sh, which fails before this change, and wire it into scripts/test.sh. Part of #1696. Signed-off-by: Anton Standrik <astandrik@yandex-team.ru> |