mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-04 18:58:00 +00:00
main
33
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
8fae39e294 |
Merge origin/main into fix/adr-reindex-race
tests/test_pipeline.c: main and this branch added tests at the same spot; both are kept unchanged. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7716153630 |
Merge origin/main into fix/home-user-symlink-alias
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
04cb937236 |
Merge origin/main into fix/adr-reindex-race
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
5350b5b83e |
fix(cli): render the Copilot manifest through the memory core
The PostToolUse change rendered the manifest twice per install and uninstall (current and previous generation) and freed both with free(), which grew the raw allocator count in src/cli/cli.c past its baseline (lint-memory-core). The manifest is now written with cbm_json_writer_allocator() and released with cbm_free(CBM_MEM_CLASS_OTHER, ...), including the frees that predate the change, so cli.c drops from 170 to 166 raw sites and its baseline is lowered to match. The rendered bytes are unchanged, so byte-exact ownership of manifests written by earlier installs still holds. Refs #509 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
  0xb007ab1eandClaude Opus 4.8
|
001fbe269a |
fix(adr): store ADRs in a per-project sidecar DB, untouched by reindex
Reworks the ADR-loss-across-reindex fix to Alternative 1 (per the PR review):
remove the race rather than narrow it. ADRs live in a per-project sidecar SQLite
DB ("<graph_db>.adr.db"), separate from the graph DB a reindex rebuilds or
deletes, so there is no capture, re-read, re-apply, or lock — and no sleeps,
polling, or timeout deciding an outcome.
store.c:
- ADR writes (store/delete/update_sections) use a cached read-write handle opened
SQLITE_OPEN_READWRITE|CREATE and configured like the graph store (PRAGMA
busy_timeout=10000 + journal_mode=WAL) so cross-process access waits rather
than failing SQLITE_BUSY. A :memory: store keeps ADRs in the graph DB.
- Reads never write: cbm_store_adr_get uses the cached RW handle if open, else
opens the sidecar SQLITE_OPEN_READONLY (with the same busy timeout) when the
file exists, else reads the legacy graph row without creating anything. When
the sidecar exists but is NOT marked migrated (half-created), it falls back to
the legacy graph row so a partial sidecar never hides an ADR.
- One-time migration is atomic: adr_copy_legacy_once wraps the marker check,
legacy copy, and marker insert in a single BEGIN IMMEDIATE...COMMIT
(re-checking the marker under the lock; ROLLBACK->ERR on failure). New public
cbm_store_adr_migrate_once. Sidecar path via cbm_path_for_file_api (Windows
long-path). Dropped the write-only adr_db_path field.
pipeline.c: migrate a legacy ADR into the sidecar BEFORE the format-change and
forced-full routes unlink the old generation; preserve-and-abort on failure, and
do NOT fail open — a query-open returning NULL preserves unless the final DB
genuinely does not exist. Migration targets the FINAL path (staging suffix
stripped). The early-capture/re-apply machinery and its test hook are gone.
artifact.c: the exported artifact carries the ADR — snapshot_inject_adr always
DELETEs project_summaries first, then injects only when an ADR is present, so a
deleted ADR never resurrects through a lingering legacy row. Import restores the
ADR into the destination sidecar via cbm_store_adr_restore_from_graph_if_absent:
it copies the artifact's ADR (INSERT OR IGNORE — never clobbers a local one)
regardless of the migrated marker, so a teammate who already indexed the project
(marked sidecar, no ADR) still receives the shared decision record. store.c
factors the row copy into adr_copy_rows, shared by the one-time legacy migration
and the ungated restore.
mcp.c: delete_project removes "<db>.adr.db" (+ -wal/-shm/-journal) only after the
graph unlink succeeds; is_project_db_file skips *.adr.db. cli.c: exclude
*.adr.db from index listing/counting. README + memory-core baselines updated.
Tests (file-based): ADR written mid-reindex survives on BOTH the full and delta
routes (route-pinned FORCED_FULL / CLOSURE_REPAIR); a legacy row migrates before
a forced-full/format reindex with no prior read; migration failure (a directory
at <db>.adr.db) preserves-and-aborts with the old generation intact;
delete_project via the MCP handler removes the sidecar + journals; the artifact
export->import round trip keeps the ADR; a deleted ADR does not resurrect on
import; an import does not clobber a local ADR; an import into an already-indexed
destination (marked sidecar, no ADR) restores the artifact's ADR; a half-created
sidecar falls back to the legacy row. The import tests remove the destination
graph DB (keeping the sidecar + marker) before importing, matching production
(try_artifact_bootstrap imports only when the graph DB is missing) so the rename
succeeds on Windows, not only on POSIX.
Closes #516
Signed-off-by: 0xb007ab1e <134006168+0xb007ab1e@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Martin Vogel
|
631d24c790 |
Merge main and strengthen home alias ownership regression coverage
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
181837343e |
perf(cypher): stream query execution chunk by chunk from a paged node scan
On the kernel index (8.5 M nodes, 1.9 M CALLS edges) `MATCH (a)-[:CALLS]->(b)
RETURN count(*)` peaked at 18-25 GB RSS (killed in a 12 GB container): the
store materialised every start node with all its strings (4.9 GB), every one
was deep-copied into a 1.6 KB binding, and every expanded row was materialised
again before projection.
cypher: execute_single now runs the start nodes through the pipeline in chunks
of 1024 (pattern-0 hops, late WHERE, WITH, RETURN). Each hop emits a binding's
expansions in input order and every consumer (RETURN rows / star / default
projection / aggregation, WITH aggregation / projection) keeps its state
between chunks, so the concatenated output is the single-pass output: same
rows, row order, first-seen group order, ORDER BY ties, DISTINCT, LIMIT and
max_rows truncation, OPTIONAL fallback rows, warnings. Queries with additional
MATCH patterns keep one pass (their join strategy depends on the first binding
of the whole set). Start nodes and fetched neighbours move their strings into
the binding instead of being deep-copied and freed.
store: cbm_store_node_scan_open/_next/_close hand out the rows of
cbm_store_find_nodes / _by_label a page at a time. Pages are keyset queries
along the index the eager query walks (idx_nodes_file: NULL file_path rows by
id, then the rest of the last file by id, then the files after it;
idx_nodes_label: by id), each reset before it returns, so no read lock is held
between pages. open verifies the eager plan with EXPLAIN QUERY PLAN and falls
back to the eager rows for any other plan.
Kernel, same index, medians of 3 (macOS arm64):
count(*) peak RSS 24.9 GB -> 256 MB, 23.0 s -> 13.0 s
top-callee peak RSS 12.4 GB -> 1.26 GB, 11.7 s -> 10.9 s
Responses byte-identical to the baseline on django, redis and the kernel
(19 Cypher + 24 tool cases each); the streamed scan equals the eager scan
field by field, in order, for every project and label of all three indexes.
A scan that fails part-way fails the whole query ("query aborted: reading the graph failed") instead of answering from the pages read so far:
The streamed start-node scan answered from the rows of the pages it had read
when a later page failed (a SQLite step error or a page re-prepare failure):
a partial result that looked complete. A failed page, a scan that cannot
open, or a failed eager fallback now marks the start source failed;
execute_chunk returns an error and cbm_cypher_execute reports
"query aborted: reading the graph failed (<store error>)" with no rows. The
store error is logged as cypher.scan_failed.
The test seam cbm_store_node_scan_test_fail_page_after (CBM_ENABLE_TEST_SEAMS)
fails one page after a given number of successful pages; the new test fails
the second page of a label scan (rows and count(*)) and a later page of the
all-nodes scan, and asserts an error result with no rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
7729a8ee9f |
fix(mcp,cypher): allocate the new query-path memory through the memory core
lint-memory-core failed on #2642: the new search_code scanner, result index, root containment cache and Cypher sort/group/distinct scratch used malloc/calloc/realloc/free directly (mcp.c +22, cypher.c +26, git_context.c +1). They now go through cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/cbm_mem_strdup: file buffers as CBM_MEM_CLASS_EXTRACT, the rest as CBM_MEM_CLASS_OTHER, each block freed with the class it was allocated with. The result index reserves its slot before a result's strings exist, so its insert cannot fail and needs no cleanup; cbm_git_branch_resolve writes into caller buffers instead of returning heap strings. cypher.c ends below its baseline (128 < 132), which is tightened. Output unchanged: 42/42 tool and Cypher responses identical on django; mcp, cypher and git_context suites green under ASan. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
f96001f4ed |
fix(pipeline): keep both nodes when a file and a directory share a stem
A file and the directory beside it with the same stem (genproto.sh next
to genproto/) got one qualified name: the file's Module QN equals the
directory's Folder QN, and the graph buffer keeps the structural node, so
the file's Module was dropped, and with it the edges that pointed at it.
On 17 infrastructure repositories this lost 27 Modules (shell scripts,
YAML and JSON manifests, a Makefile, Rust, Elixir, Python, Terraform).
One side now yields, decided once per run from the discovered files:
- the Folder is spelled proj.pkg.foo/ (no module QN ends in a slash);
- where the directory is a Java/Go package it IS that package's module
and Go imports resolve to it, so the Folder keeps its QN and the file's
Module is spelled proj.pkg.foo#sh instead.
Symbol QNs do not change.
The closure-repair planner declines when a deletion touches a collision
on either side, since the spelling depends on the file set. The kustomize
resolver binds a directory entry to its Folder even when a same-stem
manifest sits beside it. CBM_SEMANTIC_INDEX_VERSION 7 -> 8, so existing
indexes rebuild once.
Measured on main
|
||
|
Martin Vogel
|
23e291058c |
perf(python): index unresolved submodule prefix probes
Reuse a stable arena-owned function QN order for ambiguity marking and submodule prefix lookup, preserving exact lookup policy and appended tails. Add prefix/index failure regressions and a pipeline operation-count case. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
eb4f6b99c4 |
fix(pipeline): decline the closure plan through one exit
The Python namespace checks added two more raw free(dep_targets) calls on decline paths, growing pipeline_incremental.c past its memory-core baseline. The first file that declines now stops both the changed and the deleted loop; one exit after the loops skips the dependent query and releases dep_targets and the row index for every reason, including the existing missing-row and added-name declines. closure_python_namespace_decline maps the namespace comparison to its decline reason. Decline order and reasons are unchanged. The file drops to 140 raw sites; the baseline is tightened to match. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
8a7cbab13b |
Merge origin/main into feat/infra-graph
Brings in #2596 (MSan: hidden uninitialized reads) and #2068 (CodeAtlas web). README: main's CBM Atlas line and this branch's infrastructure-as-code line are both kept. tools/list metadata: main and this branch each stayed under the 20 KiB discovery budget, but together they reached 20,809 bytes. The branch's wording is shorter now (search_graph kind/namespace, one hint per trace mode: 136 bytes less), and the reviewed budget is 21 KiB so the topology/deploy/run modes keep their usage hints. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
4d9592b5fb |
Merge pull request #2068 from DeusData/feat/codeatlas-web
CBM Lens Beta: graph exploration and architecture workspace |
||
|
Martin Vogel
|
a19d7a45fb |
Use memory core for Lens response ownership
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
2d291002df |
Merge origin/main into feat/infra-graph
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
06e69d0c56 |
doc-links(ts): bare specifiers through tsconfig paths and workspace packages
A bare import specifier was always external. The TS resolver now asks the pipeline what it already knows for import resolution, without indexing package.json or tsconfig.json: the nearest tsconfig/jsconfig `paths` and `baseUrl` (the alias collection; the first target, as TypeScript tries it first), then the workspace package whose `name` the specifier is (the package map), bound to the package's entry module (`exports`, `main`, `module`) when the repository indexes it. Every other bare specifier, a package's subpath included, stays external. Incremental runs stay exact: a tsconfig change re-resolves the files under it, a package.json change is a manifest change. path_alias gets cbm_path_alias_resolve_into, which writes into the caller's buffer; cbm_path_alias_resolve now wraps it (two raw allocations fewer, the memory-core baseline lowered to match). Also fixed: a declaration in an index file has no `index` in its QN (fqn.c strips it from a symbol's QN, not from the module's), so a reference to one missed its node; the resolver now names a file's declarations with that prefix. Proof (ship build, Apple M5 Pro / 48 GB): the TypeScript compiler b465fdbfe175 unchanged (69 edges, 57 of the prototype's 61 pairs); discord.js 4dc1acc6 unchanged (no doc reference names a workspace import; its packages' entries are build output). Tests: a tsconfig alias, a workspace package's entry and an index file's declaration each bind, each red with its rule reverted; path_alias, pipeline, edge_imports, incremental and all doc suites pass (1298). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
48278d5117 |
fix(infra): new Helm and kustomize code allocates through the memory core
The Helm file:// dependency records and the exact kustomize entry resolver added raw strdup/free sites, so pass_k8s.c went from 6 to 9 raw allocator sites and pass_pkgmap.c from 82 to 85 -- the memory-core ratchet (make lint-memory-core, part of lint-ci) fails on both files. pass_k8s.c: a dependency record now holds core-owned copies (class other) of its chart path and target QN. The QN helper copies the libc string fqn.c hands out into the core and releases it in one place, and the record array grows before anything is allocated, so the failure path frees nothing. 9 -> 6, the baseline. pass_pkgmap.c: the resolver's three frees release fqn.c's libc-allocated QNs and paths and have to stay raw. The manifest read buffer, private to this file, now comes from the core (class extract: per-file source text) and its three callers release it there. 85 -> 81; the baseline drops from 82 to 81 in this change. No graph change: kubermatic and kubernetes/k8s.io indexes are identical to the previous build node for node and edge for edge, deckhouse in every edge and node identity. Its remaining difference, the order of the variants list on QN-colliding Go methods, also appears between two runs of the same binary and is not caused here. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
40c9af3737 |
refactor(infra): delete the line-regex infra parsers nothing calls
pass_infrascan.c carried line-by-line parsers for Dockerfiles, .env
files, shell scripts and Terraform, five file classifiers and a JSON
bracket cleaner; pass_envscan.c walked a repository for URL-valued
environment assignments. No production code calls any of them: the
graph gets these files from the tree-sitter extractors, and the
envscan walk said so in its own header comment. Their only callers
were 40 tests in test_pipeline.c that asserted the parsers' output,
so the suite measured code the index never runs.
Deleted: cbm_is_dockerfile, cbm_is_compose_file,
cbm_is_cloudbuild_file, cbm_is_env_file, cbm_is_shell_script,
cbm_clean_json_brackets, cbm_parse_{dockerfile,dotenv,shell,terraform}_source,
cbm_scan_project_env_urls(_excluded), their result structs, and the
tests and test helpers that existed only for them.
Kept, all with production callers or a planned one:
cbm_is_kustomize_file and cbm_is_k8s_manifest (pass_k8s), cbm_infra_qn,
the Helm Chart.yaml parser, and the secret detectors
cbm_is_secret_binding / cbm_is_secret_value (still covered by
infra_secret_detection), which the secret-hygiene work reuses. The
secret detector's prefix offsets now have their own names instead of
borrowing the Dockerfile parser's keyword lengths.
No graph change: the deleted functions were unreachable from the
index. Suites green: pipeline 299 (339 minus the 40 deleted), platform,
grammar_probe_f/g, infrascan, discover, language, lang_contract,
incremental, integration, edge_structural, configlink.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
 BumaldaOverTheWater94
|
944cfbfd75 | Merge branch 'main' into fix/home-user-symlink-alias | ||
|
Martin Vogel
|
867e5f883d |
Merge origin/main into the smart-CI branch
Ten files changed on both sides; every conflict was a union of independent additions: - workflows (_lint.yml, _test.yml): keep the branch's lane `if:` and main's pinned ubuntu-24.04 runner with its reason. - internal/cbm/cbm.h: main's callable-identity and module-doc fields plus the branch's configured-test fields. - Makefile.cbm: both new test files; prod_lsp_all.o keeps main's .build-config dependency and the branch's coverage flags. - scripts/test.sh: main's setup-scripts contract becomes Step 0y3, ahead of the branch's 0z series and the contracts-only exit. - src/foundation/subprocess.c: main's cbm_posix_close_fds_from (it closes the error descriptor too). - src/mcp/mcp.c: main's repository default base for detect_changes, with the early returns released through the branch's detect_release. - src/pipeline/lsp_resolve.h, tests/test_main.c: both includes, suites and start-up steps (main's git environment reset first). - scripts/memory-core-baseline.txt: recounted on the merged tree and lowered (mcp.c 773, config_toml_edit.c 73). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
36e5874525 |
feat(test-impact): team artifact bundles and admitted coverage
A main commit can now publish its test-impact bundle — the commit's frozen graph, its per-test coverage map and receipt.json binding both — and a PR selection given the bundle of its merge base, verified by the caller, builds its graph incrementally from the bundle's graph and narrows suites by the admitted map. Without a verified bundle nothing changes: the graph is built from scratch and every reached suite runs whole. Producer and consumer: - `codebase-memory-mcp test-impact publish|select` (in process, no daemon): publish builds the bundle of the checked-out commit, from the previous bundle's graph when that bundle verifies; select is the selection with optional --artifact/--artifact-commit/--artifact-verified. The MCP surface is unchanged. - receipt.json binds the exact imported graph (graph digest), the graph's content and topology, the map's three files and a compatibility digest (platform label + the bytes of test_impact.coverage.compatibility_paths). - A bundle is used only when the caller verified its source, it is the merge base's own, and its imported graph is the one the receipt names; the map is admitted by the existing receipt policy (source, ancestry, content, age, compatibility, metadata). Refusals become rejection reasons in the answer's receipt. - C_eff: a changed function the map knows contributes its own id; one it does not know (new, platform-only, a type or macro) contributes its nearest mapped callers on the static inbound closure; deleted names contribute every mapped function of that name. Graph: - the frozen build accepts a base candidate and repairs it through the incremental closure route; any doubt declines to the full build. Pinned to one worker like the full frozen build. - incremental closure repair skipped the interprocedural complexity pass: re-parsed functions lost transitive_loop_depth/recursive and callers kept stale values. One traversal now serves the in-memory pass and a new store-level recompute that runs before the importance rescore. - test-impact's frozen builds use a fixed display Git context: the per-run pin path and the commit ids made two builds of one tree differ and made every commit a semantic-input change, so repair always declined. - graph content and topology digests (store_content_digest.c): the exact graph digest binds one database (ids, db_uid, generation), never equal across builds; the content digest keys rows by what they say, the topology digest by what selection reads. Measured on this repository (frozen publish, macOS arm64): a two-file C change repairs in 18-20 s instead of 75-80 s with the topology of the full build; a docs-only change repairs with identical content; selections with and without the bundle are identical on all seven replayed pairs (two repaired incrementally, five declined to the full build). Recorded, not fixed: closure repair still differs from a full build in edge PROPERTIES (argument quoting and key order, CONFIGURES args) and in the corpus-wide SIMILAR_TO/SEMANTICALLY_RELATED edges; selection reads neither, so the topology digest is the incremental-equals-full check and the receipt carries both. Tooling: scripts/test-impact/coverage-merge.py builds a push's incremental coverage map (re-run suites verbatim, the rest carried with ids remapped onto the new image's function table), with its contract as test.sh Step 0z7. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c5abb64943 |
feat(test-impact): detect_changes scope:"tests" end to end
One request pins the change (merge base .. HEAD), builds the graph and the runnable test registry of exactly the HEAD snapshot, seeds the graph from the change, walks what depends on it and answers which tests to run (cbm.test_impact). Missing or doubtful evidence never narrows: it becomes a run-all answer that names why. Snapshot (classified ledger -> frozen graph -> registry): - one shared language classifier for discovery and pinned blobs (raw bytes on every platform); the classified ledger feeds the frozen build, which indexes exactly the listed files and records partial parses as per-file PARSE_GAP instead of refusing the graph; - the pinned tree skips symlinks and submodule links (counted), as discovery never indexes them; - the test model scopes uncertainty to the suites of its file (they run whole), reads line splices where they occur, treats a mid-file `#ifndef X` + `#define X` as an ordinary condition, and no longer reads a plain identifier named like a macro as an invocation. On this repository the registry is now narrowable and matches a text census exactly (8,365 cases, 141 runner suites). Seeding and reach (port of the measured replay, rule B): - hunk -> overlapping definitions, comment-only hunks dropped, pure deletions -> enclosing definition, no-definition hunks -> whole file, deleted names found by text, header changes propagated through the include closure and the headers' macros (pastes included), file-level escalation for unclassifiable header lines; - inbound fixpoint walk over CALLS, ASYNC_CALLS, USAGE, CALL_REFERENCE, READS (+ routes); cases, suites and the runner's entry points are sinks; a reached runner main runs everything (RUNNER_REACHED); - a parse gap whose text names a reached definition reaches the definition around it (or its file); - macro-registered tests run their suites whole; cases and suites map to graph nodes by their start line. The line patterns reproduce the reference regexes exactly on 1,383,850 lines of this repository and 37,167,926 kernel lines; on 24 replayed changes the engine selects at least what the reference selected (13 identical, the rest only conservative supersets on the same graph). Also: - configured test declarations that cannot be mapped degrade their file (no configured roles, a test_declarations diagnostic) instead of aborting the whole index; - the selector keeps other runners' suites out of the selection, and the result codec judges the model by narrowability like the selector; - a change to .codebase-memory.json runs everything (built-in rule); - .codebase-memory.json with the c-cbm preset and this repository's rules. Gate fixes (suites, lint-ci, Windows cross-compile): - the request runs on its own thread with a default-size stack (daemon tool calls run on 256 KiB threads, where the in-process build died); - a request-supplied config_path is no longer followed by an unlink of an uninitialized path buffer; - the inventory's frame-local reader no longer stays reachable from the caller's source after the filter call; - FILE_ID_INFO needs _WIN32_WINNT >= 0x0602; the pinned-tree header pins it before any system header (toolchains defaulting lower failed); - memory-core ratchet: the win_utf8 converters gain core-backed `_in(cls, ...)` forms over one implementation (the legacy names keep the C library for unmigrated callers), the Git facts module uses them, and the Windows spawn path releases everything through one failure exit; pipeline, extraction and config allocations that this work touched moved to the core in matched pairs; baselines tightened; - the branch's C files formatted with the project's clang-format. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
11867dfa74 |
feat(identity): callable identity plumbing, no language enabled (#2061)
Overloads that share a base QN (Java/C#/C++/Kotlin/Swift/Scala/ObjC) merge
into one graph node today, and that node collects the CALLS of every
overload. The agreed cure is one signature-qualified callable identity for
all overloading languages:
callable_qn := base_qn [<tparams>] (params) [cvref]
This change lands the plumbing with every language still at
CBM_CALLABLE_ID_NONE, so the graph is byte-identical; each language is
switched on later in its own change (with its index-format bump).
- internal/cbm/callable_sig.{c,h}: the per-language mode side table
(NONE|TYPED|LABELED_TYPED|LABELED|ARITY; kept out of CBMLangSpec like
cbm_string_dispatch_suffixes, since ~160 positional rows would otherwise
all need the member under -Wmissing-field-initializers), the shared
builder cbm_callable_sig[_mode] for the def and the call-scope QN, and
the inverse cbm_qn_callable_base_len[_named]. Suffix contract: no '.',
'::' or '->'; whitespace-normalized; comments/annotations/attributes
dropped; qualified types keep their last segment; '->' spelled '=>';
variadic '[]' (Java) / '~'; C++ top-level cv dropped, arrays decay,
cvref kept; <= 200 bytes, longer ones keep whole leading entries and end
in '#<fnv1a64 of the full suffix>)'. The builder is iterative (cursor
walk, bounded declarator loop) and self-checks against the inverse.
- The inverse is anchored to the node's bare name wherever the name is
known: real QNs already end in ')' without being callables (a Java field
whose name is its declarator `X = f()`, make `$(VAR)` targets), and only
the anchor makes the inverse exact on them.
- Leaf splitters strip a suffix first (no-op today): registry by-name key
(indexed by the bare name for a suffixed callable), cbm_lsp_bare_segment,
cbm_pipeline_qn_class_method_tail, pxc_last_component, pxc_qn_leaf,
go_index_visit_node, collect_prod_defs, sg_qn_prefix_len.
- New unused fields: CBMDefinition.qn_sig_off, CBMResolvedCall.callee_sig
(relocated by result compaction), CBMRegisteredFunc.sig, CBMLSPDef.sig.
The lsp_surface codec is unchanged (no format bump in this change).
- get_code_snippet and trace_path gain base-QN tiers
(cbm_store_find_nodes_by_qn_base) that find nothing until suffixes exist.
- store: cbm_store_find_nodes_by_qn_suffix built its LIKE pattern in a
512-byte buffer (a long suffix was truncated into a pattern that matched
nothing) and did not escape '_'/'%' (`my_func` also matched `myXfunc`).
The pattern is now heap-built, escaped, and matched with ESCAPE '\'.
Proof: full sorted node+edge+lsp_surface dumps are byte-identical before
and after on 7 corpora (this repo, elasticsearch libs, Exposed, dotnet
System.Linq, System.Collections, coreclr/gc, a C corpus).
Refs #2061
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
 Glen KoandClaude
|
2fbf3fb977 |
fix(cli): fold per-user /home entry into the alias loop, add test seam
Address review on #2309: - Fold the /home/<name> handling into the existing #2175 alias loop via a shared activation_alias_map() that builds the mapped path in a stack buffer and returns activation_string_copy(); raw allocator sites in activation_transaction.c drop from 80 to 78 (baseline lowered). - Comment the deliberate difference: the per-user entry may resolve to a directory owned by the current account, the whole-alias case may not. - Add cbm_activation_transaction_set_home_root_for_testing() (test seams only) and a Linux test that fails without the per-user branch. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Glen Ko <gleko@amazon.com> |
||
|
Martin Vogel
|
5910c47a97 |
Merge pull request #2332 from DeusData/fix/issue-1916
fix(pipeline): compose axios.create baseURL into HTTP_CALLS (#1916) |
||
|
Martin Vogel
|
dc735c01fb |
fix(install): migrate v0.10.8 OpenCode agent profiles on upgrade (#2264)
Upgrading from v0.10.8 to v0.11.0 on a machine with OpenCode reported all
three OpenCode subagent profiles as "preserved modified profile" plus an
op=agent_install error each, and the activation stopped ("one or more agent
configurations failed; the published/current executable was kept"), although
nobody had edited those files.
Root cause: #1933 (v0.11.0) added the tool_search / tool_search_regex
permissions to the rendered OpenCode profile. Install and uninstall recognise
a profile as ours only when it matches the current rendering or one of the
known released renderings (the other access mode, the v0.9.1-rc.1 Codex
shape, the pre-tier Verify file). The v0.10.8 OpenCode shape was not in that
set, so cbm_text_migrate_owned_document classified the untouched v0.10.8
files as user-modified. Reproduced end to end: install with the v0.10.8
release binary, then install with v0.11.0/main, into an isolated HOME; the
three profiles are 1729/1880/1910 bytes, the same sizes as in the report.
cbm_render_graph_profile_opencode_v0108() renders the pre-#1933 shape
(both access modes), and install/uninstall now list it as a released
rendering, so those files are upgraded in place and uninstall removes them
as owned. A profile the user actually edited is still preserved. The
released-list assembly that install and uninstall duplicated moves into one
helper, and the two legacy renderers share one implementation (cli.c drops
three raw free sites; the memory-core baseline is tightened 173 -> 170).
The Codex and OpenCode op=mcp_install errors in the same report do not
reproduce from a pristine v0.10.8 config and depend on the reporter's own
config.toml / opencode.json; they are not addressed here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
77ab4bea87 |
fix(extract): definitions walk stack grows without a ceiling
walk_defs keeps its pending frames in a growable stack that was bounded by an 8M-frame ceiling (env CBM_WALK_DEFS_MAX). At the ceiling wd_push logged "extract.walk_defs_capped" once and then stopped pushing. Children are pushed last-to-first, so a file wider than the ceiling lost its FIRST definitions: a work cap decided graph content. Fix: no ceiling. The stack doubles on demand, heap frames now come from the memory core (cbm_realloc/cbm_free, class EXTRACT) instead of safe_realloc and raw free, and the doubling is overflow-guarded (int capacity and size_t byte count). The only thing that can stop the walk is an allocation failure, and that is loud: logged at error level as "extract.walk_stack_alloc_failed" (walker=walk_defs, pending, path) -- the same event the complexity walkers use -- and the file's result is marked has_error with "definitions walk: stack allocation failed", which the definitions passes already report as a per-file extract error. The arena-backed path (ctx->scratch) gets the same failure handling. Traversal order is unchanged. CBM_WALK_DEFS_MAX is removed. It was read only in wd_stack_max(); it was not documented anywhere (docs, --help, config, tests), so nothing else changes. extract_defs.c loses its raw free() call, so the memory-core baseline for the file drops from 5 to 4. Test (extraction suite): walk_defs_wide_file_extracts_every_definition extracts a C file with 1000 top-level functions with CBM_WALK_DEFS_MAX=256 set, and asserts every wd0..wd999 definition is present. RED on main (extract.walk_defs_capped limit=256; wd0 missing), GREEN with the fix, RED again with the production change reverted. The extraction (379), complexity (5), ast_profile (10) and semantic (35) suites pass. Real input: Linux kernel, index-only, fresh cache and runtime dirs, same machine. main and fix both produce 8,530,336 nodes and 15,764,128 edges, SEMANTICALLY_RELATED 607 each, and the full node set (label, QN, lines) and edge set (type, source QN, target QN) hash identically -- no kernel file approaches the old ceiling. Index wall time 210.0 s (main) vs 194.4 s (fix), within run-to-run noise. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
1d5ff68e50 |
chore(lint): one helper for pass_calls' transient-QN lookups
pc_module_var added a third compute-QN / find_by_qn / free sequence to pass_calls.c, growing its memory-core ratchet (23 -> 24). Route all three through pc_find_by_computed_qn, which leaves 22 raw sites, and ratchet the pass_calls.c baseline down to match. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
24afb562dc |
fix(graph): route the new scratch through the memory core
The lint chain stops at its first red target, so a clang-format failure hid the memory-core linter until PR CI: the coverage line-offset table, the Cypher direction swap and the search label each added a raw allocator site. The table now comes from cbm_alloc/cbm_free (extract class), the swap frees through safe_str_free, and the BM25 row array moves to cbm_calloc/cbm_free so mcp.c's raw-site count goes down (baseline 814 -> 813) rather than up. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
af3e003ad8 |
fix(mcp): one release point for handle_index_repository's argument strings
main fails the memory-core linter:
src/mcp/mcp.c: grew by 3 (814 -> 817)
PR #1723 (merged as
|
||
|
Martin Vogel
|
1d58385b51 |
refactor(mem): route this branch's allocations through the memory core
The memory-core linter is a ratchet: raw malloc/calloc/realloc/free/strdup sites may not grow, because allocations belong on one accounted path (src/foundation/mem_core.h) rather than spread across hundreds of call sites. This branch's own work had added 35 raw sites across nine files -- the parallel k8s prep, the configlink key matching, the worker pool's counted path, the Go shared stdlib registry, the spill namespace copy, the route service array and the namespace-name arrays. All of them now use cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/ cbm_mem_strdup. One was a latent bug rather than a style point: pass_k8s freed a buffer that k8s_read_file had allocated with raw malloc, so converting only the free would have mismatched the allocator -- the read function and every free of its result are converted together. The Windows free-space path widens its argument through the memory core instead of cbm_utf8_to_wide(), which allocates with raw malloc. Two files improved enough to tighten their baselines, which the ratchet requires so the gain cannot be quietly given back: sqlite_writer.c 92->74 (the index-cell arena), pass_k8s.c 13->6 and registry.c 19->17. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7b77fd4834 |
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (
|
||
|
Martin Vogel
|
da3258f3c7 |
feat(mem): a central allocation core, a linter that enforces it, and phase attribution in the index pipeline
Memory in this project is now allocated through ONE core. This lands the
core, the gate that keeps new code on it, and the instrumentation that lets
an index run say which class of memory grew in which pass -- the three things
the 2026-09-13 audit found missing.
WHAT THE AUDIT FOUND
There was no centralized allocation layer. foundation/mem.h was policy and
measurement only (budget, RSS, over_budget, memory map, phase marks) with no
alloc/free wrapper, so the budget could be observed after the fact and never
enforced. Raw allocator calls in the tree: 4086 across 92 files. The
extraction engine had already adopted arenas (1301 call sites); the graph
buffer -- which holds the 35 GB peak of a kernel index -- used none,
allocating every 64 B node, 48 B edge and every name/qualified_name/
properties_json individually.
Two further facts made a self-accounting core the only workable design:
1. cbm_mem_map_collect() walks THIS thread's mimalloc heap only. Walking
the process-wide heap is a data race TSan caught on macOS, so it is
deliberately not done, and an 18-worker index attributes almost
nothing -- the rest lands in residual.
2. The mimalloc global override is ON for Linux/MinGW and permanently OFF
for macOS (two-level namespace: this binary's free becomes mi_free while
system libraries keep allocating from the system zone, and a pointer
crossing that boundary aborts). On macOS ordinary malloc is served by
the system allocator; the startup audit reports owned_classes=0/6. Any
accounting that assumes mimalloc owns the pointer is blind on an entire
platform.
THE CORE (src/foundation/mem_core.{h,c})
cbm_alloc / cbm_calloc / cbm_realloc / cbm_mem_strdup / cbm_free, each tagged
with an allocation class (gbuf_node, gbuf_edge, gbuf_string, gbuf_index,
extract, semantic, dump, store, other). Per-class live bytes, live blocks and
peak, kept in atomics -- thread-safe by construction, identical on every
platform, independent of which allocator serves malloc. libc semantics are
preserved exactly (alloc(0) is freeable, free(NULL) is a no-op, realloc(NULL)
allocates, a failed realloc leaves the block intact) so adoption is a
mechanical rename and never a behaviour change.
No per-allocation header: a {class,size} prefix costs 16 bytes on ~100M blocks
at kernel scale, 1.6 GB of overhead to measure a memory problem. Sizes come
from the platform usable-size query (malloc_size / malloc_usable_size /
_msize), which is correct under either allocator and free. Where no query
exists (BSD) the request size is charged, which understates -- the safe
direction for a diagnostic. A mismatched class on free clamps at zero rather
than wrapping, so a small drift can never masquerade as a colossal leak.
Arena-backed allocators report in bulk through cbm_mem_class_add_external /
remove_external so their memory appears in the same table; the extraction
engine must not be rewritten to per-object allocation, that would undo the
batching that keeps its allocation count low.
THE LINTER (scripts/lint-memory-core.py, wired into make lint-ci)
Raw malloc/calloc/realloc/free/strdup/strndup outside the core is a defect.
4086 sites cannot migrate at once, so the gate is a ratchet: a checked-in
baseline (scripts/memory-core-baseline.txt) records each file's count and
the build goes red when any file grows, or a new file appears with any.
Files only ever go down; the baseline line is lowered in the change that
migrates the file, and --strict fails a stale baseline. Comments and string
literals are stripped first, so prose that mentions malloc( cannot trip it --
the security audit already bit on exactly that with fork(. Proven both ways:
one appended malloc turns it red naming the file (83 -> 85), restoring turns
it green. Scans src/ and internal/cbm/: cli, mcp, daemon, store, pipeline.
PHASE ATTRIBUTION (src/pipeline/pipeline.c)
cbm_mem_phase_mark and the new cbm_mem_class_log now fire at pipeline.begin
and at every pass.timing site: parallel_extract, the six sequential passes,
the eight predump passes, tests and dump_and_persist, with peaks reset per
index. Both instruments existed in foundation/ and were wired only into MCP
request handling, never into the index pipeline -- which is where the memory
is. Locating the kernel peak this week required an external RSS sampler
because nothing in-process could say which phase it was in.
PRESSURE PRIMITIVES (system_info.c, mem.{c,h}, platform.h)
cbm_system_available_ram() -- macOS host_statistics64 (free + inactive +
purgeable), Linux MemAvailable, Windows GlobalMemoryStatusEx; 0 when unknown,
never cached -- and cbm_mem_system_under_pressure(), true below 12.5% of RAM
available and false when unknown so nothing ever aborts on a guess. Landed
UNWIRED: an earlier draft used them to let the over-budget latch press on
whenever the system was not under pressure, and that permitted the very
overshoot this work exists to end (Decision A's tests at
test_pipeline.c:13481 and :13583 went red under it, correctly). They are the
backstop for the next step, admission control, which keeps a run under budget
before an allocation rather than measuring it after.
mem suite 10 new tests green; pipeline + mem 339 passed 0 failed; lint-ci
clean including the new gate.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|