The UI security audit (security-ui.sh, layer A1) forbids hardcoded
external URLs in graph-ui source — the callout's GitHub issues link
tripped it on all pr-smoke legs. Follow the established pattern for
external targets (/api/repo-info deep-links): /api/ui-config now
carries upstream_issues_url and the callout consumes it, rendering the
issue button only when the backend provides an https URL. The protocol
check uses a regex literal on purpose: a bare protocol string in source
also aborts the audit's URL extraction.
Verified: security-ui.sh passes locally; /api/ui-config serves the URL;
the callout opens with the prefilled link in the browser; frontend
build + tests green.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Clicking a node of the missed skeleton now opens a dedicated right-panel
callout instead of the standard node panel (code snippet and
callers/callees are meaningless for a not-fully-indexed file):
- explains the gap in plain words (best-effort detection; the file
content itself is ground truth)
- asks the user to have their agent summarize what fails to parse and
report it upstream so the edge case can be handled
- two working actions: a prefilled upstream GitHub issue (title
'Indexing gap: <file>'; body carries ONLY the file path and project
name, with an explicit add-snippets-only-if-shareable note) and a
copy-to-clipboard agent prompt (index_status -> summarize flagged
ranges -> file the issue) with visible copied feedback
Verified in the browser: skeleton click opens the callout with both
actions; frontend build + tests green.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Per review: no dedicated coverage tool.
- index_status now carries the coverage report (parse_partial + skipped
lists from the separate index_coverage table, plus the best-effort
coverage_note); get_index_coverage is removed. Tool description
updated so agents learn the coverage semantics from index_status;
index_repository and the response note now point there
- the read hook resolves coverage through index_status (same envelope
fields, unchanged note wording); verified end-to-end against a real
index (flagged file -> note with exact ranges, clean file -> silence)
- get_code_snippet responses from a parse_partial file now carry a
correlated coverage_note ("line range(s) X could not be parsed ...
the source above is ground truth") — the result names its file, so
the warning is precisely anchored. Entirely-skipped files cannot
appear here (no nodes), so no uncorrelated noise is possible; clean
files carry no note (asserted both ways in the e2e test)
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Check 8d still locked the matcher to the pre-#963 'Grep|Glob' and failed
the pr-smoke leg on all platforms. The matcher now includes Read (the
augmenter injects the coverage note when a not-fully-indexed file is
read; structurally non-blocking, so the issue-#362 gate hazard cannot
recur). The guard now pins the exact new matcher and still rejects
Search/catch-all creep.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Render the miss graph side by side with the code graph instead of
swapping layouts:
- /api/layout (code graph) now also attaches "missed_graph": {nodes,
edges, offset} — the shadow-project layout placed below the primary
cluster (same satellite pattern as linked_projects; -Y slot so
cross-repo satellites collide last). graph=missed stays the isolated
view for API users
- the UI paints the skeleton white and ghostly beside the galaxy,
auto-frames BOTH clusters on load, and navigates naturally: clicking
the skeleton flies the camera into it (file labels + detail panel),
clicking a code node flies back to the code side, and clicking empty
space while the skeleton has focus returns to the overview (the
galaxy can be entirely off-screen at that point)
- the sidebar toggle now shows/hides the skeleton (with a missed-file
count) rather than swapping the layout
Verified in the browser: overview composition, skeleton focus with
split.c labeled, and the empty-space return flight; ui/httpd suites and
frontend build + tests green.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Four refinements to the coverage signal:
- extraction: subtract DEFINITE recovery before flagging. Tree-sitter
error recovery plus the ERROR-descending def walker often re-extract
constructs inside a failed region (verified: `def broken(:` comes
back as a def); a region whose every line is covered by definitions
that START inside it is not a miss, and a fully recovered file is not
flagged at all. Container defs (Module/Package) don't count as
evidence, and partially covered regions stay flagged — the
#ifdef-split case keeps its flag because the first branch's function
is genuinely lost
- naming: the query_graph option is graph="missed" (the graph shows
ONLY misses — "coverage" was misleading); shadow project renamed to
"<name>::missed"; tool descriptions updated so agents discover both
the option and its semantics
- hook: the CLI-installed PreToolUse augmenter now also matches Read
and injects a coverage note when the file being read is listed as not
fully indexed ("line ranges X-Y could not be parsed — the file
content you are reading is ground truth"). Safe against the old
issue-362 hazard: the augmenter is structurally non-blocking (always
exit 0, additionalContext only), mirroring the Gemini matcher that
already includes read_file; matcher upgrade bookkeeping updated
- ui: "Missed files" toggle in the graph sidebar renders the miss
graph as a second graph option — /api/layout gains graph=missed
(same db file, shadow-project scoping; base project name validated
as before)
Tests: recovery-subtraction cases (recovered def unflagged, garbage
region flagged, trailing recovered defs keep the flag), CLI matcher
tests updated, e2e resilience fixtures switched to an unrecovered
miss; verified end-to-end: hook emits the note for a flagged file and
stays silent otherwise, /api/layout?graph=missed serves the miss graph
and the code graph is unchanged, frontend build + tests green.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Persist the parse-coverage signal outside the graph and make the misses
queryable both as a list and as a graph.
- store: new index_coverage table (project, rel_path, kind, detail) —
coverage is metadata ABOUT the graph and never mixes into the graph
tables; replace/get/free APIs with a deleted-file prune keyed off
file_hashes (the live-file set)
- store: materialize a derived miss graph under the shadow project
"<project>::coverage" (Project -> Folder chain -> File{kind,detail},
CONTAINS_FOLDER/CONTAINS_FILE edges), rebuilt from the table inside
the same transaction on every replace; invisible to list_projects
(which scans the cache directory, not the projects table)
- pipeline: coverage is written in the same persist step as every other
post-dump SQL write (hashes, ADR, FTS). Full runs replace the whole
set; incremental runs merge rows surviving from files not re-extracted
with this run's fresh entries, so a fixed file's flag clears and
deleted files prune automatically
- mcp: new get_index_coverage tool (per-project parse_partial + skipped
lists with the best-effort note); query_graph gains graph="coverage"
to run the same cypher against the miss graph; the File-node property
marker from the previous commit is dropped in favor of the separate
table — the code graph gains no coverage rows
- wording: constructs in flagged ranges MAY be missing — tree-sitter
error recovery still salvages some (verified: a function inside a
flagged #ifdef region and even a broken def were still extracted)
- tests: store round-trip/prune/shadow-graph unit test; e2e asserts the
response fields, the advertised coverage-graph query, code-graph
purity, and that the flag clears after the file is fixed (incremental
route)
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Files whose parse tree contains tree-sitter ERROR/MISSING regions were
silently indexed as if complete — constructs inside those regions are
absent from the graph with no signal anywhere (classic trigger: the
preprocessor-blind #ifdef-split-brace pattern in C).
- cbm_extract_file: set parse_incomplete and record the 1-based line
ranges of the top-most error regions (bounded at 64), serialized as
"start-end,start-end" into the result arena
- pipeline: record such files under the new phase "parse_partial"
(distinct from skips — the file IS indexed) in both the parallel and
sequential extraction paths; stamp the File node with
{parse_incomplete:true, error_ranges} on full and incremental runs
via a new cbm_gbuf_set_node_props (upsert survivor rules bypass)
- mcp: report parse_partial {files, count, truncated, note} separately
from skipped[] (skipped_count semantics unchanged); the per-run
logfile lists parse_partial rows; the index_repository tool
description documents the signal and the query_graph pattern to find
flagged files
- tests: new parse_coverage suite (RED-first, 7 cases incl. false-
positive guards and the region cap) plus an end-to-end resilience
guard asserting response fields, logfile, File-node marker, and the
advertised query_graph query
The signal is explicitly best-effort: a flag means constructs in the
listed ranges were dropped (prefer grep there); the absence of a flag
is NOT a completeness guarantee.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The npm and PyPI installers only ever fetched the standard (headless)
build, so the UI build (embedded graph visualization) was reachable only
through install.sh --ui or a manual archive download. Add a CBM_VARIANT=ui
opt-in to both installers that fetches the codebase-memory-mcp-ui-*
archive instead; the default is unchanged (standard headless build).
Document the per-channel opt-in in the README.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_cli_sha256_file shelled out to sha256sum/shasum, which is unportable:
the tool may be absent, and under cmd.exe the single-quoted path is not
dequoted so the hash of a downloaded release could never be computed on
Windows. Add a small in-process SHA-256 (FIPS 180-4, src/foundation/sha256.c)
and use it directly — no external tool, no per-OS branching, no popen. The
checksum test now covers three NIST vectors (empty, "abc", and a 56-byte
input that forces padding into a second block).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
High-density graphs washed out to a cyan-white blob: edges blend additively
and the per-node boost rewarded luminance, so ~80k overlapping edges plus the
white/yellow mid-tier stars saturated the center.
Compensate by density so contrast stays roughly constant as the graph grows:
- edges dim by ~1/sqrt(edgeCount) (they cause the blob) while nodes and bloom
stay at full strength up to 25k nodes and only ease gently past that, so the
bright-star look is preserved on moderate graphs;
- the per-node glow is now colour-aware by channel dominance instead of
luminance: blue hubs (high-degree) glow brightest, red leaves modestly,
white/yellow least — matching the star-class meaning and giving the graph
depth instead of a uniform white core.
Add a Display menu (edge brightness / node glow / bloom multipliers, 0–3×,
persisted) layered on top of the adaptive defaults, so the look is tunable
live. A selection is never density-scaled and stays bright against the
dimmed rest.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Add a node-budget control next to Refresh: type any count (snapped to
5,000 steps, 5,000 default, 10M max) and the graph reloads with that many
nodes plus every edge between them. The choice persists per project.
Loading gets real feedback: the layout response streams with a live
MB counter under an animated constellation (static under
prefers-reduced-motion), replacing the bare spinner.
Rendering scales with the budget: instance matrices are rebuilt only when
the node set or highlight changes instead of every frame, sphere
tessellation steps down as counts grow, and past 75k nodes the cloud
switches to soft point sprites — one position per node — so six-figure
budgets stay interactive. The truncation notice now names the loaded edge
count and points at the budget control.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The layout endpoint treated the default render cap (2000) as a ceiling:
an explicit max_nodes above it was silently clamped back down, so the UI
could never show more than 2000 nodes regardless of the request. Make the
default a default — an explicit request is now honored up to a hard
ceiling of 10M nodes; CBM_UI_MAX_RENDER_NODES still adjusts the ceiling
for constrained deployments. The default budget rises to 5000, and the
layout refinement pass scales its iteration count down past 100k/500k
bodies to keep very large budgets responsive. Edges need no changes: the
layout already returns every edge between loaded nodes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The policy was only attached to embedded-asset responses, but browsers
enforce a CSP from the document response; on subresources it is inert.
Attach the same header where index.html is served.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Add CBM_ALLOWED_ROOT to the environment-variable tables in README.md and
docs/CONFIGURATION.md: when set, index_repository refuses a repo_path that
resolves outside it; unset imposes no restriction.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The UI makes no external network calls, but nothing enforced that at the
browser. Emit a CSP whose directives name no external host, so a future
dependency or injected content cannot load or connect off-origin;
connect-src 'self' confines fetch/XHR/WebSocket to the local server. The
'self'/data:/blob:/inline-style/wasm allowances cover the bundled app's own
needs (React inline styles, three.js textures/workers). Header emission only,
no rendering change expected — confirm the 3D view in a browser before
release.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
index_repository canonicalizes repo_path but imposes no workspace boundary,
so a caller-supplied path can resolve anywhere on disk. Add an opt-in
CBM_ALLOWED_ROOT: when set, a repo_path that resolves outside it is refused
(reusing the same containment check the read sinks use). Unset by default, so
the standard index-the-given-path behaviour is unchanged; deployments where
repo_path may be influenced by an untrusted caller can set a boundary.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The UI handlers accumulate JSON with `pos += snprintf(buf + pos, size - pos,
...)`. snprintf returns the length it WOULD have written, so on truncation
pos ran past the end of the buffer and the next call's `size - pos` wrapped
to a huge value and wrote out of bounds — reachable through the directory
browser, whose readdir entries can fill the buffer before the trailing
parent/roots appends. Route every accumulator through a clamping
http_appendf() helper that pins pos at the buffer size on truncation. Adds a
forked-child regression test that browses a directory wide enough to fill the
buffer.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The macOS digest path invoked `shasum -a CBM_SZ_256` — a literal string
(not a macro; string literals do not expand) naming an invalid algorithm, so
the digest always failed on macOS, and the self-update path then treated a
could-not-verify result as non-fatal and installed the download regardless.
Use the correct algorithm, and install only a positively-verified download:
a mismatch, a missing checksum entry, or an unavailable hash tool now all
abort. Adds a known-vector digest regression test.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
base_branch is spliced into `git diff --name-only "<base>"...HEAD`. The
existing check rejected shell metacharacters but not a leading '-', so a
value like --output=<path> was passed to git as an option and wrote the diff
to an arbitrary file. A real git ref never starts with '-'; reject it
alongside the metacharacter check.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The graph-UI server binds to loopback only and reflected a localhost Origin
for CORS, but did not inspect the Host header — so a request routed to the
local port under a foreign name (a rebinding DNS record, a proxy) was served
normally, reaching the state-changing endpoints. Parse the Host header and
refuse any request whose Host is not a loopback address before dispatch; a
request with no Host (local HTTP/1.0 tooling) still proceeds. Adds a
live-socket regression test.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A source path never legitimately contains a newline or carriage return, and
on the Windows search filelist those bytes are the per-entry record separator
(NUL can't be used there — it breaks PowerShell Get-Content). Skip any indexed
path containing one so a crafted path cannot inject an extra file into the
scan set.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
safe_stat rejected POSIX symlinks (lstat + S_ISLNK) but on Windows returned
wide_stat directly, which follows junctions / reparse points — so a junction
inside an indexed tree could be walked into and files outside the project
root pulled into the index. Check FILE_ATTRIBUTE_REPARSE_POINT and skip such
entries, mirroring the POSIX symlink skip.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
get_code_snippet resolved a result path and confirmed it stayed under the
project root before reading, but the search_code source-attach path built
the same kind of path and read it with no such check — so a result whose
indexed path resolves outside the root (a `..` segment, or a symlink /
junction followed during discovery) could be read back into the response.
Extract the existing containment check into a shared cbm_path_within_root()
and route both read sinks through it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The import path allocated the destination from the plaintext original_size
field in artifact.json and passed that value to the decoder truncated
through int. A large real database already wraps that cast (a full-kernel
index is ~14 GB), and a doctored field could additionally desync the decoder
capacity from the actual buffer. Read the decompressed size from the zstd
frame's own content-size header, require the metadata field to match it, cap
the total, and run the decoder with a size_t capacity equal to the
allocation. cbm_zstd_decompress now takes size_t lengths and returns int64_t.
Adds a regression test that a mismatched original_size is rejected.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The result projection is materialized per row into fixed-width stack
arrays (CBM_SZ_32 columns) in execute_return_simple and its siblings,
indexed by the parsed RETURN item count — which the parser left
unbounded. A RETURN/WITH clause with more than 32 projection items
wrote past those arrays. Reject an over-wide projection at parse time,
with a fork-isolated regression test driving a 48-column RETURN.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The guard's sparse defs scaled WITH the input (one per 2000 lines), so
the fixture inherited a slice of the separate per-def sibling-scan cost
(O(defs x siblings) — its own tracked finding): an n^2/2000 term that
is negligible on clang-macOS but dominant enough on windows-CLANG64
ASan to push LINEAR walk code to a 43x measured ratio, over the 40x
bound. Ten defs at fixed positions regardless of size keep the
anti-vacuous breadth check while the sibling-scan term stays 10 x n =
linear. Measured after the change: fixed code 22x, pre-merge quadratic
walk 339x — the discriminator now has ~2x headroom on the green side
and ~8x on the red side across all three toolchains measured.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An absolute wall-clock bound conflates machine speed with complexity
class: the gcc-13-ARM ASan CI leg runs this extraction ~200x slower
than clang at measured-perfectly-linear complexity (27.1/54.2/108.3/
214.8s for 50k/100k/200k/400k siblings) and flunked a 30s bound on
LINEAR code — the leg has always been the slow one (71min in the last
fully green run, unchanged since).
Assert the complexity class instead: extract the same comment-sibling
fixture at 20k and 400k lines and bound the growth ratio at 40x.
Calibrated from measurement, not models — the ASan builds carry a large
linear per-line baseline that dilutes small-growth ratios (at 8x growth
the known-quadratic pre-merge walk measured 22.4x and slipped under a
24x bound), while at 20x growth linear code measures ~20x on both
toolchains and the quadratic walk measures 64x. Verified RED (64.4x,
fails) against the pre-merge walk_defs and GREEN (19x) on the fix; the
test now prints both timings unconditionally for future triage.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The no-skips lint gate forbids bare SKIP(); the parallel-recovery guard
is genuinely platform-specific (POSIX fork harness — Windows coverage
comes from the smoke invariants), so it uses SKIP_PLATFORM like the
neighboring supervisor guards.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An 81k-file TypeScript corpus previously never finished indexing. Three
stacked defects, found by stack-sampling the live grind and reading the
supervisor's quarantine records:
1) walk_defs pushed children via index-based ts_node_child(i), which is
O(i) per call in tree-sitter — O(n^2) per wide node. A 3.5 MB file
with ~580k flat comment siblings needed ~1.7e11 iterator steps and
hung extraction past the supervisor's quiet-timeout. All six descent
sites (including the default loop) now collect children linearly with
a TSTreeCursor (wd_collect_children); small nodes keep the direct
index path. Guard: extract_wide_flat_file_is_linear (400k comment
siblings — the monster's exact shape — bounded at 30s).
2) The sequential cross-LSP driver handed the FULL def list to a full
per-file registry build+finalize — O(files x defs); 74% of samples
sat in build_qn_index. The parallel resolve worker's dispatch
(module-def filter -> shared prebuilt registries -> filtered
fallback) is now extracted into cbm_pxc_dispatch_file and drives
BOTH pipelines — one path, one semantics. The shared registries
live in a caller-owned arena that outlives the calls pass
(resolved_calls borrow registry strings; freeing at pass end was a
use-after-free ASan caught in pass_calls). Rust is exempt from the
def filter: Cargo-manifest cross-crate resolution needs defs from
OTHER workspace crates that the own+imports filter drops (this
starvation predated the unification on the parallel path). Guard:
pipeline_seq_ts_cross_uses_shared_registry (full builds 40 -> 0,
cross-file edge preserved).
3) The crash supervisor's recovery re-ran the worker SINGLE-THREADED to
keep one exact marker. At scale that fell into the sequential crawl,
was killed as a hang mid-pass, and the stale extraction marker got
four innocent files quarantined, one 15-minute retry at a time.
Recovery (and the terminal partial run) now re-run PARALLEL; the
marker is an append journal ("S <rel>" / "D <rel>", written by
extraction and both resolve drivers), the suspect set is the open-S
in-flight set, and a file is quarantined only when it recurs across
two consecutive failed runs — one per round, oldest open S first;
disjoint consecutive sets stop the loop rather than blame an
innocent. Guard: index_recovery_parallel_quarantines_crasher
(single-threaded spawn count must stay zero; verified RED against
the old loop).
Also: parse_ts_type_text gains a dynamic work budget proportional to
the input (1M + 64x source bytes, CBM_TS_TYPE_BUDGET override, warn
once then degrade to unknown) replacing unbounded recursion cost.
Measured on microsoft/TypeScript (81,398 files): infinite -> 33s cold
full index, 295k nodes / 779k edges, zero dangling edges, zero files
skipped — the monster file itself now indexes cleanly. Full 9-language
sweep green with no quality flags; kernel unchanged at 390s / 8.5M
nodes / 0 dangling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The supervised-worker fast path deliberately skips the pipeline
teardown (the OS reclaims the process wholesale) and pairs that with
_Exit, which bypasses atexit/LeakSanitizer. The production worker in
run_cli has both halves; the test binary's worker emulation
(tf_maybe_run_index_worker) only had the first: it returned normally
through main(), LeakSanitizer ran at exit, reported the intentionally
unfreed pipeline and forced exit code 1 — so the supervisor read a
healthy index as worker_failed and IDX832 went red on exactly the
platforms whose ASan builds run a leak pass (all four Linux legs;
macOS/Windows ASan have none). Reproduced and verified fixed in an
ubuntu:22.04 arm64 container: reap outcome=clean, mcp suite 140/0.
Mirror the production exit: deliver the response file, fflush, _Exit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_mem_init is init-once (guarded by g_initialized), so the
backpressure test's setenv(CBM_MEM_BUDGET_MB=1) + re-init 'restore'
was a silent no-op: whichever init ran first fixed the budget for the
whole process. In the full runner this test's init won, locking a 1 MB
budget that leaked into every later budget consumer —
mem_over_budget_low_rss red on every platform, and all subsequent
pipeline tests silently running with backpressure permanently engaged.
Add cbm_mem_set_budget_for_tests(): writes the budget directly without
touching the init guard. The test now saves the caller-visible budget,
sets 1 MB via the hook, and restores before asserting. Env dance
removed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The deterministic same-QN collision rule picked the lexicographically
smallest (label, file_path, start_line, name), which silently inverted
the documented upsert/merge semantics: re-upserting a QN with updated
fields kept the OLD content (gbuf_upsert_updates,
gbuf_upsert_same_qn_updates_all_fields, gbuf_merge_overlapping_qns all
red on every platform).
Replace it with a single mixed-direction total order applied in both
cbm_gbuf_upsert_node and merge_update_existing: smallest file_path,
then LARGEST start_line, then largest name/label. Still a pure content
function of the two candidates (commutative, associative,
scheduling-independent — multi-threaded runs stay byte-identical,
verified by the xfs MT x4 fingerprint guard), and line-descending
within a file restores the classic contract: a later definition in the
same file replaces the earlier one, and a full tie is the same entity
refreshed in place.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The header referenced the license identifier of a third-party library's
bundled dependency while explaining that the implementation deliberately
does not use that library. License scanners match the bare identifier
string and misclassify the file. Reword to state provenance directly:
written from the published papers, nothing vendored or derived.
Comment-only change; also picks up the formatter pass on this file.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This reverts commit 0475a880e5735ef5cb0b0d99532e552096e5a2d5.
Measured on the linux kernel: peak RSS unchanged (23.95 vs 24.4 GB, within
run noise) at +22 s wall -- the extraction peak turned out to be monotonic
retained-tree/source accumulation, which is schedule-independent, not the
concurrent giant transients this queue bounded. Net negative as measured;
the retention-side fix is parked for post-release.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase-aligned RSS attribution located the global peak (24.4 GB at t=47s,
~14 GB transient above the retained floor) INSIDE parallel extraction: the
size-descending queue put the kernel's 12 largest files into 12 workers
simultaneously, each holding a multi-MB source + its 10-50x TSTree + arena.
Every post-extraction optimization sat below this waterline.
Replace the single biggest-first cursor with a dual-ended claim queue over
the same size-sorted list: big_workers = clamp(workers/6, 1, 3) claim from
the BIG end while the rest rush the small end, so at most big_workers giant
working-sets coexist -- a structural bound, not a scheduling heuristic (a
static interleave cannot bound concurrency under dynamic pulls: smalls
finish in ms and workers converge back onto the giants). Giants still start
at t=0, preserving the LPT no-straggler property; workers never idle while
either end has files. Per-file claim flags (atomic exchange) make the
crossover race-free; end-of-queue rescans are O(n) bytes once.
Intra-file parallel parsing was researched and ruled out: tree-sitter has
no such concept (incremental != parallel; one parser per thread is the
model), and data-parallel parsing theory (PAPAGENO through SLE 2025 cyclic
OPG work) requires operator-precedence grammars with local parsability --
none of the 159 vendored GLR grammars qualify, and C's preprocessor makes
byte-chunking unsound.
Output is schedule-independent since the determinism fix and verified so:
xfs sorted edge set byte-identical to the previous commit. pipeline 216
passed. Worst case (all-giant repo) unchanged by design.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two borrow-don't-copy wins from the layer-duplication analysis (a hot string
was resident 4-5x across extraction arenas, gbuf, registries, defs and
tokens). Graph output verified byte-identical.
1. Resolution registry (registry.c): every registered definition strdup'd its
qualified name TWICE (exact-map key + by_name array item) and its label
once (~8.5M label strdups). The by_name items now borrow the exact map's
single owned key (same lifetime, freed in one place) and labels intern
into a small per-registry pool (<= ~30 distinct). ~280 MB + ~17M
allocations on the kernel.
2. Semantic tokens (pass_semantic_edges.c): all_tokens held one strdup per
token INSTANCE — identical tokens ("xfs", "error", ...) duplicated across
hundreds of thousands of functions. Each tokenize worker now interns into
a per-worker pool (key == value == the single owned copy; no contention,
content-identical strings so downstream corpus/tfidf output is unchanged);
all_tokens slots borrow. Ownership moves from per-func frees to pool
frees at pass end.
Suites: semantic 33, simhash 24, pipeline 216, registry 54, incremental 161.
xfs: two MT runs byte-identical, and byte-identical to the pre-change graph.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel