The *.sh eol=lf rule landed via #1314, but the git hooks
(scripts/git-hooks/commit-msg, scripts/hooks/pre-commit) are extensionless
and were still at the mercy of core.autocrlf. Add their explicit entries,
plus the contract test from #1272 wired as scripts/test.sh Step 0t: every
shell entrypoint (*.sh + both hook directories) must carry an eol=lf
attribute, with a matched-zero-files guard so a broken glob can never pass
vacuously.
Distilled from #1272: the diagnosis (CRLF checkouts breaking shebangs under
WSL/MSYS), the hook-file coverage, and the guard design are @xumian520's;
verified RED without the *.sh rule (105 uncovered entrypoints) and green
with it (107 files).
Co-Authored-By: xumian520 <126989134+xumian520@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Second mechanism of #1196, exposed by the reporter's v0.10.6 retest: every
relationship-expansion site capped its output buffer at bind_cap*10, so edges
past the cap were silently dropped BEFORE WHERE and aggregation ever ran.
count() then reported the scanned prefix as if it were a fact — 9,360 of
13,691 DEFINES field-measured at --max-rows 1000 — and a LABEL on the source
did not protect you (the label workaround only ever fixed source
enumeration, which #1323 already made exact). max_rows is an output-row
limit per the public header; projection already enforces it.
All five capped sites now share one growable append (geometric growth,
size_t sizing): the per-hop expansion, its fixed/variable-length helpers and
process_edges, the bound-terminal driver, and the cross-join outer buffer.
Only allocation failure stops materialisation; match_count stays truthful
either way, so the #627 OPTIONAL contracts hold (a saturated buffer can no
longer exist, and the fallback rows share the same append). The #601
deadline still bounds pathological time, and hop caps keep bounding depth —
this removes only the silent row-dropping.
Regression test: 2 labeled sources x 30 edges with max_rows=2 — the old cap
returned count=20; ground truth 60 now holds, and the list form returns
exactly max_rows rows. Proven RED before and RED again on revert; cypher
181/181 and mcp suites green, including every OPTIONAL/#627 semantics test.
Fixes#1196 (together with #1323, which fixed the unlabeled source scan).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The conflict's shared tail (#endif + closing brace) was emitted once for
two adjacent Windows tests; re-add the closer so both tests compile.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Trivial insertion-point collision in tests/test_mcp.c: both this branch and
main (#1704's UTF-8 pin) add a Windows search_code test at the same spot.
Both tests kept.
PowerShell 5.1 encodes stdout for a native-process pipe in the console
OEM codepage, so raw search content containing characters the inherited
CP cannot carry (Cyrillic under CP437/850, ...) reached
collect_grep_matches as '?' — and whether it degraded depended entirely
on which console the server happened to inherit. That surfaced as the
intermittent test_mcp raw-Русский mojibake on the windows CI leg and
means real Windows users in a default console get '?' for all
non-ASCII search_code raw content.
Every generated command now pins [Console]::OutputEncoding to UTF-8, so
the pipe is codepage-independent by construction. The read side needs
no pin: Select-String decodes BOM-less UTF-8 via .NET StreamReader
defaults. A Windows-side builder test asserts all five command variants
carry the prelude.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#1181 gave list_projects pagination parameters, retiring the last
empty-properties schema — and the two #1359 regression tests leaned on
list_projects as their live zero-argument example, so main went red the
moment the merge train composed (the PR was green on its July base, which
predated these tests).
The gate's schema→decision core is split behind a CBM_CLI_ENABLE_TEST_API
seam, so the zero-argument branch stays pinned directly (empty properties,
absent properties, populated properties) regardless of what the registry
ships; list_projects now asserts its NEW truth (piped args accepted, TTY
still refused); and the schema↔gate parity sweep keeps running over every
tool without the impossible >=1 zero-argument floor. Production behavior
is unchanged — this is the tests catching up with an intended schema
change, plus a seam so they never again depend on a shipped example.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The v0.10.6 release run failed deterministically at verify:
BLOCKED: expected scan object is missing:
objects/scan-3099e91c...--codebase-memory-mcp.exe
exclude-rescanned-selected-objects.sh (added after v0.10.5, first exercised
by this release) deliberately deletes the selected executables from the
surface-scan directory — their bytes were already scanned as candidates and
re-submitting identical bytes re-rolls a probabilistic classifier — and
writes binaries/virustotal-withheld.tsv. But check-virustotal.sh still
received the pre-withhold scan-set listing all sixteen objects and failed
closed on the first missing file. The rework's two halves never talked.
The gate now accepts an optional VT_WITHHELD manifest (strict parse: v1
marker, the stated reason required, sha256-keyed rows): an expected-set row
whose hash the manifest vouches for is exempt from the on-disk and
action-output contracts, while everything else keeps the strict path.
Fail-closed properties preserved and extended:
- no VT_WITHHELD -> byte-for-byte previous behavior (candidate
stage and dry-run call sites are unaffected;
verified against the original failure)
- withheld object present -> blocked (inconsistent staging)
- hash outside the set -> blocked (spurious withhold)
- everything withheld -> blocked (scan would cover nothing)
- mismatched object name -> blocked
vt-results.tsv keeps its exact shape (scanned objects only) — the release
notes table already uses the candidate results, and the withheld manifest is
now preserved with the rest of the evidence artifacts. release.yml passes
VT_WITHHELD only in the verify stage, right after the withhold step.
Verified offline with a fixture reproducing the release failure verbatim
plus the four negative cases above; the positive case passes staging and
association validation and proceeds to VT polling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Re-stamp the runtime DACL only when it is actually wrong, and repair cache
children left unusable by the pre-v0.10.3 DACL regime.
The unconditional per-start re-stamp rewrote an already-correct security
descriptor and propagated it to children (#1601 counted eleven no-op
"Security change" USN records against one _config.db in a day), and every
rewrite is a window in which a concurrent atomic publish can be refused
DELETE on the destination (#1620). The old regime's PROTECTED,
non-inheritable ACE also left every child born unusable — the 0-byte
worker-log class behind #1416's diagnosis — so the secured directory now
walks its regular children and repairs any with an empty DACL or a foreign
owner.
The fast path is gated on the ADOPTION-level predicate, not the general
secure() check: lock-directory adoption (private_win_owner_only_dacl)
demands the exact protected owner-only single-ACE descriptor the stamp
writes, while secure() also admits SYSTEM/Administrators ACEs. A fresh
directory with an inherited DACL passed secure(), skipped the stamp, and
stranded every subsequent lock adoption — 59/77 daemon-suite failures on
the real Windows VM. With the ported predicate (SE_DACL_PROTECTED,
single non-inherited owner ACE, FILE_ALL_ACCESS/GENERIC_ALL) the same VM
runs 77/77 and the full suite 7346/0.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Codex sanitizes stdio MCP subprocess environments to the names listed in
env_vars. Since #1645 CBM_RUNTIME_DIR relocates the daemon rendezvous, so a
Codex subprocess that does not receive it looks for the daemon in the DEFAULT
location and never finds it — the same silent client/daemon split
CBM_CACHE_DIR caused in #1562. Both names decide WHICH daemon a process talks
to and are now forwarded unconditionally (forward-if-present semantics);
behavioural knobs (log level, workers, budgets) deliberately stay
unforwarded — that broader list remains #1664's open enhancement question.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
gotspatel's live opencode.json stores our entry with backslashes
(`C:\...\codebase-memory-mcp.exe`) while the installer compares its own path
with forward slashes — the same file on disk, refused over the separator
spelling, so op=mcp_install failed on a correctly-installed machine (and on
Windows the dead-path probe rightly reported the binary PRESENT, which turned
the mismatch into a hard refusal).
Ownership comparison now treats `\` and `/` as equal everywhere and folds
case on Windows only, where the filesystem is case-insensitive; POSIX
byte-exactness otherwise holds. The annotated entry that names this binary is
recognised as already satisfied and preserved byte-for-byte.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
PowerShell 5.1's `Set-Content -Encoding UTF8` writes a BOM, so real
Windows-authored Hermes configs start with EF BB BF — and both edit ops
failed content-independently (the reporter's 26-byte reproduction is their
23-byte file plus exactly this BOM; reproduced RED on macOS with the same
bytes, so the platform was never the variable).
The document read now validates past a leading BOM and yaml_doc_init treats
it as a prologue: the first line's structure starts after it, the key lookup
still sees our own section when the BOM immediately precedes it (guarded by a
dedicated no-duplicate-section test), and every edit splices interior ranges,
so the BOM survives writes byte-for-byte. Non-document inputs — keys, entry
blocks, identity scalars — keep the strict no-BOM rule.
Also makes the moved-entry cli test fixture platform-correct: the Windows
dead-path probe can only prove a fixed-drive path absent, so the Windows
branch uses one; a POSIX-shaped path is refused there by design.
Fixes#1656.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two install failures with the same root theme — the entry we write is a
compatibility contract with the agent's parser, and both sides of that
contract needed repair:
goose (#1675): ExtensionConfig::Stdio declares `name` as a required serde
field with no default, and goose's loader silently drops entries that fail to
deserialize — install reported success and the extension was invisible. The
goose block now carries `name: codebase-memory-mcp`; the non-goose YAML
schema stays name-free. A CBM_CLI_ENABLE_TEST_API seam asserts the exact
block bytes per schema.
annotated MCP entries (#1630, the deferred field-merge): dbd20eaa recognised
an entry the client annotated ("enabled": true beside our command/type) but
could only leave it untouched, because replacing the whole entry would drop
the client's keys. config_json_like gains
cbm_json_like_replace_field_raw_if_unchanged — splice ONE member's value,
preserving every other byte (comments, ordering, client keys) — and the
upsert flow uses it on the two AUTHORIZED repair channels only:
- a relocating update (the entry names the previous managed binary), and
- the existing Windows dead-path probe, which previously fell back to a
wholesale rewrite and lost the annotations.
POSIX keeps its doctrine unchanged: a config-supplied path is never trusted,
so a moved-looking entry without that authority is preserved byte-for-byte
and install fails loudly (cli_editor_mcp_preserves_unrecorded_posix_absolute_
entries_without_probe holds). All repair/refusal paths are covered by tests
proven RED on the unfixed flow.
Fixes#1675.
Fixes#1630.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Four legal-YAML constructs from the reporters' real Hermes configs made
`install` fail permanently (any one of them aborted mcp_install and/or
pre_llm_hook_install):
- exact empty flow collections as values (`plugins: []`, `tool_choice: {}`)
— now validated key-only in both the mapping-body and sequence document
scans, mirroring the #1673 empty-mapping exception; non-empty flow
collections stay rejected.
- block sequences at the same indent as their mapping key (column-0 `- item`)
— item lines directly after a value-less key are structure, not malformed
keys, in the root walker, the key matcher, and the sequence mapping-range
walker.
- double-quoted scalars continued across lines with a trailing `\` — the doc
loader now precomputes per-line continuation flags; continuation lines are
value bytes every structural walker skips, and a document ending inside an
open continuation stays an error.
- mid-word quote characters in plain scalars (`LET'S`) — quotes are scalar
indicators only at a node start (range start, after `:`, after `-`),
exactly like the #1639 anchor/alias rule; real quoted values keep their
protection.
Byte-identity alone also froze users on canonicals older releases wrote:
galaxy's entry had `command:` unquoted, and the goose block gained `name:`
(#1675), so the existing entry was declared FOREIGN forever. An entry under
our key now repairs when it parses as a known prior shape (single command
line, or the pre-name goose block) with a codebase-memory-mcp[.exe] command
basename; anything else stays FOREIGN and the file untouched.
End-to-end: both reporters' full configs (iandol 15.6 KB, galaxy 15.2 KB) now
install with zero agent_config errors, every original line byte-preserved,
and the goose upgrade path rewrites the old block in place. Each construct
carries a distilled regression test proven RED on the unfixed editor.
Fixes#1631.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
ts_eval_expr_type and ts_signature_for_call are mutually recursive: resolving
a call evaluates its argument expressions once per lookup path (method
dispatch + namespace fallback), and in tsc-compiled spread files the first
argument is itself the next nested Object.assign(...) call — the same subtree
re-evaluates once per enclosing level, 2^n total. The TS suite's
objectSpreadRepeatedComplexity.js (3.6 KB, 5 nodes) measured 20.4 s; with the
memo its eval cost is zero within measurement noise of a one-file control,
and the microsoft/TypeScript corpus drops 37.5 -> ~24 s warm (nodes
byte-identical, edges within the known scheduler jitter).
Expression types are position-pure within a file pass (one node = one scope
path; the per-file walk is single-threaded and deterministic), so one eval
per node is the correct semantics, not a cache trade-off. The memo is a
per-file, arena-backed, linear-probe table keyed on TSNode.id. Results
produced under a depth-cap or budget bail are never stored: both bail sites
bump a degradation counter, and a store only happens when the subtree
completed clean — a degraded UNKNOWN can therefore never shadow a later full
evaluation.
The regression guard asserts work, not wall-clock: the nested-Object.assign
shape must complete without exhausting the deterministic eval budget, read
back through a new CBM_ENABLE_TEST_SEAMS accessor pair. The seam lives in the
lsp_all unity object, so GRAMMAR_CFLAGS_TEST/TSAN now carry the seams define
(test artifacts always have seams; prod never does). Verified RED without the
memo (budget exhausted, suite 47.8 s) and green with it (3.2 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_arena_alloc(arena, 0) returns NULL, so the java and cs cross-registry
builders read a def_count of 0 as OOM and returned NULL — a corpus with no
files of that language silently lost its shared registry (and the seal tests
caught exactly that: cbm_cs_build_cross_registry(&arena, NULL, 0) == NULL).
Guard the partition alloc behind def_count > 0; the empty registry is still
built, finalized, and shared.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The no-skips lint gate (scripts/check-no-test-skips.sh) rightly rejected the
throughput-report test's two SKIP() calls:
- CBM_SKIP_PERF=1 is deliberate operator configuration, not a hidden
environment failure: reporting is off by request, so the test PASSes with a
stderr note instead of skipping.
- an uncreatable report dir IS an environment failure and now FAILs with the
remedy in the message (set CBM_COMPLEXITY_REPORT_DIR), per the policy text.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
ts_eval_expr_type was depth-capped but work-unbounded: crafted expressions
(the TS test suite's repeated object spreads) stay under the depth cap
while fanning out. Charge the same per-file budget the type-text parser
uses, at 16 units per entry (an eval entry does ~two orders of magnitude
more work than a text-parse unit), degrading to UNKNOWN on exhaustion.
Honest status: this hardens the documented budget design, but the known
3.6 KB spread-bomb baseline file still measures ~11 s in-corpus — its
entry path into the evaluator apparently runs unarmed and is recorded as
an open lead (zero budget warnings observed). Suites green (ts_lsp,
extraction, complexity).
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two measured changes on the TypeScript corpus (81,397 files; v0.9.0 17.9 s,
before 44.1 s, after 37.5 s):
- cov_rebuild_shadow_graph upserted every directory segment for every
failure row — 13,243 parse-partial baseline files under one tests/
subtree meant ~80k redundant node/edge round-trips, 9.1 s of a 9.2 s
coverage_replace. An in-rebuild path->id map creates each directory once;
identical graph (edges deduped by unique key before, absent now).
Coverage block 9,162 -> 2,920 ms. Sub-block timings
(publish.timing.coverage: del/rows/prune/meta/commit + row_count +
detail_bytes) are kept — the caller-level number could not name the
culprit.
- JS/TS export_statement is an import CONTEXT only in its re-export forms
(source field, or a bare specifier list without a declaration). The old
is_export_of_declaration blacklist missed TS-only forms
(ambient_declaration, function_signature, module_declaration), running
declare-heavy subtrees (.d.ts, export namespace) behind inside_import:
suppressed usages + per-identifier ancestor walks. Positive detection
replaces the blacklist; +3,954 restored usage edges on the corpus,
nodes identical. (Measured perf-neutral here — kept for correctness.)
Suites green incl. store_nodes/edges/search, mcp, extraction, ts_lsp,
complexity, and the 53-language calls-breadth contract.
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Four measured changes on the dotnet/runtime corpus (58,656 files; every
step's numbers below are from full-corpus runs on the same host, baselines
captured this session; v0.9.0 = 444.3 s):
1. Two-phase cs registry build (types -> finalize -> funcs), the same
pre-finalize linear-lookup disease fixed for Java: prepare 280 -> 140 s.
2. C# import-context carve-out: cs_import_types lists namespace_declaration
(for namespace-name mapping) and using_statement (C#'s RAII block — a
grammar-name collision), so EVERY namespaced C# file's whole body ran
with inside_import=true. That both suppressed ordinary usage extraction
under namespaces and sent every identifier through the ancestor-walking
import-binding check (tree-sitter's ts_node_parent re-descends from the
root, so wide files went quadratic: one 147 KB JIT torture file cost
490 s; 6.8 s after). Only using_directive / namespace_use_declaration
open an import scope now. Restores the suppressed usages:
edges 4,291,387 -> 5,869,093 (+37%), nodes identical.
3. The usages walker maintains call/import ancestry as enter/exit counters
on its explicit stack instead of per-node ancestor re-walks
(extract_usages.c had grown from 6 to 100 ts_node_parent calls since
v0.9.0; the two per-node gates are now O(1) with semantics preserved —
strict ancestors only, emit before self-count).
4. Registry short-name indexes replace the two remaining full scans:
cs_lookup_extension walked all 963k funcs per unresolved invocation
(now the existing free-func short-name iterator, first-match order
preserved via min-index selection), and cs_resolve_type_name's step-9
fallback scanned every type per unresolved name IN BOTH the builder and
per-file resolution (new type_short index in finalize, same
reverse-insertion ascending-order pattern, best-score ties keep the
first-in-registration-order winner). Builder 140,095 -> 500 ms; resolve
cross-LSP CPU 5.5M -> 317k ms (us_per_file_per_kdef 191 -> 11).
End state: 449.9 s wall (1.01x of v0.9.0) with +48.6% edges vs v0.9.0 —
per-edge cost 32% BETTER than v0.9.0. Extract's remaining 359 s is the
24 MB hugeexpr1.cs parse floor both versions pay.
Guarded by the complexity suite; cs_lsp/extraction/edge/lang-contract
suites green including the 53-language calls-breadth contract.
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Finding #1669 took an 11-corpus A/B across two release binaries. This
suite makes that bug class fail a unit test in seconds, on every
platform, from tiny corpora.
Method: build k and 2k REPLICATED module copies, run the full in-process
pipeline on both, assert counter RATIOS. Independent copies mean every
extensive quantity — nodes, edges, Σ per-file registry defs — must grow
linearly (ratio ~2). A files x corpus coupling makes per-file work itself
grow with k and lands at ratio ~4. Ratios expose the exponent regardless
of absolute scale, so 60-120 files suffice.
Verdicts are pure functions of (code, input): gates ride ONLY on
deterministic work counters and data-product counts, never on wall time.
Throughput (nodes/s, edges/s) is information-only, written to
private/benchmarks/complexity-<ts>.json (local, gitignored; skipped
under CBM_SKIP_PERF where rates are meaningless).
Two corpus shapes, both needed:
- Independent modules (java/py/go/ts templates): catches cross-module
contamination and dedup breakage. The #1669 bug is GREEN here — fully
closed modules filter perfectly, which is exactly why it survived.
- The growing shared package (bigpkg): one Java package whose file count
scales with k — the real-repo shape (files concentrate in large
packages). The JVM namespace filter branch makes per-file work track
package size, so this corpus is the honest #1669 reproducer:
ratio 4.00 RED on the pre-fix tree, 4.00 RED for a module-scoped
overlay, 2.00 GREEN for the own-file overlay. It discriminated the
correct fix design before the fix was written.
Both legs of every pair exceed MIN_FILES_FOR_PARALLEL(50): below it the
sequential path runs, which builds no shared registries and would be the
wrong code path to gate (its per-file cost is bounded by the 50-file
ceiling).
Recorded but deliberately NOT gated, with reasons at the case:
tail_candidates and fallback_rows are legitimately superlinear under
replication until those scans are bounded, and measured ~1 ns/unit.
Every ratio gate carries a non-vacuousness floor on the base counter so
broken counter wiring fails loudly instead of green-washing
(cbm_pxc_count_perfile_defs feeds the overlay path into the same
counter the fallback path already used; wired for Java, extend to the
TS overlay when touching ts_lsp).
Dynamic coverage: languages iterate CBM_LANG_COUNT; embedded templates
cover the LSP-hybrid languages, and tests/fixtures/complexity/<lang>/
dirs are auto-discovered so a new language joins the guard by dropping
fixtures. Uncovered languages are listed in the report with the reason.
The local report additionally carries per-language node/edge counts with
ratios and a per-pass elapsed_ms table per run (captured via a TEE log
sink during the in-process pipeline runs) — trend data for humans, still
never a gate.
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Cross-file LSP for Java rebuilt a type registry for EVERY file from a def
set the module filter reduces by a constant factor, not to a constant
size: the JVM filter branch includes every def sharing the file's
namespace, so per-file work tracks the corpus (measured defs_per_file
1,292 -> 5,031 in lockstep as defs_total went 179k -> 689k). That makes
the pass O(files x corpus_defs) — 87% of a Java index and the bulk of
the v0.9.0 -> v0.10.x slowdown.
Three changes, one architecture (the pattern Go/Python/C/C#/TS already
use):
1. cbm_java_build_cross_registry — the JVM def universe (Java + Kotlin,
for mixed source roots) built ONCE per run, sealed read-only, shared
across resolve workers. Wired into both pipeline.c and
pipeline_incremental.c.
2. Two-phase registration inside that build: all TYPES first, finalize
(hash buckets exist), then FUNCS. Func registration parses signatures,
and type-name qualification via cbm_registry_lookup_type is a LINEAR
scan until finalize — a single mixed pass measured 0.44 ms/def at
689k defs, a ~300 s sequential build that erased the sharing win.
Two-phase: 306 s -> 2.8 s. Stable partition order because overload
ties resolve to the first registered QN match.
3. cbm_run_java_lsp_cross_with_registry resolves each file against the
shared base through an overlay holding exactly THIS FILE's defs
(register_local_func_or_type_from_file). Own-file scope is the load-
bearing choice: patch_one_method refines signatures from the AST and
must write a private copy (its types live in the per-file arena, the
base is sealed), and any wider scope re-imports the quadratic — a
module/namespace-scoped overlay measured ratio 4.00 on the growing-
package corpus, own-file measures 2.00.
Elasticsearch corpus (46,477 files), same host, CBM_PROFILE=1:
v0.10.5 this change v0.9.0
wall 419.9 s 85.6 s (4.9x) 61.5 s
cross-LSP CPU 6,195,780ms 52,513ms (118x) 110,344 ms
us_per_file 207,448 1,770 3,722
us_per_file_per_kdef 300 2 —
nodes 693,100 693,100 —
edges 5,646,235 5,649,949 —
Cross-LSP CPU now beats v0.9.0. Nodes are identical; edges +0.066%,
consistent with the shared base resolving cross-package targets the old
per-file namespace/import filter could not see, plus source-order
independence from the two-phase build.
Guarded by the complexity suite's shared-package gate (RED at ratio 4.00
on the pre-change tree, 2.00 after — see the suite commit).
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The per-file cross-LSP path rebuilds a type registry for every file. Whether
that is cheap or quadratic depends entirely on how many defs the module
filter leaves, and nothing reported it — so a filter that reduces by a
constant FACTOR rather than to a constant SIZE looked identical to one that
worked.
parallel.resolve.perfile_registry reports defs_per_file next to defs_total,
plus how often the filter failed. Measured on one Java tree:
1/4 corpus: defs_total=179033 defs_per_file=1292
full corpus: defs_total=689216 defs_per_file=5031
defs_per_file grew 3.89x while the corpus grew 3.85x — lockstep, so per-file
work is proportional to the whole corpus and cross-file LSP is
O(files x corpus_defs). filter_failed=0 throughout, which is why this was
invisible: the filter always 'succeeds', it just does not bound the set.
Diagnostics only; no behaviour change.
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Formatting only, on lines added by the previous commit. Verified with the
repo's pinned Homebrew LLVM clang-format so this is the CI-canonical
result, and confirmed no unrelated file needed reformatting.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Finding the 0.10.x indexing regression took an 11-corpus A/B across two
release binaries, a subset-scaling series, and a per-pass exponent fit.
None of that should have been necessary: the pass that carried it,
cross-file LSP, was already timed and already logged its def count. The
numbers were there, just never normalised into something a reader could
judge.
Three additions, all shipped without a flag except the detailed curve:
- parallel.resolve.cross_lsp_cost — cross-LSP cost NORMALISED per file,
next to defs_total. Wall time cannot separate "big repo" from
"superlinear pass"; us_per_file can. Measured on one Java tree:
files=3710 defs=102845 us_per_file=35129 us_per_file_per_kdef=341
files=14833 defs=339866 us_per_file=106722 us_per_file_per_kdef=314
Per-file cost tripled while per-kdef stayed flat — the fingerprint of
work proportional to the whole corpus (files x defs). One grep on two
differently sized repos now answers what previously took a two-binary
bench.
- parallel.resolve.scan_cost — candidates visited per tail-match lookup,
plus fallback_rows, which the code has counted since #1085 but exposed
only to a test. On the same tree the tail scan reached 242M candidate
visits (n^2.04), worth seeing even though it proved cheap in wall time.
- cbm_scale_probe (foundation/profile.h) — samples cumulative elapsed at
1/8, 1/4, 1/2 and 1 of a pass's items and fits k in T ~ n^k, warning
once k reaches 1.35 in shipped builds. Wired into parallel_extract and
parallel_resolve.
The probe's limits are documented rather than oversold: it catches growth
WITHIN a run, and would NOT have caught this bug, whose per-item cost is
constant-but-large within any single run (it reported 1.26 while the
cross-corpus exponent was 1.86). That is exactly why us_per_item and
us_per_file are emitted alongside it.
Tests are deterministic by construction: the exponent fit is a pure
function fed synthetic points, and the checkpoint bookkeeping is asserted
directly. A test that proved the detector by generating a real quadratic
workload would be asserting on the scheduler.
No product behaviour changes; diagnostics only.
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Since #1360 routed ordinary malloc/new through mimalloc on Linux, the
arena policy governs every allocation in the process rather than just the
bound sqlite/tree_sitter populations. cbm sets arena_eager_commit=0, so
mimalloc commits sub-ranges with mprotect(PROT_READ|PROT_WRITE) over a
PROT_NONE reservation, and each partial commit SPLITS the reserved VMA.
Measured on the Go corpus, Linux arm64, shipped binaries:
v0.9.0 10 mappings, at ANY worker count
v0.10.5 ~22k mappings, peak; the count tracks CONCURRENCY
(999 at 1 worker, 8460 at 4, 11965 at 18)
Two consequences, both of which #1654 reported from a 96-CPU/376 GB host:
the mmap/mprotect churn serialises on the kernel's per-process mmap_lock,
and the VMA count climbs toward vm.max_map_count, after which mmap fails
for ANY size -- so mimalloc reported it could not allocate 10 KB while
`free -g` still showed 246 GB available.
mimalloc's own default for this option is 2, meaning "eager-commit arenas
only on an OS that overcommits (i.e. linux)", precisely because commit is
free there until pages are touched. Overriding it to 0 opted Linux out of
the default written for Linux. Restore it on Linux only; every other
platform keeps the lazy setting, where commit is NOT free and the
upfront-memory reason still holds (Windows especially, #581).
Measured effect, same corpus and host, baseline build vs this build:
mappings 22450 -> 17312 (-23%)
wall 92.4s -> 92.6s (unchanged)
peak RSS 19.14 -> 19.22 GB (unchanged)
This is a partial mitigation, not a cure: the remaining ~17k mappings are
individual 64 KB-3 MB extraction buffers, each taking its own mmap (the
worker reserves ~40 GB of address space for ~19 GB of RSS). Pooling those
is the durable fix and is deliberately left out of this change.
Guard: mem_arena_eager_commit_follows_platform_commit_cost pins the
platform split so the Linux default cannot be silently opted out again.
Reproduction and controlled 2x2 (only vm.max_map_count varied) are
recorded on #1654.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
LeakSanitizer on CI caught all five new tests leaking their CBMFileResult:
Indirect leak of 24 byte(s) ... ts_tree_new
cbm_extract_file_ex cbm.c:1256
do_extract test_parse_coverage.c:39
test_dockerfile_missing_final_newline_not_flagged_issue1610:272
SUMMARY: AddressSanitizer: 706504 byte(s) leaked in 189 allocation(s)
Every pre-existing test in this suite calls cbm_free_result before PASS; the new
ones did not. The local run could not have found it - LeakSanitizer reports
"detect_leaks is not supported on this platform" on macOS arm64, so this class
of defect is CI-only here.
Each test now captures what it asserts, frees, and only then decides, so the
early-FAIL paths do not leak either. The cross-grammar loop prints its
diagnostic before freeing so the failure message keeps naming the grammar.
While correcting the guard, a first attempt left ASSERT_TRUE(flagged ||
has_ranges || true) in real_error_before_eof_still_flagged - always true, and it
would have silently disarmed the guard that stops the EOF suppression from being
over-broad. Removed. The guard is re-proven binding: forcing
cbm_is_eof_terminator_miss to return true makes EIGHT tests fail, including both
guards, and restoring it returns the suite to green.
parse_coverage 14 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The verify pass submitted every extracted object, selected executables included,
on the stated grounds that "VirusTotal is content-addressed, so identical bytes
return the analysis it already holds instead of re-running 70+ engines".
That is measurably false. On v0.10.5 all EIGHT re-submissions produced a NEW
analysis - same VirusTotal file-id, timestamp 47 minutes later:
candidate: file-id=2c00f485... ts=1786795957 (12:12:37Z)
verify : file-id=2c00f485... ts=1786798758 (12:59:18Z)
Re-analysing identical bytes re-rolls a probabilistic classifier, and Microsoft's
ML engine answered differently within that hour, in BOTH directions:
82750cd1 (linux-amd64) microsoft-ml -> clean
6d3c5be6 (darwin-arm64) clean -> microsoft-ml
The published notes are generated from the candidate scan, so v0.10.5 shipped a
table calling linux-amd64 flagged when VirusTotal had it clean, and darwin-arm64
clean when VirusTotal was reporting Trojan:Script/Wacatac.B!ml. Every hash in
that table links to the page that contradicted it. Corrected in place after
publication; this removes the cause.
The second scan proved nothing the first did not. Identity is settled by hash
before this step runs: verify-release-selection.py reconciles every published
container to the selected bytes, and checksums.txt binds the same digests
publicly. A re-scan adds no assurance - only another roll.
What still gets scanned is exactly what the candidate pass never saw: install.sh,
install.ps1, LICENSE, THIRD_PARTY_NOTICES.md, the MCPB manifest.json and the
unpacked UI assets. install.sh and install.ps1 are the highest-consequence
non-executable bytes we publish - users pipe them straight into a shell - and
that coverage is untouched. Measured on the v0.10.5 object set: 16 objects in,
8 withheld, 8 still scanned.
The withheld set is recorded as evidence (cbm-virustotal-withheld-v1) naming each
sha256 and pointing at virustotal-candidate-results.tsv, so the published
evidence still accounts for every shipped object.
Fails closed three ways, each with an actionable message: no object matches a
selected sha (the containers do not carry the recorded bytes), everything is
withheld (the surface scan would be a no-op), or the selection names no shas at
all. The zero-match grep is wrapped rather than left to pipefail, because a
guard that aborts silently is not a guard - found by testing the guards rather
than assuming them.
Also drops 8 VirusTotal submissions per release.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A file that does not end with a newline leaves the grammar's mandatory line
terminator MISSING. cbm_collect_error_regions counted that node, so the file was
reported parse_partial with the last line as its error range.
It is not a miss. The node is ZERO-WIDTH and sits at EOF: the parser consumed no
source for it, so by construction nothing was dropped - no construct can live in
a zero-byte span - and every real instruction above it parsed normally. Proven
by dumping the tree: the reporter's two-line Dockerfile yields
(source_file (from_instruction ...) (entrypoint_instruction ...) (MISSING "\n"))
with both instructions intact and the MISSING node spanning bytes 73-73.
It was never Dockerfile-specific. Stripping the trailing newline from the 156
linkable grammar fixtures flips 13 of them to has_error, and SIX produce regions:
dockerfile, tcl, fish, gomod, hyprlang - and makefile, which is a genuinely
different case (its ERROR has WIDTH; the recipe really is lost).
Worse, the ones that stayed silent did so for no principled reason. ini, fsharp,
beancount, requirements, gitignore, sshconfig and kconfig omit the same
terminator, but theirs is a HIDDEN node and hidden nodes are invisible to
ts_node_child(). Whether a user was told their file was partially parsed came
down to whether that grammar's author declared the terminator visible.
The cost was not cosmetic: a phantom parse_partial writes a "<project>::missed"
shadow row, and until #1609 that row made the project fail cross-repo validation
as BOTH source and target. A single absent byte could remove an entire
repository from cross-repo intelligence with no error shown anywhere.
The suppression is deliberately narrow - zero-width AND at EOF. A MISSING or
ERROR node with width still counts even at EOF, and anything before EOF is
untouched. Both callers pass the raw root, so one source_len is correct for
both; verified rather than assumed, since root is bound once and never
reassigned.
Reported by @vitaliy-shatskiy, who could not share the original file and instead
rebuilt the property from scratch with a byte-exact script - an editor would
have silently re-added the newline and hidden it. Their isolation matrix ruled
out BOM, CRLF vs LF, exec-form vs shell-form and file length before we looked at
it once.
Reproduce-first, revert-checked: the Dockerfile and cross-grammar tests fail on
the previous tree and pass with the fix; forcing the new predicate to return
false brings the identical REDs back. Two guards pin the boundary and hold in
both directions - a width-bearing failure at EOF (makefile) and a real
mid-file ERROR in a file that ALSO lacks its final newline (built from
C_IFDEF_SPLIT, the fixture this suite already proves is flagged).
parse_coverage 14, extraction 276, language 217, infrascan 3,
grammar_regression 1 - 511 passed, 0 failed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`update` hands off to install.sh (install.ps1 on Windows) and prints the command
to run. It built that command from cbm_detect_self_path - the BINARY's directory
- and treated "I resolved my own location" as "the installer is beside me".
Those are different questions. install.sh is placed beside the binary by
install.sh itself, but a binary that was moved, packaged by a distro, or built
from source has no installer next to it. We printed the path anyway:
bash "/home/<user>/.local/bin/install.sh"
/usr/bin/bash: /home/<user>/.local/bin/install.sh: No such file or directory
Reported on discussion #1560 (#1632) by a user who was already three releases
deep in install trouble and had just been told, by us, to run a file that does
not exist.
`update` exists to tell someone how to proceed. Ending the interaction on a
command that cannot run is the one outcome it must not produce - and the
fallback text was already there and already correct, naming install.sh as
shipping in the release archive without asserting a path.
The probe goes through cbm_path_info_utf8 so a non-ASCII install directory
resolves on Windows, and rejects a DIRECTORY of that name, because `bash <dir>`
is not a command either. A symlink still counts: it is reported rather than
followed, and the shell runs it perfectly well.
The Windows branch gets the same treatment; it had the identical assumption
about install.ps1.
Reproduce-first and revert-checked: with the probe forced to return true - the
old behaviour - the new test fails with "a directory with no installer must not
be named as one", and passes once it is restored. cli: 276 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
audit-license-provenance.py ran `curl https://www.apache.org/licenses/LICENSE-2.0.txt`
every time the gate executed and byte-compared the result against
vendored/nomic/LICENSE, with capture_output and no error check. Any fetch
failure therefore produced an empty string, compared unequal, and reported:
vendored/nomic: DIFFERS [apache.org canonical LICENSE-2.0.txt]
PROVENANCE AUDIT FAILED: 1 unexplained verdict(s)
which is indistinguishable from a real licence discrepancy.
A gating verdict must be a pure function of the tree, not of whether a web
server answered. This one was neither reproducible nor attributable: it
reddened `security / license-gate` on PR #1337 - a branch touching cli.h,
hook_augment.c and test_cli.c, and no licence at all - and left that
contributor blocked for over two weeks on a signal that had nothing to do with
their change.
Verified before pinning: our vendored copy is byte-identical to the upstream
canonical text, 11358 bytes on both sides, diff clean. The Apache-2.0 text is
immutable and versioned, so a digest is the honest way to express "this is that
text". A mismatch now means our vendored copy changed, which is exactly - and
only - what this audit exists to detect.
The audit passes locally with no network access on the nomic entry.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The ui-* archives are byte-identical copies of the canonical ones, published
after verify so they inherit the hash-bound VirusTotal verdicts. They were
absent from checksums.txt, and publish-legacy-aliases.sh documented that as
intentional: "checksums.txt covers the canonical names current installers
request."
That reasoning has a hole. The aliases exist only for 0.9.x updaters (#1538),
and those verify the NAME they asked for. So the alias fixed the 404 and moved
the failure one step later - the updater downloads the archive, cannot find its
name in checksums.txt, and refuses:
warning: codebase-memory-mcp-ui-darwin-arm64.tar.gz not found in checksums.txt
error: refusing to install an unverified download
Reported by AmooAti in #1134. Confirmed on the live v0.10.4 release: eight ui-*
archives published, zero of them listed. Every pre-0.10 user who answered the
old variant chooser with "ui" is hard-blocked from updating by any path.
The same digest is now emitted under the legacy name before the attestation
step, so the attested artifact covers both names. No new bytes and no new scan
surface: an alias is a copy, so its sha256 is by construction the one already
computed.
The rule lives in scripts/ci/append-legacy-alias-checksums.sh rather than inline
in the workflow, because the venue-parity contract requires it: a venue may
provision, plumb artifacts, or call a canonical leg script, and text
transformation is none of those. Keeping it beside publish-legacy-aliases.sh
also puts the two halves of the alias rule in one place, which matters because
they must stay in step - .tar.gz and .zip only, never an already-ui-* name. It
fails closed when it matches nothing, since a name with no asset is as broken as
an asset with no name.
Validated against the real v0.10.4 checksums file: the generated set is exactly
the eight ui-* assets that release published - no phantom names, none missing -
and the empty case exits non-zero.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Indexing writes an internal "<name>::missed" miss-graph row into the SAME db as
the primary project whenever a file parses partially. cr_store_has_exact_project
required `count == 1` over ALL rows returned by cbm_store_list_projects, which
does not filter those rows - so any project that had ever recorded a parse miss
failed validation, as SOURCE and as TARGET, and the whole feature reported:
project is not indexed
for a project that plainly was. There is no user-level workaround: a partial
parse is not something the operator controls, and re-indexing reproduces the
shadow row.
This is the same defect mcp.c fixed for list_projects in #1044 ("requiring
n == 1 over ALL rows made every project with a miss graph vanish"); the
cross-repo site never learned it. The fix ports that primary-row filter.
The single-primary requirement itself is deliberately kept: it is what proves
the db belongs to the project we were asked about rather than being a shared or
mislabelled store. Only "::" shadow rows stop counting toward it.
Reported by vitaliy-shatskiy in #1609, whose diagnosis named the exact function
and the exact reason.
Reproduce-first, and revert-checked both ways:
- the new test fails on origin/main with `ASSERT(!(result.failed))`
(tests/test_cross_repo.c), for the behaviour under test rather than a setup
error;
- it passes with the fix;
- reverting ONLY src/pipeline/pass_cross_repo.c and keeping the test brings
the identical RED back, so the test binds to the production change.
The existing pair without shadow rows is the control: those tests already prove
that path returns edges, so this cannot pass vacuously on a fixture that never
matched.
cross_repo 8 passed; pipeline 249 passed; store_edges 25 and store_nodes 67
passed - no collateral change.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`codebase-memory-mcp uninstall --help` removed the binary and every agent
configuration (#1038). It did the destructive thing to someone asking what the
command does.
The top-level dispatcher matches the subcommand at argv[1] and forwards the rest,
so its own --help check at src/main.c:1047 never sees argv[2]. Nothing downstream
looked either, and cbm_cmd_uninstall went straight to parse_auto_answer.
The guard is checked FIRST, before parse_auto_answer, so a `-y` elsewhere on the
line cannot auto-confirm the destruction we are trying to prevent. It prints real
usage, states plainly that the command is destructive, and points at --dry-run.
--help is the flag a person types precisely BECAUSE they are unsure what a
command does. It must never be the thing that destroys their install.
cli suite: 271 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
posix_directory_parent_secure(current_fd) validates the directory we are ALREADY
IN, but the refusal message printed `component` - the child about to be entered.
Every reporter has therefore been sent to inspect the wrong directory:
#1537 read "ancestor '.cache'" when /Users/<user> was refusing
#1621 read "ancestor 'cbm-daemon-501'" when /private/tmp was refusing
Both inspected the named directory, found it clean, and said so. They were
right. #1537 has been open for weeks with the reporter repeatedly confirming a
correct `.cache` - including on v0.10.4 today - because we kept pointing at it.
The message now names the CONTAINING directory and says explicitly not to check
the component itself.
This fixes no permission logic. It changes weeks of talking past each other into
a report someone can act on in a minute, which for this class of bug is the
whole game: the refusal is invisible from outside, so the message IS the
diagnosis.
Found while reviewing the CBM_RUNTIME_DIR work; the original detail was added in
the earlier half of #1537 and named the wrong variable from the start.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The daemon/CLI rendezvous directory is created under %LOCALAPPDATA% (Windows) or
/tmp -- /private/tmp on macOS -- and every ancestor of it must pass the
private-directory walk. That ancestry is not always acceptable, and when it is
not, EVERY invocation fails, `config list` included, so the settings surface
cannot be reached either:
codebase-memory-mcp: secure daemon endpoint could not be created
#1623 narrowed the Windows side of this by admitting AppContainer package and
capability SIDs on ancestors, and named the remainder explicitly: a live local
group, Authenticated Users inherited from a secondary volume root, and orphaned
unresolvable SIDs still refuse, and "those need CBM_RUNTIME_DIR or a separate
change". #1621 is the POSIX shape of the same dead end -- /private/tmp/cbm-daemon-<uid>
refused with no way to move it.
There was no way to move it in a shipped build. The only relocation hook,
CBM_TEST_DAEMON_RUNTIME_PARENT, is compiled out unless CBM_ENABLE_TEST_SEAMS is
defined, so a test build started while the shipped build did not; CBM_CACHE_DIR
is no help either, because it moves the cache and never the rendezvous.
CBM_RUNTIME_DIR names the parent directory the rendezvous is created under. It
does NOT relax the check: the directory it names goes through exactly the same
validation as the default -- ancestors owned by you or root, not world-writable,
no allow-ACL; the rendezvous directory itself still forced to owner-only -- and a
value that fails is refused rather than silently replaced by the default. The
operator only chooses an ancestry that passes. cbm_safe_getenv never truncates,
so no half of an over-long value can become a runtime parent.
The override is resolved in cbm_daemon_bootstrap_endpoint_new(), the one function
every product endpoint goes through: the daemon, the MCP client, the local CLI,
the index worker, and the install/update/uninstall activation path in cli.c. No
call site can silently keep the default, and the detached daemon inherits the
value with the rest of its environment. An explicit parent still wins, so the
compile-time test seam and the lifecycle guards' isolated namespace behave
exactly as before.
Approach and variable name from #1576 by Leonardo trindade miranda, resolved one
layer lower so the activation path is covered too.
Refs #1574
Refs #1621
Co-Authored-By: Leonardo trindade miranda <tmonestudio@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Six reports describe the same dead end: the indexing worker dies, the
supervisor points at `logs/.worker-<pid>.log`, and the file is 0 bytes.
Nothing was ever flushed, so not one of #1070, #1130, #1132, #1133,
#1145 or #1450 is reproducible or attributable -- the hint says "crashed
on a file" and the file is never named.
Root cause is buffering, not the crash. The worker's stderr is redirected
to that log file by the supervisor, and every diagnostic goes through
`fprintf(stderr, ...)` with no flush. The C standard only promises stderr
is "not fully buffered"; the Windows CRT gives a redirected stderr FULL
buffering, which is why five of the six reports are Windows. A worker
that aborts, is SIGKILLed by the OOM killer, or is terminated after a
hang takes its whole buffer with it.
- cbm_log_set_crash_durable(): setvbuf(_IONBF) plus a per-line flush in
emit_line. Both, deliberately -- setvbuf covers every writer to the
stream including the plain fprintf startup errors that explain a
worker which never got as far as logging, and the flush covers the
case where setvbuf is refused because the stream was already written
to. Enabled for the worker role only, claimed in main() before the
process writes anything.
- cbm_index_worker_log_begin(): a startup header written and flushed as
the worker's first act -- version, build fingerprint, pid, repo path,
and the worker's own arguments. A control record rather than an info
line, so CBM_LOG_LEVEL cannot restore the 0-byte log.
- Under a crash-durable log the parallel extract pass logs every file
it starts, not just the first two rounds of workers, so the log ends
with the files that were in flight when the worker died. One line per
file, never per node; unchanged for every non-worker caller.
This fixes no crash. It converts six unreproducible reports into reports
we can act on, and every future one arrives with the run identified.
Test: a worker started with a fully buffered stderr -- the state the
Windows CRT hands it, forced on POSIX so the contract binds on all three
legs -- writes diagnostics and is then SIGKILLed. The retained log must
be non-empty, carry the header with the version, pid, repo path and args,
and still hold the line written after it. Reverting the production change
leaves that log at 0 bytes.
The probe kills rather than aborts because Darwin's abort() runs the
stdio cleanup handler and flushes the very buffer the repro depends on
stranding: under abort the reverted build still produced a populated log
and the test was quietly toothless. SIGKILL runs no cleanup, and is
literally #1070's death (signal=9).
Refs #1070
Refs #1130
Refs #1132
Refs #1133
Refs #1145
Refs #1450
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`cli <tool>` resolved its JSON arguments from stdin whenever stdin was not a
terminal and no --args-file, raw-JSON positional or --flag form was supplied,
and cli_slurp_stream reads to EOF. An ordinary automation caller never sends
that EOF: Node's child_process.spawn defaults to stdio ['pipe','pipe','pipe']
and the parent must call child.stdin.end() explicitly, which almost nobody does
for a command it is not writing to. fd 0 then stays open with no writer and the
read never returns. The reporter's shell script was still parked in fread(0)
fourteen minutes later.
`list_projects` advertises "properties":{} — stdin could never have carried
anything it accepts, so the read was pure deadlock with nothing to gain. Gate
the stdin path on the tool's input_schema actually declaring properties, in a
seam (cbm_cli_args_from_stdin_allowed) that main.c's resolution chain calls.
An unknown tool also stops blocking: dispatch rejects it by name and no stdin
content can change that verdict. Tools that do declare properties keep the
documented `echo '<json>' | cli <tool>` channel untouched, and interactive runs
are unchanged because isatty(0) already short-circuited them.
Measured on the pre-fix binary, stdin held open by a writer that never writes,
15-second watchdog: `cli list_projects` was killed at 15s with 0 bytes of
output, while `cli list_projects < /dev/null` returned in 3s with 90 bytes.
After the fix the same open-pipe invocation returns in 3s with 90 bytes.
Regression tests (tests/test_cli.c): the reported hang, the piped-argument
channel that must survive, the interactive path, unknown/NULL tool names, and a
sweep asserting the gate tracks the advertised schema across the whole tool
table so a future zero-argument tool cannot reintroduce this silently.
Revert-check: with the schema gate reverted to the pre-fix `return
!stdin_is_tty`, both tests fail on the list_projects assertion and the suite
reports 2 failed; with the gate applied, 272 passed / 0 failed.
Known limit, not addressed here: a tool that DOES declare properties, invoked
with its required flag omitted (`cli index_status` without --project), still
blocks on an open pipe. Distinguishing "arguments are coming" from "nobody will
ever write" needs a bounded wait, which is a public-surface decision (deadline
value, and the cross-platform pipe-readiness probe Windows would need) rather
than a mechanical fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
OpenCode writes "enabled": true beside the "command" and "type" we write. Our
ownership check required the entry's key set to match EXACTLY
(config_json_like.c: member_count != found_count), so a three-key entry with two
recognised keys was classified as FOREIGN - and we refused to touch an entry we
had written ourselves. install then failed with:
error: agent_config agent=OpenCode op=mcp_install path=.../opencode.json
Confirmed on two independent configs: Linux (#1630) and Windows (#1582). In
gotspatel's file EVERY MCP server carries the key - mssql, forgetful,
chrome-devtools and ours - so this is OpenCode's normal shape, not an unusual
hand-edit. Anyone who has ever toggled a server on or off in the UI was hit.
Two of my own hypotheses were wrong before the reporters' files settled it: it is
not JSONC comment parsing, and it is not the .jsonc targeting that #1575 fixed.
#1575 fixed WHICH file we open; this happens after, on what we find inside.
The distinction now reported is MATCH_WITH_EXTRAS, and the caller treats it as
ALREADY SATISFIED - success, without touching the file. That is deliberate and it
is the safe half of the fix: cbm_json_like_upsert_entry REPLACES an entry
wholesale, so writing our canonical shape over an annotated entry would silently
delete the client's keys. A refusal the user can see beats a deletion they
cannot. Doing nothing is also correct on the merits: the entry already names this
binary with the right type, which is the entire content of the install.
Merging our fields into an annotated entry while preserving the rest is the
fuller fix and stays tracked in #1630. This makes the common case work without
risking anyone's configuration tonight.
Ownership is NOT loosened otherwise: an entry whose command points at a different
binary is still foreign and still refused, byte-identically, and that direction
is pinned by its own test.
Tests use the reporters' actual entry shape. cli suite: 272 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
yaml_range_has_unsupported rejected `&` and `*` anywhere in a plain scalar, with
no positional test at all. So a value containing prose asterisks - `use
*emphasis* here`, a kaomoji, a glob inside a description - was refused as if it
were an alias.
That is one of four constructs that made a real 16 KB hand-maintained Hermes
config permanently un-editable by us (#1631). It is not YAML we cannot model; it
is YAML we declined to read.
An indicator only counts where a NODE begins. The test is now positional and
judged by what PRECEDES the character, because the scanned range covers a whole
line including its key: in `command: &shared` the `&` is an anchor even though
`command:` came first. A node begins at the range start, after a mapping colon,
or after a block-sequence dash. Everything else is text.
`{` and `}` keep their existing treatment. Empty flow mappings (`key: {}`) are
the next item in #1631 and carry their own semantics; bundling them here would
mix a positional correction with a value-semantics change.
Root-caused by @rg6304, who reproduced it in isolation, read this source, and
corrected the maintainer's hypothesis - the constructs I had guessed (inline
comments, anchors, `---` separators) appear nowhere in the failing file.
TESTING NOTE, recorded because it nearly went wrong: the first version of the
regression test put the asterisk in an untouched foreign section and passed with
the fix REVERTED - the editor only validates the range it writes, so the test
proved nothing. Rewritten onto the entry block, it now fails without the fix
(`== -1, expected 0`) and passes with it. The companion test pins the other
direction: a LEADING `*` is still an alias and still refused, byte-identically.
config_yaml_edit + yaml suites: 117 passed, 0 failed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`error: agent_config agent=OpenCode op=mcp_install path=...` is the whole
diagnostic today. The config editors collapse nine distinct fail-closed
conditions into a single -1 - unsupported structure, an inline comment on a
field line, a non-single-link file, unsafe metadata, lock contention, I/O - and
the message separates none of them.
Four failures of exactly this shape are sitting in discussion #1560 right now,
unresolved across five releases: OpenCode mcp_install on Linux and Windows,
Hermes mcp_install and pre_llm_hook_install on both, and a Codex hook_preflight
that is new in v0.10.4. One reporter has hit install failures on every release
from 0.10.0 to 0.10.4 and is still filing them. Nobody can tell whether these
are one bug or four.
The refusal now reports what the target actually is: kind and size, observed at
the moment of failure. That is enough to split the space - a 19 KB regular file
that we refuse to write is a structural refusal, not a permissions or
missing-file problem, and that alone rules out most of the candidates.
Deliberately NOT errno. This function is called from 119 sites and errno may be
stale from an unrelated call. Printing it would repeat #1537 exactly, where a
permission decision surfaced a fabricated ENOENT and sent the reporter hunting
for a file that was never missing. Only observed facts, or nothing.
This is the first step, not the whole fix: the editors should carry their own
refusal reason out, the way cbm_daemon_ipc_validation_detail() already does for
the daemon. That is 89 failure returns in the YAML editor alone, so it lands
separately rather than as an unreviewable diff.
cli suite: 270 passed, 0 failed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#1620: every index_repository run fails, every pass succeeds, the worker exits
0, no error-level line appears anywhere, and the user is told:
Pipeline failed. Check repo_path exists and contains source files.
The repository is fine. The publish is failing on a filesystem permission
problem, and two separate defects conspired to hide it.
cbm_rename_replace discarded GetLastError() on Windows and returned a bare
CBM_NOT_FOUND. Callers then log errno - see finalize.rename_failed - so the one
field that should explain the failure held whatever an unrelated CRT call left
there. The Win32 error is now translated to errno, with ERROR_ACCESS_DENIED the
interesting case: MoveFileEx needs DELETE on the destination, which a cache file
created under an empty or foreign DACL does not grant.
cbm_pipeline_finalize_staged_generation returned PERSIST_FAILED on a failed
sidecar removal with no log at all. That is the silent path: a publish that
fails must say so.
Neither change fixes the underlying ACL problem. Together they turn a report
that misdirects the user at their own repository into one naming a permission
failure on a specific file - which is the difference between an unactionable
issue and a five-minute one.
platform + pipeline suites: 268 passed, 0 failed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
'secure CLI coordination could not be created (endpoint)' is what four separate
reporters were left with in #1533 and #1574. On those machines every mode fails,
config list included, so the product cannot be reconfigured out of the failure,
and CBM_LOG_LEVEL=debug adds nothing. One reporter had to build an instrumented
binary to discover the cause was a single ACE on an ancestor of %LOCALAPPDATA%.
cbm_daemon_ipc_validation_detail() already holds exactly what they needed - the
refusing directory, the offending SID and the right it granted - and the
daemon-endpoint path has printed it since #1582. This path never did, so the same
refusal was actionable through one entry point and opaque through the other.
Quoting one of them: printing the refusing ancestor and the offending SID in the
default error would turn a multi-hour investigation into a one-line fix.
Fixes nothing by itself; it is the difference between a report we can act on and
a report that costs a reporter an evening. Builds clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
install.ps1 created its staging directory with New-Item, which inherits whatever
%TEMP% carries. It then ran the downloaded binary from that directory, and the
binary validates its own directory and refuses inherited cross-account mutation
grants. So the installer handed our own validator a directory the installer had
made wrong:
error: failed to stage install candidate: activation transaction I/O failed:
acl-grants-cross-account-mutation to S-1-5-21-...-1003
Five reporters, three different offending identities - a CodexSandboxUsers local
group, a synthesized SID recorded in .codex/cap_sid, and orphaned SIDs left by
uninstalled software - all naming an ACE the installer inherited rather than
anything cbm wrote. Still live on v0.10.4 (#1614). Closes#1529, #1614, #1571.
cbm's own C staging already creates its directory with a protected owner-only
DACL (win_mkdtemp_private_create). install.ps1 was the single path that skipped
it, which is exactly why redirecting TMP/TEMP to a fresh directory worked around
the failure - that workaround is now unnecessary.
Applied after creation rather than atomically on purpose: the CreateDirectory
overload taking a DirectorySecurity exists on Windows PowerShell 5.1 but not on
PowerShell 7, and Set-Acl works on both. The directory name is unpredictable and
nothing is written into it before the download, so the window is not usefully
attackable.
Best-effort by design: a filesystem that cannot carry a DACL must not fail the
install, and if the hardening does not take, the binary's own validation still
refuses - the honest outcome rather than a silent downgrade.
Pure ASCII, as PowerShell 5.1 decodes a BOM-less .ps1 as ANSI.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
