scripts/soak-test.sh claimed to isolate daemon coordination from
interactive sessions through a private CBM_CACHE_DIR, but only
CBM_RUNTIME_DIR moves the daemon rendezvous, so the soak shared the
operator's account daemon and asserted that its own shutdown stopped it.
Source scripts/test-runtime.sh for a harness-owned runtime and cache,
and let the helper stop the private daemon before the root is removed.
Add tests/test_soak_runtime_isolation_contract.sh, which fails before
this change, and wire it into scripts/test.sh.
Part of #1696.
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
The codeql-gate has never checked anything. It declared no permissions block,
so it inherited the workflow's contents:read and the code-scanning alert API
answered 403. gh writes the API error BODY to stdout, so
ALERTS=$(gh api '.../code-scanning/alerts?state=open' --jq 'length' 2>/dev/null || echo "0")
left ALERTS holding
{"message":"Resource not accessible by integration",...,"status":"403"}0
every [ ... -gt ... ] then failed as a non-integer comparison, the if took its
false branch, and the step printed "CodeQL gate passed (0 alerts)" and exited 0.
Observed in the log of a GREEN run (34700514102, job 103571322660) while the
repository had an open high-severity alert. The `|| echo "0"` reads as defensive
and is the exact opposite: it converts "I cannot see the alerts" into "there are
no alerts", which is the one direction a security gate must never fail in.
Grant the job security-events:read (and actions:read, which the CodeQL-run poll
needs for the same reason), and replace the fallback with a helper that treats a
non-zero gh exit or a non-numeric body as a hard failure. The helper returns
rather than exits because it runs inside a command substitution, where an exit
would end only the subshell and let the caller continue with an empty count --
the same silent pass in a new costume; both call sites check the status.
tests/test_security_gate_fail_closed.sh pins both halves and fails with six
assertions against the previous workflow.
Also bumps graph-ui's vitest to ^4.1.11 (GHSA-82fw-gwwq-j7x9, path traversal via
@vitest/mocker, CVSS 5.9). Development scope, and graph-ui ships in no release
artifact, so nothing released was ever exposed -- but it was the sole open alert
and the gate above is worthless while it stands unresolved. Raising the floor
past the vulnerable range also stops a fresh install resolving back into it.
graph-ui: npm ci clean, 47 tests in 12 files pass on 4.1.11.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The macos-15-intel leg killed the cli suite at its 900s wall clock while
reporting 7537 passed, 0 failed. Two separate things were wrong.
The suite was misclassified. cli spends 497s of the 900s default on macos-14,
the FASTEST macOS runner, while macos-15-intel in the same matrix is 2.4-3.6x
slower on comparable suites (daemon_runtime 842s vs 349s, stack_overflow_b
277s vs 76s). 497s at that ratio cannot fit. daemon_runtime, at 842s on that
same runner, passes only because it was already in SLOW_SUITES. cli belongs
there too -- it is deterministic and simply large, which is the only thing that
list is allowed to mean.
The drain test was also waiting on a clock instead of on the system. Profiling
the suite per test (315 tests, 300s wall / 166s cpu) put
cli_install_into_host_namespace_still_drains_host_cohort at 42s wall for 19s of
cpu -- 23s idle, the largest single idle block, where the next worst was 11s.
The existing cli_install_force_quiesces_active_cohort_before_replacing_binary
drains a cohort too and idles 0.5s, which is what pointed at the difference.
The cost was the host_serving probe. Its generous budget exists for the
POSITIVE question -- is this daemon still up? -- where a slow reply on a loaded
runner must not be misread as drained; that fixed a real flake and is kept.
Asked in the negative it inverts: no reply is coming, so the full 15s is spent
establishing silence and ASSERT_FALSE is decided by the timeout expiring rather
than by the daemon's behaviour. The drain is already proven positively in that
test -- install returns 0 only after the activation completed, and the host
child exits CLI_SCOPE_HOST_DRAINED -- so the probe only has to confirm it.
Also shortens the drained child's teardown budget, which retried service_free /
lease_release / manager_free against 10s on a path where the activation had
already torn the service down. Behaviour at the deadline is unchanged; it is
reached sooner when nothing is wedged. Measured at only -3s on its own, kept
because it is correct and free.
Local, same machine, ASan+UBSan: the drain test 42.0s -> 25.8s (-38%), the
suite 300s -> 279s, 315 passed before and after -- no test removed, no
assertion weakened. PR #2188, which runs the same leg without these two tests,
passed macos-15-intel in 37m34s, so the suite was not already over budget on
main.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A reporter brought a full ProcMon trace to #1856 and still could not say which
check refused their install. That is not their failing: the binary discards the
answer before printing it.
cli_activation_production_context_init() validates the cache, rendezvous and
log directories -- it is the emitter MOST likely to hold a useful detail -- and
it printed CLI_ACTIVATION_REFUSED_MESSAGE bare, through the raw sink rather
than cli_activation_diagnostic(). So the reader got "Check the errors above"
with nothing above. That is the exact dead end #1416 and #1537 already fixed;
the property was repaired on the paths that had tests and left broken on this
sibling, which had none. Route it through the attributing helper: the refusal
now names the transaction refusal note or cbm_daemon_ipc_validation_detail().
The remedy was also wrong for the reported shape. `icacls <dir> /remove:g <sid>`
cannot remove an INHERITED ACE, and the stock C:\ grant for Authenticated Users
reaches every new child directory exactly that way -- so a reporter following
our advice watches the command succeed and the refusal persist. The refusal now
says whether the ACE was inherited, and the advice names
`icacls <dir> /inheritance:r /grant:r "%USERNAME%":(OI)(CI)F` for that case.
No safety check is relaxed. An install directory writable by other accounts is
still refused; it is now refused in terms the reader can act on.
The new contract test guards the CLASS rather than this one call site: any
emitter of a refusal constant that bypasses the attributing helper fails it,
which is what would have caught this. It fails on the previous tree naming
src/cli/cli.c:750, plus the missing inherited-ACE report and the missing
/inheritance:r remedy.
Local: cli 314 passed; activation_transaction daemon_ipc daemon_bootstrap
daemon_version 110 passed, 1 skipped (user namespaces are Linux-only, runs on
the Linux leg); make -f Makefile.cbm lint-ci clean.
Reported by zaferavci1.
Refs #1856
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Indexing accepts whatever a repository contains. A tree carrying a vendored
monorepo, a generated dump, or a runaway build directory is discovered in
full, and the first sign of trouble is a host under memory pressure with
nothing that attributes it to indexing.
Add two opt-in limits evaluated during discovery against accepted source
files only: index_max_files and index_max_source_mb. Both default to off, so
nothing changes until an operator sets one. Crossing a limit fails the whole
attempt with a structured resource_limit_exceeded result naming the resource,
the observed value and the limit; no partial graph is published, and an
existing serving index keeps answering.
Limits are read from the CLI-managed _config.db and are not MCP request
arguments. A supervised parent replaces any caller-supplied policy before
spawning its worker, and the worker rejects a missing or incomplete contract,
so the CLI, the daemon and the supervised worker all enforce the same
decision.
Both keys reach an operator through the existing config get/set/list/reset
with no new subcommand. `set` suppresses its own generic message for them
because the policy writer names the precise reason -- so that writer speaks
on every failure it can return, including a validated value whose write then
fails on a database that cannot be written. Exiting non-zero in silence is
not an acceptable answer from a CLI.
The two shell regressions that hand-roll the supervisor's worker argv carry
that contract as well. Without it the worker exits before either guard can
observe anything, and the guard would go quietly vacuous.
Signed-off-by: 刘冲 <mail@liuchong.dev>
Add Explore, Galaxy, Architecture, Agents, and System workspaces with a
persistent, resizable local chat dock. Attach exact source selections and
graph context, render Markdown, and offer pinned opt-in browser models.
Keep the galaxy visible below chat and focused on the active file or code
selection. Restore hierarchy root selection, resolve projection identities
by symbol, and retain loaded search matches after an empty index reply.
Expose process measurement units, embed browser runtime MJS/WASM assets,
and constrain binary-audit exceptions to reviewed dependency strings.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The daemon-spawned index worker inherited the daemon process environment and
re-ran the allowed-root check without a session policy. The daemon's session
policy is installed only in application.c (the sole production caller of
cbm_mcp_server_set_session_context); after the daemon admits an
index_repository request, index_supervisor.c builds the worker argv
(`cli --index-worker --index-worker-build <fp> index_repository <args_json>
--response-out ...`, no root anywhere) and subprocess.c spawns it with the
daemon's environ. The worker then creates its own server with
cbm_mcp_server_new(NULL), so handle_index_repository runs with no session
policy and falls back to getenv("CBM_ALLOWED_ROOT") - the environment of
whoever STARTED the daemon (bootstrap.c spawns it with the starter's environ).
Observed: a daemon started with CBM_ALLOWED_ROOT=<rootA>; a client session for
<rootB>/tiny is admitted by the daemon (index.supervisor.reap outcome=clean
exit_code=0) and then refused by the worker with "<rootB>/tiny is outside the
allowed root. To allow it, run: codebase-memory-mcp allow-root ...". Any daemon
whose starter had CBM_ALLOWED_ROOT set broke every other session outside it.
Fix: in the worker arm of run_cli, right after the worker's server is created
and before any tool runs, read repo_path from the request args (both daemon
spawn paths rewrite the worker args with the canonical repo_path), canonicalize
it and install it explicitly as both session root and allowed root. The daemon
already authorized the request; the worker only executes it. A repository that
cannot be canonicalized does not exist and can only have been admitted by a
session with no declared boundary (containment needs a real path), so the
worker mirrors that with an explicit unrestricted policy - still never the
environment fallback - and the pipeline keeps reporting the missing repository
as the tool error it always was. A request without repo_path fails closed with
"request workspace scope invalid". No argv or IPC change, no process
environment mutation.
Regression test: tests/test_worker_session_scope.sh drives the real binary as a
supervised worker with CBM_ALLOWED_ROOT=<rootA> and repo_path=<rootB>/tiny and
requires an indexed response with no boundary refusal; a worker without
repo_path must exit nonzero on the scope check. Wired as Step 5f of
scripts/test.sh next to the other real-binary worker contracts. RED on main
(the exact "outside the allowed root" refusal above), GREEN with this change.
Distilled from #1925 with co-author credit.
Verification: test-runner suites index_supervisor, daemon_application, daemon,
mcp, cli green; tests/test_worker_error_response.sh, test_worker_watchdog.sh
and test_parent_watchdog.sh green against the fixed binary; make lint-ci and
scripts/check-no-test-skips.sh clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: Zhiyu <zhiyuzhang001@gmail.com>
Re-stacks the CodeAtlasWeb branch on the updated base (feat/atlas-r1 at
a44732b6, which merges main 7b0f553c). Ten conflicts, all in the seam
between the previous frontend and its replacement:
- README.md: main's surface count (45) with this branch's Atlas line.
- graph-ui/package-lock.json, graph-ui/src/App.tsx: CodeAtlasWeb's own
files stay; the previous frontend's edits to the same paths do not
apply.
- Seven previous-frontend files main had modified (GraphTab,
NodeDetailPanel and its test, StatsTab test, useProjects, rationale
and its test) stay deleted, and two tests main added for that
frontend (App.test.tsx, useProjects.test.tsx) are not brought in;
their subjects no longer exist in this tree.
Nothing under graph-ui changes relative to this branch; the C side
takes main's lean-output work and the branch's merge resolution.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three findings scripts/security-ui.sh raised on this branch, none new to
the merge:
- The analytics pattern matched the English word "plausible" in the
wiki text, and `\s` inside an ERE bracket is a literal backslash and
's', which cut every localhost URL at its first 's' and then failed
the allowlist. Same two regex fixes as on the stacked branch
(plausible.io; [:space:]).
- graph-ui/src/lib/rationale.ts carried a literal GitHub URL in a doc
comment, and its test built forge links from a literal GitHub base.
The audit forbids external URLs anywhere in the UI source, tests
included; the comment now describes the shape, and the test composes
the same links from a loopback base (the composition is
host-agnostic).
Audit passes; rationale tests 5 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings the Atlas branch up to current main (7b0f553c). Eight files
conflicted; how each was settled:
- src/mcp/mcp.c: the real ingest_traces description stays, in main's tool
row shape (main dropped the title column); index_repository keeps the
truthful abort branches (aborted_previous_preserved, persist_failed)
and gains main's format_migration flag in the success branch.
- src/ui/http_server.c and tests/test_httpd.c: the branch's twelve-tool
read allowlist for /rpc (#1663) over main's three, with main's %zu
and initialised counter.
- README.md: main's surface count (45), the branch's Atlas line.
- graph-ui (the previous frontend, replaced by the stacked #2068 but
kept building and green here): the branch's node-selection wiring
(onSelectNode) is kept consistently, so main's duplicate
handleSelectPath and its 3-argument call go; main's version chip is
added to the branch's header; useProjects keeps the branch's
render-first flow on top of main's paginated JSON helpers
(fetchAllProjects, fetchFullSchema), which the lean output contract
requires, and a project whose size the server did not send now gets
its schema too. main's GraphTab.selection.test.tsx, written against
main's GraphTab, is dropped in favour of the branch's own #1197
coverage in GraphTab.regions.test.tsx ("reports selected nodes to
the route for deep links"); main's useProjects pagination test waits
for the lazy schema enrichment instead of asserting before it runs.
Verified: mcp, httpd and ui suites 414 passed; the old frontend builds
and its 149 tests pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Path.write_text is open-then-write: it creates and truncates the file
first and the content lands afterwards. The parallel harness contract
polls for `<suite>.ready` to appear and then parses its content as the
leader pid, so a reader that wins that window sees an empty file and
fails with `ValueError: invalid literal for int() with base 10: ''` --
a spurious FAIL of the harness contract with a scheduler-side cause.
The same shape applied to `<suite>.leader-exited`.
Publish the barrier files through a helper that writes a same-directory
temp file and os.replace()s it onto the destination, which is atomic on
POSIX and Windows: a poller now sees either no file or complete content.
All three barrier publications go through it.
Pin the contract structurally (no timing): the scheduler may not write a
barrier file in place, and it must rename into place instead. The pin is
RED on the previous scheduler.
Distilled from #1188 with co-author credit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: Kody <kaidi.shi.1121@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
tests/test_venue_parity_contract.sh holds every venue to one harness: a
workflow step may provision, move artifacts, or call a canonical leg
entry, and the product-exercising commands live inside scripts/. The
test-ui job ran npm inline and tripped it on the local ladder.
scripts/ci/test-ui.sh is now the whole leg (install without a browser
download, vitest, the style and promise scans, the portable frozen
acceptance checks, the production build); the job calls it, and running
the same script locally is the same leg. Contract green again
(22 workflows marker-checked, 10 walked).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Maintainer glue for the imported frontend; no product behaviour of the
UI itself changes.
- Makefile.cbm: `npm ci` runs with PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1.
The UI's dev dependencies include playwright for its local browser
proofs, and a plain install would download browsers that no build or
gate launches.
- graph-ui/vite.config.ts: the dev server listens on 5173 with the two
literal loopback origins, which is what
tests/test_ui_dev_proxy_security.sh pins.
- graph-ui/index.html: lang="en"; the catalog is English.
- CSP: the directive string moves to http_server.h as CBM_UI_CSP_VALUE
so a test can pin it, and connect-src now admits the two loopback
services the UI may talk to when the reader starts them, the
local-model sidecar on 127.0.0.1:4141 and the agent-event bridge on
127.0.0.1:4142. Under connect-src 'self' both fetches were silently
blocked in the embedded build. Every other host stays forbidden; a
new httpd test asserts that the only hosts in the policy are those
two loopback ports.
- scripts/security-ui.sh: two false positives that bit on the imported
source. The analytics pattern matched the English word "plausible"
(now plausible.io), and `\s` inside an ERE bracket is a literal
backslash and 's', which cut every localhost URL at its first 's' and
then failed the allowlist (now [:space:]).
Verified: httpd suite 64 passed; dev-proxy contract and UI security
audit pass; graph-ui builds and its 2024 vitest tests pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fix-forward after rebasing onto main (545 commits, 26 conflicted-file
resolutions). Where main had shipped a behaviour in the meantime, main
wins; the PR keeps its output contract on top of it:
- search_code: the scan runs under main's supervised, deadline- and
output-bounded runner and feeds the branch's streaming classifier from
the runner's output file; the POSIX wrappers map grep's no-match to 0,
so every non-zero exit fails closed (no exit-1 special case).
- trace_path: strategy/confidence precede args in header and rows (#1542),
in both the tree table and the json legs.
- cypher: relationship expansion is never capped (a cap falsified
aggregates) and unlabeled MATCH scans every candidate; `truncated` is
raised only by a real ceiling — a clamped hop range is probed one hop
beyond the cap and reports truncated only when a candidate exists there.
The plain BFS and the trail BFS both probe one row past max_results so
saturation is observable instead of silent.
- search_code full mode and the final json use main's U+FFFD sanitizer
instead of '?' replacement.
- search_graph json keeps an empty `groups` array for semantic-only calls;
list_projects json echoes offset/limit next to next_offset and accepts
main's include_details spelling for the stats projection.
- discovery budget: main added get_file_outline, compare_graphs, manage_adr
set_sections and the debug/include_details parameters; their descriptions
are trimmed to the lean style and the ceiling moves from 15 to 18 KiB.
Tests that asserted a JSON default now request format:"json"; the
fail-closed search test uses an unreadable regular file (a missing or
non-regular indexed path is deliberately not a scan operand on main); the
non-regular fixture gives its real sources File nodes, as every indexed
project has; list_projects total counts listable projects (the 0-byte
ghost is never a row, so it is not a page either).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The rebase onto main brought the Bash-enabled PreToolUse matcher
(Grep|Glob|Bash), which the exec-form tests still counted as Grep|Glob.
The Windows lifecycle test seeded the .cmd shell spelling as the converged
identity; the shell-free registration converges every owned shell spelling
onto one exec-form entry per matcher instead. The test now asserts that,
plus their removal on uninstall, and hands cbm_install_agent_configs the
'/'-spelled binary path cbm_cmd_install registers (cbm_mkdtemp returns a
native '\' temp root on Windows, which uninstall never derives).
Smoke 8aq proved Claude's gate registration by grepping the shim name out
of settings.json; the exec form carries no shim name, so it now checks the
binary-plus-'hook-augment' PreToolUse hook. 9d's shim-name marker would
otherwise pass vacuously after uninstall; the 'hook-augment' argument is an
owned marker there too.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Step 0o (MCPB bundle contract) and Step 0q (release candidate derivation)
hand the composition gate a stub compiled straight from a .c file with the
compiler's defaults. That stub exists to exercise packaging, format detection
and the needle scans; it is not linked with -z now and never ships. A1d
measured eager binding on it anyway, so on a toolchain that binds lazily by
default (test-diag: clang-22) Step 0o died deterministically with
A1d-bind-now stage/codebase-memory-mcp: GOT section(s) .got.plt fall
outside PT_GNU_RELRO [15824,16384) — they stay WRITABLE after relocation
The property A1d protects is a property of the release link (-z now in
Makefile.cbm), so the assertion is scoped to it: a caller that feeds the gate
a packaging fixture declares so with CBM_COMPOSITION_FIXTURE=1 and A1d is
reported n/a — printed on every run as a named exemption, never a silent
skip. Every other assertion still runs on the fixture unchanged, and release
derivation never sets the variable.
Verified in the Linux arm64 container (Ubuntu 24.04, gcc 13.3): a stub
linked -z lazy reproduces the CI failure text on the gate without the
variable and reports n/a with it; both contracts pass under a CC wrapper
that links lazily (the shape that was red), and unchanged on macOS.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
When a suite leader has already exited, taskkill /T cannot walk its tree,
so run-test-wave.py proves cleanup by asking Windows whether anything is
still parented to the dead PID. That probe was timed with --kill-grace,
which the parallel gate passes as 1s. A cold powershell.exe + CIM start
routinely costs more than that on a runner, so TimeoutExpired -- a fact
about interpreter start-up, not about the process tree -- was read as
"assume the worst" and failed an already-clean shard with exit 2.
Give the probe its own stable-state budget (15s) plus one retry, keeping
the fail-closed semantics: an exhausted or failed probe still counts as
"cannot prove absence". The refusal now names which of the two happened,
an unfinished probe or a counted set of live descendants, instead of
reporting both as a leaked tree.
The contract test pins this structurally -- no sleeps, no timing
thresholds: the probe must be bounded by the module constant rather than
by the kill-grace argument, must retry a timed-out probe, must never read
an unusable probe as absence, and must report the two failures
distinctly. Its Windows leader-exit case now waits on the scheduler's
refusal rather than on a deadline shorter than the new budget.
Refs #1741
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The 8al-node probe imports the generated Pi extension in bare Node with an
empty node_modules. That was a sound stand-in for the Pi host only while
the extension imported nothing but node: builtins; the first package
import (`typebox`, Pi's schema library for tool parameters — a real Pi
dependency, verified against pi-coding-agent 0.84.4 on #1809) can never
resolve there, so the probe fails on a correct extension and would pass a
wrong one just as readily.
Give the probe what the host gives the extension: a Proxy-backed
`typebox` stub in the probe's node_modules (any Type.X(...) returns a
descriptor, so the probe never tracks which helpers the emitter uses — it
only exercises the tool lifecycle), plus an explicit allowlist: every
import specifier in the generated file must be a node: builtin or a
package on PI_HOST_PACKAGES, otherwise the probe fails naming the
specifier. That turns the guarantee the empty node_modules gave by
accident (no surprise dependency) into an asserted contract.
Verified locally by running only the 8al-node block against extensions
generated by two binaries:
main@5802ccdd extension, old probe: OK; new probe: OK
#1809 extension, old probe: ERR_MODULE_NOT_FOUND typebox (the CI red)
#1809 extension, new probe: OK
#1809 extension + an injected 'lodash' import, new probe:
FAIL 8al-node: ... packages the Pi host does not provide: lodash
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The vendoring table claims our copy of a grammar matches the pinned
upstream commit. A local patch makes that claim false unless the patch
is written down, so a future re-vendor would drop the fix without
anyone noticing.
The row covers both changed lines, because both are the same defect in
the same file: OP_SYMBOL_SUPPRESSOR at line 131 and eat_operators at
line 514.
The row also names the two upstream commits that already carry these
changes, and the table intro gains one clause for the case. Upstream
fixed eat_operators in fb63a7004f07 on 2026-04-06 (their issue #558)
and the OP_SYMBOL_SUPPRESSOR entry in 6ab8d1d74ebd on 2026-08-10. Our
pin 8abb3e8b3325 is from 2026-03-20 and predates both, so these are
backports rather than local inventions.
That distinction changes what a re-vendor should do. The table's
instruction is "must re-apply these", which is right for the other four
rows -- upstream never took those. For this row it would mean
hand-applying a change the newer file already has. The row now says to
delete it instead, and the intro allows for that case.
MANIFEST.md is itself listed in scripts/vendored-checksums.txt, so its
own recorded checksum moves with the edit.
Refs #1892
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
scanner.c line 131 shifts a literal 1UL by 32 to build the entry that
suppresses the bang of a try!. unsigned long is 64 bits on Linux and
macOS but 32 bits on Windows, so there the shift count equals the width
of the type, which is undefined.
1ULL is 64 bits on every target this project builds for.
Nothing reports it today. The expression is a compile-time constant, so
no sanitizer sees it run, and Makefile.cbm:719 builds vendored grammars
with -w, which switches off -Wshift-count-overflow.
Upstream tree-sitter-swift made the same change in 6ab8d1d74ebd, after
the commit this grammar is pinned to.
Refs #1892
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
The Swift scanner keeps a 64-bit mask of the symbols that suppress a
match -- the rule that stops `try!` emitting its `!` as a token of its
own. It tests one bit per candidate:
uint64_t suppressing_symbols = OP_SYMBOL_SUPPRESSOR[full_match];
for (uint64_t suppressor = 0; suppressor < TOKEN_COUNT; suppressor++) {
if (!(suppressing_symbols & 1 << suppressor)) {
The mask is uint64_t but the literal `1` is an int, so the shift is an
int shift. TOKEN_COUNT is larger than 32, so once suppressor reaches 31
the shift runs past the width of the type. That is undefined behavior,
and every bit above 31 is tested against a value the standard does not
define.
Nothing caught it because nothing in the tree reached the suppressor
path. Any Swift force-unwrap does: `cached!` is enough.
UBSan reports it as:
scanner.c:514:47: runtime error: left shift of 1 by 31 places
cannot be represented in type 'int'
`1ULL` makes the literal as wide as the mask it is tested against.
The new test in tests/test_extraction.c cannot go red on its own. The
normal test build prints the UBSan message and carries on, which is why
this survived. The Windows CLANGARM64 leg runs UBSan in trap mode, and
there the same shift is an illegal-instruction crash -- so the test
exists to make sure that leg keeps parsing a force-unwrap at all.
scripts/vendored-checksums.txt records the new hash for the one changed
file, as scripts/security-vendored.sh --update writes it. Layer 8 of the
security gate compares vendored content against that manifest, so the
edit and its recorded hash belong in the same commit.
Found while adding Swift URL extraction in #1892 / #1976, and split out
of that PR so the vendored change can be reviewed on its own.
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
Refresh the complete generated Perl grammar bundle from upstream v1.2.1 at c3e17b31179bf8f658c9f37c7a3ea6a202212d5a, raising it from ABI 14 to supported ABI 15. Preserve the existing license and record the source and per-file provenance.
Add deterministic coverage for a valid format followed by a named sub, while retaining a malformed-source control.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Adds -Wl,-z,relro and -Wl,-z,now to ELF_HARDENING_FLAGS, and asserts the
resulting property on the produced artifact rather than on the compiler's
willingness to accept a flag.
The shipped Linux binaries measurably lacked the property. Built the way
.github/workflows/_build.yml builds them (STATIC=1, gcc 13.3 / GNU ld 2.42,
Ubuntu 24.04 aarch64), PT_GNU_RELRO ends at 0x11DA0000 in both cases:
without -z now: .got 0x11d9ec20+0x13c8, .got.plt 0x11d9ffe8+0x40
-> .got.plt ends at 0x11DA0028, 40 bytes PAST the window.
Those GOT slots stayed writable for the whole process
lifetime, in every Linux binary shipped to date.
with -z now: .got.plt folds into .got, the whole GOT sits inside the
window and is re-mapped read-only after startup.
Both flags are free at runtime for a static binary -- everything resolves at
link time, so there is no lazy binding left to pay for. -z relro is already
this toolchain's default and changes nothing here; it is named anyway so the
property stops depending on one distro's spec file, since the musl/portable
and glibc-floor images are different toolchains and a default is not a
guarantee.
A1c/A1d in scripts/ci/check-binary-composition.sh assert the OUTCOME on the
binary via readelf, for the same reason A1 exists beside the .note.GNU-stack
annotation: a flag the linker accepts is not evidence the artifact gained
anything. PR #1138 demonstrated that directly -- its -pie was silently
discarded under -static, and its -D_FORTIFY_SOURCE=2 silently DOWNGRADED
Ubuntu's default of 3, both while every compile probe passed.
The readers use awk rather than a short-circuiting reader on purpose: a
reader that exits at the first match leaves the upstream taking EPIPE, and
under `set -o pipefail` the satisfied case is reported as the failing one.
Deliberately NOT taken from #1138: -D_FORTIFY_SOURCE=2 (a downgrade on the
release platform), -fPIE/-pie (inert under STATIC=1), -fstack-protector-strong
(real but unmeasured cost, and #1138 injected it into SQLite, tree-sitter and
mimalloc -- the indexing hot loop), and the $(shell) probe machinery (~10
compiler spawns per Makefile parse, making the binary a function of the
builder's toolchain).
Co-authored-by: PR9000 <119280965+PR9000@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Seven packaging surfaces declared 0.8.1 while v0.10.8 was the shipped
release — nine releases of drift in metadata published to package
registries, and the third version-metadata drift this month.
pkg/go was not cosmetic: it hardcodes its version into the release URL
it downloads from, so `go install` fetched v0.8.1 binaries.
Chocolatey had to move as a unit. The nuspec is version-only but
chocolateyInstall.ps1 pins the asset sha256, so bumping the nuspec alone
would ship a package declaring 0.10.8 while installing 0.8.1. The new
checksum is the v0.10.8 windows-amd64.zip digest, verified by downloading
the published asset and hashing it, not merely copied from checksums.txt.
The gate (tests/test_version_metadata_contract.sh, scripts/test.sh Step
0x) checks against the newest stable v-prefixed tag, because nothing in
the tree declares the version: CBM_VERSION is injected at build time and
the release is dispatched with a version input. That absence is exactly
why these files drifted. Mutual consistency across the surfaces needs no
git and always runs, so a shallow clone degrades the gate loudly instead
of passing silently.
Homebrew (0.10.3), Scoop and AUR (0.8.1) are left pinned: each embeds
per-release asset checksums, so bumping them is a distribution chore
rather than a metadata edit. They are still gated — internally
consistent, and never naming a release that does not exist — and each
carries a recorded reason for being frozen.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The line read `7x ABI-13, 86x ABI-14, 65x ABI-15`, which sums to 158 against
161 vendored grammars. Counting `#define LANGUAGE_VERSION` in each parser.c
gives 9 / 79 / 73 = 161, so the line was already wrong before Chialisp; bumping
the ABI-14 bucket by one, as the Chialisp change did, would have carried a
four-off error forward under a fresh timestamp.
Recount with Chialisp included: 9 / 80 / 73 = 162, which matches both the
grammar-directory count and the language-count contract. The regeneration
command is now recorded next to the numbers so the next person does not have to
rediscover it:
grep -h '#define LANGUAGE_VERSION' internal/cbm/vendored/grammars/*/parser.c \
| sort | uniq -c
Vendored-integrity gate and the language-count contract both re-run green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only
public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five
reference files from chia-blockchain@main it produced 3-34 ERROR nodes each.
Three defects account for all of them, and each desynchronised the rest of the
file rather than losing one node:
* comments were `/;.*\r\n/` -- CRLF-only, while real files are LF;
* `(include foo.clib)` was rejected because of the dot in the filename;
* `(defconstant NAME <expr>)` accepted only a primitive after the name.
So the grammar is written here instead, in tools/tree-sitter-chialisp/, and
files as first-party: its vendored LICENSE is the project's own, byte-identical
to the repository root, carrying no third-party copyright.
It is deliberately GENERIC -- source_file / list / symbol / string / number /
hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp
form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head
symbols; which lists are definitions is decided in extract_defs.c. A dialect
that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs
no regenerated parser. Acceptance: all five reference files parse with zero
ERROR and zero MISSING nodes; the three defects above are pinned as tests.
Extraction, and the four decisions in it:
* `Constant` is admitted to the cross-file name registry
(cbm_label_is_registry_symbol). Chialisp's condition codes live as
`(defconstant ...)` in a shared .clib and are referenced by name from every
puzzle; labelled but unregistered they are seeded nowhere, and roughly half
of a Chialisp graph would be present as nodes yet never the target of an
edge. All three registry-seeding sites route through that one predicate, so
they cannot diverge. No other language emits `Constant`, so the blast radius
is exactly Chialisp. Pinned by a two-file graph test that goes red when the
label is removed from the predicate.
* `export` / `namespace` are NOT definition heads. `(export foo)` names a
function already defined in the same file, so treating it as a def head
mints a second node for one symbol. They stay in the not-a-call filter.
* A parameter list is a `list` too, but it binds rather than invokes. Without
that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a
phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions.
* `(include *standard-cl-26*)` is a dialect directive, not a file dependency;
recording it would invent a dependency on a file that does not exist.
Three helpers that the defs, calls and unified walks all need -- the quote-data
gate, comment-skipping named-child access, and the Chialisp def-head set -- live
in helpers.c rather than being copied per translation unit. They previously
existed as byte-identical duplicates that had already drifted: extract_defs.c
skipped comments when reading a def head while its mirror in extract_unified.c
did not, so a comment between a head and its name desynchronised defs from call
scope. Both now use the same accessor.
Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87,
first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165.
Distilled from PR #919 by darkverbito, whose extraction logic -- the CLVM
not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and
the include/embed import handling -- is kept and is the substance of this change.
The grammar that PR vendored is not used.
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The watcher-poll family gets its production fix and its doctrine
rebuild in one pass:
- the dirty-state signature statted files through stat(), whose
st_mtime is SECOND-granularity on Windows — two same-second edits
hashed identically and a reindex went missing (the Windows leg
caught it as test_watcher:2045). The signature now reads
cbm_path_info_utf8 (FILETIME 100ns ticks, wide-path correct) and
retries once when a just-written file is transiently unreadable
under AV real-time scanning
- a test seam exposes the committed/pending signatures, and the suite
gains self-diagnosing count asserts — the family blocked two
releases because a bare count mismatch could not say whether
detection missed or the reindex path misbehaved; now it names which
- reach-assertions get a bounded wait-for-state helper (safe: an
unchanged dirty tree never increments, so extra polls cannot
overshoot); must-stay-quiet asserts keep their fixed polls
py_lsp_scale's quadratic detector samples min-of-N so scheduler noise
on a loaded host cannot inflate the calibrated ratio (noise only adds
time; the minimum approximates uncontended cost — the assert failed
the 18-job mac leg while passing isolated on the same tree).
The full-pipeline probe suites join the slow scheduler tier with their
reasons stated: a fresh checkout's first run pays a deterministic
AV-scan-cold tax that killed node_creation_probe at exactly 900s.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Seven surfaces disagreed with reality and with each other: README.md,
docs/index.html, docs/llms.txt and pkg/npm/README.md said 158, while
server.json, the chocolatey nuspec and scripts/package-release.sh said 159.
161 grammars are vendored.
21 individual claims corrected. The count is now gated by
tests/test_language_count_contract.sh, so the next language to land cannot
ship a stale figure.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The count was hand-maintained on seven separate surfaces and had drifted apart
from reality AND from itself: README, docs/index.html, docs/llms.txt and the npm
README said 158, while server.json, the chocolatey nuspec and package-release.sh
said 159.
Derived from the vendored grammar directories rather than CBM_LANG_COUNT,
because the registry contains entries with no parser at all: CBM_LANG_NIM
(grammar removed 2026-06-12) and CBM_LANG_OBJECTSCRIPT_EXPORT have no grammar
directory, no tree_sitter function and no MANIFEST row, yet files still route to
them by extension. Publishing CBM_LANG_COUNT would claim languages we do not
parse. A vendored grammar directory demonstrably exists and is checksum-gated.
Accepted trade-off, stated in the test: this UNDERCOUNTS languages that share
one grammar (TypeScript/TSX, objectscript_udl/_routine). We publish the number
we can prove rather than the larger number we cannot.
The contract binds four things so it cannot go quietly vacuous:
- the grammar directory count is sanity-bounded, so a layout change fails loudly
instead of making every check below it trivially true
- MANIFEST.md's declared figure must already agree with the directory count
- every registered surface must state exactly that number, and a registered
surface that states NO number fails (the wording changed, or it should be
deregistered)
- any UNREGISTERED file stating a language count fails, so an eighth surface
cannot reintroduce the drift this exists to prevent
Files that legitimately say "<N> languages" about something else — benchmark
corpus subsets, probe scopes, the version-pinned winget manifests — are listed
with what their number actually counts, so no exemption is unexplained.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Vendor AndreasMaierDe/tree-sitter-plsql @ 28aebef209be (MIT, ABI 14, no
external scanner) and wire CBM_LANG_PLSQL end to end:
- extraction onto existing labels only: package spec/body -> Class,
procedure/function (create/definition/declaration) -> Function,
file -> Module; callee extraction for ref_call incl. the
package-qualified pkg.name form; identifier usage vocabulary with the
upstream 'parameter' kind (a ref_call ARGUMENT wrapper) carved out of
the whole-binding rule
- discover: .pks/.pkb/.pck/.pls/.plb/.plsql/.fnc/.trg/.bdy/.tps/.tpb
(.sql stays SQL and .prc stays FORM, both asserted negatively)
- ledgers: CALL_WITHOUT_REFERENCE_VOCAB(PLSQL), call-node manifest
ref_call row (historical total 219 -> 220), call-argument matrix B row
- tests: extraction triple (package body + package-qualified call,
standalone function, honest negative pin of the upstream
CREATE TYPE ... AS OBJECT ERROR-node limitation), grammar-regression
case, label golden Class:1,Function:1,Module:1, MUST_EXTRACT_DEFS,
extension mapping, bug-repro board battery row
- MANIFEST.md (community-niche verdict, local include-quote patch row),
THIRD_PARTY.md section, refreshed vendored checksums
PL/SQL support was contributed by Oguz (@ouzsrcm) in #1033; re-vendored
from upstream per vendoring policy with his language wiring distilled on
top.
Co-authored-by: Oğuz <oguzhan.saricam@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Grok Build (xAI's grok CLI/TUI) already reached codebase-memory-mcp through
its Claude/Cursor compat layer, but a Grok-only user got nothing. Detect
$GROK_HOME (default ~/.grok) or a grok CLI and install:
- an owned [mcp_servers.codebase-memory-mcp] managed block in config.toml;
a same-name foreign table is left byte-identical because a second header
would make Grok reject the whole file, while the known owned shape written
by `grok mcp add` is adopted. No env forwarding: grok 1.0.5 passes the
full parent environment to stdio servers (verified), unlike Codex (#1562).
- an owned rules file under $GROK_HOME/rules/ (always scanned, all projects)
- the shared skill under $GROK_HOME/skills/
- Scout/Verify/Auditor agents with named-server mcpInheritance and the tier
allowlist spelled out as the server__tool ids Grok's search_tool/use_tool
dispatcher accepts (inheritance filters per server, never per tool)
- hooks withheld: Grok's passive events discard stdout and PreToolUse honors
only deny/rewrite decisions, so there is no context-output contract.
Verified against grok 1.0.5: inspect discovers the server, agents, skill and
rules; mcp doctor handshakes and discovers 15 tools. --clients=grok token,
README/npm/llms/landing-page matrices (43 -> 44 surfaces), and smoke coverage.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Anton Standrik
Martin Vogel
刘冲
Joshua Richter
DavidHLP
