Brings #2198 (benchmark harness runtime isolation, #1696) up to date
with main a5b248f4.
No conflicts. Of the files this branch touches, main changed only
scripts/test.sh since the merge base 8530c8cd: it adds Step 5-alloc
(production allocator contract), far from this branch's Step 0i3,
which is unchanged. scripts/test-runtime.sh, which the harnesses and
their contract test source, is unchanged on main. No caller of either
harness outside docs/EVALUATION_PLAN.md exists on main.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
With CBM_BENCH_KEEP_RUNTIME=1, bench_finish ran `daemon stop` with its
output and exit status discarded and wrote runtime-root.txt regardless.
`daemon stop` refuses while a committed client still uses the daemon and
fails when the daemon does not answer; the handoff then promised a
stopped daemon that was still running.
A refused stop now prints the explanation of the product, writes no
handoff, keeps the root without cleaning it up (the daemon may still be
using it) and exits non-zero. A stop with nothing to stop still exits 0,
so a clean run is unaffected.
Section 7 step 8 of the evaluation plan follows the same rule: the kept
root is removed only if its daemon stopped, and is kept with a message
otherwise.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Both harnesses discarded the output of `daemon start`, so a refused
start was reported only as "private daemon did not start", the symptom
without its cause. The output now goes to a run-private file next to the
index stderr, the shape both scripts already use for the index call, and
is printed under "--- daemon start output ---" when the start fails.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
benchmark-index.sh removed the runtime-root.txt of an earlier run only
after the repository path, the private runtime and the temp directory had
been set up. A run that failed in one of those steps exited non-zero but left
the previous handoff in the reused results directory, and section 7 of
the evaluation plan sources that file unconditionally, so the graph
session would be pointed at a root that step 8 has already removed.
The results directory is now created and the stale handoff removed right
after argument parsing, before any setup step that can fail.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings the branch up to current main (it was 2061 commits behind) so
the new Step 0x2 runs inside the current scripts/test.sh. No conflicts.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/test.sh lists every shell contract test explicitly and does
not glob tests/, so an unlisted test never ran on any leg. Register
tests/test_install_android_guard.sh as Step 0x2, right after Step 0x,
in the same two-line shape as its neighbours.
Give the two PATH stubs (uname, curl) the same shebang every other
stub under tests/ uses, so they are proper executables on every leg
instead of relying on the shell fallback for files without one.
Refs #1202
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
tests/test_pipeline.c: main and this branch added tests at the same spot;
both are kept unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Smoke check 8d required Claude's PostToolUse matcher to be exactly Read. The
drift note widens it on purpose to Read|mcp__.*codebase-memory-mcp__.*, so a
graph tool call can reset the search count, and the macOS PR smoke failed on
it. 8d now requires exactly the new matcher, and besides a Search matcher it
also refuses a catch-all ("*" or empty) matcher, as its comment promised.
Checked against real installs: the previous binary's Read-only settings fail
8d, the new install passes, and the new install plus a catch-all PostToolUse
entry fails.
Refs #509
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
bench_finish keeps the private root on a zero exit status, but the index
call swallowed its failure (`|| echo '{"error":"index failed"}'`), so a
refused index exited 0 and, with CBM_BENCH_KEEP_RUNTIME set, kept an
empty root and wrote runtime-root.txt for an index that does not exist.
benchmark-index.sh now records the CLI's exit status, keeps 00-index.json
valid JSON on failure ({"error":"index failed","exit":N}), and exits 1
once every per-run file — the three timing files included — is written,
when the index failed or reported no project (the search-graph twin's own
test). bench_finish then cleans up as its comment always promised, and
the caller's loop notices. The index call keeps its stderr in a
run-private temp file, the twin's shape, and the failure prints it under
`--- index stderr ---` with the first 500 bytes of the response: for a
one-shot CLI, stderr is the only channel a refusal is reported on. A
stale runtime-root.txt from an earlier run in the same results directory
is removed at start, so a failed run leaves no handoff at all.
The contract test's fixture answers `cli index_repository` with a
minimal envelope when CBM_BENCH_PROBE_INDEX_OK is set, so the keep run
covers the successful-index handoff and asserts project.txt; the
unflagged run asserts a non-zero exit and the surfaced cause after a
refused index; a new keep run with a refused index asserts a non-zero
exit, the timing files, no surviving root and no handoff, a pre-seeded
stale one included. Against the previous head the test fails on
"benchmark-index exited 0 after a refused index".
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
The evaluation plan (docs/EVALUATION_PLAN.md §7) indexes a language with
benchmark-index.sh and then answers graph questions against that index
from its own MCP session. With a run-private runtime (#1696) the index
was gone before that session could start, so the handoff is now explicit
and opt-in.
With CBM_BENCH_KEEP_RUNTIME set, a SUCCESSFUL run stops its daemon,
leaves the private root in place and records the paths that reach it in
<results>/<lang>/runtime-root.txt (sourceable: CBM_BENCH_RUNTIME_ROOT,
CBM_RUNTIME_DIR, CBM_CACHE_DIR). Ownership of that root, including its
removal, passes to the caller. A failed run cleans up regardless: there
is no index worth keeping and nothing may leak. Nothing changes in the
default path or in the three timing files.
§7's skeleton sets the flag, sources the file before the graph session
and removes the root in step 8 instead of the live-store *.db files.
The contract test drives a kept run and requires the root to survive,
the handoff file to be sourceable, and its paths to be the ones the
product processes actually used; the unflagged run still asserts the
root is gone. The kept-run assertions fail against the previous head.
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
Review follow-up for the benchmark harness isolation.
- Both benchmark scripts build the index request with a JSON-escaped
repository path (python3 json.dumps, the spelling soak-test.sh already
uses): a path containing a quote or a backslash produced a payload the
server could not parse.
- benchmark-index.sh reads its three timestamps from one helper backed by
time.monotonic_ns(); an NTP step mid-run no longer skews or negates a
figure whose purpose is comparison across runs.
- benchmark-search-graph.sh keeps the stderr of the index call and of the
response parse in a run-private temp file (removed by the EXIT trap) and
prints it, plus the first 500 bytes of the response, under the existing
"did not report a project" line.
- The contract test now refuses the index from its fixture and requires the
cause to reach the output, and (POSIX only) drives a repository path with
a quote and a backslash and requires the recorded request to parse as
JSON with the resolved path. Both assertions fail against the previous
head of this branch.
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
scripts/benchmark-index.sh and scripts/benchmark-search-graph.sh ran the
product with no runtime or cache of their own: the benchmark repository
was indexed into the operator's live store and every one-shot joined the
operator's account daemon. Source scripts/test-runtime.sh in both, start
the private daemon before timing, record setup-time.txt and
total-time.txt beside index-time.txt, and index from a repository path
instead of querying a project in the live store.
Add tests/test_benchmark_runtime_isolation_contract.sh, which fails
before this change, and wire it into scripts/test.sh.
Part of #1696.
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
The PostToolUse change rendered the manifest twice per install and uninstall
(current and previous generation) and freed both with free(), which grew the
raw allocator count in src/cli/cli.c past its baseline (lint-memory-core).
The manifest is now written with cbm_json_writer_allocator() and released with
cbm_free(CBM_MEM_CLASS_OTHER, ...), including the frees that predate the
change, so cli.c drops from 170 to 166 raw sites and its baseline is lowered
to match. The rendered bytes are unchanged, so byte-exact ownership of
manifests written by earlier installs still holds.
Refs #509
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reworks the ADR-loss-across-reindex fix to Alternative 1 (per the PR review):
remove the race rather than narrow it. ADRs live in a per-project sidecar SQLite
DB ("<graph_db>.adr.db"), separate from the graph DB a reindex rebuilds or
deletes, so there is no capture, re-read, re-apply, or lock — and no sleeps,
polling, or timeout deciding an outcome.
store.c:
- ADR writes (store/delete/update_sections) use a cached read-write handle opened
SQLITE_OPEN_READWRITE|CREATE and configured like the graph store (PRAGMA
busy_timeout=10000 + journal_mode=WAL) so cross-process access waits rather
than failing SQLITE_BUSY. A :memory: store keeps ADRs in the graph DB.
- Reads never write: cbm_store_adr_get uses the cached RW handle if open, else
opens the sidecar SQLITE_OPEN_READONLY (with the same busy timeout) when the
file exists, else reads the legacy graph row without creating anything. When
the sidecar exists but is NOT marked migrated (half-created), it falls back to
the legacy graph row so a partial sidecar never hides an ADR.
- One-time migration is atomic: adr_copy_legacy_once wraps the marker check,
legacy copy, and marker insert in a single BEGIN IMMEDIATE...COMMIT
(re-checking the marker under the lock; ROLLBACK->ERR on failure). New public
cbm_store_adr_migrate_once. Sidecar path via cbm_path_for_file_api (Windows
long-path). Dropped the write-only adr_db_path field.
pipeline.c: migrate a legacy ADR into the sidecar BEFORE the format-change and
forced-full routes unlink the old generation; preserve-and-abort on failure, and
do NOT fail open — a query-open returning NULL preserves unless the final DB
genuinely does not exist. Migration targets the FINAL path (staging suffix
stripped). The early-capture/re-apply machinery and its test hook are gone.
artifact.c: the exported artifact carries the ADR — snapshot_inject_adr always
DELETEs project_summaries first, then injects only when an ADR is present, so a
deleted ADR never resurrects through a lingering legacy row. Import restores the
ADR into the destination sidecar via cbm_store_adr_restore_from_graph_if_absent:
it copies the artifact's ADR (INSERT OR IGNORE — never clobbers a local one)
regardless of the migrated marker, so a teammate who already indexed the project
(marked sidecar, no ADR) still receives the shared decision record. store.c
factors the row copy into adr_copy_rows, shared by the one-time legacy migration
and the ungated restore.
mcp.c: delete_project removes "<db>.adr.db" (+ -wal/-shm/-journal) only after the
graph unlink succeeds; is_project_db_file skips *.adr.db. cli.c: exclude
*.adr.db from index listing/counting. README + memory-core baselines updated.
Tests (file-based): ADR written mid-reindex survives on BOTH the full and delta
routes (route-pinned FORCED_FULL / CLOSURE_REPAIR); a legacy row migrates before
a forced-full/format reindex with no prior read; migration failure (a directory
at <db>.adr.db) preserves-and-aborts with the old generation intact;
delete_project via the MCP handler removes the sidecar + journals; the artifact
export->import round trip keeps the ADR; a deleted ADR does not resurrect on
import; an import does not clobber a local ADR; an import into an already-indexed
destination (marked sidecar, no ADR) restores the artifact's ADR; a half-created
sidecar falls back to the legacy row. The import tests remove the destination
graph DB (keeping the sidecar + marker) before importing, matching production
(try_artifact_bootstrap imports only when the graph DB is missing) so the rename
succeeds on Windows, not only on POSIX.
Closes#516
Signed-off-by: 0xb007ab1e <134006168+0xb007ab1e@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The unit suites link mimalloc's debug build without the global override,
so they could not see the #2562 start-up crash, and nothing guarded the
vendored allocator's local behaviour: prod_mimalloc.o is compiled with -w
and without CFLAGS_COMMON, so -Wdate-time never reaches the vendored
sources, and the macOS recommit patch had no test at all.
tests/test_prod_allocator_contract.sh links three small fixtures against
$BUILD_DIR/prod_mimalloc.o itself:
1. free(NULL) from an ELF .preinit_array hook, before every constructor
(Linux). Also asserts malloc returns a mimalloc-owned block, so a
fixture that did not link the override fails instead of passing.
2. mi_options_print_out's first line is exactly "vX.Y.Z" with no build
clock, and the object is the release build (debug level 0). Upstream
dropped the clock in v3.4.5; this keeps a refresh from bringing it back.
The banner also has no "profiling" line, which pins -DMI_PROFILE=0.
3. _mi_prim_decommit sets needs_recommit on macOS and leaves upstream's
release value elsewhere (local patch 1 in vendored/mimalloc/PATCHES.md).
scripts/test.sh runs it right after Step 5 builds the production binary,
on every Linux and macOS leg; other platforms skip with the reason.
Linux arm64 (glibc 2.39), gcc 13.3 and clang 18.1, 4 runs each: main's
3.3.2 object segfaults in fixture 1 (exit 139) 4/4 with both compilers;
this branch passes all three fixtures 4/4 with both. macOS arm64:
reverting the recommit patch turns fixture 3 red (needs_recommit=0);
an object with a git-describe suffix in the banner, built with
MI_DEBUG=2, or built with MI_PROFILE=1 turns fixture 2 red.
The preinit reproduction follows the test Felix Rath wrote for #2562.
Co-authored-by: Felix Rath <felixm.rath@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
With mimalloc 3.3.2's release build, free(NULL) before the allocator's
constructor dereferenced a page map that does not exist yet. glibc 2.44's
newlocale() does exactly that during libstdc++ start-up, so every Linux
binary died with SIGSEGV before main (reported with a reproduction in
#2562). Upstream fixed it in b2226c19 (mimalloc #1341), first released in
v3.4.4; v3.5.3, upstream's latest published release, contains it.
Vendored from microsoft/mimalloc tag v3.5.3 (tag object deac3dcf, commit
d4881d338125e1cb7c47ba4cfb398d6f7c0c8d45): LICENSE, include/ and src/,
written blob by blob from a bare clone. Before the local patches were
reapplied, the staged LICENSE blob and include/ and src/ trees had the
tag's exact object ids (50 files; src/arena-meta.c is gone upstream).
Afterwards the only differences are two documented hunks:
- src/prim/unix/prim.c: macOS always recommits on decommit, carried from
63fb7b3d (the phys_footprint fix).
- src/prim/osx/alloc-override-zone.c: one comment reworded, because
scripts/security-vendored.sh rejects the upstream "fork()" spelling
even inside a comment. No code change.
The reproducible-banner patch (no __DATE__/__TIME__) is dropped: upstream
removed the build clock in v3.4.5 (mimalloc #1355). The two Makefile
comments that pointed at that patch now say so.
The production object is built with -DMI_PROFILE=0: mimalloc 3.5.2+
compiles its allocation-sampling profiler in by default, with a sample
countdown on the allocation path, and the product never uses it.
vendored/mimalloc/PATCHES.md records the pin, a re-verification recipe,
the check that pins each patch and the refresh procedure; THIRD_PARTY.md
points at it, the SBOM version is 3.5.3 and scripts/vendored-checksums.txt
is regenerated (mimalloc lines only; security-vendored.sh passes).
Measured on Apple M5 Pro (macOS 26.6), full indexes, 2 interleaved reps,
main fceb25d8 (3.3.2) -> this change: TypeScript peak phys_footprint
1.07/1.01 -> 1.04/1.04 GB, peak RSS 1.87/1.84 -> 1.85/1.86 GB, wall
30.6/29.1 -> 29.3/30.0 s; django footprint 0.41/0.41 -> 0.38/0.39 GB,
RSS 0.59/0.59 -> 0.58/0.60 GB, wall 6.5/6.2 -> 6.5/6.5 s. The graphs are
identical: full node and edge hashes, node and edge census, no dangling
edges (django 68022 nodes / 322464 edges, TypeScript 306239 / 835984).
The same tree without the recommit patch peaks at 2.09/2.04 GB footprint
on TypeScript, so 63fb7b3d still does its job. Linux arm64 (glibc 2.39,
gcc 13.3): both trees build without warnings; django peak RSS 0.67/0.68
GB on main, 0.69/0.68 GB with this change.
Co-authored-by: Felix Rath <felixm.rath@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Bring CodeQL back to pull requests as jobs of PR CI itself (needed by
ci-ok, no polling), narrowed to first-party code, while push to main,
the weekly run, the dry run and the release keep the full scan.
- codeql.yml: one leg per language (c-cpp, javascript-typescript,
python, actions), scope in .github/codeql/codeql-config.yml. The full
scan traces the whole build, vendored C included. The PR scan
(light: true) analyses each selected language's whole first-party
scope; c-cpp compiles the third-party objects before tracing
(scripts/build.sh --vendored-only) and then traces the first-party
build plus the vendored files the PR changes (--reuse-vendored
--retrace-from, make -W). Diff-informed analysis (codeql-action
v4.38.1, forced on for PR calls) only restricts reported alerts.
The alert check moves into each leg and reads the analysed ref.
Push and PR runs cancel superseded runs per language; dispatched
runs never cancel. Compiler cache off so every compile is traced.
- pr.yml: a codeql job runs the legs of the selected codeql-* lanes;
ci-ok needs it.
- select-lanes.sh: codeql-cpp/-js/-py/-actions lanes, selected by a
change in their scope (any C/C++ incl. vendored; JS/TS, Python and
workflows inside the config's paths minus paths-ignore) and by every
full run (undeterminable diff, workflows, build, CodeQL machinery).
--codeql-scope LANGUAGE prints a list's in-scope paths.
- codeql-retrace-vendored.sh: the vendored C files a PR changes (empty,
never narrowing, when the diff cannot be determined).
- Makefile.cbm: cbm-third-party builds the vendored prod objects only.
- tests: the gate contract pins the new layout; the selector contract
pins that a C change always selects the C scan, an undeterminable
diff selects every language, and a vendored-only change retraces
exactly the changed vendored files. Selector fixtures, ci-ok's job
map, the replay's codeql-action failures (lane codeql-cpp) and the
interface probes follow.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
CodeQL ran on every pull_request push, and PR CI's security / codeql-gate
job polled up to 150 min for that run before ci-ok could pass. With ~30 PRs
pushed at once, the gate jobs held runners while the CodeQL runs they waited
for queued behind them, and PR CI starved for hours.
- codeql.yml: triggers are push to main, a weekly schedule, workflow_call
and workflow_dispatch; pull_request is gone. The open-alert check moves
here, unchanged and still fail-closed, as the `alerts` job (needs
analyze), so every main push and every weekly run fails while any
code-scanning alert is open. Push runs keep cancel-in-progress; called and
scheduled runs get a group of their own. The analysis key is pinned to the
push configuration so dry-run/release uploads do not open configurations
of their own on main.
- _security.yml: codeql-gate removed; callers grant contents: read only.
- dry-run.yml, release.yml: a `codeql` job calls codeql.yml on the
dispatched commit; release verify needs it and requires success.
- select-lanes.sh, require-all-green.sh: the codeql-gate lane is gone from
the selector universe and from ci-ok's job map; no other lane changes.
- tests: the fail-closed contract follows the check to codeql.yml and now
also pins the triggers, the derived callers' grants and release verify;
lane fixtures drop codeql-gate.
- SECURITY.md, MAINTAINERS.md: say where CodeQL runs now.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
README.md, the site files and pkg/npm/README.md are read by test_cli, so
a change to one now runs the test legs (T1-tests). The September replay
held 15 README pushes to the docs tier the analysis assigned and failed
every test leg's contract step. Those rows are now a documented
deviation, read-by-tests, which like makefile-by-name may only raise a
tier: 767 pushes as expected, 53 documented fail-safe deviations, all 22
real failures still caught.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
Ben Fairless
Anton Standrik
wargloom
0xb007ab1eandClaude Opus 4.8
Attila Oláh
