Bring CodeQL back to pull requests as jobs of PR CI itself (needed by
ci-ok, no polling), narrowed to first-party code, while push to main,
the weekly run, the dry run and the release keep the full scan.
- codeql.yml: one leg per language (c-cpp, javascript-typescript,
python, actions), scope in .github/codeql/codeql-config.yml. The full
scan traces the whole build, vendored C included. The PR scan
(light: true) analyses each selected language's whole first-party
scope; c-cpp compiles the third-party objects before tracing
(scripts/build.sh --vendored-only) and then traces the first-party
build plus the vendored files the PR changes (--reuse-vendored
--retrace-from, make -W). Diff-informed analysis (codeql-action
v4.38.1, forced on for PR calls) only restricts reported alerts.
The alert check moves into each leg and reads the analysed ref.
Push and PR runs cancel superseded runs per language; dispatched
runs never cancel. Compiler cache off so every compile is traced.
- pr.yml: a codeql job runs the legs of the selected codeql-* lanes;
ci-ok needs it.
- select-lanes.sh: codeql-cpp/-js/-py/-actions lanes, selected by a
change in their scope (any C/C++ incl. vendored; JS/TS, Python and
workflows inside the config's paths minus paths-ignore) and by every
full run (undeterminable diff, workflows, build, CodeQL machinery).
--codeql-scope LANGUAGE prints a list's in-scope paths.
- codeql-retrace-vendored.sh: the vendored C files a PR changes (empty,
never narrowing, when the diff cannot be determined).
- Makefile.cbm: cbm-third-party builds the vendored prod objects only.
- tests: the gate contract pins the new layout; the selector contract
pins that a C change always selects the C scan, an undeterminable
diff selects every language, and a vendored-only change retraces
exactly the changed vendored files. Selector fixtures, ci-ok's job
map, the replay's codeql-action failures (lane codeql-cpp) and the
interface probes follow.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
CodeQL ran on every pull_request push, and PR CI's security / codeql-gate
job polled up to 150 min for that run before ci-ok could pass. With ~30 PRs
pushed at once, the gate jobs held runners while the CodeQL runs they waited
for queued behind them, and PR CI starved for hours.
- codeql.yml: triggers are push to main, a weekly schedule, workflow_call
and workflow_dispatch; pull_request is gone. The open-alert check moves
here, unchanged and still fail-closed, as the `alerts` job (needs
analyze), so every main push and every weekly run fails while any
code-scanning alert is open. Push runs keep cancel-in-progress; called and
scheduled runs get a group of their own. The analysis key is pinned to the
push configuration so dry-run/release uploads do not open configurations
of their own on main.
- _security.yml: codeql-gate removed; callers grant contents: read only.
- dry-run.yml, release.yml: a `codeql` job calls codeql.yml on the
dispatched commit; release verify needs it and requires success.
- select-lanes.sh, require-all-green.sh: the codeql-gate lane is gone from
the selector universe and from ci-ok's job map; no other lane changes.
- tests: the fail-closed contract follows the check to codeql.yml and now
also pins the triggers, the derived callers' grants and release verify;
lane fixtures drop codeql-gate.
- SECURITY.md, MAINTAINERS.md: say where CodeQL runs now.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The first commit of this branch is the Smart CI change as of the rebase
onto 9f85f104. A review of it found places where the engine could run
fewer tests than a change needs; each now runs everything or selects the
right tests, with a test that fails when its fix is reverted.
Engine:
- A `#define NAME "literal"` line counted as naming a role but selected
nobody, so the tests that start the blocking-git probe through their
marker macros would not have run. The scan now follows NAME, as a whole
word, to the code that uses it, through chains of up to 4 macros; a
define it cannot follow runs everything.
- main dispatches through the role table, so no edge leads from main to a
handler. A table entry the engine cannot read, or whose handler is not
exactly one function, was skipped; it now runs everything.
- In a runner file, only a test case names a role's literal. The runner's
own helper that reads the marker no longer counts as a spawner, so a
role only the runner names runs everything again.
- A handler reached first by a parse gap or by the spawn lift was never
lifted. Role lifts and parse gaps now run to a fixpoint, before the
runner check and again after the spawn lift.
- A reached sink that main calls (the blocking-git probe) counted as a way
into main and hid the real one: #2644 now names the start-up path main
-> cbm_index_supervisor_capture_build_fingerprint.
- No pointer past the end of a role line; the RUNNER_REACHED note clamps
its prefix.
CI:
- pkg/npm/README.md, which test_cli reads, runs the test legs instead of
the docs tier.
- The build rule lists the harness scripts the test legs run:
run-test-wave.py, path-safety.sh, clean.sh.
- The revert guard rejects the ancestor's coverage when a diff fails.
- The vetoed selection is uploaded (test-selection-vetoed).
Docs: SMART_CI.md states the new fail-safe cases, the version that lacks
the new config keys, the selection base, and that the narrowed leg builds
with gcc while the coverage comes from clang.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
v0.12.0 could not be installed through npm, pnpm, pip or the Go wrapper on
any platform (#2657): each wrapper refused an archive member above 256 MiB,
and every v0.12.0 binary is 331-335 MB. The v0.11.0 binaries were already
about 300 MB, so the Go wrapper, which has had the cap since v0.10.0, was
broken before; npm and PyPI gained it in v0.12.0. The archive is verified
against the release's checksums.txt before it is unpacked, so a byte cap
only ever refused what we ship.
- npm, PyPI, Go: delete every byte-size cap (compressed archive, per member,
aggregate, checksum manifest, download size). The checksum, the exact
member set, path and link refusal, the member count, HTTPS and the
--version check stay.
- All three honour CBM_DOWNLOAD_URL like install.sh, for testing: https, or
plain http to an exact loopback authority with no proxy and no redirect.
- Smoke: scripts/smoke-wrappers.sh installs the three wrappers from the
checkout against the release fixture and runs the binary each one
installed. smoke-test.sh Phase 13w runs it; SMOKE_REQUIRE_WRAPPERS=1 makes
it required on the PR smoke lanes and in the release smoke, which gates
publishing.
Other install paths found broken while checking every channel:
- Homebrew: the formula was still at 0.10.3 and installed the dynamic Linux
build, which needs glibc 2.38 (Ubuntu 22.04, Debian 12 and RHEL 9 lack it).
It now installs 0.12.0 with the static -portable Linux assets and tracks
the newest release in the version contract like the other surfaces.
- Go: `go install .../pkg/go/cmd/codebase-memory-mcp@vX.Y.Z` never worked
(no pkg/go/v* tags, so Go fetched the whole repository, over its 500 MB
module limit), and @latest built main's wrapper, whose version constant
lags each release. The release now also tags pkg/go/vX.Y.Z, and the
wrapper installs its own module version (the constant only for local and
untagged builds).
- npm: a stable release also moves the `next` dist-tag to itself; `next`
still pointed to 0.9.1-rc.1.
- Docs: winget and Chocolatey were listed as install channels but are not
published; they are removed until they are. Windows arm64 ships and is
now listed.
Fixes#2657.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Under R1, a T0b push (test-infrastructure/, lint config, graph-ui, memwaste
tooling) selected lint and no test leg, so the static contract steps that
police exactly those files -- the VM scripts, the compose and run.sh wiring,
the venue-parity rules -- ran nowhere. Replayed over September, 13 of the 18
T0b pushes ran no contract step at all.
The R1 table now gives T0b lint + contracts, so those PRs get the
contracts job (scripts/test.sh --contracts-only) as T0a PRs do. The
fixture table's six T0b cases expect it, and the history replay asserts
that every push runs the contract steps (the contracts lane or any test
leg). Comments that described the job as docs-only are updated.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The pulls/files API lists at most 3000 files and says nothing when it stops,
and the lane decision is only as complete as that list: a cut list can drop
exactly the file that needed a lane.
select-lanes.sh takes --expect-files N (the PR's changed_files; batch rows
take "expect_files") and decides full with a file_list_truncated reason when
the list has fewer distinct paths than N (a complete list never does: renames
only add their old path), when N is at or over the 3000-file cap, or when the
list itself reaches the cap. The tier still classifies what was listed. An
empty or non-numeric count is a usage error, so a workflow that loses the
count fails loudly instead of skipping the check.
pr.yml passes github.event.pull_request.changed_files. The selector contract
covers short, complete, rename-inflated, at-cap and just-under-cap lists and
the github-output and batch channels; the wiring contract requires the count
to be passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/ci/select-lanes.sh maps a PR's changed-file list to a tier, a lane
set, a full flag and the reasons: each path gets a category, the categories
give a tier (T0a docs .. T5 core), the tier gives the R1 lanes, and a few
categories add lanes of their own (pkg -> wrappers + 3-OS smoke, graph-ui ->
Windows guards, memory core -> memwaste, ...). It fails safe: tier T5, an
unmatched path, an empty list, the selector itself, any workflow, the test
harness, Makefile.cbm and vendored code select every lane. The output is a
pure function of the file set.
tests/test_select_lanes.sh pins the decision table (43 cases), determinism
(order, duplicates, CRLF), the input channels and the lane universe.
scripts/test-impact/replay-selector.sh replays all 767 September 2026 PR
pushes and the 22 real failures from a compact fixture: every push gets its
expected tier (729 equal to the analysis, 38 documented deviations where the
analysis split Makefile.cbm diffs by content and the selector, seeing names
only, is conservatively full), and every real failure's lane is selected.
Both run as Step 0z/0z2 of scripts/test.sh.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel