fix(test-impact): close the role table's fail-opens, follow literals through macros

The first commit of this branch is the Smart CI change as of the rebase
onto 9f85f104. A review of it found places where the engine could run
fewer tests than a change needs; each now runs everything or selects the
right tests, with a test that fails when its fix is reverted.

Engine:
- A `#define NAME "literal"` line counted as naming a role but selected
  nobody, so the tests that start the blocking-git probe through their
  marker macros would not have run. The scan now follows NAME, as a whole
  word, to the code that uses it, through chains of up to 4 macros; a
  define it cannot follow runs everything.
- main dispatches through the role table, so no edge leads from main to a
  handler. A table entry the engine cannot read, or whose handler is not
  exactly one function, was skipped; it now runs everything.
- In a runner file, only a test case names a role's literal. The runner's
  own helper that reads the marker no longer counts as a spawner, so a
  role only the runner names runs everything again.
- A handler reached first by a parse gap or by the spawn lift was never
  lifted. Role lifts and parse gaps now run to a fixpoint, before the
  runner check and again after the spawn lift.
- A reached sink that main calls (the blocking-git probe) counted as a way
  into main and hid the real one: #2644 now names the start-up path main
  -> cbm_index_supervisor_capture_build_fingerprint.
- No pointer past the end of a role line; the RUNNER_REACHED note clamps
  its prefix.

CI:
- pkg/npm/README.md, which test_cli reads, runs the test legs instead of
  the docs tier.
- The build rule lists the harness scripts the test legs run:
  run-test-wave.py, path-safety.sh, clean.sh.
- The revert guard rejects the ancestor's coverage when a diff fails.
- The vetoed selection is uploaded (test-selection-vetoed).

Docs: SMART_CI.md states the new fail-safe cases, the version that lacks
the new config keys, the selection base, and that the narrowed leg builds
with gcc while the coverage comes from clang.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
Martin Vogel committed 2026-10-17 20:44:27 +02:00
1 parent 7f6f567840
commit 57b557ced8
8 files changed
+444 -58

No files matched your search

+20 -12
View File
@@ -26,7 +26,10 @@ cannot be affected. When it is unsure, it runs everything:
- a shallow clone, a diff that cannot be read in full, or an invalid config;
- no coverage map, or a coverage map that is not admitted (wrong commit, more
than 7 days old, made for another platform or harness);
- a change that reaches the test runner's `main`;
- a change that reaches the test runner's `main`, or a test-runner role
(below) that no test names;
- a role table entry the engine cannot read, or a macro naming a role's
literal that it cannot follow;
- any failure of the selection job itself.
Each of these makes the run cost more CI minutes. None of them loses coverage.
@@ -37,7 +40,7 @@ Each of these makes the run cost more CI minutes. None of them loses coverage.
changed files ──► Layer 1: lane selector ──► which CI jobs run
(scripts/ci/select-lanes.sh, your own rules)
│
▼ (Linux test legs only, in this repo)
▼ (the Linux x86_64 test leg only, in this repo)
merge base + PR ─► Layer 2: test-impact engine ─► which suites / tests run
(codebase-memory-mcp test-impact select
+ team-artifact bundle from main)
@@ -93,9 +96,9 @@ reports `run_all`. Static reachability alone never omits a suite.
list, `--version`); `--skip-config` skips agent configuration, which a CI
runner does not need. It has no version flag, so it is not a pin.
**Version note.** `v0.12.0` has `test-impact select`/`publish`. It does
not have `key: "path"`, `unreferenced`, `--vetoed-out` or `run_all_rules`;
they ship in the first release after `v0.12.0`. With `v0.12.0`, a config
**Version note.** `v0.12.0` and `v0.12.1` have `test-impact select`/`publish`.
They do not have `key: "path"`, `unreferenced`, `--vetoed-out` or
`run_all_rules`; those ship in a later release. With either version, a config
that uses those keys is invalid, and every answer is `run_all`. Until that
release, build the binary from source (`make -f Makefile.cbm cbm`), as this
repository's producer and `select-tests` jobs do.
@@ -239,8 +242,9 @@ Leave this job out of your aggregate required check. What
4. Runs `test-impact select` and writes `selection-engine.json`. It also
writes a Markdown summary to the job's step summary.
The engine reads `.codebase-memory.json` from the **merge base**, not from the
PR. A PR cannot loosen its own selection. A PR that changes the config runs
The engine reads `.codebase-memory.json` from the **selection base** (the merge
base, or the ancestor whose bundle was admitted), not from the PR. A PR cannot
loosen its own selection. A PR that changes the config runs
everything (`builtin:test-impact-config`).
### (c) Gate job and the test job
@@ -328,16 +332,20 @@ TF_ROLE(TF_PHASE_SPAWNER, "CBM_INDEX_SUPERVISOR", tf_maybe_run_index_worker),
The engine reads the table: a change that only a probe's handler reaches
selects the tests whose source names one of that handler's literals (the argv
the child gets, or a switch that lets production code start it). A reached
handler that no test names, and a change reaching `main` any other way, still
run everything; the engine note names the call chain into `main`.
the child gets, or a switch that lets production code start it), also through
a `#define` that names the literal. The runner's own use of a literal names no
spawner. A reached handler that no test names, a table entry the engine cannot
read, and a change reaching `main` any other way still run everything; the
engine note names the call chain into `main`.
**Only the legs the coverage speaks for should narrow.** Here the coverage
comes from one Linux x86_64 clang-21 producer. Only the Linux x86_64 test leg
downloads the selection (`runner.os == 'Linux' && runner.arch == 'X64'` in
`_test.yml`). The arm64, macOS and Windows legs always run everything,
because code under architecture or platform `#ifdef`s has static evidence
only there.
only there. That leg builds with gcc, not clang: code under a compiler `#if`
has static evidence only too, and `--platform` names the producer's platform,
so the engine's platform check admits the map by construction.
### (d) Flip the variable
@@ -423,7 +431,7 @@ Built-in rules, applied after yours:
| `fixtures` → `referencing_tests`, `path_segment:2` | `tests/fixtures/<name>/...` selects the tests that mention the fixture, down to `tests/fixtures/<name>`. |
| `contracts` → `ignore` | Shell, Python and data contract tests run in every leg's contract step, which is never narrowed. |
| `notices` → `ignore` | Release notices. |
| `repo-data` → `referencing_tests`, `path`, `unreferenced: ignore` | Scripts, docs and packaging. A C test that reads one of these files is selected. A file that no test names selects nothing. |
| `repo-data` → `referencing_tests`, `path`, `unreferenced: ignore` | Scripts, docs and packaging. A C test whose source names one of these files (its path, as text) is selected. A file that no test names selects nothing. |
`coverage.compatibility_paths` lists the harness and coverage tools. Changing
the bytes of any of them, or the `--platform` label, makes the existing coverage