mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-04 10:48:20 +00:00
fix(test-impact): close the role table's fail-opens, follow literals through macros
The first commit of this branch is the Smart CI change as of the rebase
onto 9f85f104. A review of it found places where the engine could run
fewer tests than a change needs; each now runs everything or selects the
right tests, with a test that fails when its fix is reverted.
Engine:
- A `#define NAME "literal"` line counted as naming a role but selected
nobody, so the tests that start the blocking-git probe through their
marker macros would not have run. The scan now follows NAME, as a whole
word, to the code that uses it, through chains of up to 4 macros; a
define it cannot follow runs everything.
- main dispatches through the role table, so no edge leads from main to a
handler. A table entry the engine cannot read, or whose handler is not
exactly one function, was skipped; it now runs everything.
- In a runner file, only a test case names a role's literal. The runner's
own helper that reads the marker no longer counts as a spawner, so a
role only the runner names runs everything again.
- A handler reached first by a parse gap or by the spawn lift was never
lifted. Role lifts and parse gaps now run to a fixpoint, before the
runner check and again after the spawn lift.
- A reached sink that main calls (the blocking-git probe) counted as a way
into main and hid the real one: #2644 now names the start-up path main
-> cbm_index_supervisor_capture_build_fingerprint.
- No pointer past the end of a role line; the RUNNER_REACHED note clamps
its prefix.
CI:
- pkg/npm/README.md, which test_cli reads, runs the test legs instead of
the docs tier.
- The build rule lists the harness scripts the test legs run:
run-test-wave.py, path-safety.sh, clean.sh.
- The revert guard rejects the ancestor's coverage when a diff fails.
- The vetoed selection is uploaded (test-selection-vetoed).
Docs: SMART_CI.md states the new fail-safe cases, the version that lacks
the new config keys, the selection base, and that the narrowed leg builds
with gcc while the coverage comes from clang.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
1 parent
7f6f567840
commit
57b557ced8
8 files changed
+444
-58
No files matched your search
+20
-12
@@ -26,7 +26,10 @@ cannot be affected. When it is unsure, it runs everything:
|
||||
- a shallow clone, a diff that cannot be read in full, or an invalid config;
|
||||
- no coverage map, or a coverage map that is not admitted (wrong commit, more
|
||||
than 7 days old, made for another platform or harness);
|
||||
- a change that reaches the test runner's `main`;
|
||||
- a change that reaches the test runner's `main`, or a test-runner role
|
||||
(below) that no test names;
|
||||
- a role table entry the engine cannot read, or a macro naming a role's
|
||||
literal that it cannot follow;
|
||||
- any failure of the selection job itself.
|
||||
|
||||
Each of these makes the run cost more CI minutes. None of them loses coverage.
|
||||
@@ -37,7 +40,7 @@ Each of these makes the run cost more CI minutes. None of them loses coverage.
|
||||
changed files ──► Layer 1: lane selector ──► which CI jobs run
|
||||
(scripts/ci/select-lanes.sh, your own rules)
|
||||
│
|
||||
▼ (Linux test legs only, in this repo)
|
||||
▼ (the Linux x86_64 test leg only, in this repo)
|
||||
merge base + PR ─► Layer 2: test-impact engine ─► which suites / tests run
|
||||
(codebase-memory-mcp test-impact select
|
||||
+ team-artifact bundle from main)
|
||||
@@ -93,9 +96,9 @@ reports `run_all`. Static reachability alone never omits a suite.
|
||||
list, `--version`); `--skip-config` skips agent configuration, which a CI
|
||||
runner does not need. It has no version flag, so it is not a pin.
|
||||
|
||||
**Version note.** `v0.12.0` has `test-impact select`/`publish`. It does
|
||||
not have `key: "path"`, `unreferenced`, `--vetoed-out` or `run_all_rules`;
|
||||
they ship in the first release after `v0.12.0`. With `v0.12.0`, a config
|
||||
**Version note.** `v0.12.0` and `v0.12.1` have `test-impact select`/`publish`.
|
||||
They do not have `key: "path"`, `unreferenced`, `--vetoed-out` or
|
||||
`run_all_rules`; those ship in a later release. With either version, a config
|
||||
that uses those keys is invalid, and every answer is `run_all`. Until that
|
||||
release, build the binary from source (`make -f Makefile.cbm cbm`), as this
|
||||
repository's producer and `select-tests` jobs do.
|
||||
@@ -239,8 +242,9 @@ Leave this job out of your aggregate required check. What
|
||||
4. Runs `test-impact select` and writes `selection-engine.json`. It also
|
||||
writes a Markdown summary to the job's step summary.
|
||||
|
||||
The engine reads `.codebase-memory.json` from the **merge base**, not from the
|
||||
PR. A PR cannot loosen its own selection. A PR that changes the config runs
|
||||
The engine reads `.codebase-memory.json` from the **selection base** (the merge
|
||||
base, or the ancestor whose bundle was admitted), not from the PR. A PR cannot
|
||||
loosen its own selection. A PR that changes the config runs
|
||||
everything (`builtin:test-impact-config`).
|
||||
|
||||
### (c) Gate job and the test job
|
||||
@@ -328,16 +332,20 @@ TF_ROLE(TF_PHASE_SPAWNER, "CBM_INDEX_SUPERVISOR", tf_maybe_run_index_worker),
|
||||
|
||||
The engine reads the table: a change that only a probe's handler reaches
|
||||
selects the tests whose source names one of that handler's literals (the argv
|
||||
the child gets, or a switch that lets production code start it). A reached
|
||||
handler that no test names, and a change reaching `main` any other way, still
|
||||
run everything; the engine note names the call chain into `main`.
|
||||
the child gets, or a switch that lets production code start it), also through
|
||||
a `#define` that names the literal. The runner's own use of a literal names no
|
||||
spawner. A reached handler that no test names, a table entry the engine cannot
|
||||
read, and a change reaching `main` any other way still run everything; the
|
||||
engine note names the call chain into `main`.
|
||||
|
||||
**Only the legs the coverage speaks for should narrow.** Here the coverage
|
||||
comes from one Linux x86_64 clang-21 producer. Only the Linux x86_64 test leg
|
||||
downloads the selection (`runner.os == 'Linux' && runner.arch == 'X64'` in
|
||||
`_test.yml`). The arm64, macOS and Windows legs always run everything,
|
||||
because code under architecture or platform `#ifdef`s has static evidence
|
||||
only there.
|
||||
only there. That leg builds with gcc, not clang: code under a compiler `#if`
|
||||
has static evidence only too, and `--platform` names the producer's platform,
|
||||
so the engine's platform check admits the map by construction.
|
||||
|
||||
### (d) Flip the variable
|
||||
|
||||
@@ -423,7 +431,7 @@ Built-in rules, applied after yours:
|
||||
| `fixtures` → `referencing_tests`, `path_segment:2` | `tests/fixtures/<name>/...` selects the tests that mention the fixture, down to `tests/fixtures/<name>`. |
|
||||
| `contracts` → `ignore` | Shell, Python and data contract tests run in every leg's contract step, which is never narrowed. |
|
||||
| `notices` → `ignore` | Release notices. |
|
||||
| `repo-data` → `referencing_tests`, `path`, `unreferenced: ignore` | Scripts, docs and packaging. A C test that reads one of these files is selected. A file that no test names selects nothing. |
|
||||
| `repo-data` → `referencing_tests`, `path`, `unreferenced: ignore` | Scripts, docs and packaging. A C test whose source names one of these files (its path, as text) is selected. A file that no test names selects nothing. |
|
||||
|
||||
`coverage.compatibility_paths` lists the harness and coverage tools. Changing
|
||||
the bytes of any of them, or the `--platform` label, makes the existing coverage
|
||||
|
||||
Reference in new issue
Block a user