The test-impact engine certifies the history it diffs and refuses a shallow
clone ("Graft or shallow state prevents history certification"). The shadow
job, the gate-mode select-tests job and the main-branch producer all checked
out with fetch-depth 2. So every shadow answer since #2458 was run_all:
GIT_UNAVAILABLE, then POLICY_UNAVAILABLE, because the merge base's
.codebase-memory.json is read through the same git facts. That was 24 of 24
measured PR runs. All three jobs now check out full history (about 400 MiB,
these jobs only).
The producer never published a bundle either. coverage-map.py exits 1 when a
suite fails under coverage: the map is still written and that suite's tests
are incomplete. test-impact-publish.sh ran it under set -e, so the publish
aborted. The failing suite was test_impact_git. Its hostile-diff probe
re-enters the runner as a child, which inherited CBM_TEST_COVERAGE_DIR,
started a coverage run of its own, and refused the per-test LLVM_PROFILE_FILE
it inherits.
Two changes:
- The probe now clears CBM_TEST_COVERAGE_DIR, as it already clears
CBM_TEST_ONLY_FILE. The child's profile then goes to the spawning test, like
any other child process.
- publish treats exit 1 as a written, conservative map (incomplete tests run
whole), with a warning that names the suite. It fails only on exit 2.
Proof:
- Depth-2 clone at f9ce3865: test-impact select answers run_all
[GIT_UNAVAILABLE, POLICY_UNAVAILABLE] with "Graft or shallow state ...", as
in CI. Same commit with full history: graph certified, run_all
[RUNNER_REACHED] (the change reaches the runner's main).
- Coverage runner (Homebrew LLVM, macOS), test_impact_git: before, 43 passed
and 2 failed ("a coverage run starts with LLVM_PROFILE_FILE=/dev/null");
after, 45 passed. coverage-map.py --suite test_impact_git: 45 tests,
0 incomplete, exit 0.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Git facts fixture ran every Git command with a 10 s quiet timeout. The
diff-driver controls run `git diff --ext-diff` / `--textconv` with a fresh
copy of the test runner as the driver; on a loaded host that copy takes over
10 s to start and finish, Git prints nothing meanwhile, and the timeout killed
it (outcome HANG). test-windows (x86_64) failed
test_git_ancestor_changes_history_and_external_driver_guards_apply that way
(controls[0]), and locally 12 concurrent runs of that test failed 8 times,
every failure HANG at ~10,020 ms. Without the quiet timeout the same run
failed 0 of 12.
The fixture now waits for Git to finish; the suite's own ceiling stays the
backstop. No fixture caller expects a hang.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
macos-15-intel CI fails a different Git-backed fixture on each run of this
branch (spawn ESRCH on 6a29a19c, the native inventory fixture on bb1f6238,
two test_impact_git blob-batch checks on dd598ef4: shallow_before and the
budget variants' controls), and these left nothing to attribute: the facts
open, the inventory control and the blob read returned false silently. They
pass on Apple Silicon, Linux and Windows.
The open, inventory and blob-batch failures of these fixtures now print their
stage and the Git facts error (status, exit code, diagnostic). No verdict
changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The subprocess capture probes, the runner-filter tests and the git-facts diff
probe start (or copy) the test runner itself, by argv[0]. On Windows that is
the caller's spelling: the native python scheduler (run-test-wave.py) finds
the runner by search and passes it without ".exe", and an exact spawn or copy
of that path fails. Started that way, 12 subprocess tests failed and the
runner-filter tests with them; started with ".exe", the same build passed.
tf_runner_image() returns the module path on Windows (cbm_module_path_utf8)
and argv[0] elsewhere; the three sites use it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
PR CI on the first push of these commits found what no local leg had
built yet:
- test_impact_result_validate.c: initialize the count GCC's
-Wmaybe-uninitialized flags (-Werror on every Linux build: smoke,
shadow, analyze).
- tests: split one-line `if` chains GCC reports as misleading
indentation, and drop a redundant `= {0}` it reads as missing braces
(the GCC test runner did not compile).
- tests/test_test_impact_engine.c: realpath() into a PATH_MAX buffer;
glibc's _FORTIFY_SOURCE aborts on anything smaller, as
test_index_resilience.c already notes.
- test_impact_git.c: pass --no-lazy-fetch only when the git executable
takes it (git 2.44+, probed once per session with the option on
--version). Older git rejected it and every engine command failed;
Ubuntu 24.04 ships git 2.43. `-c protocol.allow=never` already turns a
lazy fetch into an error there, which is what the option asks for.
- test_impact_engine.c: the selection and publish job threads release
their thread-local node-type bitset cache before they end, as the
parallel workers and the MCP index thread do. LeakSanitizer reported it
(488 bytes in 13 allocations, test-diag); the Linux arm64 ASan leg
reproduced exactly that and is clean with the fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Producer. test-impact-artifact.yml publishes, on every push to main, the
commit's team artifact bundle: its frozen graph, built incrementally from the
previous bundle, and its per-test coverage map, re-run only for the suites
the push selects (scripts/ci/test-impact-publish.sh). A weekly run rebuilds
both from scratch and fails when the incremental graph's topology differs
from the full one. Bundles are Actions artifacts kept 30 days, never
committed.
Shadow. The report-only test-impact-shadow job also builds the checkout and
runs `test-impact select` against the base side of the PR merge commit
(scripts/ci/test-impact-select.sh). A bundle is used only when it comes from
a successful main-branch producer run for exactly that commit
(scripts/ci/test-impact-bundle.sh). The job stays continue-on-error and
outside ci-ok.
Narrowing switch, PR CI only. While the repository variable
TEST_IMPACT_MODE is `gate`, a select-tests job turns the engine's answer into
the tests the test-unix and test-windows legs run. Unset, nothing changes.
It fails open by construction:
- scripts/test-impact/selection.py is the only place an answer becomes a
runner filter. Only a well-formed selected lane whose filter matches its
own suite list narrows; run-all answers, a lane that selects nothing,
unknown schemas or tokens, and any crash run everything.
- `test` passes a selection only from a successful, narrowed select-tests
run; ci-ok does not need select-tests. A broken selection costs minutes,
never coverage.
- scripts/run-tests-parallel.sh launches only the selected suites and keeps
every guard on the selected list. A selected suite the build does not
register fails the run, unless the engine saw its registration as
conditional; then it is dropped with a notice, exactly as a full run of
that build never runs it. Each runner judges its own tokens, so a selected
test that does not exist fails its suite.
- The shard manifest records the selection's digest, and
verify-shard-union.sh requires every shard of a leg to agree on it.
- Dry runs and releases never pass a selection (lane-wiring contract).
Also fixed on the way:
- publish crashed on any run-all push: a run-all lane carries
"suites": null.
- the git-facts hostile-diff probe re-ran the runner with the parent's
CBM_TEST_ONLY_FILE still set; under a narrowed run naming its suite it
re-ran the spawning test, which spawned again. The child now clears it
(a new test fails without that).
- the select summary crashed on the same null suites.
Contracts: tests/test_test_impact_selection.sh (Step 0z8),
tests/test_harness_selection.sh (Step 2c, the real runner through the real
harness), and new cases in the lane-gate and lane-wiring contracts.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve the current selector, frozen-input inventory, result codec,
coverage collection, runner filtering, and isolated Windows venue work.
Component-level regression proofs are recorded in the PR handover.
The live classified provider, public execution integration, artifact
admission, fresh oracle/shadow evidence, and full platform gates remain
incomplete. This checkpoint does not enable or claim verified narrowing.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel