Files
codebase-memory-mcp/scripts/ci/test-impact-select.sh
T
Martin Vogel 776bb119ac ci(test-impact): team artifact producer, PR shadow and the PR-CI narrowing switch
Producer. test-impact-artifact.yml publishes, on every push to main, the
commit's team artifact bundle: its frozen graph, built incrementally from the
previous bundle, and its per-test coverage map, re-run only for the suites
the push selects (scripts/ci/test-impact-publish.sh). A weekly run rebuilds
both from scratch and fails when the incremental graph's topology differs
from the full one. Bundles are Actions artifacts kept 30 days, never
committed.

Shadow. The report-only test-impact-shadow job also builds the checkout and
runs `test-impact select` against the base side of the PR merge commit
(scripts/ci/test-impact-select.sh). A bundle is used only when it comes from
a successful main-branch producer run for exactly that commit
(scripts/ci/test-impact-bundle.sh). The job stays continue-on-error and
outside ci-ok.

Narrowing switch, PR CI only. While the repository variable
TEST_IMPACT_MODE is `gate`, a select-tests job turns the engine's answer into
the tests the test-unix and test-windows legs run. Unset, nothing changes.
It fails open by construction:
- scripts/test-impact/selection.py is the only place an answer becomes a
  runner filter. Only a well-formed selected lane whose filter matches its
  own suite list narrows; run-all answers, a lane that selects nothing,
  unknown schemas or tokens, and any crash run everything.
- `test` passes a selection only from a successful, narrowed select-tests
  run; ci-ok does not need select-tests. A broken selection costs minutes,
  never coverage.
- scripts/run-tests-parallel.sh launches only the selected suites and keeps
  every guard on the selected list. A selected suite the build does not
  register fails the run, unless the engine saw its registration as
  conditional; then it is dropped with a notice, exactly as a full run of
  that build never runs it. Each runner judges its own tokens, so a selected
  test that does not exist fails its suite.
- The shard manifest records the selection's digest, and
  verify-shard-union.sh requires every shard of a leg to agree on it.
- Dry runs and releases never pass a selection (lane-wiring contract).

Also fixed on the way:
- publish crashed on any run-all push: a run-all lane carries
  "suites": null.
- the git-facts hostile-diff probe re-ran the runner with the parent's
  CBM_TEST_ONLY_FILE still set; under a narrowed run naming its suite it
  re-ran the spawning test, which spawned again. The child now clears it
  (a new test fails without that).
- the select summary crashed on the same null suites.

Contracts: tests/test_test_impact_selection.sh (Step 0z8),
tests/test_harness_selection.sh (Step 2c, the real runner through the real
harness), and new cases in the lane-gate and lane-wiring contracts.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-10-05 18:52:31 +02:00

103 lines
4.6 KiB
Bash

#!/usr/bin/env bash
# test-impact-select.sh — the PR selection with the team artifact: what
# `codebase-memory-mcp test-impact select` runs for this change.
#
# Shadow (default): REPORT-ONLY; pr.yml's test-impact-shadow job runs it
# continue-on-error and ci-ok does not need it.
# --gate DIR (pr.yml's select-tests job, only when the repository variable
# TEST_IMPACT_MODE is `gate`): also turns the answer into what the test legs
# run (scripts/test-impact/selection.py) — DIR/test-only.txt and
# DIR/optional-suites.txt for a narrowed selection — and writes
# `selection=narrowed` or `selection=all` to $GITHUB_OUTPUT. Anything but a
# well-formed narrowed answer, a lane that selects nothing included, is `all`.
#
# The base is the base side of the PR merge commit (HEAD^1 by default). Its
# bundle is fetched with scripts/ci/test-impact-bundle.sh, which only accepts
# the main-branch producer's own run for exactly that commit; only then does
# the selection pass --artifact-verified. Without a bundle the engine builds
# the graph itself and admits no coverage: every reached suite runs whole.
#
# Usage: scripts/ci/test-impact-select.sh --binary PATH --out DIR [--base REV]
# [--platform LABEL] [--gate DIR]
# Writes DIR/selection-engine.json and DIR/summary-engine.md (appended
# to $GITHUB_STEP_SUMMARY when set).
# Exit: 0 = selection written · 1 = the tooling broke · 2 = usage error.
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
BINARY="" OUT="" BASE="HEAD^1" PLATFORM="linux-x86_64-clang21" GATE=""
while [ $# -gt 0 ]; do
case "$1" in
--binary) BINARY="$2"; shift 2 ;;
--out) OUT="$2"; shift 2 ;;
--base) BASE="$2"; shift 2 ;;
--platform) PLATFORM="$2"; shift 2 ;;
--gate) GATE="$2"; shift 2 ;;
*) sed -n '2,24p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 2 ;;
esac
done
[ -n "$BINARY" ] && [ -n "$OUT" ] || exit 2
mkdir -p "$OUT"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT
export CBM_RUNTIME_DIR="$(mktemp -d /tmp/cbm-rt.XXXXXX)" # sun_path must fit
export CBM_CACHE_DIR="$WORK/cache"
mkdir -p "$WORK/engine" && chmod 700 "$WORK/engine"
base="$(git -C "$ROOT" rev-parse "$BASE")"
artifact=()
bundle_rc=0
"$ROOT/scripts/ci/test-impact-bundle.sh" --fetch "$base" "$WORK/bundle" || bundle_rc=$?
if [ "$bundle_rc" -eq 0 ]; then
artifact=(--artifact "$WORK/bundle" --artifact-commit "$base" --artifact-verified)
elif [ "$bundle_rc" -ne 3 ]; then
# No bundle means no coverage: more tests run, never fewer. Say so.
echo "::warning::test-impact bundle fetch failed (exit $bundle_rc): selecting without coverage"
fi
started=$(date +%s)
"$BINARY" test-impact select --repo "$ROOT" --base "$base" --work "$WORK/engine" \
--platform "$PLATFORM" ${artifact[@]+"${artifact[@]}"} > "$OUT/selection-engine.json"
elapsed=$(( $(date +%s) - started ))
python3 - "$OUT/selection-engine.json" "$elapsed" "${#artifact[@]}" > "$OUT/summary-engine.md" <<'PY'
import json, sys
answer = json.load(open(sys.argv[1]))
coverage = answer.get("receipt", {}).get("coverage", {})
lines = ["### test-impact engine", "",
f"- decision: `{answer.get('decision')}` in {sys.argv[2]} s",
f"- team artifact: {'fetched' if sys.argv[3] != '0' else 'none'}; coverage "
f"`{coverage.get('state')}` {coverage.get('rejection_reasons') or ''}"]
for lane in answer.get("lanes") or []:
suites = lane.get("suites") or []
whole = [s["suite"] for s in suites if s.get("mode") == "whole"]
narrowed = {s["suite"]: len(s.get("tests") or []) for s in suites if s.get("mode") == "tests"}
lines.append(f"- lane `{lane.get('lane')}` ({lane.get('decision')}): {len(whole)} whole, "
f"{len(narrowed)} narrowed ({sum(narrowed.values())} tests)")
print("\n".join(lines))
PY
if [ -n "$GATE" ]; then
mkdir -p "$GATE"
selection=all
filter_rc=0
python3 "$ROOT/scripts/test-impact/selection.py" filter "$OUT/selection-engine.json" \
--lane unit --out "$GATE/test-only.txt" --optional-out "$GATE/optional-suites.txt" ||
filter_rc=$?
if [ "$filter_rc" -eq 0 ]; then
selection=narrowed
echo "- PR CI runs $(wc -l < "$GATE/test-only.txt" | tr -d ' ') selection tokens" \
>> "$OUT/summary-engine.md"
else
rm -f "$GATE/test-only.txt" "$GATE/optional-suites.txt"
echo "- PR CI runs every test (selection exit $filter_rc)" >> "$OUT/summary-engine.md"
fi
if [ -n "${GITHUB_OUTPUT:-}" ]; then
echo "selection=$selection" >> "$GITHUB_OUTPUT"
fi
fi
cat "$OUT/summary-engine.md"
if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
cat "$OUT/summary-engine.md" >> "$GITHUB_STEP_SUMMARY"
fi