The hosted Ubuntu x64 runners read the archive through
mirror+file:/etc/apt/apt-mirrors.txt. With Acquire::Retries above 0
(including the apt default of 3), a failover from a dead first entry
of that list (connection refused, DNS failure or a stalled host) to
the next entry goes idle: apt-get fetches a few InRelease files from
the second mirror and then waits until the step timeout. With
Acquire::Retries=0 the same failover completes. Seen on noble
(apt 2.8.3) and jammy (apt 2.4.14).
Every apt step this branch hardened now picks the retry count at
runtime, applied to both update and install: 0 when the runner has the
mirror list (moving to the next mirror is the retry), 3 otherwise
(arm64 runners and any host with plain sources keep their retries).
--error-on=any, the http and https timeouts and the step timeouts are
unchanged.
Proof: the step text extracted from the edited _build.yml (update plus
install of zlib1g-dev), run under bash -e in ubuntu:24.04 and
ubuntu:22.04 containers with runner-shaped mirror lists:
- first mirror refused: exit 0 in 8 s / 9 s, every index and deb
fetched from the second mirror
- first mirror stalled (192.0.2.1): exit 0 in 68 s / 69 s, one 30 s
timeout each for update and install, then the second mirror
- healthy first mirror: exit 0 in 7 s / 9 s
- no mirror list, plain sources: the prefix keeps Retries=3, exit 0
in 8 s / 7 s
- every mirror dead: update exits 100 in under 1 s, install skipped
- revert check, the previous step text with a dead first mirror
(refused or stalled): idle with no Packages index at the 180 s cap
on both releases
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
apt-get update exits 0 when index fetches fail: it only prints
"W: Some index files failed to download. They have been ignored, or
old ones used instead." and the job breaks later at install time with
a misleading "Unable to locate package", or installs from stale lists.
Add --error-on=any to every apt-get update this branch hardens, placed
after the -o options as in test-infrastructure/Dockerfile.msan, so the
update step itself fails when an index cannot be fetched.
Proof in ubuntu:24.04 (apt 2.8.3) and ubuntu:22.04 (apt 2.4.14)
containers with a runner-shaped mirror+file:/etc/apt/apt-mirrors.txt
source and the same -o options CI passes:
- every mirror unreachable: exit 100 with the flag (E: lines), exit 0
without it (W: lines, zero Packages lists), on both apt versions
- first mirror refused or stalled (30 s connect timeout) and the real
archive second: the Ign: failover lines do not trip the flag, exit 0
with every index fetched (measured with Acquire::Retries=0)
- the image sources with the flag: exit 0 and no Ign/W/E lines, so the
option is understood by apt 2.4 and 2.8
- green hosted-runner logs (ubuntu-24.04 and ubuntu-22.04 legs) show no
Ign/Err/W/E lines in the update step
With Acquire::Retries=3 a mirror failover leaves apt idle after the
InRelease failover, with or without the flag and also for a plain
apt-get update; that apt behaviour predates this change and is
unchanged by it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On 2026-11-01 two Linux jobs hung in `sudo apt-get update` right after
"Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease" until
their job limit; the build never started in either:
- PR 2641, test-tsan x86 (run 41251045506, job 124688326533): the
"Install deps (Ubuntu)" step ran 2 h until the 120-minute job limit.
- PR 2395, test-unix gcc x86 (run 41256855177, job 124705168264): the
same step ran 4 h until the 240-minute job limit.
Both logs show the azure mirror ignored after about 15 s, the fallback to
archive.ubuntu.com, and then no output until the cancel. The default apt
http timeout (120 s) did not end the stall.
Rule applied to every workflow step whose run block calls apt:
- A step-level timeout-minutes. 10 for steps that only provision packages
(green runs on 2026-11-01 and the last green dry run: 5 to 76 s); 20 for
the three pr.yml steps that also run scripts/build.sh (green: 246 to
331 s). A stalled mirror now fails the step in minutes instead of
holding the runner until the job limit.
- `-o Acquire::Retries=3 -o Acquire::http::Timeout=30
-o Acquire::https::Timeout=30` on every apt-get update and install,
with the -o options before the subcommand as in
test-infrastructure/Dockerfile.msan.
Nothing else changes: no action, runner, package or step-order changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
GitHub retires the macos-14 runner image on 2026-11-02, and its brownouts
already cancel our macos-14 jobs. Move every macos-14 runner reference to
macos-15 (arm64): the release darwin-arm64 build, the unix-macos14 and
tsan-mac test legs, the smoke-mac PR leg (its two matrix.os guards move in
lockstep), the release smoke, both soak legs, the memwaste event lane, the
bug-repro board, the cache warmer and the all-platform smoke. Lane names
stay unchanged: select-lanes.sh, ci-ok and the lane contracts key on them,
never on the runner label.
Two broad legs already ran on macos-15 (_test.yml BROAD_UNIX, _smoke.yml
BROAD_UNIX) and smoke.yml listed both images, so the moved legs would have
duplicated them. Those duplicates are dropped; PR CI keeps exactly the
same legs.
Without an explicit minimum, clang stamps the binary with the SDK's
default (LC_BUILD_VERSION minos), so a release built on macos-15 would
refuse to start on macOS 14. Makefile.cbm now folds
-mmacosx-version-min=14.0 into CC/CXX next to ARCHFLAGS for Apple targets
(detected from the compiler's target, not the host), so it reaches every
compile and link, vendored objects included, on every build path.
Proof on macOS 26.6 / SDK 26.5: the production binary built from
origin/main reports minos 26.0; with this change it reports minos 14.0,
with zero warnings under -Werror, no weak imports and an identical
undefined-symbol set. The test runner (direct make, no env.sh) is minos
14.0 too; platform and daemon_ipc suites pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Bring CodeQL back to pull requests as jobs of PR CI itself (needed by
ci-ok, no polling), narrowed to first-party code, while push to main,
the weekly run, the dry run and the release keep the full scan.
- codeql.yml: one leg per language (c-cpp, javascript-typescript,
python, actions), scope in .github/codeql/codeql-config.yml. The full
scan traces the whole build, vendored C included. The PR scan
(light: true) analyses each selected language's whole first-party
scope; c-cpp compiles the third-party objects before tracing
(scripts/build.sh --vendored-only) and then traces the first-party
build plus the vendored files the PR changes (--reuse-vendored
--retrace-from, make -W). Diff-informed analysis (codeql-action
v4.38.1, forced on for PR calls) only restricts reported alerts.
The alert check moves into each leg and reads the analysed ref.
Push and PR runs cancel superseded runs per language; dispatched
runs never cancel. Compiler cache off so every compile is traced.
- pr.yml: a codeql job runs the legs of the selected codeql-* lanes;
ci-ok needs it.
- select-lanes.sh: codeql-cpp/-js/-py/-actions lanes, selected by a
change in their scope (any C/C++ incl. vendored; JS/TS, Python and
workflows inside the config's paths minus paths-ignore) and by every
full run (undeterminable diff, workflows, build, CodeQL machinery).
--codeql-scope LANGUAGE prints a list's in-scope paths.
- codeql-retrace-vendored.sh: the vendored C files a PR changes (empty,
never narrowing, when the diff cannot be determined).
- Makefile.cbm: cbm-third-party builds the vendored prod objects only.
- tests: the gate contract pins the new layout; the selector contract
pins that a C change always selects the C scan, an undeterminable
diff selects every language, and a vendored-only change retraces
exactly the changed vendored files. Selector fixtures, ci-ok's job
map, the replay's codeql-action failures (lane codeql-cpp) and the
interface probes follow.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
CodeQL ran on every pull_request push, and PR CI's security / codeql-gate
job polled up to 150 min for that run before ci-ok could pass. With ~30 PRs
pushed at once, the gate jobs held runners while the CodeQL runs they waited
for queued behind them, and PR CI starved for hours.
- codeql.yml: triggers are push to main, a weekly schedule, workflow_call
and workflow_dispatch; pull_request is gone. The open-alert check moves
here, unchanged and still fail-closed, as the `alerts` job (needs
analyze), so every main push and every weekly run fails while any
code-scanning alert is open. Push runs keep cancel-in-progress; called and
scheduled runs get a group of their own. The analysis key is pinned to the
push configuration so dry-run/release uploads do not open configurations
of their own on main.
- _security.yml: codeql-gate removed; callers grant contents: read only.
- dry-run.yml, release.yml: a `codeql` job calls codeql.yml on the
dispatched commit; release verify needs it and requires success.
- select-lanes.sh, require-all-green.sh: the codeql-gate lane is gone from
the selector universe and from ci-ok's job map; no other lane changes.
- tests: the fail-closed contract follows the check to codeql.yml and now
also pins the triggers, the derived callers' grants and release verify;
lane fixtures drop codeql-gate.
- SECURITY.md, MAINTAINERS.md: say where CodeQL runs now.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The first commit of this branch is the Smart CI change as of the rebase
onto 9f85f104. A review of it found places where the engine could run
fewer tests than a change needs; each now runs everything or selects the
right tests, with a test that fails when its fix is reverted.
Engine:
- A `#define NAME "literal"` line counted as naming a role but selected
nobody, so the tests that start the blocking-git probe through their
marker macros would not have run. The scan now follows NAME, as a whole
word, to the code that uses it, through chains of up to 4 macros; a
define it cannot follow runs everything.
- main dispatches through the role table, so no edge leads from main to a
handler. A table entry the engine cannot read, or whose handler is not
exactly one function, was skipped; it now runs everything.
- In a runner file, only a test case names a role's literal. The runner's
own helper that reads the marker no longer counts as a spawner, so a
role only the runner names runs everything again.
- A handler reached first by a parse gap or by the spawn lift was never
lifted. Role lifts and parse gaps now run to a fixpoint, before the
runner check and again after the spawn lift.
- A reached sink that main calls (the blocking-git probe) counted as a way
into main and hid the real one: #2644 now names the start-up path main
-> cbm_index_supervisor_capture_build_fingerprint.
- No pointer past the end of a role line; the RUNNER_REACHED note clamps
its prefix.
CI:
- pkg/npm/README.md, which test_cli reads, runs the test legs instead of
the docs tier.
- The build rule lists the harness scripts the test legs run:
run-test-wave.py, path-safety.sh, clean.sh.
- The revert guard rejects the ancestor's coverage when a diff fails.
- The vetoed selection is uploaded (test-selection-vetoed).
Docs: SMART_CI.md states the new fail-safe cases, the version that lacks
the new config keys, the selection base, and that the narrowed leg builds
with gcc while the coverage comes from clang.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The v0.12.1 release smoke failed on every leg in Phase 13w, the
package-wrapper install gate added in #2664:
- macOS: the runner images have no Go on PATH, and
SMOKE_REQUIRE_WRAPPERS=1 rightly fails without one. pr-smoke and the
three release smoke jobs now set up Go from pkg/go/go.mod, with the
same pinned step the wrapper test job uses.
- Windows: `mkdir -m 700` changes the mode in a second step, which MSYS2
cannot do under the smoke profile's owner-only DACL. mktemp -d sets
the mode at creation.
- Linux: the runner's older Go downloaded the 1.26.1 toolchain into the
smoke profile's HOME. Go writes its module cache read-only, so the
harness could not delete its profile afterwards. All Go state now stays
in the wrapper's own work tree, and its cleanup restores write
permission before removing it.
npm and PyPI passed on every leg that reached them; the gate itself is
unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
v0.12.0 could not be installed through npm, pnpm, pip or the Go wrapper on
any platform (#2657): each wrapper refused an archive member above 256 MiB,
and every v0.12.0 binary is 331-335 MB. The v0.11.0 binaries were already
about 300 MB, so the Go wrapper, which has had the cap since v0.10.0, was
broken before; npm and PyPI gained it in v0.12.0. The archive is verified
against the release's checksums.txt before it is unpacked, so a byte cap
only ever refused what we ship.
- npm, PyPI, Go: delete every byte-size cap (compressed archive, per member,
aggregate, checksum manifest, download size). The checksum, the exact
member set, path and link refusal, the member count, HTTPS and the
--version check stay.
- All three honour CBM_DOWNLOAD_URL like install.sh, for testing: https, or
plain http to an exact loopback authority with no proxy and no redirect.
- Smoke: scripts/smoke-wrappers.sh installs the three wrappers from the
checkout against the release fixture and runs the binary each one
installed. smoke-test.sh Phase 13w runs it; SMOKE_REQUIRE_WRAPPERS=1 makes
it required on the PR smoke lanes and in the release smoke, which gates
publishing.
Other install paths found broken while checking every channel:
- Homebrew: the formula was still at 0.10.3 and installed the dynamic Linux
build, which needs glibc 2.38 (Ubuntu 22.04, Debian 12 and RHEL 9 lack it).
It now installs 0.12.0 with the static -portable Linux assets and tracks
the newest release in the version contract like the other surfaces.
- Go: `go install .../pkg/go/cmd/codebase-memory-mcp@vX.Y.Z` never worked
(no pkg/go/v* tags, so Go fetched the whole repository, over its 500 MB
module limit), and @latest built main's wrapper, whose version constant
lags each release. The release now also tags pkg/go/vX.Y.Z, and the
wrapper installs its own module version (the constant only for local and
untagged builds).
- npm: a stable release also moves the `next` dist-tag to itself; `next`
still pointed to 0.9.1-rc.1.
- Docs: winget and Chocolatey were listed as install channels but are not
published; they are removed until they are. Windows arm64 ships and is
now listed.
Fixes#2657.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The test-impact engine certifies the history it diffs and refuses a shallow
clone ("Graft or shallow state prevents history certification"). The shadow
job, the gate-mode select-tests job and the main-branch producer all checked
out with fetch-depth 2. So every shadow answer since #2458 was run_all:
GIT_UNAVAILABLE, then POLICY_UNAVAILABLE, because the merge base's
.codebase-memory.json is read through the same git facts. That was 24 of 24
measured PR runs. All three jobs now check out full history (about 400 MiB,
these jobs only).
The producer never published a bundle either. coverage-map.py exits 1 when a
suite fails under coverage: the map is still written and that suite's tests
are incomplete. test-impact-publish.sh ran it under set -e, so the publish
aborted. The failing suite was test_impact_git. Its hostile-diff probe
re-enters the runner as a child, which inherited CBM_TEST_COVERAGE_DIR,
started a coverage run of its own, and refused the per-test LLVM_PROFILE_FILE
it inherits.
Two changes:
- The probe now clears CBM_TEST_COVERAGE_DIR, as it already clears
CBM_TEST_ONLY_FILE. The child's profile then goes to the spawning test, like
any other child process.
- publish treats exit 1 as a written, conservative map (incomplete tests run
whole), with a warning that names the suite. It fails only on exit 2.
Proof:
- Depth-2 clone at f9ce3865: test-impact select answers run_all
[GIT_UNAVAILABLE, POLICY_UNAVAILABLE] with "Graft or shallow state ...", as
in CI. Same commit with full history: graph certified, run_all
[RUNNER_REACHED] (the change reaches the runner's main).
- Coverage runner (Homebrew LLVM, macOS), test_impact_git: before, 43 passed
and 2 failed ("a coverage run starts with LLVM_PROFILE_FILE=/dev/null");
after, 45 passed. coverage-map.py --suite test_impact_git: 45 tests,
0 incomplete, exit 0.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Producer. test-impact-artifact.yml publishes, on every push to main, the
commit's team artifact bundle: its frozen graph, built incrementally from the
previous bundle, and its per-test coverage map, re-run only for the suites
the push selects (scripts/ci/test-impact-publish.sh). A weekly run rebuilds
both from scratch and fails when the incremental graph's topology differs
from the full one. Bundles are Actions artifacts kept 30 days, never
committed.
Shadow. The report-only test-impact-shadow job also builds the checkout and
runs `test-impact select` against the base side of the PR merge commit
(scripts/ci/test-impact-select.sh). A bundle is used only when it comes from
a successful main-branch producer run for exactly that commit
(scripts/ci/test-impact-bundle.sh). The job stays continue-on-error and
outside ci-ok.
Narrowing switch, PR CI only. While the repository variable
TEST_IMPACT_MODE is `gate`, a select-tests job turns the engine's answer into
the tests the test-unix and test-windows legs run. Unset, nothing changes.
It fails open by construction:
- scripts/test-impact/selection.py is the only place an answer becomes a
runner filter. Only a well-formed selected lane whose filter matches its
own suite list narrows; run-all answers, a lane that selects nothing,
unknown schemas or tokens, and any crash run everything.
- `test` passes a selection only from a successful, narrowed select-tests
run; ci-ok does not need select-tests. A broken selection costs minutes,
never coverage.
- scripts/run-tests-parallel.sh launches only the selected suites and keeps
every guard on the selected list. A selected suite the build does not
register fails the run, unless the engine saw its registration as
conditional; then it is dropped with a notice, exactly as a full run of
that build never runs it. Each runner judges its own tokens, so a selected
test that does not exist fails its suite.
- The shard manifest records the selection's digest, and
verify-shard-union.sh requires every shard of a leg to agree on it.
- Dry runs and releases never pass a selection (lane-wiring contract).
Also fixed on the way:
- publish crashed on any run-all push: a run-all lane carries
"suites": null.
- the git-facts hostile-diff probe re-ran the runner with the parent's
CBM_TEST_ONLY_FILE still set; under a narrowed run naming its suite it
re-ran the spawning test, which spawned again. The child now clears it
(a new test fails without that).
- the select summary crashed on the same null suites.
Contracts: tests/test_test_impact_selection.sh (Step 0z8),
tests/test_harness_selection.sh (Step 2c, the real runner through the real
harness), and new cases in the lane-gate and lane-wiring contracts.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Under R1, a T0b push (test-infrastructure/, lint config, graph-ui, memwaste
tooling) selected lint and no test leg, so the static contract steps that
police exactly those files -- the VM scripts, the compose and run.sh wiring,
the venue-parity rules -- ran nowhere. Replayed over September, 13 of the 18
T0b pushes ran no contract step at all.
The R1 table now gives T0b lint + contracts, so those PRs get the
contracts job (scripts/test.sh --contracts-only) as T0a PRs do. The
fixture table's six T0b cases expect it, and the history replay asserts
that every push runs the contract steps (the contracts lane or any test
leg). Comments that described the job as docs-only are updated.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The pulls/files API lists at most 3000 files and says nothing when it stops,
and the lane decision is only as complete as that list: a cut list can drop
exactly the file that needed a lane.
select-lanes.sh takes --expect-files N (the PR's changed_files; batch rows
take "expect_files") and decides full with a file_list_truncated reason when
the list has fewer distinct paths than N (a complete list never does: renames
only add their old path), when N is at or over the 3000-file cap, or when the
list itself reaches the cap. The tier still classifies what was listed. An
empty or non-numeric count is a usage error, so a workflow that loses the
count fails loudly instead of skipping the check.
pr.yml passes github.event.pull_request.changed_files. The selector contract
covers short, complete, rename-inflated, at-cap and just-under-cap lists and
the github-output and batch channels; the wiring contract requires the count
to be passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
_test.yml takes a shard_profile input: "standard" (the default; dry run and
release pass nothing and keep ubuntu x3 / windows x2, byte-identical
matrices) or "pr", which pr.yml now uses: ubuntu x86 and Windows run as one
job each and only arm64, the slowest leg, splits in two. A PR runs few legs
and every shard re-pays its setup, so fewer, longer jobs cost less. An
unknown profile is an error. The wiring contract pins both the default and
PR's use of the profile.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A new pr.yml job, test-impact-shadow, predicts from the code graph which C
test suites a change can affect and uploads the prediction, so graph-based
selection can be judged against what actually failed before anything gates
on it. It is continue-on-error and not in ci-ok's needs: it can never block
or pass a merge.
scripts/ci/test-impact-shadow.sh --fetch-release downloads the latest
published release archive for the host plus checksums.txt with the job's
read-only token and verifies the SHA-256 (seconds instead of a build, and a
PR that breaks the indexer cannot corrupt its own prediction); pr.yml builds
the checkout only if that fails. --binary/--out then indexes the checkout
into a throwaway HOME/cache with a private CBM_RUNTIME_DIR (short, under
/tmp, so the daemon socket fits sun_path), runs detect_changes against the
merge commit's base at depth 15 and hands it to predict.py. The graph is
only consulted when C sources changed.
scripts/test-impact/predict.py ports the replay's mapping: impacted test
functions -> registered suites (perf excluded; nested RUN_SUITE children
resolve to the registered suite that runs them), changed test files ->
whole suite, fixtures -> suites that name them, grammars -> grammar suites,
and run-all on the selector's full, a test header, test infrastructure, or
a saturated/truncated impact set. It writes prediction.json and a short
step-summary table.
tests/test_test_impact_predict.sh (Step 0z5) pins the mapping on a
synthetic tree; the wiring contract requires the job to stay
continue-on-error; the script joins the venue-parity interface probes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
pr.yml's `changes` job now lists the changed files (renames with their old
path too), runs scripts/ci/select-lanes.sh in a credential-less step and
outputs tier, full, lanes and the pr-smoke matrix. security, lint and test
pass the lanes to their reusable workflows, which take a `lanes` input
defaulting to "all": dry-run.yml and release.yml pass none and run exactly
as before (setup-matrix output for every broad/shard combination is
byte-identical; the TSan matrix moved into setup-matrix with the same legs).
Under a selection, setup-matrix keeps only the tagged legs of the selected
lanes (an untagged leg is an error, never a silent drop) and each lane job
has an `if:` on its lane. pr-smoke and memwaste run on their lanes instead of
the old product regex. Docs-only PRs (T0a) get a `contracts` job running the
new `scripts/test.sh --contracts-only` (Step 0 contracts, no build). ci-ok
needs contracts, lists the run's jobs (actions: read) and passes LANES/JOBS
to require-all-green.sh; shard-completeness passes --lanes.
tests/test_lane_wiring_contract.sh (Step 0z4) pins it: dry run and release
pass no lanes and the inputs default to all, pr.yml wires the selection
through, and every lane is served by a tagged leg or an `if:` naming it.
The smoke-fixture contract now checks the selector's decisions for the
smoke-harness paths instead of the removed regex; select-lanes.sh joins the
venue-parity interface probes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Found while proving the previous commit, in the same step, and the worse
of the two defects because it fails OPEN. The classification was
if printf '%s\n' "$FILES" | grep -qE '^(src/|internal/|...)'; then
under the runner's `bash -eo pipefail`. grep -q exits on its first match;
once the list is larger than the pipe buffer printf is still writing,
dies of SIGPIPE, and pipefail turns the MATCH into a failed pipeline. The
else branch then writes product=false, and pr-smoke and memwaste are
skipped -- on exactly the PRs that change the most files. The previous
commit's message flagged this as a follow-up; it is fixed here instead.
The list now reaches grep through a here-string, so there is no pipe to
break. The regex is unchanged.
Proof, by executing the workflow's own run block under
`bash -eo pipefail` with a stub gh, ten runs each:
849 KB list, first line src/a.c before: product=false 10/10
after: product=true 10/10
849 KB list, docs only product=false before and after
small docs-only list product=false
422 fallback on the real #2246 product=true
fallback, nonexistent merge ref rc 128, no output written
It takes roughly 1,500 changed files to cross the buffer, which is why
ordinary PRs never showed it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The `changes` job decides whether a PR touches product code, and ci-ok
requires it. It asks GitHub for the PR's file list. An earlier fix moved
it off the .diff endpoint, which rejects PRs over 20k lines, onto the
paginated pulls/files endpoint -- but that endpoint renders the diff
server-side as well, and for a PR that regenerates a vendored parser it
answers
HTTP 422: Sorry, this diff is taking too long to generate.
Observed on #2246 (a 42 MB sql/parser.c): `changes` red, therefore ci-ok
red, for a reason that has nothing to do with the contribution. The
compare endpoint fails the same way, so there is no API route to the
file list for such a PR. Every grammar refresh would hit this.
When the API call fails, the job now takes the list from git instead.
The PR merge ref's first parent is the base, so a name-only diff across
that one merge commit is exactly the PR's changed files. The fetch is
depth 2 with --filter=blob:none into a bare repository under RUNNER_TEMP:
commits and trees only, no blobs, nothing checked out and nothing from
the PR executed. No new action, no new permission, and the API path and
the classification regex are untouched, so ordinary PRs behave as before.
A ::notice:: line records when the fallback was used. If the fetch fails
too, the step exits non-zero and writes no output -- the gate fails
closed, as it did before.
Proof, by executing the workflow's own run block under
`bash -eo pipefail` with a stub gh:
API path, docs-only list rc 0 product=false
fallback, real #2246 rc 0 product=true, the same five
files `git diff --numstat`
reports for the PR; 0.85 s,
188 KB fetched
fallback, nonexistent merge ref rc 128 no output written
Contract tests that read pr.yml -- security_gate_fail_closed,
smoke_fixture_contract, windows_bundle_contract, venue_parity_contract --
pass.
Not addressed here, noted for a follow-up: the classification pipes the
list into `grep -q` under pipefail, which can report a match as a
failure once the list outgrows the pipe buffer.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Answers a question the existing tooling could not: of everything we
allocate, how much is never used?
Lanes, all driven by scripts/memwaste.sh:
event every allocation observed through the memory core — size, age,
churn, and whether the bytes were ever written
access SanitizerCoverage load/store instrumentation, so "allocated and
never touched" is measured rather than inferred
scaling k vs 2k replicas through clang profile counters, to catch
super-linear work per file
padding allocator slack per size class
ratchet the numbers recorded per venue so a regression is visible
The event lane needed a foreign-free counter: on Windows the override sees
frees for blocks it never allocated (they come from the CRT), and counting
those as untracked tripped the soundness gate. mem_events now separates
them (14 daemon / 5 worker on the Go corpus).
Two portability fixes the lanes surfaced, both real:
- the enable decision must not be cached before `environ` exists. Under
-fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our
wrapper before glibc sets up environ, so the lane silently disabled
itself and produced no dump at all on Linux.
- the access map needs 48 bits: arm64 Linux hands out addresses above
2^47, which a 47-bit map dropped on the floor.
mem_profile is retired into this: it measured a strict subset and had its
own sampling story.
Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on
all three platforms report-only, and pr.yml/release.yml carry the wiring.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The codeql-gate has never checked anything. It declared no permissions block,
so it inherited the workflow's contents:read and the code-scanning alert API
answered 403. gh writes the API error BODY to stdout, so
ALERTS=$(gh api '.../code-scanning/alerts?state=open' --jq 'length' 2>/dev/null || echo "0")
left ALERTS holding
{"message":"Resource not accessible by integration",...,"status":"403"}0
every [ ... -gt ... ] then failed as a non-integer comparison, the if took its
false branch, and the step printed "CodeQL gate passed (0 alerts)" and exited 0.
Observed in the log of a GREEN run (34700514102, job 103571322660) while the
repository had an open high-severity alert. The `|| echo "0"` reads as defensive
and is the exact opposite: it converts "I cannot see the alerts" into "there are
no alerts", which is the one direction a security gate must never fail in.
Grant the job security-events:read (and actions:read, which the CodeQL-run poll
needs for the same reason), and replace the fallback with a helper that treats a
non-zero gh exit or a non-numeric body as a hard failure. The helper returns
rather than exits because it runs inside a command substitution, where an exit
would end only the subshell and let the caller continue with an empty count --
the same silent pass in a new costume; both call sites check the status.
tests/test_security_gate_fail_closed.sh pins both halves and fails with six
assertions against the previous workflow.
Also bumps graph-ui's vitest to ^4.1.11 (GHSA-82fw-gwwq-j7x9, path traversal via
@vitest/mocker, CVSS 5.9). Development scope, and graph-ui ships in no release
artifact, so nothing released was ever exposed -- but it was the sole open alert
and the gate above is worthless while it stands unresolved. Raising the floor
past the vulnerable range also stops a fresh install resolving back into it.
graph-ui: npm ci clean, 47 tests in 12 files pass on 4.1.11.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dry run 30225685743 failed at the ensure-defender step on every
Windows job: the hosted images policy-lock real-time protection OFF
(WinDefend starts, Set-MpPreference accepts, RTP stays off —
untamperable from the job).
Revert to the pre-gate runner behavior: no ensure-defender step in
runner jobs, the release Defender scan is on-demand and fail-soft
again. The local VM preflight KEEPS the fail-closed Defender-ON gate
(same canonical script) — AV-interaction coverage is a deliberate
local superset where the platform allows it.
The venue-parity contract now enforces the INVERSE for runners
(ensure-defender must not gate hosted jobs) and still requires the VM
preflight gate; VM README documents the verified posture delta.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes the closable local-vs-remote venue gaps so that, outside arch
legs / YAML glue / release plumbing, every local red is a remote red
and vice versa on shared legs:
- scripts/package-release.sh: THE canonical archive step (names +
five-file Windows bundle layout defined once); _build.yml's eight
inline archive blocks become calls to it.
- scripts/ci/smoke-artifact.sh: the artifact-flow smoke lane — build,
package, extract, then the canonical wrapper in artifact mode; wired
as run.sh smoke-artifact (compose service), win.sh smoke-artifact,
and directly runnable on macOS. Archive-layout bugs now surface
locally instead of in a release dry run.
- glibc-floor leg (Dockerfile.glibc22 + compose + run.sh): portable
binary smokes on ubuntu-22.04/glibc 2.35; the dynamic binary must
refuse there (2.38+ floor by design).
- Defender-ON parity (user directive): scripts/ci/ensure-defender.ps1
enables + VERIFIES real-time protection, fail-closed; runs in every
Windows CI job (_test x2, _soak x3, _smoke, pr.yml) AND in the VM
preflight; _smoke.yml's scan engine-failure soft-skip becomes a red
gate. Expected cost: slower Windows jobs (AV scanning during
install/build/test I/O); the next dry run proves the runner side.
- Contracts: launcher-bundle five-file check retargeted onto
package-release.sh + per-archive canonical-call association;
venue-parity contract requires the new lanes, counts one
ensure-defender step per Windows job, adds --help probes for the new
entries. Extended contract fails on the pre-change tree (verified:
20 violations on HEAD).
- VM README: ephemerality/Defender posture documented — utmctl has no
snapshot verb, so per-run revert stays a manual qcow2 option; the
sweep preflight remains the standing mechanism.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The staged venue-unification bundle: _smoke.yml onto the shared
wrappers with an extracted-artifact input (CBM_SMOKE_ARTIFACT_DIR),
soak-legs.sh as the one canonical soak entry (quick + query-leak legs
across all venues), protected per-user TEMP roots shared across
venues, clean-disk preflights before every VM/Docker run, win.sh and
run.sh routed through the canonical scripts, soak.yml retired, and
the venue-parity contract (whitelist walker: workflows may provision
or call canonical entries, nothing else).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Flip shard_suites on for all three callers of the reusable test
workflow. This commit is deliberately the only topology change:
reverting it alone restores the previous single-job-per-leg CI shape
while keeping the (inert without it) sharding machinery and the
cache-warm pipeline.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Combines the ccache steps with this branch's Windows hardening: the
protected temp root and profile staging keep their roles, the cache
steps slot alongside them, and the strictly per-ref keys apply
unchanged. The daemon branch's own CI legs warm from the second push
on.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
ccache with CCACHE_COMPILERCHECK=content everywhere: every cache entry
is keyed on the CONTENT of the compiler binary plus the fully
preprocessed translation unit, so a hit is provably the identical
compilation - a stale, foreign, or corrupted cache can only MISS, never
return wrong output. No CCACHE_BASEDIR and no path rewriting: debug
info and sanitizer report paths stay exact.
Locally scripts/env.sh routes compilers through ccache's masquerade
directories when present (opt-out CBM_NO_CCACHE=1) - $CC keeps its
plain name, so verify_compiler, make, and link lines are untouched.
CI caches are strictly per-ref by policy on top of GitHub's own branch
scoping: keys embed github.ref, so no base-branch fallback - a new PR
builds cold once and only its own pushes warm it. Cached: the four
test jobs and the pr-smoke matrix. Release builds in _build.yml stay
deliberately uncached.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
PR CI ran tests but never built the binary or smoke-tested it, so built-binary
regressions only surfaced at the release dry run — e.g. the Windows
CreateProcess argv-quoting bug (worker got mangled JSON) slipped all the way to
the release gate.
Add a path-gated light smoke: on PRs that touch product code (src/, internal/,
Makefile.cbm, build.sh, smoke-test.sh, env.sh), build the PRODUCTION binary and
run scripts/smoke-test.sh on ubuntu-latest + macos-14 + windows-latest (native,
reliable — not the flaky broad/emulated legs). Every leg gates. Docs/CI/test PRs
skip it and stay fast (gh pr diff --name-only filter). The full broad/emulated
smoke stays in the release dry run.
SMOKE_DOWNLOAD_URL is intentionally unset so smoke-test.sh's download/update
phases self-skip; the core index/search/trace phases (which catch the
CreateProcess class) still run. ci-ok now treats skipped as OK so PRs that skip
smoke still pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A new push to a PR (or ref) now cancels the in-progress validation for
the previous commit instead of letting obsolete pipelines run to
completion. Drafts deliberately keep the full gate set — multi-platform
feedback matters most during iteration.
Signed-off-by: Martin Vogel <martin.vogel@datadice.io>
PRs now run security gates, lint, and the full test suite (perf
assertions excluded — they stay in dry runs and releases where a
timing-flaky red cannot block a merge). Builds, smoke and soak remain
maintainer-driven.
Signed-off-by: Martin Vogel <martin.vogel@datadice.io>
Pull requests run the security island only — static audit, license
gates, and the CodeQL gate plus the ci-ok summary; the full dry-run
chain (lint/test/build/smoke) stays maintainer-driven via
workflow_dispatch. The CodeQL gate now resolves the PR head SHA instead
of the synthetic merge commit so it can find the analysis run.
Signed-off-by: Martin Vogel <martin.vogel@datadice.io>
Pull requests now execute the dry-run chain (security including the
license gates, lint, full test suite, all build legs, smoke) plus a
single ci-ok summary job that fails unless every stage succeeded.
Branch protection requires dco + ci-ok, so nothing unverified can merge
— including from admins. CodeQL also runs on pull requests so the
security island completes on PR commits.
Signed-off-by: Martin Vogel <martin.vogel@datadice.io>
Martin Vogel
dependabot[bot]
Martin Vogel