mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-04 10:48:20 +00:00
Martin Vogel
0d783bcca4
The parallel-scheduler contract failed twice on test-windows CLANG64 while running fake suites only: - PR #2394 (run 36364479687): "Windows timeout race refused without a surviving descendant to refuse over". The refusal came 38s after the forced leader exit (a cold powershell/CIM descendant probe, once in the wave loop and again in the cleanup pass), while the fixture's descendant was a `time.sleep(30)` that had already exited on its own. - PR #2345 (run 36199749518): "suite 'hang_after_summary' taskkill could not prove process-tree cleanup". The scheduler bounded taskkill.exe with --kill-grace, which the contract passes as 1s, so a slow cold start of the tool was reported as a failed cleanup. Both verdicts were a race between a clock and runner latency. An audit of the fixture found more of the same shape: - Every hanging fixture ended on its own after 30s. A starved scheduler could therefore see a "hung" leader exit 0, and a descendant the scheduler failed to kill could vanish before the leak check looked. - The 1s --timeout could kill a slow leader before it had printed its summary (asserted pass=1), spawned its descendant or written the descendant pid. That pid was then read back with int() from a file written create-then-write, so a reader could see no file or an empty one. The empty-file read recorded earlier was the scheduler's ready file, which3b9052c9already publishes atomically; the fixture's own writer had the same shape. - The descendant installed its SIGTERM-ignore during its own start-up. A SIGTERM that won that race skipped the SIGKILL escalation the POSIX leg exists to cover, silently and with the verdict unchanged. - On Windows, the stubborn-tree leak check was a single look right after taskkill, but TerminateProcess returns before its target has exited. Fix it by construction, not by budget: - A gate. Every process that must outlive the scheduler's decision blocks reading stdin. That is a pipe whose only write end the contract holds, handed down through the scheduler, and the read returns only at EOF. The contract closes the pipe once the verdict is recorded. If the contract dies, EOF releases everything, so nothing is orphaned and no fixture ends on a clock. - The three hanging scenarios run under the scheduler's existing pre-terminate barrier. They are released only once the fixture has published `<suite>.established`, which happens after the summary is flushed, the descendant has confirmed it is armed, and its pid is known. The marker is written to a temp file and moved into place with os.replace, so a reader sees either no file or the whole file. - The Windows leak check waits on the descendant's process handle. That wait is exact, not a race: the descendant is held by the gate, so the scheduler's kill is the only way it can end. - run-test-wave.py runs taskkill through windows_taskkill_tree() with its own stable-state budget (WINDOWS_TASKKILL_SECONDS = 15), as132e8fc3did for the descendant probe. --kill-grace keeps bounding what its name says. Real runs already pass --kill-grace 15, so they are unchanged. The contract pins this structurally, the same way it pins the probe: exactly one taskkill site, no kill-grace there, and a helper bounded by the constant that fails closed on a timeout, a start failure or a non-zero exit. Every assertion keeps its contract: - A leader that hangs after a green summary is recorded as rc=124 pass=1. - A zero-test child is recorded as rc=97. - The wave continues after a bounded failure. - A SIGTERM-resistant tree is killed in full. - A leader that exits between the timeout decision and termination is still cleaned up on POSIX. On Windows the scheduler refuses (rc=2, naming the cleanup failure) over a descendant that is really alive. A new self-check fails loudly if a held leader or descendant is not live while the barrier holds, that is, if the fixture no longer exercises what those assertions claim. Proof, on macOS arm64 and a Linux arm64 container. The Windows branch was reasoned through; the VM is down. - Torture hook (proof only) that delays each hanging leader 2s, past the 1s timeout. The old fixture is RED 3/3 on every scenario (pass=0, FileNotFoundError on descendant.pid, "lost its bounded result"). The new one is green, including at 8s, and 10/10 under the hook plus CPU load. - The #2394 mechanism: after an injected 31s refusal latency the old descendant is gone and the new one is live. It exits 0.02s after the gate closes. - The taskkill pin is RED on the old scheduler and green on the new one. - Stress, 20 runs each under CPU load. The old full contract passed 20/20 on macOS and 20/20 on Linux, so these hosts do not reproduce the CI timing. The rebuilt held scenarios passed 20/20 on macOS and 20/20 on Linux. The full contract after the change passes a single run on macOS and on Linux, and 20/20 on each leg under CPU load (4 load workers; the Linux container is capped at 4 CPUs). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/ — the canonical leg entries
The doctrine (enforced, not advisory): a venue may provision a machine;
only a canonical leg script may exercise the product. Local CI, PR CI, the
dry run and the release all call the SAME files below — venues differ only in
host specs, architecture and inputs, never in what a leg does. Platform
differences (arm64 sanitizer flags, native Windows behavior, the Linux portable
binary) live INSIDE these scripts, once, never per-venue.
tests/test_venue_parity_contract.sh (run as Step 0j of every test leg) turns
any violation — inline harness logic in a workflow, a bypassed wrapper, a
missing --help — into a red build.
Every entry answers --help (authoritative, agent-oriented: modes, env, exit
codes) and rejects unknown flags with exit 2 + Please consult --help.
The legs
| leg | entry | what a run gives you |
|---|---|---|
| test | test.sh |
DEFAULT = the venue leg: static contracts (Step 0a–0r) + CLEAN ASan+UBSan build + all suites via the parallel harness + prod-binary guards. --suites a,b = iteration mode (incremental rebuild, subset, seconds). --tsan = the ThreadSanitizer leg. CLANGARM64 gets CI's trap-UBSan flags automatically. |
| package wrappers | ci/test-package-wrappers.sh |
Go, npm, and PyPI runtime-set publication/lock suites on the current host. CI runs the same entry on Linux and Windows so both platform lock implementations gate. |
| build | build.sh |
CLEAN production runtime set (native executable + authenticated integration asset; --with-ui adds one content-addressed UI pack). ccache via env.sh makes repeats fast; CCACHE_COMPILERCHECK=content guarantees a hit is byte-identical to a cold compile — never stale. --version, STATIC=1, BUILD_DIR=. |
| lint | lint.sh |
clang-tidy + cppcheck + clang-format (+ no-skips policy). --ci = the CI gate set (no clang-tidy). Drives the same make targets as make lint/lint-ci. |
| smoke (unix) | smoke-local.sh |
Stages a full release fixture, serves it on a kernel-assigned port, runs smoke-test.sh (ALL phases incl. download/install/update E2E) inside a disposable HOME/XDG/TMP sandbox. ui variant makes a missing verified UI pack a FAILURE. CBM_SMOKE_ARTIFACT_DIR = smoke an extracted release artifact verbatim (release mode). |
| smoke (windows) | ../test-infrastructure/vm/vm-smoke.sh |
Same verified runtime-set contract on the real Windows VM, plus the user-PATH registry guard (prepare/verify/cleanup). |
| smoke-invariants | smoke-invariants.sh |
Production-path resilience battery (MCP handshake, all tools invocable, malformed-input handling, supervised crash/hang recovery) — no fixture server or install E2E. smoke.yml runs an explicitly seam-enabled build on the WIDEST source matrix; release artifacts remain seam-free and use the release-shaped smoke legs above. |
| soak | soak-legs.sh |
The release-gating soak SEQUENCE: quick then query-leak (the #581 detector — never reindexes, so RSS growth = query-path leak), each guarded by a completion-summary check. --legs quick for the ASan single-leg variant. Duration is per leg. |
Internal harnesses — never called directly by a venue (the contract forbids
it): smoke-test.sh (phases; wrappers provide fixture server + sandbox),
soak-test.sh (one soak run; soak-legs.sh provides the sequence + guards),
run-tests-parallel.sh (reached through test.sh).
Conventions
- Exit codes: 0 = pass · 2 = usage error · 90 = guard (a run died without its completion summary — never counts as green) · anything else = the leg's real failure.
- Iteration is a flag, not a side-tool: the fast paths (
--suites,--legs) are modes of the SAME entry the gates run, so a dev loop can never drift from the venue behaviour. - Env sandboxing: the smoke wrappers neutralize every agent-config destination override; a smoke can never scribble on your real config.
Recommended workflows
- Iterating on a change:
scripts/test.sh --suites <suite>(seconds, incremental, same ASan+UBSan flags as the gate). List suites:build/c/test-runner --list-suites. Debugging a Windows-on-ARM trap: re-run withSANITIZE=for a plain build, or use the emulatedwin.sh ubsan-*pair for full diagnostics. - Before any push (the 3-OS ladder):
scripts/test.sh(macOS, full) →./test-infrastructure/run.sh full(Linux + TSan + smoke) →test-infrastructure/vm/win.sh test-par+guards+smoke-install(+soakwhen the change touches memory/daemon paths). - Concurrency-touching change: add
scripts/test.sh --tsanearly — the same leg CI gates on. - Release-shaped verification:
CBM_SMOKE_ARTIFACT_DIR=<extracted artifact> scripts/smoke-local.sh <binary> [ui]smokes exactly what would ship. - A leg is red in CI but green locally: first suspect environment shape,
not code — the preflights (
win.shautomatic;scripts/ci/preflight-docker.sh) andtest-infrastructure/README.md's residuals list cover the knowable differences.
See scripts/ci/README.md for the CI plumbing and
test-infrastructure/README.md for the venue map.