test(cli): stop a negative probe from spending 15s proving silence

The macos-15-intel leg killed the cli suite at its 900s wall clock while
reporting 7537 passed, 0 failed. Two separate things were wrong.

The suite was misclassified. cli spends 497s of the 900s default on macos-14,
the FASTEST macOS runner, while macos-15-intel in the same matrix is 2.4-3.6x
slower on comparable suites (daemon_runtime 842s vs 349s, stack_overflow_b
277s vs 76s). 497s at that ratio cannot fit. daemon_runtime, at 842s on that
same runner, passes only because it was already in SLOW_SUITES. cli belongs
there too -- it is deterministic and simply large, which is the only thing that
list is allowed to mean.

The drain test was also waiting on a clock instead of on the system. Profiling
the suite per test (315 tests, 300s wall / 166s cpu) put
cli_install_into_host_namespace_still_drains_host_cohort at 42s wall for 19s of
cpu -- 23s idle, the largest single idle block, where the next worst was 11s.
The existing cli_install_force_quiesces_active_cohort_before_replacing_binary
drains a cohort too and idles 0.5s, which is what pointed at the difference.

The cost was the host_serving probe. Its generous budget exists for the
POSITIVE question -- is this daemon still up? -- where a slow reply on a loaded
runner must not be misread as drained; that fixed a real flake and is kept.
Asked in the negative it inverts: no reply is coming, so the full 15s is spent
establishing silence and ASSERT_FALSE is decided by the timeout expiring rather
than by the daemon's behaviour. The drain is already proven positively in that
test -- install returns 0 only after the activation completed, and the host
child exits CLI_SCOPE_HOST_DRAINED -- so the probe only has to confirm it.

Also shortens the drained child's teardown budget, which retried service_free /
lease_release / manager_free against 10s on a path where the activation had
already torn the service down. Behaviour at the deadline is unchanged; it is
reached sooner when nothing is wedged. Measured at only -3s on its own, kept
because it is correct and free.

Local, same machine, ASan+UBSan: the drain test 42.0s -> 25.8s (-38%), the
suite 300s -> 279s, 315 passed before and after -- no test removed, no
assertion weakened. PR #2188, which runs the same leg without these two tests,
passed macos-15-intel in 37m34s, so the suite was not already over budget on
main.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
Martin Vogel committed 2026-09-12 18:01:46 +02:00
1 parent 9a460b389c
commit fa99ff64eb
2 files changed
+58 -8

No files matched your search

+13 -1
View File
@@ -26,7 +26,19 @@ SUITE_NAME = re.compile(r"^[a-z0-9_]+$")
SUMMARY = re.compile(r"^ (?P<passed>[0-9]+) passed")
FAILED = re.compile(r"(?:^|, )(?P<failed>[0-9]+) failed")
SKIPPED = re.compile(r"(?:^|, )(?P<skipped>[0-9]+) skipped")
SLOW_SUITES = frozenset(("incremental", "store_arch", "daemon_runtime"))
# Suites whose honest runtime does not fit the default per-suite budget, and so
# get --slow-timeout instead. This is a statement about SIZE, never about
# flakiness: every suite here is deterministic and simply long, and a racy suite
# must be made deterministic rather than given more clock.
#
# `cli` joined the list because the classification had gone stale, not because
# anything regressed. It spends 497s of the 900s default on macos-14 -- the
# FASTEST macOS runner -- while the macos-15-intel runner in the same matrix is
# 2.4-3.6x slower on comparable suites (daemon_runtime 842s vs 349s,
# stack_overflow_b 277s vs 76s). 497s at that ratio cannot fit, so the suite was
# killed at 900s and reported as hung. daemon_runtime, at 842s on that same
# runner, survives only because it was already listed here.
SLOW_SUITES = frozenset(("incremental", "store_arch", "daemon_runtime", "cli"))
POLL_SECONDS = 0.05
# WHY: the Windows descendant probe below is a cold `powershell.exe` + CIM