fix(test): the ENOSPC bootstrap test no longer lets a deadline decide its verdict

daemon_bootstrap_fails_fast_when_daemon_dies_at_publication went red on
the MSan lane of three PRs that touch no daemon code at all (#2245,
#2158, #2140) with

    tests/test_daemon_bootstrap.c:1116: ASSERT(daemon_named_cause)

and green on a rerun of the identical commit.

Cause, attributed 2026-09-20. The test forks a real daemon host whose
record publication fails with ENOSPC, and asserts that the client names
that cause instead of waiting out its 30 s startup deadline. But before
it listens, the host SHA-256s its own executable image, and the test
runner is ~450 MB: ~3 s natively, 20-32 s under MSan (48 CI samples,
median 25.8 s). The spawn override returned as soon as it had forked, so
the client's 30 s clock raced the host's hash. On a slow run the
deadline fired first, the reaper SIGKILLed a host that had not failed
yet, and the log had no listen_failed line to find. Nothing was wrong
with the product: with a 28 s artificial delay the client does learn the
recorded ENOSPC cause.

The spawn override now reaps the forked host before it returns. The
host writes its start-failure record and only then releases its lifetime
reservation and exits, so "record on disk, reservation released" is a
stable state pinned by construction when the client takes its first look
-- and that first look is guaranteed: the client's post-spawn wait loop
runs at least once even past the deadline, and its first failure check
after a spawn is unthrottled. How long the host takes no longer matters.
No production code changes; the assertions are unchanged.

A regression to the pre-#1828 behaviour (a client that ignores the
record) still fails the unchanged assertions: its message no longer
names the failure, the errno and the path.

Proof, one build with temporary env-switched instrumentation that is
not part of this commit (a sleep in the forked host before it starts,
and a switch back to the old fire-and-forget spawn):

  spawn          forked host delayed   daemon_bootstrap suite
  synchronous    no                    28 passed
  old (async)    31 s                  27 passed, 1 failed:
                                       ASSERT(log_read)
  synchronous    31 s                  28 passed

The old spawn fails under the delay one assertion earlier than CI did
(log_read, not daemon_named_cause): the injected sleep sits before the
host opens its log, whereas in CI the host was killed mid-hash with the
log already open. Same mechanism -- the client's deadline beating the
host -- and the 2026-09-20 attribution reproduced the CI message itself
with a delay placed after the log is opened.

The final tree, without the instrumentation: daemon_bootstrap 28 passed,
three runs in a row.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
Martin Vogel committed 2026-09-22 00:48:20 +02:00
1 parent bf4c476d5e
commit da4bd25a01
1 file changed
+30 -7
+30 -7
View File
@@ -893,6 +893,8 @@ typedef struct {
char parent[BOOTSTRAP_TEST_PATH_CAP];
cbm_daemon_build_identity_t identity;
pid_t children[BOOTSTRAP_ENOSPC_MAX_CHILDREN];
int exit_status[BOOTSTRAP_ENOSPC_MAX_CHILDREN];
bool reaped[BOOTSTRAP_ENOSPC_MAX_CHILDREN];
size_t child_count;
size_t spawn_calls;
} bootstrap_enospc_host_t;
@@ -928,19 +930,40 @@ static bool bootstrap_enospc_host_spawn(void *opaque,
int run_result = endpoint ? cbm_daemon_host_run(&config) : 0;
_exit(run_result == -1 ? 0 : 50);
}
state->children[state->child_count++] = child;
/* SYNCHRONOUS on purpose. The daemon host writes its start-failure record
* and only then lets go of its lifetime reservation and exits, so once it
* has been reaped "record on disk, reservation released" is a STABLE state
* pinned by construction. The client's first look after the spawn (the
* wait loop always runs once and its first failure check is unthrottled)
* therefore finds the record however long the host took to get there.
*
* Returning while the host was still running made the verdict ride on the
* client's 30 s startup deadline: before it listens the host SHA-256s its
* own image, which for the ~450 MB test runner is ~3 s natively and ~26 s
* under MSan (48 CI samples: 20-32 s). The deadline won that race on slow
* runs and the reaper then SIGKILLed a host that had not failed yet --
* `ASSERT(daemon_named_cause)` red on PRs that touch no C at all (#2245,
* #2158, #2140). A deadline must never decide a test (O9). */
size_t slot = state->child_count++;
state->children[slot] = child;
int status = 0;
pid_t waited;
do {
waited = waitpid(child, &status, 0);
} while (waited < 0 && errno == EINTR);
state->reaped[slot] = waited == child;
state->exit_status[slot] = status;
return true;
}
static void bootstrap_enospc_reap(bootstrap_enospc_host_t *state, int *nonzero_exits) {
*nonzero_exits = 0;
for (size_t i = 0; i < state->child_count; i++) {
int status = 0;
pid_t waited;
do {
waited = waitpid(state->children[i], &status, WNOHANG);
} while (waited < 0 && errno == EINTR);
if (waited == 0) {
int status = state->exit_status[i];
if (!state->reaped[i]) {
/* Only reachable when the synchronous wait in the spawn itself
* failed; never leave a host behind. */
pid_t waited;
(void)kill(state->children[i], SIGKILL);
do {
waited = waitpid(state->children[i], &status, 0);