The supervisor creates the worker's response file before spawning it. A
worker that exited 0 without writing that file left it empty, and
slurp_worker_response read the empty file as "" with status OK. The run
then counted as a SUCCESS with an empty response (the daemon marked the
job successful). The clean-exit branch of worker_terminal_log also
deleted the worker log, the only record of where the run stopped. Only
an oversized response was treated as a failure.
- index_supervisor: an empty response file is WORKER_RESPONSE_READ_EMPTY.
A clean exit with no readable response sets response_missing, keeps the
worker log, records its path, and records the last structured event
the log holds (text or JSON format, last 64 KiB) as last_phase. It is
logged as index.supervisor.no_response with the log path and the last
phase. The outcome stays CLEAN, so the disposition is FALLBACK: an
explicit error, with no in-process fallback and no recovery re-run of a
worker that did not crash.
- mcp: cbm_mcp_index_worker_no_response_failure builds the error
(reason "no_response", last_phase, worker_log, hint). It is used for
the first attempt and for a recovery run.
- daemon application: the job failure message says the worker exited
cleanly without writing a response, and gives the last phase and the
log to inspect.
- pipeline_incremental: the legacy per-file purge loop now logs
incremental.purge.start (changed/deleted counts) and
incremental.purge.progress. Progress lines carry counts only, use a
ceiling stride (at most ~20 lines plus the final one) and never log one
line per file. Before this, nothing was logged between
incremental.edge_snapshot and incremental.purge.
Tests (RED before, green after, RED on revert):
- index_supervisor_clean_exit_without_response_is_named_failure_issue1300
(a real worker subprocess logs a phase and exits 0 without a response)
- daemon_application_clean_exit_without_response_is_named_failure_issue1300
- incr_purge_reports_bounded_progress_issue1300
- index_supervisor_clean_exit_without_response_response_issue1300 (the
contract of the response builder)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Six reports describe the same dead end: the indexing worker dies, the
supervisor points at `logs/.worker-<pid>.log`, and the file is 0 bytes.
Nothing was ever flushed, so not one of #1070, #1130, #1132, #1133,
#1145 or #1450 is reproducible or attributable -- the hint says "crashed
on a file" and the file is never named.
Root cause is buffering, not the crash. The worker's stderr is redirected
to that log file by the supervisor, and every diagnostic goes through
`fprintf(stderr, ...)` with no flush. The C standard only promises stderr
is "not fully buffered"; the Windows CRT gives a redirected stderr FULL
buffering, which is why five of the six reports are Windows. A worker
that aborts, is SIGKILLed by the OOM killer, or is terminated after a
hang takes its whole buffer with it.
- cbm_log_set_crash_durable(): setvbuf(_IONBF) plus a per-line flush in
emit_line. Both, deliberately -- setvbuf covers every writer to the
stream including the plain fprintf startup errors that explain a
worker which never got as far as logging, and the flush covers the
case where setvbuf is refused because the stream was already written
to. Enabled for the worker role only, claimed in main() before the
process writes anything.
- cbm_index_worker_log_begin(): a startup header written and flushed as
the worker's first act -- version, build fingerprint, pid, repo path,
and the worker's own arguments. A control record rather than an info
line, so CBM_LOG_LEVEL cannot restore the 0-byte log.
- Under a crash-durable log the parallel extract pass logs every file
it starts, not just the first two rounds of workers, so the log ends
with the files that were in flight when the worker died. One line per
file, never per node; unchanged for every non-worker caller.
This fixes no crash. It converts six unreproducible reports into reports
we can act on, and every future one arrives with the run identified.
Test: a worker started with a fully buffered stderr -- the state the
Windows CRT hands it, forced on POSIX so the contract binds on all three
legs -- writes diagnostics and is then SIGKILLed. The retained log must
be non-empty, carry the header with the version, pid, repo path and args,
and still hold the line written after it. Reverting the production change
leaves that log at 0 bytes.
The probe kills rather than aborts because Darwin's abort() runs the
stdio cleanup handler and flushes the very buffer the repro depends on
stranding: under abort the reverted build still produced a populated log
and the test was quietly toothless. SIGKILL runs no cleanup, and is
literally #1070's death (signal=9).
Refs #1070
Refs #1130
Refs #1132
Refs #1133
Refs #1145
Refs #1450
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The macOS legs went green with the prior calibration but the 3-core
ubuntu-24.04-arm runner is slower still: each spawned worker re-hashes
the entire ASan test-runner image (hundreds of MB) for the exact-build
fingerprint before signalling readiness, which there takes tens of
seconds. Raise the hang-guard deadlines generously - index_supervisor
60s ready / 90s terminal, frontend 30-90s, cli 45s. These are hang
guards, not benchmarks: large values only slow genuine-hang detection,
never mask a regression. (Root-cause fix - a test-only fingerprint
injection seam that skips the hash - is tracked for follow-up.)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The worker-log dumps settled the CI macOS failures: workers were alive
with created-but-EMPTY logs when the 3 s readiness / 5 s terminal
deadlines expired - busy in startup, not hung and not refused. Worker
startup re-hashes the entire executable image for the exact-build
fingerprint before its first write; for the ASan test-runner that is a
multi-hundred-MB read+SHA-256, which the 3-core CI runner VMs with cold
IO cannot finish inside the old budgets, while a warm 18-thread dev
machine finishes in well under a second.
Raise the fixture budgets to 15 s readiness / 20 s terminal. These are
hang guards, not benchmarks: a wedged worker still fails loudly, and
the failure path now dumps the (empty) worker log as proof either way.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The four readiness/terminal failures on the CI macOS legs persist even
with the suite serialized on a quiet machine, so the worker is exiting
rather than starving - and its exit reason lives in the worker log and
response files, which teardown deletes before the asserts print.
Dump them at each failure site (async readiness, probe non-terminal,
backlog logging, oversize containment) so the next failing run names
the worker's actual error instead of a bare ASSERT.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
Subgurim