Three more cases in the benchmark runtime isolation contract, each RED
against the harness without its fix:
- a stale runtime-root.txt is gone after a run that fails during setup
(a repository path that does not exist);
- with the keep flag and a `daemon stop` that refuses, the run fails,
writes no handoff, keeps its root and prints the refusal; the test
removes that root itself, before any verdict, so a failing assertion
leaks nothing;
- a refused `daemon start` fails both harnesses, prints the cause and
leaves no root behind.
The fixture refuses the start on stderr and the stop on stdout, the
channel the product uses for a busy refusal.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
bench_finish keeps the private root on a zero exit status, but the index
call swallowed its failure (`|| echo '{"error":"index failed"}'`), so a
refused index exited 0 and, with CBM_BENCH_KEEP_RUNTIME set, kept an
empty root and wrote runtime-root.txt for an index that does not exist.
benchmark-index.sh now records the CLI's exit status, keeps 00-index.json
valid JSON on failure ({"error":"index failed","exit":N}), and exits 1
once every per-run file — the three timing files included — is written,
when the index failed or reported no project (the search-graph twin's own
test). bench_finish then cleans up as its comment always promised, and
the caller's loop notices. The index call keeps its stderr in a
run-private temp file, the twin's shape, and the failure prints it under
`--- index stderr ---` with the first 500 bytes of the response: for a
one-shot CLI, stderr is the only channel a refusal is reported on. A
stale runtime-root.txt from an earlier run in the same results directory
is removed at start, so a failed run leaves no handoff at all.
The contract test's fixture answers `cli index_repository` with a
minimal envelope when CBM_BENCH_PROBE_INDEX_OK is set, so the keep run
covers the successful-index handoff and asserts project.txt; the
unflagged run asserts a non-zero exit and the surfaced cause after a
refused index; a new keep run with a refused index asserts a non-zero
exit, the timing files, no surviving root and no handoff, a pre-seeded
stale one included. Against the previous head the test fails on
"benchmark-index exited 0 after a refused index".
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
The evaluation plan (docs/EVALUATION_PLAN.md §7) indexes a language with
benchmark-index.sh and then answers graph questions against that index
from its own MCP session. With a run-private runtime (#1696) the index
was gone before that session could start, so the handoff is now explicit
and opt-in.
With CBM_BENCH_KEEP_RUNTIME set, a SUCCESSFUL run stops its daemon,
leaves the private root in place and records the paths that reach it in
<results>/<lang>/runtime-root.txt (sourceable: CBM_BENCH_RUNTIME_ROOT,
CBM_RUNTIME_DIR, CBM_CACHE_DIR). Ownership of that root, including its
removal, passes to the caller. A failed run cleans up regardless: there
is no index worth keeping and nothing may leak. Nothing changes in the
default path or in the three timing files.
§7's skeleton sets the flag, sources the file before the graph session
and removes the root in step 8 instead of the live-store *.db files.
The contract test drives a kept run and requires the root to survive,
the handoff file to be sourceable, and its paths to be the ones the
product processes actually used; the unflagged run still asserts the
root is gone. The kept-run assertions fail against the previous head.
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
Review follow-up for the benchmark harness isolation.
- Both benchmark scripts build the index request with a JSON-escaped
repository path (python3 json.dumps, the spelling soak-test.sh already
uses): a path containing a quote or a backslash produced a payload the
server could not parse.
- benchmark-index.sh reads its three timestamps from one helper backed by
time.monotonic_ns(); an NTP step mid-run no longer skews or negates a
figure whose purpose is comparison across runs.
- benchmark-search-graph.sh keeps the stderr of the index call and of the
response parse in a run-private temp file (removed by the EXIT trap) and
prints it, plus the first 500 bytes of the response, under the existing
"did not report a project" line.
- The contract test now refuses the index from its fixture and requires the
cause to reach the output, and (POSIX only) drives a repository path with
a quote and a backslash and requires the recorded request to parse as
JSON with the resolved path. Both assertions fail against the previous
head of this branch.
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
scripts/benchmark-index.sh and scripts/benchmark-search-graph.sh ran the
product with no runtime or cache of their own: the benchmark repository
was indexed into the operator's live store and every one-shot joined the
operator's account daemon. Source scripts/test-runtime.sh in both, start
the private daemon before timing, record setup-time.txt and
total-time.txt beside index-time.txt, and index from a repository path
instead of querying a project in the live store.
Add tests/test_benchmark_runtime_isolation_contract.sh, which fails
before this change, and wire it into scripts/test.sh.
Part of #1696.
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
Martin Vogel
Anton Standrik