Section 7 still began with rm -f ~/.cache/codebase-memory-mcp/*.db.
Every index in the loop now lands in a run-private cache (#1696), so
that line no longer made any run colder; it only deleted the indexes in
the live store of the operator. Dropped, and the comment above it now
names only what the setup still does.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
With CBM_BENCH_KEEP_RUNTIME=1, bench_finish ran `daemon stop` with its
output and exit status discarded and wrote runtime-root.txt regardless.
`daemon stop` refuses while a committed client still uses the daemon and
fails when the daemon does not answer; the handoff then promised a
stopped daemon that was still running.
A refused stop now prints the explanation of the product, writes no
handoff, keeps the root without cleaning it up (the daemon may still be
using it) and exits non-zero. A stop with nothing to stop still exits 0,
so a clean run is unaffected.
Section 7 step 8 of the evaluation plan follows the same rule: the kept
root is removed only if its daemon stopped, and is kept with a message
otherwise.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
§7 step 8 removed the kept runtime root while the graph session of
steps 4-7 may still have a daemon running under it; stop that daemon
under the sourced CBM_RUNTIME_DIR first. §13 Reproducibility still
cleared ~/.cache/codebase-memory-mcp/*.db and called the harness
unflagged, which no longer describes what the harness does: it indexes
into a run-private root, and only CBM_BENCH_KEEP_RUNTIME=1 hands that
root, via runtime-root.txt, to the graph phase.
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
The evaluation plan (docs/EVALUATION_PLAN.md §7) indexes a language with
benchmark-index.sh and then answers graph questions against that index
from its own MCP session. With a run-private runtime (#1696) the index
was gone before that session could start, so the handoff is now explicit
and opt-in.
With CBM_BENCH_KEEP_RUNTIME set, a SUCCESSFUL run stops its daemon,
leaves the private root in place and records the paths that reach it in
<results>/<lang>/runtime-root.txt (sourceable: CBM_BENCH_RUNTIME_ROOT,
CBM_RUNTIME_DIR, CBM_CACHE_DIR). Ownership of that root, including its
removal, passes to the caller. A failed run cleans up regardless: there
is no index worth keeping and nothing may leak. Nothing changes in the
default path or in the three timing files.
§7's skeleton sets the flag, sources the file before the graph session
and removes the root in step 8 instead of the live-store *.db files.
The contract test drives a kept run and requires the root to survive,
the handoff file to be sourceable, and its paths to be the ones the
product processes actually used; the unflagged run still asserts the
root is gone. The kept-run assertions fail against the previous head.
Signed-off-by: Anton Standrik <astandrik@yandex-team.ru>
A reference written in a doc comment is extracted with the definition it
documents (internal/cbm/doclink.c; the C# scanner doclink_cs.c) and
resolved per file in the pipeline (src/pipeline/doc_links.c,
doc_links_cs.c, doc_links_msbuild.c). A reference that names a definition
of the repository with certainty becomes a MENTIONS edge from the
documented definition (the File node for a file-level comment) with the
properties via, syntax, tier, line and count. Any other reference becomes
a row of the new doc_link_unresolved table with its reason: missing,
ambiguous, external, test_only_target, graph_gap, unparseable or
below_bar_tier. Nothing is guessed. index_status reports the layer in a
doc_links block.
C# binds a cref by the compiler's lookup rule: the first scope with the
name decides, a cref never binds an inherited member, System is outside
the repository. Assemblies come from project files, and MSBuild global
usings (<Using>, ImplicitUsings, Directory.Build.props/.targets, imports,
conditions) are evaluated from the project files' content as data the
indexer has already read; the layer opens no file. Two declarations of a
name in different assemblies make the reference ambiguous.
Each file's scope is stored line-free with its LSP surface ("dl"), so an
incremental run re-resolves the files a change can reach and falls back
to a full run for a change that can re-route references anywhere (a
project file, a global using, a deleted file).
The scanner and the resolver treat the repository as untrusted input:
bounded nesting, no input-driven recursion, work that grows linearly
(held by test seams, never by a clock), no silent limits, and a damaged
or foreign stored scope fails closed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The per-language chapters were LLM-drafted to establish format and
dimension mapping — their symbol targets are not yet ground-truth-derived
and must be regenerated at execution time per §3.1/§12. Make that status
explicit at the top of Appendix A so the drafted symbol names are not
read as validated.
Supersedes #420 (same diff; recreated for DCO author/sign-off).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Address peer-review feedback on question provenance/external validity.
- Anchor the five dimensions to Sillito, Murphy & De Volder's catalogue of
questions programmers ask during software-evolution tasks (FSE'06/TSE'08),
with LaToza & Myers (2010) and Ko et al. (ICSE'07) as support; add a
D1-D5 -> Sillito-group mapping table
- Adopt published repo-level QA benchmarks for the major languages: reuse and
compare against SWE-QA (primary), with CoReQA, CodeRepoQA and RepoQA as
references; note snippet-level sets (CodeQA/CS1QA/CoSQA) as the contrast
that motivates structural retrieval
- Policy: anchor to Sillito; reuse SWE-QA for major languages; generate the
multilingual remainder from independent LSP/git ground truth, never from
the model under test
Rename TEST_PLAN.md to EVALUATION_PLAN.md and rebuild it as a peer-review
evaluation specification covering all 159 supported languages.
- Graph (MCP) vs Explorer (grep/glob/read) per language, 5 bespoke questions
mapped to 5 universal dimensions (D1-D5) for cross-language aggregation
- Sequential, main-channel execution: one language at a time, Explorer the
only subagent, per-language index lifecycle (cold index -> eval -> delete)
to avoid concurrent-indexing memory blowups; index time is a key metric
- LLM-as-a-Judge grading (single disclosed model, blind A/B, median of passes)
- Per-language node-type and edge-type histograms with every one of the 32
edge types listed, zeros kept (a zero is a real extraction-gap signal)
- Cross-repo intelligence + semantic/similarity deep-dive for the 9 LSP
languages, with recall/precision ground truth
- Single result tree (eval-results/), retiring the versioned-dir scheme
- Master repo assignment for all 159 languages; per-language chapters in
Appendix A
Martin Vogel
Anton Standrik
Martin Vogel