search_code matched file_pattern against the file basename only, so any
pattern containing a directory (src/cli/*.c, apps/web/src/**/*.ts)
silently returned 0 results, while search_graph's file_pattern matches
the full repo-relative path.
A file_pattern containing '/' is now matched in C against the
repo-relative path of every indexed file, on all platforms, with
search_graph-equivalent semantics: '*' and '**' match across '/', '**/'
also matches zero directories, '?' matches one character, and a pattern
without wildcards is a substring match. Patterns without '/' keep the
existing basename behaviour. On Windows such patterns are scanned in
process from the already-filtered list instead of the PowerShell -like
post-filter. A search with a '/' pattern that returns nothing now carries
a hint explaining whole-path matching and pointing at path_filter, and
the schema describes both parameters.
Refs #2836
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
tests/test_pipeline.c: main and this branch added tests at the same spot;
both are kept unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Split the PR along the line the review asked for. The request timing and
transport fields stay: every mcp.request log line carries
handler_duration_ms, response_serialization_ms, cancelled and timed_out,
and a search that reaches its deadline logs status=timeout.
The behaviour changes go back to the base text and need a separate
maintainer decision: stricter jsonrpc/id parsing, dropping "result":null,
NULL returns from the response formatters, leaving the stdio loop on a
missing response or an input/output failure, the removed blocking-poll
fallback, the HTTP 500 path, the daemon session end on a missing response,
and the update-notice warning. Their tests go with them. The list_projects
sort guard is dropped as well (main has the same fix).
cbm_mcp_server_handle formats its three responses with
cbm_jsonrpc_format_response directly; a NULL reply still marks the request
as an error. The cbm_log_mcp_request checks in tests/test_log.c are back
with the new signature, plus checks for the new fields and for
status=timeout. clang-format on the new lines.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reworks the ADR-loss-across-reindex fix to Alternative 1 (per the PR review):
remove the race rather than narrow it. ADRs live in a per-project sidecar SQLite
DB ("<graph_db>.adr.db"), separate from the graph DB a reindex rebuilds or
deletes, so there is no capture, re-read, re-apply, or lock — and no sleeps,
polling, or timeout deciding an outcome.
store.c:
- ADR writes (store/delete/update_sections) use a cached read-write handle opened
SQLITE_OPEN_READWRITE|CREATE and configured like the graph store (PRAGMA
busy_timeout=10000 + journal_mode=WAL) so cross-process access waits rather
than failing SQLITE_BUSY. A :memory: store keeps ADRs in the graph DB.
- Reads never write: cbm_store_adr_get uses the cached RW handle if open, else
opens the sidecar SQLITE_OPEN_READONLY (with the same busy timeout) when the
file exists, else reads the legacy graph row without creating anything. When
the sidecar exists but is NOT marked migrated (half-created), it falls back to
the legacy graph row so a partial sidecar never hides an ADR.
- One-time migration is atomic: adr_copy_legacy_once wraps the marker check,
legacy copy, and marker insert in a single BEGIN IMMEDIATE...COMMIT
(re-checking the marker under the lock; ROLLBACK->ERR on failure). New public
cbm_store_adr_migrate_once. Sidecar path via cbm_path_for_file_api (Windows
long-path). Dropped the write-only adr_db_path field.
pipeline.c: migrate a legacy ADR into the sidecar BEFORE the format-change and
forced-full routes unlink the old generation; preserve-and-abort on failure, and
do NOT fail open — a query-open returning NULL preserves unless the final DB
genuinely does not exist. Migration targets the FINAL path (staging suffix
stripped). The early-capture/re-apply machinery and its test hook are gone.
artifact.c: the exported artifact carries the ADR — snapshot_inject_adr always
DELETEs project_summaries first, then injects only when an ADR is present, so a
deleted ADR never resurrects through a lingering legacy row. Import restores the
ADR into the destination sidecar via cbm_store_adr_restore_from_graph_if_absent:
it copies the artifact's ADR (INSERT OR IGNORE — never clobbers a local one)
regardless of the migrated marker, so a teammate who already indexed the project
(marked sidecar, no ADR) still receives the shared decision record. store.c
factors the row copy into adr_copy_rows, shared by the one-time legacy migration
and the ungated restore.
mcp.c: delete_project removes "<db>.adr.db" (+ -wal/-shm/-journal) only after the
graph unlink succeeds; is_project_db_file skips *.adr.db. cli.c: exclude
*.adr.db from index listing/counting. README + memory-core baselines updated.
Tests (file-based): ADR written mid-reindex survives on BOTH the full and delta
routes (route-pinned FORCED_FULL / CLOSURE_REPAIR); a legacy row migrates before
a forced-full/format reindex with no prior read; migration failure (a directory
at <db>.adr.db) preserves-and-aborts with the old generation intact;
delete_project via the MCP handler removes the sidecar + journals; the artifact
export->import round trip keeps the ADR; a deleted ADR does not resurrect on
import; an import does not clobber a local ADR; an import into an already-indexed
destination (marked sidecar, no ADR) restores the artifact's ADR; a half-created
sidecar falls back to the legacy row. The import tests remove the destination
graph DB (keeping the sidecar + marker) before importing, matching production
(try_artifact_bootstrap imports only when the graph DB is missing) so the rename
succeeds on Windows, not only on POSIX.
Closes#516
Signed-off-by: 0xb007ab1e <134006168+0xb007ab1e@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
file_pattern, kind and namespace are applied after the 2000-row BM25
candidate window: unlike the label they cannot go into the FTS5 MATCH. With a
common query token and a narrow filter the window held none of the filtered
rows, and search_graph answered 0 rows -- or a few rows of an under-counted
total -- although matching nodes exist. An unlabelled query whose window held
only File/Folder/Variable/Project rows had the same failure.
When the window is saturated and such a filter is set, or the window held no
row passing the filters, the query is now ranked over every candidate
(inner LIMIT -1): the FTS-first subquery plan stays, the extra work is linear
in the token's match count (the saturation probe already orders every match),
and the total is exact. The decision depends only on the query, never on the
page, so offset pages stay consistent. Unfiltered and label-only queries that
find rows keep the window and are byte-identical.
The filter binds, the count query and the saturation probe move into helpers
shared by the ranked and the count query.
Measured on Django, meilisearch and kubernetes indexes (9 / 9 / 24 filtered
queries): zero results 6 -> 0, 6 -> 0, 23 -> 0; rows returned 52 -> 156,
3 -> 303, 19 -> 807; exact totals 0 -> 9, 0 -> 9, 0 -> 24. The graph is
unchanged (search-only change).
Refs #2386
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
In query (BM25) mode an explicit label naming File, Folder, Variable or
Project returned 0 rows -- often with an exact "eq" total -- although those
nodes are in nodes_fts and match the query: the noise exclusion meant for
unlabelled queries was applied to labelled queries too. It now applies only
when no label is given (one shared SQL fragment for the ranked and the count
query), so such a label is served from its own label-scoped candidate window
like any other label. Unlabelled queries are unchanged.
A label no node carries as spelled (labels are case-sensitive) still answers
0 rows, as structural mode does, but no longer silently: the result carries a
hint naming the stored spelling, or get_graph_schema when the label is
unknown. An existing label with no matching row stays a plain zero.
Measured on Django and meilisearch indexes (81 / 114 label queries): zero
results 18 -> 0 and 21 -> 0 (8 and 18 of them reported an exact "eq" total);
every other label returns byte-identical results, and unlabelled queries are
byte-identical to main.
Refs #2386
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Replace the PATH_ALLOW_BROAD environment variable with a recorded grant,
the shape agreed in review: `codebase-memory-mcp allow-root
--approve-broad <path>` records an explicit human approval for one root
refused as too broad (CBM_WS_DENY_TOO_SHALLOW), mirroring
--approve-sensitive end to end.
- cbm_workspace_grant_add() takes approve_broad next to
approve_sensitive. Each flag lifts only its own verdict, absolute
denials stay unliftable, and a refusal names the flag that would
lift it.
- The approval is stored with a '^' marker, listed by allow-root --list
as "(approved broad)", and honoured by cbm_workspace_root_allowed()
only for CBM_WS_DENY_TOO_SHALLOW and only on an exact path match,
keeping the exact-match rule of the original change.
- cbm_workspace_root_allowed() returns to its previous signature. Every
caller (MCP index_repository and index-job start, auto-index, daemon
sessions, the HTTP UI) reads the same grant store, so stdio and
daemon sessions decide identically and nothing depends on the
environment a daemon was started with.
- The too-broad refusal names `allow-root --approve-broad <path>`, a
command a person runs.
- cbm_workspace_classify_root() and cbm_workspace_verdict_is_overridable()
are unchanged.
The regression test keeps its name and its exact-match, sibling,
absolute and sensitive checks, now against the grant store, and adds
that a broad marker never lifts a sensitive verdict.
Co-authored-by: Mark Murawski <github@kobaz.net>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The store-switch regression for #1115 was added inside
tool_list_projects_tree_uses_one_stable_header_and_keeps_json_direct
and that test was renamed after it, so the name no longer said what
its seven list_projects shape assertions check.
Restore that test exactly as it is on main and move the switch loop
unchanged into tool_project_discovery_agrees_across_store_switches_issue1115
with its own twelve-project fixture. The new test also checks
list_projects before the switches, so it covers both the first-touch
index_status/query case and the before/after list agreement the issue
asked for.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The read-only analysis profile and the fast scout profile serve the graph
subagents (codebase-memory-auditor, codebase-memory-scout). Their server
instructions listed tools only. Like the default profile, they now state as
facts which structural questions the graph answers (where a symbol is defined,
who calls a function, what it calls, and for analysis also call chains, data
flow and change impact), keywords first: analysis 190 -> 648 characters, scout
199 -> 427, well within Claude Code's 4,096-character cut. The profile names,
check_index_coverage and the absence of index_repository are unchanged.
Stacked on #2698.
Refs #509
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Claude Code defers every MCP tool behind ToolSearch by default: at session
start the model sees only tool names and server instructions, while Grep and
Read are always loaded. Codex does the same and shows the server instructions
as the namespace description of the deferred tools. Agents therefore reached
for grep before the graph tools were even loaded (#509, discussion #2025).
- tools/list: search_graph, trace_path and get_code_snippet carry
_meta["anthropic/alwaysLoad"] = true (a documented per-tool hint), so the
structural discovery path loads upfront while the other 14 tools stay
deferred. About 4.6K characters of schema; clients that do not know the key
ignore it, as MCP reserves _meta.
- Server instructions (default profile) now state, keywords first and as
facts, which structural questions the graph answers and where grep fits
better: 309 -> 999 characters, within Claude Code's 4,096-character cut.
- SessionStart/SubagentStart: the untrusted-metadata label now covers the
graph-derived project name only. Wrapping the routing sentences too told the
model to treat cbm's own guidance as data. The other sentences are unchanged;
sanitization is unchanged.
The initialize byte budget is raised deliberately from 512 B to 1.5 KiB (1,188
bytes now); tools/list stays within its 21 KiB budget (20,938 bytes).
Refs #509
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
search_code
- Scan the indexed files in process instead of through grep (POSIX) or a
PowerShell Select-String pipeline (Windows), with the same grep -f
semantics: one pattern per line, ERE regex, binary-file and NUL handling,
-Hn records. The subprocess runner stays for the unindexed recursive
fallback and a Windows file_pattern it cannot prefilter.
- Windows now matches like macOS and Linux: Select-String matched
case-insensitively, used .NET regex syntax and read NUL-containing
files as text. A trailing CR is still ignored there, so `foo$` keeps
matching CRLF checkouts.
- Result merge, root containment and source reads no longer scale with
hits x results: a node-id index, one canonical root per request with a
per-file verdict cache, and fread spans instead of per-byte fgetc.
query_graph
- ORDER BY (RETURN and WITH) was a bubble sort. A stable merge sort keeps
the same comparisons and so the same order; WITH keys are resolved once
per row instead of twice per comparison.
- GROUP BY, DISTINCT (RETURN and WITH) and COUNT(DISTINCT) looked every
row up in a linear list. They hash the same keys now.
Store reads
- Read tools run their statements in one deferred read transaction,
renewed every 100 ms, instead of taking the database file lock once per
statement.
list_projects
- Resolve only the branch the listing prints: 2 git processes per project
instead of 7.
On macOS and Linux every response is byte-identical to before:
search_code 30/30 (django), 28/28 (this repo) and 28/28 (Linux, GNU
grep); all tools and the Cypher cases 86/86 on shared django and redis
indexes. On django: search_code 908 -> 355 ms, WITH ... ORDER BY
41 s -> 62 ms, count(*) 508 -> 184 ms, list_projects 94 -> 40 ms. On the
Windows ARM VM: search_code 1.6-3.4 s -> 0.2-0.5 s.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Doc-section -> code candidates (doc_link_candidates, #2574/#2610) are the
documentation-in-the-graph feature of this release. They now run by
default for full indexes and not for moderate or fast ones; an explicit
`doc_candidates` argument, or a `doc_candidates` setting the user set,
overrides the default either way in every mode.
- pipeline: the switch is unset/on/off; unset means on only when the
effective mode is full.
- MCP: forwards only an explicit choice (argument, then a user-set config
value) to daemon jobs and workers; daemon auto-index and watcher/UI jobs
now carry a user-set value too (their workers have no config, so
`doc_candidates false` was ignored there).
- fast + doc_candidates on runs the semantic pass's corpus phases and the
doc phase only: candidates, and still no SIMILAR_TO or
SEMANTICALLY_RELATED edges.
- config: `doc_candidates` joins the config key table (default `auto`);
`config set doc_candidates ...` was refused as an unknown key before.
README, docs/CONFIGURATION.md and the MCP schema describe the default.
Proof (macOS arm64, django fb11376):
full 11,304 rows · moderate 0 · fast 0 · full+off 0 · moderate+on 1 ·
fast+on 1 (moderate/fast discovery skips docs/) -- nodes and edges are
identical between on and off in every mode; fast+on adds 0 semantic
edges.
- New tests (RED on the old code): pipeline
pipeline_doc_candidates_default_follows_mode,
pipeline_doc_candidates_explicit_wins_over_mode,
pipeline_doc_candidates_fast_on_legacy_incremental; mcp
tool_index_repository_doc_candidates_follow_mode,
tool_index_repository_doc_candidates_forwarded_only_when_explicit;
daemon_application
daemon_application_programmatic_index_forwards_set_doc_candidates.
- mcp 374 passed (4 Windows-only skips), daemon_application 73 passed,
pipeline/store_arch/cli/runtime_settings/index_policy/pipeline_frozen
green on the release tree.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The doc<->code candidate scoring (#2574, doc_section_worker in
semantic_edges) scores every documentation section against the functions
that share a key term with it, key terms being those held by at most
funcs/20 functions. Its cost therefore grows with sections x functions.
Measured on 11 corpora (10-09 main vs 4d9592b5, semantic_edges): under a
second up to ~100k functions, +12.6 s on java (325k), +14.6 s on csharp
(451k), +174 s on the Linux kernel (784k functions, 28k sections), almost
all of the kernel's regression against v0.11.0.
A new boolean setting, doc_candidates (default true), and an optional
index_repository argument of the same name switch the scoring off: no
candidate rows are stored and nodes and edges are unchanged. The argument
wins over the setting. The index handler resolves the value once and
passes it to a daemon job or supervised worker as an argument, the same way
as the resource policy, so every route sees one decision; the session
auto-index reads the setting. With the switch off, a delta-route reindex
replaces the candidate rows it would otherwise keep, so turning it off
leaves no stale candidates. The default output is unchanged.
Tests (each fails on its assertion when the switch has no effect):
- pipeline_doc_candidates_off_stores_none_and_keeps_graph: no rows, same
node and edge counts as the default run, which has rows;
- pipeline_doc_candidates_off_clears_rows_on_delta: a closure-repair
reindex with the switch off removes the earlier rows;
- tool_index_repository_doc_candidates_switch: argument false -> none;
setting default -> rows; setting false -> none; argument true over
setting false -> rows.
Proof, release-built binaries on macOS arm64 (M5 Pro 18c/48 GB):
- django, main 4d9592b5 vs this change with defaults: nodes (63,441),
edges (307,584) and doc_link_candidates rows (11,304) hash-identical;
- django with doc_candidates false: nodes and edges hash-identical, no
candidate rows;
- Linux kernel with doc_candidates false: semantic_edges 17.9 s (191.9 s on
main, 17.6 s before #2574), whole index 210.9 s (v0.11.0: ~234 s), no
candidate rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
Shane McCarron
0xb007ab1eandClaude Opus 4.8
Pcristin
harshitaajoshi