mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-06 16:53:56 +00:00
a35671e962f1f2fb4e94deaf55e71cf05f6b3828
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
28b41c35b7 |
feat(mcp): opt-in async index_repository with status polling (#2144)
index_repository blocks until the whole index completes. MCP clients with a per-call deadline (Copilot for IntelliJ) give up on a large repository; their cancel drops the daemon job's last subscriber, the daemon cancels the worker, and every retry starts over, so the index never finishes. Add an opt-in async mode and a status query to the same tool; the default synchronous behaviour is unchanged: - async: true starts the project's index job in the daemon, or joins the one already running for it, and returns at once with its state. The application holds one subscriber reference for an async job until its terminal publish, so the request returning, the client cancelling or the starting session closing no longer cancels it. Only daemon shutdown (or the final session of a non-permanent generation) ends it. Async requests never queue behind the physical job limit; a full daemon answers busy. async/status are stripped from the worker args, so an async request coalesces with an identical synchronous one. - status: true reports the running or last job for the project (queued/running/cancelling/succeeded/failed/cancelled, started_at, finished_at, error summary) from a small per-project record kept in the daemon's job registry after the job itself is reaped. An unknown project is an error; a project indexed before this daemon generation reports idle. No freshness key is added: index_status keeps the #1561 freshness object. - async + status together, non-boolean values, and async with cross-repo-intelligence are refused. In-process servers (index worker, embedders) refuse both modes: nothing there outlives the call. - On a temporary (non-permanent) daemon, async is refused when the request arrives on the one-shot `cli` tool channel and that session is the daemon's only live session: the daemon would stop and cancel the job the moment the command exits. The error points at `daemon start` (or an open MCP session). A long-lived MCP session stays a valid host even when it is the only one, which is the IDE case this exists for; status stays allowed everywhere. The daemon lifecycle is unchanged. - New allocations go through the memory core (cbm_alloc/cbm_calloc/ cbm_mem_strdup/cbm_free); blocks returned by non-core APIs are released with safe_free, so no file's raw allocator count grows. - A cut-short synchronous call now offers the async alternative: in the cancellation text the daemon or frontend delivers (including the -32800 reply to a cancelled index_repository), and, because a timed-out client never reads that reply, as a notice on the next index_repository or status call for the project. - The tool description and schema, the CLI help and the README explain the flags, the deadline problem and the polling pattern. No server-side timeout or budget is introduced. Tests (deterministic, held fake worker, bounded observation waits): async returns before completion and status observes running then succeeded; an async job survives its session closing while a sync job in the same shape is still cancelled; validation errors; the cancellation reply and the next-call notice carry the advice; a lone one-shot client of a temporary daemon is refused async while status and an MCP session on the same daemon are allowed; in-process refusal; schema; notice helper. Refs #2031 Fixes #2144 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
|
7710f86e92 |
fix(daemon): avoid duplicate idle maintenance polling
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
|
c059396eb2 |
fix(build): give sanitized-build detection one spelling and a backstop
Four places ask "is this binary instrumented?" and each asked it differently.
One of them, the C# LSP bench, only recognised ASan, so TSan and MSan measured
an instrumented parse against the NATIVE 200ms budget. The other three carried
a hand-copied list of `__SANITIZE_*__` macros that nobody kept in sync.
That drift is what the TSan gap was made of. CFLAGS_TSAN never passed
SANITIZED_DEFINE, and the per-site conditions could not cover for it: they test
`__SANITIZE_THREAD__`, which is GCC's spelling. Clang — the compiler that leg
uses — announces thread instrumentation through `__has_feature(thread_sanitizer)`
only, and no site consulted it. The claim in
|
||
|
|
8d487b7da0 |
test(daemon): scale the frontend announce backstop with the build
`test-msan` failed the two backpressure cases while every other lane was green, and the same suite passes under MSan locally (12/12). The wait that fired is a LIVENESS BACKSTOP, not a race budget: the daemon child either announces on its ready pipe or it does not, and the test asserts the announcement itself, never a timing window. A backstop only does its job if it never fires on a legitimately slow start. 30s was already the second value here — the first was below the daemon's own 12s cohort-claim budget and SIGKILLed healthy startups on oversubscribed runners. Under MemorySanitizer, with instrumented libc++ and origin tracking, startup runs several times slower than that same budget it has to clear, so 30s stopped being a backstop and started being a verdict. Sanitized builds now get 180s, matching the existing sanitizer-aware pattern in test_cs_lsp_bench.c. This costs nothing when the daemon is healthy: a passing run returns the moment the byte arrives, whatever the ceiling. Nothing about the asserted state changed, and the non-sanitized bound is untouched. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
|
f59d24fd10 |
fix(mcp): make every tool reply client-usable — structuredContent, pipelining, config get (#1522)
Three regressions shipped in 0.10.0 share one failure shape: an empty result
with a success status, indistinguishable from "nothing found" for the LLM
clients that are cbm's primary consumers.
1) structuredContent {} on the whole tree-format surface (#1522 bug 1).
#1488 replaced the duplicated payload with an EMPTY structuredContent
object while every tool still declared a blanket permissive outputSchema.
Spec-honoring clients (Claude Code among them) treat structuredContent as
THE result when a schema is declared, so search_graph, trace_path,
query_graph, get_architecture, search_code, and detect_changes all rendered
as literally "{}" on their DEFAULT format, on every platform. The corrected
contract: no tool declares an outputSchema (tool output is
format-parameter-polymorphic — no static schema is truthful), JSON-object
payloads keep their parsed structuredContent, errors keep
structuredContent.error, and text-shaped payloads carry NO structuredContent
key at all — which also preserves #1375's no-duplication win.
2) Frontend queue overflow killed the session (found by the #1522 sweep).
Any 7+ requests pipelined in one stdin burst — an agent issuing parallel
tool calls does exactly this — overflowed the 8-frame frontend queue, which
failed the whole session: rc=1 with ZERO bytes of output, every buffered
response lost. A full queue is now backpressure: the stdin reader blocks
until the worker drains (bounded by the same stop/fail flags every teardown
path already sets); only a single frame larger than the entire 12 MiB byte
budget — which could never be admitted — remains a hard failure.
3) config get printed "" with exit 0 for every unset and every unknown key
(#1522 bug 2). list printed stored-or-DEFAULT while get printed
stored-or-EMPTY, and no subcommand validated key names, so a typo was
indistinguishable from a correctly-read setting. One config-key table now
drives help, list, get, set, and reset: get prints the stored value or the
key's real default (the same fallback the runtime readers use), and unknown
keys error with exit 1 on get, set, and reset alike.
Tests — each RED on the pre-fix tree and RED again on revert:
* test_mcp.c: text results carry no structuredContent key; tools/list
declares no outputSchema; the tool-table guard now binds all three
branches (absent / parsed-object / error) for every registered tool.
* test_daemon_frontend.c: the over-capacity contract flips from
"session fails" to "backpressure without loss" — the held first request
plus all 32 over-capacity frames are answered and the run closes cleanly.
* test_cli.c: the config command contract — defaults, round-trip, reset,
and unknown-key rejection on all three subcommands.
* smoke-test.sh: Phase 3z rewritten to the corrected structuredContent
contract, new Phase 3z1 (default-format replies usable in schema-honoring
clients, no outputSchema advertised), 3z2 (24 pipelined calls all
answered), 3z3 (config defaults + unknown-key rejection) — all asserted
against the SHIPPED artifact, where #1488's smoke phase previously pinned
the empty-object behavior as correct.
Verified end-to-end on the locally built production binary: all 15 tools
declare no schema; text tools return ABSENT structuredContent, object tools
populated, error envelopes intact; 7/24/64-deep pipelined bursts all answered
with rc=0; config get prints real defaults and exits 1 on unknown keys.
hook_augment (structuredContent.projects) and index_resilience
(structuredContent.status) consume object payloads and are unaffected.
Fixes #1522.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
|
0baddf7887 |
daemon: permanent lifecycle, daemon-backed CLI/hooks, real-Windows hardening, long-path launcher transactions
Daemon lifecycle and Windows correctness, verified on a real Windows 11 ARM64 VM through the maintained test-infrastructure/vm drivers, plus the macOS and Linux arm64 suites and the container lint gate. Daemon lifecycle: - daemon start/stop/status subcommands. `daemon start` launches a PERMANENT daemon (spawn shape is byte-exact argv; survives idle periods and session ends) and reports an already-active daemon instead of failing. Permanence is honored at every stop latch: last-committed-client disconnect, host initial-client window, coordinator release, and application final-session close — a permanent daemon also keeps admitting new sessions after its last one closes. - daemon stop refuses while sessions are active and lists the blocking peers (pid/role) that must finish first; an idle daemon drains through the activation-shutdown machinery with the ACK ordered after connection interrupts. A second stop is idempotent. The wire ops are no-cohort first-frame requests with peer fingerprint authentication, so stop/status never conflict with an exact-build admission gate. - One-shot CLI commands now execute through the daemon (index workers keep their local supervised path). A cold CLI run that had to spawn a temporary daemon prints a hint that `daemon start` removes the per-command startup tax; a warm daemon is recycled silently. - Hooks are connect-only fail-open: with no daemon present the hook emits a visible, rate-limited notice (Claude-dialect systemMessage plus stderr for other dialects) and always exits 0 — augmentation is never allowed to block the caller's tool use. - Version skew: a newer-build client automatically drains an older-build permanent daemon (strict semantic-version triples only; dev builds never auto-drain) and the build-conflict message names `cbm daemon stop` as the manual escape hatch. Windows IPC/runtime (real-VM verified): - ipc(win): persistent pending overlapped ConnectNamedPipe. The accept path used to destroy its listening pipe instance on every 20 ms poll timeout; a client attaching in the teardown window was severed or left on an orphaned pipe object whose HELLO no server handle could ever read, absorbing the connect until the client's own timeout expired. The pending connect now survives poll timeouts and nothing is destroyed while a client could be attaching. - ipc(win): drain-before-close for final responses. Closing a named-pipe server handle can discard a just-sent response before the peer reads it (POSIX stream sockets never lose buffered data on close). A bounded cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX, immediate on interrupted connections) now precedes close in runtime_worker_finish and runtime_reject_inline, so hello-conflict, capacity and disconnect acknowledgements reliably reach the peer. - runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no transport half-close, so close_begin now announces departure with an explicit frame (ordered after APPLICATION_CANCEL, before the local interrupt); the server releases the client's admission on receipt instead of waiting for the handle to close. Admission-drop timing is now identical to POSIX shutdown() semantics on every platform. - runtime(win): client close cancellation. close_begin serializes with request publication under the send lock, best-effort sends the active token's APPLICATION_CANCEL frame, then interrupts local I/O; the server cancels MCP/subprocess work promptly. Contract tests accept both correct outcomes (interrupted transport or decoded CANCELLED). - runtime: activation acknowledgement ordering. The activation ACK is the requester's license to act on "snapshotted and draining", so every connection interrupt is now initiated before the ACK is sent; a session could previously get one more request serviced after the requester observed the ACK. - service(win): deadline-bounded private-file prepare. The conflict-log prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms timer granularity) burned ~1.6 s against permanently obstructed paths, stalling hello rejections past the client's timeout. The retry budget is now a 250 ms deadline; transient share collisions still retry. - subprocess(win): cmd.exe /C payload encoder quotes metacharacters correctly (root cause of the git-on-Windows failure cluster). - watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native Windows stop/unwatch cancellation test with exact-image verification. - httpd: send_all writes in bounded 64 KiB slices. A single giant nonblocking send() on Windows is absorbed wholesale into AFD kernel buffering regardless of SO_SNDBUF, so send deadlines and interrupts could never engage against a slow peer (and the full payload was pinned in nonpaged pool). Slicing restores a deterministic backpressure point; a test hook pins SO_SNDBUF for the deadline and interrupt tests. - ui/http: shutdown lifecycle — interrupt checks, response-wide send deadline, explicit connection states, refusal to free a server while a listener-owned connection is active. Windows long-path support: - Central path-aware wide conversion (canonicalize via GetFullPathNameW and prepend the extended-length prefix for absolute paths >=240) at the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite store opens, and the daemon build-fingerprint/log paths. Deep managed installs (a 64-hex generation directory routinely exceeds MAX_PATH) now index, stage and activate correctly. - activation transaction: its own file APIs and the component-walking ancestry validators now operate in the extended-length namespace; the launcher path is canonicalized (and prefixed when deep) once at entry so every downstream exact-string comparison stays form-consistent. - Executable self-resolution uses the wide APIs (GetModuleFileNameW, GetFileAttributesW) so non-ASCII install paths survive argv[0] resolution. Windows launcher install/uninstall transaction: - FileRenameInfoEx names are NUL-terminated in an over-allocated buffer. FileNameLength governs per the contract, but filter drivers read FileName as NUL-terminated and appended adjacent heap bytes to created names — a flaky, garbage-suffixed rename target. Both the CLI and the launcher rename helpers are fixed. - Uninstall retires state via rename-aside (.cbm -> .cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex chars so the bare rename target stays under the FileRenameInfoEx NT-conversion ceiling at guard depths; 64 bits still uniquely identify the generation. - When the running launcher's mapped generation backings pin .cbm against rename, the backings are relocated to activation-<pid>-N .retired tombstones beside the install (a mapped image may be renamed, never deleted; the launcher's liveness-guarded sweep reclaims stale tombstones). Every relocation is recorded, and a FAILED uninstall reverses the moves after restoring .cbm — via MoveFileExW with extended-length paths on both arguments, since the deep generation target is beyond the handle-based rename's bare-path reach — so a restored install keeps its generation backings and stays runnable. - After a committed uninstall the retired tree's backings are relocated out so the tree is shallow enough for the detached cleanup's rd, and the cleanup's working directory strips the extended-length prefix (CreateProcessW lpCurrentDirectory silently ignores prefixed paths). - Files created under Administrators-default-owner directories (CopyFileW destinations, CREATE_NEW tombstones, probe directories) are explicitly owner-stamped so the exact-owner validators hold on runner images; guard fixtures stamp hand-built trees the same way. Diagnostics, tests and infra: - diagnostics: discovery is now an always-delivered JSON control record (new cbm_log_control) that survives CBM_LOG_LEVEL suppression and paths containing spaces; placement honors $TMPDIR with /tmp fallback via a diagnostics-local helper; the soak parser reads the JSON record; documented in docs/CONFIGURATION.md. Red-first coverage for suppressed log levels, TMPDIR-with-spaces, and native Windows output-contract assertions. - tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint parents with production-shaped ancestry (LocalAppData on Windows, via th_secure_runtime_parent_new) — the runtime ancestry validation correctly refuses temp roots whose ancestors grant mutation rights to Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and drive the documented startup-owner publication flow before reading generation-bound endpoint addresses. This turns the 26 Windows failures previously visible in CI's full-test job green without weakening any validation. - tests(win): the launcher guard covers the full permanent-launcher contract including failed-uninstall restore and immediate reinstall after uninstall; new daemon lifecycle and reworked hook-augment guards run the start/recycle/stop flow end to end. - tests: CBM_SKIP_PERF is now actually consumed by the test runner (it was set by CI but never read, so perf suites ran everywhere); four throughput/bench suites are classified as perf, the heavy store_arch suite moved to the slow-timeout tier, and two wall-clock-sensitive assertions were rewritten as invariant checks with coarse hang-detector backstops. - build/test infra: build-dir safety contract, UI dev-proxy security contract, soak daemon-recovery contract, path-safety helper, the Windows VM worktree-sync contract wired into scripts/test.sh, and vm/win.sh guards building its clean embedded-UI product in an isolated BUILD_DIR so it cannot clobber the incremental test build. provision-windows.sh now installs Node.js for the guards UI build. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
|
85c2d99480 |
test(daemon): raise the frontend announce deadline above the claim budget
The backpressure/EOF fixtures waited only 10s for the forked daemon to announce readiness, but the daemon's own startup includes the cohort-claim path whose internal budget is HOST_DAEMON_CLAIM_TIMEOUT_MS (12s). Under the 4-core oversubscription of CI runners a slow-but-legitimate startup blew past 10s and was SIGKILLed before it could announce, so daemon_frontend backpressure failed almost every ubuntu run while passing on fast machines. Raise the announce deadline to 30s, comfortably above the daemon's own worst-case budget. Verified: the backpressure failures no longer reproduce under a local CI-fidelity 4-core run. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
|
3325b31b37 |
test(daemon): skip the build-fingerprint hash in tests (root-cause fix)
Every spawned worker/daemon re-hashes the entire executable image for the exact-build fingerprint; for the ASan test-runner (hundreds of MB) that is tens of seconds per spawn on constrained CI runners and was the sole cause of the whole daemon-family readiness-timeout flake class (index_supervisor, daemon_frontend, cli, and much of the Windows daemon surface). Add a test-only seam under CBM_CLI_ENABLE_TEST_API: when CBM_TEST_BUILD_FINGERPRINT holds a valid stub the hash is skipped. test_main.c sets it once; every forked child and re-exec'd worker inherits it, so exact-build match AND mismatch behaviour is preserved (a mismatch test still passes a different fingerprint via argv). Never compiled into production. Local mac run: index_supervisor 6/6, the four daemon suites 111/0, exact-build test green. The generous frontend catastrophic-alarm values are kept as backstops. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
|
8d465bc9ce |
test(daemon): widen worker/frontend hang-guard deadlines for slow ARM CI
The macOS legs went green with the prior calibration but the 3-core ubuntu-24.04-arm runner is slower still: each spawned worker re-hashes the entire ASan test-runner image (hundreds of MB) for the exact-build fingerprint before signalling readiness, which there takes tens of seconds. Raise the hang-guard deadlines generously - index_supervisor 60s ready / 90s terminal, frontend 30-90s, cli 45s. These are hang guards, not benchmarks: large values only slow genuine-hang detection, never mask a regression. (Root-cause fix - a test-only fingerprint injection seam that skips the hash - is tracked for follow-up.) Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
|
1fbfb1f69c |
test(daemon): calibrate frontend and cli-quiesce budgets for starved runners
Same class as the index-supervisor calibration, proven by the serial tail: the daemon_frontend backpressure/EOF tests and the cli forced- install quiesce child failed on the 4-core ubuntu leg even running alone on a quiet machine. Their 2 s request/cleanup/acquire budgets are below real startup cost there (identity capture, frontend spawn, lock acquisition on cold IO). Raised to 10-30 s; still hang guards - a wedged run fails loudly. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
|
ad2874bcff |
fix: close remaining daemon coordination races
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
|
83c137d2a5 |
feat: complete shared daemon lifecycle
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
|
0e00ef5702 |
feat: coordinate concurrent CBM sessions
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |