Commit Graph
13 Commits
Author SHA1 Message Date
Martin Vogel 28b41c35b7 feat(mcp): opt-in async index_repository with status polling (#2144)
index_repository blocks until the whole index completes. MCP clients with
a per-call deadline (Copilot for IntelliJ) give up on a large repository;
their cancel drops the daemon job's last subscriber, the daemon cancels
the worker, and every retry starts over, so the index never finishes.

Add an opt-in async mode and a status query to the same tool; the default
synchronous behaviour is unchanged:

- async: true starts the project's index job in the daemon, or joins the
  one already running for it, and returns at once with its state. The
  application holds one subscriber reference for an async job until its
  terminal publish, so the request returning, the client cancelling or
  the starting session closing no longer cancels it. Only daemon shutdown
  (or the final session of a non-permanent generation) ends it. Async
  requests never queue behind the physical job limit; a full daemon
  answers busy. async/status are stripped from the worker args, so an
  async request coalesces with an identical synchronous one.
- status: true reports the running or last job for the project
  (queued/running/cancelling/succeeded/failed/cancelled, started_at,
  finished_at, error summary) from a small per-project record kept in the
  daemon's job registry after the job itself is reaped. An unknown
  project is an error; a project indexed before this daemon generation
  reports idle. No freshness key is added: index_status keeps the #1561
  freshness object.
- async + status together, non-boolean values, and async with
  cross-repo-intelligence are refused. In-process servers (index worker,
  embedders) refuse both modes: nothing there outlives the call.
- On a temporary (non-permanent) daemon, async is refused when the
  request arrives on the one-shot `cli` tool channel and that session is
  the daemon's only live session: the daemon would stop and cancel the
  job the moment the command exits. The error points at `daemon start`
  (or an open MCP session). A long-lived MCP session stays a valid host
  even when it is the only one, which is the IDE case this exists for;
  status stays allowed everywhere. The daemon lifecycle is unchanged.
- New allocations go through the memory core (cbm_alloc/cbm_calloc/
  cbm_mem_strdup/cbm_free); blocks returned by non-core APIs are released
  with safe_free, so no file's raw allocator count grows.
- A cut-short synchronous call now offers the async alternative: in the
  cancellation text the daemon or frontend delivers (including the
  -32800 reply to a cancelled index_repository), and, because a timed-out
  client never reads that reply, as a notice on the next index_repository
  or status call for the project.
- The tool description and schema, the CLI help and the README explain
  the flags, the deadline problem and the polling pattern.

No server-side timeout or budget is introduced.

Tests (deterministic, held fake worker, bounded observation waits):
async returns before completion and status observes running then
succeeded; an async job survives its session closing while a sync job in
the same shape is still cancelled; validation errors; the cancellation
reply and the next-call notice carry the advice; a lone one-shot client
of a temporary daemon is refused async while status and an MCP session
on the same daemon are allowed; in-process refusal; schema; notice
helper.

Refs #2031

Fixes #2144

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-25 23:03:56 +02:00
Martin Vogel 7710f86e92 fix(daemon): avoid duplicate idle maintenance polling
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-27 20:32:19 +02:00
Mauricio Offermann c059396eb2 fix(build): give sanitized-build detection one spelling and a backstop
Four places ask "is this binary instrumented?" and each asked it differently.
One of them, the C# LSP bench, only recognised ASan, so TSan and MSan measured
an instrumented parse against the NATIVE 200ms budget. The other three carried
a hand-copied list of `__SANITIZE_*__` macros that nobody kept in sync.

That drift is what the TSan gap was made of. CFLAGS_TSAN never passed
SANITIZED_DEFINE, and the per-site conditions could not cover for it: they test
`__SANITIZE_THREAD__`, which is GCC's spelling. Clang — the compiler that leg
uses — announces thread instrumentation through `__has_feature(thread_sanitizer)`
only, and no site consulted it. The claim in 0a163d4f that compiler probes could
not have helped is true of the probes we had, not of the one clang actually
offers.

src/foundation/sanitized.h now answers the question once, as CBM_SANITIZED,
from two sources with distinct jobs:

  - CBM_SANITIZED_BUILD from the build system stays the source of truth, and is
    the ONLY thing that can answer for UBSan and trap-UBSan: undefined-behaviour
    instrumentation leaves no macro and no __has_feature bit to probe.
  - The clang and GCC probes are the backstop for the three sanitizers that do
    announce themselves, so a lane that forgets the define still gets correct
    budgets instead of native ones on an instrumented binary.

Deliberately no #error when a probe fires without the define: promoting leaves
the binary correct while the lane gets fixed, and it does not break an
out-of-tree `make CFLAGS_EXTRA=-fsanitize=address` that never went near
Makefile.cbm.

__has_feature is defined away where it does not exist rather than guarded with
`#elif defined(__has_feature)`. The guarded form compiles everywhere but fails
cppcheck, which walks every configuration and rejects the file with "failed to
evaluate #if condition, undefined function-like macro invocation". The
define-away idiom is what clang documents and what tests/test_mem.c already
uses; `defined(...) && __has_feature(...)` is not an option at all, since && does
not spare a preprocessor without the builtin from parsing `0 (0)`.

Verified the resolution rather than assuming it (clang 22, -dM -E):

  native                                  CBM_SANITIZED 0
  -DCBM_SANITIZED_BUILD=1                 CBM_SANITIZED 1
  -fsanitize=address                      CBM_SANITIZED 1
  -fsanitize=thread   (linux target)      CBM_SANITIZED 1
  -fsanitize=memory   (linux target)      CBM_SANITIZED 1
  -fsanitize=undefined                    CBM_SANITIZED 0   <- define-only, as designed

The third row is the one that matters: the TSan failure this header is named
after would have self-healed.

Also wired the define into the instrumented flag sets of our own code that
still lacked it — CXXFLAGS_TSAN (preprocessor.cpp is ours, and CXXFLAGS_TEST
already had it), GRAMMAR_CFLAGS_TEST and GRAMMAR_CFLAGS_TSAN. Neither tree can
include the header today (no -Isrc), so this is the build system keeping its
own promise rather than a behaviour change. Vendored flag sets are untouched:
mimalloc, sqlite3, tre, zstd, lz4 and tree-sitter read no macro of ours.

Behaviour change worth naming: test_cs_lsp_bench now allows 2000ms on the TSan
and MSan lanes instead of 200ms. It loosens a bound that was being applied to
an instrumented binary by accident; it never tightens one.

Not verified locally: this machine has no POSIX-target compiler, so the POSIX
half of subprocess.c was not compiled here. The full Windows test-runner builds
clean with -Werror and the subprocess suite is green (14 passed, 17 skipped);
clang-format clean; macro matrix as above.

Signed-off-by: Mauricio Offermann <mauricio.offermann@gocode.cl>
2026-08-14 06:46:05 -04:00
Martin Vogel 8d487b7da0 test(daemon): scale the frontend announce backstop with the build
`test-msan` failed the two backpressure cases while every other lane was green,
and the same suite passes under MSan locally (12/12). The wait that fired is a
LIVENESS BACKSTOP, not a race budget: the daemon child either announces on its
ready pipe or it does not, and the test asserts the announcement itself, never a
timing window.

A backstop only does its job if it never fires on a legitimately slow start.
30s was already the second value here — the first was below the daemon's own
12s cohort-claim budget and SIGKILLed healthy startups on oversubscribed
runners. Under MemorySanitizer, with instrumented libc++ and origin tracking,
startup runs several times slower than that same budget it has to clear, so 30s
stopped being a backstop and started being a verdict.

Sanitized builds now get 180s, matching the existing sanitizer-aware pattern in
test_cs_lsp_bench.c. This costs nothing when the daemon is healthy: a passing
run returns the moment the byte arrives, whatever the ceiling. Nothing about the
asserted state changed, and the non-sanitized bound is untouched.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-12 11:52:38 +02:00
Martin Vogel f59d24fd10 fix(mcp): make every tool reply client-usable — structuredContent, pipelining, config get (#1522)
Three regressions shipped in 0.10.0 share one failure shape: an empty result
with a success status, indistinguishable from "nothing found" for the LLM
clients that are cbm's primary consumers.

1) structuredContent {} on the whole tree-format surface (#1522 bug 1).
   #1488 replaced the duplicated payload with an EMPTY structuredContent
   object while every tool still declared a blanket permissive outputSchema.
   Spec-honoring clients (Claude Code among them) treat structuredContent as
   THE result when a schema is declared, so search_graph, trace_path,
   query_graph, get_architecture, search_code, and detect_changes all rendered
   as literally "{}" on their DEFAULT format, on every platform. The corrected
   contract: no tool declares an outputSchema (tool output is
   format-parameter-polymorphic — no static schema is truthful), JSON-object
   payloads keep their parsed structuredContent, errors keep
   structuredContent.error, and text-shaped payloads carry NO structuredContent
   key at all — which also preserves #1375's no-duplication win.

2) Frontend queue overflow killed the session (found by the #1522 sweep).
   Any 7+ requests pipelined in one stdin burst — an agent issuing parallel
   tool calls does exactly this — overflowed the 8-frame frontend queue, which
   failed the whole session: rc=1 with ZERO bytes of output, every buffered
   response lost. A full queue is now backpressure: the stdin reader blocks
   until the worker drains (bounded by the same stop/fail flags every teardown
   path already sets); only a single frame larger than the entire 12 MiB byte
   budget — which could never be admitted — remains a hard failure.

3) config get printed "" with exit 0 for every unset and every unknown key
   (#1522 bug 2). list printed stored-or-DEFAULT while get printed
   stored-or-EMPTY, and no subcommand validated key names, so a typo was
   indistinguishable from a correctly-read setting. One config-key table now
   drives help, list, get, set, and reset: get prints the stored value or the
   key's real default (the same fallback the runtime readers use), and unknown
   keys error with exit 1 on get, set, and reset alike.

Tests — each RED on the pre-fix tree and RED again on revert:
  * test_mcp.c: text results carry no structuredContent key; tools/list
    declares no outputSchema; the tool-table guard now binds all three
    branches (absent / parsed-object / error) for every registered tool.
  * test_daemon_frontend.c: the over-capacity contract flips from
    "session fails" to "backpressure without loss" — the held first request
    plus all 32 over-capacity frames are answered and the run closes cleanly.
  * test_cli.c: the config command contract — defaults, round-trip, reset,
    and unknown-key rejection on all three subcommands.
  * smoke-test.sh: Phase 3z rewritten to the corrected structuredContent
    contract, new Phase 3z1 (default-format replies usable in schema-honoring
    clients, no outputSchema advertised), 3z2 (24 pipelined calls all
    answered), 3z3 (config defaults + unknown-key rejection) — all asserted
    against the SHIPPED artifact, where #1488's smoke phase previously pinned
    the empty-object behavior as correct.

Verified end-to-end on the locally built production binary: all 15 tools
declare no schema; text tools return ABSENT structuredContent, object tools
populated, error envelopes intact; 7/24/64-deep pipelined bursts all answered
with rc=0; config get prints real defaults and exits 1 on unknown keys.
hook_augment (structuredContent.projects) and index_resilience
(structuredContent.status) consume object payloads and are unaffected.

Fixes #1522.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-11 01:12:59 +02:00
Martin Vogel 0baddf7887 daemon: permanent lifecycle, daemon-backed CLI/hooks, real-Windows hardening, long-path launcher transactions
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.

Daemon lifecycle:

- daemon start/stop/status subcommands. `daemon start` launches a
  PERMANENT daemon (spawn shape is byte-exact argv; survives idle
  periods and session ends) and reports an already-active daemon
  instead of failing. Permanence is honored at every stop latch:
  last-committed-client disconnect, host initial-client window,
  coordinator release, and application final-session close — a
  permanent daemon also keeps admitting new sessions after its last
  one closes.
- daemon stop refuses while sessions are active and lists the blocking
  peers (pid/role) that must finish first; an idle daemon drains
  through the activation-shutdown machinery with the ACK ordered after
  connection interrupts. A second stop is idempotent. The wire ops are
  no-cohort first-frame requests with peer fingerprint authentication,
  so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
  keep their local supervised path). A cold CLI run that had to spawn a
  temporary daemon prints a hint that `daemon start` removes the
  per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
  emits a visible, rate-limited notice (Claude-dialect systemMessage
  plus stderr for other dialects) and always exits 0 — augmentation is
  never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
  older-build permanent daemon (strict semantic-version triples only;
  dev builds never auto-drain) and the build-conflict message names
  `cbm daemon stop` as the manual escape hatch.

Windows IPC/runtime (real-VM verified):

- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
  path used to destroy its listening pipe instance on every 20 ms poll
  timeout; a client attaching in the teardown window was severed or left
  on an orphaned pipe object whose HELLO no server handle could ever
  read, absorbing the connect until the client's own timeout expired.
  The pending connect now survives poll timeouts and nothing is
  destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
  server handle can discard a just-sent response before the peer reads
  it (POSIX stream sockets never lose buffered data on close). A bounded
  cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
  immediate on interrupted connections) now precedes close in
  runtime_worker_finish and runtime_reject_inline, so hello-conflict,
  capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
  transport half-close, so close_begin now announces departure with an
  explicit frame (ordered after APPLICATION_CANCEL, before the local
  interrupt); the server releases the client's admission on receipt
  instead of waiting for the handle to close. Admission-drop timing is
  now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
  request publication under the send lock, best-effort sends the active
  token's APPLICATION_CANCEL frame, then interrupts local I/O; the
  server cancels MCP/subprocess work promptly. Contract tests accept
  both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
  the requester's license to act on "snapshotted and draining", so every
  connection interrupt is now initiated before the ACK is sent; a
  session could previously get one more request serviced after the
  requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
  prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
  timer granularity) burned ~1.6 s against permanently obstructed paths,
  stalling hello rejections past the client's timeout. The retry budget
  is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
  correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
  Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
  nonblocking send() on Windows is absorbed wholesale into AFD kernel
  buffering regardless of SO_SNDBUF, so send deadlines and interrupts
  could never engage against a slow peer (and the full payload was
  pinned in nonpaged pool). Slicing restores a deterministic
  backpressure point; a test hook pins SO_SNDBUF for the deadline and
  interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
  deadline, explicit connection states, refusal to free a server while
  a listener-owned connection is active.

Windows long-path support:

- Central path-aware wide conversion (canonicalize via GetFullPathNameW
  and prepend the extended-length prefix for absolute paths >=240) at
  the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
  store opens, and the daemon build-fingerprint/log paths. Deep managed
  installs (a 64-hex generation directory routinely exceeds MAX_PATH)
  now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
  ancestry validators now operate in the extended-length namespace;
  the launcher path is canonicalized (and prefixed when deep) once at
  entry so every downstream exact-string comparison stays
  form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
  GetFileAttributesW) so non-ASCII install paths survive argv[0]
  resolution.

Windows launcher install/uninstall transaction:

- FileRenameInfoEx names are NUL-terminated in an over-allocated
  buffer. FileNameLength governs per the contract, but filter drivers
  read FileName as NUL-terminated and appended adjacent heap bytes to
  created names — a flaky, garbage-suffixed rename target. Both the CLI
  and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
  .cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
  chars so the bare rename target stays under the FileRenameInfoEx
  NT-conversion ceiling at guard depths; 64 bits still uniquely
  identify the generation.
- When the running launcher's mapped generation backings pin .cbm
  against rename, the backings are relocated to activation-<pid>-N
  .retired tombstones beside the install (a mapped image may be renamed,
  never deleted; the launcher's liveness-guarded sweep reclaims stale
  tombstones). Every relocation is recorded, and a FAILED uninstall
  reverses the moves after restoring .cbm — via MoveFileExW with
  extended-length paths on both arguments, since the deep generation
  target is beyond the handle-based rename's bare-path reach — so a
  restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
  out so the tree is shallow enough for the detached cleanup's rd, and
  the cleanup's working directory strips the extended-length prefix
  (CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
  (CopyFileW destinations, CREATE_NEW tombstones, probe directories)
  are explicitly owner-stamped so the exact-owner validators hold on
  runner images; guard fixtures stamp hand-built trees the same way.

Diagnostics, tests and infra:

- diagnostics: discovery is now an always-delivered JSON control record
  (new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
  paths containing spaces; placement honors $TMPDIR with /tmp fallback
  via a diagnostics-local helper; the soak parser reads the JSON record;
  documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
  log levels, TMPDIR-with-spaces, and native Windows output-contract
  assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
  parents with production-shaped ancestry (LocalAppData on Windows, via
  th_secure_runtime_parent_new) — the runtime ancestry validation
  correctly refuses temp roots whose ancestors grant mutation rights to
  Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
  drive the documented startup-owner publication flow before reading
  generation-bound endpoint addresses. This turns the 26 Windows
  failures previously visible in CI's full-test job green without
  weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
  contract including failed-uninstall restore and immediate reinstall
  after uninstall; new daemon lifecycle and reworked hook-augment
  guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
  (it was set by CI but never read, so perf suites ran everywhere);
  four throughput/bench suites are classified as perf, the heavy
  store_arch suite moved to the slow-timeout tier, and two
  wall-clock-sensitive assertions were rewritten as invariant checks
  with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
  contract, soak daemon-recovery contract, path-safety helper, the
  Windows VM worktree-sync contract wired into scripts/test.sh, and
  vm/win.sh guards building its clean embedded-UI product in an
  isolated BUILD_DIR so it cannot clobber the incremental test build.
  provision-windows.sh now installs Node.js for the guards UI build.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-21 15:32:15 +02:00
Martin Vogel 85c2d99480 test(daemon): raise the frontend announce deadline above the claim budget
The backpressure/EOF fixtures waited only 10s for the forked daemon to
announce readiness, but the daemon's own startup includes the cohort-claim
path whose internal budget is HOST_DAEMON_CLAIM_TIMEOUT_MS (12s). Under the
4-core oversubscription of CI runners a slow-but-legitimate startup blew
past 10s and was SIGKILLed before it could announce, so daemon_frontend
backpressure failed almost every ubuntu run while passing on fast machines.
Raise the announce deadline to 30s, comfortably above the daemon's own
worst-case budget. Verified: the backpressure failures no longer reproduce
under a local CI-fidelity 4-core run.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-19 16:55:44 +02:00
Martin Vogel 3325b31b37 test(daemon): skip the build-fingerprint hash in tests (root-cause fix)
Every spawned worker/daemon re-hashes the entire executable image for
the exact-build fingerprint; for the ASan test-runner (hundreds of MB)
that is tens of seconds per spawn on constrained CI runners and was the
sole cause of the whole daemon-family readiness-timeout flake class
(index_supervisor, daemon_frontend, cli, and much of the Windows
daemon surface). Add a test-only seam under CBM_CLI_ENABLE_TEST_API:
when CBM_TEST_BUILD_FINGERPRINT holds a valid stub the hash is skipped.
test_main.c sets it once; every forked child and re-exec'd worker
inherits it, so exact-build match AND mismatch behaviour is preserved
(a mismatch test still passes a different fingerprint via argv). Never
compiled into production. Local mac run: index_supervisor 6/6, the four
daemon suites 111/0, exact-build test green. The generous frontend
catastrophic-alarm values are kept as backstops.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-19 06:35:31 +02:00
Martin Vogel 8d465bc9ce test(daemon): widen worker/frontend hang-guard deadlines for slow ARM CI
The macOS legs went green with the prior calibration but the 3-core
ubuntu-24.04-arm runner is slower still: each spawned worker re-hashes
the entire ASan test-runner image (hundreds of MB) for the exact-build
fingerprint before signalling readiness, which there takes tens of
seconds. Raise the hang-guard deadlines generously - index_supervisor
60s ready / 90s terminal, frontend 30-90s, cli 45s. These are hang
guards, not benchmarks: large values only slow genuine-hang detection,
never mask a regression. (Root-cause fix - a test-only fingerprint
injection seam that skips the hash - is tracked for follow-up.)

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-19 00:42:19 +02:00
Martin Vogel 1fbfb1f69c test(daemon): calibrate frontend and cli-quiesce budgets for starved runners
Same class as the index-supervisor calibration, proven by the serial
tail: the daemon_frontend backpressure/EOF tests and the cli forced-
install quiesce child failed on the 4-core ubuntu leg even running
alone on a quiet machine. Their 2 s request/cleanup/acquire budgets are
below real startup cost there (identity capture, frontend spawn, lock
acquisition on cold IO). Raised to 10-30 s; still hang guards - a
wedged run fails loudly.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 19:37:02 +02:00
Martin Vogel ad2874bcff fix: close remaining daemon coordination races
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 15:26:38 +02:00
Martin Vogel 83c137d2a5 feat: complete shared daemon lifecycle
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-18 01:26:08 +02:00
Martin Vogel 0e00ef5702 feat: coordinate concurrent CBM sessions
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-07-16 19:20:46 +02:00