Brings the branch up to main 9c19b5dd.
Two same-spot insertion conflicts, each resolved as the main version
of the file plus the own diff of this branch:
- src/daemon/application.c: the job struct gained watch_owners on main
and request_started_ms plus the request facts here; both kept.
- tests/test_daemon_application.c: main added the #1818
autoindex-skipped helpers where this branch added the worker resource
failure test; both kept, the test placed after the doc_candidates
test.
Everything else merged cleanly. src/main.c changed on both sides: main
(#2838) lowers the worker CPU priority inside the worker role block,
and this branch strips the publication fence flag before that block
and installs the fence after the worker role options; the hunks are
disjoint and both are kept. tests/test_pipeline.c gained the Next.js
route tests (#2747) on main and the publication gate test here, also
disjoint. No call site needed adapting: no new main code calls a
function this branch re-signed, and main re-signed none that this
branch calls.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
test_daemon_application.c: the new auto-watch test passes false for the approve_broad parameter that #2017 added to cbm_workspace_grant_add, the same value the existing callers use; the merged tree did not compile without it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Under `daemon start` the session watch release kept every watch whose last
subscriber left, so the daemon could keep watching after the session ended.
The same release also runs when a session's root stops being allowed by
workspace policy (an approval revoked, a boundary declared). That is not a
disconnect: the session is still open and its root is refused, so keeping
the watch there let a permanent daemon go on indexing a root the user had
just withdrawn.
application_release_session_watch_locked() now takes session_ending. The
cancel and close paths pass true and keep the daemon-owned watch exactly as
before; the workspace-policy branch of the watch refresh passes false and
removes the watch in both modes, as it did before permanent watches.
Test: daemon_application_permanent_workspace_release_still_ends_watch
revokes the sensitive-root approval of a permanent session's root and sends
one request; its synchronous refresh must end the watch (RED when the
policy branch passes true).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Under `daemon start` a watch now outlives its last session. delete_project
from another session or the CLI removes the project's DB but meets no owner
of the watch: daemon MCP sessions carry no watcher pointer, so nothing
unwatches it, and the next change in that root re-indexed the deleted
project.
When a permanent watch has no subscribers, no job for its project is
running and its DB is gone, the watcher callback now removes the watch and
skips, the rule a session refresh already applies to a deleted project.
The running-job guard is needed because a full reindex unlinks the DB while
it rebuilds it: a second change during that rebuild must join the job, not
end the watch. Session-owned watches are unchanged.
Tests: daemon_application_permanent_watch_ends_when_project_db_is_deleted
(RED before: the deleted project was re-indexed) and
daemon_application_permanent_watch_kept_while_its_reindex_runs (RED
without the running-job guard: the second change ended the watch).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Bring the permanent-daemon watch ownership change up to current main.
Three conflicts in src/daemon/application.c:
- Struct comments: keep main's index-record bookkeeping struct as is,
then the branch's comment on watch-job subscriptions describing the
permanent-mode ownership.
- application_release_session_watch_locked: main now releases every
watch membership a session holds. Keep that loop and apply the
branch's rule inside it: a permanent daemon keeps the physical watch
when its last subscriber leaves; non-permanent applications still
remove it.
- application_background_index_admit: watch_live keeps main's
application_watch_root_matches() and the index-args encoding that
follows it, and additionally admits a permanent watch with no
subscribers.
Ownership hand-off: main split application_background_index into
_admit and _wait, so the watcher-ownership flag set at admission could
no longer reach the terminal unsubscribe in the wait loop. _admit now
reports it through a bool out-parameter, application_background_index
passes it to _wait, and _wait releases the retained subscription once
when the job is terminal. The terminal block always breaks out of the
loop, so the old reset of the flag after that release is gone.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
main moved 12 merges past this branch's last base (72c36044). One
conflict, in src/daemon/application.c: #2737 split
application_background_index() into admit/wait helpers and gave it
named result codes. Resolved by taking main's structure and keeping
this branch's one change there: the automatic job's args carry
_cbm_background after the policy call. The watcher path re-encodes
mode/persistence/doc_candidates into the same document, so the marker
survives the re-encode, and the job comparison still drops it.
The net diff against main is unchanged: the same 13 files and the
same added and removed lines as before the merge.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
When daemon-backed auto-indexing refuses a repository, the
daemon.autoindex.skipped warning reported the tracked-file count but not
the limit that caused the refusal. The MCP autoindex.skip event already
carries limit=<n> (#1466); add the same field to the daemon warning,
using the auto_index_limit value the admission check already used.
Admission behavior is unchanged.
daemon_application_auto_index_honors_tracked_file_limit now captures the
warning through a log sink and asserts reason=too_many_files, files=1 and
limit=0 as whole fields.
Fixes#1818
Signed-off-by: Gaurav Dubey <gauravdubey0107@gmail.com>
Match canonical Windows separators across watch ownership and admission,
drain all session watches when auto_watch is disabled, and retain the
resolved doc_candidates choice for later refreshes. Add deterministic
regressions for the five observed failures and an absent-option guard.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Automatic jobs (session auto-index, watcher re-index) now carry
_cbm_background, and the job registry decides whether a new request may
join the job already running for its project by comparing the index
arguments after normalising defaults. An explicit index_repository for
the same root therefore stopped matching the automatic job and was
refused with "another index operation for this project is active with
different options" -- the refusal the repo_path separator fold
(9846c3f1) removed on Windows, now on every platform.
_cbm_background chooses the worker count and never what is indexed, so
drop it from the comparison the same way default-valued options are
dropped. The explicit request joins the running job again.
The new daemon_application test holds an auto-index job open with the
fake worker, checks that its args carry the marker, and sends an
explicit async index_repository for the same root. Without this change
it is refused as an options conflict; with it the request joins (one
worker start, one active job). The async fixture gains an optional
stored config so the test can enable auto_index.
Refs #1084
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
main moved 1,029 commits past this branch's base (96c3f41c). Six
same-spot conflicts, each resolved by keeping both sides:
- src/mcp/mcp.c, index_run_supervised_path: the supervised worker args
keep main's doc_candidates forwarding and add _cbm_background after it.
- src/mcp/mcp.c, autoindex_thread: main's index_doc_candidates_apply()
runs first, then the committed-index headroom decision.
- src/pipeline/pipeline.c, run_extraction_phase: a frozen pipeline still
forces one worker and records its route; every other pipeline takes
cbm_pipeline_worker_count().
- src/pipeline/pipeline_incremental.c, run_extract_resolve: same shape,
the frozen check first, then the pipeline-aware count.
- src/pipeline/pipeline_incremental.c, closure_probe_surfaces: main's
def_count < 0 abort is kept; the surface rows borrow probe_ctx with the
pipeline set only around that one call, as before. probe_ctx never
allows spilling, so the result lookup takes the same branch it took
with a NULL context.
- tests/test_mcp.c: both RUN_TEST blocks are kept.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Replace the PATH_ALLOW_BROAD environment variable with a recorded grant,
the shape agreed in review: `codebase-memory-mcp allow-root
--approve-broad <path>` records an explicit human approval for one root
refused as too broad (CBM_WS_DENY_TOO_SHALLOW), mirroring
--approve-sensitive end to end.
- cbm_workspace_grant_add() takes approve_broad next to
approve_sensitive. Each flag lifts only its own verdict, absolute
denials stay unliftable, and a refusal names the flag that would
lift it.
- The approval is stored with a '^' marker, listed by allow-root --list
as "(approved broad)", and honoured by cbm_workspace_root_allowed()
only for CBM_WS_DENY_TOO_SHALLOW and only on an exact path match,
keeping the exact-match rule of the original change.
- cbm_workspace_root_allowed() returns to its previous signature. Every
caller (MCP index_repository and index-job start, auto-index, daemon
sessions, the HTTP UI) reads the same grant store, so stdio and
daemon sessions decide identically and nothing depends on the
environment a daemon was started with.
- The too-broad refusal names `allow-root --approve-broad <path>`, a
command a person runs.
- cbm_workspace_classify_root() and cbm_workspace_verdict_is_overridable()
are unchanged.
The regression test keeps its name and its exact-match, sibling,
absolute and sensitive checks, now against the grant store, and adds
that a broad marker never lifts a sensitive verdict.
Co-authored-by: Mark Murawski <github@kobaz.net>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Watchdogs now sample the OS-charged footprint instead of resident size on
Darwin, and index_max_duration_seconds covers the whole request so a
crash/hang respawn cannot reset the clock.
Signed-off-by: 刘冲 <mail@liuchong.dev>
Discovery limits bound what indexing accepts, not what it then costs. A
repository well inside those bounds can still exhaust the host through parser
memory, or simply never finish, and a supervised worker that hangs leaves the
parent waiting with nothing to report.
Add index_max_rss_mb and index_max_duration_seconds, enforced by the parent
against the worker process tree rather than the worker process alone, so a
runaway child cannot hide behind a small parent. Resident memory is sampled
through the platform interface on macOS, Linux and Windows. Crossing a limit
terminates the tree and yields one trusted, structured terminal result that
attributes the failure to the resource that caused it.
Both limits default to off. A measurement that cannot be taken fails the
attempt instead of passing it: a watchdog that quietly stops watching is
worse than no watchdog at all.
The shell fixture that stands in for the supervisor names the two new keys.
The worker accepts only a policy that spells out every key it knows, which is
what keeps a stale supervisor from starting a worker it cannot bound.
Signed-off-by: 刘冲 <mail@liuchong.dev>
The daemon keeps adjacency layers (store/graph_mem.h) resident across
requests -- 237 MB for the kernel -- with no end of life but LRU pressure.
- Idle eviction: a layer nobody has held for CBM_GRAPH_MEM_IDLE_EVICT_MS
(10 minutes, measured from its last release) is evicted on the next
request: cbm_mcp_handle_tool sweeps after releasing the request store, and
every acquire sweeps too. No timer thread. Ten minutes outlasts the gaps
between an agent's graph calls within one task, while a session that moved
on stops pinning the memory; a wrong guess costs one rebuild (1.2-1.5 s on
the 17 GB kernel index, tens of ms on ordinary repos). The clock is a test
seam (cbm_graph_mem_set_clock_for_testing), so tests move time instead of
waiting.
- Worker budget: application_worker_memory_slice_locked hands index workers
the aggregate minus the resident layer bytes (cbm_graph_mem_cached_bytes),
at least one byte per active job so a slice never reads as "no cap".
- Memory class: layers are charged to the new CBM_MEM_CLASS_GRAPH_CACHE
("graph_cache" in census and class logs) instead of STORE.
Responses stay byte-identical to the baseline on django and redis (tools
and Cypher) and on the kernel trace/hub and extra BFS/degree cases.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Doc-section -> code candidates (doc_link_candidates, #2574/#2610) are the
documentation-in-the-graph feature of this release. They now run by
default for full indexes and not for moderate or fast ones; an explicit
`doc_candidates` argument, or a `doc_candidates` setting the user set,
overrides the default either way in every mode.
- pipeline: the switch is unset/on/off; unset means on only when the
effective mode is full.
- MCP: forwards only an explicit choice (argument, then a user-set config
value) to daemon jobs and workers; daemon auto-index and watcher/UI jobs
now carry a user-set value too (their workers have no config, so
`doc_candidates false` was ignored there).
- fast + doc_candidates on runs the semantic pass's corpus phases and the
doc phase only: candidates, and still no SIMILAR_TO or
SEMANTICALLY_RELATED edges.
- config: `doc_candidates` joins the config key table (default `auto`);
`config set doc_candidates ...` was refused as an unknown key before.
README, docs/CONFIGURATION.md and the MCP schema describe the default.
Proof (macOS arm64, django fb11376):
full 11,304 rows · moderate 0 · fast 0 · full+off 0 · moderate+on 1 ·
fast+on 1 (moderate/fast discovery skips docs/) -- nodes and edges are
identical between on and off in every mode; fast+on adds 0 semantic
edges.
- New tests (RED on the old code): pipeline
pipeline_doc_candidates_default_follows_mode,
pipeline_doc_candidates_explicit_wins_over_mode,
pipeline_doc_candidates_fast_on_legacy_incremental; mcp
tool_index_repository_doc_candidates_follow_mode,
tool_index_repository_doc_candidates_forwarded_only_when_explicit;
daemon_application
daemon_application_programmatic_index_forwards_set_doc_candidates.
- mcp 374 passed (4 Windows-only skips), daemon_application 73 passed,
pipeline/store_arch/cli/runtime_settings/index_policy/pipeline_frozen
green on the release tree.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Since #1948 a non-git root can be polled by a file-tree scan when
watch_non_git is on. index_status still reported such a root as
strategy "none", which the enum documents as "not a git project: never
polled", so a root that is actively polled looked unwatched.
publish_status now maps a tree-polled root to the new
CBM_WATCHER_STRATEGY_TREE, rendered as "tree" (the value the
watcher.baseline log already uses). "none" keeps its meaning: a
registered non-git root that is never polled. CONFIGURATION.md lists
the four values.
Test: daemon_application_index_status_reports_tree_strategy_issue2167
fails without the production change and passes with it; suites
daemon_application, watcher and mcp: 507 passed, 4 platform skips.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The background watcher only watches the project an open MCP session is
rooted in, and drops the watch when the last session for that project
closes. A repository indexed with `cli index_repository` (or from a
session rooted elsewhere) is never watched, but README promised
"Auto-sync keeps it fresh after that" for every index_repository target
and the MCP instructions said "Indexes auto-refresh". Nothing in the
status output said which projects were actually watched, so a stale
CLI-indexed repo looked like a watcher bug.
What gets watched is unchanged. Instead:
- index_status gains a `watch` object: `watched`, plus `strategy`,
`poll_interval_ms` and `last_scan_at` when watched, or a `reason`
(not_session_project, cli_session, auto_watch_off, watcher_disabled,
not_registered, no_watcher) and a re-index hint when not. It does not
touch the #1561 `freshness` key.
- The watcher publishes per-project strategy, cadence and last completed
scan time through atomics and exposes them via
cbm_watcher_project_info(); the daemon answers index_status through a
session-scoped provider.
- README, docs/CONFIGURATION.md, docs/index.html, the index_status tool
description and the MCP instructions now state the real scope.
`watcher_enabled` already shows in `config list` since v0.11.0 (680bc72e).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Addresses the four blockers from review:
- Capture the expected worker count while CBM_WORKERS is still unset in
mcp_auto_index_in_process_uses_background_worker_policy, so the assertion no
longer reads the lane override restored just before it.
- Give pipeline_background_policy_preserves_index_results a fixture past
MIN_FILES_FOR_PARALLEL and drop any inherited CBM_WORKERS /
CBM_INDEX_SINGLE_THREAD pin, so it really compares an all-core index against
a headroom index instead of two sequential runs.
- Route the two remaining all-core sites through cbm_pipeline_worker_count:
the LSP surface pass and incremental manifest hashing. The pipeline is
plumbed into cbm_pipeline_build_semantic_manifest for that.
- Rename the hidden key _background to _cbm_background, matching the existing
_cbm_index_policy convention, across every producer and consumer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Du315CaKLufAYPEdEwcocq
Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
With #2134, an unnamed index_repository of a root that another indexed
project already owns adopts that owner's name, and the daemon keys the
job by it (application_index_project_key reads the forwarded "name").
The unnamed status poll that the #2144 async reply asks for ("the same
repo_path and status: true") still derived its key from the path alone
in index_status_project_key. For an owned root it never found the job
it had just started and answered "no index job or index is known for
project '<path-derived>'"; an ambiguous root got that same misleading
answer instead of the loud error the start returns.
index_status_project_key now runs the start path's
index_root_owner_resolve on the same canonical repo_path: an explicit
name still wins, a single other owner is the key, several owners fail
with the same error naming them, otherwise the path-derived key stays.
The status error buffer grows to hold that owner list.
The daemon's other readers of the key (job dedupe in
application_job_subscribe_locked, the synchronous cut-short marker set
and taken around application_index_run_sync) derive it from the
forwarded args, which already carry the owner name, so they agree with
the start; status was the only reader that diverged.
Refs #2134, #2144
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The supervisor creates the worker's response file before spawning it. A
worker that exited 0 without writing that file left it empty, and
slurp_worker_response read the empty file as "" with status OK. The run
then counted as a SUCCESS with an empty response (the daemon marked the
job successful). The clean-exit branch of worker_terminal_log also
deleted the worker log, the only record of where the run stopped. Only
an oversized response was treated as a failure.
- index_supervisor: an empty response file is WORKER_RESPONSE_READ_EMPTY.
A clean exit with no readable response sets response_missing, keeps the
worker log, records its path, and records the last structured event
the log holds (text or JSON format, last 64 KiB) as last_phase. It is
logged as index.supervisor.no_response with the log path and the last
phase. The outcome stays CLEAN, so the disposition is FALLBACK: an
explicit error, with no in-process fallback and no recovery re-run of a
worker that did not crash.
- mcp: cbm_mcp_index_worker_no_response_failure builds the error
(reason "no_response", last_phase, worker_log, hint). It is used for
the first attempt and for a recovery run.
- daemon application: the job failure message says the worker exited
cleanly without writing a response, and gives the last phase and the
log to inspect.
- pipeline_incremental: the legacy per-file purge loop now logs
incremental.purge.start (changed/deleted counts) and
incremental.purge.progress. Progress lines carry counts only, use a
ceiling stride (at most ~20 lines plus the final one) and never log one
line per file. Before this, nothing was logged between
incremental.edge_snapshot and incremental.purge.
Tests (RED before, green after, RED on revert):
- index_supervisor_clean_exit_without_response_is_named_failure_issue1300
(a real worker subprocess logs a phase and exits 0 without a response)
- daemon_application_clean_exit_without_response_is_named_failure_issue1300
- incr_purge_reports_bounded_progress_issue1300
- index_supervisor_clean_exit_without_response_response_issue1300 (the
contract of the response builder)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolve the RUN_TEST list conflict in tests/test_daemon_application.c by keeping both sides' new tests.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
index_repository blocks until the whole index completes. MCP clients with
a per-call deadline (Copilot for IntelliJ) give up on a large repository;
their cancel drops the daemon job's last subscriber, the daemon cancels
the worker, and every retry starts over, so the index never finishes.
Add an opt-in async mode and a status query to the same tool; the default
synchronous behaviour is unchanged:
- async: true starts the project's index job in the daemon, or joins the
one already running for it, and returns at once with its state. The
application holds one subscriber reference for an async job until its
terminal publish, so the request returning, the client cancelling or
the starting session closing no longer cancels it. Only daemon shutdown
(or the final session of a non-permanent generation) ends it. Async
requests never queue behind the physical job limit; a full daemon
answers busy. async/status are stripped from the worker args, so an
async request coalesces with an identical synchronous one.
- status: true reports the running or last job for the project
(queued/running/cancelling/succeeded/failed/cancelled, started_at,
finished_at, error summary) from a small per-project record kept in the
daemon's job registry after the job itself is reaped. An unknown
project is an error; a project indexed before this daemon generation
reports idle. No freshness key is added: index_status keeps the #1561
freshness object.
- async + status together, non-boolean values, and async with
cross-repo-intelligence are refused. In-process servers (index worker,
embedders) refuse both modes: nothing there outlives the call.
- On a temporary (non-permanent) daemon, async is refused when the
request arrives on the one-shot `cli` tool channel and that session is
the daemon's only live session: the daemon would stop and cancel the
job the moment the command exits. The error points at `daemon start`
(or an open MCP session). A long-lived MCP session stays a valid host
even when it is the only one, which is the IDE case this exists for;
status stays allowed everywhere. The daemon lifecycle is unchanged.
- New allocations go through the memory core (cbm_alloc/cbm_calloc/
cbm_mem_strdup/cbm_free); blocks returned by non-core APIs are released
with safe_free, so no file's raw allocator count grows.
- A cut-short synchronous call now offers the async alternative: in the
cancellation text the daemon or frontend delivers (including the
-32800 reply to a cancelled index_repository), and, because a timed-out
client never reads that reply, as a notice on the next index_repository
or status call for the project.
- The tool description and schema, the CLI help and the README explain
the flags, the deadline problem and the polling pattern.
No server-side timeout or budget is introduced.
Tests (deterministic, held fake worker, bounded observation waits):
async returns before completion and status observes running then
succeeded; an async job survives its session closing while a sync job in
the same shape is still cancelled; validation errors; the cancellation
reply and the next-call notice carry the advice; a lone one-shot client
of a temporary daemon is refused async while status and an MCP session
on the same daemon are allowed; in-process refusal; schema; notice
helper.
Refs #2031Fixes#2144
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Add source-backed repository navigation, selection context and a shared change
analysis workspace for symbols, working-tree changes and Git comparisons.
Persist agent events and project logs through the existing daemon with SQLite
schema 4, bounded polling, retention and restart-safe deduplication.
Repair compact RPC decoding, path-search races, source pagination and identity,
large map responses, live error propagation and explicit local diagnosis flows.
Include the real-repository browser runners and focused regression coverage.
Validation: 2473 frontend tests, 203 acceptance checks, production build,
style/promise gates, 5 reference-boundary checks and 7 local hook tests pass.
The complete repository-map browser flow also passes after daemon restart.
Global C lint and cross-platform release validation remain open and are
recorded in the accompanying handoff report.
Signed-off-by: Bernhard Jackiewicz <bj@techport.io>
Indexing accepts whatever a repository contains. A tree carrying a vendored
monorepo, a generated dump, or a runaway build directory is discovered in
full, and the first sign of trouble is a host under memory pressure with
nothing that attributes it to indexing.
Add two opt-in limits evaluated during discovery against accepted source
files only: index_max_files and index_max_source_mb. Both default to off, so
nothing changes until an operator sets one. Crossing a limit fails the whole
attempt with a structured resource_limit_exceeded result naming the resource,
the observed value and the limit; no partial graph is published, and an
existing serving index keeps answering.
Limits are read from the CLI-managed _config.db and are not MCP request
arguments. A supervised parent replaces any caller-supplied policy before
spawning its worker, and the worker rejects a missing or incomplete contract,
so the CLI, the daemon and the supervised worker all enforce the same
decision.
Both keys reach an operator through the existing config get/set/list/reset
with no new subcommand. `set` suppresses its own generic message for them
because the policy writer names the precise reason -- so that writer speaks
on every failure it can return, including a validated value whose write then
fails on a database that cannot be written. Exiting non-zero in silence is
not an acceptable answer from a CLI.
The two shell regressions that hand-roll the supervisor's worker argv carry
that contract as well. Without it the worker exits before either guard can
observe anything, and the guard would go quietly vacuous.
Signed-off-by: 刘冲 <mail@liuchong.dev>
The daemon handed every index worker aggregate/physical_job_limit (a
fixed quarter by default) regardless of how many jobs were running, so a
lone worker on a host like #1864 was capped at a quarter of the memory
the daemon itself was allowed — and with the budget now failing whole
(decision A) that fixed slice would have turned an idle host into a
guaranteed over_memory_budget. Decision 2 (#1997, #832): the slice is
decided at spawn time as aggregate / jobs active right then (the job
being spawned included), so a lone worker may use the whole aggregate
and concurrent jobs split it. The explicit CBM_MEM_BUDGET_MB override
keeps its precedence inside the worker (bf46b91f); the sub-byte-per-slot
guard is unchanged. The spawn line logs project, active_jobs,
aggregate_mb and slice_mb so a fail-whole verdict can be read against
the budget the worker really had.
Tests: sequenced admission proves first=aggregate, second=aggregate/2;
the racing 4-way admission test accepts any aggregate/1..4 slice (the
divisor is scheduling-dependent there); a scripted CLEAN worker
returning the over_memory_budget error response is passed through
verbatim on the first attempt — no recovery loop, no quarantine.
Refs #1997#832#1864
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The first cut on this branch (c1b9c451) respelled the session root and
the allowed root with forward slashes on the way into the session policy
(cbm_mcp_server_set_session_context) and at the three application.c sites
that canonicalize a root. CI's Windows shard then failed two tests:
tests/test_daemon_application.c:2256: ASSERT(sensitive_blocked)
tests/test_daemon_application.c:2380: ASSERT(sensitive_blocked)
Both set HOME to the cbm_canonical_path spelling of a directory - the
platform's native form, backslashes on Windows - and expect a session
rooted there to be refused for auto-index and for watch. The sensitive-root
and allowed-root containment walk (ws_is_ancestor_or_equal in
src/foundation/workspace.c) compares prefixes byte-exact, so a root that
now read C:/Users/... no longer matched a HOME of C:\Users\...: the
sensitive refusal was lost and $HOME was admitted. Respelling the policy
root was the wrong layer.
This commit keeps the policy roots in their canonical native spelling
(mcp.c and the three application.c sites are back to what main runs) and
folds the separator spelling at the one comparison the original defect
lives in: application_index_args_equal, where the running job's args are
compared with an explicit index_repository request's. The auto-index job
spells repo_path the way the session policy holds it; the handler spells
it with forward slashes. On Windows the two never matched, and the request
was refused as OPTIONS_CONFLICT instead of joining the job already running
for its root. The fold acts on the parsed copies only (yyjson_mut_obj_replace
with a cbm_normalize_path_sep'd string), so nothing the daemon stores
changes spelling and every other option stays exact.
Why this is correct on Windows without a local Windows run: the policy
spelling is untouched, so the containment path is byte-for-byte what main
runs there today; cbm_normalize_path_sep folds backslashes on every
platform, so the new fold is exercised by the new test wherever the suite
runs.
Tests: daemon_session_context_keeps_one_spelling_of_a_root asserted that
the policy respells a root with forward slashes. That was the wrong
contract: the policy's containment checks compare its spelling byte-exact
against HOME and the granted roots, so a respelled root is exactly what
stops matching them. It is replaced by
daemon_session_context_keeps_the_policy_spelling_of_a_root, which binds
the opposite: the policy stores the root as given. The comparison is exposed through
application_internal.h as cbm_daemon_application_index_args_equal_for_test
and bound by daemon_application_index_args_compare_repo_path_separator_equivalently:
a backslash and a forward-slash repo_path compare equal; a different path,
a sub-path, and a different mode still do not.
Refs #1726
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A single query_graph reply larger than one 10 MiB frame killed the MCP server:
exit code 1, no JSON-RPC error, nothing on stderr, no crash report. Under a host
that does not respawn the server, every tool from it was gone for the rest of
the session — the reporter lost their tooling mid-investigation to one dead-code
query and finished by driving the binary over stdio by hand.
Root cause is a conflation, not the limit itself. application_mcp_request
checked only response_length > UINT32_MAX and passed anything smaller on, so an
oversized reply reached the transport, which rejected the frame. The frontend
worker cannot tell that rejection from a dead socket, and for a dead socket it
deliberately _Exit()s the process — closing the kernel IPC handle is the only
portable way to cancel daemon session ownership from a thread blocked in stdio.
Correct for a broken transport; a fatal over-reaction to a large answer.
The substitution therefore has to happen in application_mcp_request, which is
the last layer still holding the request id. Replies that fit are returned
byte-identical; a reply that cannot be framed becomes a JSON-RPC -32603 naming
both numbers and what to do ("narrow the projection, add LIMIT, or paginate"),
because the failure it replaces was a silent exit and an opaque "internal error"
would be no improvement.
Reproduced first, on a locally generated 20k-node fixture rather than the
reporter's repo (external code stays unverified until reviewed):
before LIMIT 10000 -> ok, 8,529,990 bytes
LIMIT 20000 -> SERVER DIED exit=1 stderr=''
after LIMIT 20000 -> JSON-RPC -32603, and a follow-up query on the SAME
session succeeds — which is the property that actually matters
The decision and the substitution live in one function on purpose. The first
version of the test built the error object directly and PASSED with the size
check removed: it proved the error was well-formed while the server still died.
Testing application_framable_response instead binds the branch — verified by
removing the guard, which reddens it at ASSERT(replaced != big).
Not addressed here, deliberately: every reply still carries its payload twice
(content[0].text and structuredContent.text are verbatim duplicates, ~2.05x
measured), which is why the ceiling is reached at ~4 MB of real content. That is
a wire-format question affecting every tool and every caller's token bill, so it
belongs in its own change with a decision behind it, not smuggled into a crash
fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
All three cost real release cycles, and none is fixed by widening a budget.
daemon_application_cancels_physical_job_only_after_final_session waited for the
SUBSCRIBER COUNT to reach 2, then cancelled both sessions and asserted the
physical job had started exactly once. The job starts asynchronously after
subscription, so both cancels could land first and leave starts == 0 --
arguably the correct outcome. It now waits for the state the assertions
actually require. Verified 47/47 on Windows, the only platform it ever failed.
The parallel harness refused outright when the suite leader had already exited,
because taskkill /T cannot walk a tree from a dead PID. But the leader can exit
between the timeout decision and that call, so the harness itself lost a race:
a natural exit at the wrong moment failed the whole wave. It now proves cleanup
the only way still available -- nothing parented to that PID -- and its contract
asserts the PROPERTY rather than the phrase "tree cleanup" it used to grep for.
That string pin is what broke when the guard was reworded while behaving
correctly; the contract now checks rc==2 AND that the descendant really did
survive, which would also catch a guard that claims to fail closed while
leaking.
extract_wide_flat_file_is_linear took ONE sample per size, so the ratio carried
the noise of both. On a loaded Windows VM linear code measured 51x against a 40x
bound (184ms -> 9387ms). Best-of-N instead: timing noise only ever adds time, so
the minimum is the cheapest good estimate of the noise-free cost. The bound is
deliberately unchanged -- it sits where linear (~20x) and quadratic (~128x) are
each >=2x away, so raising it would move the test toward the very signal it
exists to catch. Now measures 19.1x on Windows, 21.4x on macOS.
Also: the smoke's `cli` helper redirected stderr to a file and discarded it, so
any of the 10 bare `VAR=$(cli ...)` assignments could kill the run under
`set -euo pipefail` printing NOTHING. One such abort cost a full Windows cycle
just to locate and still could not be attributed. It now surfaces the command
and its stderr. Neutral wording on purpose: one call site expects a non-zero
exit and must not read as a failure.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The fresh-request guard test waited for 'job terminal AND >= N/2 subscribers
still attached'. Publish flips terminal and releases the blocked prior
requests in the same locked step, so that conjunction is a transient window
whose width is thread-wakeup scheduling: the busy-spin caught it by luck,
a 1ms-yield poll (#1307) sampled past it on fast x64 runners, and no budget
can pin it (release runs 30305464193 and 30309182389 failed it from both
directions).
The production guard never consults subscriber counts —
application_find_active_job_locked skips any terminal job — and the test's
downstream assertions (starts==2, destroys==2, stale/fresh response
separation) catch a terminal-job reuse in every interleaving. So wait only
for the stable end-state (active jobs == 0) and drop the racy helper.
47/47 locally.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
app_wait_for_terminal_job_with_subscribers was the only app_wait_for_*
helper polling without cbm_usleep(1000): a sleepless spin pins a core for
up to 10s and can starve the daemon threads it is polling on scarce-CPU
runners. Release run 30305464193 hit exactly this on windows-11-arm
(test_daemon_application.c:2671 ASSERT(terminal_with_prior_subscribers));
the suite passes 47/47 locally with the yield in place.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two changes to the index coordinator, developed together and validated as
one state:
Determinism seams. A cancel racing a worker start was only observable on a
loaded runner; now the interleaving is forced by construction: a test hold
parks every job thread before its first pre-start cancel check, and a
background-initialize completion counter turns sleep-then-assert negatives
into positive-signal waits. The cancel-before-worker-start case is pinned
by a dedicated test; three auto-index admission tests now wait for the
admission tail instead of sleeping through it.
Queued admission. An explicit index_repository request that hits the
physical job limit used to surface a raw busy error. The request thread
already blocks for the whole index, so it now queues: a cancellable,
shutdown-aware wait-and-resubscribe loop replaces the error path, with a
wait counter as the parked-state observable. The programmatic coordinator
API keeps its non-blocking busy answer for callers with their own retry
policy (watcher, auto-index). The job-limit test asserts the new contract:
parked without a second worker start, admitted and completed once the
occupying job finishes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The suite's observation waits shared a 2000 ms budget that is a hang
detector, not a latency assertion — every waiter returns the moment its
condition holds, so green runs never pay it. Under ThreadSanitizer's
5-20x slowdown on a loaded 3-core macOS CI runner the cancel-delivery
wait lost the tail of that distribution once in seven otherwise-green
TSan rounds (the cancel itself had succeeded; only the fake worker's
counter observation timed out). The budget is now 10000 ms.
Verified: nine consecutive daemon_application runs under native macOS
TSan with the widened budget, all green, zero ThreadSanitizer
warnings.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.
Daemon lifecycle:
- daemon start/stop/status subcommands. `daemon start` launches a
PERMANENT daemon (spawn shape is byte-exact argv; survives idle
periods and session ends) and reports an already-active daemon
instead of failing. Permanence is honored at every stop latch:
last-committed-client disconnect, host initial-client window,
coordinator release, and application final-session close — a
permanent daemon also keeps admitting new sessions after its last
one closes.
- daemon stop refuses while sessions are active and lists the blocking
peers (pid/role) that must finish first; an idle daemon drains
through the activation-shutdown machinery with the ACK ordered after
connection interrupts. A second stop is idempotent. The wire ops are
no-cohort first-frame requests with peer fingerprint authentication,
so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
keep their local supervised path). A cold CLI run that had to spawn a
temporary daemon prints a hint that `daemon start` removes the
per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
emits a visible, rate-limited notice (Claude-dialect systemMessage
plus stderr for other dialects) and always exits 0 — augmentation is
never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
older-build permanent daemon (strict semantic-version triples only;
dev builds never auto-drain) and the build-conflict message names
`cbm daemon stop` as the manual escape hatch.
Windows IPC/runtime (real-VM verified):
- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
path used to destroy its listening pipe instance on every 20 ms poll
timeout; a client attaching in the teardown window was severed or left
on an orphaned pipe object whose HELLO no server handle could ever
read, absorbing the connect until the client's own timeout expired.
The pending connect now survives poll timeouts and nothing is
destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
server handle can discard a just-sent response before the peer reads
it (POSIX stream sockets never lose buffered data on close). A bounded
cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
immediate on interrupted connections) now precedes close in
runtime_worker_finish and runtime_reject_inline, so hello-conflict,
capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
transport half-close, so close_begin now announces departure with an
explicit frame (ordered after APPLICATION_CANCEL, before the local
interrupt); the server releases the client's admission on receipt
instead of waiting for the handle to close. Admission-drop timing is
now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
request publication under the send lock, best-effort sends the active
token's APPLICATION_CANCEL frame, then interrupts local I/O; the
server cancels MCP/subprocess work promptly. Contract tests accept
both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
the requester's license to act on "snapshotted and draining", so every
connection interrupt is now initiated before the ACK is sent; a
session could previously get one more request serviced after the
requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
timer granularity) burned ~1.6 s against permanently obstructed paths,
stalling hello rejections past the client's timeout. The retry budget
is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
nonblocking send() on Windows is absorbed wholesale into AFD kernel
buffering regardless of SO_SNDBUF, so send deadlines and interrupts
could never engage against a slow peer (and the full payload was
pinned in nonpaged pool). Slicing restores a deterministic
backpressure point; a test hook pins SO_SNDBUF for the deadline and
interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
deadline, explicit connection states, refusal to free a server while
a listener-owned connection is active.
Windows long-path support:
- Central path-aware wide conversion (canonicalize via GetFullPathNameW
and prepend the extended-length prefix for absolute paths >=240) at
the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
store opens, and the daemon build-fingerprint/log paths. Deep managed
installs (a 64-hex generation directory routinely exceeds MAX_PATH)
now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
ancestry validators now operate in the extended-length namespace;
the launcher path is canonicalized (and prefixed when deep) once at
entry so every downstream exact-string comparison stays
form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
GetFileAttributesW) so non-ASCII install paths survive argv[0]
resolution.
Windows launcher install/uninstall transaction:
- FileRenameInfoEx names are NUL-terminated in an over-allocated
buffer. FileNameLength governs per the contract, but filter drivers
read FileName as NUL-terminated and appended adjacent heap bytes to
created names — a flaky, garbage-suffixed rename target. Both the CLI
and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
.cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
chars so the bare rename target stays under the FileRenameInfoEx
NT-conversion ceiling at guard depths; 64 bits still uniquely
identify the generation.
- When the running launcher's mapped generation backings pin .cbm
against rename, the backings are relocated to activation-<pid>-N
.retired tombstones beside the install (a mapped image may be renamed,
never deleted; the launcher's liveness-guarded sweep reclaims stale
tombstones). Every relocation is recorded, and a FAILED uninstall
reverses the moves after restoring .cbm — via MoveFileExW with
extended-length paths on both arguments, since the deep generation
target is beyond the handle-based rename's bare-path reach — so a
restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
out so the tree is shallow enough for the detached cleanup's rd, and
the cleanup's working directory strips the extended-length prefix
(CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
(CopyFileW destinations, CREATE_NEW tombstones, probe directories)
are explicitly owner-stamped so the exact-owner validators hold on
runner images; guard fixtures stamp hand-built trees the same way.
Diagnostics, tests and infra:
- diagnostics: discovery is now an always-delivered JSON control record
(new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
paths containing spaces; placement honors $TMPDIR with /tmp fallback
via a diagnostics-local helper; the soak parser reads the JSON record;
documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
log levels, TMPDIR-with-spaces, and native Windows output-contract
assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
parents with production-shaped ancestry (LocalAppData on Windows, via
th_secure_runtime_parent_new) — the runtime ancestry validation
correctly refuses temp roots whose ancestors grant mutation rights to
Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
drive the documented startup-owner publication flow before reading
generation-bound endpoint addresses. This turns the 26 Windows
failures previously visible in CI's full-test job green without
weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
contract including failed-uninstall restore and immediate reinstall
after uninstall; new daemon lifecycle and reworked hook-augment
guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
(it was set by CI but never read, so perf suites ran everywhere);
four throughput/bench suites are classified as perf, the heavy
store_arch suite moved to the slow-timeout tier, and two
wall-clock-sensitive assertions were rewritten as invariant checks
with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
contract, soak daemon-recovery contract, path-safety helper, the
Windows VM worktree-sync contract wired into scripts/test.sh, and
vm/win.sh guards building its clean embedded-UI product in an
isolated BUILD_DIR so it cannot clobber the incremental test build.
provision-windows.sh now installs Node.js for the guards UI build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
Gaurav Dubey
wargloom
刘冲
Matt Van HornandClaude Opus 5
Bernhard Jackiewicz
Daichi Matsumoto
Ertan
LeoLan Digital