feat(index): add worker resource watchdogs

Discovery limits bound what indexing accepts, not what it then costs. A
repository well inside those bounds can still exhaust the host through parser
memory, or simply never finish, and a supervised worker that hangs leaves the
parent waiting with nothing to report.

Add index_max_rss_mb and index_max_duration_seconds, enforced by the parent
against the worker process tree rather than the worker process alone, so a
runaway child cannot hide behind a small parent. Resident memory is sampled
through the platform interface on macOS, Linux and Windows. Crossing a limit
terminates the tree and yields one trusted, structured terminal result that
attributes the failure to the resource that caused it.

Both limits default to off. A measurement that cannot be taken fails the
attempt instead of passing it: a watchdog that quietly stops watching is
worse than no watchdog at all.

The shell fixture that stands in for the supervisor names the two new keys.
The worker accepts only a policy that spells out every key it knows, which is
what keeps a stale supervisor from starting a worker it cannot bound.

Signed-off-by: 刘冲 <mail@liuchong.dev>
This commit is contained in:
刘冲 committed 2026-10-19 17:18:00 +08:00
1 parent d4980ad757
commit de1615264a
21 files changed
+1472 -111

No files matched your search

+44 -4
View File
@@ -1,8 +1,8 @@
# Index resource limits
Index resource limits are optional operator controls for repositories whose
discovery breadth is not known in advance. They are disabled by default so
existing large-repository workloads retain their current behavior.
discovery breadth or worker runtime is not known in advance. They are disabled
by default so existing large-repository workloads retain their current behavior.
## Discovery settings
@@ -23,6 +23,39 @@ Values use base-10 integers. MiB means 1,048,576 bytes. Empty values, zero,
negative values, suffixes, trailing characters, and values outside the stated
ranges are rejected without changing the stored value.
## Worker settings
| Key | Default | Accepted value | Protects |
|---|---:|---:|---|
| `index_max_rss_mb` | `off` | `off` or `64..1048576` | Current RSS of the complete worker process tree |
| `index_max_duration_seconds` | `off` | `off` or `1..86400` | Total worker wall-clock duration |
```bash
codebase-memory-mcp config set index_max_rss_mb 8192
codebase-memory-mcp config set index_max_duration_seconds 3600
```
RSS is the current resident memory of the contained worker and every descendant,
not the worker's allocation budget and not peak memory. This hard watchdog is
separate from the internal `CBM_MEM_BUDGET_MB` soft budget. The supervisor
samples RSS at most once every 250 milliseconds so the watchdog does not turn
full process-table enumeration into a busy loop.
Duration uses a monotonic clock from successful spawn. It is independent of the
existing 15-minute quiet timeout: continuous log progress does not reset total
duration, while the quiet timeout continues to identify a worker that stops
making progress.
Equality is allowed. The first RSS or elapsed-duration observation above its
limit starts the existing graceful-to-force process-tree shutdown. CBM reports
terminal only after the tree is quiescent or a bounded containment failure is
explicitly surfaced. Resource termination is not retried and does not
quarantine a source file.
If RSS is enabled and three consecutive probes cannot obtain any trustworthy
tree measurement while the root worker is still running, CBM fails closed with
`code=resource_probe_failed`.
## Counting and failure semantics
`index_max_files` counts a file only after it passes directory pruning, ignore
@@ -55,6 +88,12 @@ The previous database remains available because publication occurs only after a
complete discovery and successful staged build. If no previous database exists,
`serving_index_preserved` is false.
Worker limit failures use the same shape with `stage=worker`,
`resource=rss_bytes` and `unit=bytes`, or `resource=duration_ms` and
`unit=milliseconds`. RSS measurement failures use `code=resource_probe_failed`
and omit `observed`, `limit`, and `unit` because no trustworthy observation was
available.
## Trust and compatibility
Limits are read from the CLI-managed `_config.db`; they are not MCP request
@@ -64,5 +103,6 @@ parent policy.
These settings do not replace or increase `auto_index_limit`, change the 512 MiB
single-file cap, alter workspace-root authorization, or affect
`cross-repo-intelligence`. With both settings `off`, discovery follows the
existing unbounded path.
`cross-repo-intelligence`. With all settings `off`, discovery follows the
existing path and the supervisor performs no periodic RSS probe or total-duration
termination.