eval_condition_partial (from #2253) answers UNKNOWN for a condition whose
alias is not bound yet, but it only looked at c->variable. A function
condition is filed under its first argument's alias, so in
MATCH (a)-[:CALLS]->(b) WHERE coalesce(a.no_such_prop, b.name) = 'x'
the seed row saw a bound a, read coalesce(NULL, <unbound b>) as a
definite mismatch and pruned every row before CALLS expansion: zero rows,
while the same predicate after WITH returned the right ones.
Check every function argument's alias as well, the way this branch's
original fix did (condition_bindings_available), inside main's
three-valued leaf.
RED before: cypher_exec_where_func_arg_on_unbound_target, row_count 0,
expected 1. GREEN after; cypher suite 237 passed.
Co-authored-by: lorenzozanee <wyz0707@proton.me>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Conflict in tests/test_cypher.c: both sides added WHERE NOT regression
tests right after cypher_exec_where_not. Kept both sets, main's four
#2253 tests first, then this branch's four, with their RUN_TEST lines in
the same order. src/cypher/cypher.c merged cleanly to main's version.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On the kernel index (8.5 M nodes, 1.9 M CALLS edges) `MATCH (a)-[:CALLS]->(b)
RETURN count(*)` peaked at 18-25 GB RSS (killed in a 12 GB container): the
store materialised every start node with all its strings (4.9 GB), every one
was deep-copied into a 1.6 KB binding, and every expanded row was materialised
again before projection.
cypher: execute_single now runs the start nodes through the pipeline in chunks
of 1024 (pattern-0 hops, late WHERE, WITH, RETURN). Each hop emits a binding's
expansions in input order and every consumer (RETURN rows / star / default
projection / aggregation, WITH aggregation / projection) keeps its state
between chunks, so the concatenated output is the single-pass output: same
rows, row order, first-seen group order, ORDER BY ties, DISTINCT, LIMIT and
max_rows truncation, OPTIONAL fallback rows, warnings. Queries with additional
MATCH patterns keep one pass (their join strategy depends on the first binding
of the whole set). Start nodes and fetched neighbours move their strings into
the binding instead of being deep-copied and freed.
store: cbm_store_node_scan_open/_next/_close hand out the rows of
cbm_store_find_nodes / _by_label a page at a time. Pages are keyset queries
along the index the eager query walks (idx_nodes_file: NULL file_path rows by
id, then the rest of the last file by id, then the files after it;
idx_nodes_label: by id), each reset before it returns, so no read lock is held
between pages. open verifies the eager plan with EXPLAIN QUERY PLAN and falls
back to the eager rows for any other plan.
Kernel, same index, medians of 3 (macOS arm64):
count(*) peak RSS 24.9 GB -> 256 MB, 23.0 s -> 13.0 s
top-callee peak RSS 12.4 GB -> 1.26 GB, 11.7 s -> 10.9 s
Responses byte-identical to the baseline on django, redis and the kernel
(19 Cypher + 24 tool cases each); the streamed scan equals the eager scan
field by field, in order, for every project and label of all three indexes.
A scan that fails part-way fails the whole query ("query aborted: reading the graph failed") instead of answering from the pages read so far:
The streamed start-node scan answered from the rows of the pages it had read
when a later page failed (a SQLite step error or a page re-prepare failure):
a partial result that looked complete. A failed page, a scan that cannot
open, or a failed eager fallback now marks the start source failed;
execute_chunk returns an error and cbm_cypher_execute reports
"query aborted: reading the graph failed (<store error>)" with no rows. The
store error is logged as cypher.scan_failed.
The test seam cbm_store_node_scan_test_fail_page_after (CBM_ENABLE_TEST_SEAMS)
fails one page after a given number of successful pages; the new test fails
the second page of a label scan (rows and count(*)) and a later page of the
all-nodes scan, and asserts an error result with no rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
search_code
- Scan the indexed files in process instead of through grep (POSIX) or a
PowerShell Select-String pipeline (Windows), with the same grep -f
semantics: one pattern per line, ERE regex, binary-file and NUL handling,
-Hn records. The subprocess runner stays for the unindexed recursive
fallback and a Windows file_pattern it cannot prefilter.
- Windows now matches like macOS and Linux: Select-String matched
case-insensitively, used .NET regex syntax and read NUL-containing
files as text. A trailing CR is still ignored there, so `foo$` keeps
matching CRLF checkouts.
- Result merge, root containment and source reads no longer scale with
hits x results: a node-id index, one canonical root per request with a
per-file verdict cache, and fread spans instead of per-byte fgetc.
query_graph
- ORDER BY (RETURN and WITH) was a bubble sort. A stable merge sort keeps
the same comparisons and so the same order; WITH keys are resolved once
per row instead of twice per comparison.
- GROUP BY, DISTINCT (RETURN and WITH) and COUNT(DISTINCT) looked every
row up in a linear list. They hash the same keys now.
Store reads
- Read tools run their statements in one deferred read transaction,
renewed every 100 ms, instead of taking the database file lock once per
statement.
list_projects
- Resolve only the branch the listing prints: 2 git processes per project
instead of 7.
On macOS and Linux every response is byte-identical to before:
search_code 30/30 (django), 28/28 (this repo) and 28/28 (Linux, GNU
grep); all tools and the Cypher cases 86/86 on shared django and redis
indexes. On django: search_code 908 -> 355 ms, WITH ... ORDER BY
41 s -> 62 ms, count(*) 508 -> 184 ms, list_projects 94 -> 40 ms. On the
Windows ARM VM: search_code 1.6-3.4 s -> 0.2-0.5 s.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
eval_expr_partial, the three-valued evaluator that execute_single runs on
every seed row before relationship expansion, recursed into the right
child of every AND / OR / XOR node. With operator chains linked along the
right child, that is one stack frame per operand: the 50,000-operand OR
and AND chain tests overflowed the 256 KiB query worker stack in this
early pass, before the already iterative eval_expr ever ran.
Walk a chain the way eval_expr does: recurse into NOT and into each
operand, and follow the links in a loop. The operands are read in the
same order and with the same short-circuits as the nested form (AND stops
at the first definite false, OR at the first definite true, XOR reads
every operand), and the running value is combined three-valued, so an
UNKNOWN operand anywhere in the chain keeps the seed for the full
evaluation after expansion.
A new test runs three-operand OR / XOR / AND chains whose UNKNOWN operand
sits on a chain link and not only on the chain's head. A loop that lets a
later operand overwrite the UNKNOWN prunes the seed and fails it.
eval_expr_partial moves into the whitelist's chain-walker group with
eval_expr and cypher_expr_selectivity: it still recurses, but only as
deep as the parse-depth cap allows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Use a thread-local test clock for the large UNION structural regression
so sanitizer overhead cannot consume its execution deadline. Preserve
all row, ordering, and constrained-stack checks. Add an exact positive
budget expiry case alongside the existing deadline tests.
Request the fixture's exact 20000-row budget to avoid reserving the
default 100000 binding slots for every branch, and assert no truncation.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A WHERE chain such as `a OR b OR c ...` was folded into a left-deep tree,
so every walker over the expression tree recursed once per operand: the
evaluator, the seed-selectivity pass of the planner and, every 128 nodes,
the free routine. UNION branches were parsed by a nested cbm_parse call
per branch, so the parser recursed once per branch as well. Stack use
therefore followed the length of a query, not its nesting, and the
daemon evaluates queries on a worker thread with a 256 KiB stack. A long
but flat query, such as a few thousand `OR name = ...` terms, needs to
work there.
Chains of one operator are now linked along the right child at parse
time, OP(a, OP(b, OP(c, ...))), and eval_expr and cypher_expr_selectivity
follow the links in a loop, recursing only into the operands, whose
nesting the existing parse-depth cap bounds. Operand order and the
left-to-right short-circuit of AND and OR are the same as in the nested
form, XOR keeps its running parity, and a missing value takes part in a
chain exactly as before. expr_free already walks with an explicit stack
and needs no change for a right-linked chain.
UNION branches are read by one parser in a loop in cbm_parse and linked
as they come; parse_post_where no longer parses the branch after the
keyword, and the ALL flag still sits on the branch before it.
Tests: 50,000-operand OR and AND chains and 20,000 UNION ALL branches run
on a thread with the worker stack size inside a forked child, with the
expected rows computed by the test; a chain with operands over a missing
property is compared with its parenthesised forms in both associations;
50 UNION and UNION ALL branches return the right rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
parse_match_pattern allocated the pattern's node and relationship arrays
and returned on a parse error without freeing them. Its callers count a
pattern only after it parsed, so cbm_query_free never saw it: every query
whose pattern failed partway leaked, including every named path
(`MATCH p = (...)`, not supported, fails at the first node). The element
whose parse failed leaked too (its variable, label or properties), and
parse_props freed only the array, not the entries it had already parsed.
The fuzz job found it on its first run, from a checked-in seed:
`MATCH p = (a:Class)-[:INHERITS*1..3]->(b:Class) RETURN a.name, length(p)`
(LeakSanitizer: 288 bytes in 2 allocations).
parse_match_pattern now frees and zeroes the pattern on every failure,
counting the failed element (parse_node and parse_rel zero it first, so
only what they allocated is freed). parse_props frees its parsed entries
on every failure through one helper.
New test cypher_parse_error_frees_partial_pattern drives five failure
points: a named path, a property list after one entry, a cut-off
relationship, a missing label after a relationship, and a second MATCH
clause's properties.
Checked with the fuzz binary (clang 18 ASan/LSan, arm64 container) on the
five queries: before this change the first four leak (288, 306, 339 and
348 bytes); after it all five are clean, and scripts/fuzz.sh all --runs
20000 --seed 1 passes for extract, cypher and config. The fifth goes
through the same parse_props failure as the second. macOS: the cypher
suite passes, 217 tests.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_regcomp() handed every pattern straight to the platform compiler
(libc regcomp on POSIX, the vendored TRE on Windows). Both expand bounded
repetition at compile time, so the compiled size of a pattern is not
proportional to its length. Measured on macOS libc (TRE-derived) with
REG_EXTENDED|REG_NOSUB, the compiled form costs about 0.6 KB per expanded
atom (1.1 KB with submatch tracking, 1.7 KB with REG_ICASE), a K-way
alternation adds about 70 B x K^2 for its position sets, and an
alternation under an interval pays that for every copy.
The wrapper now sizes a pattern in one linear pass before compiling it:
an atom costs 1, a group costs its content, an interval multiplies the
element it applies to by its upper bound, and a K-way alternation is
charged K^2/8 on top of its branches. Arithmetic saturates and group
nesting is bounded. A pattern longer than 32768 bytes, or whose estimate
exceeds 32768 units, is refused with CBM_REG_ETOOBIG before anything is
allocated. The heaviest accepted compile is a repeated alternation at
about 30 MB with REG_NOSUB (about 90 MB under the ICASE-plus-submatch
flag set no call site uses). Ordinary intervals, the pipeline's fixed
patterns and an alternation of 400 twenty-character identifiers remain
accepted.
The refusal reads as a normal compile error at every call site.
search_code now reports an unusable path_filter as a tool error instead
of silently dropping the filter, search_graph reports a failed row scan
instead of returning an empty result, and query_graph carries the reason
in the result's warning field; all three name it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The search_code `context` argument had no upper bound while every sibling
sizing argument (result_limit, match_limit, source_max_lines, raw_limit,
directory_limit) is clamped on both sides. A large value made every hit
return its file from the match to the end, and the window arithmetic in
attach_result_source could overflow int. `context` now gets the same
0..200 clamp as source_max_lines (MCP_SEARCH_CONTEXT_MAX_LINES), the
window computation is overflow-safe on its own, and the tool schema states
default, minimum and maximum like its siblings. Values inside the bound
return exactly the lines they did before.
Sink ceilings: cbm_cypher_execute clamps a caller-supplied max_rows to
CYPHER_RESULT_CEILING (nothing above it is ever materialized) and
execute_single checks the binding array it sizes from that value instead
of writing through a failed allocation.
Tests pin the context window and the extreme values (INT_MAX, negative),
the query_graph max_rows ceiling, the search_graph limit ceiling, the
trace cursor token truncated after the leg character, the quote rejection
in file_pattern, and the engine-side max_rows clamp.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An OPTIONAL MATCH with a relationship, whose start and terminal
variables are not bound by an earlier MATCH and whose start node
pattern matches no node, is joined in cross_join_with_rels() with an
empty candidate list. That path sized its output for a single binding,
but the OPTIONAL fallback then kept one row per existing binding with a
plain indexed store, so every row after the first was written past the
end of the allocation.
Route the fallback through binding_out_append(), the growing writer
that the expansion branch of the same function and the other OPTIONAL
fallbacks already use. The node-only join (cross_join_nodes) was fixed
earlier by sizing its buffer for the fallback rows; the relationship
variant kept the unchecked store.
Add a cypher regression test that runs such a query over four bound
functions and asserts one row per function with the optional variables
unbound. Before this change the test runner reported a
heap-buffer-overflow in cross_join_with_rels under ASan.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #1245 by Andrew Hundt (upstream b835b5e8), re-derived on
main's cypher.c rather than cherry-picked -- the file has moved ~9k lines
since the branch point.
The WHERE clause runs twice: once early, on seed rows where only the
first alias is bound, to prune; and once late, when every alias is bound,
to decide. The early pass used the same two-valued evaluator as the late
one. That evaluator answers "true" for a comparison on an alias it cannot
see -- the permissive default that makes the late pass work -- but NOT and
XOR invert whatever they are given, so
MATCH (a)-[:CALLS]->(b) WHERE NOT b.name CONTAINS "x"
evaluated NOT(true) on every seed and pruned every row. Zero results, no
error, for a query that is correct.
The early pass now uses a three-valued evaluator. A condition on an alias
that is not yet bound is UNKNOWN; NOT of UNKNOWN stays UNKNOWN; AND and OR
short-circuit only on a definite operand; XOR with any UNKNOWN operand is
UNKNOWN. Only a definite FALSE prunes. The late pass is unchanged and
still gives the final two-valued answer, so nothing that was correct
before changes.
Three deliberate differences from the upstream diff:
- the unbound-alias rule lives in one leaf, eval_condition_partial, so
the expression-tree path and main's legacy flat AND/OR path cannot
drift apart;
- that leaf guards c->variable == NULL. On main a condition whose left
side is a function of literals, coalesce(0, 1) >= 2, carries no
variable at all; the upstream version would have passed NULL to
strcmp;
- the flat path keeps main's rule that an empty condition list is true.
RED before the fix, all four with row_count == 0, expected 1:
cypher_exec_where_not_on_relationship_target,
cypher_exec_where_mixed_alias_and, ..._or, ..._xor. GREEN after: cypher
208 passed; cypher + mcp 526 passed, 4 skipped (the Windows-only skips).
eval_expr_partial recurses and is added to CBM_RECURSION_WHITELIST.
Co-authored-by: Andrew Hundt <ATHundt@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
RETURN DISTINCT projected up to the engine row cap (max_rows, 100k by
default) and only then removed duplicates in rb_apply_distinct. A value
that first appeared past the cap was never seen. On a 150k-edge index
`MATCH ()-[r]->() RETURN DISTINCT type(r)` returned 21 of 22 types:
HAS_BRANCH, a single edge, was missing, and the result was flagged
truncated:true / engine_limit.
The projection loops (plain RETURN, RETURN *, RETURN * after WITH) now
de-duplicate while projecting, through a hash set of length-prefixed row
keys. The cap therefore counts distinct rows. Memory stays bounded by the
cap: only admitted rows are remembered. Once the cap is full, the rest of
the bindings are only probed (no insert), to decide whether a new
distinct row exists, which is when `truncated` is set. An explicit LIMIT
on DISTINCT now also counts distinct rows and can stop the projection
early. ORDER BY and SKIP/LIMIT keep running after DISTINCT on the
already-distinct rows. The O(n^2) post-pass remains only for aggregate
rows, which are distinct by group key.
Proof on django (55,377 nodes / 294,528 edges, 23 edge types in
get_graph_schema), same index, before vs after:
before: 22 types, HAS_BRANCH missing, truncated:true engine_limit
after: 23 types, truncated:false (same 5.3 s wall time)
Test: cypher_return_distinct_counts_distinct_rows_against_cap_issue1364
(rare type past a max_rows=5 cap; ORDER BY, SKIP/LIMIT and LIMIT variants;
more distinct rows than the cap still reports truncated).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A node variable carried in scope through WITH (`WITH f`, `WITH f AS g`,
`WITH f, count(r) AS refs`) was projected as a scalar virtual var: the
node's name in .name and the variable name in .qualified_name (used as
the owned storage for the binding's name), start_line 0. RETURN
f.qualified_name therefore returned the literal variable name "f" and
f.start_line "0"; only f.name was right. The aggregate path tagged the
stub with the node id so node_prop could re-fetch some fields, but the
re-fetch only ran for empty string fields, so the non-empty
qualified_name and the numeric start_line never reached it.
The aggregate path also grouped a node by its display name, so several
same-named functions collapsed into one row carrying one node's
properties and the summed count.
Fix:
- A bare, bound node item in WITH is now carried as a deep copy of the
node, bound under its AST-owned alias, on both the simple and the
aggregate path; every property then resolves exactly as it does
without the WITH. Unbound OPTIONAL vars and scalar items stay scalar.
- Aggregate WITH groups node items by node id, not name.
- Scalar virtual vars keep their owned alias string in .project (never
exposed as a property) instead of .qualified_name, so
n.qualified_name / keys(n) no longer return the variable name.
- WITH ... ORDER BY g.<prop> resolves the property on the carried node.
- The node_prop stub re-fetch heuristic is removed: nothing produces
id-tagged stubs anymore.
Tests (cypher suite): cypher_issue2208_with_agg_carried_node_props,
cypher_issue2208_with_agg_groups_by_node_identity,
cypher_issue2208_with_simple_carried_node_props.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Nine findings from the 2026-09-16 graph-quality probe over the eleven bench corpora, each with a
test that fails without its fix and a before/after measurement on the real corpus.
Deterministic resolution (O9). Two indexes of one kernel tree differed in 5,821 IMPORTS and 632
CALLS edges. Three order/load dependencies carried it: the `#include` target was the first by-name
hit whose path ends with the include path — a bucket in worker merge order (include/linux/device.h
vs tools/virtio/linux/device.h); the registry broke suffix_match score ties by bucket position; the
LSP budget-share rule measured wall time. Include targets now prefer the File node, the least nested
path, then the lexicographic path; registry ties prefer the least nested QN, then the smaller QN; the
budget share is measured in thread CPU time like the budget itself. Two kernel indexes are now
identical (8,529,723 nodes, 1,974,940 CALLS, zero churn), as are two elasticsearch indexes.
Findability. search_graph's BM25 path honours the `label` filter and ranks an exact-name definition
above test methods that repeat the token (15/15 probe queries).
Coverage. The parse-coverage gap tree subtracts definitions the walker recovered inside an error
region (multi-line only); kernel/sched/core.c reports 476 flagged lines instead of 7,674.
Precision. Config-language callers (Makefile, CMake, YAML) no longer bind unique_name into code
(kernel Makefile -> C CALLS 195 -> 0). JS-family token names ({}, 1, (x: T) => U) are not
definitions; computed, quoted and escaped member keys stay (TypeScript 204 junk -> 0, 1,782 real
names kept). Names are cut at their first line (elasticsearch YAML 8,306 -> 0); a comment inside an
argument list is not an argument, so no Route node is named by a Java block comment; the wildcard
route `/*` is still a route. HTML/Vue/Svelte/Astro embedded member calls stay unbound.
Recall. A Python call through a self- or cls-rooted attribute keeps a specific unique_name member
target (Django +751, elasticsearch +3,288 CALLS, no new callee names).
Robustness. Edge properties never carry a split UTF-8 sequence (the 120-byte expression cut and the
JSON escaper both trim a partial code point). CLI array flags accept a JSON-array literal. Cypher
seeds a single-hop pattern from its selective end (kernel `MATCH (a)-[:CALLS]->(b) WHERE b.name =
'_printk'`: execution limit -> 7.9 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The head of the first pattern is bound whether the query names it or not:
execute_single falls back to the synthetic name "_n0". The capacity check
counted only names, so an unnamed head plus 16 named nodes passed it and
binding_set then dropped the 16th name without a word — the same silence
this guard exists to remove.
Count that slot, and name the synthetic variable once instead of twice.
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
check_projection_scope models declared names in a fixed 32-entry array
and skipped the check entirely when a query declared more. Nothing
bounded how many variables a pattern could declare, so the out-of-scope
refusal added by #1922 switched itself off on a wide query: the same
undeclared name was refused at 10 declared names and quietly accepted at
35, answering a column of empty strings.
Skipping was deliberate — a wrong refusal costs the caller a working
query, which is worse than the silence. The premise was wrong. A binding
holds CYP_MAX_VARS node variables and CYP_MAX_EDGE_VARS edge variables in
plain arrays, and binding_set and binding_set_edge drop anything past
those without a word, so a query naming more cannot be answered at all.
Its extra names bind to nothing and project as blanks, which reads as
"the graph holds no such data". The choice was never refuse-or-stay-quiet;
it was refuse, or answer wrong.
So bound the input. check_pattern_var_capacity counts distinct node and
edge variables across every pattern — they share one binding, and an
OPTIONAL MATCH pattern sits in the same array — and refuses beyond what a
binding holds, naming the limit and how to get under it. Leaving the name
off a node frees its slot, because an unnamed node takes none.
Bounding the input rather than each consumer also makes three other silent
drops unreachable: binding_set past 16, binding_set_edge past 8, and the
column collection in execute_default_projection. Those would otherwise
need an error path threaded up through every caller, since both binding
setters return void.
The two `< 0` branches in check_projection_scope stay, with comments
saying they can no longer fire. They keep the guard standing if either
bound ever moves.
Fixes#1995
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
Brings the Atlas branch up to current main (7b0f553c). Eight files
conflicted; how each was settled:
- src/mcp/mcp.c: the real ingest_traces description stays, in main's tool
row shape (main dropped the title column); index_repository keeps the
truthful abort branches (aborted_previous_preserved, persist_failed)
and gains main's format_migration flag in the success branch.
- src/ui/http_server.c and tests/test_httpd.c: the branch's twelve-tool
read allowlist for /rpc (#1663) over main's three, with main's %zu
and initialised counter.
- README.md: main's surface count (45), the branch's Atlas line.
- graph-ui (the previous frontend, replaced by the stacked #2068 but
kept building and green here): the branch's node-selection wiring
(onSelectNode) is kept consistently, so main's duplicate
handleSelectPath and its 3-argument call go; main's version chip is
added to the branch's header; useProjects keeps the branch's
render-first flow on top of main's paginated JSON helpers
(fetchAllProjects, fetchFullSchema), which the lean output contract
requires, and a project whose size the server did not send now gets
its schema too. main's GraphTab.selection.test.tsx, written against
main's GraphTab, is dropped in favour of the branch's own #1197
coverage in GraphTab.regions.test.tsx ("reports selected nodes to
the route for deep links"); main's useProjects pagination test waits
for the lazy schema enrichment instead of asserting before it runs.
Verified: mcp, httpd and ui suites 414 passed; the old frontend builds
and its 149 tests pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fix-forward after rebasing onto main (545 commits, 26 conflicted-file
resolutions). Where main had shipped a behaviour in the meantime, main
wins; the PR keeps its output contract on top of it:
- search_code: the scan runs under main's supervised, deadline- and
output-bounded runner and feeds the branch's streaming classifier from
the runner's output file; the POSIX wrappers map grep's no-match to 0,
so every non-zero exit fails closed (no exit-1 special case).
- trace_path: strategy/confidence precede args in header and rows (#1542),
in both the tree table and the json legs.
- cypher: relationship expansion is never capped (a cap falsified
aggregates) and unlabeled MATCH scans every candidate; `truncated` is
raised only by a real ceiling — a clamped hop range is probed one hop
beyond the cap and reports truncated only when a candidate exists there.
The plain BFS and the trail BFS both probe one row past max_results so
saturation is observable instead of silent.
- search_code full mode and the final json use main's U+FFFD sanitizer
instead of '?' replacement.
- search_graph json keeps an empty `groups` array for semantic-only calls;
list_projects json echoes offset/limit next to next_offset and accepts
main's include_details spelling for the stats projection.
- discovery budget: main added get_file_outline, compare_graphs, manage_adr
set_sections and the debug/include_details parameters; their descriptions
are trimmed to the lean style and the ceiling moves from 15 to 18 KiB.
Tests that asserted a JSON default now request format:"json"; the
fail-closed search test uses an unreadable regular file (a missing or
non-regular indexed path is deliberately not a scan operand on main); the
non-regular fixture gives its real sources File nodes, as every indexed
project has; list_projects total counts listable projects (the 0-byte
ghost is never a row, so it is not a page either).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
execute_return_star_after_with sized its column arrays at CYP_MAX_VARS
while the parser allowed up to CBM_SZ_32 items. A WITH naming 17 to 32
aliases parsed cleanly, then answered 16 columns and dropped the rest
with no error. The caller could not tell that from a query that genuinely
had 16 columns.
Widening those arrays to 32 is the wrong fix and would make the result
worse. A binding holds exactly CYP_MAX_VARS variables, and
with_add_vbinding_var drops any alias past the 16th on the way in. The
projection would then report 20 columns of which 4 are always blank —
silent wrong data in place of a silent short answer. The 16 was not an
arbitrary undersize; it matched what the binding can carry.
So bound the WITH where the RETURN is already bounded, in
parse_return_or_with, and refuse the query instead of answering it
short. The clamp in execute_return_star_after_with stays: it can no
longer fire, and it is what keeps that function correct if the bound
ever moves.
The test asserts both halves — a 20-alias WITH is refused, and a
16-alias WITH still succeeds with 16 columns — so the guard rejects only
what the binding genuinely cannot hold.
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
RETURN * built its columns from the variables the query pattern named,
never from the bindings it was about to project. One line caused two
separate wrong answers, and neither one reported an error.
After a WITH, the pattern's variables are out of scope — the WITH
replaced them with the names it made. The old code still asked for the
old names, found none of them, and answered a full result of empty
strings. This query used to print twelve columns of nothing:
MATCH (f:Function) OPTIONAL MATCH (f)-[:CALLS]->(g)
WITH f.name AS caller, g.name AS callee RETURN *
It now prints two columns, caller and callee, holding their values. A
name the WITH made holds one value rather than a node, so it gets one
column, not the four a node variable gets.
Separately, collect_pattern_vars appended every pattern's variables with
no repeat check. A variable named in two patterns got its four columns
twice, which the OPTIONAL MATCH above does with f.
Two tests cover both faults and fail against the old code:
cypher_return_star_dedups_repeated_pattern_var col_count 12, want 8
cypher_return_star_after_with_names_aliases col_count 8, want 2
Cypher suite: 185 passed, 0 failed. clang-format clean on both files.
Reported alongside a second fault this does NOT fix: a variable the WITH
dropped is still accepted afterwards and renders empty, because nothing
checks a projected name against the live scope. See
.agents/research/2026-08-29-cypher-return-star-and-with-scope.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
expect() fills p->error and returns NULL when the token after SKIP or LIMIT
is not a number, but both call sites only tested `if (num)` and then fell
through to `*out = r; return 0;`. r->limit kept the -1 "unset" initialiser,
which execution reads as "no LIMIT":
rb_apply_skip_limit(rb, ret->skip, ret->limit >= 0 ? ret->limit : max_rows);
So `RETURN n.name LIMIT $limit` ran unbounded and answered isError:false with
the whole result set. `SKIP $offset LIMIT 10` lost both clauses at once,
because the orphaned operand also blocked the following match(p, TOK_LIMIT).
Propagate the failure the way the ORDER BY branch directly above already
does. This is the failure mode #1334 banned, reached by a different route.
Fixes#1994
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: metehanulusoy <ulusoy.metehan03@gmail.com>
cbm_parse built a query and returned success without checking that it
had read every token. The grammar accepts at most one WITH and treats
RETURN as optional, so the parser stopped at the first thing it did not
understand and reported success anyway.
The dropped tail took the filter and the RETURN with it. The engine then
answered from the fragment it had parsed, using its default projection.
It reported success and returned wrong rows, which is worse than a
refusal, because nothing tells the caller to look.
Two shapes hit this:
MATCH (f:Function) WHERE f.name = 'x' RETURN f.name AS n BANANA SPLIT 99
-> one row, no error, the trailing words silently dropped
MATCH (f:Function)
OPTIONAL MATCH (a)-[:CALLS]->(f)
WITH f, count(a) AS calls
OPTIONAL MATCH (b)-[:USAGE]->(f)
WITH f, calls, count(b) AS usages
WHERE calls = 0 AND usages = 0
RETURN f.name AS n
-> every function, unfiltered, under the default column names
cbm_parse now checks that the cursor sits on the end-of-input token
before it returns success, and names the leftover token when it does not.
The message adds a note about the one-WITH limit, but only when a
standalone WITH really sits in the part that went unread. Appending it
every time pointed the reader at WITH when the problem was a typo -- the
BANANA query above holds no WITH anywhere. The test cannot simply look at
the token the parse stopped on either: on the second-WITH query above it
stops at OPTIONAL rather than at WITH, because parse_post_where consumes
the first WITH and then has no way to take another MATCH stage. So it
scans the unread tail, and skips a WITH straight after STARTS, which is
the STARTS WITH operator rather than a clause.
The check exposed a second defect. parse_post_where parses the branch
after UNION by calling cbm_parse on a slice of the same tokens, using a
separate parser. The outer parser's cursor never moved past the UNION
keyword, so a valid UNION query looked unfinished. That cursor was
already wrong; nothing caught it, because nothing checked where the
cursor ended up. The UNION branch now moves the cursor to the end after
the sub-parse succeeds, which is sound because that sub-parse no longer
returns success with tokens left over.
Three tests cover both directions. Removing only the guard turns the two
rejection tests red with rc == 0 -- the parser reporting success on input
it never finished reading -- while the acceptance test stays green, so
the guard is load-bearing for exactly these two behaviours.
The two rejection tests also pin the message. The BANANA case must name
BANANA and must not mention WITH; that assertion was seen red first, at
tests/test_cypher.c:226. The second-WITH case must still carry the note;
that one passed before and after, and is the control that stops the note
being suppressed everywhere instead of only where it misleads.
Verified on macOS with Apple clang:
make -f Makefile.cbm test-focused TEST_SUITES=cypher -> 186 passed
guard removed, tests kept -> 184 passed, 2 failed
make -f Makefile.cbm cbm -> exit 0, no warnings
The full suite reports 7623 passed, 2 failed. Both failures are in
tests/test_cli.c (lines 1748 and 6723) and reproduce on a clean tree with
this change stashed out. They depend on the coding agents installed on
the machine, not on this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
A RETURN could name a variable the WITH before it dropped. The query ran,
exited clean, printed a column for that name, and filled it with empty
strings. Nothing said anything was wrong.
That silence is the harm. A reader sees a column of nothing and reads it as
"the graph holds no such data". The true answer is "your query named
something that is out of scope". For a code graph, a false "nothing calls X"
is the worst answer it can give.
The check runs on the parsed query rather than on the run-time bindings, and
that split is the whole point. It asks whether the query DECLARED the name,
which is a different question from whether a row happened to bind it. So the
existing convention is untouched: an OPTIONAL MATCH target that matched
nothing is still declared, still legal, and still projects "".
Two placeholders are skipped, because neither names anything the query
declared: count(*) stores "*", and a CASE expression stores "CASE".
A query with more names than the guard can model skips the check rather than
guessing. A wrong refusal costs the caller a working query, which is worse
than the silence this removes.
Same failure shape as #373, and the same answer: say so out loud.
Fixes#1919
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
Three test families leaned on ingest_traces being a discard-stub:
- the daemon-runtime usability probe sent traces:[] and grepped for the
stub's traces_received; it now sends a valid minimal ingest for the
seeded project (the daemon requires the project argument) and checks
the honest pairs_unmatched report
- cypher_wide_return_projection_bounded forked the sanitized runner for
crash isolation; a fork that then does real store setup is
byte-fragile on macOS — a 1.5KB schema addition got the child killed
before the query ran (bisected: any equal growth flips it, even with
a 1-column query). The parse-time bound is the contract; assert it
in-process. A regression still fails loudly, and the test now runs
on Windows too instead of skipping
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Second mechanism of #1196, exposed by the reporter's v0.10.6 retest: every
relationship-expansion site capped its output buffer at bind_cap*10, so edges
past the cap were silently dropped BEFORE WHERE and aggregation ever ran.
count() then reported the scanned prefix as if it were a fact — 9,360 of
13,691 DEFINES field-measured at --max-rows 1000 — and a LABEL on the source
did not protect you (the label workaround only ever fixed source
enumeration, which #1323 already made exact). max_rows is an output-row
limit per the public header; projection already enforces it.
All five capped sites now share one growable append (geometric growth,
size_t sizing): the per-hop expansion, its fixed/variable-length helpers and
process_edges, the bound-terminal driver, and the cross-join outer buffer.
Only allocation failure stops materialisation; match_count stays truthful
either way, so the #627 OPTIONAL contracts hold (a saturated buffer can no
longer exist, and the fallback rows share the same append). The #601
deadline still bounds pathological time, and hop caps keep bounding depth —
this removes only the silent row-dropping.
Regression test: 2 labeled sources x 30 edges with max_rows=2 — the old cap
returned count=20; ground truth 60 now holds, and the list form returns
exactly max_rows rows. Proven RED before and RED again on revert; cypher
181/181 and mcp suites green, including every OPTIONAL/#627 semantics test.
Fixes#1196 (together with #1323, which fixed the unlabeled source scan).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A pass over places where a bound was either miscalculated or missing.
UI response builders (src/ui/http_server.c) mixed the clamping helper
http_appendf with raw cursor indexing. Route the per-line separators and
quotes through the helper too, so the cursor cannot leave the buffer once
output saturates it, and terminate explicitly afterwards — the helper pins
the cursor and then writes nothing, which would otherwise leave the "%s"
reply with no terminator in range. The log buffer was budgeted at
LOG_LINE_MAX + 10 per line while JSON escaping can double every byte, so
size it for the escaped worst case. Job status entries now escape their path
and error fields, which were interpolated raw.
Cypher (src/cypher/cypher.c): the WHERE grammar descends once per nested '('
and once per NOT, so parse depth followed the query text rather than
anything bounded. Track depth on parser_t and refuse past 256 levels with a
parse error. Bounding parse depth also bounds the resulting tree, so the
recursive evaluator inherits the limit.
Gitignore (src/discover/gitignore.c): '**' retries the remainder at every
position and consecutive groups multiply, so match cost is exponential in
the number of groups. Thread a step budget through the matcher and give up
past 20000 steps, reporting no-match so a pathological pattern fails to
ignore rather than ignoring the wrong files. glob_match keeps its name so
recursion_whitelist.h still describes the functions that recurse; the new
non-recursive wrapper that seeds the budget is glob_match_bounded.
Call extraction (internal/cbm/extract_calls.c): extract_fp_callee recursed
once per applied argument, so stack use followed the parse-tree depth of the
indexed file. Rewritten as a left-spine loop. Reassigning node/nk before
continuing leaves the fall-through cases where the recursive form left them,
so behaviour is unchanged and the extraction suite is untouched.
CALLS props (src/pipeline/pass_calls.c): `cap - pos - PAIR_LEN` wraps once
pos reaches cap - PAIR_LEN and stops bounding the copy below. Guard
additively, matching the closing write in the same function.
Shell-bound paths (src/foundation/str_util.*): the three git shell-out sites
each wrap a repo path in cmd.exe-compatible double quotes, where %VAR%,
!VAR! and ^ remain active. Two carried private copies of the check that
rejects those three on Windows and the third used the bare validator.
Promote the stricter form to cbm_validate_shell_path_arg and route all three
through it, so the copies cannot drift apart again.
Tests cover the buffer bounds, the parse-depth refusal, matcher termination
and the validator. The bounds tests run in a forked child so a violation
surfaces as a signal instead of silently.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Cypher request model stored a single order_by expression; on
ORDER BY key1 DESC, key2 ASC LIMIT n the parser consumed only key1,
left ', key2 ... LIMIT n' unparsed, and the query silently returned the
entire result set (6326 rows / 117 KB instead of 5 on the reporter's
graph - a token flood straight into agent context).
The return clause now models up to CBM_CYPHER_ORDER_KEYS_MAX (8) sort
keys with per-key direction (Cypher semantics). Both sort sites -
rb_apply_order_by for RETURN and sort_bindings for the WITH pipeline -
compare key-by-key with later keys breaking ties. More keys than the
modeled maximum is a loud parse error, never a silently dropped
remainder. The no-ORDER-BY projection fast path is preserved via
order_key_count == 0.
Tests: parse-level (2 keys, per-key direction, LIMIT consumed;
9-key over-cap rejected), exec-level RETURN (limit kept, tiebreak
ordering, per-key direction), and exec-level WITH pipeline (limit kept).
All RED before the fix (row_count 4 instead of 2), GREEN after.
Recorded, pre-existing and unchanged: an ORDER BY key that is not part
of the projection is silently skipped (single-key main behaves the
same); sorting by unprojected properties needs hidden-column projection
and is a separate issue.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogelandlorenzozanee
Joshua Richter
metehanulusoyandClaude Opus 5
King Star
imi
SEPURI-SAI-KRISHNA