The "<project>::missed" shadow graph lists the files the indexer did not
fully cover. unresolved_calls rows describe files that were indexed; their
call evidence is reported by check_index_coverage and trace_path. The
miss-graph builder skipped only not_indexed_* rows, so every file with an
unresolved call became a missed File node: on a Prometheus checkout the
shadow graph grew from 30 nodes / 29 edges to 961 / 960. A file with both
a parse_partial and an unresolved_calls row also lost its parse_partial
node, because rows arrive ordered by (rel_path, kind) and the File upsert
keeps the later one.
Treat unresolved_calls like not_indexed_* in all three places the builder
filters rows: the failure fingerprint, the failure count that decides
whether a shadow graph exists at all, and the per-row loop.
store_coverage_shadow_excludes_unresolved_calls covers a project whose
only rows are unresolved_calls (no shadow project), a file with both kinds
(its parse_partial node survives), and the fingerprint (equal to the same
failure rows without the evidence). Reverting each filter on its own turns
its check red.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On the kernel index (8.5 M nodes, 1.9 M CALLS edges) `MATCH (a)-[:CALLS]->(b)
RETURN count(*)` peaked at 18-25 GB RSS (killed in a 12 GB container): the
store materialised every start node with all its strings (4.9 GB), every one
was deep-copied into a 1.6 KB binding, and every expanded row was materialised
again before projection.
cypher: execute_single now runs the start nodes through the pipeline in chunks
of 1024 (pattern-0 hops, late WHERE, WITH, RETURN). Each hop emits a binding's
expansions in input order and every consumer (RETURN rows / star / default
projection / aggregation, WITH aggregation / projection) keeps its state
between chunks, so the concatenated output is the single-pass output: same
rows, row order, first-seen group order, ORDER BY ties, DISTINCT, LIMIT and
max_rows truncation, OPTIONAL fallback rows, warnings. Queries with additional
MATCH patterns keep one pass (their join strategy depends on the first binding
of the whole set). Start nodes and fetched neighbours move their strings into
the binding instead of being deep-copied and freed.
store: cbm_store_node_scan_open/_next/_close hand out the rows of
cbm_store_find_nodes / _by_label a page at a time. Pages are keyset queries
along the index the eager query walks (idx_nodes_file: NULL file_path rows by
id, then the rest of the last file by id, then the files after it;
idx_nodes_label: by id), each reset before it returns, so no read lock is held
between pages. open verifies the eager plan with EXPLAIN QUERY PLAN and falls
back to the eager rows for any other plan.
Kernel, same index, medians of 3 (macOS arm64):
count(*) peak RSS 24.9 GB -> 256 MB, 23.0 s -> 13.0 s
top-callee peak RSS 12.4 GB -> 1.26 GB, 11.7 s -> 10.9 s
Responses byte-identical to the baseline on django, redis and the kernel
(19 Cypher + 24 tool cases each); the streamed scan equals the eager scan
field by field, in order, for every project and label of all three indexes.
A scan that fails part-way fails the whole query ("query aborted: reading the graph failed") instead of answering from the pages read so far:
The streamed start-node scan answered from the rows of the pages it had read
when a later page failed (a SQLite step error or a page re-prepare failure):
a partial result that looked complete. A failed page, a scan that cannot
open, or a failed eager fallback now marks the start source failed;
execute_chunk returns an error and cbm_cypher_execute reports
"query aborted: reading the graph failed (<store error>)" with no rows. The
store error is logged as cypher.scan_failed.
The test seam cbm_store_node_scan_test_fail_page_after (CBM_ENABLE_TEST_SEAMS)
fails one page after a given number of successful pages; the new test fails
the second page of a label scan (rows and count(*)) and a later page of the
all-nodes scan, and asserts an error result with no rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Overloads that share a base QN (Java/C#/C++/Kotlin/Swift/Scala/ObjC) merge
into one graph node today, and that node collects the CALLS of every
overload. The agreed cure is one signature-qualified callable identity for
all overloading languages:
callable_qn := base_qn [<tparams>] (params) [cvref]
This change lands the plumbing with every language still at
CBM_CALLABLE_ID_NONE, so the graph is byte-identical; each language is
switched on later in its own change (with its index-format bump).
- internal/cbm/callable_sig.{c,h}: the per-language mode side table
(NONE|TYPED|LABELED_TYPED|LABELED|ARITY; kept out of CBMLangSpec like
cbm_string_dispatch_suffixes, since ~160 positional rows would otherwise
all need the member under -Wmissing-field-initializers), the shared
builder cbm_callable_sig[_mode] for the def and the call-scope QN, and
the inverse cbm_qn_callable_base_len[_named]. Suffix contract: no '.',
'::' or '->'; whitespace-normalized; comments/annotations/attributes
dropped; qualified types keep their last segment; '->' spelled '=>';
variadic '[]' (Java) / '~'; C++ top-level cv dropped, arrays decay,
cvref kept; <= 200 bytes, longer ones keep whole leading entries and end
in '#<fnv1a64 of the full suffix>)'. The builder is iterative (cursor
walk, bounded declarator loop) and self-checks against the inverse.
- The inverse is anchored to the node's bare name wherever the name is
known: real QNs already end in ')' without being callables (a Java field
whose name is its declarator `X = f()`, make `$(VAR)` targets), and only
the anchor makes the inverse exact on them.
- Leaf splitters strip a suffix first (no-op today): registry by-name key
(indexed by the bare name for a suffixed callable), cbm_lsp_bare_segment,
cbm_pipeline_qn_class_method_tail, pxc_last_component, pxc_qn_leaf,
go_index_visit_node, collect_prod_defs, sg_qn_prefix_len.
- New unused fields: CBMDefinition.qn_sig_off, CBMResolvedCall.callee_sig
(relocated by result compaction), CBMRegisteredFunc.sig, CBMLSPDef.sig.
The lsp_surface codec is unchanged (no format bump in this change).
- get_code_snippet and trace_path gain base-QN tiers
(cbm_store_find_nodes_by_qn_base) that find nothing until suffixes exist.
- store: cbm_store_find_nodes_by_qn_suffix built its LIKE pattern in a
512-byte buffer (a long suffix was truncated into a pattern that matched
nothing) and did not escape '_'/'%' (`my_func` also matched `myXfunc`).
The pattern is now heap-built, escaped, and matched with ESCAPE '\'.
Proof: full sorted node+edge+lsp_surface dumps are byte-identical before
and after on 7 corpora (this repo, elasticsearch libs, Exposed, dotnet
System.Linq, System.Collections, coreclr/gc, a C corpus).
Refs #2061
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_store_count_nodes and cbm_store_count_edges treat every sqlite3_step
result other than SQLITE_ROW as a count of zero. A read that failed —
SQLITE_CORRUPT, SQLITE_BUSY, SQLITE_IOERR — is therefore indistinguishable
from a project that genuinely holds no rows, and index_status renders it
as the positive assertion status "empty". A user or agent reading that
concludes the repository was never indexed and starts a multi-minute
re-index, while the corruption itself is never surfaced.
Both functions already have an error channel: each returns CBM_STORE_ERR
when prepare_cached fails. Only the step result was not reported through
it. index_status is already written for that value — it sets degraded on
a negative node count and clamps a negative edge count — so the guard
existed and simply never fired.
Initialise count to CBM_STORE_ERR so a non-row step is reported as a
failed read. A successful step still overwrites it with the real count,
so the healthy path is unchanged.
Closes#2012
Signed-off-by: kavish-19 <63698788+kavish-19@users.noreply.github.com>
A dbt model is an ordinary .sql file whose dependencies are written
{{ ref('other_model') }} or {{ source('group','table') }}, never as
literal table names. The SQL grammar cannot read those -- FROM {{ ref(
'x') }} is a parse error to it -- so the dependency structure of an
entire dbt project was invisible to the graph.
extract_dbt.c runs as a sub-extractor inside the normal indexing
pipeline. Per qualifying file it emits one Model definition (named by
the file stem, dbt's own model identity) plus one usage per ref()/
source() call; pass_usages resolves those into model -> relation
lineage edges like any other reference.
Model is a relation label alongside Table and View, which is what makes
the rest work without new plumbing: registry seeding, the central
relation veto, the incremental surface hash, search ranking and the
architecture queries all pick it up from cbm_label_is_relation. Sharing
one label class also means a model's source('raw','customers') resolves
onto a Table declared in a plain DDL migration in the same repository,
so dbt lineage and SQL lineage form one graph rather than two, while
the veto keeps model names out of every non-lineage consumer.
The pass is self-gating: SQL files only, and only those carrying a real
ref()/source() call. The dbt builtins are themselves the evidence, which
is cheaper than a dbt_project.yml lookup and more precise -- templated
SQL that is not dbt (an Airflow {{ ds }} parameter) produces no Model
node and no usages even inside a dbt repository.
{% macro %} definitions are deliberately excluded: the vendored
tree-sitter-jinja2 grammar has no node types for {% %} statements, so
they can only be recovered by a hand-written scanner that cannot see
comments or find {% endmacro %} for a correct span. Filed as follow-up
rather than approximated.
Tests cover the lineage, the last-string-argument semantics of both
builtins, the gate (against a plain-SQL control extraction), plain DDL
staying untouched, and an end-to-end pipeline case asserting model ->
model across files, model -> Table onto plain DDL, and cross-language
isolation. Disabling the pass reddens three of them; removing Model from
the relation set breaks lineage outright.
Implements the lineage half of #575.
Co-authored-by: alexisperinger-ux <alexis.peringer@iss-stoxx.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
CREATE TABLE / CREATE VIEW / CREATE MATERIALIZED VIEW now produce Table
and View nodes (previously generic Variable), CREATE PROCEDURE produces
a Function, and schema-qualified DDL names (schema.table) are named by
the table identifier instead of the schema. A view's FROM/JOIN relations
are emitted as usages and resolve into view -> table USAGE lineage
edges.
Relations join the cross-file name registry so lineage can resolve, with
two structural safeguards:
- Registry membership is defined once by cbm_label_is_registry_symbol
(helpers.c); the full, parallel and incremental seed sites all call
it, ending the KEEP-IN-SYNC copies the old label lists required.
- The default cbm_registry_resolve vetoes relation-labeled results:
common table names (users, orders, config) collide with code
identifiers in every language, and no CALLS/USAGE/READS/WRITES/THROWS/
handler/decorator consumer may bind them. The SQL lineage path opts in
through the new cbm_registry_resolve_lineage.
Table/View also join the registry-only per-file LSP surface labels so a
table rename invalidates dependent SQL files on incremental (no stale
lineage edges), rank with the type tier in BM25 search, and appear in
the architecture boundary/package/cluster queries via the pinned
CBM_SQL_RELATION_LABELS fragment.
Tests: extraction trio (labels, lineage usages, schema-qualified names),
grammar golden + probe updates, relation-label contract pin, and two
pipeline tests: cross-language isolation (binding: fails without the
veto) and incremental table-rename stale-lineage.
Closes#574.
Co-authored-by: alexisperinger-ux <alexis.peringer@iss-stoxx.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#1545 fixed the defect that let concurrent instances quarantine each other's
healthy databases, but shipped without tests — and this is a data-loss path, so
it gets coverage before it ships.
Three cases, one per verdict, chosen so a regression in either direction fails:
healthy -> OK (the baseline the other two are measured against)
bad root_path-> CORRUPT (a verdict that never says CORRUPT would protect
broken databases instead of users)
no handle -> TRANSIENT (a file we could not open tells us NOTHING about
its contents; answering CORRUPT there is how a
database nobody had read got renamed and rebuilt)
Co-Authored-By: LynxBay <LynxBay@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The routing follow-through on the +92% warm-reindex finding: a semantic
manifest delta no longer unconditionally rebuilds the world. The planner
recomputes exactly the changed files plus the recorded consumers of any
changed SURFACE, and the executor resolves them against cross registries
rehydrated from the persisted per-file surfaces -- the same registration
code a full build feeds from fresh parses, which is what makes the output
converge instead of drift.
Routing, in order: exact manifest match stays a no-op; a delta first
offers itself to the closure planner; every uncertain case declines to
the full rebuild that was yesterday's only behaviour. Declines: virtual/
config manifest entries, new files, ADDED definition names (yesterday's
graph cannot know who would resolve to a name that did not exist -- the
write-the-caller-first flow and shadowing both live here), missing or
undecodable surface rows, dependents outside discovery, and a budget of
30% of files with an 8-file floor (a percentage alone starves small
repos: 1 changed file in 3 is 33%).
Two structural facts carry the correctness argument. Per-file extraction
is a pure function of file content, so an unchanged dependent can never
be surface-changed in turn -- the closure is depth-1 by construction, no
fixpoint. And a body edit reserializes to the identical surface bytes,
so its closure is the file itself. The dependent set comes from one
indexed query over the previous generation's edges (structural
Folder/Project containment excluded -- a container is not a consumer).
The executor is the existing partial machinery, parameterized: re-parse
list = closure; inbound-edge snapshot/re-link keeps only sources OUTSIDE
the closure (sound because every referencer of a surface-changed file is
inside it by construction); cbm_parallel_resolve now receives real cross
registries built from stored-surface defs plus this run's fresh parses;
publication merges surviving surface rows with the re-parsed files'
fresh ones inside the same generation. The legacy test-only route
publishes no surface rows at all -- a stale row that satisfies a future
closure plan with yesterday's surface would be worse than the full
rebuild an empty table forces.
Tests pin route AND convergence together (route equality matters because
a full rebuild satisfies any convergence assertion vacuously): body edit
routes CLOSURE_REPAIR with node/edge/CALL_REFERENCE counts equal to a
fresh full index; REMOVING a definition keeps the closure route and
drops the dependent's stale CALL_REFERENCE -- the assertion the legacy
QN-keyed re-link could never pass; added-name, new-file and budget cases
decline; the existing Go content-change test now routes CLOSURE_REPAIR
with its convergence assertions unchanged, making it the Go-language
proof of the same machinery.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
First piece of the closure-repair incremental route (the follow-through
on the +92% warm-reindex finding): a per-file lsp_surface row holding
the file's serialized cross-file definition set -- exactly what
pass_lsp_cross registration consumes -- plus the metadata the routing
decision needs: the surface sha (early-cutoff key: a body edit leaves it
unchanged, so no dependent recomputation is owed), a referenced-name
bloom (added-symbol trigger), and a governing-config context hash.
The store treats defs_json and the bloom as opaque; the codec lives with
pass_lsp_cross, which is their only writer and reader. A project with no
rows reads back as OK/0 -- callers treat that as "no surface data" and
route to a full rebuild, which is also how databases written before this
table existed upgrade themselves.
Table appears via the CREATE IF NOT EXISTS schema on store open, so the
raw dump writer needs no change: publication opens the staging DB with
the store right after the dump, which applies the schema.
Round-trip covered in store_nodes: batch upsert, ordering, binary bloom
with embedded NUL, NULL bloom, whole-row conflict replacement including
bloom removal, project-scoped delete, and the empty-project signal.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm.h documents cbm_label_is_type_like() as the single source of truth for
type-like labels, "so adding a new type-like label (e.g. Struct for
Rust/Go/Swift/D structs) updates them all at once instead of scattering
|| strcmp(label,"Struct")==0 across the tree".
A SQL string literal cannot call it. Four queries in store.c and the BM25
ranking in mcp.c carried their own hardcoded label lists, so they silently
opted out of that contract and stopped matching the moment Struct, Interface,
Enum, Type and Trait began being emitted:
store.c:5279 arch_boundaries ('Function','Method','Class')
store.c:5422 arch_packages_from_qn ('Function','Method','Class')
store.c:6689 arch_clusters ('Function','Method','Class')
store.c:7742 cbm_store_vector_search ('Function','Method','Class')
mcp.c:2759 BM25 ranking ('Class','Interface','Type','Enum')
Effect on main today: get_architecture (boundaries, packages, clusters) and
vector search drop every Struct, Interface, Enum, Type and Trait node in the
project, and search_code under-ranks structs -- for Rust, Go, Swift and D,
where struct is the primary type declaration. The BM25 list had drifted
differently again, omitting Struct and Trait but not Interface.
The fix is not a wider literal. CBM_SQL_TYPE_LIKE_LABELS and
CBM_SQL_CALLABLE_OR_TYPE_LABELS live next to each other in constants.h and are
pinned to cbm_label_is_type_like() by a test that checks both directions:
every label the C predicate accepts appears in the SQL fragment, and nothing it
rejects is smuggled in. Adding a type-like label without updating the SQL now
fails that test instead of quietly shrinking query results.
No new labels are introduced. Every label added here is one the extractors
already emit.
Reported by @win4r in #568, whose audit found four of the five sites. Their
patch widened the literals in place and added Actor, a label main does not
emit; this routes through the canonical set instead and also covers Interface,
Type and Trait, which that patch still omitted.
Co-Authored-By: win4r <win4r@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the #963 coverage taxonomy: alongside the best-effort FAILURE
signals (parse_partial, skipped), the report now lists what was
deliberately NOT indexed — a deterministic, by-design class, never mixed
into the failure surfaces.
- discover: file skips are classified by reason (gitignore / cbmignore /
skip-list / ignored-suffix / fast-pattern / size-cap) and collected via
the new cbm_discover_ex2 (per-file entries capped at 2000 with the
uncapped total reported — truncation is explicit, never silent; whole
excluded subtrees stay exhaustive as before). Unsupported-extension
files are deliberately NOT recorded: no grammar for an extension is
not an ignore decision, and listing every README/asset would drown
the signal
- pipeline: ignored files captured per run (full + incremental) and
persisted into the index_coverage table as not_indexed_file rows,
excluded dirs as not_indexed_dir; incremental runs rewrite the
by-design set fresh from discovery instead of carrying old rows
- store: the deleted-file prune exempts not_indexed_* (deliberately
unindexed paths never have hash rows); the missed shadow graph keeps
showing FAILURES only — the UI's report-an-edge-case callout can
never fire for gitignored paths, and a project whose only coverage
rows are by-design gets no miss graph at all
- mcp: index_repository response gains not_indexed_files {files, count,
truncated, note} (dirs stay under excluded); index_status's coverage
report gains a not_indexed section (dirs + files + note) and no longer
lumps by-design rows into skipped; tool descriptions updated — the
by-design class is presented as deliberate and deterministic (change
the ignore rules and re-index), with no best-effort hedging
- tests: e2e guard (gitignored file + dir reported with reasons on both
surfaces, failure surfaces stay clean, missed graph excludes them,
rows survive an incremental re-index, ignored constructs genuinely
absent from the graph); store unit test covers the prune exemption
and shadow-graph exclusion
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Four refinements to the coverage signal:
- extraction: subtract DEFINITE recovery before flagging. Tree-sitter
error recovery plus the ERROR-descending def walker often re-extract
constructs inside a failed region (verified: `def broken(:` comes
back as a def); a region whose every line is covered by definitions
that START inside it is not a miss, and a fully recovered file is not
flagged at all. Container defs (Module/Package) don't count as
evidence, and partially covered regions stay flagged — the
#ifdef-split case keeps its flag because the first branch's function
is genuinely lost
- naming: the query_graph option is graph="missed" (the graph shows
ONLY misses — "coverage" was misleading); shadow project renamed to
"<name>::missed"; tool descriptions updated so agents discover both
the option and its semantics
- hook: the CLI-installed PreToolUse augmenter now also matches Read
and injects a coverage note when the file being read is listed as not
fully indexed ("line ranges X-Y could not be parsed — the file
content you are reading is ground truth"). Safe against the old
issue-362 hazard: the augmenter is structurally non-blocking (always
exit 0, additionalContext only), mirroring the Gemini matcher that
already includes read_file; matcher upgrade bookkeeping updated
- ui: "Missed files" toggle in the graph sidebar renders the miss
graph as a second graph option — /api/layout gains graph=missed
(same db file, shadow-project scoping; base project name validated
as before)
Tests: recovery-subtraction cases (recovered def unflagged, garbage
region flagged, trailing recovered defs keep the flag), CLI matcher
tests updated, e2e resilience fixtures switched to an unrecovered
miss; verified end-to-end: hook emits the note for a flagged file and
stays silent otherwise, /api/layout?graph=missed serves the miss graph
and the code graph is unchanged, frontend build + tests green.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Persist the parse-coverage signal outside the graph and make the misses
queryable both as a list and as a graph.
- store: new index_coverage table (project, rel_path, kind, detail) —
coverage is metadata ABOUT the graph and never mixes into the graph
tables; replace/get/free APIs with a deleted-file prune keyed off
file_hashes (the live-file set)
- store: materialize a derived miss graph under the shadow project
"<project>::coverage" (Project -> Folder chain -> File{kind,detail},
CONTAINS_FOLDER/CONTAINS_FILE edges), rebuilt from the table inside
the same transaction on every replace; invisible to list_projects
(which scans the cache directory, not the projects table)
- pipeline: coverage is written in the same persist step as every other
post-dump SQL write (hashes, ADR, FTS). Full runs replace the whole
set; incremental runs merge rows surviving from files not re-extracted
with this run's fresh entries, so a fixed file's flag clears and
deleted files prune automatically
- mcp: new get_index_coverage tool (per-project parse_partial + skipped
lists with the best-effort note); query_graph gains graph="coverage"
to run the same cypher against the miss graph; the File-node property
marker from the previous commit is dropped in favor of the separate
table — the code graph gains no coverage rows
- wording: constructs in flagged ranges MAY be missing — tree-sitter
error recovery still salvages some (verified: a function inside a
flagged #ifdef region and even a broken def were still extracted)
- tests: store round-trip/prune/shadow-graph unit test; e2e asserts the
response fields, the advertised coverage-graph query, code-graph
purity, and that the flag clears after the file is fixed (incremental
route)
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
dump_and_persist_hashes upserted file hashes one statement per file --
~89k autocommit transactions (one fsync each) on the linux kernel. Switch to
the existing cbm_store_upsert_file_hash_batch (same cached upsert inside one
begin/commit, rollback on error): identical tuples and replace semantics, one
fsync. OOM fallback to the per-file path retained.
Also instrument the whole post-dump tail with CBM_PROFILE spans
(persist/1_reopen .. 6_artifact_export) -- this stretch was ~60 s unattributed
on kernel-scale runs; the FTS5 backfill over all node names is now measurable
before any optimization decision.
New deterministic guard: store_file_hash_batch_equals_loop asserts the batched
path produces byte-identical file_hashes rows to the per-file loop. Suites:
pipeline 216, store_nodes 56, store_edges 25, store_search 65, store_arch 52,
store_bulk 3, store_pragmas 7, store_checkpoint 1, incremental 161 -- zero
failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The projects.root_path integrity check only accepted '/' or an uppercase
'A'-'Z' first character. On Windows, drive letters are commonly lowercase
(c:/repo, y:/share), so such a path was flagged store.corrupt and the DB
was auto-deleted on open — a likely cause of the mapped-drive DB deletion
in #227/#367. Accept 'a'-'z' as well.
Relates to #227, #367.
cbm_pipeline_run_incremental was destructively overwriting a project's
graph when the index mode was downgraded. find_deleted_files classified
ANY stored file absent from the current discovery list as deleted and
purged its nodes via cbm_gbuf_delete_by_file. In a full → fast mode
downgrade, directories matching FAST_SKIP_DIRS (tools, scripts, bin,
build, docs, __tests__, migrations, ...) are excluded from discovery
but still exist on disk — so they were getting silently destroyed.
Real-world incident, 2026-04-13: the Skyline graph collapsed from 2819
nodes (full-mode nightly) to 2303 mid-day when a concurrent fast-mode
reindex obliterated packages/mcp/src/tools/ (~500 nodes / 18 files).
Fix:
- find_deleted_files now stat()s each stored-but-missing file. Files
that exist on disk are mode-skipped (nodes AND hash rows preserved);
only ENOENT/ENOTDIR results purge. Other errnos (EACCES, EIO, ELOOP,
transient NFS, ...) preserve. NULL repo_path preserves. snprintf
truncation preserves. strdup OOM mid-record drops the entry rather
than persist a NULL rel_path that would silently fail upsert. stat()
not lstat() is intentional — the indexer follows symlinks during
discovery so a stale symlink has no source to parse.
- Mode-skipped hash rows are threaded through dump_and_persist →
persist_hashes and re-upserted into the rebuilt DB. Without this,
the next reindex's stored hashes wouldn't include the file → the
deletion-detection contract breaks → orphaned graph nodes survive
forever once the file is eventually deleted.
- persist_hashes now checks the upsert return code on both the
current-files loop and the new mode_skipped loop and logs a warning
with scope/rel_path/rc on failure. Partial-failure policy: a single
bad row no longer silently drops, but does not abort the whole
reindex either — partial preservation beats total loss.
Tests:
- incremental_fast_preserves_mode_skipped_tools_dir walks the failure
end-to-end across four phases: full index, fast reindex preserves
the tools dir, mutate-and-fast-reindex exercises dump_and_persist's
hash carry (NOT the noop fast-path that would hide bugs there),
then delete-on-disk full reindex confirms the deletion-detection
contract still holds.
- store_file_hash_upsert_rejects_null_required_fields pins the API
contract that cbm_store_upsert_file_hash returns CBM_STORE_ERR on
NULL required fields, so a future schema change relaxing any NOT
NULL is caught at the producer layer rather than letting the new
warning go silent.
Conflict resolution against current main: the dump_and_persist
signature now takes both mode_skipped/mode_skipped_count (this PR) and
repo_path (added on main for cbm_artifact_export); the call site
passes cbm_pipeline_repo_path(p) alongside the mode_skipped pair.
Full suite: 2837 passed, 0 failed under ASan + UBSan + LeakSanitizer.
Closes#251.
Co-authored-by: Peter Cox <info@petercox.ie>
Add cbm_store_check_integrity() that validates:
- projects table has <= 5 rows (1 project per .db file)
- root_path starts with / or drive letter (not numeric garbage)
Wire into resolve_store (MCP queries) and pipeline routing:
- MCP: corrupt DB auto-deleted with ERROR log, returns "not indexed"
- Pipeline: corrupt DB deleted, falls through to full reindex
Root cause: user-reported corrupt DBs had 48K rows in projects table
with node IDs in text columns. Exact write path TBD, but detection
+ auto-clean prevents the symptom from persisting.
5 unit tests: clean DB, empty DB, bad path, too many rows, NULL.
- Arena-allocate pair_field_types in cpp_stdlib_data.c (was stack array
escaping scope via registry pointer — caught by GCC ASan)
- Add sysfs fallback for L2 cache on Linux ARM (sysconf returns 0)
- Expose cbm_node_free_fields() and fix 12+ leak sites in tests where
find_node_by_id/qn results were never freed (caught by LeakSanitizer)
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
Martin Vogel
kavish-19
Martin Vogel
Peter Cox