Reworks the ADR-loss-across-reindex fix to Alternative 1 (per the PR review):
remove the race rather than narrow it. ADRs live in a per-project sidecar SQLite
DB ("<graph_db>.adr.db"), separate from the graph DB a reindex rebuilds or
deletes, so there is no capture, re-read, re-apply, or lock — and no sleeps,
polling, or timeout deciding an outcome.
store.c:
- ADR writes (store/delete/update_sections) use a cached read-write handle opened
SQLITE_OPEN_READWRITE|CREATE and configured like the graph store (PRAGMA
busy_timeout=10000 + journal_mode=WAL) so cross-process access waits rather
than failing SQLITE_BUSY. A :memory: store keeps ADRs in the graph DB.
- Reads never write: cbm_store_adr_get uses the cached RW handle if open, else
opens the sidecar SQLITE_OPEN_READONLY (with the same busy timeout) when the
file exists, else reads the legacy graph row without creating anything. When
the sidecar exists but is NOT marked migrated (half-created), it falls back to
the legacy graph row so a partial sidecar never hides an ADR.
- One-time migration is atomic: adr_copy_legacy_once wraps the marker check,
legacy copy, and marker insert in a single BEGIN IMMEDIATE...COMMIT
(re-checking the marker under the lock; ROLLBACK->ERR on failure). New public
cbm_store_adr_migrate_once. Sidecar path via cbm_path_for_file_api (Windows
long-path). Dropped the write-only adr_db_path field.
pipeline.c: migrate a legacy ADR into the sidecar BEFORE the format-change and
forced-full routes unlink the old generation; preserve-and-abort on failure, and
do NOT fail open — a query-open returning NULL preserves unless the final DB
genuinely does not exist. Migration targets the FINAL path (staging suffix
stripped). The early-capture/re-apply machinery and its test hook are gone.
artifact.c: the exported artifact carries the ADR — snapshot_inject_adr always
DELETEs project_summaries first, then injects only when an ADR is present, so a
deleted ADR never resurrects through a lingering legacy row. Import restores the
ADR into the destination sidecar via cbm_store_adr_restore_from_graph_if_absent:
it copies the artifact's ADR (INSERT OR IGNORE — never clobbers a local one)
regardless of the migrated marker, so a teammate who already indexed the project
(marked sidecar, no ADR) still receives the shared decision record. store.c
factors the row copy into adr_copy_rows, shared by the one-time legacy migration
and the ungated restore.
mcp.c: delete_project removes "<db>.adr.db" (+ -wal/-shm/-journal) only after the
graph unlink succeeds; is_project_db_file skips *.adr.db. cli.c: exclude
*.adr.db from index listing/counting. README + memory-core baselines updated.
Tests (file-based): ADR written mid-reindex survives on BOTH the full and delta
routes (route-pinned FORCED_FULL / CLOSURE_REPAIR); a legacy row migrates before
a forced-full/format reindex with no prior read; migration failure (a directory
at <db>.adr.db) preserves-and-aborts with the old generation intact;
delete_project via the MCP handler removes the sidecar + journals; the artifact
export->import round trip keeps the ADR; a deleted ADR does not resurrect on
import; an import does not clobber a local ADR; an import into an already-indexed
destination (marked sidecar, no ADR) restores the artifact's ADR; a half-created
sidecar falls back to the legacy row. The import tests remove the destination
graph DB (keeping the sidecar + marker) before importing, matching production
(try_artifact_bootstrap imports only when the graph DB is missing) so the rename
succeeds on Windows, not only on POSIX.
Closes#516
Signed-off-by: 0xb007ab1e <134006168+0xb007ab1e@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The three exported release functions added for #1762 freed raw heap
blocks. Since the memory-core ratchet landed, the allocating side has to
go through the core as well, and on current main cbm_leiden already
allocates its result in CBM_MEM_CLASS_OTHER, so a raw free() in
cbm_leiden_free left the class counters drifting.
- cbm_store_fetch_call_edges: allocate and grow the source/target
arrays with cbm_alloc/cbm_realloc in CBM_MEM_CLASS_STORE. An
allocation failure now finalizes the statement, releases both arrays
and returns CBM_STORE_ERR instead of writing through a NULL array.
- cbm_store_free_call_edges, cbm_leiden_free and cbm_layout_free_json
release with cbm_free in the class the matching allocation used.
- cbm_layout_to_json writes its output through cbm_json_writer_allocator,
so the string is an OTHER-class core block; the UI server's five
release sites now call cbm_layout_free_json.
- The header comments name the paired free instead of "caller frees".
- Tests: ownership checks for all three pairs measure the class counters
across the allocating call and the paired free, so a raw allocation on
either side fails; the call-edge check grows past the initial capacity.
Refs #1762
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings the branch up to main at a2227157. Two conflicts, both resolved by
keeping both sides:
- src/ui/layout3d.h: the new cbm_layout_free_json declaration sits
before main's region-level layout section.
- tests/test_store_arch.c: the Leiden/Louvain tests release their
results through the new cbm_leiden_free; main's
leiden_result_memory_core_ownership test is kept.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three query rewrites over the existing indexes; every MCP response stays
byte-identical to c9cacdf0 (tools + Cypher suites on django, redis and the
linux kernel, same index each).
- arch_packages: GROUP BY +n.name. With a bare column the planner walked
idx_nodes_name over every node to get groups in name order, with a table
lookup per node to test the label (6.3 s of the 7.3 s get_architecture on
the 8.5M-node kernel graph, for 5 Package nodes). The unary plus seeks
idx_nodes_label instead; the name tie-break is explicit because a sorted
GROUP BY follows the ORDER BY direction and reversed tied counts.
- get_schema_impl: label / edge-type counts step through the distinct values
(MIN(col) > previous) and count each value's index range instead of a
GROUP BY scan: kernel 0.26 -> 0.16 s and 0.54 -> 0.32 s. Values reach
ORDER BY in the same ascending order, so ties keep their order.
- cbm_store_search: an anchored name_pattern ("^schedule$", "^kmalloc")
adds idx_nodes_name ranges for its literal prefix (every ASCII letter case
of the first three letters under iregexp, which runs in the C locale),
one UNION ALL arm per range merged by ORDER BY name, id without a sort;
the count sums the ranges. LIKE hints and the regex stay in the WHERE, so
the rows are those of the plain scan. Patterns that do not compile keep
the plain query (and its error).
Kernel, interleaved medians n=5: get_architecture 9.2-10.3 s -> 1.03 s,
get_graph_schema 0.76 s -> 0.48 s, cycles 2.24 s -> 2.04 s, search_graph
^schedule$ 1.15 s -> 14 ms, ^kmalloc + label 0.64 s -> 14 ms. Peak daemon
RSS for get_architecture +67 MB -> +22 MB.
The ranged SQL is built with a bounds-checked append (no snprintf-derived sizes) into CBM_MEM_CLASS_STORE buffers; a build that does not fit rolls its binds back and the plain query runs.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Doc -> code candidates now carry a p judged for their kind, their document
format and where the document sits.
Format (sample 6, 234 blind-judged candidates): reStructuredText, AsciiDoc and
PDF sections each get their own curve (reST stored from tfidf 0.30, AsciiDoc
from 0.20, PDF not measured and not stored). A candidate that repeats an
exact MENTIONS link of the same section is not stored.
Position (sample 7, 490 blind-judged candidates from four monorepos): every
document gets a home folder -- its folder; a docs folder documents its
parent; a folder without code hands over to the nearest one with code; the
repository root means the whole project. Markdown candidates outside the
document's home were right 6 of 90 and are never stored; inside it they are
stored from tfidf 0.30 (0.33 / 0.53); for documents of the whole project
samples 4 and 7 pooled give 0.22 / 0.51 from 0.30.
New kinds, all in doc_link_candidates (evidence gains "kind" and
"position"; no new edge type, no schema change):
- local: up to two home functions ranked below the top five (0.32);
- file: a whole file, its vector the sum of its functions' terms and its
definitions' names, config keys included, cut to 64 terms (0.30, from
tfidf 0.40);
- folder: a README or index document -> its home folder (0.87).
get_code_snippet on a section lists every kind with its label;
get_file_outline shows the sections about the file or a folder holding it
(cbm_store_doc_candidates_for_path).
cbm_sem_p_2dp rounded 0.325 and 0.525 down (their float values sit 1.2e-6
below the half); its epsilon is 1e-4 on the x100 scale.
Replayed on the sample repositories, the build stores exactly the rows the
judged curves select (950 rows, 0 differences).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A documentation section that explains a function rarely names it, so
the deterministic doc links cannot connect them. The semantic pass now
ranks, for every Markdown section that can be about code, the functions
whose TF-IDF terms best match the section's heading and body, over the
functions' own corpus (function-pair scores are untouched):
- gate: no candidates for release-number or project-meta headings
(changelog, licence, contributing, links, contents, install, ...),
bodies under 20 words, or bodies that are mostly links;
- targets: functions and methods, never tests (by path or name);
- stored: a section's best five with tfidf >= 0.20, each with p, the
probability that the section is about the function, from 274
blind-judged candidates in five held-out repositories (three reading
rounds): 0.36 below 0.30, 0.42 below 0.40, else 0.55. Below 0.20 the
judged share was 0.06, so nothing is stored there.
These are candidates for an agent to verify, not graph edges: at best
about half are right, far below the bar for a link. They live in a side
table, doc_link_candidates (section and function qualified names, rank,
score, p, evidence), created when a generation is published, so older
indexes read as "no candidates". Full and legacy-partial reindexes
replace a project's rows; the delta route keeps the previous rows (its
proxy buffer has no section text or function tokens to recompute
them), and a row whose node is gone is skipped when read.
get_code_snippet shows them: on a Section, "possibly_about" (its
candidate functions); on a function or method, "possibly_described_in"
(the sections), at most five, highest p then score first, with a
candidates_note saying what they are.
Measured on fastapi (6,144 gated sections): 24,293 rows, identical to
the generator's dump; index time within noise; the database grows by
the table (37 MB there, with a 148-character project prefix on every
qualified name).
Tests: semantic:sem_doc_calibrated_p_bands,
store_arch:doc_candidates_replace_and_get (RED when a replace keeps old
rows), pipeline:pipeline_doc_candidates_published_and_kept_by_delta
(RED with test functions admitted, the heading gate off, the publish
write skipped, or the delta route replacing rows),
mcp:snippet_doc_candidates_both_directions (RED with stale rows shown).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_store_fetch_call_edges, cbm_leiden/cbm_louvain, and
cbm_layout_to_json document caller-owned memory ("caller frees
both" / "the caller frees it" / "Caller must free()") but exported
no matching free function, unlike the rest of the API
(cbm_store_find_nodes -> cbm_store_free_nodes,
cbm_layout_compute -> cbm_layout_free, etc.). From C a bare free()
works via static linking, but a caller across an FFI boundary (e.g.
C# P/Invoke) has no way to reach the DLL's internal free() unless
it is exported by name.
Adds three paired free functions following the existing pattern:
- cbm_store_free_call_edges(src, tgt)
- cbm_leiden_free(result)
- cbm_layout_free_json(json)
Updates the one internal production caller (mcp.c SCC analysis) to
use cbm_store_free_call_edges instead of a bare free(), and the
existing unit tests for cbm_leiden/cbm_louvain/cbm_layout_to_json
to use the new paired frees. Adds fetch_call_edges_basic,
fetch_call_edges_truncated, and fetch_call_edges_empty_project to
test_store_arch.c, covering cbm_store_fetch_call_edges /
cbm_store_free_call_edges, which had no prior test coverage.
Fixes#1762
Signed-off-by: Jumaga2015 <jumaga2015@gmail.com>
Every ADR fixture in the tree used \n, so nothing exercised a CRLF-stored
document — and CI could not have found this, because the fixtures are LF
whichever platform runs them. A red would only have appeared once a real
user's CRLF ADR met this code.
The exposure is specific to splicing. The old rebuild normalised everything
on the way out, so line-ending confusion was invisible; a byte-span
replacement can miscount a separator run, and byte-identity is the property
the splice design exists to provide.
Two of the three paths were already correct, and now say so in tests:
- Replacing a section preserves CRLF exactly. The walk-back over a trailing
break run steps across both '\r' and '\n', so it cannot cut mid-pair.
- Heading matching is ending-agnostic: "## PURPOSE\r\n" locates the same as
"## PURPOSE\n", because the scanner trims '\r' from the name. Trimming and
matching were different claims and only trimming was pinned.
Appending a new section was wrong. It counted trailing '\n' characters, so a
document ending "...Foo\r\n" was read as having one break but then had a
BARE '\n' appended, and the "## NAME" line it wrote used '\n' too — leaving a
mixed document that neither the author nor the tool asked for. A CRLF file
ending in a blank line also gained a spurious extra break.
Appends now count a "\r\n" pair as one break, and write the separator and
heading line using the document's own ending.
The contract, now that a document can be mixed:
- Bytes that already exist are never rewritten, so each line keeps whatever
ending it had. This follows from splicing and is what the acceptance
property already guaranteed.
- Text this code writes itself — the separator and the "## NAME" line —
follows the document's majority ending, LF on a tie.
- A caller's body is inserted verbatim. Rewriting bytes a caller supplied
would be the same silent modification this whole change set removes.
Co-authored-by: andis777 <24672074+andis777@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
manage_adr mode='set_sections' rebuilt the ADR from cbm_adr_parse_sections()
and cbm_adr_render(). That model is lossy, so the rebuild silently rewrote
text nobody asked to change. Measured on the previous commit:
- Everything before the first recognised heading was dropped.
- A mis-cased '## Purpose' was not a heading, so when it came first its
whole block was dropped with the preamble — an ordinary typo cost the
author a section.
- Sections stored out of canonical order were reordered.
mode='update' never had this exposure because it writes the caller's bytes
verbatim; the section merge introduced it.
The merge now locates the target heading's byte span and replaces only that
span, appending a block when the heading is absent. Bytes outside the span
are never reconstructed, so a preamble, a code fence, an unrecognised
heading and the author's section order all survive byte-for-byte. That
property is the acceptance test: every document in
adr_splice_preserves_untouched_bytes is a case the rebuild corrupted.
Arbitrary headings now work, which is what an "add an entry" mode needs.
The canonical six become a convention rather than a privilege: they are
still what ADR_EMPTY_HINT suggests, but they get no ordering priority and
none is required. Under splicing, ordering is whatever the document says,
and any privileged reordering would reintroduce the data loss above.
Names match exactly, including case, so '## Purpose' and '## PURPOSE' are
different sections. Folding them would silently merge two blocks the
author chose to keep apart — the same class of loss this commit removes.
cbm_adr_validate_section_keys() no longer enforces the canonical set. It
now rejects only names that could not round-trip through a '## NAME' line
(empty, '#'-leading, newline-bearing, edge-whitespace, over-long), because
such a name scans back as a different heading or as none, and a second
identical write would append a duplicate instead of being a no-op.
A heading inside a fenced code block is a code sample, not a section. An
unterminated fence hides every heading after it, so a write would append a
duplicate rather than replace: that is refused explicitly, and mode='sections'
reports the same ambiguity instead of answering with a partial list.
manage_adr mode='sections' now uses cbm_adr_scan_headings(), the same
classifier the write path splices with. It used to list any '#'-prefixed
line, so a '## Foo' in prose or inside a fence was reported as a section no
write could target. Two components disagreeing about what a section is was
how a section write came to be able to destroy one.
cbm_adr_parse_sections() is unchanged, so documents parse exactly as before.
TEST(adr_parse_sections_non_canonical) still passes untouched.
Co-authored-by: andis777 <24672074+andis777@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The absolute 8s bound failed on the shared ubuntu-arm UBSan runner, which measured 25.9s for work that takes 207ms locally (125x spread). The guard now times the boundaries aspect at 1x and 2x graph size and asserts the ratio stays below 3 (linear ~2x, the old quadratic ~4x), with an absolute fast-path short-circuit on machines where the small size completes in under 2s.
get_architecture's arch_boundaries looked up the package of every CALLS edge endpoint with a linear scan over all Function/Method/Class nodes (lookup_pkg over parallel arrays) — O(edges * nodes). On the Linux kernel graph (~1.4M defs, ~1.4M CALLS edges) the boundaries aspect spun >10 minutes at 100% CPU; latent until C call extraction was fixed (the kernel previously produced almost no CALLS edges). The node query now orders by id and lookup_pkg binary-searches: 80k nodes/160k edges 16.6s -> 207ms in the regression test; kernel-graph get_architecture(all) >10min -> 23s. Reproduce-first: arch_boundaries_no_quadratic_scan (red 16566ms pre-fix vs 8000ms bound, green 207ms post-fix).
Wire the Leiden detector into get_architecture via a new 'clusters' aspect.
arch_clusters loads Function/Method/Class nodes + CALLS edges (capped for very
large graphs), runs cbm_leiden, and reports the top-N communities compactly —
each with a dominant-package label, member count, cohesion (internal vs boundary
edge ratio), representative top nodes (by degree), and the packages spanned.
Singletons are skipped to avoid noise. The mcp.c serializer already emitted the
clusters array; it is now populated.
Tests: arch_clusters_basic (two cliques + a bridge -> >=2 connected clusters).
Generous smoke coverage added for the full Cypher surface (labels/type/id/keys/
properties/size/reverse/replace/left/coalesce/substring, NOT EXISTS dead-code,
CASE, unsupported-function error) plus the get_architecture clusters aspect.
The previous community detection ran a single local-moving pass with no
aggregation or refinement, so it fragmented graphs into hundreds of tiny,
sometimes internally-disconnected clusters (e.g. 275 on a mid-size repo).
Implement the Leiden algorithm (Traag, Waltman & van Eck 2019,
arXiv:1810.08473): local moving (fast, queue-based) -> refinement -> graph
aggregation, repeated until the partition can no longer be coarsened. The
refinement phase re-derives each community bottom-up from singletons, merging
only along edges, which guarantees every reported community is internally
connected. Aggregation seeds each level from the prior move-phase partition so
coarse structure carries over. A resolution parameter controls granularity.
Internals use a CSR graph (cbm_lg_t); self-loops are folded into the weighted
degree rather than materialised, so the modularity gain uses degree for the
null model and the adjacency for connectivity. cbm_louvain is preserved as a
thin wrapper over cbm_leiden(resolution=1.0) for API compatibility.
Tests: multi-level collapse (32-node clique chain -> few connected
communities), resolution control (path graph granularity), and a connectivity
checker proving the refinement guarantee; existing Louvain tests still pass.
Closes#57
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
Martin Vogel
0xb007ab1eandClaude Opus 4.8
Jumaga2015
SEPURI-SAI-KRISHNA
Berserk Agent
Martin Vogel