sqlite_writer.c computed page offsets as (long)(page - 1) * 65536 and
passed them to fseek()/ftell(). long is 32 bits on Windows, so once a
database grows past 2 GiB the offset goes negative and fseek() fails;
the result is ignored, so pages land wherever the stream happens to be.
Past 4 GiB the offset wraps back to small positive values and the
writer overwrites the start of the file. The staging database stops
at exactly 4 GiB, is corrupt, and the run ends in persist_failed.
Route every seek/tell in the writer through 64-bit wrappers
(_fseeki64/_ftelli64 on Windows, fseeko/ftello elsewhere) and compute
offsets with a new cbm_writer_page_offset() helper that widens before
multiplying.
Fixes#2399
Signed-off-by: lixsp11 <30531760+Lixsp11@users.noreply.github.com>
Distills PR #770 (graph-buffer dedup key) and completes the edge-uniqueness
contract it left partial: the contract lives in THREE places, and changing
only the buffer ships databases that violate their own UNIQUE constraint
(PRAGMA integrity_check: non-unique entry in sqlite_autoindex_edges_1).
A single 'import { A, B } from ./lib' produced ONE IMPORTS edge: the graph
buffer dedups edges on (source_id, target_id, type) and merge-replaces
properties on collision, so the second symbol silently overwrote the first.
pass_calls.c, pass_usages.c, pass_semantic.c and pass_lsp_cross.c parse one
local_name per IMPORTS edge for cross-file resolution, so the dropped
symbol's calls failed to resolve too — not just "who imports X" queries.
Changes, kept in sync across all three sites:
- graph_buffer.c (from #770): make_edge_key() folds local_name into the
dedup key for IMPORTS edges only; all three key call-sites updated.
Hardened beyond #770: EDGE_KEY_BUF bumped to 256 and oversized
local_names are re-keyed with an FNV-1a hash of the full name instead
of being silently truncated (two long names sharing a prefix must not
collide back into one edge).
- store.c: edges gains local_name_gen, a VIRTUAL generated column
(IMPORTS -> coalesce(json_extract(properties,'$.local_name'),''),
else '' — NOT NULL because NULLs never conflict in a UNIQUE index);
uniqueness widened to UNIQUE(source_id, target_id, type,
local_name_gen) and the insert upsert's conflict target matches.
init_schema probes pre-#768 DBs (no local_name_gen) and fails the
open: SQLite cannot ALTER a table constraint in place, and an
unopenable DB already takes the existing repair path — full index
deletes + rebuilds, artifact import refuses and falls back to a
reindex. Read-only query opens skip init_schema and keep working.
- sqlite_writer.c/.h: dump DDL matches the widened schema; the hand-built
sqlite_autoindex_edges_1 comparator and entry builder include the
local_name column; CBMDumpEdge carries local_name extracted via real
JSON parsing (yyjson) so index entries match json_extract's unescaped
values exactly, per the idx_edges_url_path precedent.
- artifact.h: CBM_ARTIFACT_SCHEMA_VERSION 1 -> 2 so old binaries refuse
artifacts carrying the widened schema (their 3-column conflict target
can no longer prepare against it).
Tests (reproduce-first, all red on the unfixed code):
- gbuf tier: multi-symbol import -> 2 IMPORTS edges; long-local_name
truncation guard.
- store tier: distinct local_name coexists as 2 rows, same local_name
still upserts, non-IMPORTS dedup unchanged.
- writer tier: dumped DB passes integrity_check with 2 sibling imports
and exposes matching local_name_gen values.
- end-to-end: TS fixture through the real pipeline -> 2 queryable
IMPORTS edges AND integrity_check ok; with only the buffer half of
the fix this test still fails on integrity_check, proving the schema
half is required.
Closes#768.
Co-authored-by: Alexandros Pappas <11921291+apappas1129@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Split cbm_write_db's monolithic build into an incremental API:
cbm_writer_open / cbm_writer_append_nodes / cbm_writer_finalize. Node
rows are written to the table B-tree across append calls via a
persistent page builder, so callers can flush and free node batches
(including the heavy properties column) without materializing every
record at once. Everything after the nodes table — edges, vectors,
metadata, indexes, and the sqlite_master page — is emitted at finalize,
as before.
cbm_write_db is now a one-shot wrapper (open, append all nodes,
finalize) producing byte-identical output; the full test suite and
PRAGMA integrity_check confirm the on-disk format is unchanged.
Major quality improvement: replace random index vectors with pre-trained
Nomic nomic-embed-code token embeddings. Tokens like 'error' and 'exception' now start
with similar vectors (learned from millions of code repos) instead of
arbitrary random projections. Co-occurrence enrichment adds project-specific
context on top.
Architecture:
- vendored/nomic/code_vectors.bin: 37.7MB raw int8 vectors
- vendored/nomic/code_vectors_blob.S: assembler .incbin (instant build)
- vendored/nomic/code_vectors.h: extern declarations + pretrained_vec_at()
- vendored/nomic/code_tokens.h: 40856 token strings (575KB)
- semantic.c: cbm_sem_random_index() now looks up pretrained vectors first,
falls back to sparse random for unknown tokens
- CBM_SEM_DIM raised from 256 to 768 to match Nomic nomic-embed-code
Also: RRI (Reflective Random Indexing), code pattern vocabulary injection,
120+ abbreviation expansions, callee/caller/body token enrichment,
label filter (Function/Method/Class only) in vector search SQL.
Binary size: 136MB → 169MB (+33MB from embedded vectors).
Search quality: keyword queries return relevant error-handling functions.
Domain-specific keyword queries return the expected functions.
Vector search pipeline (end-to-end):
- store.c: cbm_cosine_i8 custom SQLite function (int8 dot product with
sqrt magnitude), registered at store open time
- store.c: cbm_store_vector_search() builds merged query vector from
keywords via base RI vectors (xxHash), int8-quantizes, runs SQL query
with JOIN nodes ON node_id = id for metadata
- mcp.c: search_graph gains semantic_query param (array of strings),
returns semantic_results alongside regular results
- sqlite_writer.c: fixed build_vector_record to include node_id in
record body (matches build_node_record pattern)
- mcp.c: use yyjson_mut_obj_add_strcpy (not add_str) to avoid
use-after-free when vector results are freed before JSON serialization
Quality note: base RI vectors at query time don't capture co-occurrence
enrichment — results are moderate quality. Storing enriched token vectors
would improve this significantly (future work).
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
lixsp11
Martin VogelandAlexandros Pappas
Martin Vogel
