- delete_project now calls cbm_mem_collect() after closing the database
to return freed pages to OS immediately
- Set purge_delay=0 (was 1000ms) so mimalloc purges freed pages without delay
- Set purge_decommits=1 explicitly so macOS uses MADV_FREE_REUSABLE
(immediate RSS accounting) instead of MADV_FREE (lazy, RSS stays high)
Before: deleting Linux kernel index left 8.1G RSS (2.1G dirty pages)
After: dirty pages purged immediately, RSS reflects actual usage
VT's status=completed is final — no more engines will report. The script
was polling indefinitely when ARM binaries only reached 50/76 engines.
Now accepts any completed scan, logs a NOTE when below MIN_ENGINES.
Nested .gitignore support (fixes#178):
- Load per-subdirectory .gitignore during walk, match paths relative to
the gitignore's directory via local_rel_path()
- Root and nested gitignores stack independently
- Owned gitignores collected and freed at walk end (avoids use-after-free
from borrowed pointers on the iterative stack)
Security:
- Run security-strings/install/network + ClamAV + Windows Defender on
ALL binary variants (standard + UI), not just standard
- Whitelist UI bundle URLs (React, Three.js, Google Fonts, Tailwind, W3C)
Co-Authored-By: dLo999 <dLo999@users.noreply.github.com>
gcc/glibc embed bug tracker URLs (bugs.launchpad.net, gcc.gnu.org,
sourceware.org) into statically-linked binaries. These are compiler
artifacts, not our code.
check_already_latest() exits early before printing dry-run info when
the running version matches the latest release. Skip this check in
dry-run mode so the user sees what would happen.
Portable binaries now get the same security-strings, security-install,
and security-network audits as regular Linux/macOS builds. Closes gap
where the portable (musl static) binary was smoke-tested but not
security-audited.
Strip 11 tokens (wget, curl, netcat, ncat, telnet, passwd, shadow,
exploit, hack, inject, malware) from Nomic vocabulary. These fall back
to sparse random vectors — negligible quality impact. Removes all
security audit exceptions: zero allowlists, zero suppressions.
The embedded Nomic code token vocabulary (40K tokens) includes words like
"wget" as code tokens. Filter out bare single-word matches (2-10 lowercase
chars) since real dangerous strings appear in command context, not as
standalone vocabulary entries.
- Set APPDATA/LOCALAPPDATA env vars for Windows so cbm_app_config_dir()
and cbm_app_local_dir() resolve to FAKE_HOME paths
- Add Windows (*.exe) branches for Zed, KiloCode, VSCode config checks
- Add macOS branch for KiloCode (was hardcoded to .config/ on all platforms)
- Create platform-correct detection dirs (AppData on Windows, Library on
macOS, .config on Linux) so agent detection + install paths match
- Update skill check from old 4-skill names to consolidated codebase-memory
(old dirs are cleaned up during install since skill consolidation)
On Alpine musl (portable builds), cbm_app_config_dir() needs explicit
XDG_CONFIG_HOME to resolve to FAKE_HOME/.config. Without it, the install
writes to the wrong path and KiloCode/Zed config checks fail.
- semantic.c: free strdup'd keys+values in token_map via cbm_ht_foreach
before cbm_ht_free (44KB leak in 7920 allocations)
- code_vectors_blob.S: add Windows COFF branch (.rdata,"dr") alongside
macOS Mach-O and Linux ELF sections
- code_vectors_blob.S: preprocessor conditionals for macOS (Mach-O
__DATA,__const + underscore prefix) vs Linux (ELF .rodata, no prefix)
- Makefile: use $(CC) -c instead of $(AS) to enable preprocessor on .S
- Add vendored/nomic to KNOWN_VENDORED security allowlist (pure int8
vector data, zero executable code)
- Update vendored checksums
Ports four targeted improvements from #162 (Koolerx):
- FTS5 BM25 ranked search via a new `query` parameter on search_graph.
Uses a contentless nodes_fts virtual table populated at dump time with
a cbm_camel_split() SQL function that splits camelCase identifiers
(updateCloudClient -> update Cloud Client) so tokenized word queries
match identifier internals. BM25 rank is boosted by node label
(Function/Method +10, Route +8, Class/Interface/Type/Enum +5) and
noise labels (File/Folder/Module/Variable) are filtered out. The new
path is opt-in via the query parameter; regex name_pattern and vector
semantic_query paths are untouched.
- Enable SQLITE_ENABLE_FTS5 in the vendored sqlite3 build flags.
- Rebuild nodes_fts after incremental reindex. The btree dump path
bypasses any FTS5 triggers, so the index is wiped and repopulated via
delete-all + INSERT .. SELECT after each incremental merge.
- Add Interface to the label filter on cbm_registry_add in both the
parallel and sequential definition passes, so C#/Java class
declarations like Foo : IBar can resolve IBar to an INHERITS edge
target during enrichment.
- Add an explicit C# base_list handler to extract_base_classes() that
iterates named children directly instead of returning the raw node
text. Without this the fallback path returned ": IExamService" (with
the leading separator) and registry lookup always failed, so C#
INHERITS edges to interfaces never resolved.
Co-Authored-By: Koolerx <Koolerx@users.noreply.github.com>
- Parallelize pass_similarity and pass_semantic_edges via worker pool with
thread-local edge buffers; sequential final merge since gbuf is not
thread-safe. Adds cbm_lsh_query_into() as a thread-safe variant with
caller-provided candidate buffer.
- Add activatable profiling subsystem (CBM_PROFILE=1 env or --profile flag)
for step-level timing of extract, resolve, corpus build, vector phases,
and sqlite dump. Zero overhead when disabled.
- Fix cbm_index_mode_t enum mismatch between pipeline.h (FULL=0, MODERATE=1,
FAST=2) and discover.h (FULL=0, FAST=1). mode=fast silently no-op'd
fast-discovery filtering because discover.c compared against the wrong
value. Linux kernel fast mode went 1:40 -> 3:11 as a result; now back to
1:40. Broaden the filter guard to mode != CBM_MODE_FULL so MODERATE and
FAST both get aggressive discovery.
- Clamp cbm_sem_combined_score output to [0, 1]. The proximity multiplier
returns up to 1.10 as a same-file boost which could push the final
cosine score above 1.0.
- Short-circuit semantic scoring when MinHash jaccard >= 0.95. Exact
near-clones are already emitted as SIMILAR_TO edges; returning 0 here
avoids flooding SEMANTICALLY_RELATED with cross-service copy-paste
boilerplate and frees the edge budget for genuine vocabulary-bridged
relations.
- Validate search_graph semantic_query as an array of strings and return
a clear error for a single-string input. Update the tool description
to spell out the requirement explicitly with an example.
- JSON-escape user-controlled strings (callee names, call arguments,
URL paths, import local_name) in call/argument properties. Introduces
cbm_json_escape() in foundation/str_util.
- Skip SQLite pending_byte_page (file offset 0x40000000) during raw page
writes in sqlite_writer to avoid corrupting databases that cross the
1 GiB boundary.
- Migrate pretrained vector blob from UniXcoder (51K tokens) to
nomic-embed-code (40856 tokens x 768d int8). Includes the extraction
script under scripts/extract_nomic_vectors.py.
Major quality improvement: replace random index vectors with pre-trained
Nomic nomic-embed-code token embeddings. Tokens like 'error' and 'exception' now start
with similar vectors (learned from millions of code repos) instead of
arbitrary random projections. Co-occurrence enrichment adds project-specific
context on top.
Architecture:
- vendored/nomic/code_vectors.bin: 37.7MB raw int8 vectors
- vendored/nomic/code_vectors_blob.S: assembler .incbin (instant build)
- vendored/nomic/code_vectors.h: extern declarations + pretrained_vec_at()
- vendored/nomic/code_tokens.h: 40856 token strings (575KB)
- semantic.c: cbm_sem_random_index() now looks up pretrained vectors first,
falls back to sparse random for unknown tokens
- CBM_SEM_DIM raised from 256 to 768 to match Nomic nomic-embed-code
Also: RRI (Reflective Random Indexing), code pattern vocabulary injection,
120+ abbreviation expansions, callee/caller/body token enrichment,
label filter (Function/Method/Class only) in vector search SQL.
Binary size: 136MB → 169MB (+33MB from embedded vectors).
Search quality: keyword queries return relevant error-handling functions.
Domain-specific keyword queries return the expected functions.
Vector search pipeline (end-to-end):
- store.c: cbm_cosine_i8 custom SQLite function (int8 dot product with
sqrt magnitude), registered at store open time
- store.c: cbm_store_vector_search() builds merged query vector from
keywords via base RI vectors (xxHash), int8-quantizes, runs SQL query
with JOIN nodes ON node_id = id for metadata
- mcp.c: search_graph gains semantic_query param (array of strings),
returns semantic_results alongside regular results
- sqlite_writer.c: fixed build_vector_record to include node_id in
record body (matches build_node_record pattern)
- mcp.c: use yyjson_mut_obj_add_strcpy (not add_str) to avoid
use-after-free when vector results are freed before JSON serialization
Quality note: base RI vectors at query time don't capture co-occurrence
enrichment — results are moderate quality. Storing enriched token vectors
would improve this significantly (future work).
Four changes to reduce false positives from 382K to 58K SIMILAR_TO edges
on the Linux kernel while maintaining ~98% precision:
1. MIN_NODES raised to 30 (leaf tokens) — aligns with BigCloneBench
standard of 50 raw source tokens / 6 lines minimum clone size
2. collect_ast_tokens now records LEAF nodes only, not internal grammar
nodes. Leaf-only counting is language-agnostic: leaves correspond to
actual source tokens regardless of grammar verbosity (C tree-sitter
produces 2x more internal nodes than Python for the same logic)
3. hash_trigrams tracks unique structural trigram count via hash set.
Functions with < 32 unique structural trigrams are skipped — too few
distinct features for K=64 MinHash to produce reliable estimates
4. Structural-weighted MinHash via repetition: each trigram gets weight
0-3 based on count of non-normalised tokens (I/S/N/T = normalised).
Weight-0 trigrams (all-normalised, e.g. I|I|I) are pure noise and
skipped entirely. Weight 1-3 trigrams use repetition-based weighted
MinHash — higher weight = more hash attempts per seed = more likely
to dominate the signature. Achieves IDF-like effect without needing
corpus-wide statistics.
Also adds O(1) hash-set dedup in LSH query (replacing O(n) linear scan)
and bucket size cap of 200 to skip oversized buckets.
Linux kernel: 1:41, similarity 7.7s, 395K fp, 58K edges, ~98% precision.
Tests updated with larger Go functions to meet new thresholds.
Compute K=64 MinHash signatures from normalized AST node-type trigrams
during function extraction, then generate SIMILAR_TO edges via LSH
(b=32, r=2) for function pairs with Jaccard >= 0.95.
- src/simhash/minhash.{h,c}: MinHash compute, Jaccard, hex encode/decode,
LSH index with band hashing for O(n) candidate generation
- src/pipeline/pass_similarity.c: post-pass reads fingerprints from node
properties, builds LSH index, emits SIMILAR_TO edges with jaccard and
same_file metadata. Same-language only, max 10 edges per node.
- internal/cbm/cbm.h: fingerprint fields on CBMDefinition
- internal/cbm/extract_defs.c: compute_fingerprint() hook at 3 extraction
sites after complexity, skip functions with < 10 AST body nodes
- pass_definitions.c + pass_parallel.c: serialize fingerprint to "fp" hex
in properties_json for both sequential and parallel pipeline paths
- pipeline.c + pipeline_incremental.c: register pass_similarity in both
full and incremental post-pass lists
- tests/test_simhash.c: 28 tests across 4 suites (core, LSH, edge gen,
pipeline integration with generated Go project + incremental)
All hardcoded ~/.cache/codebase-memory-mcp paths now route through
cbm_resolve_cache_dir() in platform.c. Priority: CBM_CACHE_DIR env
var > default ~/.cache/codebase-memory-mcp.
Added db_path_for_project() helper in http_server.c to eliminate 5
copies of the same path construction pattern.
Fixes#154
Co-Authored-By: dLo999 <dLo999@users.noreply.github.com>
Check GitHub releases/latest redirect header before downloading.
Saves bandwidth and avoids unnecessary index rebuilds.
- "Already up to date" when version matches or is ahead
- --force flag to bypass the check
- Graceful degradation when network is unavailable
- Uses same curl dependency as the download itself
Fixes#142
Co-Authored-By: dLo999 <dLo999@users.noreply.github.com>
One non-UTF-8 node name crashed the entire /api/layout endpoint
(HTTP 500). Force layout can also produce NaN/Inf positions.
- Use YYJSON_WRITE_ALLOW_INVALID_UNICODE (matches MCP serializer)
- Sanitize NaN/Inf: positions default to 0.0, size to 1.0
- Log yyjson error code on serialization failure
Co-Authored-By: Selene29 <Selene29@users.noreply.github.com>
PATH uses ; on Windows, not : — strtok with : splits C:\Users at
the drive letter colon, producing garbage paths. Also skip S_IXUSR
check on Windows since stat() doesn't set Unix permission bits.
Fixes#159
Co-Authored-By: slvnlrt <slvnlrt@users.noreply.github.com>
Replaces hardcoded ~/.local/bin/ with OS-native runtime detection:
- Windows: GetModuleFileNameA()
- macOS: _NSGetExecutablePath()
- Linux: readlink("/proc/self/exe")
Falls back to ~/.local/bin/ if detection fails.
Fixes install.ps1 placing binary in $LOCALAPPDATA/Programs/ while
configs point to non-existent ~/.local/bin/ path.
Fixes#158
Co-Authored-By: slvnlrt <slvnlrt@users.noreply.github.com>
KiloCode is a VS Code extension — uses the same base config dir.
Apply same platform-aware paths as Zed/VS Code: macOS uses
$HOME/Library/Application Support/Code/, Windows uses %APPDATA%/Code/.
cbm_store_bfs doesn't support direction="both" — it silently falls
through to outbound only. Run two BFS passes (inbound + outbound)
and merge results, matching how handle_trace_call_path works.
Also fix use-after-free: use strcpy for BFS node names that get
freed before JSON serialization.
When include_tests=false (default), test files are filtered from trace
results. When true, test nodes appear with is_test=true marker. Helps
answer "which tests cover this function's blast radius?"
Also adds tests for include_tests and risk_labels params.
Co-Authored-By: maplenk <maplenk@users.noreply.github.com>
The watcher's git_is_dirty() only ran git status on the parent repo,
missing uncommitted changes inside submodules. Add a second check via
git submodule foreach --recursive (portable, works on Apple Git which
lacks --recurse-submodules).
Fixes#71
Co-Authored-By: halindrome <halindrome@users.noreply.github.com>
Merge codebase-memory-exploring, codebase-memory-tracing,
codebase-memory-quality, codebase-memory-reference into single
codebase-memory skill with decision matrix, workflows, and gotchas.
Old skill directories are cleaned up automatically during install.
Factual corrections vs original PR: trace_call_path -> trace_path,
removed non-existent tools (read_file, list_directory) and edge types
(OVERRIDE, USAGE, FILE_CHANGES_WITH), removed risk_labels parameter.
Co-Authored-By: gdilla <gdilla@users.noreply.github.com>
- Pin Dockerfile.alpine to SHA digest (PinnedDependenciesID)
- Narrow workflow-level permissions to contents:read (TokenPermissionsID)
- Elevated permissions only on release-draft and verify jobs
Local test infra now covers 4 platforms:
- Linux arm64 (Ubuntu, ASan+LeakSan)
- Linux portable (Alpine musl, static binary)
- Windows (mingw cross-compile)
- macOS (native)
Fixes:
- Remove sys/unistd.h and sys/poll.h (glibc-only, musl lacks them)
- Add STATIC=1 support to Makefile (appends -static to LDFLAGS)
- CI portable build uses docker run alpine instead of container directive
(GitHub Actions JS actions break in Alpine containers on ARM64)
DeusData
Martin Vogel
