Systematic audit of every grammar's parser.c against the current
lang_specs.c entries. For each language, identified all primary
extraction-relevant node types (function/class/call/import) present
in the grammar but missing from the spec.
Changes:
- 44 new node type arrays created for languages that had empty_types
where the grammar provides extractable constructs
- 121 existing arrays extended with additional node types
- Covers all 153 vendored grammars
Examples of additions:
- Go: import, import_declaration added to import_types
- Python: import, import_from_statement, import_statement added
- JavaScript: new_expression, call_expression added to call_types
- PHP: namespace_use_declaration added (was completely missing)
- Rust: macro_invocation added to call_types
- C/C++: struct_specifier, enum_specifier, union_specifier added
All 2797 tests pass with zero regressions.
Fill in function, class, field, call, import, branch, variable,
assignment, throw, and decorator node types for all new languages
based on per-grammar AST analysis. Previously all 89 new languages
had empty_types (module detection only, no extraction).
Programming languages with full specs (31):
Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim,
Crystal, Odin, Ada, Sway, Cairo, Move, Hare, Pony, AWK, Fish,
Zsh, Tcl, Fennel, ReScript, PureScript, Nickel, Teal, Luau,
Squirrel, ISPC, Starlark, Puppet, LLVM IR
IDL/config with type/field specs (12):
Prisma, Thrift, Cap'n Proto, Smithy, WIT, Bicep, WGSL, HLSL,
VHDL, SystemVerilog, Apex, Kconfig
Remaining languages have module_types only (config/data formats
where function/class extraction is not applicable).
Add automated extension-to-language mapping tests for all new grammars
that have file extensions. Removes conflicting extensions:
- .blade.php: compound extension, handled by userconfig system
- .env*: filenames, not extensions (no auto-discovery for dotfiles)
- .sv/.svh: already mapped to Verilog, SystemVerilog uses same grammar
Migrate the lang_specs[] array from implicit positional indexing to
C99 designated initializers ([CBM_LANG_X] = {...}), eliminating a
class of silent ordering bugs when adding new languages.
Changes:
- Add ts_factory function pointer to CBMLangSpec struct
- Convert all 66 entries to designated initializer syntax
- Add _Static_assert guard: array size must match CBM_LANG_COUNT
- Replace 140-line cbm_ts_language() switch with 3-line factory dispatch
- Net reduction: 127 lines removed
This is prerequisite infrastructure for the ~90 new language expansion.
- Add SHA256 checksum verification against checksums.txt (was missing,
all other installers had it)
- Validate URL scheme on every redirect hop (HTTPS-only, max 5 redirects)
- Replace execSync string interpolation with execFileSync array args
(eliminates shell injection vector)
- Add path traversal check on extracted binary (tar-slip defense)
install.ps1:
- Remove both Unblock-File calls (Defender ClickFix.R!ml signal)
- Remove ExecutionPolicy ByPass from usage comment (scanned by Defender)
- Add HTTPS-only URL scheme check on CBM_DOWNLOAD_URL
install.sh:
- Add HTTPS-only URL scheme check on CBM_DOWNLOAD_URL
pkg/go wrapper:
- Add validateURLScheme() to httpGet and fetchChecksums (defense-in-depth,
matching Python wrapper's _validate_url_scheme from PR #248)
All installers now allow localhost/127.0.0.1 for testing but reject
http://, ftp://, file:// and other schemes.
Closes#246.
Three security improvements to pkg/pypi/src/codebase_memory_mcp/_cli.py:
1. tar-slip mitigation (S202) — the previous `tf.extractall(tmp)` /
`zf.extractall(tmp)` calls trusted the archive not to contain absolute
paths or `..` components. Replaced with _safe_extract_tar and
_safe_extract_zip:
- On Python 3.12+, tarfile uses the new `filter='data'` (PEP 706).
- On Python 3.8–3.11 and for zipfile (no built-in filter), each member
path is validated against the destination via os.path.abspath checks;
symlinks and hardlinks in tarballs are rejected outright.
2. URL scheme allowlist (S310) — _validate_url_scheme rejects anything
that isn't https before urlretrieve runs. The URL is constructed from
hardcoded constants today, but this is a cheap defense-in-depth check
against future config sources, redirects, or typos.
3. Justified noqa suppressions on the exec path — os.execv and
subprocess.run receive args as a list (not a shell string) and neither
uses shell=True, so there is no injection vector. Comments document
the invariant so future changes don't reintroduce a shell form.
Verification:
- `ruff check _cli.py --select S` — passes (was 5 findings, now 0)
- Syntax check passes under Python 3.14
- Behavior preserved: download/extract/exec flow is unchanged on happy path
- New failure modes: clear `sys.exit` with actionable message when a
malicious member path or non-https URL is detected
No new dependencies; all changes stdlib-only and compatible with the
project's declared `requires-python = ">=3.8"`.
- SECURITY.md: new Transparency & Disclaimer section at the top with
honest description of filesystem access patterns and liability scope;
new Help Us Stay Secure section actively inviting security research
with a defined scope and promise of fast response + public credit
- README.md: Security & Trust callout after the intro paragraph — brief,
direct, links to SECURITY.md for full context
checksums.txt contains hashes of .tar.gz/.zip archives; VirusTotal
indexes the extracted binaries. Updated README table and badge to use
correct binary SHA-256 hashes for v0.6.0. Updated release.yml to hash
extracted binaries from the binaries/ directory for future releases.
- Adds VirusTotal (0/72) and SLSA Level 3 badges to the header
- Adds Security section with per-binary VirusTotal scan links for v0.6.0
- Adds step in release.yml to auto-append a Security Verification table
with per-binary VirusTotal links to every release's notes going forward
Adds a README.md to pkg/npm so the npmjs.com package page shows
documentation. Updates the files list in package.json to include it.
Will be published with the next version bump.
Adds a thin Go wrapper at pkg/go/cmd/codebase-memory-mcp that detects
the current platform and architecture, downloads the matching pre-built
binary from GitHub Releases, verifies its SHA-256 checksum, caches it,
and replaces the current process via exec.
Install via:
go install github.com/DeusData/codebase-memory-mcp/pkg/go/cmd/codebase-memory-mcp@latest
The Go module proxy indexes this automatically once live on main.
Introduces pkg/ as a self-contained home for all package registry wrappers.
Each subdirectory is a ready-to-publish package that downloads the appropriate
pre-built binary from GitHub Releases — no source compilation at install time.
- pkg/npm/ — postinstall downloads binary; bin.js shim for CLI
- pkg/pypi/ — lazy download on first run via _cli.py; pipx-friendly
- pkg/homebrew/ — multi-arch formula (darwin/linux × arm64/amd64)
- pkg/scoop/ — Windows bucket manifest with autoupdate + checksum
- pkg/winget/ — v1.6.0 manifests (version + installer + locale)
- pkg/chocolatey/ — nuspec + install/uninstall scripts
- pkg/aur/ — PKGBUILD + .SRCINFO for x86_64 and aarch64
All hashes are real SHA-256 checksums from the v0.6.0 release.
- delete_project now calls cbm_mem_collect() after closing the database
to return freed pages to OS immediately
- Set purge_delay=0 (was 1000ms) so mimalloc purges freed pages without delay
- Set purge_decommits=1 explicitly so macOS uses MADV_FREE_REUSABLE
(immediate RSS accounting) instead of MADV_FREE (lazy, RSS stays high)
Before: deleting Linux kernel index left 8.1G RSS (2.1G dirty pages)
After: dirty pages purged immediately, RSS reflects actual usage
VT's status=completed is final — no more engines will report. The script
was polling indefinitely when ARM binaries only reached 50/76 engines.
Now accepts any completed scan, logs a NOTE when below MIN_ENGINES.
Nested .gitignore support (fixes#178):
- Load per-subdirectory .gitignore during walk, match paths relative to
the gitignore's directory via local_rel_path()
- Root and nested gitignores stack independently
- Owned gitignores collected and freed at walk end (avoids use-after-free
from borrowed pointers on the iterative stack)
Security:
- Run security-strings/install/network + ClamAV + Windows Defender on
ALL binary variants (standard + UI), not just standard
- Whitelist UI bundle URLs (React, Three.js, Google Fonts, Tailwind, W3C)
Co-Authored-By: dLo999 <dLo999@users.noreply.github.com>
gcc/glibc embed bug tracker URLs (bugs.launchpad.net, gcc.gnu.org,
sourceware.org) into statically-linked binaries. These are compiler
artifacts, not our code.
check_already_latest() exits early before printing dry-run info when
the running version matches the latest release. Skip this check in
dry-run mode so the user sees what would happen.
Portable binaries now get the same security-strings, security-install,
and security-network audits as regular Linux/macOS builds. Closes gap
where the portable (musl static) binary was smoke-tested but not
security-audited.
Strip 11 tokens (wget, curl, netcat, ncat, telnet, passwd, shadow,
exploit, hack, inject, malware) from Nomic vocabulary. These fall back
to sparse random vectors — negligible quality impact. Removes all
security audit exceptions: zero allowlists, zero suppressions.
The embedded Nomic code token vocabulary (40K tokens) includes words like
"wget" as code tokens. Filter out bare single-word matches (2-10 lowercase
chars) since real dangerous strings appear in command context, not as
standalone vocabulary entries.
- Set APPDATA/LOCALAPPDATA env vars for Windows so cbm_app_config_dir()
and cbm_app_local_dir() resolve to FAKE_HOME paths
- Add Windows (*.exe) branches for Zed, KiloCode, VSCode config checks
- Add macOS branch for KiloCode (was hardcoded to .config/ on all platforms)
- Create platform-correct detection dirs (AppData on Windows, Library on
macOS, .config on Linux) so agent detection + install paths match
- Update skill check from old 4-skill names to consolidated codebase-memory
(old dirs are cleaned up during install since skill consolidation)
On Alpine musl (portable builds), cbm_app_config_dir() needs explicit
XDG_CONFIG_HOME to resolve to FAKE_HOME/.config. Without it, the install
writes to the wrong path and KiloCode/Zed config checks fail.
- semantic.c: free strdup'd keys+values in token_map via cbm_ht_foreach
before cbm_ht_free (44KB leak in 7920 allocations)
- code_vectors_blob.S: add Windows COFF branch (.rdata,"dr") alongside
macOS Mach-O and Linux ELF sections
- code_vectors_blob.S: preprocessor conditionals for macOS (Mach-O
__DATA,__const + underscore prefix) vs Linux (ELF .rodata, no prefix)
- Makefile: use $(CC) -c instead of $(AS) to enable preprocessor on .S
- Add vendored/nomic to KNOWN_VENDORED security allowlist (pure int8
vector data, zero executable code)
- Update vendored checksums
Ports four targeted improvements from #162 (Koolerx):
- FTS5 BM25 ranked search via a new `query` parameter on search_graph.
Uses a contentless nodes_fts virtual table populated at dump time with
a cbm_camel_split() SQL function that splits camelCase identifiers
(updateCloudClient -> update Cloud Client) so tokenized word queries
match identifier internals. BM25 rank is boosted by node label
(Function/Method +10, Route +8, Class/Interface/Type/Enum +5) and
noise labels (File/Folder/Module/Variable) are filtered out. The new
path is opt-in via the query parameter; regex name_pattern and vector
semantic_query paths are untouched.
- Enable SQLITE_ENABLE_FTS5 in the vendored sqlite3 build flags.
- Rebuild nodes_fts after incremental reindex. The btree dump path
bypasses any FTS5 triggers, so the index is wiped and repopulated via
delete-all + INSERT .. SELECT after each incremental merge.
- Add Interface to the label filter on cbm_registry_add in both the
parallel and sequential definition passes, so C#/Java class
declarations like Foo : IBar can resolve IBar to an INHERITS edge
target during enrichment.
- Add an explicit C# base_list handler to extract_base_classes() that
iterates named children directly instead of returning the raw node
text. Without this the fallback path returned ": IExamService" (with
the leading separator) and registry lookup always failed, so C#
INHERITS edges to interfaces never resolved.
Co-Authored-By: Koolerx <Koolerx@users.noreply.github.com>
- Parallelize pass_similarity and pass_semantic_edges via worker pool with
thread-local edge buffers; sequential final merge since gbuf is not
thread-safe. Adds cbm_lsh_query_into() as a thread-safe variant with
caller-provided candidate buffer.
- Add activatable profiling subsystem (CBM_PROFILE=1 env or --profile flag)
for step-level timing of extract, resolve, corpus build, vector phases,
and sqlite dump. Zero overhead when disabled.
- Fix cbm_index_mode_t enum mismatch between pipeline.h (FULL=0, MODERATE=1,
FAST=2) and discover.h (FULL=0, FAST=1). mode=fast silently no-op'd
fast-discovery filtering because discover.c compared against the wrong
value. Linux kernel fast mode went 1:40 -> 3:11 as a result; now back to
1:40. Broaden the filter guard to mode != CBM_MODE_FULL so MODERATE and
FAST both get aggressive discovery.
- Clamp cbm_sem_combined_score output to [0, 1]. The proximity multiplier
returns up to 1.10 as a same-file boost which could push the final
cosine score above 1.0.
- Short-circuit semantic scoring when MinHash jaccard >= 0.95. Exact
near-clones are already emitted as SIMILAR_TO edges; returning 0 here
avoids flooding SEMANTICALLY_RELATED with cross-service copy-paste
boilerplate and frees the edge budget for genuine vocabulary-bridged
relations.
- Validate search_graph semantic_query as an array of strings and return
a clear error for a single-string input. Update the tool description
to spell out the requirement explicitly with an example.
- JSON-escape user-controlled strings (callee names, call arguments,
URL paths, import local_name) in call/argument properties. Introduces
cbm_json_escape() in foundation/str_util.
- Skip SQLite pending_byte_page (file offset 0x40000000) during raw page
writes in sqlite_writer to avoid corrupting databases that cross the
1 GiB boundary.
- Migrate pretrained vector blob from UniXcoder (51K tokens) to
nomic-embed-code (40856 tokens x 768d int8). Includes the extraction
script under scripts/extract_nomic_vectors.py.
Major quality improvement: replace random index vectors with pre-trained
Nomic nomic-embed-code token embeddings. Tokens like 'error' and 'exception' now start
with similar vectors (learned from millions of code repos) instead of
arbitrary random projections. Co-occurrence enrichment adds project-specific
context on top.
Architecture:
- vendored/nomic/code_vectors.bin: 37.7MB raw int8 vectors
- vendored/nomic/code_vectors_blob.S: assembler .incbin (instant build)
- vendored/nomic/code_vectors.h: extern declarations + pretrained_vec_at()
- vendored/nomic/code_tokens.h: 40856 token strings (575KB)
- semantic.c: cbm_sem_random_index() now looks up pretrained vectors first,
falls back to sparse random for unknown tokens
- CBM_SEM_DIM raised from 256 to 768 to match Nomic nomic-embed-code
Also: RRI (Reflective Random Indexing), code pattern vocabulary injection,
120+ abbreviation expansions, callee/caller/body token enrichment,
label filter (Function/Method/Class only) in vector search SQL.
Binary size: 136MB → 169MB (+33MB from embedded vectors).
Search quality: keyword queries return relevant error-handling functions.
Domain-specific keyword queries return the expected functions.
Vector search pipeline (end-to-end):
- store.c: cbm_cosine_i8 custom SQLite function (int8 dot product with
sqrt magnitude), registered at store open time
- store.c: cbm_store_vector_search() builds merged query vector from
keywords via base RI vectors (xxHash), int8-quantizes, runs SQL query
with JOIN nodes ON node_id = id for metadata
- mcp.c: search_graph gains semantic_query param (array of strings),
returns semantic_results alongside regular results
- sqlite_writer.c: fixed build_vector_record to include node_id in
record body (matches build_node_record pattern)
- mcp.c: use yyjson_mut_obj_add_strcpy (not add_str) to avoid
use-after-free when vector results are freed before JSON serialization
Quality note: base RI vectors at query time don't capture co-occurrence
enrichment — results are moderate quality. Storing enriched token vectors
would improve this significantly (future work).
test
Martin Vogel
DeusData
Dustin
