Below a JVM source root, a package named like a fast-mode skip dir
(integration/) is kept in moderate mode, while a top-level integration/
stays skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: mbrl <mbrl@local>
Below src/<set>/{java,kotlin,scala,groovy}/ every directory is a package,
so a built-in skip name there (target, tmp, vendor, deploy, env, dist...)
is a package name, not build output. Discovery and the inventory filter
now keep such directories; the root itself, its siblings and the safety
core (.git, worktrees) stay skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Leonardo trindade miranda <tmonestudio@gmail.com>
Brings #2161 up to date with main a2227157 (1035 commits since the
last refresh at bedbc0ac). The merge is clean: main's own changes to
src/discover/discover.c and tests/test_discover.c sit in other
regions, so the merged tree is main plus exactly this PR's commit
80f8bda5.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings #2421 (dep-info .d files kept out of the D grammar, direction
approved in #2414) up to date with main a2227157.
One textual conflict, in the SUITE(pipeline) RUN_TEST list of
tests/test_pipeline.c: main added
pipeline_stem_folder_collision_keeps_both_nodes and
pipeline_kustomize_entry_beside_same_stem_file right after
pipeline_structure_nodes, where this branch adds
pipeline_d_dep_info_leaves_no_nodes. Resolved by keeping all three
RUN_TEST lines, main's first.
src/discover/language.c, src/discover/discover.h, tests/test_discover.c
and tests/test_language.c merged cleanly. The merged tree adds exactly
this branch's 310 lines to main and deletes nothing.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
safe_stat reports a symlink as CBM_NOT_FOUND by policy, and
walk_dir_process_entry turned any CBM_NOT_FOUND into out->failed when
count_only was set. One symlink anywhere in a tree therefore aborted the
whole count.
cbm_discover_count_bounded is what auto-index admission calls (the MCP
session path and the daemon path), so any repository containing any
symlink could never auto-index: admission logged
reason=unsafe_or_unavailable_path files=-1 and returned. Nothing is unsafe
and the walk is not slow; the count simply refused at the first symlink it
met. Explicit index_repository was unaffected, because count_only is false
there and the same entry is skipped rather than fatal.
A count now skips symlinks and still aborts on an entry it genuinely cannot
observe, which is what the admission guard wants.
Carried from #1803 by Logan G onto current main. #1822 has since given
safe_stat an is_symlink out-param for the ignored-files report, so this
reuses it instead of adding a second is_link parameter; the production
change is the one condition. The regression test
discover_count_bounded_survives_a_symlink is the contributor's, aligned
with the neighbouring symlink tests: a visible SKIP_PLATFORM on Windows
instead of compiling it out, no wall-clock deadline (0 = none), and
th_cleanup of its temp tree.
Co-authored-by: Logan G <logan@gosolto.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
git applies ignore rules to untracked paths only, but discovery applied
them to every path. A nested .gitignore line `manager`, meant for a
built binary, hid cluster-api's tracked config/manager/ kustomization,
and with it the Deployment its overlays point at.
When the indexed directory is in a git repository, discovery now reads
`git ls-files -z` once. A directory that git's rules ignore (.gitignore
files, info/exclude, the global excludes file) is still entered when it
holds tracked files. Inside it, untracked files and directories stay
ignored, as in git. A tracked file is kept although a rule matches it.
The built-in skip list, .cbmignore and the size cap apply as before. If
git cannot run, nothing changes.
On the 17 infra corpora git lists 34 tracked files that an ignore rule
matches. The fix indexes 26 of them, all from that list. The other 8 are
never indexed for other reasons: no language (.tfvars, .gitkeep) or a
built-in skip directory (vendor/). No file is lost and no untracked
file appears. cluster-api gains the manager kustomization (4 Resources,
its overlay IMPORTS), helm-charts a Chart, and otel-demo a Route.
Test: discover_gitignore_never_hides_tracked_files builds a real git
repository. It fails without the fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
.d mapped to D unconditionally, so make-style dependency files written by
cargo, rustc, gcc/clang -MD and CMake were parsed with the D grammar and
walked at superlinear cost. Add cbm_disambiguate_d, a 4 KB first-line sniff
in the shape of the existing .frm/.cls/.m disambiguators, that returns
CBM_LANG_COUNT only for a make rule whose targets look like paths and keeps D
on any doubt.
A 200-file cargo dep-info corpus goes from 108.9 s to 1.27 s to index; 0 of
12,062 real D files are misclassified.
Signed-off-by: Nelson Spence <nelson@projectnavi.ai>
storage/framework/views/ holds Laravel's compiled Blade templates:
generated PHP with no source value. It was excluded only through
Laravel's stock .gitignore in that folder; when that file is missing,
every compiled view was indexed and reported as parse_partial by the
php grammar, burying real coverage gaps under generated noise.
Every built-in skip rule matched a directory BASENAME, and "views" is
far too common to skip everywhere. Add ALWAYS_SKIP_DIR_SUFFIXES, a
built-in rule list matched against the relative path suffix on a
'/'-component boundary, so storage/framework/views (at any depth, e.g.
backend/storage/framework/views) is skipped while resources/views,
app/views, app/framework/views, mystorage/framework/views and
storage/framework/views_old stay indexed. The rule is applied in
should_skip_directory next to the basename skip list, so it shares its
semantics: the subtree is reported as an excluded dir (#411) in the
index result and as not_indexed_dir "excluded subtree" in
check_index_coverage, and a .cbmignore negation
(!storage/framework/views/) can un-skip it like any non-safety-core
built-in skip (#500).
Test: discover::discover_skips_laravel_compiled_views_issue1735 (RED
before, green after, RED on revert) and
discover::discover_cbmignore_negates_laravel_compiled_views_issue1735.
Docs: docs/cbmignore.md precedence layer 1.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The previous commit found the enclosing repository when a git-less subfolder
is indexed, but merged its patterns into the matcher whose rel_path is
relative to repo_path. gitignore.c matches a rooted pattern with
`glob_match_bounded(p->pattern, rel_path)`, so every anchored pattern of the
enclosing repository was silently re-anchored onto the indexed subfolder.
That is wrong in both directions, and both were checked against real
`git check-ignore` run inside the subfolder:
* enclosing root .gitignore `/secret.py`, indexing pkg/
git INDEXES pkg/secret.py — we HID it. A rooted pattern means "in the
repository root", and re-anchored it became "in pkg". Silent index loss:
the file is gone from the graph with no diagnostic, the worst failure
mode discovery has.
* enclosing .git/info/exclude `pkg/scratch/`, indexing pkg/
git IGNORES pkg/scratch/tmp.py — we INDEXED it. The pattern is anchored
THROUGH pkg, so once rel_path lost the "pkg/" prefix it stopped matching
and an excluded directory got walked.
Only the enclosing root's .gitignore was consulted, too; git reads the
.gitignore of every directory between the repository root and the indexed
one, deeper overriding shallower.
Mechanism. discover.c already had the chain that encodes "the deepest file
with an opinion wins": gitignore_link_t carries a `prefix`, the walk-relative
directory a nested matcher came from, which local_rel_path() strips off. An
ancestor is the mirror image of that, so the chain is extended rather than
duplicated: each link now also carries a `base`, the walk root's path
relative to the directory the matcher came from, which gitignore_chain_result
PREPENDS instead of stripping (discover.c:569, 610). Exactly one of the two
offsets is ever non-empty, and both live in the link's flexible tail, so no
allocation site is added.
* ancestor_ignores_build() (discover.c:1282) loads the .gitignore of every
directory from the enclosing repository root down to — but not including —
the indexed one, shallow to deep, and chains them so the deepest wins,
negations included. The enclosing repository's info/exclude is merged into
the ROOT ancestor's matcher, which is where git anchors it, keeping the
precedence info/exclude already has over .gitignore at the same level.
* the indexed directory's own .gitignore stays its own matcher at the leaf
of that chain (discover.c:1480), so it still overrides every ancestor.
Nothing is merged into it any more, which also removes the earlier merge's
failure mode of dropping the local patterns and keeping the ancestor's.
* repo_path is canonicalized with cbm_canonical_path() before the lexical
parent walk (discover.c:1414), so `cbm index pkg` and
`cbm index /abs/path/pkg` discover the same files; pipeline.c strdups the
caller's path unchanged.
* walk_owned_gitignore_free() (discover.c:1019) is factored out of walk_dir
and shared with the ancestor chain, which owns its matchers the same way.
Proof on real input, before the suites. Indexing scripts/ of this checkout —
a git-less subfolder whose root .gitignore carries the rooted patterns
`/memlab-*` and `/soak*/`:
before 87 files memlab-drive.py MISSING memlab-report.py MISSING
after 89 files memlab-drive.py present memlab-report.py present
`git check-ignore` inside scripts/ indexes both, so the two recovered files
are exactly the ones git keeps. No file is lost in the other direction.
RED on the previous commit, four new tests, each direction pinned to a
`git check-ignore` verdict on an identical fixture:
test_discover.c:1471: count == 1, expected 2 == 2 rooted_pattern_not_reanchored
test_discover.c:1498: count == 2, expected 1 == 1 info_exclude_rooted_subpath
test_discover.c:1526: count == 2, expected 1 == 1 intermediate_gitignore
test_discover.c:1555: count == 0, expected 1 == 1 deeper_ancestor_negation_wins
discover: 120 passed, 4 failed
The three tests already on the branch use unanchored patterns, which match at
any depth and so cannot see this bug; they pass before and after.
GREEN: discover 124 passed, 0 failed; discover + gitignore + pipeline +
git_context 425 passed, 0 failed.
Revert-check: reverting only the discover.c hunks and rebuilding puts the
four new tests RED again on the same four lines with the same counts
(120 passed, 4 failed); restoring returns 425 passed. Memory-core linter:
none grew (31 raw sites in discover.c before and after). clang-format clean.
Co-authored-by: Amir Fathi <amirfathi.me@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`external` was listed in FAST_SKIP_DIRS, so fast and moderate discovery
silently dropped every directory named `external`. In many codebases
that name holds first-party code (external-facing APIs, adapters,
integration layers) rather than vendored third-party sources, and the
dedicated `vendor` / `third_party` / `3rdparty` entries already cover
the vendored case. Remove `external` from the fast skip list so those
sources are indexed in every mode.
The shortened list is regrouped with category comments in the same
style as ALWAYS_SKIP_DIRS; element order is unchanged.
Signed-off-by: Hubert Tarnacki <hubert.tarnacki@gmail.com>
resolve_git_common_dir() only stats repo_path/.git directly, so indexing a
subfolder with no .git of its own never consulted any ancestor repo's
.gitignore. Add resolve_enclosing_git_root() to walk up parent directories
the way git itself does, and merge that ancestor's .gitignore (and
info/exclude) ahead of the subfolder's own so the more specific file still
wins on conflict.
Fixes#510
Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
Indexing accepts whatever a repository contains. A tree carrying a vendored
monorepo, a generated dump, or a runaway build directory is discovered in
full, and the first sign of trouble is a host under memory pressure with
nothing that attributes it to indexing.
Add two opt-in limits evaluated during discovery against accepted source
files only: index_max_files and index_max_source_mb. Both default to off, so
nothing changes until an operator sets one. Crossing a limit fails the whole
attempt with a structured resource_limit_exceeded result naming the resource,
the observed value and the limit; no partial graph is published, and an
existing serving index keeps answering.
Limits are read from the CLI-managed _config.db and are not MCP request
arguments. A supervised parent replaces any caller-supplied policy before
spawning its worker, and the worker rejects a missing or incomplete contract,
so the CLI, the daemon and the supervised worker all enforce the same
decision.
Both keys reach an operator through the existing config get/set/list/reset
with no new subcommand. `set` suppresses its own generic message for them
because the policy writer names the precise reason -- so that writer speaks
on every failure it can return, including a validated value whose write then
fails on a database that cannot be written. Exiting non-zero in silence is
not an acceptable answer from a CLI.
The two shell regressions that hand-roll the supervisor's worker argv carry
that contract as well. Without it the worker exits before either guard can
observe anything, and the guard would go quietly vacuous.
Signed-off-by: 刘冲 <mail@liuchong.dev>
try_load_nested_gitignore() returned NULL as soon as a walk frame already
carried a local matcher, and that matcher was inherited by every child
frame. So only the SHALLOWEST nested .gitignore on a path was ever loaded;
any deeper one was silently dropped — an EMPTY storage/.gitignore was
enough to disable storage/dump/.gitignore's "*". On the standard Laravel
layout (storage/.gitignore above storage/<dir>/.gitignore) that meant
thousands of git-ignored cache/debugbar dumps were discovered and indexed,
which on the reporter's app grew to 25,638 discovered files against 6,682
git-visible ones and ended in an OOM kill (#1973, discovery half; also the
§2 root cause of #530).
Discovery now keeps a root->leaf chain of .gitignore links per directory
frame: every directory below the root loads its own .gitignore, a frame
without one shares its parent's link, and a path is judged by walking the
chain leaf->root with git's precedence — the deepest file with an opinion
wins (so a deeper negation re-includes what an ancestor ignores), and inside
one file the last matching pattern wins. The repository root's .gitignore
(merged with info/exclude) is the chain head, so the same precedence holds
between root and nested files. Cost is one match per .gitignore on the
path — O(depth) per entry, no rescans; the walk frame shrinks by the 4 KB
local-prefix buffer it no longer needs.
Tests pin the reporter's layout (empty ancestor + deeper "*"), a deeper
negation overriding an ancestor pattern, and three nesting levels with an
empty middle file. Measured on a Laravel app (koel) seeded with typical
compiled-view/cache artifacts: 2,885 discovered files with 450 git-ignored
-> 2,435 with 0; clean clones of linux, dotnet/runtime, koel and redis
discover byte-identical file sets before and after.
Fixes the discovery half of #1973.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
walk_dir_process_entry dropped a safe_stat() symlink/reparse-point rejection
with a bare return, unlike its two sibling skip paths (directory exclusion,
file skip_reason), which both call file_list_add_ignored. A file reachable
only through a symlinked directory vanished from the index with no trace in
not_indexed, skipped, or parse_partial, so a symlink-heavy subtree looked
identical to a complete index.
safe_stat now reports whether a rejection was specifically the
symlink/reparse-point check via an out-param, and walk_dir_process_entry
records it as an ignored file with reason "symlink" when so. The
count_only path and the traversal-following decision itself are untouched.
Fixes#1815
Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
Two loose ends from the bounds work.
The decompression side already took size_t so a >2 GiB capacity could not wrap
through int, but compression still took int for both the source length and the
destination capacity, and the artifact export cast a size_t database size down
to reach it. A database past 2 GiB would have handed the encoder a negative
length. Both lengths and the bound helper are size_t now, and the function
returns int64_t like its decompressing counterpart.
The discovery walk now prunes the cache directory by absolute path. A custom
CBM_CACHE_DIR may sit inside a repository — tests do it routinely — and walking
into it pulls every other project's graph database into this project's file
list. This is the narrow form of a concern that was briefly implemented as
refusing any root that contained the cache; refusing a whole root was too blunt,
and not walking the cache is what the concern actually asks for.
The test fails without the prune: a .go file planted under the cache is
otherwise discovered, and "cache" is not in the built-in skip list, so the
assertion is not vacuous.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A directory with more immediate subdirectories than the fixed 512-frame
walk stack made the whole discovery walk hard-fail with files=0 and a
misleading 'check repo_path' hint — dotnet/runtime carries 855 sibling
dirs in one JIT regression folder and could not be indexed at all. The
stack now grows geometrically from the same initial capacity; the parent
frame is a popped copy, so growth cannot dangle it. Covered by a 600-dir
fanout reproduction.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A .cbmignore negation pattern (e.g. "!obj/") whose rule is the last
match for a directory now un-skips built-in skip-list dirs, so projects
can index sources that live under names like obj/, target/, or docs/
(issue #500). Distills the design and test intents of PR #543 onto the
current 3-state cbm_gitignore_match_result API: instead of a second
inverted-polarity matcher, should_skip_directory consults the existing
last-match-wins result (<0 = negated) inside the built-in branch and
falls through so .gitignore/global/local rules still apply.
Non-negatable safety core: .git, node_modules, and the worktree-internal
dirs (.worktrees, .claude-worktrees) can never be un-skipped — a
repo-committed .cbmignore must not defeat OOM/safety skips (#489, #802).
Un-skipped dirs are walked normally and therefore no longer appear in
the excluded-subtree list (#411 coherence).
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
In a linked worktree, <repo>/.git is a regular file ("gitdir: <path>")
rather than a directory. cbm_discover_ex gated all per-clone ignore
sources on .git being a directory, so inside a worktree it skipped
.git/info/exclude and the global core.excludesfile entirely and walked
into directories the user had excluded there (e.g. build/, twister-out/).
Committed .gitignore still applied via the issue #510 path, which is why
the gap was easy to miss.
Resolve the git common dir up front: when .git is a gitlink file, parse
'gitdir:', then follow <gitdir>/commondir to the shared dir where git
keeps info/exclude and config (falling back to the gitdir when no
commondir file is present). info/exclude and config are then read from
that common dir, so a worktree behaves like an ordinary checkout.
Adds discover_worktree_info_exclude (fails before this change) and
discover_worktree_committed_gitignore.
Signed-off-by: Nahim El Atmani <2959826+Naam@users.noreply.github.com>
Drop the repo-local .git/config pass in resolve_global_excludes_path() so
an untrusted repo cannot point core.excludesFile at an arbitrary file (e.g.
~/.ssh/id_rsa) and have cbm open and read it merely by being indexed. git is
safe here because it only trusts a repo's .git/config after a deliberate
clone/init; the indexer has no such trust step. Global excludes now resolve
only from the user-owned ~/.gitconfig and $XDG_CONFIG_HOME/git/config. A
repo's own ignores remain handled by .gitignore / .cbmignore.
Add a guard test asserting a repo-local .git/config whose excludesFile points
outside the repo is ignored, not read.
Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* fix(discover): honor .git/info/exclude during file discovery
index_repository only loaded .gitignore when building the exclusion
matcher. .git/info/exclude — the per-clone exclude file that Git treats
as authoritative — was never read, so paths excluded only there were
walked in full. On repos with Sandcastle worktrees this caused the
indexer to traverse 661K files / 170 GB and OOM (issue #489).
Add cbm_gitignore_merge() to append patterns from one matcher into
another. In cbm_discover_ex, after loading .gitignore, also load
.git/info/exclude and merge its patterns in. If only the exclude file
exists, use it directly. No downstream call paths change.
Add three unit tests for cbm_gitignore_merge and two integration tests
that reproduce the issue scenario (exclude-only and exclude-stacked-
with-gitignore).
Signed-off-by: ShauryaaSharma <shauryasofficial27@gmail.com>
* style(discover): fix clang-format alignment in cbm_gitignore_merge
Signed-off-by: ShauryaaSharma <shauryasofficial27@gmail.com>
---------
Signed-off-by: ShauryaaSharma <shauryasofficial27@gmail.com>
deploy/ and deployed/ build artifact directories were in FAST_SKIP_DIRS,
which only applies in fast/moderate indexing modes. In the default full
mode they were indexed alongside source files, creating duplicate
Class/Method rows with stale data. Any MATCH (c:Class {name:'Foo'})
without a file_path filter returned multiple rows — one from source,
one from the build copy.
Move deploy and deployed to ALWAYS_SKIP_DIRS so they are excluded in all
indexing modes.
Regression test: discover_deploy_excluded_full_mode in test_discover.c.
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
Add scripts/check-no-test-skips.sh (run from lint) which fails the lint phase on any plain SKIP() or direct tf_skip_count manipulation; only SKIP_PLATFORM() (for genuinely platform-specific tests) is tolerated. Add FAIL() and SKIP_PLATFORM() helpers to the test framework and convert the remaining SKIP()/perf-gated skips across the suite into pass-or-fail assertions, so a suite that cannot meet its preconditions reports a red failure instead of a silent skip.
The v0.6.0 behavior where a vendor/ directory in the root .gitignore was
still indexed is fixed on the v0.7.0 discovery rework. Add a faithful
regression test (PHP project, untracked vendor/ in root .gitignore) asserting
nothing under vendor/ is discovered, so it cannot regress.
- cbm_arena_alloc returns NULL on a NULL arena (both arena.c copies)
instead of dereferencing it — defense-in-depth against the NULL-arena
type-allocation path that could crash the LSP type layer.
- discover: add "vendored" to the always-skip directory list.
Distilled from #374, taking the parts not already on main. The PR's
ts_lsp tuple-arena allocation, the C/C++ template formal-count clamp,
and the type_rep unbound-param preservation it also carried all landed
independently in v0.7.0 / via #322 / #360, so only these two
defensive improvements remain. Relates to #390.
Nested .gitignore support (fixes#178):
- Load per-subdirectory .gitignore during walk, match paths relative to
the gitignore's directory via local_rel_path()
- Root and nested gitignores stack independently
- Owned gitignores collected and freed at walk end (avoids use-after-free
from borrowed pointers on the iterative stack)
Security:
- Run security-strings/install/network + ClamAV + Windows Defender on
ALL binary variants (standard + UI), not just standard
- Whitelist UI bundle URLs (React, Three.js, Google Fonts, Tailwind, W3C)
Co-Authored-By: dLo999 <dLo999@users.noreply.github.com>
Discover integration tests use system("rm -rf && mkdir -p && echo >")
which are Unix shell commands. These tests are properly skipped on
Windows with SKIP_UNIX_SHELL macro. The remaining 4 Windows test
failures (store_dump, cli_find, ui_config x2) need individual fixes.
TODO: Port discover test helpers to use C file operations.
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
Martin Vogel
mbrlandClaude Opus 5.5
Leonardo trindade mirandaandClaude Opus 5.5
Nelson Spence
Hubert Tarnacki
刘冲
ibaldr89
Nahim El Atmani
Shaurya
Thomas Dyar
Martin Vogel
Ivan Porto Carrero
