walk_yaml_mapping (dotted key paths, 256 frames) and
scan_yaml_for_infra_bindings (512 nodes) kept their pending nodes in
fixed-size stacks and stopped pushing once full. A mapping with more than
256 nested sibling mappings lost the full key_path of every leaf past the
cap: `k299.url` was only recorded as the shorter path that the walk of a
nested block emits on its own.
The infra Route pass decides on those paths. A `data:` payload block with
300 entries lost `data.svcN.webhook_url` for N >= 256, the payload deny
never saw those values, and 44 of them were minted as infra Route nodes.
Both walkers now start on the same-sized on-stack chunk and grow through
cbm_walk_stack_reserve / cbm_walk_stack_release (#2375). If the stack
cannot grow, the failure is logged as extract.walk_stack_alloc_failed and
the walk stops, instead of truncating silently.
Part of #2388.
Signed-off-by: Gaurav Dubey <gauravdubey0107@gmail.com>
A URL in a config file became an infra Route -- "an endpoint this service
exposes" -- unless its key was on a short deny list (jwks, registry,
upstream, ...). In 17 infra repositories almost every such Route was a
reference instead: changelog `pull_request` links (deckhouse 6,240), docs
`link`s, alert `runbook_url`s, chart `home`/`source`, kubeconfig
`server`s, Argo `repoURL`s, Helm repository `url`s. Until the previous
commit one root handler took a HANDLES edge to each of them.
The rule is now an allow list: the key's leaf, compared ignoring case,
`_` and `-`, must name an endpoint other systems call on this service --
push_endpoint, post_url, callback(_url), webhook(_url) -- the keys the
deny list's own comment named as the ones to keep.
On real input, infra Routes: deckhouse 6,380 -> 3 (three `webhook`
keys), kubermatic 228 -> 0, prometheus-community/helm-charts 124 -> 0,
gardener 26 -> 0, kubernetes/k8s.io 48 -> 0, onedr0p/home-ops 100 -> 0.
HANDLES and HTTP_CALLS are unchanged in all six.
A new contract test with those reference keys fails on the deny list;
another pins the camelCase and _url endpoint spellings; the #999 guard
(push_endpoint still mints) passes unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A reference written in a doc comment is extracted with the definition it
documents (internal/cbm/doclink.c; the C# scanner doclink_cs.c) and
resolved per file in the pipeline (src/pipeline/doc_links.c,
doc_links_cs.c, doc_links_msbuild.c). A reference that names a definition
of the repository with certainty becomes a MENTIONS edge from the
documented definition (the File node for a file-level comment) with the
properties via, syntax, tier, line and count. Any other reference becomes
a row of the new doc_link_unresolved table with its reason: missing,
ambiguous, external, test_only_target, graph_gap, unparseable or
below_bar_tier. Nothing is guessed. index_status reports the layer in a
doc_links block.
C# binds a cref by the compiler's lookup rule: the first scope with the
name decides, a cref never binds an inherited member, System is outside
the repository. Assemblies come from project files, and MSBuild global
usings (<Using>, ImplicitUsings, Directory.Build.props/.targets, imports,
conditions) are evaluated from the project files' content as data the
indexer has already read; the layer opens no file. Two declarations of a
name in different assemblies make the reference ambiguous.
Each file's scope is stored line-free with its LSP surface ("dl"), so an
incremental run re-resolves the files a change can reach and falls back
to a full run for a change that can re-route references anywhere (a
project file, a global using, a deleted file).
The scanner and the resolver treat the repository as untrusted input:
bounded nesting, no input-driven recursion, work that grows linearly
(held by test seams, never by a clock), no silent limits, and a damaged
or foreign stored scope fails closed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The test-impact engine selects tests by walking the graph, so it can only
be as right as the graph. Three gaps made it wrong:
One identity for a definition's mutually exclusive variants
- A definition written once per #if/#else branch, per #[cfg], per
platform file (foo_linux.c / foo_win.c, platform directories) or per Go
build tag is ONE node. The graph buffer keeps every variant's span in a
`variants` node property (definition labels only), so a change in any
variant maps to the definition.
- Extraction now sees definitions inside conditional blocks where it
skipped them: C++/C#/ObjC class members under #if (one cursor walk,
conditional block types per language in lang_specs), Elixir defs under
if/unless, Fortran .F90/.F95/.F03/.F08, Rust #[cfg] twins (no more
`name#cfg(...)` QNs that lost every call), C platform files and
directories (non-static file-scope functions).
- Haskell: the vendored scanner reads only an #if group's first branch.
The grammar stays upstream; every further branch is parsed as its own
same-length view (cpp_branch_views.c) and adds what the raw parse
missed. haskell/bytestring: no new partial parses, +27 functions,
IMPORTS 322 -> 329, 131 definitions with variant spans.
- Incremental closure repair re-parses every file of such a definition
when one of them changes (variant partners through DEFINES), so a
repair equals a full index.
No cross-language name guesses
- A name-only resolution (unique_name, suffix_match, qualified_suffix,
field_type_hint, and same_module across files that only share a path
stem) never binds a symbol another language defines.
- Bash imports are the files a script sources: every top-level command
was read as an import of its name (`set -e` imported "set", bound to a
TSX method in 137 scripts here).
- This repository: cross-language CALLS 5,916 -> 5,091; the remainder are
C/C++ pairs, builtins and same-stem module collisions (recorded).
SPAWNS edges (per language, like the HTTP client patterns)
- A call that starts a program (subprocess.run, exec.Command,
child_process.spawn, posix_spawn, system, Process.Start, Command::new,
... about 30 languages, through the file's own imports) gets a SPAWNS
edge to a Process node named after the program (literal first word, or
<dynamic>) instead of a CALLS guess. No target resolution. Both
resolvers ask at the same point (parity test).
The engine follows the graph
- Seeds map changed lines through every variant span; a test written
once per #if branch is one test; when a change reaches a non-test
`main`, the tests that spawn processes are reached too (the process
runs main, and main reaches everything).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Markdown docs reference other repo files constantly (a coding-standards
doc links the modules it governs, a README points at entry points) but
none of that surfaced as edges, so fan-in queries were blind to
documentation hubs. In an A/B retrieval benchmark I ran in August on a
docs-heavy repo, the graph's top fan-in answer was off by roughly 17x:
the most-referenced file was a standards doc with 173 references and
zero inbound edges.
New pre-dump pass, pass_doclinks.c, modeled on pass_configlink.c: three
strategies emit REFERENCES_FILE edges between existing File nodes only
(unresolvable targets are dropped, the pass never invents nodes):
inline links, backtick paths, bare path mentions. Targets resolve
against the referencing file's directory and the repo root; repeated
references collapse into one edge carrying strategy, confidence and
count. Registered in the pre-dump sequence after configlink and in the
incremental post-passes; REFERENCES_FILE added to the skill's edge-type
list and the structural/language contract tests.
Tests mirror test_configlink.c (real files in a tmpdir, File nodes in a
gbuf, run the pass, assert edges): inline link, backtick, bare mention,
http/anchor ignored, anchor-suffixed file link, dedupe with count,
relative-vs-root resolution, unresolvable-target guard, NULL repo_path
skip.
A companion change adds the same linking for shell files (source lines
and script invocations); split out to keep each change reviewable.
Validation: scripts/build.sh clean; focused serial runner (doclinks,
configlink, pipeline, edge_structural, lang_contract) 340 passed under
ASan/UBSan; cppcheck clean on the new files with the repo's flags.
Signed-off-by: Clément Deust <clement.deust@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cross-file Constant resolution test walks USAGE edges and calls
cbm_store_find_node_by_id() for each source, which populates the caller's
cbm_node_t with heap_strdup'd fields (store.c:195 via scan_node). It freed
the arrays it owned -- consts and edges -- but never released the per-edge
node, leaking one set of fields per iteration.
LeakSanitizer caught it in every sanitizer-enabled lane while the plain
test shards and both Windows legs stayed green, which is why the local
test-runner could not see it:
test-lsan-macos 417 bytes in 6 allocations
test-unix (ubuntu-latest, 2/3) 401 bytes in 6 allocations
test-unix (ubuntu-24.04-arm, 2/3) same signature
test-diag same signature
Use cbm_node_free_fields(), not cbm_store_free_nodes(). The latter frees
the array pointer itself after releasing each element's fields, so calling
it on a stack-allocated node is a bad free -- ASan aborts with
"bad-free test_lang_contract.c" if you try. cbm_node_free_fields() releases
the strdup'd members and leaves the struct alone, which is the idiom the
neighbouring probes already use (test_edge_types_probe.c:245).
The three other find_node_by_id call sites in this file predate this branch
and already handle their own cleanup; only the new Chialisp contract leaked.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Chialisp commit was formatted with the pinned Homebrew clang-format, which
is right for the production sources but wrong for the three test files it also
touched: tests/ is not in LINT_SRCS, so those files have never been under the
format gate and were already format-dirty on main. Running the formatter over
them rewrote code that had nothing to do with Chialisp — 478 whitespace-only
lines in repro_invariant_breadth.c alone, plus 50 in test_extraction.c and 14 in
test_lang_contract.c — which is a review burden and hides the actual change.
Restore those three files to main's bytes and re-apply only the Chialisp
additions, in each file's own house style. The result is token-identical to the
version that was tested, apart from one em-dash written as `--` in a comment.
Re-verified after the rewrite: the six chialisp extraction tests, the cross-file
Constant graph test, the matrix-B row, the breadth case
(`[INV-BREADTH] PASS chialisp calls=1 callable=1 module=0`), and `make lint-ci`.
The production sources keep their formatting: they ARE in LINT_SRCS, and the
only whitespace they carry is the trailing-comment realignment in cbm.h that
adding a longer enumerator forces, plus the re-indent in extract_defs.c from
moving the name-resolution block into an else branch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only
public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five
reference files from chia-blockchain@main it produced 3-34 ERROR nodes each.
Three defects account for all of them, and each desynchronised the rest of the
file rather than losing one node:
* comments were `/;.*\r\n/` -- CRLF-only, while real files are LF;
* `(include foo.clib)` was rejected because of the dot in the filename;
* `(defconstant NAME <expr>)` accepted only a primitive after the name.
So the grammar is written here instead, in tools/tree-sitter-chialisp/, and
files as first-party: its vendored LICENSE is the project's own, byte-identical
to the repository root, carrying no third-party copyright.
It is deliberately GENERIC -- source_file / list / symbol / string / number /
hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp
form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head
symbols; which lists are definitions is decided in extract_defs.c. A dialect
that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs
no regenerated parser. Acceptance: all five reference files parse with zero
ERROR and zero MISSING nodes; the three defects above are pinned as tests.
Extraction, and the four decisions in it:
* `Constant` is admitted to the cross-file name registry
(cbm_label_is_registry_symbol). Chialisp's condition codes live as
`(defconstant ...)` in a shared .clib and are referenced by name from every
puzzle; labelled but unregistered they are seeded nowhere, and roughly half
of a Chialisp graph would be present as nodes yet never the target of an
edge. All three registry-seeding sites route through that one predicate, so
they cannot diverge. No other language emits `Constant`, so the blast radius
is exactly Chialisp. Pinned by a two-file graph test that goes red when the
label is removed from the predicate.
* `export` / `namespace` are NOT definition heads. `(export foo)` names a
function already defined in the same file, so treating it as a def head
mints a second node for one symbol. They stay in the not-a-call filter.
* A parameter list is a `list` too, but it binds rather than invokes. Without
that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a
phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions.
* `(include *standard-cl-26*)` is a dialect directive, not a file dependency;
recording it would invent a dependency on a file that does not exist.
Three helpers that the defs, calls and unified walks all need -- the quote-data
gate, comment-skipping named-child access, and the Chialisp def-head set -- live
in helpers.c rather than being copied per translation unit. They previously
existed as byte-identical duplicates that had already drifted: extract_defs.c
skipped comments when reading a def head while its mirror in extract_unified.c
did not, so a comment between a head and its name desynchronised defs from call
scope. Both now use the same accessor.
Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87,
first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165.
Distilled from PR #919 by darkverbito, whose extraction logic -- the CLVM
not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and
the include/embed import handling -- is kept and is the substance of this change.
The grammar that PR vendored is not used.
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Escape import aliases and GraphQL operation strings before serializing edge properties, preventing raw control characters and quotes from producing malformed JSON.
Add a reachable single-line GraphQL regression fixture and assert that emitted edge properties remain valid JSON.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: tmonestudio <tmonestudio@users.noreply.github.com>
Three language fixtures assign cbm_project_name_from_path into lp->project each
time they open an indexed store. Teardown frees the last one, so a fixture that
indexes more than once dropped every earlier heap name — 68 allocations across the
run, which the macOS leak lane reports at process exit.
Pre-existing rather than introduced here: none of these files or fqn.c are in this
branch's diff, and the leak lane passed on the previous PR. Adding a suite shifted
what the leak job sees, which is how it surfaced. Fixing it here rather than
recording it, since a gate that only stays green by composition accident is not a
gate.
Freeing before reassigning also makes the ownership obvious at the point it
matters, instead of relying on the reader noticing the teardown three functions
away.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Unblock CI lint so the test job runs. Prefer Homebrew LLVM formatting
on the member IMPORTS retarget. Add contract that
`from ghost_module import nope as alias; alias()` emits no CALLS.
Relabel registry alias pin as adversarial (behavior already on main).
Signed-off-by: 涂名恩 <125283161+Joseph-MingEn@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
resolve_module often lands on the Module for pkg.mod.symbol paths.
When the member Function/Class exists, IMPORTS must target it so
\`from M import f as g\` + g() can CALLS→f (Yui bridge_execute shape).
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: 涂名恩 <125283161+Joseph-MingEn@users.noreply.github.com>
from M import f as g + g() must emit CALLS→f, not invent M.g.
Use resolve_import_node in sequential import maps, and prefer an
exact registry hit when the IMPORTS target is already the def QN.
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: 涂名恩 <125283161+Joseph-MingEn@users.noreply.github.com>
Run the test process under a private temporary HOME and clear inherited CBM_CACHE_DIR so production index paths and legacy fixtures resolve to the same isolated cache.
Make lang-contract fixtures honor an explicit cache override and remove the sentinel tree at process exit, including after early assertion returns.
Validated on Windows with the full runner: 5828 passed, 22 skipped; a 500-project user cache had zero files added or removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: tmonestudio <tmonestudio@users.noreply.github.com>
from m import f as g; g() produced no CALLS edge at all. Two stacked
defects: (1) py_lsp_bind_imports keyed from-style detection on the
local name matching the module-path tail - an alias never matches, so
g was bound MODULE(m.f) and the call missed; (2) even with the binding
fixed, the pass join compares the bare tail of the resolved QN (f)
against the textual callee (g) and dropped the resolution.
Fix: (1) a dotted module path whose tail differs from the local name
is an aliased binding - bind NAMED(qn) and let phase 6 upgrade it to
the registered function/class (from-import alias) or module (import
a.b as z); (2) when the identifier-callee NAMED branch resolves a name
whose textual form differs from the QN tail, emit strategy
lsp_import_alias with the textual name stashed in reason, and gate
that strategy in the join's reason list - the exact pattern already
used by lsp_dict_dispatch and lsp_func_ptr. The parallel path's hash
index misses the alias key and falls through to the same gated linear
scan.
The contract test pins all four import forms (aliased from-import,
plain from-import, single-segment module alias, dotted module alias)
so the flip from MODULE to NAMED binding cannot regress the module-
alias attribute chains.
Closes#988
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
URLs in .pre-commit-config.yaml, .github/workflows and similar CI
configs are repository/action/registry references for the development
toolchain, never endpoints the indexed service exposes. They matched
is_infra_file (.yaml) and passed the #521 value guards, so each minted
a __route__infra__<url> Route node; the route matcher's root-service
heuristic then attached every handler of an ambiguous '/' route to each
junk URL - 6-7 junk HANDLES targets per handler on plain pallets/flask,
with tie-break-dependent winners feeding HANDLES churn between builds.
Deny by file identity (well-known CI/tooling basenames plus .github/,
.gitlab/, .circleci/ trees) in cbm_pipeline_extract_infra_routes, not
by URL shape: deployment configs (Cloud Scheduler push endpoints,
compose) keep minting their genuine endpoints, guarded by an inverse
test.
Closes#999
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_store_find_node_by_id heap-allocates the node's string fields;
the #871 helper left them unfreed, tripping LSan on the Linux legs.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
const foo = require('./foo') emitted a Variable definition for the
binding, so calls to foo() resolved to that same-module Variable (an
import alias, not a definition) instead of following the recorded
import to the exported function - which stayed at zero inbound CALLS
and surfaced as a disconnected Variable node. ESM import bindings never
materialize as Variables; the identifier-bound require form now behaves
identically: extract_js_vars skips the Variable def exactly when the
declarator value is a require() call with a string-literal path (the
form extract_imports records with local_name = the identifier), letting
resolution fall through to the import map. Destructured requires keep
their Variable defs since the import row only records the module.
Closes#871
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A wildcard tsconfig path alias (e.g. "@lib/*" -> "./src/lib/*") matches
any import specifier sharing that prefix, including an unrelated scoped
npm package meant to resolve from node_modules. The direct match
already failed cleanly (no node exists at the substituted path), but
cbm_pipeline_resolve_import_node's Strategy 4 - a generic "strip a
trailing path segment and retry" fallback applied to every language,
not just its intended Rust crate::/self::/super:: case - re-resolves
the truncated import string and can land on the tsconfig's OTHER, bare
alias entry that maps straight to a directory. The resulting QN
collides with that directory's real Folder node (cbm_pipeline_fqn_module
and cbm_pipeline_fqn_folder both dot-join path segments with no type
tag), and Strategy 4 accepted whatever node it found there.
Strategy 3 (symbol-name fallback) already guards its matches with
import_targetable_label(), which excludes Folder/Project/etc. Strategy
1b (resolve_sibling_file, used only for markup/build-tool sibling
imports - SCSS/Just/BitBake/Meson/Pony, none of which are directory-
module languages) and Strategy 4 lacked the same guard; both now use
it too.
Strategy 1 (the primary, direct resolution path) intentionally keeps
no such filter: Go and Java are directory-module languages in this
pipeline (cbm_lang_module_is_dir), so a bare package import correctly
and deliberately resolves straight to a Folder node there - adding the
guard to Strategy 1 broke that (verified via the Go/Java integration
tests) before being scoped down to just the fallback strategies where
the collision actually originates.
Closes#767.
Signed-off-by: Alexandros Pappas <11921291+apappas1129@users.noreply.github.com>
Removes the heaviest vendored grammar (66 MB) along with its language
wiring and test fixtures; the supported-language count moves to 158
across README, site, and manifest. The grammar manifest additionally
records the canonical-source decisions for the five
registry-disagreement grammars and the per-directory license files
restored earlier this week.
A Lerna/Yarn workspaces monorepo where packages/b imports a sibling by package name (@org/a) should resolve via package.json name->dir into a cross-package IMPORTS edge. Reproduction contract; verifies whether the generic workspaces case resolves (cbm_pipeline_resolve_module already looks up bare specifiers in the pkgmap) or exposes a gap.
git is now installed on every CI platform (incl. the Windows msys2 env) and run_git uses portable git -C, so a failed init is a broken-env failure, not a platform skip.
fopen text mode rewrites newlines to CRLF on Windows, making line-ending-sensitive grammars under-extract (below_min/calls_breadth). Open fixtures with wb. Also skip the git-history contract when git init fails (platform limitation) rather than failing.
cbm_mkdtemp on msys2 yields a backslash Windows path; embedding it in the JSON repo_path produced invalid escapes so index_repository failed (count=-1) for every harness contract, plus mixed-separator git -C args. Normalize to forward slashes after mkdtemp — valid in JSON and accepted by Windows file APIs and git.
extract_crashes uses fork/waitpid/<sys/wait.h>, which msys2-clang lacks, so the Windows test build failed with 'sys/wait.h not found'. Guard the include for _WIN32 and run extraction in-process there (fixtures are crash-free; the fork-isolated check still runs on the POSIX CI legs).
GCC at -O1 flags the partially-filled LangFile arrays in contract_all_grammars_in_graph and index_parallel_fixture as maybe-used-uninitialized when passed to lang_index_files (clang does not), failing the -Werror Linux test build. Zero-initialize both; only files[0..nfiles-1] are ever read, so behavior is unchanged.
Index a fixture for every grammar through the full pipeline and assert graph-level invariants: golden node-label histograms, def/node breadth, CALLS and IMPORTS breadth, presence of all 26 pipeline edge types (structural, type-relationship, service/dataflow, and similarity), no-crash, and call resolution. Reproduced quality gaps (under-extraction, import-extraction, and CALLS-resolution) are kept as hard failures for the fix phase rather than skipped. Adds a real-repo scale tier (scale_contract.sh) and ignores local-only private/ scratch.
Gaurav Dubey
Martin Vogel
Clément DeustandClaude Fable 5
tmonestudioandClaude Opus 4.8
涂名恩andCursor
Alexandros Pappas
Martin Vogel