WHY: the check read an import's module path with strcspn, which runs to a NUL, but a tree-sitter source slice need not be NUL-terminated: the first segment is now found with memchr inside the slice. The leading dots move into py_ns_leading_dots; written inline, cppcheck 2.20 (the CI version) took the loop to always end at the slice length and reported three knownConditionTrueFalse findings on the branches that follow it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A top-level call withdrew the whole namespace (UNKNOWN, reason EFFECT): sqlalchemy's sql/__init__.py ends with __go(locals()) and lost all 108 re-exports. The call is now allowed when the callee is the latest binding of its name, a proven same-module function with no star import or unknown effect since; every argument is a safe expression or a bare locals()/globals() the module does not rebind; and the body (nested scopes included) has no global/nonlocal and never names a parameter, a function or class the module defined earlier, the module's own name, or a name that reaches a namespace (globals, vars, locals, exec, eval, setattr, delattr, __import__, sys, importlib, __name__, __dict__, __globals__, __builtins__, builtins, __self__, inspect, f_globals, f_locals, f_back, gc). Each submodule of the package the body imports becomes a package attribute and is shadowed at the call; in a plain module the imports install nothing. Every other call stays a whole-namespace effect. The function table and verdicts live in the extraction scratch; each callee is checked once.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm.h conflict: #2700's cbm_import_binds_runtime_name helper and the facts version 2 comment are adjacent; both kept.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The grammar rejects from __future__ import * as Python does, so the file carries an error node and its whole namespace is a PARSE unknown: a stronger barrier than the UNSUPPORTED statement the previous commit guarded for. That guard could never run and is dropped; the control asserts the parse barrier the parser actually yields.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The namespace facts proved only plain one-line strings safe, so every triple-quoted string statement (a module, class or attribute docstring) was an unknown EFFECT, and from __future__ import x was UNSUPPORTED. A barrier withdraws every name bound before it: sqlalchemy's sql/expression.py lost its 149 re-exports to the docstring after nullsfirst = nulls_first, and a docstring made an otherwise plain class unproven. A string literal now evaluates nothing in any quoting or prefix (r, b, u, rb), implicit concatenations included; an f-string with a replacement field stays an effect (__format__/__str__ may run code). from __future__ import x [as y] binds like a from-import of module __future__; a star import from __future__ is a syntax error and stays unsupported.
CBM_PY_NAMESPACE_FACTS_VERSION moves to 2. The version travels in the lsp_surface py payload and a payload of another version does not decode, so the first incremental run on an index of an older binary falls back to a full re-index (delta.rehydrate_failed -> CBM_PIPELINE_FORCE_FULL_REINDEX) instead of mixing old and new facts. The codec test now spells the current version and rejects a version-1 payload.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
py_lsp.c: the module-member lookup of this branch and the C3 super() lookup of #2696 were added at the same place; both are kept.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A namespace fact stream is checked by cbm_py_namespace_facts_valid, which
accepts a shadow only for the value, decorated, unproven-class and
unsupported reasons. The withdrawn TYPE_CHECKING re-export carried the
compound reason, so the whole stream was rejected and the package published
no names at all (a lazy-module package lost its TYPE_CHECKING exports). It
now carries the unsupported reason: the builder cannot tell which source the
name has at runtime. python_namespace_facts_type_checking_alternatives pins
both shapes as complete, valid streams.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A package namespace read in the type-checker view takes the consequence of
`if TYPE_CHECKING:` as executed (lazy-module packages publish their names
there). The compat shape imports a name there and again in `else: try:
from runtime.place import name`: the module's runtime export comes from the
else branch, so publishing the TYPE_CHECKING source as the re-export target
was a guess. Every name an elif/else import binds now becomes a shadow fact
(the re-export is withdrawn, as for a later assignment); a star import or an
import form the builder does not read there makes the whole namespace
unknown. The branch is walked iteratively.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A package `__init__.py` re-exports its submodules' names (`from .fields
import *`, `from .x import Orig as Renamed`), and callers reach them as
module attributes (`models.CharField()`) or through `from pkg import Name`.
The Python namespace facts already capture each module's top-level bindings
in order; nothing consumed them, so those calls fell to name guesses that
pick the first same-named definition under the package.
- Records: cbm_pxc_collect_all_defs replays each module's COMPLETE fact
stream into re-export, star-import and `__all__` records next to its
defs, including for an import-only `__init__.py` (zero defs). Records
carry the raw module spec; the Python registry resolves it (relative
exactly, absolute by unique scope tail, else unknown).
- Fold: the shared Python registry folds the records once, after finalize
and before the seal, into a sorted per-module export table (binary
search; no lookup in the registration loop). Order-deterministic; a
cycle contributes what it has so far.
- States (conservative): an incomplete fact stream publishes nothing; a
shadow, delete or whole-namespace unknown (a top-level try/if/with)
withdraws the names it covers; a star whose source cannot be enumerated
publishes nothing; a literal `__all__` restricts a star, else public
names only.
- Consult: the Python LSP looks a module attribute up in the export table
on a miss (calls, expression types, callable values) and when binding
`from pkg import Y` where Y is no def of pkg.
- Namespace facts read a module-level `if TYPE_CHECKING:` consequence as
executed: lazy-module packages publish their names there.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The compat shape imports a name under `if TYPE_CHECKING:` and again in the
else branch, usually inside try/except ImportError. The name is bound at
runtime there, so its TYPE_CHECKING import is no annotation-only import:
recording it flagged made every CALLS import map skip the name and lost the
calls and bases that bind through the runtime import (airflow mask_secret,
OpenLineageRedactor(SecretsMasker)). The imports of the elif/else branches
are collected (any depth, iteratively), every TYPE_CHECKING import whose
local name they bind is dropped with them, and a star import there drops the
whole block: those names keep the behaviour they had before #1277.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The import an annotated field's type comes from usually sits under a
module-level `if TYPE_CHECKING:` (to avoid an import cycle).
parse_python_imports walked only the module's direct statements, so that
import was never recorded and the reporter's chain
trainer.strategies.process_batch() lost its type exactly where real code
writes it.
- Extraction records the imports of a module-level `if TYPE_CHECKING:` /
`if <module>.TYPE_CHECKING:` consequence block, flagged
CBMImport.type_checking_only. elif/else branches stay unrecorded, as
every other compound statement's imports do.
- One rule for every CALLS import-map builder:
cbm_import_binds_runtime_name() (false for a flagged import). The
sequential build_import_map applies it in its fast and slow paths; the
shared cbm_pxc_build_import_map (= _ex(..., false)) applies it in both
loops, so no CALLS import map ever binds an annotation-only name.
- The annotation consumers take cbm_pxc_build_import_map_ex(..., true):
the cross-file base/field-type resolution and the Python LSP dispatch.
The sequential dispatch always takes it; the parallel one builds that
map only for a file that records a flagged import and otherwise passes
its CALLS map, which is identical there by construction.
- The Python LSP binds those names into an annotation-only frame consulted
by annotation resolution after the ordinary scope chain: a runtime call
through such a name never resolves.
Tests: python_type_checking_imports_flagged_issue1277 (extraction),
python_type_checking_typed_field_issue1277(_parallel) (pipeline, both
venues; controls: no import_map and no LSP CALLS edge through the name).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A package-relative from-import (`from . import x`, `from .. import x`,
`from . import x as y`) has a module made only of dots. The extractor joined
it to the imported name with another '.', recording "..x" for `from . import x`:
the parent package. The import resolver then missed the real binding and fell
to the symbol-name guess (the lexicographically first same-named definition in
the project), so the import map, and the Python LSP binding that consumes it,
sent every call to an unrelated function at import_map / lsp_callable_alias
confidence. On home-assistant, 7,175 caller/callee pairs bound to another
integration's function (webhook's `async_register` to accuweather's).
Join without the extra separator when the module already ends in '.'.
`from .m import f` (".m.f") and absolute imports are unchanged.
Tests: python_package_relative_import_paths (extraction: the recorded paths);
pipeline_python_package_relative_imports{,_parallel} (all three forms resolve
to the own / parent package, never to the namesake that sorts first, without a
name guess, on both resolvers); pipeline_python_from_import_bare_call_issue1237
{,_parallel} (#1237 guard: a plain from-import and bare calls bind the imported
def and keep it across a re-index that adds a file).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A call no project symbol resolves, whose callee an import binds to an
external library, is now spelled through that import and classified with
the existing service patterns: `send` under
`from paho.mqtt import publish as send` is paho.mqtt.publish (ASYNC_CALLS
to its topic), `navigate` under `import { get as navigate } from 'axios'`
is axios.get (a typed GET HTTP_CALLS edge). Before, such calls had no
service edge at all.
- CBMImport gains `member`: the exported name an ES named import binds
(`get` in `{ get as navigate }`), NULL for default, namespace and other
imports and for every other language. Filled by process_named_imports,
carried through result compaction (and so through spill), counted in the
extraction census with the other import strings.
- One helper, cbm_pipeline_external_import_service, serves both call
resolvers: a per-file index (local name -> binding import), built on the
first unresolved call that asks, then one hash lookup per call.
- Precision: only unresolved calls, after the callee-name, route-suffix and
global-fetch fallbacks; never a relative import, an import an IMPORTS
edge materialized (a project module), a name bound by two different
imports, or a Python callee rebound as a parameter. An HTTP edge needs a
request verb in the spelled name, so urllib.parse.urljoin is not a
request. Python, JavaScript, TypeScript and TSX.
Tests: extraction_imports covers the member and its survival through
compaction; a pipeline test runs both resolvers over a typed GET, a typed
POST and a publish through aliases, and over controls that must stay
silent (urljoin, lodash and json aliases, a local definition and a
parameter shadowing the import, a materialized project package named
httpx). Each guard was reverted once and its control went red.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The __all__ literal guard compared a uint32_t child count against
SIZE_MAX / sizeof(char *), which is always false on LP64 and failed the
GCC build (-Werror=type-limits). One bound now covers both limits on
every platform: the names are counted in an int, and
count * sizeof(*names) <= INT_MAX also fits a 32-bit size_t.
Building one import fact moves into py_ns_import_fact, and the
module-name child is skipped with ts_node_eq: for a plain import the
module is the null node, which equals no child. The loop no longer tests
`from` after the from-only prelude, which cppcheck 2.20 misread as
always false (knownConditionTrueFalse). Emitted facts are unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Capture ordered Python namespace facts and retain them through compact, spill,
and versioned LSP surfaces. Reject unavailable metadata before incremental
reuse and rebuild on Python namespace changes or deletion.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Make rules were read with three gaps:
- A rule naming several targets (`seed-up seed-dev seed-down: ...`,
`kind-% seed-%:`) became one Function named after its first target:
`seed-down` had no node, so nothing could link `make seed-down` and a
search for it found nothing. Every target now gets a Function over the
rule's span (without fingerprint, structural profile or body tokens, so
the copies of one recipe do not read as similar code), and the recipe's
DEPLOYS / SPAWNS edges leave from each of them.
- A target-specific variable line (`kind-%: export X := y`) was taken for
a rule: a phantom Function for its first target, or a variant span
merged into the real rule of that name. Such a line is no rule now. Its
tree-sitter ERROR region is no longer masked, so the file reports it
as partially parsed, as for any other unparsed region.
- `include hack/tools.mk` imported a Folder `hack/tools` (local name
"mk\n\n": the generic import reader kept the directive's newlines and
split at the extension). Make includes are read on their own now: every
name of include / -include / sinclude, outside recipes, imports that
file; a name with a variable reference is left out.
Measured on 17 repositories against an independent reading: make runs
linked 298 of 300 (was 282), all 298 correct; the 2 left pick their
target in an if/else, which stays unknown by design. Graph changes are
confined to Makefiles (gardener +26 targets, 13 phantoms removed;
deckhouse +4), their edges, and two side effects of new nodes:
SEMANTICALLY_RELATED edges elsewhere re-rank, and one Go member
reference in deckhouse lost its unique-name binding to a new Make rule
of the same name. Three runs build identical graphs; index time unchanged (deckhouse 9.61 -> 9.60 s,
gardener 5.24 -> 5.27 s, cluster-api 2.68 -> 2.68 s, medians);
renaming a target, editing a Makefile or an included file leaves the
graph a fresh index builds (gardener's Makefile edit differed before,
through the phantom nodes, and no longer does).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The test-impact engine selects tests by walking the graph, so it can only
be as right as the graph. Three gaps made it wrong:
One identity for a definition's mutually exclusive variants
- A definition written once per #if/#else branch, per #[cfg], per
platform file (foo_linux.c / foo_win.c, platform directories) or per Go
build tag is ONE node. The graph buffer keeps every variant's span in a
`variants` node property (definition labels only), so a change in any
variant maps to the definition.
- Extraction now sees definitions inside conditional blocks where it
skipped them: C++/C#/ObjC class members under #if (one cursor walk,
conditional block types per language in lang_specs), Elixir defs under
if/unless, Fortran .F90/.F95/.F03/.F08, Rust #[cfg] twins (no more
`name#cfg(...)` QNs that lost every call), C platform files and
directories (non-static file-scope functions).
- Haskell: the vendored scanner reads only an #if group's first branch.
The grammar stays upstream; every further branch is parsed as its own
same-length view (cpp_branch_views.c) and adds what the raw parse
missed. haskell/bytestring: no new partial parses, +27 functions,
IMPORTS 322 -> 329, 131 definitions with variant spans.
- Incremental closure repair re-parses every file of such a definition
when one of them changes (variant partners through DEFINES), so a
repair equals a full index.
No cross-language name guesses
- A name-only resolution (unique_name, suffix_match, qualified_suffix,
field_type_hint, and same_module across files that only share a path
stem) never binds a symbol another language defines.
- Bash imports are the files a script sources: every top-level command
was read as an import of its name (`set -e` imported "set", bound to a
TSX method in 137 scripts here).
- This repository: cross-language CALLS 5,916 -> 5,091; the remainder are
C/C++ pairs, builtins and same-stem module collisions (recorded).
SPAWNS edges (per language, like the HTTP client patterns)
- A call that starts a program (subprocess.run, exec.Command,
child_process.spawn, posix_spawn, system, Process.Start, Command::new,
... about 30 languages, through the file's own imports) gets a SPAWNS
edge to a Process node named after the program (literal first word, or
<dynamic>) instead of a CALLS guess. No target resolution. Both
resolvers ask at the same point (parity test).
The engine follows the graph
- Seeds map changed lines through every variant span; a test written
once per #if branch is one test; when a change reaches a non-test
`main`, the tests that spawn processes are reached too (the process
runs main, and main reaches everything).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Ten files changed on both sides; every conflict was a union of independent
additions:
- workflows (_lint.yml, _test.yml): keep the branch's lane `if:` and main's
pinned ubuntu-24.04 runner with its reason.
- internal/cbm/cbm.h: main's callable-identity and module-doc fields plus the
branch's configured-test fields.
- Makefile.cbm: both new test files; prod_lsp_all.o keeps main's
.build-config dependency and the branch's coverage flags.
- scripts/test.sh: main's setup-scripts contract becomes Step 0y3, ahead of
the branch's 0z series and the contracts-only exit.
- src/foundation/subprocess.c: main's cbm_posix_close_fds_from (it closes the
error descriptor too).
- src/mcp/mcp.c: main's repository default base for detect_changes, with the
early returns released through the branch's detect_release.
- src/pipeline/lsp_resolve.h, tests/test_main.c: both includes, suites and
start-up steps (main's git environment reset first).
- scripts/memory-core-baseline.txt: recounted on the merged tree and lowered
(mcp.c 773, config_toml_edit.c 73).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve the original staged graph and LSP work separately from the
smart-CI continuation. This is an unfinished checkpoint; final
integration and cross-platform verification remain outstanding.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve import kinds and resolve class imports through Composer prefixes
to the existing target class file, including autoload-dev mappings.
Refs #1186
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Vue-admin wrapper pattern -- `const api = axios.create({ baseURL:
'/api' }); export default api` in one module, `api.get('/orders')` in
another -- produced no HTTP_CALLS edge, so cross-repo-intelligence found
nothing to link to the backend's GET /api/orders.
Root cause: nothing recorded that `api` is an axios instance. The call
`api.get('/orders')` fell through to the route-registration suffix
fallback (`.get` + a '/'-leading argument), which treats it as an Express
`app.get` REGISTRATION: a CALLS edge to a phantom server-side Route
`/orders` (via=route_registration), and never an HTTP_CALLS edge. The
base path was unknowable anyway.
Fix, reusing HTTP_CALLS/Route (no new labels):
- extraction (JS/TS/TSX/ArkTS): a module-level binding initialised by
`axios.create(...)` records http_client="axios" and the literal
`baseURL` (NULL when absent or non-literal, never guessed); the Module
def carries the same pair when its default export is such a client
(`export default api` / `export default axios.create(...)`). ES default
imports are flagged (CBMImport.is_default).
- both call resolvers (pass_calls.c, pass_parallel.c), before the #523
callee-spelling check and the route-registration fallback: a
`<recv>.<verb>(path)` whose receiver binds -- same file, named import, or
default import -- to a client emits HTTP_CALLS to base + path (axios
combineURLs: trailing '/' trimmed, absolute URLs kept as-is, path kept
when the base is unknown). Shared helper
cbm_pipeline_http_client_call_url keeps the two paths identical.
- incremental: the client pair joins the file's LSP surface (only for files
that have one, so every other surface sha is unchanged), so a baseURL
edit pulls importers into the closure; delta preseed proxies carry the
two keys so a caller re-resolved alone still sees its unchanged wrapper.
- result_compact relocates the two new strings (ASan caught the dangling
worker-arena pointer).
Look-alikes (`factory.create({ baseURL })`, a plain object's `.get`) are
unchanged.
Test: pipeline_axios_wrapper_baseurl_composes_http_calls_issue1916 --
default/named/same-file/dynamic-base/look-alike cases, sequential vs
parallel resolver equality, and same-DB closure-repair reindex (wrapper
edit, then caller-only edit) converging with fresh indexes. RED on origin/
main (seq.wrapper_http == 0); each of the parallel, surface and preseed
pieces reverted alone turns its own assertion RED.
Fixes#1916
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Vue change (#1852) built the shared embedded-script seam — the
included-ranges parse over the original buffer and the lang=/src=
resolver — but gated structural extraction on the host being Vue. A
Svelte component, an HTML page or an Astro file therefore still yielded
only File + Module nodes and IMPORTS edges from its <script> blocks: no
Function definitions, no CALLS, and a false dead-code verdict for
anything used only from a component.
Widen the seam to every host that declares an embedded spec:
- Structural extraction (definitions + calls) runs for every embedded
block; the per-host boolean is gone.
- The Vue attribute resolver becomes script_embedded_language() and
runs for all four markup hosts. It starts from the spec row's language
instead of hard-coding JavaScript, and learns HTML's type= attribute:
absent or empty, module, text/javascript and application/javascript
pass; application/json, importmap, text/x-template and the like bail,
so data blocks cannot leak garbage definitions into the graph. src=
and unsupported lang= values bail as before. CFML's <cfscript> carries
no attributes and keeps its spec-fixed language.
- Astro's frontmatter fence and <script> bodies default to TypeScript:
Astro type-checks both as TS with no lang= attribute to say so, and
the fence cannot carry one. The TypeScript grammar parses untyped
JavaScript unchanged.
Tests: the #1852 control that pinned Svelte/HTML/Astro at zero functions
and zero calls now asserts the real symbols; stay-at-zero controls cover
type="application/json" and <script src> in every host; both Svelte
blocks (context="module", and Svelte 5's bare module) contribute in
host-file coordinates; HTML's accepted type= forms and Astro's TS
frontmatter are covered; an end-to-end CALLS edge inside a .svelte file
is asserted. The Svelte and Astro label goldens gain the Variable their
fixtures always declared.
Verified on a 213-component Vue corpus: node counts per label and per
host language, edge counts and trace_path output are identical before
and after.
Fixes#1807
Signed-off-by: Brian McNaboe <bmcnaboe@gmail.com>
Fixes#2010. Refs #1997.
Indexing a 14k-file TypeScript repository holds 13.3 GB resident at
post_extract on Linux aarch64, and about 3.1 GB of it is AST traversal
scratch that no CBMFileResult points at.
ts_nstack_init cut its stacks from ctx->arena, which is result->arena. In
the parallel path that arena is stored into ec->result_cache[file_idx]
(pass_parallel.c:963) and freed only after parallel_resolve and the infra
passes (pipeline.c:1324), so a structure written for a one-file lifetime is
in fact held for the whole index, once per file, concurrently. The header
claimed otherwise and was wrong: "freed when the arena is destroyed at end
of file extraction".
The largest instance is channels. CHAN_STACK_CAP is 4096 and sizeof(TSNode)
is 32, so scan_string_consts_js (extract_channels.c:104) and
extract_channels_js (:376) take 128 KB each, per JavaScript or TypeScript
file. Across 12,673 such files that is 3,168 MB, and it produced 23 Channel
nodes out of 164,684.
CBMExtractCtx gains a scratch arena, created and destroyed by the
cbm_extract_file_ex call that builds the context. Nothing else changes about
where anything lives: ctx->arena still owns every string a CBMFileResult
points at, and ctx->scratch owns only traversal stacks, which nothing points
at.
Nothing a CBMFileResult points at moved. A context built without a scratch,
and a file whose scratch arena fails to allocate, both fall back to
ctx->arena, which is exactly the behaviour that shipped before this change,
so no path can be handed a NULL arena and silently lose nodes.
The signature change is what makes the mistake unrepresentable.
ts_nstack_init takes the context rather than an arena, so handing it
ctx->arena is an incompatible pointer type; push and push_children read the
arena the stack recorded, so a stale argument is an arity error. Both fail
under -Werror rather than silently retaining memory.
The 55 init sites are NOT one uniform sed. 29 pass ctx->arena and 26 pass a
local alias, and that alias is shared with result allocations: in
parse_zig_imports (extract_imports.c:1093) the same `a` feeds cbm_node_text,
strip_quotes, path_last and cbm_imports_push (extract_imports.c:1101, 1106,
1108, 1109). Rebinding it would have moved import names and module paths
into memory that dies with the call while the result kept pointing at them.
Each site was edited on its own and no alias was rebound. Re-derive the set
with:
grep -rn 'ts_nstack_init' internal/cbm/extract_*.c
grep -rn 'ts_nstack_push' internal/cbm/extract_*.c
parse_lisp_imports used its alias only for the stack, so the alias goes too.
push_nested_class_nodes and push_class_body_children took an arena purely to
reach the stack, so they take the context now; the one call site passes ctx.
Two measurements, from two different builds, and they are not the same
number.
A macOS census build, which instruments every cbm_arena_alloc and sums the
blocks behind every live CBMFileResult arena, reports the retained per-file
arena set dropping from 11,226 MB mapped to 5,680 MB, and blocks from 48,045
to 34,186. Those are mapped bytes, not resident: that build logs
mem.allocator.bound_populations_only, so untouched block tails never commit.
The resident figures come from Linux aarch64, gcc 13.3, static build, where
the build logs mem.allocator.owned classes=all and mapped equals resident.
Five runs per side on the full 13,999-file tree: mem.collect post_extract
rss_mb 13293 -> 6752 (-49.2%), mem.phase peak_mb 14840 -> 8565 (-42.3%).
nodes 164684, parse_partial 280 and registry entries/defines/imports
787954/804999/100027 identical on all ten runs; edges 835869-835874 against
835867-835881, overlapping ranges, so run to run noise. Wall clock means
35651 ms against 30502 ms, but the ranges overlap and the host had other
load, so the only supportable claim is no regression. ObjectScript Studio
Export on a small fixture: nodes 18, edges 25, both sides.
Scratch block size is 512 KB, chosen from measurement. arena_grow fires on
one file in 12,000 at both 512 KB and 1 MB, and constantly at 256 KB, where
the two channel stacks alone are exactly 262144 bytes. 512 KB therefore
costs half the resident block per worker for the same growth behaviour, and
stays under MI_LARGE_MAX_OBJ_SIZE in the vendored mimalloc, which is
MI_LARGE_PAGE_SIZE/8, exactly 524288 bytes (types.h:426, with
MI_ENABLE_LARGE_PAGES defaulting to 1 at :115 and not overridden by this
build), so a 512 KB block is still bin-allocated from a 4 MiB large page.
Growth is not free at that size for the same reason: arena_grow doubles to 1
MiB, above that bound and therefore a singleton OS allocation. One file in
twelve thousand pays it, which is why it is accepted. The largest single
file's scratch high water mark on this corpus is 33.5 MB, now transient
rather than retained.
Tests. Nothing in the suite asserted on result->arena size, block count or
total_alloc after cbm_extract_file, which is why this went unnoticed for the
life of extract_node_stack.h. traversal_stack_not_in_result_arena_issue2010
pins the byte budget: extracting "export const x = 1;" as TypeScript charged
365984 bytes to the result arena before and charges 87456 after, a
difference of 278528, exactly the two 4096-entry channel walks plus the
512-entry ES import walk at 32 bytes per TSNode. Of the 87456 that remain,
7680 is the defs item array at GROW_ARRAY's starting capacity of 32 times
sizeof(CBMDefinition) 240 and the other 79776 is everything else the
extraction interns, none of it traversal scratch; the 128 KB bound is
therefore above the new figure with room and a factor of four below the old
one. extract_traversal_stacks_come_from_ctx_scratch_issue2010 in test_mem.c
pins the lifetime, since a byte budget alone would also be satisfied by
shrinking CHAN_STACK_CAP: it builds the extraction context directly over two
arenas it owns, runs cbm_extract_channels, and asserts the scratch took at
least the 262144 bytes of the two walks while the result arena did not.
extract_c_macro_hidden_call_survives_preprocessed_pass_issue2010 is not a
scratch test. pp_ctx carries ctx->scratch so every context in the file is
uniform, but nothing reads it there: pp_ctx reaches only cbm_extract_unified
and cbm_run_c_lsp, and neither extract_unified.c nor anything under
internal/cbm/lsp/ includes extract_node_stack.h, so no traversal stack is
built on that path today. The test guards the macro-expansion path itself,
which had no assertion on a call that exists only after expansion.
ObjectScript composite extraction is already covered by
pipeline_objectscript_export_preserves_calls_sequential_parallel and
iris_export_xml_multi_class, so no new test is added there; a per-call arena
is correct on that path by construction, since each call owns and destroys
its own.
Local: scripts/test.sh, 7558 passed, 0 failed, 8 skipped, 139 suites.
Alternatives considered
Thread-local scratch arena, reset per file, with a depth guard for nesting.
Rejected on three counts. This codebase never uses thread-local storage for
owned bulk memory; every existing _Thread_local here is a small cache, a
guard or a log buffer, and every per-worker resource is a field in a context
struct. It would have needed cbm_arena_reset, which has no production caller
and a documented subtlety with grown block sizes (arena.c:213-217). And
extraction threads are created per cbm_parallel_for call and joined
(worker_pool.c:71,88) with no thread-exit hook in this project's own code,
so the arena would have leaked once per thread per dispatch and turned the
leak lane red.
Save and restore mark and release on the arena. Rejected because a per-call
arena already makes nesting correct by construction, and a mark type plus
two functions would have to be added to both arena headers.
Per-walk arenas, one per traversal rather than one per file. Rejected on
cost: 55 sites would each need an explicit span, for the same bound.
Right-sizing the initial capacities alone. CHAN_STACK_CAP at 256 rather than
4096 would cut those two sites from 3,168 MB to about 198 MB, which is real,
but it tunes the constant while the lifetime defect remains at all 55 sites,
and growth is not free: 256 doubling to 4096 abandons 122,880 bytes, and one
file in this corpus took 33.5 MB of stacks. The cap and the scratch block
size are coupled, which is a further reason to change one at a time. Worth
doing separately.
Routing to a scratch inside extract_node_stack.h while keeping the arena
parameters. Rejected because a parameter that is ignored lies about the API,
and the alias hazard above would stay a review obligation instead of a
compile error.
Freeing the per-file result arena earlier. Rejected: the results are read by
parallel_resolve and the infra passes. That is the retention half #1925
addresses, and this change is independent of it.
Signed-off-by: Daniel Ochoa <daniel.ochoa@closedloop.ai>
`import "C"` is not an import: "C" is reserved by go/build and no package
with that path can exist — it only tells the toolchain to compile the
preceding comment as C. cbm emitted it as an ordinary import, so the
import resolver fell through to its symbol-name fallback
(cbm_pipeline_resolve_import_node Strategy 3, pass_pkgmap.c), matched the
literal name "C" against every project node called C, and picked the
lexicographically smallest one.
Measured on a real Go+C repo: all 27 files carrying `import "C"` pointed
their IMPORTS edge at the same unrelated `C` member of a test helper —
27 false edges, and 27 files whose import map named a package that does
not exist.
Skip the spec in parse_go_import_spec. Node counts are unchanged and the
file's real imports are untouched; only the false edge disappears (a
3-file fixture goes 22 edges -> 21, 14 nodes -> 14; the real repo goes
27 -> 0 such edges).
Reproduce-first: go_cgo_pseudo_import_dropped is RED without the
extractor change (the "C" import is still in imports[]) and GREEN with
it, while still asserting the file's real `fmt` import survives.
Signed-off-by: Ilya Brykau <ilya.brykau@orca.security>
Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only
public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five
reference files from chia-blockchain@main it produced 3-34 ERROR nodes each.
Three defects account for all of them, and each desynchronised the rest of the
file rather than losing one node:
* comments were `/;.*\r\n/` -- CRLF-only, while real files are LF;
* `(include foo.clib)` was rejected because of the dot in the filename;
* `(defconstant NAME <expr>)` accepted only a primitive after the name.
So the grammar is written here instead, in tools/tree-sitter-chialisp/, and
files as first-party: its vendored LICENSE is the project's own, byte-identical
to the repository root, carrying no third-party copyright.
It is deliberately GENERIC -- source_file / list / symbol / string / number /
hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp
form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head
symbols; which lists are definitions is decided in extract_defs.c. A dialect
that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs
no regenerated parser. Acceptance: all five reference files parse with zero
ERROR and zero MISSING nodes; the three defects above are pinned as tests.
Extraction, and the four decisions in it:
* `Constant` is admitted to the cross-file name registry
(cbm_label_is_registry_symbol). Chialisp's condition codes live as
`(defconstant ...)` in a shared .clib and are referenced by name from every
puzzle; labelled but unregistered they are seeded nowhere, and roughly half
of a Chialisp graph would be present as nodes yet never the target of an
edge. All three registry-seeding sites route through that one predicate, so
they cannot diverge. No other language emits `Constant`, so the blast radius
is exactly Chialisp. Pinned by a two-file graph test that goes red when the
label is removed from the predicate.
* `export` / `namespace` are NOT definition heads. `(export foo)` names a
function already defined in the same file, so treating it as a def head
mints a second node for one symbol. They stay in the not-a-call filter.
* A parameter list is a `list` too, but it binds rather than invokes. Without
that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a
phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions.
* `(include *standard-cl-26*)` is a dialect directive, not a file dependency;
recording it would invent a dependency on a file that does not exist.
Three helpers that the defs, calls and unified walks all need -- the quote-data
gate, comment-skipping named-child access, and the Chialisp def-head set -- live
in helpers.c rather than being copied per translation unit. They previously
existed as byte-identical duplicates that had already drifted: extract_defs.c
skipped comments when reading a def head while its mirror in extract_unified.c
did not, so a comment between a head and its name desynchronised defs from call
scope. Both now use the same accessor.
Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87,
first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165.
Distilled from PR #919 by darkverbito, whose extraction logic -- the CLVM
not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and
the include/embed import handling -- is kept and is the substance of this change.
The grammar that PR vendored is not used.
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #1412 by @allanoepping. Fixes a real recall hole: tag-dialect
.cfc components were routed to the cfscript grammar (extracting nothing),
and <cfscript> bodies inside tag-dialect files are opaque cf_script_content
to the cfml grammar — whole legacy components produced zero functions.
Fix 1 — his .cfc dialect sniffer, nearly verbatim: cbm_disambiguate_cfc()
reads a 16K head via cbm_fopen; <cfcomponent>/<cffunction> anywhere means
tag dialect; a leading <cfscript (after whitespace and <!--- ---> comments)
or no markup at all keeps the old CFSCRIPT routing, so script-dialect files
and read failures behave exactly as before. Wired in detect_file_language
beside the existing _m/_cls/_inc sniffers, with his 8 direct tests.
Fix 2 — his embedded-defs capability, re-based onto the shared machinery
that superseded his mechanism mid-flight (#1852, merged after his last
rebase): a cfml_embedded_imports spec row (cf_script_tag/cf_script_content
-> CFSCRIPT), CFML added to the embedded-imports dispatch, and the
extract_structure gate extended to CFML. Included ranges on the original
buffer give absolute coordinates for free, so his line-number assertions
hold verbatim, and calls inside blocks become extractable as a bonus. His
~90-line slice+shift implementation is deliberately NOT carried — it
mirrored the sanctioned shape of its day, which #1852 replaced.
One correctness guard the port needed: the Vue attribute resolver is now
gated to Vue — cf_script_tag carries no lang=/src= attributes, and running
the resolver would have silently rewritten CFML's embedded language from
CFSCRIPT to the resolver's JavaScript default.
His embedded extraction test (line remap + <cfsetting> ERROR-cascade case)
carries over verbatim and binds: reverting the spec/gate wiring fails it at
ASSERT(has_def(r, "Function", "greet")). Suites: extraction, language,
edge_imports, lang_contract, grammar_labels, grammar_regression — 637/0.
Co-authored-by: Allan Oepping <allan@intellipay.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Vue single-file components yielded a Module node and IMPORTS edges but no
Function definitions or CALLS edges from the `<script>` block, so a component's
call graph was empty (#1410).
Adds a control test pinning current behaviour for Svelte, HTML and Astro, which
still extract imports but no functions or calls. That is deliberate scoping,
not an oversight: it documents the remaining gap rather than hiding it, and it
proves this fix does not leak into sibling languages. Svelte is tracked by
#1807 and will be a follow-up.
Re-opened from #1843, which carried no Signed-off-by and so failed DCO. The
tree is byte-identical to that commit; only the sign-off was added.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Routing .ets through the plain TypeScript grammar produced no definition
node for any @Component struct (0/19 across the OpenHarmony sample corpus),
parsed struct members as statement labels and build() as a call expression,
and registered ArkUI builtins (Column, Row, ListItem, ...) as user function
definitions with 28% fabricated cross-file CALLS edges. This adds a
dedicated ArkTS grammar that keeps everything TypeScript already handles
and models the ArkTS syntax TS cannot parse.
Grammar (tools/tree-sitter-arkts/, generated parser vendored to
internal/cbm/vendored/grammars/arkts/, ABI 15 via tree-sitter-cli 0.25.10):
first-party derivative of tree-sitter/tree-sitter-typescript v0.23.2
@ 75b3874edb2d (the commit our vendored typescript/tsx come from) on the
tree-sitter-javascript v0.23.1 base @ 3a837b6f3658 (the version upstream's
package-lock pins). LICENSE carries the upstream MIT text verbatim plus a
derivative notice; the external scanner is a symbol-rename trampoline over
a byte-identical copy of the already-vendored typescript scanner header.
ArkTS additions: decorated struct component declarations with class-shaped
bodies; UI-DSL trailing-closure component calls with attribute chains after
the closing brace; decorators on function declarations (@Builder,
@Extend(T), @Styles); @Extend/@Styles leading-dot attribute-chain bodies;
anonymous stateStyles style blocks; import lazy named imports.
Wiring: CBM_LANG_ARKTS enum + lang spec (struct_declaration as a class
kind labeled Struct, public_field_definition fields), .ets extension,
LANG_NAMES/userconfig entries, grammar shim, and TS-family joins across
the extractors and the name-resolution guards (parse_es_imports, JS
keywords/test-file/func-kind/scope tables, TS heritage/param/receiver/
namespace/type-ref/channel/usage paths, js_ts_family and the weak-method-
match suppression). Field definitions now also carry their decorators, so
@State/@Prop/@Link members are findable with their roles.
Tests: grammar corpus (35 cases), per-language regression + label golden +
import net + extension cases, seven ArkTS extraction tests including the
phantom-builtin regression (ArkUI builtins must never be definitions), and
capability-ledger / call-node-manifest / call-argument-matrix rows. On the
14-file OpenHarmony sample corpus the TS-grammar baseline had 12/14 files
with parse errors (231 ERROR nodes); the arkts grammar parses 14/14 clean,
and an end-to-end index yields all 19 components as Struct nodes, 68
methods, 119 fields, and zero phantom builtin definitions.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Remove leftover getenv(CBM_DBG_*) debug fprintf blocks in extract_calls.c/pass_calls.c (cppcheck always-true). Convert parse_lisp_imports to a stack-based walk (no recursion → no NOLINT, avoids deep-nesting overflow). Apply Homebrew clang-format to all touched sources. Local suite 5562/0, lint-ci clean.
Imports: per-grammar parsers for hare/pascal/powershell/scheme/racket/starlark/tcl/teal/zsh + css/html/cmake/bitbake/kconfig/gn/just/nix/jsonnet/pkl/nickel/thrift/capnp/d/tablegen/crystal/fsharp/ada; node-extraction for D/PowerShell/Pascal/Julia/F#; INHERITS for those. Pipeline edges: register Variable/Field defs for WRITES/READS resolution; CONFIGURES pass consuming env_accesses; DEPENDS_ON from go.mod/requirements.txt; SQS dot-form QN. Golden updates: julia +Class, just +Function (correct new extraction).
parse_generic_imports only scans direct children of the root, which missed three grammars: Dart wraps imports in import_or_export (the old dispatch matched the never-emitted import_declaration), Haskell nests import nodes under an imports container, and Zig's @import is a builtin_function nested inside a variable_declaration. Add dedicated parsers (dart finds the nested string_literal URI, haskell descends the imports container reusing the module field, zig DFS-finds @import/@cImport builtin_functions) so all three extract their imports.
Several grammars (clojure/racket/scheme, ada/pascal/fortran/fsharp, cairo/d/odin/squirrel/rescript, and the hlsl/ispc/slang C-family) extracted only a Module node. Add per-grammar def-name resolution in extract_defs.c with matching lang_specs entries so their functions reach the graph, and descend Kotlin import_list/import_header in extract_imports.c so Kotlin imports are captured. The vendored grammar MANIFEST documents each custom case.
R had no import extraction at all (not in the import dispatch), so library(),
require(), source() AND box::use() all produced zero IMPORTS edges; and
module\$fn() calls were dropped because the call-callee resolver did not handle
R extract_operator function nodes.
#218: add parse_r_imports — recursively scan for call nodes and emit IMPORTS for
library/require/requireNamespace/loadNamespace/source (first arg) and for each
box::use(mod[syms], pkg/path[syms], ...) argument (module path = the spec text
before the [symbols] list). AST node types confirmed via a tree-sitter-R probe.
#219: handle extract_operator (R \$) as a call function node in
extract_callee_from_fields → emit "module.fn" so module\$fn() yields a CALLS
edge like other member calls.
Tests cover box::use + library + source imports and the \$-call; r_collect_imports
added to the recursion whitelist.
Format drift had accumulated across recent commits (the dry-run lint enforces clang-format-20, which had not run on these commits yet). No functional changes — whitespace/layout only, verified by a clean cppcheck + clang-format + NOLINT lint-ci pass.
tree-sitter's ts_node_child(node, i) is O(i) — it walks the child iterator from the first child on every call — so the common 'for (i=0; i<count; i++) ts_node_child(node, i)' idiom is O(n^2) over a node's children. On a file whose top level is very wide (e.g. a 580k-line fixture parsed as hundreds of thousands of sibling comment nodes) this made extraction run for over an hour.
Add ts_nstack_push_children(), which enumerates a node's children in a single O(n) cursor pass, and use it in the whole-tree DFS walkers (calls, channels, env-accesses, imports, type-assigns). Convert the module-level variable walk to a cursor too, and add cbm_is_module_level_p() so callers that already hold the parent node skip the O(n) ts_node_parent() rescan.
Indexing microsoft/TypeScript drops from ~5100s to ~50s with identical graph output.
Pure rename — no logic change. The header is consumed only by
internal/cbm/extract_*.c, depends on tree_sitter/api.h, and is purpose-
built for AST traversal during extraction. The "extract_" prefix makes
the scope explicit and keeps the file naming consistent with its
neighbours (extract_calls.c, extract_imports.c, ...).
- internal/cbm/ts_node_stack.h -> internal/cbm/extract_node_stack.h
- Include guard CBM_TS_NODE_STACK_H -> CBM_EXTRACT_NODE_STACK_H
- Header banner comment updated
- 9 call sites in internal/cbm/extract_*.c updated to the new path
Full suite: 2824 passed, 0 failed.
Generalizes #224's Svelte/Vue script-block import extraction into a
spec-driven embedded-language pattern that also covers HTML and Astro,
so any future host grammar that keeps sub-language source as raw text
gets coverage with a one-line declaration.
The existing CBMLangSpec already exposes the per-language tree-sitter
grammar via cbm_ts_language(); add a parallel embedded_imports field
that lists which AST node types hold which sub-language. The new
generic walker in extract_imports.c reads that list, locates each
embedded content node in the host AST, re-parses the slice with the
embedded grammar resolved through cbm_ts_language(), and runs the
standard ES import extractor (walk_es_imports) over the inner AST.
No grammar symbols are referenced from the extraction file.
- internal/cbm/lang_specs.h: add CBMEmbeddedLangSpec + embedded_imports
field on CBMLangSpec.
- internal/cbm/lang_specs.c: declare vue/svelte/html/astro embed lists
({"script_element", "raw_text", CBM_LANG_JAVASCRIPT}); set the new
field on those four spec entries; explicit NULL initializer on every
other entry to satisfy -Wmissing-field-initializers.
- internal/cbm/extract_imports.c: add embedded_collect_content_nodes
+ parse_embedded_imports; route SVELTE/VUE/HTML/ASTRO through the
generic helper.
- tests/test_extraction.c: cover Svelte (with and without <script>),
Vue, and HTML.
Full suite: 2824 passed, 0 failed.
Closes#223. Closes#224.
Co-authored-by: James <jmcmacnz@gmail.com>
AST traversal functions used fixed-size TSNode stack[] arrays. When the
DFS stack filled up, the child-push loop exited silently, dropping
entire subtrees without warning (e.g. 600 TS imports → 511, 130 Express
routes capped at 512). Reported in #199; also addresses #213 (large
TypeScript files producing zero nodes from stack exhaustion) and #215
(SEGV in template calls from stack overflow).
Adds ts_node_stack.h — a growable stack backed by the existing arena
allocator. Initial capacity matches previous fixed caps so small files
allocate no extra memory; doubles on overflow instead of truncating.
Old blocks are abandoned in the arena and freed on arena_destroy at
the end of file extraction (no realloc, no mixed lifetimes).
Applied to all 14 TSNode fixed stacks across 9 extraction files
(extract_calls, extract_channels, extract_defs, extract_env_accesses,
extract_imports, extract_semantic, extract_type_assigns, extract_type_refs,
extract_usages). walk_defs (uses walk_defs_frame_t, different struct
type) deliberately left as-is.
7 regression tests in tests/test_stack_overflow.c cover TS imports >512,
JS/Python calls >512, Go calls >1024, Express routes >150, deeply
nested calls, and YAML vars >256.
Cherry-picked from #217 with merge-conflict resolution against post-#206/#207
main:
- Makefile.cbm: keep TEST_STACK_OVERFLOW_SRCS plus main's new
TEST_ZSTD_SRCS / TEST_ARTIFACT_SRCS.
- extract_channels.c: keep main's scan_string_consts_js name (main also
added scan_string_consts_python after the PR was opened); declare
CHAN_STACK_CAP in the enum block.
- test_stack_overflow.c: replace sprintf with snprintf using the actual
remaining buffer size, since macOS clang rejects sprintf and naive
snprintf with sizeof(p) on a char* fails -Wformat-truncation.
Closes#199, #213, #215. Closes#217.
Martin Vogel
Brian McNaboe
Daniel Ochoa
Ilya Brykau
Martin Vogel
Ahmed Mohammed
