The #961/#1989 rescue re-adopts definitions from the preprocessed second
pass when they sit in a raw parse error, and drops a copy only when its QN
equals a raw one. When the two parses scope the same definition
differently, both copies survived:
- Linux lib/tests/randstruct_kunit.c: a macro-generated struct body makes
the raw parse nest every later struct inside one ERROR region; the flat
expanded copies were adopted next to the mis-nested raw ones (5
duplicate groups).
- coreclr/rapidjson: the expanded parse nests utilcode.h classes wrongly
or loses rapidjson's scopes, and both copies survived.
One rule at one decision point: an expanded and a raw definition with the
same label, name and start line but different QNs are one definition. The
raw copy wins and whatever the expanded pass attributed to the expanded QN
moves to it -- unless
- the raw copy's nesting exists only because of a raw ERROR region that
the expanded parse resolved (the randstruct shape, six conditions in
cbm_misnested_raw_twin): then the flat expanded copy replaces the raw
one and the raw children (Fields, nested defs, call/usage scopes) are
re-parented to it; or
- the raw copy's QN is already held by a raw definition of another label
(coreclr inc/holder.h: the raw parse reads a FORCEINLINE constructor as
a Function of the class's scope-less QN): then the expanded copy keeps
its own, correct QN instead of vanishing into the Function's node.
Proof (macOS arm64):
- randstruct_kunit.c: 5 -> 0 duplicate groups, 12 Fields re-parented to
the flat structs, CALLS/USAGE/WRITES unchanged.
- dotnet/runtime native subset (src/coreclr + src/native): 700 -> 183
groups; the remaining groups are no raw/expanded pairs at one site.
- redis: node and edge sets byte-identical.
- New tests (RED without the change): extraction
extract_c_misnested_raw_struct_replaced_by_flat_copy,
extract_cpp_preprocessed_rescue_skips_raw_def_on_same_line,
extract_cpp_rescue_keeps_expanded_class_when_raw_qn_is_taken (fixture
delta-minimized from holder.h).
- conditional_variants, extraction, c_lsp, pipeline, edge_types_probe,
parse_coverage, lsp_resolution_probe: 1854 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
42a93202 gave a non-static C-family function under a platform directory
(arch/x86/, src/win/, ...) or file suffix a platform-neutral QN, so the
platform variants of one function shared a node. On the Linux kernel that
merged only 644 of 17,413 such definitions; the other 16,769 lost their
path for nothing (`arch/arm64/mm/init.c` mem_init became
`arch.mm.init.mem_init`, and a lookup by its path QN found nothing),
arches missing from the token list kept their paths, and calls bound to
merged nodes produced impossible arch -> other-arch CALLS.
Every definition keeps its path-derived QN again (the v0.11.0 identity).
The platform-neutral QN is now a key: definitions carrying one get a
`variant_key` node property, and nodes of a project sharing a key are one
variant group. The test-impact walk (store_impact `follow_variant_keys`,
enabled by the test-impact engine) treats a reached variant as reaching its
whole group, at seeding and at every step, so a change confined to the
Windows variant still selects the tests that reach the POSIX variant's
callers. The key is a pure function of a definition's own path, name and
linkage: incremental re-parsing of the changed file writes what a full
index writes. Same-QN variants (#if/#else, Rust cfg, Go build tags,
.c/.h pairs) and their `variants` property are unchanged. No new edge type.
Proof (macOS arm64):
- Linux kernel subset (arch, include, kernel, mm, lib; 27,005 files):
arch functions whose QN is not under their own path 12,873 -> 0;
arch/A -> arch/B CALLS 10,353 -> 7,101; 581 variant groups (1,815
members), each one of the former cross-file merges; the path QN of
arch/arm64's mem_init resolves.
- This repository: 13 groups (test_impact_tree_{posix,windows},
mem_override_{posix,win}); before, 22 of 49 platform-path functions had a
stripped QN.
- New/rewritten tests (RED on the old tree or without the walk
expansion): conditional_variants cv_c_platform_files,
cv_c_platform_directories, cv_incremental_platform_files;
test_impact_engine
test_impact_engine_platform_variants_reach_each_others_callers;
store_impact impact_walk_crosses_platform_variants_only_when_asked;
cv_incremental_variant_partners re-fixtured to Go build tags.
- 18 affected suites: 2710 passed, 4 Windows-only skips.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
doclink_rst.c sorted d.toks unconditionally; the vector stays NULL until
the first token, and qsort's base is declared nonnull even for zero
elements. GCC UBSan on Linux amd64 reported it in rst_section_text_cut and
diagrams_rst ("null pointer passed as argument 1, which is declared to
never be null", doclink_rst.c:1767) -- the same class as the k8s pass fix.
Sort only when there are at least two tokens, as the file already does for
its section titles.
Proof: doc_links_rst on Linux amd64 (GCC, ASan+UBSan) reports the
violation without the change and none with it; doc_links_rst + diagrams on
macOS: 27 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The preprocessed second pass rebuilt the expanded source with simplecpp's
TokenList::stringify(), which puts one space between every pair of tokens:
`template<> struct bitonic<int64_t, AVX2>` became
`bitonic < int64_t , AVX2 >`. Every QN builder takes the class segment from
node text, so a definition the #961 rescue re-adopted from that buffer
(vxsort's INLINE/NOINLINE macros break the raw parse on every method line)
got a second, token-spaced QN, and the QN-equality dedup kept both. CALLS
split across the two spellings (dotnet/runtime: 902 compact + 996 spaced
into the bitonic sorters).
The expanded buffer now keeps the source spacing: tokens that touched in
the source (same file and line, no macro expansion between them) stay
touching, every other gap is still one space, and the line structure and
#line directives are unchanged, so every line remap holds. One fix point
instead of a normaliser at each QN builder; no vendored code changes.
Proof (macOS arm64, dotnet/runtime 60108ba6, two identical runs each):
- duplicate definition groups 706 -> 217 (v0.11.0: 545); nodes with a
token-spaced QN 495 -> 0; CALLS into bitonic 902 + 996 -> 998, one
spelling. The remaining groups are coreclr gc.cpp's SVR/WKS builds (184)
and 33 scope disagreements between the two parses.
- Side effect: rapidjson document.h's GenericValue members now exist once,
with the raw parse's scope (the token-spaced copy that carried the
nested scope is gone); `operator[]` / `operator()` in the expanded pass
now parse as single tokens.
- New test (RED without the change): extraction
extract_cpp_template_specialization_preprocessed_qn_single_spelling.
Three c_lsp collision fixtures recalibrated to the new byte offsets
(comment padding and offsets only, assertions unchanged).
- extraction, c_lsp, pipeline, edge_types_probe, parse_coverage,
conditional_variants, lsp_resolution_probe: 1852 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#2513 (77bc4927) computes `url_context` -- is the callee an HTTP client or a
global fetch? -- for every positional argument of every call, in every
language, through cbm_service_pattern_match: a linear strstr over every
service pattern. Only the URL-folding functions read it, and they return
NULL outside Python and JS/TS; the one other reader is the template-string
branch. On the C# corpus (dotnet/runtime, 451k functions) extraction went
from 24.7 s to 37.4 s (+51 %); a macOS sample put the difference in
handle_calls -> cbm_service_pattern_match -> match_qn -> strstr.
A per-call url_context_t now computes the answer at most once and only when
an argument needs it: the folding paths ask only in folding languages, the
template-string branch asks for itself.
Proof (macOS arm64, Apple M5 Pro):
- C# parallel_extract 37.4 s -> 24.2 s, wall 127-132 s -> 119.7 s (the
pre-#2513 level, 24.7 s / 121.9 s).
- Byte-identical graphs: the full node and edge sets (properties included)
hash the same before and after on django (Python), typescript (TS) and
csharp, HTTP_CALLS included (457 / 283 / 39).
- extraction, extraction_imports, pipeline, parallel, infrascan and
edge_types_probe: 1009 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The C# and Java Tier-2 registry builders register types, finalize, then
register the rest from `defs + type_count`. With no definitions of that
language the array is NULL, and NULL + 0 is undefined: clang's UBSan
reports "applying zero offset to null pointer" at cs_lsp.c:3886 on the
macOS legs of dry run 38321896354 (reached by
incremental_k8s_edits_reach_other_files, a project without C# files).
The second registration now runs only when there is something to register;
both finalize calls are unchanged.
cs_lsp, cs_lsp_bench, java_lsp, java_lsp_coverage, lsp_resolution_probe:
529 passed (macOS). The report reproduces on CI's macOS clang, not on the
local toolchain, so CI's macOS legs are the check.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A call no project symbol resolves, whose callee an import binds to an
external library, is now spelled through that import and classified with
the existing service patterns: `send` under
`from paho.mqtt import publish as send` is paho.mqtt.publish (ASYNC_CALLS
to its topic), `navigate` under `import { get as navigate } from 'axios'`
is axios.get (a typed GET HTTP_CALLS edge). Before, such calls had no
service edge at all.
- CBMImport gains `member`: the exported name an ES named import binds
(`get` in `{ get as navigate }`), NULL for default, namespace and other
imports and for every other language. Filled by process_named_imports,
carried through result compaction (and so through spill), counted in the
extraction census with the other import strings.
- One helper, cbm_pipeline_external_import_service, serves both call
resolvers: a per-file index (local name -> binding import), built on the
first unresolved call that asks, then one hash lookup per call.
- Precision: only unresolved calls, after the callee-name, route-suffix and
global-fetch fallbacks; never a relative import, an import an IMPORTS
edge materialized (a project module), a name bound by two different
imports, or a Python callee rebound as a parameter. An HTTP edge needs a
request verb in the spelled name, so urllib.parse.urljoin is not a
request. Python, JavaScript, TypeScript and TSX.
Tests: extraction_imports covers the member and its survival through
compaction; a pipeline test runs both resolvers over a typed GET, a typed
POST and a publish through aliases, and over controls that must stay
silent (urljoin, lodash and json aliases, a local definition and a
parameter shadowing the import, a materialized project package named
httpx). Each guard was reverted once and its control went red.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Refs #2291, #706, #1147. Preserve raw argument values and carry URL projections separately through compaction and spill.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve registry array order and full-scan fallback when optional query
storage is unavailable. Keep temporary range storage scoped to each module
operation and shared type indexes in the registry owner arena.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
lookup_func_self guards a NULL qualified_name in its indexed bucket scan
and its post-finalize tail scan, but not in the linear scan an
unfinalized registry takes. Iterating a fallback chain checks every base
entry for shadowing against the head, so a head holding a func without a
qualified name crashed in strcmp (ASan SEGV in
pylsp_prefix_order_matches_iterator_and_chain_shadowing). The linear
scan now skips such entries like the other two.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two gaps let a call on a vendor class escape the #1186 evidence:
- A vendor class the PHP LSP knows from its seeded library data (for
example GuzzleHttp.Client) resolved as php_static_resolved or
php_method_typed. Its target has no graph node either, but without the
php_external_receiver strategy the pipeline fell through to a
name-only registry binding. Methods found on a class reached through a
vendor `use` now carry php_external_receiver as well.
- A vendor call skips the registry, and the extractor names a scoped
call by its method alone (only a literal `Route::` scope is kept), so
`GuzzleClient::get('/x')` reached the route-verb fallback and became a
route registration instead of an HTTP call. When the callee text names
no service, the service patterns now also see the vendor method the
LSP proved (`GuzzleHttp.Client.get`, `Kafka.Producer.publish`), in
both the sequential and the parallel pass.
Refs #1186
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
collect_use_declaration dropped the group prefix of `use A\B\{C as D}`
and read the function/const keyword only from the declaration, so a
grouped alias mapped to the bare `C` with the declaration's kind. The
cross-file LSP then could not recognise a grouped vendor class import as
external, and a grouped `function`/`const` alias was registered as a
class. Prefix each grouped clause with the group namespace and take the
keyword from the clause when the declaration has none, mirroring
emit_php_use_decl in extract_imports.c.
Refs #1186
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reuse a stable arena-owned function QN order for ambiguity marking and
submodule prefix lookup, preserving exact lookup policy and appended tails.
Add prefix/index failure regressions and a pipeline operation-count case.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The __all__ literal guard compared a uint32_t child count against
SIZE_MAX / sizeof(char *), which is always false on LP64 and failed the
GCC build (-Werror=type-limits). One bound now covers both limits on
every platform: the names are counted in an int, and
count * sizeof(*names) <= INT_MAX also fits a 32-bit size_t.
Building one import fact moves into py_ns_import_fact, and the
module-name child is skipped with ts_node_eq: for a plain import the
module is the null node, which equals no child. The loop no longer tests
`from` after the from-only prelude, which cppcheck 2.20 misread as
always false (knownConditionTrueFalse). Emitted facts are unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Capture ordered Python namespace facts and retain them through compact, spill,
and versioned LSP surfaces. Reject unavailable metadata before incremental
reuse and rebuild on Python namespace changes or deletion.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolve symbols declared in __init__.py under their package while retaining
the raw module identity for file and top-level caller nodes. Carry explicit
language through registry callers, select Python package imports, preserve
long exact names, and report scope allocation failures without partial output.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve the arena-owned raw Module caller name so top-level resolutions
join the extractor in both pipeline drivers. Add occurrence and ownership
controls, including package modules and named lambdas.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The depth guard of the C/C++ and Python expression evaluators
(C_EVAL_DEPTH_LIMIT, PY_LSP_MAX_EVAL_DEPTH = 256) now reports a sticky
per-file failure that stops the walk and drops every LSP resolution of the
file. Both evaluators recursed once per chain link, so a 300-term sum or a
C++ fluent chain of ~130 calls counted as 256 levels of nesting: one long
expression cost its whole file every LSP call edge. A chain's length is not
nesting, and the guard must not decide graph content for it.
c_eval_expr_type and py_eval_expr_type now first walk down the chain
operands below the node (the operand the evaluator evaluates first and
derives the link's type from), then evaluate them innermost first at the
caller's depth, keeping pending links on a small arena-owned stack in the
context that nested evaluations share. When a link is evaluated, its chain
operand is already memoized, so the evaluator's own recursive call into it
is a memo hit one frame deep. Every node is still evaluated by the
unchanged inner evaluator, evaluation does not mutate scope within one
outermost call, and only operands the recursion itself evaluates are
pre-evaluated, so the types are exactly the recursive ones.
No longer counted as depth (any length):
- C/C++: binary operators other than == != < > <= >= && || (those yield
bool without evaluating an operand), member access (. and ->), calls on
a non-name callee, subscripts, conditional consequences, parentheses,
comma lists.
- Python: binary operators, attribute access, calls on obj.m(...), f()()
and (expr)(), subscripts, the else branch of a conditional, parentheses.
Still counted, unchanged guard and reporting (genuine nesting):
- C/C++: call arguments used for template argument deduction,
std::move/std::forward operands, unary, pointer and update operands,
assignment left sides, lambda return expressions, co_await, fold and
_Generic operands.
- Python: tuple/list/set elements, dictionary keys and values, the true
branch of a conditional, await operands, next()/iter()/assert_type()
arguments, called lambdas.
Proof:
- New c_lsp and py_lsp tests build chains of 300, 2,000 and 20,000 terms
(a binary + chain and a .a() receiver chain) next to an unrelated s.m().
On the previous evaluator, and with this change reverted, all four fail
at 300 terms: the depth error is recorded and the file resolves 0 calls
(C++ per-file; Python per-file and through the cross-file dispatch).
With it, every size records no error, resolves run -> S.m, and the
chain's result type resolves x.done() / the final .done().
- Real input, isolated cache, before and after:
redis 4f20cb48 (C): 36,876 CALLS, 10,699 LSP-resolved;
django fb113765 (Python): 60,123 CALLS, 25,153 LSP-resolved.
Identical CALLS edge sets on both (0 lost, 0 gained, 0 strategy
changes), 0 files with an LSP error before and after, wall time
unchanged (redis 5.5 s, django 9.1-10.2 s both ways).
Refs #1527
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Memoize C expression evaluation, index repeated scope and registry queries,
and report incomplete walks explicitly on memo or depth failure.
Refs #1527
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Four syntax-tree walkers recursed once per nesting level of the indexed
file, so their stack use followed the depth of the parse tree rather than
anything bounded. They walk iteratively now, with identical results.
- terminal_callee_leaf (extract_calls.c) and its PHP LSP mirror
php_reference_callee_leaf descended one wrapper node per frame on a
callee such as ((((f))))(...). Both are loops now: each step either
returns or moves to the child the search continues in, so the nodes
visited and the leaf chosen are unchanged.
- cbm_collect_error_regions and cbm_mark_pp_error_rows (cbm.c) recursed
into every child that contains an error. One cursor walk now drives
both. It stops on exactly the visible children ts_node_child
enumerated, in the same order, and never enters an ERROR subtree.
- is_direct_argument_value (extract_usages.c) moved from a labeled
argument's value to the argument itself by recursing. The grammar bounds
that step, but it is a loop now, taking the slow-parent test note once
per level as before.
calls_append_args (pass_calls.c) formatted each argument into a 512-byte
local and assumed the result fit. An argument that does not fit now ends
the array like one the props buffer cannot take. The parallel twin formats in
place and already stops on truncation.
Indexing this repository's src/ and internal/cbm/ with and without the
change gives byte-identical sorted dumps of nodes, edges per type, CALLS
edges and parse-coverage rows.
CBM_WORKER_STACK_SIZE moves to worker_pool.h so the new extraction tests
run on a thread of exactly the pipeline's size, in a forked child on POSIX
so an overflow is a signal instead of a dead runner. They nest 200,000
parentheses around a first-class callable and around an invalid token,
pin the labeled-argument classification across the loop conversion, and
hand the formatter an argument longer than its local buffer.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Delphi code that calls 1C's OLE API through late-bound Variants uses
Cyrillic member names (v8_Base.<Cyrillic>.<Cyrillic>('1', Now),
NM.<Cyrillic>()). Every such name produced a tree-sitter ERROR node, so the
unit was reported as parse_partial and a procedure whose own name is
Cyrillic was dropped from the graph.
Root cause is upstream tree-sitter-pascal (Isopod, the pin we vendor):
its identifier token is ASCII-only, /[&]?[a-zA-Z_]+[0-9_a-zA-Z]*/, and
there is no other token a non-ASCII letter can lex as.
Fix: pascal becomes a self-maintained fork of the pin we already vendored
(Isopod/tree-sitter-pascal @ 042119eca2e1, v0.10.2 = upstream master),
following the arkts model. One patch
(tools/tree-sitter-pascal/patches/unicode-identifiers.patch) changes only
that token to /[&]?[\p{L}_][\p{L}\p{M}\p{Nd}_]*/. Keywords are still
extracted from this `word` token, so begin/end/procedure lex as before
(`begin` glued to Cyrillic letters is one identifier; `&begin` stays an
escaped identifier). Node types are unchanged. The \p{} classes compile
to numeric code-point ranges: the vendored parser.c has 0 non-ASCII bytes.
Regenerated with tree-sitter-cli 0.25.10 --abi 14 (ABI unchanged, no
external scanner; tree_sitter/parser.h is byte-identical to chialisp's).
Fidelity: the UNPATCHED grammar regenerated with 0.25.10 gives the same
symbol/token/field/production counts as upstream's shipped parser and
byte-identical parse trees on upstream's 88 corpus cases and its three
examples/*.pas (37,773 tree lines for codetoolmanager.pas); the patched
parser gives the same identical trees on all of those ASCII inputs and
passes all 88 corpus cases.
Proof (7 Pascal files: the report's snippet, a Cyrillic procedure with
keyword-boundary cases, an ASCII control, a broken unit, upstream's three
examples; fresh caches): parse_partial 3 -> 1 (only the genuinely broken
unit remains), nodes 581 -> 582 (the Cyrillic-named procedure), edges
1480 -> 1482 (+1 DEFINES, +1 SEMANTICALLY_RELATED); per-file function
counts for the ASCII files are unchanged.
Registered like arkts: MANIFEST.md fork row + note, audit FORKS map,
THIRD_PARTY.md, vendored checksums. LICENSE stays upstream's MIT,
byte-identical.
Tests: parse_coverage gains the report's snippet and a Cyrillic-named
procedure (not partial, definitions extracted), an ASCII control, and a
guard that broken Pascal is still flagged; the fork carries
test/corpus/unicode_identifiers.txt.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Why: a function-like macro in a definition's declarator,
`void CBasePlayer::__API_HOOK(ReloadWeapons)(int slot)` or
`static void FNAME(first_caller)(void)`, parses as a function_declarator
nested directly inside the definition's own function_declarator. The
declarator walk descended to the terminal identifier, which is the MACRO,
so every definition wrapped by the same macro got one qualified name.
ReGameDLL_CS's hook definitions landed as 20 nodes literally named
__API_HOOK, and the Linux kernel's arch/x86/kvm/mmu/paging_tmpl.h (19
FNAME(...) definitions) as a single Function FNAME. The last definition
won, calls in every other body were sourced to it with a call line
outside its range, and callers never reached the real functions. The C
LSP had its own copy of the walk: it named the caller after the macro and
bound the macro's argument list as the function's parameters, so member
calls through a real parameter went unresolved.
Fix: cbm_c_macro_wrapped_name_node() recognises the shape. Valid C/C++
never nests two function_declarators directly (a function cannot return a
function; a function-pointer return has a parenthesized_declarator in
between), so the nesting always means a macro call. When that macro is a
plain or class-qualified name taking exactly one bare-name argument, the
definition is named after the argument; any other shape (multi-argument
wrappers, non-name arguments) keeps today's walk. The shared C-family
declarator resolver uses it, so the def name, the call-scope QN and the
unified extractor agree; the C LSP uses it too and keeps the outer
(real) parameter list. The class scope of `Cls::MACRO(x)` was already
derived from the qualified name and is unchanged.
Proof: dev builds of origin/main 1f9b4db1 (before) and this change
(after), same machine, fresh indexes.
rehlds/ReGameDLL_CS @ 4a50c42 (#946):
- nodes named __API_HOOK: 20 -> 0
- Function 986 -> 1003, Method 3768 -> 3859: 129 hook definitions now
have their own node, and every added definition starts on a
`__API_HOOK(<name>)(` line (0 false definitions). Removed: the 20
__API_HOOK nodes, plus CGrenade::Explode(Vector, Vector), which now
shares CGrenade.Explode with its hooked overload, as any C++ overload
pair does today (one node per QN).
- CALLS 9436 -> 9916; CALLS sourced from __API_HOOK 410 -> 0; callable-
sourced CALLS whose call line lies outside the caller's range 565 -> 198.
- parse_partial files 131 -> 131.
- 5 samples checked against source (CBasePlayer::Spawn 5771-6142 with 32
in-range out-calls incl. ReloadWeapons via lsp_implicit_this,
CBasePlayer::TakeDamage, PM_Move, CGrenade::ShootSmokeGrenade,
CBotManager::OnEvent).
- 18 of the 147 hook definitions are still not extracted, before and
after: the parser loses them to raw ERROR regions (a `T *EXT_FUNC`
pointer return misparsed as a multiplication, a preceding
semicolon-less LINK_HOOK_*(...) invocation, an #ifdef-split
signature). That is parser recovery, not naming, and is left for a
follow-up.
Linux arch/x86/kvm/mmu @ 2d2338c93 (#1957):
- paging_tmpl.h: 1 Function FNAME -> 19 distinct functions (all 19
FNAME(...) definitions; 0 false definitions in the directory).
- CALLS from paging_tmpl.h functions with the call line outside the
caller's range: 32 -> 0 (directory: 35 -> 3, the 3 pre-existing in
mmu.c); e.g. fetch 651-786 now owns its 13 calls, sync_spte 934-1005
its 8.
- 5 samples checked against source (prefetch_gpte, walk_addr_generic,
fetch, sync_spte, page_fault).
- #1957's own reproduction: first_caller and later_function are distinct
Functions, sink() at line 9 is sourced to first_caller (7-10), and
main -> first_caller resolves.
Tests: extraction::extract_cpp_macro_wrapped_declarator_names_def_issue946
(qualified + unqualified + EXT_FUNC hooks, macro in a header: def names,
QNs, range, real parameter count, per-body call scope),
extraction::extract_c_fname_macro_definitions_stay_distinct_issue1957
(the reported C reproduction incl. the LSP caller QN, and the kernel
template shape with a token-pasting FNAME defined only under #if),
extraction::extract_c_macro_wrapped_declarator_controls_issue946
(precision controls: file-scope and in-body macro call statements, a call
through a returned function pointer, a real function returning a function
pointer, a macro-wrapped prototype and a two-argument wrapper mint no
definition named after a macro argument), and
c_lsp::clsp_macro_wrapped_declarator_caller_qn_issue946 (LSP caller QN,
lsp_implicit_this, typed parameter). RED 3x before, GREEN after, RED on
revert; the LSP half alone binds the c_lsp test and the #1957 test.
Fixes#946Fixes#1957
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings in #2596 (MSan: hidden uninitialized reads) and #2068 (CodeAtlas
web).
README: main's CBM Atlas line and this branch's infrastructure-as-code
line are both kept.
tools/list metadata: main and this branch each stayed under the 20 KiB
discovery budget, but together they reached 20,809 bytes. The branch's
wording is shorter now (search_graph kind/namespace, one hint per trace
mode: 136 bytes less), and the reviewed budget is 21 KiB so the
topology/deploy/run modes keep their usage hints.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
tree_sitter_wit_external_scanner_create() is declared to return void *
but had an empty body. tree-sitter's runtime stores its result as the
scanner payload (parser.c ts_parser__external_scanner_create), so every
WIT parse used an indeterminate value, which C11 6.9.1p12 makes
undefined. The scanner never reads its payload, so ordinary builds were
unaffected, and the vendored build's -w hid -Wreturn-type.
MemorySanitizer reports it at wit/scanner.c:13 from grammar_labels,
grammar_probe_e, grammar_regression and lang_contract. Those four suites
were excluded from the MSan lane because the report itself crashed and
posed as a stack overflow (fixed in the msan.sh commit of this series).
Recorded in MANIFEST.md's local-patch table. Upstream
bytecodealliance/tree-sitter-wit still has the empty body at
059bfdd1af72, the last change to src/scanner.c.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Each variable a K8s container's env (containers, initContainers,
ephemeralContainers at any depth), a compose service's environment (map
or list) or a Dockerfile's last stage ENV (with the stages it is built
FROM) sets is now an EnvVar of that container -- of the image, for ENV:
`<owner QN>#env/<container | ->/<name>` {env_key, scope, container,
source}, owned by the declaring file, and the owner CONFIGURES it
{strategy: env_set, source, line}. `source` says where the value comes
from (value, configMapKeyRef, secretKeyRef, fieldRef, resourceFieldRef,
fileKeyRef, compose, dockerfile); the value itself is never read into
the graph.
Two containers setting PORT are two nodes, and neither is joined to the
global EnvVar PORT a program's readers share: the same name is read by
other programs (measured on 17 infra repositories, a join through the
global node is in the setter's own program for 23 % of the pairs).
Docker drops comment lines before it joins continued lines, but the
grammar ends an instruction at one; the Dockerfile pass now reads such a
comment line as a bare continuation (rows and offsets unchanged), so
`ENV A=1 \` / `# url` / `B=2` sets B too.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Kubernetes object running a container now links to the Image of its
literal image reference: containers, initContainers and
ephemeralContainers at any depth, so pod templates of workloads,
CronJobs and custom resources all count. It is the node a Dockerfile
building on, or building, that image joins -- the run half of
Service -> ... -> main().
- extract_k8s: a new CBM_K8S_FACT_IMAGE for a literal image leaf;
a shell substitution ($(X), ${X}) or a template expression is none.
- pass_k8s: the facts become USAGE {ref: image, image} edges from the
object's Resource to the shared Image node.
- pass_docker: cbm_pipeline_image_node() is the one place an image
reference becomes an Image (normalised repository); a reference with
a character no image reference can hold -- a Helm template's
placeholder such as <.Values.image.repository> -- is no image.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The held-out audit of the eight new languages is scored. Each tier is a
language and a reference form, read on a sample of 150 links (census below
150) from repositories never used to build or tune the resolvers. Three
blind reading rounds; bar: Wilson 95% lower bound >= 0.90 and point >= 0.95.
tier population judged correct Wilson low
go doc_link 5980 150 150 0.975
java link 23934 150 150 0.975
java linkplain 1372 150 150 0.975
java see 7711 150 150 0.975
java throws 650 150 150 0.975
java value 79 79 79 0.954
kotlin kdoc_link 3888 150 150 0.975
kotlin sample 279 150 150 0.975
kotlin see 369 150 150 0.975
kotlin throws 73 73 73 0.950
rust intra_doc 14119 150 149 0.963
rust link_def 1491 150 150 0.975
perl pod_L 2255 150 150 0.975
ts/js link 1222 150 150 0.975
php see 5171 150 150 0.975
c doxygen_ref 463 150 150 0.975
c see 509 150 145 0.924
These 17 families now ship: a resolved reference is a MENTIONS edge.
TS/JS {@linkcode} (10 held-out links) and @see (18) were all judged
correct, but are too few to reach the bar; they stay held, as do the
diagram and OpenAPI link families (3 links on 21 held-out repositories).
The Go ship-gate test turned its family off and restored only the held
families afterwards; with Go shipping it left Go off for the tests that
follow it. It restores the production gates first now.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Markdown Section's docstring is its body collapsed and cut at 500 bytes.
A diagram fence in that body put the format's syntax into it (`-->`,
`participant`, `classDiagram`, `subgraph`), which names no code, and pushed
the section's prose out of the cap. The doc -> code candidates are scored
on that text.
Each top-level diagram fence (mermaid, plantuml, puml, uml, dot, graphviz,
d2) is now replaced in the section's text by the labels its diagram parser
reads: the names of its elements, then its arrow labels, in source order.
The parse runs into a scratch result, so the file's own Diagram and
DiagramElement nodes are untouched. A section without such a fence keeps
its text byte for byte; other fences stay as written.
Measured before it was chosen, on held-out repositories (147 sections that
embed diagrams, 503 section/candidate pairs, three blind reading rounds):
fence as written 277 stored precision 0.45 42 sections with a correct candidate
labels (this) 310 stored precision 0.54 56 sections with a correct candidate
fence dropped 127 stored precision 0.54 29 sections with a correct candidate
Labels gave 20 sections a correct candidate that the fence as written did
not, and lost 6 (exact McNemar p = 0.009). Dropping the fence lost more
than it gained.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dockerfiles were Module + ENV/ARG Variables only: the 261 FROM and
114 COPY --from lines of the infra corpora joined nothing, and build
files named for a variant were not indexed at all.
- pass_docker (new, end of the K8s pass): FROM <image> and
COPY --from=<image> -> DEPENDS_ON {via: from | copy-from, image,
stage, line} from the Dockerfile's Module to an Image node. ARG
defaults are expanded (global ARGs in FROM, a stage's ARG / ENV in
COPY); a stage (by name or index), scratch and an ARG without a value
are no image.
- Image (new label, as approved): one node per repository, normalised
(docker.io for no registry host, library/ for an official image, a
registry host and port kept); tag and digest stay on the edge.
- Detection: Dockerfile.<x>, Dockerfile-<x>, Dockerfile_<x>,
Containerfile[.<x>] and <x>.Dockerfile are Dockerfiles, unless the
suffix is a language extension of its own (Dockerfile.j2 stays a
Jinja template) or a BuildKit ignore file.
- QNs: a Dockerfile.<variant> keeps its suffix (fqn.c and helpers.c,
which must agree); dropping it gave the variant the plain
Dockerfile's QN, merging both Modules. Covered by this branch's
index-format bump.
- Incremental: an Image no DEPENDS_ON / USAGE edge touches any more is
dropped on both routes, like a Process without SPAWNS; the
unreferenced-node GC takes a list of edge types.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Flux HelmRelease that names its chart through `spec.chartRef` (kind
OCIRepository) instead of `spec.chart.spec` drew only a USAGE edge to the
OCIRepository: "which releases deploy chart X" had no answer for it. In
onedr0p/home-ops every one of 68 releases is written this way.
The K8s extractor now records the URL an OCIRepository fetches; the K8s
pass follows each HelmRelease's resolved chartRef to that repository and
draws DEPLOYS to the shared external Chart of the oci:// URL's last
segment (properties: via helmrelease, chart, external, url). A URL with
a variable, or a chartRef naming no repository, draws no edge.
Measured on 17 repositories against an independent reading of the Flux
objects: GitOps DEPLOYS 199 of 199 linked and correct (was 129 of 164;
home-ops 160, flux2-kustomize-helm-example 3). Only DEPLOYS (+70) and
external Charts (+44) changed; three runs build identical graphs;
index time unchanged (deckhouse 9.63 -> 9.60 s, k8s.io 3.71 -> 3.70 s,
home-ops 0.88 -> 0.87 s, medians); editing the OCIRepository's URL, the HelmRelease's chartRef,
or deleting the repository leaves the graph a fresh index builds.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Make rules were read with three gaps:
- A rule naming several targets (`seed-up seed-dev seed-down: ...`,
`kind-% seed-%:`) became one Function named after its first target:
`seed-down` had no node, so nothing could link `make seed-down` and a
search for it found nothing. Every target now gets a Function over the
rule's span (without fingerprint, structural profile or body tokens, so
the copies of one recipe do not read as similar code), and the recipe's
DEPLOYS / SPAWNS edges leave from each of them.
- A target-specific variable line (`kind-%: export X := y`) was taken for
a rule: a phantom Function for its first target, or a variant span
merged into the real rule of that name. Such a line is no rule now. Its
tree-sitter ERROR region is no longer masked, so the file reports it
as partially parsed, as for any other unparsed region.
- `include hack/tools.mk` imported a Folder `hack/tools` (local name
"mk\n\n": the generic import reader kept the directive's newlines and
split at the extension). Make includes are read on their own now: every
name of include / -include / sinclude, outside recipes, imports that
file; a name with a variable reference is left out.
Measured on 17 repositories against an independent reading: make runs
linked 298 of 300 (was 282), all 298 correct; the 2 left pick their
target in an if/else, which stays unknown by design. Graph changes are
confined to Makefiles (gardener +26 targets, 13 phantoms removed;
deckhouse +4), their edges, and two side effects of new nodes:
SEMANTICALLY_RELATED edges elsewhere re-rank, and one Go member
reference in deckhouse lost its unique-name binding to a new Make rule
of the same name. Three runs build identical graphs; index time unchanged (deckhouse 9.61 -> 9.60 s,
gardener 5.24 -> 5.27 s, cluster-api 2.68 -> 2.68 s, medians);
renaming a target, editing a Makefile or an included file leaves the
graph a fresh index builds (gardener's Makefile edit differed before,
through the phantom nodes, and no longer does).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Diagrams name the parts of a system and how they connect; an OpenAPI
document names its endpoints and their handlers. Both now enter the graph.
Diagrams:
- a Diagram node per diagram: a fenced block of a Markdown file (under its
Section), a directive of a reST file (`.. mermaid::`, `.. uml::`,
`.. graphviz::`, Sphinx `.. digraph::`), an Asciidoctor Diagram block of
an AsciiDoc file (`[plantuml]`, `[source,mermaid]`, ...), or a diagram
file of its own, named by its title, else by its file;
- a DiagramElement node per element, DEFINES from its Diagram, and a
DIAGRAM_ARROW edge per arrow (label, style, line);
- formats: Mermaid (flowchart, sequence, class, ER, state, C4,
architecture), PlantUML, Graphviz DOT, D2, draw.io (plain, compressed,
.drawio.svg) and Excalidraw (.excalidraw, .excalidraw.json,
.excalidraw.svg); .drawio.svg and .excalidraw.svg are no longer dropped
as images;
- an element's explicit link (Mermaid click/link, PlantUML [[url]], url of
and !include, draw.io and Excalidraw link, DOT URL/href, D2 link) is a
doc link read as a Markdown link, MENTIONS when it resolves.
OpenAPI:
- each operation of an OpenAPI or Swagger document (openapi.json and
swagger.json, now read with the JSON grammar but no node per key, and
any YAML or JSON file whose root says openapi/swagger) is a Route, named
by the same QN a route its handler registers in code gets
(cbm_route_canon_path, moved into the extraction library);
- its operationId is a doc link to the handler: a dotted one (Connexion,
with x-openapi-router-controller) as a qualified code name, a bare one
when exactly one function or method of the repository has that name.
The eight new link families are held until their held-out audit passes.
Read on real repositories, which shaped the rules: Excalidraw drawings put
text over shapes and draw arrows up to them without binding, so loose text
names the smallest box it sits in and a loose arrow end meets the smallest
named box it touches (the drawing's frame only at its border); arrows bound
to free text meet the text; D2 block strings and nested blocks; DOT line
breaks and sentence-long ids; PlantUML kinds of classes inside packages
and nested classes (`..+`, nested -> container, so they do not collide with
the container's association). Connexion's 24 specs: 160 operationIds, all
linked to their handlers.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Flux Kustomization, an Argo Application and a HelmRelease each name what
they put on a cluster -- a directory of the repository or a chart -- and
none of it was an edge: "what deploys this kustomization" or "which
release installs this chart" had no answer in the graph.
DEPLOYS (a new edge type, approved for this program) links them to
existing nodes only:
- a Flux Kustomization / Argo Application -> the units its path applies:
the outermost kustomizations below it (their Kustomization node) and
the manifests outside any (their File), as the deploy scopes already
read them; a path that is a chart dir -> that Chart;
- a HelmRelease -> its chart: the repository chart dir of a Git or Bucket
source, else the shared external Chart of that name (the node chart
dependencies already use) for a Helm repository.
Properties carry via (flux / argo / helmrelease) and the path or chart.
Measured on 17 repositories against an independent reading of the
manifests (units derived from kustomize closures, chart dirs and
external chart names): 129 of 129 deploys linked, 129 of 129 correct
(home-ops 92, k8s.io 36, flux example 1). Only DEPLOYS edges (and one
external Chart) were added; three runs build identical graphs.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Most of an infrastructure repository's Kubernetes objects are written as
templates -- Helm charts, Go-template addons -- and those are YAML only
once rendered: `{{- if .Values.x }}` opens the document and `name: {{
include "app.fullname" . }}` is a flow mapping to a YAML parser. Across 17
repositories 92 % of the objects written in chart templates (1,127 of
1,233) never became Resources.
A templated manifest is now extracted from its skeleton: the text with
every action rewritten line for line, so document lines stay exact. A line
holding nothing but actions is blanked; an action inside text becomes a
symbolic scalar -- `<app.fullname>` for an include/template call, the first
`.`/`$` path otherwise (`<.Values.image.tag>`), else `<tpl>`. Objects get
the chart-scoped symbolic names IDENTITY.md describes
(`<app.fullname>-config`), and a reference or selector written with the same
expression meets them inside the chart. A templated namespace is the
deploy's; a templated apiVersion leaves the group unknown; a templated kind
is unidentifiable and skipped.
Measured on 17 repositories: objects written in chart templates as
Resources 8 % -> 94 % (1,166 of 1,233: prometheus-community 0 -> 343 of
357, microservices-demo 0 -> 84 of 84, deckhouse 431 of 479, gardener 149
of 151, kubermatic 133 of 136); kubermatic's templated addon objects 88 % ->
97 % (708 of 727). References and selectors touching templated objects,
201 sampled across 6 repositories and judged against the original template
text: 201 correct (4 by hand, quoting the sampler did not normalise). The
plain-manifest gates, include resolution and selectors are unchanged; three
runs build identical graphs; editing a template keeps incremental equal to
a full rebuild.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel