Threads main's Go rel_path (#1910) through the proof-source and import-snapshot variants, keeps main's ADR sidecar so the rebuild no longer re-applies ADRs, and keeps the feature's removal of the import name fallback and JVM tail join, inside which main had added a generated-member filter. Main took semantic index version 12 for Markdown Section names, so the resolver generation becomes version 13.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings #2419 onto main a2227157. The merge is clean: since the
branch's base 0f52d30c, main has not changed
python_default_value_reference, lexical_ancestor_kind,
usage_lexical_scope_id_for_node, record_lexical_binding or
handle_usages, nor the occurrence-cursor helpers the change builds
on (reset_occurrence_cursor, occurrence_parent, the slow-parent test
counter). The merged tree is main plus exactly this PR's three files
(internal/cbm/extract_usages.c +57 -16,
tests/repro/repro_lexical_binding_precision.c +54,
tests/test_extraction.c +44).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up to #2161, which removes `external` from FAST_SKIP_DIRS
(#2151). The repro board's invariant_discovery_fast_skip_dirs still
listed it among the directories MODERATE mode must skip, so with the
change the board reported "REGRESSION external not skipped in MODERATE
mode". That runner is outside the gating test target, so PR CI could
not see it.
The table matches FAST_SKIP_DIRS again. The new behaviour stays pinned
in the gating discover suite by no_skip_fast_external.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Terraform references were resolved by the generic name matcher: by their
last name segment, across the whole repository. `format()` CALLS a
variable named format of another module, `local.x` meets a resource named
local, `.id` an output id anywhere. Measured on the 7 Terraform
repositories of 17 infrastructure corpora: 12,437 edges, recall 0.63 and
precision 0.39 against an independent reading of the references, and not
one of them left the block holding its reference.
A reference -- var.X, local.X, module.X, data.T.N, T.N -- now USAGEs
{ref, via: terraform, line}, from the top-level block holding it, the
block declaring it in the same module: the .tf / .tofu files of its
directory (a locals entry: its locals block); an entry of the block's
depends_on list DEPENDS_ON the object instead. The generic matcher makes
no edge for a .tf / .tofu file any more; a name only mentioned in string
text is no reference, nor is a block's reference to itself. A changed
file re-resolves its whole directory, so an unchanged file's reference
meets the declaration a change adds and loses the one it removes.
A bare block label (`variable region {}`, valid HCL) now names its block
like a quoted one; such blocks used to share the keyword's name and QN.
Measured: 7,771 of 7,771 file-level and 13,397 of 13,397 block-level
reference pairs (edge type included) and nothing else; the 10 other
corpora are unchanged; deterministic over three runs; the incremental
re-index equals a full index in 7 scenarios (rename a variable, a module,
a locals entry; delete the variables file; add a file; add a declaration
an unchanged file references; quote a bare label); index time unchanged
(within 1 % on the four largest Terraform corpora).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Diagrams name the parts of a system and how they connect; an OpenAPI
document names its endpoints and their handlers. Both now enter the graph.
Diagrams:
- a Diagram node per diagram: a fenced block of a Markdown file (under its
Section), a directive of a reST file (`.. mermaid::`, `.. uml::`,
`.. graphviz::`, Sphinx `.. digraph::`), an Asciidoctor Diagram block of
an AsciiDoc file (`[plantuml]`, `[source,mermaid]`, ...), or a diagram
file of its own, named by its title, else by its file;
- a DiagramElement node per element, DEFINES from its Diagram, and a
DIAGRAM_ARROW edge per arrow (label, style, line);
- formats: Mermaid (flowchart, sequence, class, ER, state, C4,
architecture), PlantUML, Graphviz DOT, D2, draw.io (plain, compressed,
.drawio.svg) and Excalidraw (.excalidraw, .excalidraw.json,
.excalidraw.svg); .drawio.svg and .excalidraw.svg are no longer dropped
as images;
- an element's explicit link (Mermaid click/link, PlantUML [[url]], url of
and !include, draw.io and Excalidraw link, DOT URL/href, D2 link) is a
doc link read as a Markdown link, MENTIONS when it resolves.
OpenAPI:
- each operation of an OpenAPI or Swagger document (openapi.json and
swagger.json, now read with the JSON grammar but no node per key, and
any YAML or JSON file whose root says openapi/swagger) is a Route, named
by the same QN a route its handler registers in code gets
(cbm_route_canon_path, moved into the extraction library);
- its operationId is a doc link to the handler: a dotted one (Connexion,
with x-openapi-router-controller) as a qualified code name, a bare one
when exactly one function or method of the repository has that name.
The eight new link families are held until their held-out audit passes.
Read on real repositories, which shaped the rules: Excalidraw drawings put
text over shapes and draw arrows up to them without binding, so loose text
names the smallest box it sits in and a loose arrow end meets the smallest
named box it touches (the drawing's frame only at its border); arrows bound
to free text meet the text; D2 block strings and nested blocks; DOT line
breaks and sentence-long ids; PlantUML kinds of classes inside packages
and nested classes (`..+`, nested -> container, so they do not collide with
the container's association). Connexion's 24 specs: 160 operationIds, all
linked to their handlers.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
AsciiDoc (.adoc, .asciidoc) is now indexed, without a grammar like PDF: a
Section per heading (`=` .. `======`, or `#`), the text up to the next
heading as its docstring, delimited blocks respected (listing, literal,
comment, passthrough, example, sidebar, quote, table, fences). Its
references become MENTIONS edges, via "asciidoc", from the section they are
written in:
include include::target[...] -- Asciidoctor's preprocessor directive,
read in every block but a comment block; the page's own
attributes, then the Antora component's and the playbook's,
substituted
attribute an {attribute} whose value is an Antora javadoc link
({javadoc-root}/<module>/<package path>/<Type>.html[#member])
binds the Java or Kotlin type, or its one member, that the
repository's main sources declare under that path
code_path, monospace spans naming a path or a qualified name (the
code_name Markdown code-span rules)
The rules are the field test's (H8 Asciidoctor/Antora harvester, audited
~100 % precise): an Antora resource ID ([component:][module:]family$path)
names <component>/modules/<module>/<family dir>/<path>, or the file a
collector scan contributes there (a scan of build output is generated: no
row); any other target is relative to the including file. `lines=` and
`tag=`/`tags=` regions bind by H8's rule: edge blank and comment lines
trimmed, the innermost enclosing definition, unless that is a type (or there
is none) and the definitions inside cover every other line (an annotation
right above one counts) -- then that definition; tag regions come from the
included file's own `tag::x[]` / `end::x[]` comments, read from that indexed
file; a tag the file lacks binds the file. One reference makes one edge: a
region holding several top-level definitions binds the file.
Antora settings travel as YAML scope blobs (antora.yml: component name,
asciidoc attributes, collector scans; antora-playbook*.yml: attributes),
read as text, so an incremental run resolves as a full one; a change to them
is GLOBAL. AsciiDoc joins the documents bound by lines: a body edit of an
included file re-resolves its pages.
Also: the Markdown resolver's module index no longer takes AsciiDoc or PDF
files for code modules.
Proof on junit-team/junit-framework at 6c8ca116 (H8's corpus and commit),
C against H8's recorded links: javadoc attributes 362/362 identical;
includes 292 identical, 60 bind the type enclosing the several definitions
H8 names, 7 regions with several top-level definitions bind the file. 317
sections, 668 edges, no include rows (9 generated partials, 1 file type the
indexer does not read); monospace spans 30/30 correct by hand. 8.3 s for the
whole index.
Tests: suite doc_links_adoc (structure and tokens, the Antora blob, the
pipeline: resource IDs through a collector scan, tag regions, a region with
its imports binding its class, lines=, javadoc attributes, spans; incremental
== full: a moved tag region, a changed component attribute). Revert check:
without H8's region rule and without AsciiDoc among the line-bound documents
exactly their assertions fail (observed).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A PDF is now indexed: its text layer is read by a dependency-free extractor
(internal/cbm/pdf, zlib only) and every page becomes a Section node ("page N",
the page's text as its docstring, property page), so search finds what a page
says and get_code_snippet shows a page's text, never the file's bytes. The code
a page names structurally becomes MENTIONS edges from that page, via "pdf":
pdf_path a repository path ("pkg/server/handler.go", "./x/y")
pdf_file a file name with a code extension ("models.py", "Pool.sol")
pdf_qn a qualified name ("app.models.User.save", "a::b", "x->y")
Bare identifiers never become edges (field test E6c: a type-target rule failed
its pre-registered bar), and only EXACT or UNIQUE resolutions link: the field
test's tiers (path-exact, component-aligned path-suffix, qn-exact, qn-suffix,
the latter also through a node's owner -- a Go method's QN has no receiver --
and with the same one-entity collapses). A mention split across a line break is
proposed as one reference and replaces its fragments only when it resolves.
Hygiene as measured: no links from PDFs under test or fixture directories (R1),
none to data files (R2), none for keyword arguments (R3); a test, mock or
fixture target is a test_only_target row (R4, with the test directories the
field test found missing). A path or name whose first component is not of this
repository is no row at all.
The extractor is a port of the field-tested prototype: xref tables and
streams, /Prev chains, repair near a wrong offset and reconstruction from the
object headers, object streams, Flate (with partial recovery), LZW, ASCII85,
ASCIIHex, RunLength and PNG/TIFF predictors, simple fonts (base encodings,
/Differences, built-in Type1 encodings, standard-14 widths, Type3), ToUnicode
and encoding CMaps, Type0/CID fonts with the embedded TrueType cmap read
backwards, Form XObjects, inline images skipped, word and line breaks from
glyph geometry, right-to-left runs reversed. Its geometry is bit-exact on every
platform (no FMA contraction; CPython's hypot). Untrusted input: every read is
bounds-checked, recursion is depth-limited, integers from the file saturate, a
decoded stream is cut at 512 MiB, and every lookup that could rescan the file
per object or per glyph is answered from an index built once. Not read:
encrypted documents (reported as such); scanned pages have no text layer.
The mention scanner is the field test's (idents.py) with CPython's semantics:
NFKC per code point plus canonical composition and the str predicates, from
generated Unicode 16.0 tables (pdf_unicode.c), and re's leftmost-first
matching.
Pipeline: CBM_LANG_PDF (no grammar; the registry ledger gets a DOCUMENT
capability, the call-node manifest exempts it like the other grammar-free
classification), extraction before any grammar lookup, a resolver hook
file_prepare (a file's state from all its references before any resolves).
Incremental: a proxy node carries parent_class where owner + name differs
from its QN (Go methods, class-body variables), so a re-resolved document binds
a Go method as a full build does.
Proof against the field-test prototype on its corpus (749 PDFs from 21
repositories, restored at the recorded commits, sha256-verified): text
byte-identical on all 1,492 pages of the 720 comparable PDFs (0.73 s vs 14.7 s);
mention tokens identical on all 960 page-text sets of E6, E6b and E6c (26,435
tokens); links identical to the frozen Python pipeline run over the same graph
on semantic-kernel 5/5, cosmos-sdk 30/30, Polly 11/11, mediastore3 3/3, dask
4/4. 44,070 seeded mutants (truncations, flips, nesting and keyword bombs, on
the originals and on uncompressed copies) under ASan and UBSan: 0 findings
after the first campaign's integer overflows were fixed as a class.
Tests: suite doc_links_pdf (layout, fonts and CMaps, object and xref streams,
xref repair and reconstruction, Form XObjects incl. a self-referencing one,
inline images, filters, hostile input, the scanner against the field test's
output, the pipeline's tiers, hygiene and rows, back navigation, the snippet,
incremental == full).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Ten files changed on both sides; every conflict was a union of independent
additions:
- workflows (_lint.yml, _test.yml): keep the branch's lane `if:` and main's
pinned ubuntu-24.04 runner with its reason.
- internal/cbm/cbm.h: main's callable-identity and module-doc fields plus the
branch's configured-test fields.
- Makefile.cbm: both new test files; prod_lsp_all.o keeps main's
.build-config dependency and the branch's coverage flags.
- scripts/test.sh: main's setup-scripts contract becomes Step 0y3, ahead of
the branch's 0z series and the contracts-only exit.
- src/foundation/subprocess.c: main's cbm_posix_close_fds_from (it closes the
error descriptor too).
- src/mcp/mcp.c: main's repository default base for detect_changes, with the
early returns released through the branch's detect_release.
- src/pipeline/lsp_resolve.h, tests/test_main.c: both includes, suites and
start-up steps (main's git environment reset first).
- scripts/memory-core-baseline.txt: recounted on the merged tree and lowered
(mcp.c 773, config_toml_edit.c 73).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Since #2133 a cross-repo result owns heap memory: a wildcard run records
each store it skips in result.skipped_projects, and the caller releases
it with cbm_cross_repo_result_free(). Fifteen tests in test_cross_repo.c
received a result and never released it. The one that seeds a pre-#768
store next to a wildcard target records a skip, so it leaked the skip
array and the skipped project name: 32 bytes in 2 allocations, allocated
in cr_record_skip.
Every assertion still passed. The macOS leak lane (test-lsan-macos) was
the run that reported it, because it builds with Homebrew LLVM and sets
detect_leaks=1. The ordinary macOS runs, the local ones included, are
built with Apple clang, where leak detection is off and cannot be
turned on, so they stayed green.
Each test now releases its result right after the fixture is torn down
and before the first assertion, which is the shape the two tests that
already released theirs use. The release therefore also runs when an
assertion fails, and the scalar fields the assertions read stay valid
after it. The repro_issue523 cases release their results the same way.
They name their targets, so they record no skip and did not leak, but
they hold a result and follow the same ownership rule.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The test-impact selector has to hand the C runner individual tests, and the
map behind it has to know what each test executes. The runner could do
neither: argv selects whole suites.
Selection. CBM_TEST_ONLY (comma-separated) and CBM_TEST_ONLY_FILE (one per
line; blank lines and # comments allowed) take suite:test tokens, and are
unioned when both are set. A suite no token names never executes its body.
Inside a named suite only the named tests run; the rest are counted as
deselected and the summary reports the count. Argv still selects suites and
the tokens narrow inside that selection, so the parallel harness hands every
per-suite process the same list with no change of its own.
Tokens are strict: one that names a suite this build does not register, or a
test its suite never ran, is printed and fails the run. An unreadable file, a
malformed token, and a selection that names no test stop the run before any
suite. Without a selection the output is byte-identical to before, and
--list-suites ignores the selection.
Coverage. TEST_COVERAGE=1 compiles clang source coverage into our own code
(the one-shot sources, the LSP unity object, the preprocessor; vendored
objects stay uninstrumented) and the runner's hooks into the test runner.
`make test-runner-cov` and `make cbm-cov` build the instrumented runner and
product binary into build/cov. With CBM_TEST_COVERAGE_DIR set the runner
writes <suite>/<test>.parent.profraw for itself around each test, points
LLVM_PROFILE_FILE at <suite>/<test>.%p.profraw for the children the test
spawns, and sends what runs between tests to _setup.<n>.profraw. No other
build compiles any of it.
tests/test_harness_test_only.sh pins the selection contract against the built
runner and runs as Step 2b of scripts/test.sh, before the suites.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Git exports repository-local variables (GIT_DIR, GIT_INDEX_FILE,
GIT_WORK_TREE, ...) into hooks, and they take precedence over
`git -C <dir>`. Every fixture git command in the test runner therefore
acted on the caller's real repository whenever the suite ran with those
variables set: with GIT_DIR alone, fixture assertions failed and commits,
branches and linked worktrees were written into the caller's clone; with
the full hook set, fixture commits failed and git-dependent tests degraded
to "git not available" SKIPs. The pre-commit hook already clears the
variables (7883507d), but any other caller (a shell with GIT_DIR set, an
editor hook, `GIT_DIR=... scripts/test.sh`) still hit it.
The test runner and the repro runner now clear git's local-env list
(`git rev-parse --local-env-vars`) once at startup, before any fixture or
re-exec'd child spawns git, so every fixture scopes to its own repository
no matter who invokes the suite.
tests/test_git_env_isolation_contract.sh (Step 3a of scripts/test.sh)
drives the real runner's git_context suite with GIT_DIR, and with the full
hook set, aimed at a throwaway decoy repository. It requires the same
pass/skip summary as a clean run and a decoy left untouched (no refs, no
linked worktrees).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
For nearly every Python identifier, lexical_ancestor_kind and
python_default_value_reference climbed toward the root with ts_node_parent,
which re-descends from the root on every step, so a deep expression was
cubic. In error-free trees, climb the walk's occurrence cursor instead and
take a single parent step for global/nonlocal names, which the grammar makes
direct children; trees with errors keep the original climb.
sympy's polys/numberfields/resolvent_lookup.py drops from about 8.7 s to
54 ms in the definitions pass, ordinary Python extraction is about 25%
faster, and the graph is byte-identical.
Signed-off-by: Nelson Spence <nelson@projectnavi.ai>
Why: the test runner isolated HOME and CBM_CACHE_DIR but not
CBM_RUNTIME_DIR, and neither of the first two moves the daemon
rendezvous. Its key is a constant product hash (cbm_daemon_rendezvous_key,
"c888acc3ae367a1e"), and with no explicit parent and CBM_RUNTIME_DIR unset
cbm_daemon_bootstrap_endpoint_new falls back to the account-wide default
parent (/private/tmp on macOS, /tmp on Linux, LocalAppData on Windows).
Every endpoint a test resolved without an explicit parent was therefore
byte-identical to the developer's live daemon socket
(/private/tmp/cbm-daemon-<uid>/cbm-c888acc3ae367a1e.sock), whatever HOME
said.
The only in-runner product caller is CLI activation
(cli_activation_production_context_init, reached from cbm_cmd_install,
cbm_cmd_uninstall and update). Only a process-global seam kept it off the
default: SUITE(cli) sets that seam on entry and clears it on exit. On the
default endpoint, activation admits into the live daemon's cohort. When
the cohort is ours or busy, or the active cache fingerprint is unreadable,
activation prints "Stopping active CBM sessions" and drains them. That is
the same class of failure as an old binary's install, which stopped a
user daemon and six MCP sessions. One test outside SUITE(cli), or one
fixture that drops the variable, was enough to reach it.
daemon_bootstrap_runtime_dir_env_relocates_rendezvous did exactly that:
it restored by unsetting CBM_RUNTIME_DIR.
Fix:
- tests/test_main.c: tf_setup_cache_sentinel creates a fresh per-run
runtime parent (th_secure_runtime_parent_new "run", under /tmp on POSIX
and LocalAppData on Windows) and sets CBM_RUNTIME_DIR to it for the
whole run. It replaces any inherited value, the rule
scripts/test-runtime.sh applies to the shell harness. The existing
atexit cleanup removes the directory.
- src/daemon/bootstrap.c, CBM_ENABLE_TEST_SEAMS only: once a test runner
arms the guard, an endpoint request with no explicit parent and no
CBM_RUNTIME_DIR is refused with a stderr diagnostic instead of
resolving the default. No product binary arms it.
- tests/repro/repro_main.c arms the same guard. No reproduction needs the
rendezvous.
- tests/test_daemon_bootstrap.c: the relocation test restores the value
it found instead of unsetting it.
- tests/test_helpers.h: th_secure_runtime_base() names the anchor that
th_secure_runtime_parent_new already used.
Proof:
- Before, on origin/main 1f9b4db1: runner_isolation is RED 3/3 at its
first assertion (CBM_RUNTIME_DIR is NULL during the run).
- After: green 3/3. The endpoint resolved through the product resolver
lies inside /private/tmp/cbm-runtime-run-XXXXXX/, never at
/private/tmp/cbm-daemon-<uid>. All 43 activation-guard scopes logged
by the suites below are fixture or cli-suite directories, and the
guard never fired. The per-run directory is gone after exit, and the
live daemon and its runtime directory are unchanged.
- Revert: dropping the setenv gives RED (value is NULL). Disarming the
guard gives RED (refused_without).
Tests:
- runner_isolation: runner_isolation_daemon_runtime_dir_is_private,
runner_isolation_refuses_default_daemon_runtime
- Green on macOS: cli (328), mcp + mcp_mutation_guard (344),
daemon_runtime + daemon_application (106), daemon + daemon_version +
daemon_frontend + daemon_bootstrap + daemon_ipc (117), runner_isolation
+ activation_transaction + version_cohort + project_lock +
index_supervisor + private_file_lock + lock_registry (73). All skips
are existing Windows- or Linux-only platform skips.
- make -f Makefile.cbm lint-ci passes. test-repro-runner builds.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings the branch onto current main (fe85a6b2) as promised on the PR: the
only conflict was internal/cbm/helpers.h, where both sides added
declarations after cbm_find_child_by_kind; both blocks are kept (the PR's
cbm_find_children_by_kind prototype, then main's Lisp-family gates).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolves conflicts with the Chialisp language addition (#1920) in three
repro ledgers. Both branches added one call-capable language, so beyond
the textual conflicts, the auto-merged shared totals had to be bumped a
second time:
- repro_language_registry.c: capability partition now 88/27/49,
EXPECTED_CALL_CAPABLE_LANGUAGES 115, EXPECTED_NON_CALL_LANGUAGES 51,
EXPECTED_MATRIX_B_ROWS 49
- repro_call_node_manifest.c: historical total 226, active primary 196,
direct calls 160, constructor calls 22, primary owners 196
- repro_call_argument_matrix_b.c: ROUTINE_ARGUMENT_LANGUAGE_COUNT 39,
matrix static assert 49 (RACKET..CHIALISP)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Chialisp commit was formatted with the pinned Homebrew clang-format, which
is right for the production sources but wrong for the three test files it also
touched: tests/ is not in LINT_SRCS, so those files have never been under the
format gate and were already format-dirty on main. Running the formatter over
them rewrote code that had nothing to do with Chialisp — 478 whitespace-only
lines in repro_invariant_breadth.c alone, plus 50 in test_extraction.c and 14 in
test_lang_contract.c — which is a review burden and hides the actual change.
Restore those three files to main's bytes and re-apply only the Chialisp
additions, in each file's own house style. The result is token-identical to the
version that was tested, apart from one em-dash written as `--` in a comment.
Re-verified after the rewrite: the six chialisp extraction tests, the cross-file
Constant graph test, the matrix-B row, the breadth case
(`[INV-BREADTH] PASS chialisp calls=1 callable=1 module=0`), and `make lint-ci`.
The production sources keep their formatting: they ARE in LINT_SRCS, and the
only whitespace they carry is the trailing-comment realignment in cbm.h that
adding a longer enumerator forces, plus the re-indent in extract_defs.c from
moving the name-resolution block into an else branch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only
public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five
reference files from chia-blockchain@main it produced 3-34 ERROR nodes each.
Three defects account for all of them, and each desynchronised the rest of the
file rather than losing one node:
* comments were `/;.*\r\n/` -- CRLF-only, while real files are LF;
* `(include foo.clib)` was rejected because of the dot in the filename;
* `(defconstant NAME <expr>)` accepted only a primitive after the name.
So the grammar is written here instead, in tools/tree-sitter-chialisp/, and
files as first-party: its vendored LICENSE is the project's own, byte-identical
to the repository root, carrying no third-party copyright.
It is deliberately GENERIC -- source_file / list / symbol / string / number /
hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp
form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head
symbols; which lists are definitions is decided in extract_defs.c. A dialect
that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs
no regenerated parser. Acceptance: all five reference files parse with zero
ERROR and zero MISSING nodes; the three defects above are pinned as tests.
Extraction, and the four decisions in it:
* `Constant` is admitted to the cross-file name registry
(cbm_label_is_registry_symbol). Chialisp's condition codes live as
`(defconstant ...)` in a shared .clib and are referenced by name from every
puzzle; labelled but unregistered they are seeded nowhere, and roughly half
of a Chialisp graph would be present as nodes yet never the target of an
edge. All three registry-seeding sites route through that one predicate, so
they cannot diverge. No other language emits `Constant`, so the blast radius
is exactly Chialisp. Pinned by a two-file graph test that goes red when the
label is removed from the predicate.
* `export` / `namespace` are NOT definition heads. `(export foo)` names a
function already defined in the same file, so treating it as a def head
mints a second node for one symbol. They stay in the not-a-call filter.
* A parameter list is a `list` too, but it binds rather than invokes. Without
that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a
phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions.
* `(include *standard-cl-26*)` is a dialect directive, not a file dependency;
recording it would invent a dependency on a file that does not exist.
Three helpers that the defs, calls and unified walks all need -- the quote-data
gate, comment-skipping named-child access, and the Chialisp def-head set -- live
in helpers.c rather than being copied per translation unit. They previously
existed as byte-identical duplicates that had already drifted: extract_defs.c
skipped comments when reading a def head while its mirror in extract_unified.c
did not, so a comment between a head and its name desynchronised defs from call
scope. Both now use the same accessor.
Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87,
first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165.
Distilled from PR #919 by darkverbito, whose extraction logic -- the CLVM
not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and
the include/embed import handling -- is kept and is the substance of this change.
The grammar that PR vendored is not used.
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Merging main brought in three contracts this branch violated.
repro_call_node_manifest requires a table row for every live
call_node_types entry, so Pkl's unqualifiedAccessExpr/qualifiedAccessExpr
(direct) and newExpr (constructor) each get one, with the partition totals
moved to match. The historical-snapshot column is the audited ledger, not
an archaeology record -- the suite hard-fails on a row that claims it was
absent -- so a newly registered kind joins the snapshot with the change
that registers it.
repro_language_registry then requires every call-capable language to own
exactly one call-argument matrix row, which turned out to be a real
extractor gap rather than a bookkeeping entry. Pkl's grammar labels no
production with a `name` field, so the generic declared-container rule
never recognised a Pkl binding: method names, parameter names, and
property names were each re-emitted as an ordinary read of themselves,
which Go and Nickel do not do. Add a Pkl occurrence policy that binds the
declared identifier of methodHeader, typedIdentifier, classProperty,
objectProperty, clazz, and typeAlias, resolved against the nearest such
container so annotations, defaults, and bodies stay reads. Recording the
binding also supplies the lexical-shadow proof the matrix row asserts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Tony Thayer-Osborne <tony.thayerosborne@conductorone.com>
Pkl was wired end-to-end but shallow: on a real-world 172-file Pkl corpus
the graph had 386 IMPORTS edges and 0 CALLS edges, because pkl_call_types
was empty_types.
Pkl has no dedicated call node. `unqualifiedAccessExpr` and
`qualifiedAccessExpr` are the same node whether they are a call
(`helper(a)`) or a bare property read (`host`) — the only discriminator is
an `argumentList` child. extract_pkl_callee gates on that child, and is
dispatched from extract_callee_name with an unconditional return: falling
through to field-based or generic first-identifier resolution would mint a
CALLS edge for every property read in every Pkl file, since a bare access
expr's first child is an identifier.
Callee resolution:
helper(a) -> "helper"
utils.fallback(a) -> "utils.fallback" (module-qualified; cbm.c
shortens to the last dotted segment)
s.trim().toLowerCase() -> "toLowerCase" (a receiver that is itself a
call has parens in its text and is not prefixed)
new Server { ... } -> "Server" (links to the class def)
Also fills the other empty slots in the Pkl spec, all confirmed against
real parse trees rather than guessed: typeAlias (class), importGlobClause /
importExpr (import), ifExpr / whenGenerator / forGenerator (branch),
throwExpr (throw), annotation (decorator). forGenerator is additionally
registered in cbm_is_loop_node_type so `for (x in xs)` counts toward
loop-nesting depth; the name is Pkl-unique so no other grammar collides.
The Pkl repro test moves from the structural battery (dims 1-5) to the
full callable battery (dims 1-8) and adds an inline negative assertion
that bare property reads do NOT become CALLS edges — the regression the
argumentList gate exists to prevent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Tony Thayer-Osborne <tony.thayerosborne@conductorone.com>
The language-count contract shipped this morning in #1859 had a hole: it
matched only the prose form "<N> languages", so README's shields badge
("languages-158-orange") never matched and the gate reported success while
the badge sat three behind the prose in the same file.
The cause is worth recording. The surfaces were enumerated with the same
regex the gate then used, so the verification could only ever confirm what
the enumeration had already seen. A gate and its own completeness check must
not share a pattern.
Both claim forms are now matched, in the per-surface check and in the
unregistered-file sweep, so a new file stating a count in either form still
fails. README's badge is corrected to 161.
Also corrects a stale range comment left by the PL/SQL merge: matrix B's
header said "CBMLanguage 80..163" when PL/SQL had made it 164. That is the
comment variant of the additive-counter trap — the numeric constants were
audited across that rebase, the prose describing them was not. The two
sibling range descriptors (GO..ARKTS = 68 rows, RACKET..PLSQL = 47 rows)
were checked and are correct.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Vendor AndreasMaierDe/tree-sitter-plsql @ 28aebef209be (MIT, ABI 14, no
external scanner) and wire CBM_LANG_PLSQL end to end:
- extraction onto existing labels only: package spec/body -> Class,
procedure/function (create/definition/declaration) -> Function,
file -> Module; callee extraction for ref_call incl. the
package-qualified pkg.name form; identifier usage vocabulary with the
upstream 'parameter' kind (a ref_call ARGUMENT wrapper) carved out of
the whole-binding rule
- discover: .pks/.pkb/.pck/.pls/.plb/.plsql/.fnc/.trg/.bdy/.tps/.tpb
(.sql stays SQL and .prc stays FORM, both asserted negatively)
- ledgers: CALL_WITHOUT_REFERENCE_VOCAB(PLSQL), call-node manifest
ref_call row (historical total 219 -> 220), call-argument matrix B row
- tests: extraction triple (package body + package-qualified call,
standalone function, honest negative pin of the upstream
CREATE TYPE ... AS OBJECT ERROR-node limitation), grammar-regression
case, label golden Class:1,Function:1,Module:1, MUST_EXTRACT_DEFS,
extension mapping, bug-repro board battery row
- MANIFEST.md (community-niche verdict, local include-quote patch row),
THIRD_PARTY.md section, refreshed vendored checksums
PL/SQL support was contributed by Oguz (@ouzsrcm) in #1033; re-vendored
from upstream per vendoring policy with his language wiring distilled on
top.
Co-authored-by: Oğuz <oguzhan.saricam@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Routing .ets through the plain TypeScript grammar produced no definition
node for any @Component struct (0/19 across the OpenHarmony sample corpus),
parsed struct members as statement labels and build() as a call expression,
and registered ArkUI builtins (Column, Row, ListItem, ...) as user function
definitions with 28% fabricated cross-file CALLS edges. This adds a
dedicated ArkTS grammar that keeps everything TypeScript already handles
and models the ArkTS syntax TS cannot parse.
Grammar (tools/tree-sitter-arkts/, generated parser vendored to
internal/cbm/vendored/grammars/arkts/, ABI 15 via tree-sitter-cli 0.25.10):
first-party derivative of tree-sitter/tree-sitter-typescript v0.23.2
@ 75b3874edb2d (the commit our vendored typescript/tsx come from) on the
tree-sitter-javascript v0.23.1 base @ 3a837b6f3658 (the version upstream's
package-lock pins). LICENSE carries the upstream MIT text verbatim plus a
derivative notice; the external scanner is a symbol-rename trampoline over
a byte-identical copy of the already-vendored typescript scanner header.
ArkTS additions: decorated struct component declarations with class-shaped
bodies; UI-DSL trailing-closure component calls with attribute chains after
the closing brace; decorators on function declarations (@Builder,
@Extend(T), @Styles); @Extend/@Styles leading-dot attribute-chain bodies;
anonymous stateStyles style blocks; import lazy named imports.
Wiring: CBM_LANG_ARKTS enum + lang spec (struct_declaration as a class
kind labeled Struct, public_field_definition fields), .ets extension,
LANG_NAMES/userconfig entries, grammar shim, and TS-family joins across
the extractors and the name-resolution guards (parse_es_imports, JS
keywords/test-file/func-kind/scope tables, TS heritage/param/receiver/
namespace/type-ref/channel/usage paths, js_ts_family and the weak-method-
match suppression). Field definitions now also carry their decorators, so
@State/@Prop/@Link members are findable with their roles.
Tests: grammar corpus (35 cases), per-language regression + label golden +
import net + extension cases, seven ArkTS extraction tests including the
phantom-builtin regression (ArkUI builtins must never be definitions), and
capability-ledger / call-node-manifest / call-argument-matrix rows. On the
14-file OpenHarmony sample corpus the TS-grammar baseline had 12/14 files
with parse errors (231 ERROR nodes); the arkts grammar parses 14/14 clean,
and an end-to-end index yields all 19 components as Struct nodes, 68
methods, 119 fields, and zero phantom builtin definitions.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A C# method carrying three stacked attributes ([Foo][Bar("x")][Baz]) must
end up with three DECORATES edges, one per bracket group. The fixture is
indexed through the real production pipeline (rh_index) and asserts
rh_count_edges(..., "DECORATES") == 3, so it exercises the MCP path rather
than the extractor in isolation.
Verified to actually reproduce: with find_jvm_modifiers() reverted to its
first-child lookup the suite goes red (decorates=1, expected 3), and green
again with the fix in place.
C#-only by design. Parsing the equivalent fixtures with the vendored
grammars shows `[A] [B] [C]` yields three sibling `attribute_list` nodes in
C#, while PHP's `#[A] #[B] #[C]` yields ONE `attribute_list` holding three
`attribute_group` children — the pre-fix lookup already found all of them,
so PHP could not hit this bug and a stacked-PHP fixture would pass either
way. Java/Kotlin/Swift use the `modifiers` wrapper, a single node that never
repeats. PHP's DECORATES path stays covered by mkc_c3_php8_attribute
(tests/test_matrix_known_classes.c).
Registered in repro_main.c and Makefile.cbm's TEST_REPRO_SRCS.
Signed-off-by: Jumaga2015 <jumaga2015@gmail.com>
setenv/unsetenv do not exist on Windows, so the bug-repro runner failed to
COMPILE there -- this is the 'runner did not execute' failure on the
repro-windows CI job and the local Windows board alike. The compat layer's
cbm_setenv/cbm_unsetenv (already included via repro_harness.h) are the repo
idiom; test_pipeline.c uses them for the same variable.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Maintainer decision (option B) closing the Kotlin property-reference repro's
platform divergence at its root instead of tie-breaking the fallback.
`holder::handler` (parsed as navigation by the vendored grammar) produced a
USAGE edge whose target was chosen by the name-only registry fallback: with a
same-named property (Holder.handler) and function (Functions.handler) in the
project, the winner was registration order -- readdir order -- so Windows
(lexicographic NTFS) borrowed the FUNCTION while macOS happened to pick the
property. The graph's answer must not depend on directory enumeration.
The occurrence is now claimed by exact, receiver-typed resolution. Five links,
each of which was missing:
1. extract side: a Kotlin navigation member read is a semantic-reference
candidate at the member occurrence, so an LSP row can claim it. With no row
the join finds nothing -- but a candidate no longer falls back to the
name-only registry guess, which is the fail-closed direction this PR is
built on.
2. kotlin cross resolution: a property READ with a proven receiver emits a
CALL_REFERENCE row against the property. The property is not a callable
target, so the join can only produce USAGE, never a fabricated
CALL_REFERENCE. Value reads only; calls stay with the invocation machinery.
3. receiver typing across files: kotlin_resolve_class_name composed
<this module>.<name> for an unimported cross-file type, which can never
name a type defined in another file. A per-call unique-short-name map
(built from the project defs, ambiguous names fail closed) resolves the
annotation to the real registered QN. Hash lookup, no scans.
4. cross registry fields: pxc_map_label dropped Variable defs entirely, so no
cross registry ever saw a property. They now flow through (every language's
registrar filters by explicit label, so only Kotlin consumes them) and the
Kotlin registrar attaches them as fields of their receiver type,
hash-bucketed -- a per-type scan would be the registry-tail-scan quadratic
pattern.
5. def side: class-body variables now record their declaring class
(parent_class) -- previously only methods did. The QN stays module-level,
so this is additive metadata: the only structural parent_class consumer is
Method-gated (DEFINES_METHOD), verified in pass_definitions/pass_parallel/
pipeline_incremental.
Emission targets each field's REAL def QN carried through the field map:
kotlin class properties are minted with module-level QNs (proj.Holder.handler,
not proj.Holder.Holder.handler), so the composed form would name a node that
does not exist and the join would silently drop the edge.
The repro now proves the property edge on every platform for the same reason,
not by racing readdir.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Maintainer decision closing the last open RED from PR item 2.
`callback = handler` contains a genuine value occurrence of handler, and its
exact callable identity is proven -- the alias binding below could not be
created otherwise. The policy of this PR is proven exact value =>
CALL_REFERENCE, so that occurrence now emits a resolved reference row instead
of remaining an ordinary USAGE. Bare identifier RHS only, and the usage-carrier
rule in extract_usages.c is extended in lockstep, Python-gated -- a candidate
the LSP never matches would change nothing, but the asymmetry would be a trap.
The strategy distinction mirrors the argument path: a source name that is
itself a local alias claims lsp_callable_alias, so the local-shadow guard
admits exactly what it admits for arguments and refuses everything else.
In the graph the two occurrences inside `argument` -- the RHS and the alias
use in accept(callback) -- surface as ONE CALL_REFERENCE edge and zero USAGE
edges, because edge dedup is by (src, tgt, type) and collapsing multiple sites
between the same pair is its existing, intended behavior. That is precisely
what the fixture always asserted, so it needed no relaxation, only the comment
recording the decision.
Also: GCC rejects repro_call_argument_matrix_b's _Static_assert comparing
enumerators of two different anonymous enums (-Werror=enum-compare); clang
accepts it, so only the Linux leg saw it. Cast both sides to int.
repro_reference_precision + call_reference_contract: 100 passed, 0 failed.
This closes all five item-2 REDs.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
PR item 2. Four of its five REDs; the fifth is a policy question left open.
THREE SHADOW REDS -- a rebound name kept looking exactly bound.
An imported `callback` that Python has already rebound must lose exact callable
proof, or the graph reports a reference to a callable the name no longer refers
to. Three binders were invisible:
(callback := 0) module-level walrus
[(callback := 0) for _ in (0,)] walrus inside a comprehension
type callback = int PEP 695 type alias
The first two because a module-level expression statement is routed to call
resolution and never reaches the binding scanner at all, so no binder inside one
was ever seen. The third because the scanner had no branch for
type_alias_statement, and its generic recursion only descends -- an identifier
alone binds nothing.
The comprehension case needed care in the other direction: the whole node was
skipped, correctly, because its iteration variables are private to it. But an
assignment expression binds in the CONTAINING scope (PEP 572), which is exactly
what separates it from every other binder a comprehension can hold. It is now
scanned for walrus targets only.
PARENTHESISED CALLABLE ARGUMENT -- `accept((handler))` produced no reference.
The semantic row was recorded on the outermost parenthesised argument while the
usage carrier stopped at the parentheses and was never marked a callable-value
candidate at all, so the occurrence-exact join had nothing to join. A bare
identifier now climbs to the same wrapper python_direct_callable_attribute_site
already returns for the bound-method form, so `accept((handler))` and
`accept((service.handler))` agree on one occurrence instead of disagreeing.
That climb uses the cursor the caller already holds and checks the parent kind
before doing anything. Walking up with ts_node_parent instead costs one slow
fallback per identifier in the file, which the linearity guard in
test_extraction.c rejects -- correctly, and it caught exactly that here.
The fixture's expected span was also wrong: "((handler))" is the argument LIST,
one byte wider on each side than either party, and per argument it would collide
for any call with more than one argument. It now names "(handler)" -- the
occurrence both sides actually use. Production was genuinely broken; the fixture
was additionally wrong about where.
LOCAL CALLABLE ALIAS -- a proven alias was refused by the shadow guard.
`callback = handler` makes callback a local, so the usage is a local shadow, and
a shadowed usage may only claim a semantic reference through the alias strategy.
Python emitted lsp_callable_value_reference for every callable argument, so the
one case the guard exists to admit could never satisfy it.
A lexical binding that does not simply name the module symbol of the same
spelling is an alias introduced in this body, and only that now claims the alias
strategy. A module function referenced by its own name keeps the strategy it has
always had, so the guard still refuses everything it refused before.
STILL OPEN: the same fixture asserts no ordinary USAGE from `argument`, and the
right-hand side of `callback = handler` is a second genuine occurrence of
handler as a value. Whether that should become a CALL_REFERENCE (this PR's own
proven-value policy, making the count 2) or emit nothing is a graph-content
decision, not a defect, and is left for the maintainer.
repro_reference_precision, call_reference_contract, extraction: 371 passed,
1 failed (that fixture).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
mikemikimike
Nelson Spence
Tony Thayer-OsborneandClaude Fable 5
Jumaga2015