mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-06 16:53:56 +00:00
a35671e962f1f2fb4e94deaf55e71cf05f6b3828
291
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
7a13f401eb |
wip(resolver): JavaScript and TypeScript modules, scopes and exports
Checkpoint against a usage limit. Proved on express and kilocode subtrees; the test suites have not run on this tree yet, so later commits may adjust it. - Real lexical scopes replace the module-wide veto on any local name. - CommonJS syntax, module extensions, package.json type and TypeScript prove a module; a classic browser script stays unproved. - require bindings and an exported-name table: aliases, default exports, barrels, export *, export * as, module.exports and exports.x. - Directory imports resolve through main and index. - TS method proofs survive destructuring, catch, generics and decorators, take own and inherited methods, this.field and new X() receivers, and have no budget. - TS annotations give exact type-name occurrences. - A registered route suppresses the unresolved diagnostic again. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e79f61db49 |
fix(resolver): keep JSON strict when source text is not UTF-8
yyjson's strict writer refuses a whole document over one undecodable byte. Source files can be in any encoding, and the strict resolver puts source text in JSON on its way to publication, treating a NULL write as fatal: record_properties call arguments and callee -> aborted_previous_preserved fact_json proof lookup keys and scopes -> persist_failed pass_pkgmap import paths and names -> aborted_previous_preserved lsp_surface definitions and imports -> every file's surface row dropped Neovim and redis took the first route: a latin-1 string argument in a test file stopped the whole index. Main writes these same bytes into edge properties raw, so it published, with invalid UTF-8 inside that JSON. Add cbm_json_write_text: valid documents are written unchanged at no extra cost, and only a document the writer refused has its strings repaired, replacing undecodable bytes with U+FFFD. That covers every field of a document, rather than each field that happens to carry source text today. The exact-bytes columns (resolver_occurrences spelling, resolver_lookups key) keep the original bytes; the MCP layer normalizes those at its own encoding boundary. The pending unresolved-call key is compared against text read back from that JSON, so it applies the same replacement to both sides. The UTF-8 sequence check moves from mcp.c to foundation/str_util so there is one copy. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
6f9374281f |
Merge main a6d0d6c9 into the strict shared resolver
Threads main's Go rel_path (#1910) through the proof-source and import-snapshot variants, keeps main's ADR sidecar so the rebuild no longer re-applies ADRs, and keeps the feature's removal of the import name fallback and JVM tail join, inside which main had added a generated-member filter. Main took semantic index version 12 for Markdown Section names, so the resolver generation becomes version 13. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e9426010c1 |
test(graph_buffer): a failed edge append must not report success
Adds a class-filtered, thread-local, one-shot realloc failure hook compiled only with CBM_MEM_TEST_API, and a regression that fails the primary edge array's first growth. The old code registered the secondary and dedup entries and returned a positive id while the array the dump reads stayed empty, so a retry appeared to succeed and the edge was lost and leaked. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
9bb891b669 |
ci: move macOS runners to macos-15 and pin the deployment target to 14.0
GitHub retires the macos-14 runner image on 2026-11-02, and its brownouts already cancel our macos-14 jobs. Move every macos-14 runner reference to macos-15 (arm64): the release darwin-arm64 build, the unix-macos14 and tsan-mac test legs, the smoke-mac PR leg (its two matrix.os guards move in lockstep), the release smoke, both soak legs, the memwaste event lane, the bug-repro board, the cache warmer and the all-platform smoke. Lane names stay unchanged: select-lanes.sh, ci-ok and the lane contracts key on them, never on the runner label. Two broad legs already ran on macos-15 (_test.yml BROAD_UNIX, _smoke.yml BROAD_UNIX) and smoke.yml listed both images, so the moved legs would have duplicated them. Those duplicates are dropped; PR CI keeps exactly the same legs. Without an explicit minimum, clang stamps the binary with the SDK's default (LC_BUILD_VERSION minos), so a release built on macos-15 would refuse to start on macOS 14. Makefile.cbm now folds -mmacosx-version-min=14.0 into CC/CXX next to ARCHFLAGS for Apple targets (detected from the compiler's target, not the host), so it reaches every compile and link, vendored objects included, on every build path. Proof on macOS 26.6 / SDK 26.5: the production binary built from origin/main reports minos 26.0; with this change it reports minos 14.0, with zero warnings under -Werror, no weak imports and an identical undefined-symbol set. The test runner (direct make, no env.sh) is minos 14.0 too; platform and daemon_ipc suites pass. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b821d1506f |
Complete strict resolver proof bridges and type propagation
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
642fe3fe46 |
Complete proof bridges and migrate resolver quality contracts
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3d65c609e1 |
fix(mem): define MI_MINGW_UCRT64 for MinGW mimalloc builds
mimalloc 3.5.3 treats a MinGW build as having a process-attach hook only when MI_MINGW_UCRT64 is defined (src/prim/windows/prim.c). Upstream's CMake sets it; Makefile.cbm did not. src/prim/prim.c therefore also compiled its constructor, and _mi_auto_process_init ran twice on Windows: once from that constructor and once from the CRT TLS callback. The second run flushes the delayed-output buffer to stderr again, and the first flush had left a newline in it, so every Windows run started with a blank line on stderr. That is what failed smoke-win's one-line --version check. In the MI_DEBUG=2 test build the same second run reaches mi_assert_internal(mi_out_default == NULL) in options.c, which aborts. 3.3.2 had no such condition. Define it for IS_MINGW builds in both the production and the test mimalloc flags. Every MinGW toolchain this repository builds with (MSYS2 CLANG64 and CLANGARM64) is UCRT. No vendored source changes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7e55bc110b | Merge branch 'main' into fix/mimalloc-3.5.3-preinit-free | ||
|
Martin Vogel
|
47d5e1d618 |
Merge main into shared resolver implementation
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
891d6eb249 |
Require occurrence-bound evidence for cross-file edges
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
75f45ba248 |
Add shared semantic resolution with explicit proof admission
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
5aa31bee62 |
ci: parallelize cppcheck with isolated whole-program state
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
 Martin VogelandFelix Rath
|
f9f44b5869 |
fix(mem): vendor mimalloc 3.5.3 so free(NULL) before init is safe
With mimalloc 3.3.2's release build, free(NULL) before the allocator's constructor dereferenced a page map that does not exist yet. glibc 2.44's newlocale() does exactly that during libstdc++ start-up, so every Linux binary died with SIGSEGV before main (reported with a reproduction in #2562). Upstream fixed it in b2226c19 (mimalloc #1341), first released in v3.4.4; v3.5.3, upstream's latest published release, contains it. Vendored from microsoft/mimalloc tag v3.5.3 (tag object deac3dcf, commit d4881d338125e1cb7c47ba4cfb398d6f7c0c8d45): LICENSE, include/ and src/, written blob by blob from a bare clone. Before the local patches were reapplied, the staged LICENSE blob and include/ and src/ trees had the tag's exact object ids (50 files; src/arena-meta.c is gone upstream). Afterwards the only differences are two documented hunks: - src/prim/unix/prim.c: macOS always recommits on decommit, carried from |
||
|
Martin Vogel
|
453b21e680 |
ci(codeql): PR scan per language inside PR CI, full scan on main
Bring CodeQL back to pull requests as jobs of PR CI itself (needed by ci-ok, no polling), narrowed to first-party code, while push to main, the weekly run, the dry run and the release keep the full scan. - codeql.yml: one leg per language (c-cpp, javascript-typescript, python, actions), scope in .github/codeql/codeql-config.yml. The full scan traces the whole build, vendored C included. The PR scan (light: true) analyses each selected language's whole first-party scope; c-cpp compiles the third-party objects before tracing (scripts/build.sh --vendored-only) and then traces the first-party build plus the vendored files the PR changes (--reuse-vendored --retrace-from, make -W). Diff-informed analysis (codeql-action v4.38.1, forced on for PR calls) only restricts reported alerts. The alert check moves into each leg and reads the analysed ref. Push and PR runs cancel superseded runs per language; dispatched runs never cancel. Compiler cache off so every compile is traced. - pr.yml: a codeql job runs the legs of the selected codeql-* lanes; ci-ok needs it. - select-lanes.sh: codeql-cpp/-js/-py/-actions lanes, selected by a change in their scope (any C/C++ incl. vendored; JS/TS, Python and workflows inside the config's paths minus paths-ignore) and by every full run (undeterminable diff, workflows, build, CodeQL machinery). --codeql-scope LANGUAGE prints a list's in-scope paths. - codeql-retrace-vendored.sh: the vendored C files a PR changes (empty, never narrowing, when the diff cannot be determined). - Makefile.cbm: cbm-third-party builds the vendored prod objects only. - tests: the gate contract pins the new layout; the selector contract pins that a C change always selects the C scan, an undeterminable diff selects every language, and a vendored-only change retraces exactly the changed vendored files. Selector fixtures, ci-ok's job map, the replay's codeql-action failures (lane codeql-cpp) and the interface probes follow. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a22271570e |
Merge pull request #2677 from DeusData/feat/deploy-graph-gaps
feat(deploy): render Helm images, mark default-valued images, link crds and werf charts |
||
|
Martin Vogel
|
67ff40d5cb |
fix(kotlin): resolve calls through typed class properties (#1555)
The cross-file Kotlin walk resolved a call on a typed method parameter by type, but a call on a class property - a primary-constructor val, a body property or `this.prop` - fell to the name guess. The cross registry held each own-file class from its extraction defs, whose properties carry no type (constructor vals none at all), and the walk's typed AST registration of the same class was added behind that entry under the same QN, so every lookup returned the field-less one. Constructor injection is the common Kotlin dependency shape, so `repository.inserir(...)` on an injected port bound to whichever same-named method the guess found, including an implementation of another port (#1555). The walk now registers the file's own declarations into an overlay chained to the cross registry and checked first, indexed once the top-level declarations are collected. The sole-definer fallback walks the whole chain so its candidate set is unchanged. Typing more receivers exposed resolution gaps that the untyped receivers had hidden; each would otherwise turn a typed call into a wrong name guess: - a member of the receiver class wins over a same-named extension of the same file (Kotlin resolution order; the overlay holds both); - `x.hashCode()` / `toString()` / `equals()` on a receiver typed by another file: the per-file pass emits the kotlin.Any placeholder, the cross pass the declared override, and two targets at one occurrence are ambiguous to the call join. The per-file placeholder is dropped when the cross pass resolves the same occurrence; - the same members on a stdlib or JDK type (`count.toString()` on an Int) resolve to the kotlin.Any builtin instead of a seeded stdlib member that has no graph node; - a property READ on a typed receiver whose QN the graph spells differently (a nested class's property) never becomes a CALL_REFERENCE to a same-named function: the reference join falls back by name to non-callable targets only, which binds the property's own node; - a typed value without an `invoke` operator (`private val handshake: Handshake?`) no longer shadows the same-named member function; - inside a lambda whose typed receiver becomes `this` (`repo.let { .. }`), and inside a member extension, the enclosing class's members stay callable bare (the implicit-receiver chain); a call naming a class stays a constructor call. Fixes #1555 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
d21949ef2c |
feat(deploy): render Helm images, mark default-valued images, link crds and werf charts
Four gaps of the deployment graph, closed together.
Images named from a default value keep their Image; the edge says so.
A Dockerfile FROM / COPY --from filled by an ARG default (an ARG chain
names each ARG) carries default_of, as do compose `${X:-d}` images and
`docker build -t ${IMAGE:-x}` tags. An override found in the repository
adds its own edge: compose build.args and --build-arg make the Image the
build names DEPEND_ON the base they choose {override_of}; compose's .env
replacing an inline default marks the effective edge override_of and keeps
the default as {ref: image_default}, an image the service does not run.
Helm template images render (helm_template.c): a bounded Go
text/template + Sprig evaluator over the vendored gotmpl grammar, with
values read the way Helm reads them (go-yaml v2 scalars, float64 numbers,
Helm's coalesce). Each templated container image renders from its chart
family's defaults -- include / define helpers, with / range / if, subchart
values under their alias, global -- and the object USES it {default_of:
the values keys read}; the chart DEPLOYS it. A value that does not render
(a .Release field, lookup, a missing define, a field of a missing map, a
function not modelled) names no Image; a rendered value is checked against
the image reference grammar first. Releases render their charts with their
values: a Flux HelmRelease's spec.values, a kustomize helmCharts entry's
valuesFile + valuesInline (a chart in chartHome now links locally), a
Terraform helm_release's values + set (literals only; set_sensitive is
never read) -- each image that differs is the release's own DEPLOYS
{override_of}. A release of a chart outside the repository names images by
convention only where its values' `repository` names a registry host
{inferred: values}; a host-less one names none.
A chart DEPLOYS the CRDs of its crds/ {phase: crds}. A werf `.helm` dir
beside werf.yaml is a chart named after the werf project.
Incremental runs: a werf.yaml edit and a werf chart's values / templates
decline like a Chart.yaml chart's; a release's values file (a kustomize
valuesFile, a Terraform file()) declines too -- Terraform's now IMPORTS
{local_name: values} it, like kustomize's valuesFile.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
8df0639c62 | Break-glass merge PR #2642 | ||
|
Martin Vogel
|
8b4d5e8b5b |
perf(store): in-memory adjacency layer for trace_path and degree filters
On a kernel-scale index (8.5M nodes, 16.9M edges) trace_path's recursive CTE and search_graph's per-row degree subqueries spend seconds in B-tree seeks. graph_mem reads the edges once, in one sequential scan of the covering UNIQUE index, into a typed CSR (node id -> out/in neighbours with interned edge types) and answers both from RAM: - store_bfs (shortest-path mode): BFS over the layer with the CTE's exact semantics (minimal hop per node, the start only via a cycle back to it, (hop, id) order, probe row, JOIN-skips for ids without a node row); node rows and edge collection still come from SQL. Trail mode is unchanged. - cbm_store_search with min/max_degree: the two correlated COUNT(*) subqueries become cbm_adj_degree(); filter, order, page and total are the same SQL. Layers live in a process-wide registry (the daemon closes its stores after every request), refcounted across request threads, allocated through the memory core (CBM_MEM_CLASS_STORE) and bounded: all layers stay under a quarter of the budget, idle ones are evicted oldest-first, and a layer that does not fit is not built (SQL path, never a partial layer). A layer is keyed by db path + store generation + the rollback-journal file state (the header change counter every rollback commit bumps), read in one read transaction with the scan; only read-only query stores of sealed DBs qualify. The file state matters: the cross-repo pass writes CROSS_* edges into other projects' published DBs in place without a new generation; such a DB is in WAL mode and keeps the SQL path until the next publish. Kernel, macOS, same index, fresh daemon: trace_path schedule 7.5 s -> 21 ms warm (1.23 s first call incl. a 1.2 s build), search_graph Function min_degree 50 4.2 s -> 170 ms warm (1.35 s first). Layer: 16,890,273 edges, 237 MB. Responses byte-identical to the baseline on django, redis and the kernel (tools, Cypher and extra BFS/degree cases). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ce0ed0e3b9 |
feat(infra): an Ansible playbook or task file IMPORTS what it includes
Ansible's composition produced no structure: kubespray's 40 playbooks,
roles and task files, tied together by 217 import_playbook /
include_tasks / import_tasks / include_role / import_role / roles:
directives, were unconnected YAML files.
A playbook or task file -- a YAML file whose top level is a sequence;
manifests, values, CI and compose files are mappings -- now IMPORTS
{local_name: <directive>, via: ansible, line} what each directive names:
import_playbook a playbook from its directory; include_tasks /
import_tasks / include a task file from its directory, else from its
role's tasks/; roles: items and include_role / import_role a role's
Folder, roles/<name> at or above its directory (a role by "./" / "../"
path from the directory). The ansible.builtin. prefix is understood; a
templated target links nothing.
Measured on 17 infrastructure repositories: 211 of 211 composition pairs
of kubespray against an independent reading, nothing else in any
corpus; deterministic over three runs; the incremental re-index equals
a full index in 4 scenarios (an included file edited or deleted, an
include pointed elsewhere, a playbook edited).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
6316733532 |
feat(infra): a container's environment reaches its own program
The variables a container sets (K8s env, compose environment, Dockerfile ENV)
were container-scoped EnvVars of their own, joined to nothing: the readers of
a name point at the global EnvVar, which every program reading that name
shares, so a join through it was in the right program for 23 % of the pairs.
Each scoped EnvVar now CONFIGURES {strategy:env_read, dockerfile} the readers
of its name inside the files the container's image is built from:
- the image: a K8s container's literal image, built from exactly one
Dockerfile (Image -DEPENDS_ON {via:build}->); a compose service's own build,
else its image's; a Dockerfile's own ENV. A K8s image reference now names
its container (USAGE {ref:image, container}), so a two-container pod joins
each container to its own image.
- its files: the context sources COPY / ADD put into the final stage, the
stages it is built FROM and the stages it copies from, resolved in the
build context (the Dockerfile's directory or the repository root, the one
the sources exist under), less the context's .dockerignore, never a test
file. They are kept on the Dockerfile's Module (image_context, image_files,
image_ignore).
pass_docker now reads COPY / ADD from the instruction's own text, as Docker
does: the grammar lost a JSON form opening with `/` and, after a word with a
trailing blank, took the following instructions for more sources.
Incremental: every run recomputes every join edge. A closure repair reads the
inputs of the files it leaves alone from the store and replaces the stored
edges wholesale; a .dockerignore (now a semantic-manifest control) changing,
or a deletion of a file an image copies, rebuilds in full.
CBM_SEMANTIC_INDEX_VERSION 8 -> 9.
Measured on 17 infrastructure repositories: 80 pairs, 80 correct by hand;
the graph matches an independent reading of the Dockerfiles 80/80.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
77bc4927f9 |
fix(extraction): fold local URLs without changing topic identities
Refs #2291, #706, #1147. Preserve raw argument values and carry URL projections separately through compaction and spill. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
2cd0956560 | Break-glass merge PR #2597 | ||
|
Martin Vogel
|
8a7cbab13b |
Merge origin/main into feat/infra-graph
Brings in #2596 (MSan: hidden uninitialized reads) and #2068 (CodeAtlas web). README: main's CBM Atlas line and this branch's infrastructure-as-code line are both kept. tools/list metadata: main and this branch each stayed under the 20 KiB discovery budget, but together they reached 20,809 bytes. The branch's wording is shorter now (search_graph kind/namespace, one hint per trace mode: 136 bytes less), and the reviewed budget is 21 KiB so the topology/deploy/run modes keep their usage hints. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
738cfebb60 |
fix(fuzz): link UBSan's C++ runtime into the fuzz binary
The cbm-fuzz rule links through $(CC), the C driver. On Linux, clang's C driver links UBSan's C runtime but leaves out its C++ part (ubsan_standalone_cxx), while the C++ objects (preprocessor.cpp and the vendored simplecpp) carry -fsanitize=vptr checks. The link failed with undefined references to __ubsan_vptr_type_cache and __ubsan_handle_dynamic_type_cache_miss, so the fuzz job has never built in CI since it was added (dry runs 37212102899 and 38186491924). -fsanitize-link-c++-runtime on the link line pulls that runtime in. Checked in an ubuntu:24.04 arm64 container with CI's packages (clang 18, llvm, libclang-rt-dev): - a minimal reproduction (a C++ virtual call under -fsanitize=undefined, linked through clang) fails with exactly those two symbols without the flag, and links and runs with it; - `make fuzz` links cbm-fuzz (lsp_all.o was compiled with -g0 added: the 12.5 GB container ran out of memory on it with full debug info, which CI's runner does not); - macOS Homebrew clang accepts the flag under -Werror. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
4d9592b5fb |
Merge pull request #2068 from DeusData/feat/codeatlas-web
CBM Lens Beta: graph exploration and architecture workspace |
||
|
Martin Vogel
|
2d291002df |
Merge origin/main into feat/infra-graph
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0392068d49 |
Merge main into CBM Lens Beta
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
2ec0185b9a |
feat(docker): a Dockerfile DEPENDS_ON the images it builds on
Dockerfiles were Module + ENV/ARG Variables only: the 261 FROM and
114 COPY --from lines of the infra corpora joined nothing, and build
files named for a variant were not indexed at all.
- pass_docker (new, end of the K8s pass): FROM <image> and
COPY --from=<image> -> DEPENDS_ON {via: from | copy-from, image,
stage, line} from the Dockerfile's Module to an Image node. ARG
defaults are expanded (global ARGs in FROM, a stage's ARG / ENV in
COPY); a stage (by name or index), scratch and an ARG without a value
are no image.
- Image (new label, as approved): one node per repository, normalised
(docker.io for no registry host, library/ for an official image, a
registry host and port kept); tag and digest stay on the edge.
- Detection: Dockerfile.<x>, Dockerfile-<x>, Dockerfile_<x>,
Containerfile[.<x>] and <x>.Dockerfile are Dockerfiles, unless the
suffix is a language extension of its own (Dockerfile.j2 stays a
Jinja template) or a BuildKit ignore file.
- QNs: a Dockerfile.<variant> keeps its suffix (fqn.c and helpers.c,
which must agree); dropping it gave the variant the plain
Dockerfile's QN, merging both Modules. Covered by this branch's
index-format bump.
- Incremental: an Image no DEPENDS_ON / USAGE edge touches any more is
dropped on both routes, like a Process without SPAWNS; the
unreferenced-node GC takes a list of edge types.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
b8a5050813 |
feat(shell,ci): scripts and CI jobs SPAWNS the scripts they run
"Which CI jobs run hack/ci/run-e2e-tests.sh, and what does that script
start?" had no answer: a command running a script of the repository drew
no edge (SPAWNS reached Process nodes only, and repository scripts were
left out), and CI configurations were not read at all.
- A shell script, Make recipe or CI command that runs a script of the
repository -- by its path (./hack/x.sh, "$ROOT/hack/x.sh") or through a
shell (bash -e hack/x.sh) -- SPAWNS that script: its Module (the source
of the script's own edges, so CI -> script -> deploy chains connect),
else its File. The path must resolve to exactly one file; a program that
is no repository file keeps its Process edge.
- A new pass_ci.c reads GitHub Actions workflows and composite actions
(run: steps with their working directory, shell, env literals,
${{ github.workspace }} and ${{ github.action_path }}; no pwsh step, no
step of a Windows job) and Prow job configs (container command + args,
exec'd; bash -c TEXT as shell; a decorated job starts in the checkout).
The shell layer evaluates them like a script; their DEPLOYS and SPAWNS
edges come from the workflow / job file and carry "ci" and "job".
- env A="$X/y" PROGRAM: an assignment with an unknown value is no longer
taken for the program (it made a Process named after the value's last
segment and hid the program).
Measured on 17 repositories against an independent reading (libyaml for
the CI files, a separate shell reader for the commands): 292 of 292
script runs linked, 292 of 292 correct -- from Prow jobs (115 containers),
GitHub Actions steps, shell scripts and Make recipes, across 10
repositories (kubermatic, gardener, kube-hetzner, cluster-api, deckhouse,
k8s.io, ...); CI Process and deploy edges 4 of 4. Only SPAWNS (+295),
DEPLOYS (+1), Process (+3) and Chart (+1) changed; three runs build
identical graphs; index time unchanged (kubermatic 2.95 -> 2.93 s,
gardener 5.25 -> 5.25 s, kube-hetzner 1.13 -> 1.14 s, medians); editing or deleting scripts, workflows,
Prow jobs and Makefiles leaves the graph a fresh index builds (two
scenarios differ in SEMANTICALLY_RELATED edges and Make pattern-rule
nodes exactly as the previous build does).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
a197d0c684 |
feat(terraform): a helm_release DEPLOYS its chart
A Terraform module installing a chart with `resource "helm_release"` had no
edge to the chart: "which modules install this chart" had no answer.
A new Terraform / OpenTofu pass (run at the end of the K8s pass, like the
shell pass) reads the helm_release blocks of .tf / .tofu files and links,
to existing nodes only, from the block's node (resource.helm_release.<name>):
- chart = "./charts/x" or "${path.module}/charts/x": the Chart of that
directory of the module;
- a repository and a chart name, or chart = "oci://.../<name>": the shared
external Chart of that name (the node chart dependencies use).
Values are read from string literals and ${path.module} only; a variable
chart, or a directory without a Chart.yaml, is no edge. Properties: via
terraform, the chart, external, line.
Measured on 17 repositories against an independent reading of the
Terraform files: 21 of 21 releases linked, 21 of 21 correct (EKS
blueprints 17, k8s.io 3, kube-hetzner 1). Only these DEPLOYS edges and 8
external Charts were added; three runs build identical graphs; index time
unchanged (cloud-foundation-fabric 1.91 -> 1.89 s, EKS blueprints 0.87 ->
0.88 s, k8s.io 3.44 -> 3.45 s, medians); editing or deleting the files
leaves the graph a fresh index builds.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
3323cae0f8 |
feat(shell): shell and Make deploy commands DEPLOYS what they apply
A script or Make recipe running kubectl apply, helm upgrade or a kustomize
build piped into kubectl puts manifests, a kustomization or a chart on a
cluster, and none of it was an edge: "which script deploys this manifest"
had no answer in the graph.
A new pass (the Bash + Make layer, run at the end of the K8s pass) reads
scripts and Makefiles that name a deploy tool and links, to existing nodes
only:
- kubectl apply|create|replace -f: the File of a manifest, a directory's
manifests (-R: below it too), a glob's; -k: the Kustomization;
- kustomize build D (or kubectl kustomize D) | ... | kubectl apply -f -:
the Kustomization of D;
- helm install|upgrade [NAME] CHART, helm template piped into kubectl:
the Chart of a chart dir, else the shared external Chart of repo/name or
oci://.../name;
- find D -name G -exec kubectl apply -f {} \;: D's matching manifests.
The edge starts at the bash function or Make rule running the command,
top-level code at its file, with via (kubectl / helm / kustomize), the
deployed path or chart, and the line.
Words are evaluated by constant propagation, nothing is executed:
assignments (local / export / readonly / declare, arrays), ${X:-default},
$0 / BASH_SOURCE, $(dirname ..), $(cd .. && pwd), $(realpath ..),
$(git rev-parse --show-toplevel); cd / pushd / popd track the directory.
Anything else is unknown, and an unknown target is no edge: a value set in
a branch or loop is unknown after it, a subshell's changes do not leak, an
unquoted unknown word that may shift helm's positionals blocks the chart.
With no directory tracked, a relative path resolves against the root and
the script's directory, linked only when exactly one of them has a target.
Make: the Makefile's variables expand into each recipe line ($(shell pwd) /
$(CURDIR) = the Makefile's directory; a conditional's value is unknown),
which is then read as a shell line.
Measured on 17 repositories against an independent reading of the scripts
(its own tokenizer, variable and directory tracking, the file system): 86
of 86 deploys linked, 86 of 86 correct (k8s.io 66, kubermatic 12, gardener
8). Only DEPLOYS edges and 3 external Charts were added; three runs build
identical graphs; index time unchanged within 1 s (deckhouse 10 s, gardener
5-6 s, k8s.io 3-4 s, median of 3). Nine incremental edits (deploy lines
edited, removed and added, a Makefile, a deployed manifest, a new manifest
and a new non-manifest in an applied directory) leave the graph a fresh
index builds.
tree-sitter-bash folds `a | b` followed by a line `cat <<EOF | c` into
`a | (b cat | c)`; that shape continues the outer pipe.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
f4a49feba1 |
feat(docs): diagrams and OpenAPI operations in the graph
Diagrams name the parts of a system and how they connect; an OpenAPI document names its endpoints and their handlers. Both now enter the graph. Diagrams: - a Diagram node per diagram: a fenced block of a Markdown file (under its Section), a directive of a reST file (`.. mermaid::`, `.. uml::`, `.. graphviz::`, Sphinx `.. digraph::`), an Asciidoctor Diagram block of an AsciiDoc file (`[plantuml]`, `[source,mermaid]`, ...), or a diagram file of its own, named by its title, else by its file; - a DiagramElement node per element, DEFINES from its Diagram, and a DIAGRAM_ARROW edge per arrow (label, style, line); - formats: Mermaid (flowchart, sequence, class, ER, state, C4, architecture), PlantUML, Graphviz DOT, D2, draw.io (plain, compressed, .drawio.svg) and Excalidraw (.excalidraw, .excalidraw.json, .excalidraw.svg); .drawio.svg and .excalidraw.svg are no longer dropped as images; - an element's explicit link (Mermaid click/link, PlantUML [[url]], url of and !include, draw.io and Excalidraw link, DOT URL/href, D2 link) is a doc link read as a Markdown link, MENTIONS when it resolves. OpenAPI: - each operation of an OpenAPI or Swagger document (openapi.json and swagger.json, now read with the JSON grammar but no node per key, and any YAML or JSON file whose root says openapi/swagger) is a Route, named by the same QN a route its handler registers in code gets (cbm_route_canon_path, moved into the extraction library); - its operationId is a doc link to the handler: a dotted one (Connexion, with x-openapi-router-controller) as a qualified code name, a bare one when exactly one function or method of the repository has that name. The eight new link families are held until their held-out audit passes. Read on real repositories, which shaped the rules: Excalidraw drawings put text over shapes and draw arrows up to them without binding, so loose text names the smallest box it sits in and a loose arrow end meets the smallest named box it touches (the drawing's frame only at its border); arrows bound to free text meet the text; D2 block strings and nested blocks; DOT line breaks and sentence-long ids; PlantUML kinds of classes inside packages and nested classes (`..+`, nested -> container, so they do not collide with the container's association). Connexion's 24 specs: 160 operationIds, all linked to their handlers. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
dd3a294737 |
feat(k8s): Go operators link to the kinds they reconcile
A Kubernetes operator written with controller-runtime ties its Go code
to cluster objects in ways the type graph does not carry: the
controller builder names the kinds it reconciles, a CRD's group and
kind are backed by a Go struct, and the manifests ship inside the
binary through //go:embed. None of it was in the graph -- on the
kubermatic, gardener, cluster-api and deckhouse operators, "which
controller reconciles Seeds" or "which Go type is this CRD" had no edge
to walk.
A new operator pass (after USAGE resolution, reading each Go file once)
adds:
- HANDLES {via: controller, role: for|owns|watches} from the function
that builds a controller -- `For(&v1.X{})`, `Owns(...)`,
`Watches(...)`, `source.Kind(cache, &v1.X{})` -- to the Go type,
resolved as Go resolves it: the alias is an import of the file, and
the import path maps to a directory through the repository's go.mod
module roots (kubermatic's `k8c.io/kubermatic/sdk/v2/apis/...` lives
in `sdk/apis/...`); unqualified types are the file's own package.
gofmt chains with the '.' on an earlier line, comment lines between
links, and calls inside comments are handled;
- USAGE {ref: crd-type} from a CRD Resource (now carrying crd_group and
crd_kind) to `type <Kind> struct` in the package marked
`+groupName=<group>` or declaring `GroupName = "<group>"`;
- REFERENCES_FILE {via: go:embed} from a Go file to every file its
//go:embed patterns match (Go's path.Match per segment, hidden files
only under all:).
Go method QNs carry no receiver, so every reconciler's SetupWithManager
in one package is a single node standing for several definitions. The
graph buffer gains cbm_gbuf_foreach_span, which visits every definition
span a node stands for (collision spans not yet settled, else its
variants, else its own), so a call is placed in the right definition
whichever file it is in.
Measured on 17 infrastructure repositories against an independent
reading of the Go sources: controller HANDLES recall 343/343, precision
335/335 (kubermatic 125, gardener 79, deckhouse 80, cluster-api 59
facts); kubermatic acceptance: reconciler -> kind 13/13, CRD -> Go type
32/32, go:embed files 115/115. Compared with the previous build only
the new edges and the two CRD properties differ; three runs build
identical graphs.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
80e000a92a |
doc-links(c): Doxygen @ref and @see names become MENTIONS edges, held until their audit
A Doxygen comment in a .c or .h file (a block opening with slash-star-star
or slash-star-bang, a run of /// or //! lines) names code with `@ref name`
or `\ref name` (family doxygen_ref) and with the names an `@see`, `\see`,
`@sa` or `\sa` paragraph lists that have the shape of a symbol (`name`,
`name()`, `::name`, `Type::member`; family see). Code blocks, fences, code
spans, quoted labels and the word after \c, \p, \a, \b, \e, \em hold none; a
URL is the core's href family. A .h is parsed as C++, which does not change
the rule; other C++ and Objective-C files are not part of the leg.
C keeps no node for a function prototype, so the docs of a header's API
were never stored anywhere. The doc-link driver gets an optional loose_docs
hook, called after the definitions and the file's own doc: the C leg reads
every Doxygen block outside function bodies that no definition took, and
its references belong to the file (the File node is the source).
The scope ("c1") lists the definitions with nodes (function, variable,
type, enum constant, field, object-like and function-like macro) with their
qualified names, their linkage (static, or a type, constant or macro of a
.c file, is the file's own) and the parent of a constant or field, and the
pages, groups, anchors and sections the file's docs define.
doc_links_c.c resolves a name as Doxygen does, without imports:
- a page, group, anchor or section name is no code entity (graph_gap) and
never binds a symbol of the same name;
- then the documented file's own definitions (static ones too), its fields,
then what every file sees (external linkage, a header's types, constants
and macros); a static definition of another file is not visible;
- candidates count by qualified name (a prototype and its definition, #if
variants: one node); with `()` written a function or function-like macro
wins, then a definition wins over a macro;
- `Type::member`: a field of the struct or union Type, or a constant of the
enum Type;
- one candidate binds, several are ambiguous; a word of an @see paragraph
that names nothing and is a plain lowercase word is prose (Doxygen prints
it as text): no row; test code binds only from test code.
There is no scope_delta hook yet: a declaration or linkage change is a full
run.
The families are held (ships=false) until their held-out audit passes; the
suite ships them through the test seam.
Proof (ship build, Apple M5 Pro / 48 GB, base
|
||
|
Martin Vogel
|
7a321cac16 |
doc-links(php): PHPDoc @see references become MENTIONS edges, held until their audit
A PHP doc block names code with `@see Ref` at the start of a tag line (a
description may follow) and with inline `{@see Ref}`, both family `see`.
doclink_php.c harvests them (code spans and fenced blocks hold none; @link,
@uses and the other tags are not harvested; a URL is the core's href family)
and writes each file's scope: namespace blocks, use imports, types with
their supertypes already qualified in their own file, members, functions.
The persisted scope drops lines and imports: other files read names only.
Declarations inside a top-level conditional count as top level
(`if (!function_exists('f')) { function f() {} }`), never a function body.
doc_links_php.c resolves a reference as a FQSEN with PHP's own name rules:
- a leading backslash is taken as written; otherwise the first segment is
replaced when it is the alias of a class `use` of the namespace block the
reference is written in, else the block's namespace is put in front
(phpDocumentor's resolver);
- beyond that tool, as PHP itself has them: an unqualified function name is
taken from a `use function` import, else from the current namespace, else
from the global namespace; and self/static/parent name the documented
type and its superclass;
- a bare `name()` that would bind a function but is written in a type that
declares or inherits a method of that name is ambiguous (the method or the
function; the reference does not say which);
- members are looked up on the type, its traits, its superclass and its
interfaces (depth first); type, function and method names compare without
case, properties and constants with it;
- properties, class constants and enum cases have no graph node: graph_gap;
so has a type or function the graph folds into another one of the same
name in its file (two namespace blocks), which the scope flags;
- a name the repository does not declare is external outside the
repository's namespace roots (built-ins, Composer dependencies), else
missing. Residual: a function a dependency declares in the current
namespace is invisible, so the fallback would bind a global function of
that name.
There is no scope_delta hook yet: a declaration change is a full run.
The family is held (ships=false) until its held-out audit passes; the suite
ships it through the test seam.
Proof (ship build, Apple M5 Pro / 48 GB, base
|
||
|
Martin Vogel
|
196ca71cda |
doc-links(ts): JSDoc and TSDoc links become MENTIONS edges, held until their audit
A TypeScript, TSX or JavaScript doc comment (a block comment opening with a
slash and two stars) names code with {@link Ref}, {@linkcode Ref} (each with
an optional `|` label) and `@see Ref` at the start of a tag line that holds
no inline tag. doclink_ts.c harvests them (code spans and fenced blocks hold
none, {@linkplain} is not harvested, a hyphenated word after @see is prose, a
reference naming the declaration's own parameter is dropped, a URL is the
core's href family) and writes each file's scope: imports, exports, export *
barrels, top-level declarations, namespace bodies and functions or classes
declared inside a function body.
doc_links_ts.c resolves a reference as the TypeScript checker sees it from
the documented declaration, in one index over TS, TSX and JS files:
1. the namespaces around the reference, innermost first;
2. a function or class declared in the enclosing function body (the graph
gives it a file-level name: two of one name in one file share one node,
a graph gap);
3. the file's own top-level declarations;
4. its imports, followed through export { a as b }, export ... from and
export * barrels (cycle guard, depth bound; two barrels that answer
differently are ambiguous); a relative specifier resolves with the usual
extension and index rules, a bare one is external;
5. the standard library's global names: external;
6. the top-level declarations of script files (no import, no export): one
binds, several are ambiguous;
7. the members of the documented class or interface, `this.` included.
The other segments walk members or a module's exports. A member declared
without a node of its own is a graph gap; test code binds only from test
code. There is no scope_delta hook yet: an export change is a full run.
The families are held (ships=false) until their held-out audit passes; the
suite ships them through the test seam.
Proof: the TypeScript compiler (perf-bench/typescript b465fdbfe175), ship
build, against the field test's prototype records (cascade results left
out): 69 edges; 57 of the prototype's 61 pairs bound alike. The 4 others are
doc blocks without a definition of their own (on an `if` statement, a local
`var`, an enum member, a free-standing @typedef block), which the core does
not harvest. Of the 12 edges the prototype has no pair for, 4 are the same
targets the prototype resolved without recording a source node, and 8 bind
PropertyDescriptor to the repository's own src/lib/es5.d.ts, which the
prototype called an external library global (kept: the repository declares
it). Unresolved rows agree with the prototype's reasons apart from the
documented reason split (not_in_scope -> missing/ambiguous).
Known limit: package.json and tsconfig.json are not indexed (discovery
ignores them), so workspace packages and tsconfig `paths` stay external.
Tests (doc_mentions_ts): tokens with their lines; the scope blob and its
persisted form; every resolution rule, including a function-body
declaration that binds and two of one name that do not; incremental equals
full (a doc edit and a body edit repair one file, an export change is a full
run); 60 modules give the same edges and rows with one worker and several.
Each rule red with it reverted (the body declaration's visibility and its
line check, barrel ambiguity, the parameter rule, code spans, the library
globals).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
40c9af3737 |
refactor(infra): delete the line-regex infra parsers nothing calls
pass_infrascan.c carried line-by-line parsers for Dockerfiles, .env
files, shell scripts and Terraform, five file classifiers and a JSON
bracket cleaner; pass_envscan.c walked a repository for URL-valued
environment assignments. No production code calls any of them: the
graph gets these files from the tree-sitter extractors, and the
envscan walk said so in its own header comment. Their only callers
were 40 tests in test_pipeline.c that asserted the parsers' output,
so the suite measured code the index never runs.
Deleted: cbm_is_dockerfile, cbm_is_compose_file,
cbm_is_cloudbuild_file, cbm_is_env_file, cbm_is_shell_script,
cbm_clean_json_brackets, cbm_parse_{dockerfile,dotenv,shell,terraform}_source,
cbm_scan_project_env_urls(_excluded), their result structs, and the
tests and test helpers that existed only for them.
Kept, all with production callers or a planned one:
cbm_is_kustomize_file and cbm_is_k8s_manifest (pass_k8s), cbm_infra_qn,
the Helm Chart.yaml parser, and the secret detectors
cbm_is_secret_binding / cbm_is_secret_value (still covered by
infra_secret_detection), which the secret-hygiene work reuses. The
secret detector's prefix offsets now have their own names instead of
borrowing the Dockerfile parser's keyword lengths.
No graph change: the deleted functions were unreachable from the
index. Suites green: pipeline 299 (339 minus the 40 deleted), platform,
grammar_probe_f/g, infrascan, discover, language, lang_contract,
incremental, integration, edge_structural, configlink.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
f4277ae783 |
doc-links(perl): POD L<> codes become MENTIONS edges, held until their audit
A sub's POD section and the module's own POD (the File node's doc) name code with L<> codes: L<Module::Name>, L<Module::Name/section>, L</section>, with a `text|` prefix or not. doclink_perl.c reads them the way perlpodspec and Pod::Simple do (verbatim paragraphs and data regions hold none, an L inside an L is no link, a URL is the core's href family) and writes each file's scope: its packages, its subs and its headings. Lines are exact: a sub's doc is matched against the file's sections, and the module doc's runs (extract_defs.c) are found again in the source in order; a module doc is POD from its start, so a run that begins with text is read too. doc_links_perl.c resolves them: - a module is the file that declares its package; of several, the one the library layout names (.../Name/Space.pm), else ambiguous. The target is the file's File node, as the field test's audited prototype has it (a Module node is missing where a same-named Folder holds its qualified name, lib/Mojo.pm next to lib/Mojo/). - L<Module/section>: the sub its heading names when the module's file defines that sub; any other section is a place in the module's documentation, the module. - L</section>: a sub the heading names; a heading that names no sub is the page itself (local, neither edge nor row); a section the page has no heading for is missing. - a module the repository does not declare (CPAN, the core, a .pod page: discovery keeps no .pod file) and a man page are external; test code (t/, xt/, *.t, and cbm_is_test_file, which has no Perl rule) binds only from test code. There is no scope_delta hook yet: a change to a file's packages, subs or headings re-resolves every Perl reference. The family is held (ships=false) until its held-out audit passes; the suite ships it through the test seam. Tests (doc_mentions_perl): tokens with their lines, file-level and sub-level, each read once; every resolution rule; incremental equals full (a module doc edit and an added module-level reference repair the file, a heading change is a full run); 60 modules give the same edges and rows with one worker and several. Each rule red with it reverted (the module doc's line mapping, a module run read from its start, the missing section, the library layout). Mojolicious at 93f591742a7d (the family shipped, a measurement build) against the field test's prototype records of its .pm/.pl files: 950 (source, target) pairs both have; the 197 only the prototype has are all references within one file (570), which the core makes no edge of; 8 only here come from the extensionless scripts the prototype did not read. Every prototype row has its row here: external 102, missing 1, and the 172 targets in .pod pages external by the ruling for .pod pages. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3af25b432f |
doc-links(rust): intra-doc links become MENTIONS edges, held until their audit
Rustdoc's intra-doc links ([`Name`], [path], [text](path), the kind
disambiguators struct@, fn@, macro@ ...) and links that take their destination
from a link reference definition, bound the way rustdoc resolves them: the
item's module scope, its use declarations, the crate root and the crates
Cargo.toml names. The leg (doclink_rs.c extracts the links and each file's
scope, a Cargo.toml its crate facts; doc_links_rs.c resolves them) is ported
onto the restacked core: family rows carry their edge type, resolve() takes
the per-file state (none here). Both families are held (ships=false) until
their held-out audit passes; the suite ships them through the test seam.
The graph moved under the leg: cfg-gated twins are one definition with its
variants on one node (
|
||
|
Martin Vogel
|
a1190afbb0 |
doc-links(java, kotlin): Javadoc and KDoc references become MENTIONS edges, held until their audit
Java's {@link}, {@linkplain}, @see, @throws and {@value}, and Kotlin's [Name]
links, @see, @throws and @sample, bound the way javadoc and Dokka search for a
member (one resolver for both languages, so a Kotlin doc can name a Java type
and back). The leg (doclink_jvm.c extracts the references and each file's
scope; doc_links_jvm.c resolves them) is ported onto the restacked core:
family rows carry their edge type, resolve() takes the per-file state (none
here). All nine families are held (ships=false) until their held-out audit
passes; the suite ships them through the test seam.
The graph moved under the leg since it was written: a Java field's node is
named by the field alone (no longer by its whole declarator, `MAX = 10`), and
a field named like a method of its class has its own node,
<Owner>.<name>#field, where the method used to take the name. The tests
follow: the expected field names lose the initializer, a changed
initializer is a local change repaired file by file (it renamed the node
before, which forced a full run), and a field that shares its accessor's name
is now an edge target and a source (doc_mentions_jvm_java_families_and_reasons,
doc_mentions_jvm_node_ownership), where both were graph_gap rows.
Measured with the leg's binary (1078036f) and this one on the same input:
- elasticsearch subset (libs/ + server/src/main at f3dc4b0bd3a0, 5,956 files):
no edge lost; 66 new edges and 7 edges moved onto a field's own #field node;
81 graph_gap rows gone, no new row, no reason changed. A seeded sample of 8
of the gone rows read against the source: every new edge is right (a field
named like its accessor, enum constants, a field's own doc as the source).
- Exposed (Kotlin, c419caaab61f): identical, 896 edges and 371 rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
0c7ca34419 |
doc-links(go): Go doc links become MENTIONS edges, held until their audit
Go doc comments name code as doc links ([Name], [Recv.Name], [pkg.Name], [pkg.Recv.Name], an import path for pkg), the way go/doc/comment reads them. The Go leg (doclink_go.c extracts them and a go.mod's module path as the scope; doc_links_go.c resolves them against the package's declarations and its imports) is ported onto the restacked core: the family row carries its edge type, and resolve() takes the per-file state the core now passes (Go keeps none). The family is held (ships=false) until its held-out audit passes: a resolved doc link is a below_bar_tier row until then. The suite ships it through the test seam, like the other held families; doc_mentions_go_tokens asserts the production default. Kubernetes at 8c078bc32e90 with the family shipped (a measurement build): 1,928 edges (1,115 exact, 813 unique) and 101 rows (external 61, graph_gap 22, missing 18), the same numbers the leg measured on the old core. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
92f5edfe32 |
doc-links(asciidoc): AsciiDoc sections linked to the code they include and name
AsciiDoc (.adoc, .asciidoc) is now indexed, without a grammar like PDF: a
Section per heading (`=` .. `======`, or `#`), the text up to the next
heading as its docstring, delimited blocks respected (listing, literal,
comment, passthrough, example, sidebar, quote, table, fences). Its
references become MENTIONS edges, via "asciidoc", from the section they are
written in:
include include::target[...] -- Asciidoctor's preprocessor directive,
read in every block but a comment block; the page's own
attributes, then the Antora component's and the playbook's,
substituted
attribute an {attribute} whose value is an Antora javadoc link
({javadoc-root}/<module>/<package path>/<Type>.html[#member])
binds the Java or Kotlin type, or its one member, that the
repository's main sources declare under that path
code_path, monospace spans naming a path or a qualified name (the
code_name Markdown code-span rules)
The rules are the field test's (H8 Asciidoctor/Antora harvester, audited
~100 % precise): an Antora resource ID ([component:][module:]family$path)
names <component>/modules/<module>/<family dir>/<path>, or the file a
collector scan contributes there (a scan of build output is generated: no
row); any other target is relative to the including file. `lines=` and
`tag=`/`tags=` regions bind by H8's rule: edge blank and comment lines
trimmed, the innermost enclosing definition, unless that is a type (or there
is none) and the definitions inside cover every other line (an annotation
right above one counts) -- then that definition; tag regions come from the
included file's own `tag::x[]` / `end::x[]` comments, read from that indexed
file; a tag the file lacks binds the file. One reference makes one edge: a
region holding several top-level definitions binds the file.
Antora settings travel as YAML scope blobs (antora.yml: component name,
asciidoc attributes, collector scans; antora-playbook*.yml: attributes),
read as text, so an incremental run resolves as a full one; a change to them
is GLOBAL. AsciiDoc joins the documents bound by lines: a body edit of an
included file re-resolves its pages.
Also: the Markdown resolver's module index no longer takes AsciiDoc or PDF
files for code modules.
Proof on junit-team/junit-framework at 6c8ca116 (H8's corpus and commit),
C against H8's recorded links: javadoc attributes 362/362 identical;
includes 292 identical, 60 bind the type enclosing the several definitions
H8 names, 7 regions with several top-level definitions bind the file. 317
sections, 668 edges, no include rows (9 generated partials, 1 file type the
indexer does not read); monospace spans 30/30 correct by hand. 8.3 s for the
whole index.
Tests: suite doc_links_adoc (structure and tokens, the Antora blob, the
pipeline: resource IDs through a collector scan, tag regions, a region with
its imports binding its class, lines=, javadoc attributes, spans; incremental
== full: a moved tag region, a changed component attribute). Revert check:
without H8's region rule and without AsciiDoc among the line-bound documents
exactly their assertions fail (observed).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
1b444c6b61 |
doc-links(rst): reStructuredText sections linked to the code Sphinx markup names
A reST document now gets a Section per title (the text up to the next title
as its docstring, its lines from the title to the next one) and MENTIONS
edges, via "rst", from each section to the code it names:
role :class:`pkg.Model`, :meth:`~x.Y.z`, :func:`.f`, :c:func:`f`,
and extlink roles whose URL is a repository blob path
(:source:`django/db/models/base.py`)
object .. py:class:: / .. method:: / .. function:: ... and their
C-domain twins
autodoc .. automodule:: / .. autoclass:: ...
include, .. include:: / .. kernel-include:: of a code file,
literalinclude .. literalinclude:: (its :lines: bind the innermost
definition holding them, kept as target_lines; its
:pyobject: the named definition)
kernel_doc .. kernel-doc:: path and its :identifiers: / :functions:
code_path, inline literals naming a path or a qualified name (the
code_name Markdown code-span rules)
Structure is the field test's line model (H3): titles by adornment style,
literal blocks after `::` and in code directives, comments, directive
arguments and options, Sphinx's module / currentmodule / class contexts,
CPython's str whitespace and \w. Text before the first title belongs to the
file. A Sphinx documentation set's conf.py is read as text, never run: its
source_suffix makes discovery index e.g. Django's `.txt` documents as reST in
that set only, its primary_domain decides undomained roles, its extlinks and
intersphinx names are known to the resolver.
Python names resolve the way Sphinx imports them: the longest prefix that is
a module's import name (named from its topmost package directory), the rest
by qualified name, a package's re-exports (`from .base import Model` in its
__init__.py) followed. The re-exports and the conf.py settings travel as
Python scope blobs (doclink_py.c), so an incremental run resolves as a full
one; one resolver serves reST and Python. Forms in Sphinx's order (T, C.T,
M.T, M.C.T; a leading `.` reverses it). EXACT when a form names a node,
otherwise UNIQUE by name with the field test's fixes: a bare member
(:meth:`save`, :attr:`x`) is never searched (37 of 40 such results were
wrong), an object directive's target lies in its declared module, a module
never answers a member's role (`:meth:`dispatch`` is not tests/dispatch),
test code is no target of a search but binds when the document writes its
name in full (`django.test.TestCase`). C identifiers are whole names: the one
C/C++ definition of that name and kind. Unresolved: missing when the name or
path is the repository's, ambiguous, test_only_target; another project's
name (stdlib, intersphinx) is no row.
reST architecture decision records: the ADR detector reads reST records
like Markdown ones (titles underlined with = or -): course-discovery's 34
decision records become ADR nodes.
Proof on Django's documentation (the field test's corpus and commit), C
against the frozen H3 harvester over the same graph, per (document,
section, target): 6,695 of H3's 7,238 links identical; H3-only 543 = links
C attributes to the document because a section title repeats in it (the
Markdown rule; the same document -> code link exists), deliberate refusals
(bare members, directive searches without a module) and H3's own audited
error class (router protocol methods -> ConnectionRouter, forms Field ->
model Field); C-only = the same attribution mirrored, plus 1,053 inline
literal links (sample of 30: 30 correct). 6,574 reST sections; 12 s for the
whole index.
Tests: suite doc_links_rst (structure and tokens, the scope blobs, the
pipeline's rules, source_suffix in discovery, reST ADRs, incremental ==
full). Revert check: without the module-kind guard and without the
source_suffix mapping exactly their assertions fail (observed). Two
expectations of the old behaviour now name the new one: the reST grammar
golden and probe have the document's Sections (they recorded "reST titles
are no Sections", the gap this closes).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
033ca372a8 |
doc-links(pdf): PDF pages as Sections, linked to the code their text names
A PDF is now indexed: its text layer is read by a dependency-free extractor
(internal/cbm/pdf, zlib only) and every page becomes a Section node ("page N",
the page's text as its docstring, property page), so search finds what a page
says and get_code_snippet shows a page's text, never the file's bytes. The code
a page names structurally becomes MENTIONS edges from that page, via "pdf":
pdf_path a repository path ("pkg/server/handler.go", "./x/y")
pdf_file a file name with a code extension ("models.py", "Pool.sol")
pdf_qn a qualified name ("app.models.User.save", "a::b", "x->y")
Bare identifiers never become edges (field test E6c: a type-target rule failed
its pre-registered bar), and only EXACT or UNIQUE resolutions link: the field
test's tiers (path-exact, component-aligned path-suffix, qn-exact, qn-suffix,
the latter also through a node's owner -- a Go method's QN has no receiver --
and with the same one-entity collapses). A mention split across a line break is
proposed as one reference and replaces its fragments only when it resolves.
Hygiene as measured: no links from PDFs under test or fixture directories (R1),
none to data files (R2), none for keyword arguments (R3); a test, mock or
fixture target is a test_only_target row (R4, with the test directories the
field test found missing). A path or name whose first component is not of this
repository is no row at all.
The extractor is a port of the field-tested prototype: xref tables and
streams, /Prev chains, repair near a wrong offset and reconstruction from the
object headers, object streams, Flate (with partial recovery), LZW, ASCII85,
ASCIIHex, RunLength and PNG/TIFF predictors, simple fonts (base encodings,
/Differences, built-in Type1 encodings, standard-14 widths, Type3), ToUnicode
and encoding CMaps, Type0/CID fonts with the embedded TrueType cmap read
backwards, Form XObjects, inline images skipped, word and line breaks from
glyph geometry, right-to-left runs reversed. Its geometry is bit-exact on every
platform (no FMA contraction; CPython's hypot). Untrusted input: every read is
bounds-checked, recursion is depth-limited, integers from the file saturate, a
decoded stream is cut at 512 MiB, and every lookup that could rescan the file
per object or per glyph is answered from an index built once. Not read:
encrypted documents (reported as such); scanned pages have no text layer.
The mention scanner is the field test's (idents.py) with CPython's semantics:
NFKC per code point plus canonical composition and the str predicates, from
generated Unicode 16.0 tables (pdf_unicode.c), and re's leftmost-first
matching.
Pipeline: CBM_LANG_PDF (no grammar; the registry ledger gets a DOCUMENT
capability, the call-node manifest exempts it like the other grammar-free
classification), extraction before any grammar lookup, a resolver hook
file_prepare (a file's state from all its references before any resolves).
Incremental: a proxy node carries parent_class where owner + name differs
from its QN (Go methods, class-body variables), so a re-resolved document binds
a Go method as a full build does.
Proof against the field-test prototype on its corpus (749 PDFs from 21
repositories, restored at the recorded commits, sha256-verified): text
byte-identical on all 1,492 pages of the 720 comparable PDFs (0.73 s vs 14.7 s);
mention tokens identical on all 960 page-text sets of E6, E6b and E6c (26,435
tokens); links identical to the frozen Python pipeline run over the same graph
on semantic-kernel 5/5, cosmos-sdk 30/30, Polly 11/11, mediastore3 3/3, dask
4/4. 44,070 seeded mutants (truncations, flips, nesting and keyword bombs, on
the originals and on uncompressed copies) under ASan and UBSan: 0 findings
after the first campaign's integer overflows were fixed as a class.
Tests: suite doc_links_pdf (layout, fonts and CMaps, object and xref streams,
xref repair and reconstruction, Form XObjects incl. a self-referencing one,
inline images, filters, hostile input, the scanner against the field test's
output, the pipeline's tiers, hygiene and rows, back navigation, the snippet,
incremental == full).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
6b0761160d |
doc-links(adr): architecture decision records as ADR nodes with their facts and SUPERSEDES
A Markdown file that is an architecture decision record now also defines an
ADR node (File DEFINES ADR): its name is the record's canonical id ("ADR-12";
another prefix as written, "DEC-5"; a dated record keeps its file stem), its
qualified name "<module>.__adr__", and its properties carry the record's
facts: adr_id, title, status (normalized: proposed, accepted, implemented,
superseded, deprecated, rejected, abandoned, draft, unknown), status_text,
date, deciders, superseded_by, and the decision itself as the docstring, so
search finds a record by what it decided. Its sections link to code like any
document's (MENTIONS, previous commit).
Detection is per file, with the field tests' rules: a file in a conventional
ADR directory (adr, adrs, decisions, decision-records, ...) whose name carries
an id or whose text has ADR structure; any adr-NNN file; a numbered or dated
file with ADR structure. Never README, index or template files without an id,
never test fixtures. Facts come from front matter, then header fields
(bullets, bold, tables), then the Status section; the date also from a
"Last updated" field, the change log's first date and the file name.
An ADR's own statement that it supersedes another ("Supersedes [ADR-3](...)",
"Supersedes ADR-3"; "Replaces ADR-3" where it opens a line, never "replaces"
as a verb of prose) becomes a SUPERSEDES edge between the two ADR nodes, by
link or by id. "Superseded by" is written in the old record about another
file; an edge from the other node would be owned by a file whose text does
not hold it, so it stays a fact of the old record (status, superseded_by).
The link layer gains a per-family edge type (doclink.h: the family list's
"edge" column), so a family can produce SUPERSEDES instead of MENTIONS; edges
are grouped by (type, source, target). Definitions gain a tail field for a
document's facts (extra_props: key, value pairs), carried through result
compaction and written by both property builders.
Measured against the field-test harvester on five repositories: detection
182 of 182 records, the same files; title 182/182; status 179/182 (cosmos
ADR-050 and its annexes say "Status: ARCHIVED", kept as status_text with
status unknown); date 174/182 (semantic-kernel's "date: {2024-05-02}" is read
as a date where the harvester dropped it). SUPERSEDES: semantic-kernel
0038 -> 0015, log4brains 20201016 -> 20200926, each checked against its
sentence. Markdown MENTIONS unchanged on fastapi, curl, dapr, cosmos-sdk.
Not covered: ADRs in reStructuredText, and ADR-log configuration files
(.adr-dir, log4brains) as a detection source.
Tests: doc_links_md_adr (detection and exclusions, facts from front matter
and from a Status section, SUPERSEDES by link and by id, no edge from prose,
the record's MENTIONS, incremental == full for a status edit and for an
added record named by id).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
afedff5176 |
doc-links(md): link Markdown sections to the code files and definitions they name
A Markdown document is documentation in itself: its sections now link to the code they reference explicitly, as MENTIONS edges (via "markdown") from the Section node (the File node for text before the first heading, and for a heading whose title repeats in the file, whose Section node it does not own). An agent can go from a document to the code and back through the graph: trace_path inbound with edge_types ["MENTIONS"] lists the sections that name a definition. Families (the "syntax" property; each with its ship gate): - link: the destination of an inline link or a link reference definition (no URL, no in-page anchor, no image); - path: a bare path in prose (a slash and a file extension); - code_path: a code span or HTML <code> span holding a path, a file name or a directory, with an optional line range (#L3-L9, :3-9) or member (path::Name); - code_name: a code span holding a qualified name (a.b, a::b, A#b, A\B, mod:attr). Resolution is guess-free, with the rules the documentation field tests measured: - a path is read relative to the document and, when it has a directory part, to the repository root; a file name alone only next to the document; - a line range binds the innermost code definition holding the whole range, else the file with "target_lines" on the edge; path::member the one definition of that name in the file; - a qualified name binds the one code definition whose qualifier chain ends with the written qualifier, else the one module or package so named; a match as written outranks one that needs case folding; a field binds only through its type, never through an instance; module:attr only in Python; product documentation never binds test code; - a bare name is never an edge. Unresolved references are rows in doc_link_unresolved: missing (the directory or the repository's own root is there, the name is not), ambiguous, test_only_target. A URL, an image, a URL route (/docs), a file of a type the indexer does not read, and code of other projects are no references to the repository and leave no row. Extraction scans the document's bytes once (doclink_md.c); every scan of a line is linear, or n log n where it sorts, using position lists that grow with the bytes they track, not with the line. Front matter, fenced code, HTML comments and MDX import/export lines are not scanned. The incremental route's node proxies now carry their line numbers (pipeline_delta.c), so a re-resolved document binds a line range exactly as a full build does; the Markdown resolver reads definitions from its own by-file list of the run's nodes, never from DEFINES edges, which an incremental run holds for re-extracted files only. Measured on four repositories against the main branch (which links file to file only, REFERENCES_FILE): fastapi 2,899 MENTIONS edges (2,225 REFERENCES_FILE before), cosmos-sdk 1,286 with 205 distinct definitions reached and 188 edges from 41 ADRs into code, dapr 184, curl 52. Link pairs agree with the field-test harvester (cosmos-sdk 408 of 409; the difference is a link of a document to itself, skipped on purpose). A hand check of 45 sampled qualified-name links found one wrong, fixed by the exact-case rule above. Tests: doc_links_md (extraction, repeated headings, the span classifier table, resolution with every reason, back navigation, incremental == full for a code edit, a document edit and an added name). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ea9fe36b9e |
doc-links: doc-comment references become MENTIONS edges, C# first
A reference written in a doc comment is extracted with the definition it
documents (internal/cbm/doclink.c; the C# scanner doclink_cs.c) and
resolved per file in the pipeline (src/pipeline/doc_links.c,
doc_links_cs.c, doc_links_msbuild.c). A reference that names a definition
of the repository with certainty becomes a MENTIONS edge from the
documented definition (the File node for a file-level comment) with the
properties via, syntax, tier, line and count. Any other reference becomes
a row of the new doc_link_unresolved table with its reason: missing,
ambiguous, external, test_only_target, graph_gap, unparseable or
below_bar_tier. Nothing is guessed. index_status reports the layer in a
doc_links block.
C# binds a cref by the compiler's lookup rule: the first scope with the
name decides, a cref never binds an inherited member, System is outside
the repository. Assemblies come from project files, and MSBuild global
usings (<Using>, ImplicitUsings, Directory.Build.props/.targets, imports,
conditions) are evaluated from the project files' content as data the
indexer has already read; the layer opens no file. Two declarations of a
name in different assemblies make the reference ambiguous.
Each file's scope is stored line-free with its LSP surface ("dl"), so an
incremental run re-resolves the files a change can reach and falls back
to a full run for a change that can re-route references anywhere (a
project file, a global using, a deleted file).
The scanner and the resolver treat the repository as untrusted input:
bounded nesting, no input-driven recursion, work that grows linearly
(held by test seams, never by a clock), no silent limits, and a damaged
or foreign stored scope fails closed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
42a9320200 |
graph: one identity per conditional definition, SPAWNS edges, no cross-language guesses
The test-impact engine selects tests by walking the graph, so it can only be as right as the graph. Three gaps made it wrong: One identity for a definition's mutually exclusive variants - A definition written once per #if/#else branch, per #[cfg], per platform file (foo_linux.c / foo_win.c, platform directories) or per Go build tag is ONE node. The graph buffer keeps every variant's span in a `variants` node property (definition labels only), so a change in any variant maps to the definition. - Extraction now sees definitions inside conditional blocks where it skipped them: C++/C#/ObjC class members under #if (one cursor walk, conditional block types per language in lang_specs), Elixir defs under if/unless, Fortran .F90/.F95/.F03/.F08, Rust #[cfg] twins (no more `name#cfg(...)` QNs that lost every call), C platform files and directories (non-static file-scope functions). - Haskell: the vendored scanner reads only an #if group's first branch. The grammar stays upstream; every further branch is parsed as its own same-length view (cpp_branch_views.c) and adds what the raw parse missed. haskell/bytestring: no new partial parses, +27 functions, IMPORTS 322 -> 329, 131 definitions with variant spans. - Incremental closure repair re-parses every file of such a definition when one of them changes (variant partners through DEFINES), so a repair equals a full index. No cross-language name guesses - A name-only resolution (unique_name, suffix_match, qualified_suffix, field_type_hint, and same_module across files that only share a path stem) never binds a symbol another language defines. - Bash imports are the files a script sources: every top-level command was read as an import of its name (`set -e` imported "set", bound to a TSX method in 137 scripts here). - This repository: cross-language CALLS 5,916 -> 5,091; the remainder are C/C++ pairs, builtins and same-stem module collisions (recorded). SPAWNS edges (per language, like the HTTP client patterns) - A call that starts a program (subprocess.run, exec.Command, child_process.spawn, posix_spawn, system, Process.Start, Command::new, ... about 30 languages, through the file's own imports) gets a SPAWNS edge to a Process node named after the program (literal first word, or <dynamic>) instead of a CALLS guess. No target resolution. Both resolvers ask at the same point (parity test). The engine follows the graph - Seeds map changed lines through every variant span; a test written once per #if branch is one test; when a change reaches a non-test `main`, the tests that spawn processes are reached too (the process runs main, and main reaches everything). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |