v0.12.0 could not be installed through npm, pnpm, pip or the Go wrapper on
any platform (#2657): each wrapper refused an archive member above 256 MiB,
and every v0.12.0 binary is 331-335 MB. The v0.11.0 binaries were already
about 300 MB, so the Go wrapper, which has had the cap since v0.10.0, was
broken before; npm and PyPI gained it in v0.12.0. The archive is verified
against the release's checksums.txt before it is unpacked, so a byte cap
only ever refused what we ship.
- npm, PyPI, Go: delete every byte-size cap (compressed archive, per member,
aggregate, checksum manifest, download size). The checksum, the exact
member set, path and link refusal, the member count, HTTPS and the
--version check stay.
- All three honour CBM_DOWNLOAD_URL like install.sh, for testing: https, or
plain http to an exact loopback authority with no proxy and no redirect.
- Smoke: scripts/smoke-wrappers.sh installs the three wrappers from the
checkout against the release fixture and runs the binary each one
installed. smoke-test.sh Phase 13w runs it; SMOKE_REQUIRE_WRAPPERS=1 makes
it required on the PR smoke lanes and in the release smoke, which gates
publishing.
Other install paths found broken while checking every channel:
- Homebrew: the formula was still at 0.10.3 and installed the dynamic Linux
build, which needs glibc 2.38 (Ubuntu 22.04, Debian 12 and RHEL 9 lack it).
It now installs 0.12.0 with the static -portable Linux assets and tracks
the newest release in the version contract like the other surfaces.
- Go: `go install .../pkg/go/cmd/codebase-memory-mcp@vX.Y.Z` never worked
(no pkg/go/v* tags, so Go fetched the whole repository, over its 500 MB
module limit), and @latest built main's wrapper, whose version constant
lags each release. The release now also tags pkg/go/vX.Y.Z, and the
wrapper installs its own module version (the constant only for local and
untagged builds).
- npm: a stable release also moves the `next` dist-tag to itself; `next`
still pointed to 0.9.1-rc.1.
- Docs: winget and Chocolatey were listed as install channels but are not
published; they are removed until they are. Windows arm64 ships and is
now listed.
Fixes#2657.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The badge tracked dry-run.yml, which runs only when dispatched; its last
runs on main were cancelled or failed during the v0.12.0 preparation, so it
read "failing" although main passed the full matrix in the v0.12.0 release
run. release.yml runs the same full matrix on main for every release.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A config key linked to every code symbol whose name contained it. In
YAML, JSON and HCL files that guess was retired in WP0.9; TOML and INI
files kept it, and in 17 infrastructure repositories every one of their
72 key_symbol edges was a coincidence: `.gitleaks.toml useDefault` linked
to TestConfig_UseDefault and three other symbols, a toml-merge tool's test
data `config_path` to 46 `*ConfigPath` constants all over deckhouse,
`grafana.ini admin_user` to user-cluster constants, a Rust Cargo.toml
dependency to other services' go.mod requirements.
A .toml, .ini, .cfg or .conf key now links like an infrastructure key:
by exact name (camelCase and snake_case still normalize to one key) and
only to code inside the config file's component, its directory or the
one around a config/, conf/, settings/ or etc/ directory. .env and
.properties keys keep substring matching; the tests that covered it on a
TOML key now use a .properties key.
Measured on the 17 repositories: key_symbol 95 -> 23, the 23 being the
infrastructure edges WP0.9 reviewed, the 72 removed all TOML/INI and all
coincidences, no other edge or node changed; deterministic over three
runs; the incremental re-index equals a full index for INI, TOML, code
and tfvars edits.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An external Chart was <project>.__helm_dep__.<name>: keyed by its name,
inside one project. Two repositories naming the same published chart held
two unrelated nodes, and a project naming two charts of one name from two
publishers (kubermatic's cert-manager from charts.jetstack.io and from its
own mirror) would have merged them into one.
An external Chart whose repository URL is stated is now
__chart__<repository>/<name>, the same QN in every project, the URL in a
`repository` property: scheme and host lower-cased, a trailing slash and
`.git` dropped. A Chart.yaml dependency, a kustomize helmCharts entry, a
Flux HelmRelease chartRef and a Terraform helm_release of an oci:// chart
that name one published chart meet in one node. A chart reference (a Flux
OCIRepository url, `chart = "oci://..."`) is cut to its repository; a
repository URL never is, so repository oci://public.ecr.aws/karpenter
with chart karpenter stays oci://public.ecr.aws/karpenter/karpenter. A
repository that is no URL (an @alias, no scheme, file://, credentials in
the authority, an unexpanded ${...}) or that the pass does not read (a
Flux HelmRepository source, a Terraform `repository =`, a shell
`repo/name`) keeps the per-project node: no identity is claimed for what
was not stated. A query or fragment (`?token=...`) takes the per-project
node too, so a token never becomes part of a QN or a property.
One predicate now tells an external chart from a chart of the repository
where Terraform, shell and the Go installer index the chart directories:
an external chart a Chart.yaml depends on carries that file's path, and
must never stand in for the chart the directory declares.
Measured on 17 infrastructure repositories against an independent
reading: 141 of 141 edges into external charts reach the identity their
stated repository gives (114 by URL, 27 per project). The only
differences to the previous index are the 79 external Chart nodes that
moved to their URL identity, each with its repository property, and the
114 edges into them, re-pointed with type, source and properties
unchanged. Every chart gate (Helm includes, helmCharts, Terraform, the Go
installer, shell, CI, GitOps deploys) reads the same as before; three
runs are identical; the incremental re-index equals a full index for a
shared chart's owning file edited and deleted, a URL respelled, a
dependency moved to another publisher and to an alias, a chartRef URL
edit and a new Terraform oci:// release. The index rebuilds once on
upgrade with the other version 9 changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
kubespray applies its cluster add-ons with its own `kube` module: a
task runs kubectl apply on a manifest the play rendered from a template
a few tasks earlier. Both tasks were Functions and the template a File,
with nothing between them: no query reached the manifest a task applies.
A kube (kubespray's library module) or k8s (kubernetes.core /
community.kubernetes) task now DEPLOYS {via: ansible, module, line} the
repository file its play rendered the applied manifest from: a template
task's src (the role's templates/, else the task file's directory) or a
copy task's (its files/; never a remote_src copy, which reads the host).
The applied filename is traced through the play: a renderer whose dest
is the very string; {{ item.item.K }} over "{{ R.results }}" -> each
item of the renderer registering R; {{ item.K }} -> each of the task's
own items through the renderer of that dest; a literal filename a looped
renderer renders to; a k8s template: argument. A loop list is a literal,
a set_fact of the file, or a list of the role's defaults / vars main
file. Templates are found before copies to the same dest. remote_src is
read the way Ansible's boolean() reads it (y / yes / on / 1 / true / t,
any case): `remote_src: true` is a YAML boolean, not a string, and a
string-only read had kept every remote copy as a renderer.
The lists in a role's defaults / vars reach past the task file, so a
change to one takes the full rebuild (closure_ansible_decline); without
it, a list that names another template kept the edge to the old one. A
file holding Ansible tasks that the repair would re-walk as a dependent
-- a template it applies was deleted -- also takes the full rebuild
(closure_ansible_dependent_decline): the repair walks it without its
role's defaults, handlers and other task files, and drew none of its
list-fed edges.
Measured on 17 infrastructure repositories: 164 of 164 applied files
(52 of kubespray's 62 kube / k8s tasks, 158 templates) against an
independent reading, all in kubespray; nothing in the other 16; the rest
of the graph unchanged (+164 DEPLOYS only); deterministic over three
runs; the incremental re-index equals a full index for a template body
edit, a deleted template (with and without a same-named fallback beside
the task file), a task list edit, a copied manifest edit, a defaults
list edit and a deleted vars file. Of the 10 tasks left out, 4 act on
an object by name, one builds its definition with a template lookup,
one applies a remote_src copy, and 4 name their file through a filter, a
variable or two joined results lists.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
kubespray's 1,425 tasks and 65 handlers were text inside YAML files: a
search for "Containerd | restart containerd" found no node, and nothing
said which task triggers which handler.
Each task and handler of an Ansible file is now a Function, named by its
`name:` (else its module), {source: ansible, kind, module}, which its file
DEFINES -- a playbook's pre_tasks / tasks / post_tasks / handlers, a role's
tasks/ and handlers/ files, and the files those include; a block's block /
rescue / always tasks, never the block itself. A notify CALLS {via:
notify, line} every handler of that name or listen topic in the task's own
role (the directory before its tasks/, so nested roles stay apart), else
in its file.
A top-level YAML list is not Ansible by itself: JSON patches, image maps
and service lists are lists too, and would have made some 3,000 false
tasks in five other repositories. A file counts when it is a playbook, a
role's tasks or handlers file, or included by one (to a fixpoint).
Quoted names have their YAML escapes decoded. A notify joins two files,
so a change to Ansible content takes the full rebuild
(closure_ansible_decline); without it, renaming a listen topic or adding
a notifying task left the incremental graph different from a full index.
Measured on 17 infrastructure repositories: 1,490 of 1,490 tasks and
handlers (file, line, kind, name, module) and 276 of 276 notify edges
against an independent reading, all in kubespray; nothing in the other
16; the P7.1 include edges unchanged (211 of 211); deterministic over
three runs; the incremental re-index equals a full index for handler and
task file edits, a deleted handler file and a playbook edit. The only
other change: as every function joins the similarity corpus, four
SEMANTICALLY_RELATED scores in kubespray's Python module moved and one
pair was added.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Cluster API deploys its manager as `capi-controller-manager` in
`capi-system`: config/default/kustomization.yaml puts `namePrefix: capi-`
and `namespace: capi-system` on the `controller-manager` of
config/manager/manager.yaml. The graph only had the declared name, so the
name the cluster runs -- the one docs, scripts and tests quote -- matched
no node.
A kustomization with a namePrefix or nameSuffix now gives every object of
its build a variant: a Resource named with the deployed name, in the
namespace the build gives it, `ecosystem: kustomize`, which the
kustomization's File DEFINES and which OVERRIDEs {via: namePrefix |
nameSuffix, kustomization, line} the object. Prefixes of nested
kustomizations compose outermost first, suffixes innermost first, and the
outermost namespace wins. CustomResourceDefinitions, Namespaces and
apiregistration APIServices keep their names, as kustomize's own
transformers skip them. Variants are minted after the Kubernetes joins:
inside a build, references use the declared names.
Measured on 17 infrastructure repositories: 56 of 56 renamed objects in
the 6 renaming kustomizations against an independent reading, with
deployed name, namespace and base object; nothing else changed (56 nodes,
112 edges added, no property of an existing node); deterministic over
three runs; the incremental re-index equals a full index when a prefix
changes, goes, nests, or a renamed object is renamed or deleted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A `kubernetes_*` resource in a .tf file -- eks-blueprints' network
policies and storage classes, k8s.io's metrics-server stack -- declares
a cluster object, but the graph only had the Terraform block, named by
its Terraform address: no node carried the object's kind and name.
A kubernetes_<type> resource (also _vN) whose metadata name is a literal
is now a Resource of the same identity scheme as a manifest's object:
kind from the resource type, or a kubernetes_manifest's own apiVersion
and kind; name and namespace from the object's own metadata, never from
spec; `ecosystem: terraform` on its properties. The .tf file DEFINES it
and the block DEPLOYS it {via: terraform, line}. The objects are minted
before the Kubernetes joins, so a manifest of the same bundle references
them as it does a manifest's object (an Ingress maps to the Service). A
computed name, and a type that acts on an object declared elsewhere
(kubernetes_annotations, kubernetes_labels, ...), make none. Labels are
not read, so a Service selector does not reach a Terraform workload.
Measured on 17 infrastructure repositories: 21 of 21 literal-named
objects against an independent reading, with group, kind, name and
namespace; nothing else changed (21 nodes, 42 edges added, no property
of an existing node); deterministic over three runs; the incremental
re-index equals a full index when an object is renamed, gains a
namespace, or its file is added or deleted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A .tfvars file in a directory without Terraform -- kubespray's
sample-inventory/cluster.tfvars, eks-blueprints' spokes/workspaces/*.tfvars --
is applied to a module further up with a -var-file the repository does
not state, and linked nothing.
When the file's own directory declares none of its names, it now sets
the variables of the nearest ancestor module that does: each assignment
to a declared variable OVERRIDEs {via: tfvars, inferred: true, line} the
`variable` block, from the file's Variable. A file whose own directory
declares one of its names keeps the exact edge, without `inferred`.
Measured on 17 infrastructure repositories: 238 of 238 assignments --
124 to their own directory's module, 114 inferred to an ancestor's --
against an independent reading, with the inferred flag; nothing else
changed; deterministic over three runs; the incremental re-index equals
a full index when an ancestor-linked file changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Ansible's composition produced no structure: kubespray's 40 playbooks,
roles and task files, tied together by 217 import_playbook /
include_tasks / import_tasks / include_role / import_role / roles:
directives, were unconnected YAML files.
A playbook or task file -- a YAML file whose top level is a sequence;
manifests, values, CI and compose files are mappings -- now IMPORTS
{local_name: <directive>, via: ansible, line} what each directive names:
import_playbook a playbook from its directory; include_tasks /
import_tasks / include a task file from its directory, else from its
role's tasks/; roles: items and include_role / import_role a role's
Folder, roles/<name> at or above its directory (a role by "./" / "../"
path from the directory). The ansible.builtin. prefix is understood; a
templated target links nothing.
Measured on 17 infrastructure repositories: 211 of 211 composition pairs
of kubespray against an independent reading, nothing else in any
corpus; deterministic over three runs; the incremental re-index equals
a full index in 4 scenarios (an included file edited or deleted, an
include pointed elsewhere, a playbook edited).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Terragrunt unit (terragrunt.hcl) composes itself from other files --
`include` blocks, read_terragrunt_config calls, dependencies on other
units -- and the graph held none of it: its blocks only met same-named
blocks elsewhere through the generic name matcher.
Every unit is now evaluated in its own context, the way Terragrunt
evaluates it (also the files it includes): an include IMPORTS
{local_name, via: terragrunt, line} the file its path names; a
read_terragrunt_config IMPORTS {via: terragrunt, read, line} the file it
names -- from the block holding the call or, for a call in an included
file, from the unit's include block that brings that file in, `read`
naming the file holding the call; a dependency DEPENDS_ON the other
unit's terragrunt.hcl; a local terraform source IMPORTS its Folder.
Paths may be string templates of find_in_parent_folders (searched from
the unit's parent directory upward), dirname and get_terragrunt_dir;
anything else is unknown and links nothing. A run holding any .hcl file
re-links every unit (a .hcl change takes the full rebuild since the
previous commit).
Measured on 17 infrastructure repositories: 30 of 30 composition pairs
of the 6 units of the Terragrunt repository against an independent
evaluator, nothing else; deterministic over three runs; the incremental
re-index equals a full index in 5 scenarios (a read file edited or
deleted, the included root file edited or reading another file, a unit
edited).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A .tfvars file's keys were Variable nodes connected to nothing: nothing
said which variable `region = "eu"` sets.
An assignment to a variable the module of the file's own directory
declares now OVERRIDEs {via: tfvars, line} that `variable` block, from
the file's Variable (child to base, as a values or patch overlay). A
.tfvars file elsewhere is applied by a -var-file the repository does not
state, and links nothing. A run touching a .tfvars file re-resolves like
one touching a .tf file.
Measured on 17 infrastructure repositories: 124 of 124 assignments of
the 19 files that set variables of their own directory, nothing else;
deterministic over three runs; the incremental re-index equals a full
index in 4 scenarios (a value edited, the variables file edited, a
.tfvars file deleted).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A module call with a local source -- `module "net" { source =
"./modules/net" }` -- linked nothing: 379 such calls on the 7 Terraform
repositories of 17 infrastructure corpora, and every `module.net.id`
reached at most the calling block, never the output it reads.
Such a call now IMPORTS {local_name, via: terraform, line} the Folder of
its source directory (the repository root: the Project), when that
directory lies inside the repository and holds .tf / .tofu files; a
registry, git or http source, or a path not starting with ./ or ../,
makes none. `module.net.id` -- also `module.net[0].id`, `module.net.0.id`
and `module.net[*].id` -- USAGEs {ref: output} the `output "id"` of that
directory, beside the USAGE of `module.net`.
Output reads cross directories, so a run touching any .tf / .tofu file
now resolves every module directory of the repository: an unchanged
caller keeps its reads when the callee file is re-parsed. A full index
takes as long as before; an incremental re-index touching a .tf file
pays for parsing them all (+0.24 s on a 2,000-file repository).
Measured: 349 of the 379 calls name a directory that exists in the
repository and IMPORT it, nothing else (the 30 others name paths a test
harness creates or that are absent); 644 of 644 output reads (holding
block to output) against an independent reading; the reference pairs
of the module resolver stay exact; only edges are added; deterministic
over three runs; the incremental re-index equals a full index in 6
scenarios (an output's value edited in the callee, its outputs file
deleted, a source moved, a widely called module edited, the root
module's caller edited).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Terraform references were resolved by the generic name matcher: by their
last name segment, across the whole repository. `format()` CALLS a
variable named format of another module, `local.x` meets a resource named
local, `.id` an output id anywhere. Measured on the 7 Terraform
repositories of 17 infrastructure corpora: 12,437 edges, recall 0.63 and
precision 0.39 against an independent reading of the references, and not
one of them left the block holding its reference.
A reference -- var.X, local.X, module.X, data.T.N, T.N -- now USAGEs
{ref, via: terraform, line}, from the top-level block holding it, the
block declaring it in the same module: the .tf / .tofu files of its
directory (a locals entry: its locals block); an entry of the block's
depends_on list DEPENDS_ON the object instead. The generic matcher makes
no edge for a .tf / .tofu file any more; a name only mentioned in string
text is no reference, nor is a block's reference to itself. A changed
file re-resolves its whole directory, so an unchanged file's reference
meets the declaration a change adds and loses the one it removes.
A bare block label (`variable region {}`, valid HCL) now names its block
like a quoted one; such blocks used to share the keyword's name and QN.
Measured: 7,771 of 7,771 file-level and 13,397 of 13,397 block-level
reference pairs (edge type included) and nothing else; the 10 other
corpora are unchanged; deterministic over three runs; the incremental
re-index equals a full index in 7 scenarios (rename a variable, a module,
a locals entry; delete the variables file; add a file; add a declaration
an unchanged file references; quote a bare label); index time unchanged
(within 1 % on the four largest Terraform corpora).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Kubernetes object's metadata.labels were not in the graph: neither a
search for an object by label nor a kustomize patch whose target selects
by label could find anything.
A Resource now keeps its labels as `labels: "k=v,k2=v2"` (sorted; a
templated label, or one a selector cannot name, is left out), and a
patch target's labelSelector meets the objects of its build carrying its
terms -- `k=v` / `k==v`, `k!=v`, `k`, `!k`; a set-based term names no key
and meets nothing. A target with only a selector selects among every
kind; an annotationSelector is not evaluated and names nothing.
Measured on 17 infrastructure repositories: 1,956 Resources gain labels
and nothing else; an independent reading of metadata.labels agrees on
every one of the 1,804 objects both read (152 more are Go-templated,
which only the graph reads, through its skeleton). The 4 label-selector
patch pairs now resolve: kustomize patches 130 of 130.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A kustomization's `helmCharts` inflate Helm charts into its build -- the
fourth way a repository puts a chart on a cluster, beside a Flux
HelmRelease, a Terraform helm_release and a helm command; the graph read
none of them.
Each entry now DEPLOYS from the kustomization -- its Kustomization
Resource, else its File: the unit a Flux or Argo deploy of it reaches -- to
the shared external Chart of that name, with {via: helmcharts, external,
chart, url, version, release, namespace, values, line}. A chart name with
a path names nothing; the values file inside the repository is an IMPORTS
(local_name: valuesFile) of the kustomization.
Measured on 17 infrastructure repositories: 20 edges, every one in an
independent reading of the kustomizations (20 entries, cluster-api 10,
k8s.io 10); no other node or edge changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A kustomize overlay changes its base through patches; the graph read none of
the v5 `patches:` entries (path / patch / target mappings), so a patch file
showed up as a second copy of the object it patches, with no edge to it.
Each patch of a kustomization now OVERRIDEs, child -> base, the objects of
its build it changes, with {via: patch, kustomization, patch, line}:
- a patch file's documents: the object of the same group, kind and name
(and namespace, when the document declares one);
- an inline patch (from the kustomization's File) or a JSON6902 operation
file (from its own File): what its `target` selects -- kind, group,
version, name, namespace; a label or annotation selector is not
evaluated and names nothing;
- in the build of the kustomization stating it; a component's in the build
of each kustomization including it.
A patch file is an IMPORTS (local_name: patches) of its kustomization and
never a member of its build: its documents select nothing, are selected by
nothing.
Resources now keep the version of their apiVersion (`version`), which the
identity leaves out and a target may name.
Each build indexes its own objects by name once: a lookup in the
repository-wide name index would visit every copy of a name for every
patch. The complexity fixture gains patch files (one of four documents), an
inline and a kind-only patch.
Measured on 17 infrastructure repositories: 126 edges, every one in an
independent reading of the 173 patches (122 resolvable in the
repositories); the 4 pairs without an edge use a label selector.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Shell scripts, Make recipes and CI steps download tools and call services
with curl and wget; the graph had no edge for any of them, so neither what
a repository fetches nor the scripts that call its own services were
visible.
Such a command now HTTP_CALLS the Route of each URL it states, with
{via: curl | wget, method, url_path, host, line}:
- the method from the flags: -X / --request / --method, a body (-d, -F,
--data*, --json, --form*, wget --post-data / --post-file) POST, -T PUT,
-I HEAD, -G back to GET;
- the URL from constant words (VER=1.2.3 ... v${VER}), unknown parts `{}`
as in a Route path -- also in an unquoted word tree-sitter-bash splits
around its expansions; a URL whose host is not literal, or the value of a
proxy / referer option, is no request;
- never the user info, the query or the fragment: a token in a URL does not
reach the graph;
- the Route: path-only (__route__POST__/v1/items) when the host is a
Kubernetes Service of the repository in a service DNS form, where that
service's handler meets it as a code client's request does; any other
host -- a download, localhost, a compose service -- keeps it
(__route__GET__https://dl.k8s.io/release/{}/bin/linux/amd64/kubectl,
props {method, host, external}), and route_nodes makes no path-only twin
of it.
Incremental indexing now drops a Route without a file that no edge touches
any more -- its last caller's file changed or went. A full index never
holds one; before, the Route of a deleted code client stayed behind too.
Measured on 17 infrastructure repositories: 92 edges, every one correct by
hand; an independent reading of the scripts and Makefiles finds 82
requests, 81 of them in the graph (the last is lost in a file
tree-sitter-bash cannot parse past `[ "$v" \< "1.33" ]`). No property of
any other node or edge changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A workload's environment names the services it talks to
(PRODUCT_CATALOG_SERVICE_ADDR: productcatalogservice:3550, a URL, a
<svc>.<ns>.svc name); the graph had no edge for it, so the service topology
of an application was invisible.
A Kubernetes workload or a compose service now CONNECTS_TO the Service its
containers' environment addresses, with {via: env, key, container} -- never
the value:
- the address: the host of a URL or of host:port, a <svc>.<ns>.svc name in
the value, or the whole value when the key names an address (*_ADDR,
*_ADDRESS, *_HOST, *_URL, *_ENDPOINT, *_URI) and it is a bare DNS name; a
templated value, localhost or an IP addresses nothing, and only service DNS
forms count (svc, svc.ns, svc.ns.svc...), never another domain's host
(cbm_env_address, shared by both sides);
- Kubernetes: a reference resolved like every other one -- a Service of the
same bundle, in the named namespace or the object's own;
- compose: a service of the same file; a variable without a value takes the
project's .env, whose values compose expands against the lines before them
(CART_ADDR=cart:${CART_PORT}) -- the dotenv reader does that now too.
Consumers: trace_path's topology mode, the CLI's edge list and the README
(which also gains DEPLOYS and the env join of the previous commit).
Measured on 17 infrastructure repositories: 129 edges, every one correct by
hand; an independent reading of the values finds 133 pairs repo-wide, the 5
without an edge belong to another bundle or namespace.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolves the four README conflicts against main 49424722: the README keeps
this branch's wording and style, and takes the content main added since
4d9592b5 in that style (list items "X: ..." instead of "X — ...", mid-
sentence dashes as semicolons):
- feature list and indexing pipeline: main's infrastructure-as-code
bullets (container images, compose, Helm, Terraform, Service -> code);
- edge types: main's SPAWNS detail plus INFRA_MAPS, USAGE ref:image,
DEPLOYS, DEPENDS_ON and CONFIGURES;
- trace_path: this branch's text (depth 1 to 15, tests excluded) plus
main's topology / deploy / run modes.
Checked: the supported-agent README contract (45 surfaces, every agent),
the language-count contract (162 grammars), all 22 local links and images.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Doc-section -> code candidates (doc_link_candidates, #2574/#2610) are the
documentation-in-the-graph feature of this release. They now run by
default for full indexes and not for moderate or fast ones; an explicit
`doc_candidates` argument, or a `doc_candidates` setting the user set,
overrides the default either way in every mode.
- pipeline: the switch is unset/on/off; unset means on only when the
effective mode is full.
- MCP: forwards only an explicit choice (argument, then a user-set config
value) to daemon jobs and workers; daemon auto-index and watcher/UI jobs
now carry a user-set value too (their workers have no config, so
`doc_candidates false` was ignored there).
- fast + doc_candidates on runs the semantic pass's corpus phases and the
doc phase only: candidates, and still no SIMILAR_TO or
SEMANTICALLY_RELATED edges.
- config: `doc_candidates` joins the config key table (default `auto`);
`config set doc_candidates ...` was refused as an unknown key before.
README, docs/CONFIGURATION.md and the MCP schema describe the default.
Proof (macOS arm64, django fb11376):
full 11,304 rows · moderate 0 · fast 0 · full+off 0 · moderate+on 1 ·
fast+on 1 (moderate/fast discovery skips docs/) -- nodes and edges are
identical between on and off in every mode; fast+on adds 0 semantic
edges.
- New tests (RED on the old code): pipeline
pipeline_doc_candidates_default_follows_mode,
pipeline_doc_candidates_explicit_wins_over_mode,
pipeline_doc_candidates_fast_on_legacy_incremental; mcp
tool_index_repository_doc_candidates_follow_mode,
tool_index_repository_doc_candidates_forwarded_only_when_explicit;
daemon_application
daemon_application_programmatic_index_forwards_set_doc_candidates.
- mcp 374 passed (4 Windows-only skips), daemon_application 73 passed,
pipeline/store_arch/cli/runtime_settings/index_policy/pipeline_frozen
green on the release tree.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The doc<->code candidate scoring (#2574, doc_section_worker in
semantic_edges) scores every documentation section against the functions
that share a key term with it, key terms being those held by at most
funcs/20 functions. Its cost therefore grows with sections x functions.
Measured on 11 corpora (10-09 main vs 4d9592b5, semantic_edges): under a
second up to ~100k functions, +12.6 s on java (325k), +14.6 s on csharp
(451k), +174 s on the Linux kernel (784k functions, 28k sections), almost
all of the kernel's regression against v0.11.0.
A new boolean setting, doc_candidates (default true), and an optional
index_repository argument of the same name switch the scoring off: no
candidate rows are stored and nodes and edges are unchanged. The argument
wins over the setting. The index handler resolves the value once and
passes it to a daemon job or supervised worker as an argument, the same way
as the resource policy, so every route sees one decision; the session
auto-index reads the setting. With the switch off, a delta-route reindex
replaces the candidate rows it would otherwise keep, so turning it off
leaves no stale candidates. The default output is unchanged.
Tests (each fails on its assertion when the switch has no effect):
- pipeline_doc_candidates_off_stores_none_and_keeps_graph: no rows, same
node and edge counts as the default run, which has rows;
- pipeline_doc_candidates_off_clears_rows_on_delta: a closure-repair
reindex with the switch off removes the earlier rows;
- tool_index_repository_doc_candidates_switch: argument false -> none;
setting default -> rows; setting false -> none; argument true over
setting false -> rows.
Proof, release-built binaries on macOS arm64 (M5 Pro 18c/48 GB):
- django, main 4d9592b5 vs this change with defaults: nodes (63,441),
edges (307,584) and doc_link_candidates rows (11,304) hash-identical;
- django with doc_candidates false: nodes and edges hash-identical, no
candidate rows;
- Linux kernel with doc_candidates false: semantic_edges 17.9 s (191.9 s on
main, 17.6 s before #2574), whole index 210.9 s (v0.11.0: ~234 s), no
candidate rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings in #2596 (MSan: hidden uninitialized reads) and #2068 (CodeAtlas
web).
README: main's CBM Atlas line and this branch's infrastructure-as-code
line are both kept.
tools/list metadata: main and this branch each stayed under the 20 KiB
discovery budget, but together they reached 20,809 bytes. The branch's
wording is shorter now (search_graph kind/namespace, one hint per trace
mode: 136 bytes less), and the reviewed budget is 21 KiB so the
topology/deploy/run modes keep their usage hints.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The note on daemon-owned environment now carries the CLI rule from main.
The C test badge and counts come from a run of the suites on main:
9,388 tests in 189 suites.
Signed-off-by: Bernhard Jackiewicz <bj@techport.io>
The UI section now describes the Lens that is on main: the six
workspaces, the Architecture views, Galaxy and the local chat. Three GIFs
and six stills of this repository at 4d9592b5 replace the old screenshot
and the large WebP recordings. Statements that no longer match the UI
are corrected or removed, and the test counts come from main.
Signed-off-by: Bernhard Jackiewicz <bj@techport.io>
trace_path mode=run walks what a Service or workload runs: traffic to
the workload (INFRA_MAPS), the Image its container runs (USAGE), the
Dockerfile that builds it and what that builds on (DEPENDS_ON), the
program its ENTRYPOINT starts (SPAWNS) and the environment each
container is given (CONFIGURES) -- Service -> Deployment -> Image ->
Dockerfile -> main in one call.
The README lists the Image label, the container edges (a Dockerfile's
bases and entrypoint, image references, the build vias, compose
services, container environment) and the new mode.
The complexity guard gets the container layer: a fixture with
multi-stage Dockerfiles, an ENTRYPOINT script and a compose project
building two components of one repository, its nodes and edges gated
linear per copy, and the Image edges the component split examines
gated linear too -- every copy's builds join one repository Image by
design, so that join must stay linear.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Rename the reading workspace to CBM Lens Beta, distinguish Beta with an amber badge, and keep build diagnostics out of the header. Put architecture history beside the active context and record meaningful exploration steps.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The DEPLOYS and SPAWNS edges of the infrastructure layer had no trace
preset: "which CI job, through which script, installs this chart?" took
hand-picked edge_types.
- trace_path mode=deploy walks DEPLOYS, SPAWNS and CALLS: outbound from
a CI job, script or Make rule to the scripts, make targets and programs
it starts and the manifests, kustomizations and charts it deploys;
inbound from a chart back to the job.
- mode=topology also follows DEPLOYS, so a Flux Kustomization reaches the
kustomization its path applies and a HelmRelease its chart.
- README: the DEPLOYS edge type, make targets under SPAWNS, the trace
modes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
"Which CI jobs run hack/ci/run-e2e-tests.sh, and what does that script
start?" had no answer: a command running a script of the repository drew
no edge (SPAWNS reached Process nodes only, and repository scripts were
left out), and CI configurations were not read at all.
- A shell script, Make recipe or CI command that runs a script of the
repository -- by its path (./hack/x.sh, "$ROOT/hack/x.sh") or through a
shell (bash -e hack/x.sh) -- SPAWNS that script: its Module (the source
of the script's own edges, so CI -> script -> deploy chains connect),
else its File. The path must resolve to exactly one file; a program that
is no repository file keeps its Process edge.
- A new pass_ci.c reads GitHub Actions workflows and composite actions
(run: steps with their working directory, shell, env literals,
${{ github.workspace }} and ${{ github.action_path }}; no pwsh step, no
step of a Windows job) and Prow job configs (container command + args,
exec'd; bash -c TEXT as shell; a decorated job starts in the checkout).
The shell layer evaluates them like a script; their DEPLOYS and SPAWNS
edges come from the workflow / job file and carry "ci" and "job".
- env A="$X/y" PROGRAM: an assignment with an unknown value is no longer
taken for the program (it made a Process named after the value's last
segment and hid the program).
Measured on 17 repositories against an independent reading (libyaml for
the CI files, a separate shell reader for the commands): 292 of 292
script runs linked, 292 of 292 correct -- from Prow jobs (115 containers),
GitHub Actions steps, shell scripts and Make recipes, across 10
repositories (kubermatic, gardener, kube-hetzner, cluster-api, deckhouse,
k8s.io, ...); CI Process and deploy edges 4 of 4. Only SPAWNS (+295),
DEPLOYS (+1), Process (+3) and Chart (+1) changed; three runs build
identical graphs; index time unchanged (kubermatic 2.95 -> 2.93 s,
gardener 5.25 -> 5.25 s, kube-hetzner 1.13 -> 1.14 s, medians); editing or deleting scripts, workflows,
Prow jobs and Makefiles leaves the graph a fresh index builds (two
scenarios differ in SEMANTICALLY_RELATED edges and Make pattern-rule
nodes exactly as the previous build does).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The README's Graph Data Model named USES_TYPE, MEMBER_OF and
CONTAINS_PACKAGE, which nothing emits, and left out 14 labels and 18
edge types that real indexes hold -- among them the infrastructure ones
(Resource is listed, but not Chart, Process, EnvVar, INFRA_MAPS or
DEPENDS_ON) and Variable, Section, Field, Struct, OVERRIDE, INHERITS,
TESTS_FILE and the CROSS_* family. The skill text `install` writes for
agents carried the same CONTAINS_PACKAGE and no infra edges, so an agent
following it queried an edge that never exists and did not learn the
ones that do.
Both lists now name exactly what is produced: checked against the
labels and edge types of 17 infrastructure repositories and 86 local
indexes, plus a quoted producer in the sources for the types those
indexes do not exercise (Trait, Constant, DATA_FLOWS, CROSS_*). Against
the same data the old README had 29 mismatches, the new one none.
The edge summary also describes INFRA_MAPS (Service -> the workloads its
whole selector selects, one namespace) and DEPENDS_ON (Helm chart ->
declared charts with their repository; go.mod / requirements.txt ->
external Package), and the infrastructure bullet names Helm charts and
Terraform / OpenTofu files.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolves the index_status tool description: main's watch state (#2167) and this branch's doc_links both stay.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A reference written in a doc comment is extracted with the definition it
documents (internal/cbm/doclink.c; the C# scanner doclink_cs.c) and
resolved per file in the pipeline (src/pipeline/doc_links.c,
doc_links_cs.c, doc_links_msbuild.c). A reference that names a definition
of the repository with certainty becomes a MENTIONS edge from the
documented definition (the File node for a file-level comment) with the
properties via, syntax, tier, line and count. Any other reference becomes
a row of the new doc_link_unresolved table with its reason: missing,
ambiguous, external, test_only_target, graph_gap, unparseable or
below_bar_tier. Nothing is guessed. index_status reports the layer in a
doc_links block.
C# binds a cref by the compiler's lookup rule: the first scope with the
name decides, a cref never binds an inherited member, System is outside
the repository. Assemblies come from project files, and MSBuild global
usings (<Using>, ImplicitUsings, Directory.Build.props/.targets, imports,
conditions) are evaluated from the project files' content as data the
indexer has already read; the layer opens no file. Two declarations of a
name in different assemblies make the reference ambiguous.
Each file's scope is stored line-free with its LSP surface ("dl"), so an
incremental run re-resolves the files a change can reach and falls back
to a full run for a change that can re-route references anywhere (a
project file, a global using, a deleted file).
The scanner and the resolver treat the repository as untrusted input:
bounded nesting, no input-driven recursion, work that grows linearly
(held by test seams, never by a clock), no silent limits, and a damaged
or foreign stored scope fails closed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The test-impact engine selects tests by walking the graph, so it can only
be as right as the graph. Three gaps made it wrong:
One identity for a definition's mutually exclusive variants
- A definition written once per #if/#else branch, per #[cfg], per
platform file (foo_linux.c / foo_win.c, platform directories) or per Go
build tag is ONE node. The graph buffer keeps every variant's span in a
`variants` node property (definition labels only), so a change in any
variant maps to the definition.
- Extraction now sees definitions inside conditional blocks where it
skipped them: C++/C#/ObjC class members under #if (one cursor walk,
conditional block types per language in lang_specs), Elixir defs under
if/unless, Fortran .F90/.F95/.F03/.F08, Rust #[cfg] twins (no more
`name#cfg(...)` QNs that lost every call), C platform files and
directories (non-static file-scope functions).
- Haskell: the vendored scanner reads only an #if group's first branch.
The grammar stays upstream; every further branch is parsed as its own
same-length view (cpp_branch_views.c) and adds what the raw parse
missed. haskell/bytestring: no new partial parses, +27 functions,
IMPORTS 322 -> 329, 131 definitions with variant spans.
- Incremental closure repair re-parses every file of such a definition
when one of them changes (variant partners through DEFINES), so a
repair equals a full index.
No cross-language name guesses
- A name-only resolution (unique_name, suffix_match, qualified_suffix,
field_type_hint, and same_module across files that only share a path
stem) never binds a symbol another language defines.
- Bash imports are the files a script sources: every top-level command
was read as an import of its name (`set -e` imported "set", bound to a
TSX method in 137 scripts here).
- This repository: cross-language CALLS 5,916 -> 5,091; the remainder are
C/C++ pairs, builtins and same-stem module collisions (recorded).
SPAWNS edges (per language, like the HTTP client patterns)
- A call that starts a program (subprocess.run, exec.Command,
child_process.spawn, posix_spawn, system, Process.Start, Command::new,
... about 30 languages, through the file's own imports) gets a SPAWNS
edge to a Process node named after the program (literal first word, or
<dynamic>) instead of a CALLS guess. No target resolution. Both
resolvers ask at the same point (parity test).
The engine follows the graph
- Seeds map changed lines through every variant span; a test written
once per #if branch is one test; when a change reaches a non-test
`main`, the tests that spawn processes are reached too (the process
runs main, and main reaches everything).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The background watcher only watches the project an open MCP session is
rooted in, and drops the watch when the last session for that project
closes. A repository indexed with `cli index_repository` (or from a
session rooted elsewhere) is never watched, but README promised
"Auto-sync keeps it fresh after that" for every index_repository target
and the MCP instructions said "Indexes auto-refresh". Nothing in the
status output said which projects were actually watched, so a stale
CLI-indexed repo looked like a watcher bug.
What gets watched is unchanged. Instead:
- index_status gains a `watch` object: `watched`, plus `strategy`,
`poll_interval_ms` and `last_scan_at` when watched, or a `reason`
(not_session_project, cli_session, auto_watch_off, watcher_disabled,
not_registered, no_watcher) and a re-index hint when not. It does not
touch the #1561 `freshness` key.
- The watcher publishes per-project strategy, cadence and last completed
scan time through atomics and exposes them via
cbm_watcher_project_info(); the daemon answers index_status through a
session-scoped provider.
- README, docs/CONFIGURATION.md, docs/index.html, the index_status tool
description and the MCP instructions now state the real scope.
`watcher_enabled` already shows in `config list` since v0.11.0 (680bc72e).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Require --delete-indexes for automatic or noninteractive index removal.
Keep the separate terminal prompt with a default of retaining indexes,
and preserve dry-run and guarded activation ordering.
Refs #2200
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
README.md:132, README.md:633, README.md:761, and docs/CONFIGURATION.md:203
said one-shot `cli` commands "never start or connect to the coordination
daemon" / read "their own environment without starting the daemon". That
was already contradicted a few lines above it (README.md:126 lists
one-shot CLI commands among the processes that "share a crash-safe OS
admission barrier" with the daemon) and by the code:
main_local_cli_daemon_execute() (src/main.c) bootstraps a client
connection to the shared per-user coordination daemon via
main_client_bootstrap_with_upgrade(), spawning one (bootstrap.daemon_spawned)
when none is running, before dispatching the tool call.
Verified live: `cli --verbose list_projects` against an isolated
HOME/CBM_CACHE_DIR/CBM_RUNTIME_DIR printed "hint: this command started a
temporary CBM daemon", and its cbm-daemon.log showed
daemon.start -> daemon.runtime_stopping (reason=last_committed_client_disconnected)
-> daemon.lifetime_end, i.e. the daemon it spawned exited again once the
CLI command's own connection closed. bootstrap_production_spawn() passes
the calling process's `environ` straight to posix_spawn(), so a CLI
invocation that starts the daemon also seeds that daemon's captured
daemon-owned environment (CBM_DIAGNOSTICS, CBM_LOG_LEVEL, etc.) - it does
not read "its own environment" independently as the docs claimed.
Corrected all four passages to state the real behavior: CLI commands
connect to the shared daemon (starting one if none is running) for the
same admission barrier and per-project locks, hold a `cli_session` that
is never registered with the background watcher, and - if their own
connection is what started the daemon - that daemon exits again once the
command's connection closes and no other session is attached. No opt-out
env var or flag for an in-process/no-daemon CLI mode exists (grepped the
whole tree), so none is documented.
This is a pure documentation fix; no production behavior changed. Kept
clear of PR #2348 (fix/issue-2167), which separately rewrites the
watcher-scope paragraph at README.md:152/237/672 and
docs/CONFIGURATION.md:88 for issue #2167's auto-sync-scope correction -
none of the lines here overlap with that diff.
Refs #2183, #2167
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two follow-ups to the #2133 wildcard fix, both about what cross-repo
intelligence tells the caller.
1. "Never run" vs "ran, no links". An empty CROSS_* set used to mean
either one, and no output surface could tell them apart. Every run now
records a marker in the SOURCE store's store_meta
(key cross_repo_last_run:<project>, same per-project key pattern as
coverage_shadow_fp:<project>) once the previous CROSS_* generation has
been cleaned: last_run_at, outcome (complete|cancelled|failed),
targets, projects_scanned, total_cross_edges, skipped_projects.
index_status and the UI layout (next to linked_projects) now carry a
`cross_repo` object: {"status":"never_run"} or {"status":"ran", ...}.
The marker lives beside the edges it describes, so a reindex that
rebuilds the store drops both together. No new database files and no
new freshness key.
New store API cbm_store_meta_get/put. put seeds a missing store_meta
through the generation path, because creating the table without
db_uid/mutation_gen would make cbm_store_generation() report malformed
metadata.
2. ["*"] skips a pre-#768 store (since #2343) but only logged it. The
result now names each one, and the MCP response lists them:
skipped_projects: [{name, reason:"pre_768_schema",
hint:"reindex this project"}] (only emitted when non-empty). The
cross_repo.project_unusable log line is unchanged.
cr_project_exists() is split into cr_project_state() (usable / missing /
legacy) so that the enumerator can tell a legacy store apart from a
foreign .db file. New allocations go through mem_core.
Refs #2133
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A project indexed from a directory that is not a git repository was never
refreshed after its first index: init_baseline classified it is_git=false
and poll_project/check_changes returned early for it forever, so the graph
silently went stale. The comment in init_baseline even claimed such folders
were "polled as a plain directory", which was never true.
Add an opt-in config key, watch_non_git (default false, read once at daemon
startup like watcher_enabled). When on, a non-git root is polled on the
existing adaptive cadence (cbm_watcher_poll_interval_ms) by a tree
signature: the indexer's own cbm_discover walk (same skip lists, .gitignore
stack and .cbmignore), sorted by path and folded over (relative path, size,
mtime). A changed signature takes the existing reindex path with the same
commit-after-success discipline as the #937 dirty signature. The baseline
leaves the signature "unknown", so the first poll reindexes once
(at-least-once, covering edits made while the daemon was down).
Runaway guard: only files discovery would index enter the signature, so
cbm's own .codebase-memory output (built-in skip) and the cache directory
(pruned by absolute path) can never make a finished reindex look like a
new change; a scratch folder under an unrelated dirty repository is polled
on its own files and never inherits the ancestor's dirty state.
cbm_file_info_t gains mtime_ns, filled from the stat discovery already
performs, so the signature needs no second stat per file and stays
UTF-8-correct on Windows (discovery's wide stat).
Docs: README and docs/CONFIGURATION.md now say non-git roots are not
watched by default and document watch_non_git. The staleness signal in
index_status is left to #1561's freshness object. Refs #1948.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
Bernhard Jackiewicz