Review round 1 of #2677. Every case ends in an explicit unknown, never in a
hang, an OOM, a partial or a wrong image.
Bounds (helm_template.c):
- A per-render step and output budget (10,000,000 steps, 64 MiB): a range
over a huge int, a huge indent, defines that include each other in a fan
stop the render; every probe of the template is UNKNOWN, also one captured
before, and the trip is logged (helm.render.budget). Counted, never timed.
On the 20 infra corpora no render reaches it.
- go-yaml v2's alias limit: values whose decodes are mostly alias expansion
(a billion laughs) do not parse, as in Helm.
- --set: an index past strvals' MaxIndex (65536) or a key longer than the
parser holds is no value (the release is unknown), never a padded list or
a split key.
Reads (pass_k8s.c, pass_terraform.c): a values file, Chart.yaml, kustomize
valuesFile or Terraform file() named by path is read only when it is a
discovered file of the repository -- never through a symlink out of it --
with the size cap and tree-sitter padding of every other read. A chart file
that is there but not read makes the chart's values unknown, not empty.
Wrong images:
- Terraform number sets reach the chart as Terraform's exact decimal
(1.10 -> "1.1"), not %.17g.
- A release with a value source not read -- Flux valuesFrom, valuesFiles,
postRenderers; kustomize additionalValuesFiles, a valuesMerge other than
override, a valuesFile not read; Terraform dynamic "set" -- has no image of
its own.
- An aliased subchart renders with .Chart.Name its alias.
- An UNKNOWN if / with / range makes what its bodies assign UNKNOWN and
leaves the loop a break in them would; set / unset / merge poison the map
they change (merge: the maps within too). A change that may reach values
another part holds taints every later capture; one of a new value
(fromYaml, deepCopy, an empty dict, a part of one) does not.
Marks and secrets:
- The chart edge reads default_of as JSON (list paths kept), builds its
properties to size, and writes no edge whose mark it could not carry whole.
- Build arguments with a secret (by name or value) are never kept; build
edge properties and values-path marks are never cut.
Tests: one per bound and per wrong-image case, each RED on d21949ef by its
assertion; a Helm chart joins the Kubernetes complexity fixture.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Terraform references were resolved by the generic name matcher: by their
last name segment, across the whole repository. `format()` CALLS a
variable named format of another module, `local.x` meets a resource named
local, `.id` an output id anywhere. Measured on the 7 Terraform
repositories of 17 infrastructure corpora: 12,437 edges, recall 0.63 and
precision 0.39 against an independent reading of the references, and not
one of them left the block holding its reference.
A reference -- var.X, local.X, module.X, data.T.N, T.N -- now USAGEs
{ref, via: terraform, line}, from the top-level block holding it, the
block declaring it in the same module: the .tf / .tofu files of its
directory (a locals entry: its locals block); an entry of the block's
depends_on list DEPENDS_ON the object instead. The generic matcher makes
no edge for a .tf / .tofu file any more; a name only mentioned in string
text is no reference, nor is a block's reference to itself. A changed
file re-resolves its whole directory, so an unchanged file's reference
meets the declaration a change adds and loses the one it removes.
A bare block label (`variable region {}`, valid HCL) now names its block
like a quoted one; such blocks used to share the keyword's name and QN.
Measured: 7,771 of 7,771 file-level and 13,397 of 13,397 block-level
reference pairs (edge type included) and nothing else; the 10 other
corpora are unchanged; deterministic over three runs; the incremental
re-index equals a full index in 7 scenarios (rename a variable, a module,
a locals entry; delete the variables file; add a file; add a declaration
an unchanged file references; quote a bare label); index time unchanged
(within 1 % on the four largest Terraform corpora).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A kustomize overlay changes its base through patches; the graph read none of
the v5 `patches:` entries (path / patch / target mappings), so a patch file
showed up as a second copy of the object it patches, with no edge to it.
Each patch of a kustomization now OVERRIDEs, child -> base, the objects of
its build it changes, with {via: patch, kustomization, patch, line}:
- a patch file's documents: the object of the same group, kind and name
(and namespace, when the document declares one);
- an inline patch (from the kustomization's File) or a JSON6902 operation
file (from its own File): what its `target` selects -- kind, group,
version, name, namespace; a label or annotation selector is not
evaluated and names nothing;
- in the build of the kustomization stating it; a component's in the build
of each kustomization including it.
A patch file is an IMPORTS (local_name: patches) of its kustomization and
never a member of its build: its documents select nothing, are selected by
nothing.
Resources now keep the version of their apiVersion (`version`), which the
identity leaves out and a target may name.
Each build indexes its own objects by name once: a lookup in the
repository-wide name index would visit every copy of a name for every
patch. The complexity fixture gains patch files (one of four documents), an
inline and a kind-only patch.
Measured on 17 infrastructure repositories: 126 edges, every one in an
independent reading of the 173 patches (122 resolvable in the
repositories); the 4 pairs without an edge use a label selector.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Scorecard's PinnedDependencies check opened code-scanning alerts #82-#84
on main for the complexity corpus's Dockerfile fixtures (added by #2597),
and the CodeQL gate blocks on open alerts. The fixtures are indexer input
and are never built; pinning them costs nothing and keeps the gate clean.
Both FROM lines now name alpine:3.20 by its multi-arch index digest. The
`FROM base` stage reference stays: it names the pinned first stage.
complexity suite: 7 passed, fx_Dockerfile nodes 70 -> 139, edges 86 -> 173.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
trace_path mode=run walks what a Service or workload runs: traffic to
the workload (INFRA_MAPS), the Image its container runs (USAGE), the
Dockerfile that builds it and what that builds on (DEPENDS_ON), the
program its ENTRYPOINT starts (SPAWNS) and the environment each
container is given (CONFIGURES) -- Service -> Deployment -> Image ->
Dockerfile -> main in one call.
The README lists the Image label, the container edges (a Dockerfile's
bases and entrypoint, image references, the build vias, compose
services, container environment) and the new mode.
The complexity guard gets the container layer: a fixture with
multi-stage Dockerfiles, an ENTRYPOINT script and a compose project
building two components of one repository, its nodes and edges gated
linear per copy, and the Image edges the component split examines
gated linear too -- every copy's builds join one repository Image by
design, so that join must stay linear.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The complexity guard did not cover the shell, Make, CI, Terraform and
chartRef resolvers. Extending it found one quadratic lookup: a
helm_release's block node was looked up among every node of that name
and walked for the one in the same file, so modules sharing a release
name (the common `helm_release "this"`) made each lookup scan all of
them. tf_candidates went 12 -> 42 per doubling (3.50).
- pass_terraform: look the block up by the QN its definition got
(cbm_fqn_compute + cbm_gbuf_find_by_qn), keeping the file check.
6 -> 12 (2.00). Output identical on the three corpora with
helm_release; incremental equals full.
- Counters: g_tf_candidates (new), Make rule probes into
g_sh_candidates, chartRef in-edges into g_k8s_candidates.
- test_complexity: the fixture copier recurses into directories; new
fixtures Bash (composite action -> make -> scripts -> kubectl / helm),
HCL (helm_release of a module chart and of an external one), a Go
chart loader, and a Flux HelmRelease chartRef; sh_candidates and
tf_candidates are gated (non-vacuous, ratio), with fx_Bash and
fx_HCL nodes and edges.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The complexity guard (O(n^2) detection) had no infrastructure corpus, and
the infra resolvers looked candidates up by name across the whole
repository and then filtered them: an object reference by "Kind/name", a
selector by its rarest label pair, a custom resource by its CRD's
group/kind, a controller's kind by the Go type's name. Independent copies
share every name, so with k copies each lookup visited all k: the work
grew ~4x per doubling while the edges stayed ~2x.
Candidates are now looked up where they can match:
- K8s references, selectors and CRDs through indexes keyed by the bundle
(kustomize build root / chart / dir, and the deploy scopes for
references) and the name, type or label pair; a source visits only the
bundles it is in. The repository-wide fallback for a custom resource
without a CRD beside it stays (it stands for every copy by design);
- the operator pass through a (package dir, type) index: a controller's
kind is one lookup in the imported or own package; only an import no
module of the repository has still checks the same-named types. Group
declarations are deduplicated through a set, not a scan.
Child access in the K8s extractor goes through a tree cursor:
ts_node_named_child(n, i) walks from the first child on every call, so the
index loops over mapping pairs, sequence items and a file's documents were
quadratic in a wide node.
The guard gains fixtures (tests/fixtures/complexity/Kubernetes: a
kustomize bundle with workloads, Service, Ingress, PDB, NetworkPolicy,
HPA, RBAC, CRD + CR, cert-manager; tests/fixtures/complexity/Go: a
controller-runtime operator with +groupName types and an embedded CRD),
work counters g_k8s_candidates / g_op_candidates, and a gate on them and
on the fixtures' own node and edge growth; fixture dirs now also apply to
templated languages. An API group is one identity across a repository, so
the fixtures' CRDs use groups no copy's Go code declares -- the guard
measures accidental growth, not that by-design join. The originality
check covers the infra sources and scans them against yaml-language-server,
helm-ls, kustomize, Flux's kustomize package and controller-tools.
Before: k8s candidates 126 -> 504 (4.00x), operator 45 -> 288 (6.40x)
per doubling; after: 42 -> 84 and 6 -> 12 (2.00x), fixture edges 2.01x.
On 17 infrastructure repositories the graphs are identical to before;
three runs build identical graphs. Index wall time against origin/main,
median of three: deckhouse 9.58 -> 9.59 s, kubermatic 2.93 -> 2.93 s,
cloud-foundation-fabric 1.90 -> 1.90 s. A manifest of thousands of
documents indexes ~10 % faster (2.40 -> 2.13 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel