mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-04 10:48:20 +00:00
feat(infra): an external chart is keyed by the URL that publishes it
An external Chart was <project>.__helm_dep__.<name>: keyed by its name,
inside one project. Two repositories naming the same published chart held
two unrelated nodes, and a project naming two charts of one name from two
publishers (kubermatic's cert-manager from charts.jetstack.io and from its
own mirror) would have merged them into one.
An external Chart whose repository URL is stated is now
__chart__<repository>/<name>, the same QN in every project, the URL in a
`repository` property: scheme and host lower-cased, a trailing slash and
`.git` dropped. A Chart.yaml dependency, a kustomize helmCharts entry, a
Flux HelmRelease chartRef and a Terraform helm_release of an oci:// chart
that name one published chart meet in one node. A chart reference (a Flux
OCIRepository url, `chart = "oci://..."`) is cut to its repository; a
repository URL never is, so repository oci://public.ecr.aws/karpenter
with chart karpenter stays oci://public.ecr.aws/karpenter/karpenter. A
repository that is no URL (an @alias, no scheme, file://, credentials in
the authority, an unexpanded ${...}) or that the pass does not read (a
Flux HelmRepository source, a Terraform `repository =`, a shell
`repo/name`) keeps the per-project node: no identity is claimed for what
was not stated. A query or fragment (`?token=...`) takes the per-project
node too, so a token never becomes part of a QN or a property.
One predicate now tells an external chart from a chart of the repository
where Terraform, shell and the Go installer index the chart directories:
an external chart a Chart.yaml depends on carries that file's path, and
must never stand in for the chart the directory declares.
Measured on 17 infrastructure repositories against an independent
reading: 141 of 141 edges into external charts reach the identity their
stated repository gives (114 by URL, 27 per project). The only
differences to the previous index are the 79 external Chart nodes that
moved to their URL identity, each with its repository property, and the
114 edges into them, re-pointed with type, source and properties
unchanged. Every chart gate (Helm includes, helmCharts, Terraform, the Go
installer, shell, CI, GitOps deploys) reads the same as before; three
runs are identical; the incremental re-index equals a full index for a
shared chart's owning file edited and deleted, a URL respelled, a
dependency moved to another publisher and to an alias, a chartRef URL
edit and a new Terraform oci:// release. The index rebuilds once on
upgrade with the other version 9 changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
1 parent
150851cc05
commit
dec56c9c69
7 files changed
+351
-60
No files matched your search
@@ -333,7 +333,7 @@ The install script placed beside the binary is **reported, not deleted**: uninst
|
||||
- `INFRA_MAPS`: a Kubernetes Service to each workload whose pod labels carry every `key: value` of the Service's selector, within one namespace
|
||||
- `CONNECTS_TO`: a Kubernetes workload or compose service to the Service its containers' environment addresses (`PRODUCT_CATALOG_SERVICE_ADDR: productcatalogservice:3550`, a URL's host, `<svc>.<ns>.svc...`, a bare name on an `*_ADDR` / `*_HOST` / `*_URL` / ... key; compose: the project's `.env` interpolated): a Kubernetes Service of the same bundle and namespace, or a compose service of the same file. The edge keeps `via: env`, `key` and `container`, never the value
|
||||
- `USAGE` with `ref: image`: a Kubernetes workload (any pod template: containers, initContainers, CronJobs, custom resources) or a docker compose service to the `Image` it runs
|
||||
- `DEPLOYS`: what puts manifests, kustomizations or charts on a cluster, to what it puts there: a Flux Kustomization or Argo Application to the kustomizations / manifests / chart of its path, a Flux HelmRelease to its chart (`spec.chart` or `chartRef` → OCIRepository), a shell command or Make recipe (`kubectl apply -f/-k`, `kustomize build | kubectl apply`, `helm install/upgrade/template`), a Go installer loading a chart with the Helm SDK, a Terraform `helm_release`, a Terraform `kubernetes_*` resource the object it declares (a `Resource` like a manifest's, `ecosystem: terraform`, when its name is a literal), a kustomization inflating a chart (`helmCharts`; its values file an `IMPORTS`). A chart outside the repository is a shared external `Chart`; the edge says `via` (`flux`, `argo`, `helmrelease`, `kubectl`, `kustomize`, `helm`, `helm-sdk`, `terraform`, `helmcharts`, ...)
|
||||
- `DEPLOYS`: what puts manifests, kustomizations or charts on a cluster, to what it puts there: a Flux Kustomization or Argo Application to the kustomizations / manifests / chart of its path, a Flux HelmRelease to its chart (`spec.chart` or `chartRef` → OCIRepository), a shell command or Make recipe (`kubectl apply -f/-k`, `kustomize build | kubectl apply`, `helm install/upgrade/template`), a Go installer loading a chart with the Helm SDK, a Terraform `helm_release`, a Terraform `kubernetes_*` resource the object it declares (a `Resource` like a manifest's, `ecosystem: terraform`, when its name is a literal), a kustomization inflating a chart (`helmCharts`; its values file an `IMPORTS`). A chart outside the repository is an external `Chart`, keyed by its publisher when its repository URL is stated (`__chart__<repository>/<name>`, the same node in every project, the URL in `repository`), else one per project and name; the edge says `via` (`flux`, `argo`, `helmrelease`, `kubectl`, `kustomize`, `helm`, `helm-sdk`, `terraform`, `helmcharts`, ...)
|
||||
- `DEPENDS_ON`: a Helm chart to the charts it declares (the local chart for a resolvable `file://` repository, otherwise an external `Chart`; the edge keeps `repository`), and a `go.mod` / `requirements.txt` file to an external `Package`; a Dockerfile to the `Image`s it builds on (`via: from`, `copy-from`); an `Image` to the Dockerfile it is built from (`via: build`; `tool`: `skaffold`, `compose`, `docker-build` for `docker` / `podman` / `nerdctl` / `buildah` build commands in scripts, Make and CI); a compose service to the services its `depends_on` / `links` name
|
||||
- `CONFIGURES`: code reading an environment variable to the shared `EnvVar` of that name (`strategy: env_access`); a Kubernetes container, compose service or Dockerfile (`ENV` of the image) setting one to that container's own `EnvVar` (`strategy: env_set`, with `source`: `value`, `configMapKeyRef`, `secretKeyRef`, `fieldRef`, ..., `compose`, `dockerfile`; values are never stored); and that container's `EnvVar` to the code reading the name inside the files its image is built from (`strategy: env_read`, with `dockerfile`): the image's files are the sources its Dockerfile's `COPY` / `ADD` put into it, less the context's `.dockerignore`
|
||||
- `OVERRIDE` with `via: patch`: a Kustomize patch to each object of its kustomization's build it changes (child → base): a patch file's documents to the object of the same kind and name; an inline patch (from the kustomization) or a JSON6902 operation file to what its `target` selects (group, version, kind, name, namespace, label selector); a component's in the build of each kustomization including it. The edge keeps `kustomization`, `patch` and `line`; a patch file is an `IMPORTS` (`local_name: patches`) of its kustomization, never a member of the build
|
||||
|
||||
+177
-27
@@ -26,6 +26,7 @@
|
||||
#include "foundation/str_util.h" /* cbm_json_escape */
|
||||
#include "pipeline/worker_pool.h"
|
||||
|
||||
#include <ctype.h>
|
||||
#include <stdatomic.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
@@ -2542,14 +2543,159 @@ static char *k8s_dep_props(const char *repository) {
|
||||
return props;
|
||||
}
|
||||
|
||||
/* DEPENDS_ON from `chart_id` to the shared external Chart node of `name`
|
||||
* (one per name and project, so charts depending on the same chart meet). */
|
||||
/* ── External charts ─────────────────────────────────────────────── */
|
||||
|
||||
/* An external Chart is keyed by its publisher: `__chart__<url>/<name>`, the
|
||||
* same QN in every project, so two repositories naming one published chart
|
||||
* name one node. A chart whose repository does not resolve to a URL (a
|
||||
* `@alias`, an empty or templated repository, a source the pass cannot read)
|
||||
* keeps the per-project `<project>.__helm_dep__.<name>`: no identity is
|
||||
* claimed for what was not stated. */
|
||||
static const char K8S_CHART_QN_PREFIX[] = "__chart__";
|
||||
static const char K8S_HELM_DEP_MARK[] = ".__helm_dep__.";
|
||||
|
||||
/* A byte a stated repository URL may hold: no whitespace or control byte,
|
||||
* no quote, template or shell expansion, escape or angle bracket. */
|
||||
static bool k8s_url_byte_ok(unsigned char c) {
|
||||
return c > ' ' && c < 0x7f && !strchr("\"'`{}$<>\\", c);
|
||||
}
|
||||
|
||||
bool cbm_chart_repo_url(const char *url, char *out, size_t sz) {
|
||||
if (!url || !out || sz == 0) {
|
||||
return false;
|
||||
}
|
||||
while (*url == ' ' || *url == '\t') {
|
||||
url++;
|
||||
}
|
||||
size_t n = strlen(url);
|
||||
while (n > 0 && isspace((unsigned char)url[n - SKIP_ONE])) {
|
||||
n--;
|
||||
}
|
||||
if (n == 0 || n + SKIP_ONE > sz || !isalpha((unsigned char)url[0])) {
|
||||
return false;
|
||||
}
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
if (!k8s_url_byte_ok((unsigned char)url[i])) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
/* a query or fragment may carry a token (?token=...): like credentials
|
||||
* in the authority, never put into a QN or a property */
|
||||
if (memchr(url, '?', n) || memchr(url, '#', n)) {
|
||||
return false;
|
||||
}
|
||||
size_t s = 0; /* the scheme: [A-Za-z][A-Za-z0-9+.-]* then :// */
|
||||
while (s < n &&
|
||||
(isalnum((unsigned char)url[s]) || url[s] == '+' || url[s] == '-' || url[s] == '.')) {
|
||||
s++;
|
||||
}
|
||||
if (s + K8S_SLEN("://") > n || strncmp(url + s, "://", K8S_SLEN("://")) != 0) {
|
||||
return false;
|
||||
}
|
||||
size_t host = s + K8S_SLEN("://");
|
||||
size_t path = host;
|
||||
while (path < n && url[path] != '/') {
|
||||
path++;
|
||||
}
|
||||
/* no host, or credentials in the authority: never put into a QN */
|
||||
if (path == host || memchr(url + host, '@', path - host)) {
|
||||
return false;
|
||||
}
|
||||
for (size_t i = 0; i < path; i++) {
|
||||
out[i] = (char)tolower((unsigned char)url[i]);
|
||||
}
|
||||
memcpy(out + path, url + path, n - path);
|
||||
out[n] = '\0';
|
||||
if (strncmp(out, "file://", K8S_SLEN("file://")) == 0) {
|
||||
return false; /* a directory, not a publisher */
|
||||
}
|
||||
size_t end = n;
|
||||
while (end > path && out[end - SKIP_ONE] == '/') {
|
||||
end--;
|
||||
}
|
||||
if (end - path >= K8S_SLEN(".git") &&
|
||||
strncmp(out + end - K8S_SLEN(".git"), ".git", K8S_SLEN(".git")) == 0) {
|
||||
end -= K8S_SLEN(".git");
|
||||
while (end > path && out[end - SKIP_ONE] == '/') {
|
||||
end--;
|
||||
}
|
||||
}
|
||||
out[end] = '\0';
|
||||
return true;
|
||||
}
|
||||
|
||||
bool cbm_chart_ref_repo(const char *ref, char *out, size_t sz) {
|
||||
if (!ref || !out || sz == 0) {
|
||||
return false;
|
||||
}
|
||||
size_t end = strlen(ref);
|
||||
while (end > 0 && ref[end - SKIP_ONE] == '/') {
|
||||
end--;
|
||||
}
|
||||
while (end > 0 && ref[end - SKIP_ONE] != '/') {
|
||||
end--;
|
||||
}
|
||||
while (end > 0 && ref[end - SKIP_ONE] == '/') {
|
||||
end--;
|
||||
}
|
||||
if (end == 0 || end + SKIP_ONE > sz) {
|
||||
return false;
|
||||
}
|
||||
memcpy(out, ref, end);
|
||||
out[end] = '\0';
|
||||
return true;
|
||||
}
|
||||
|
||||
/* QN and node properties of the external Chart `name` (non-NULL) of
|
||||
* repository `url` (NULL: unknown). */
|
||||
static void k8s_chart_identity(const char *project, const char *name, const char *url, char *qn,
|
||||
size_t qsz, char *props, size_t psz) {
|
||||
char norm[CBM_SZ_1K];
|
||||
if (name[0] && !strchr(name, '/') && cbm_chart_repo_url(url, norm, sizeof(norm))) {
|
||||
int w = snprintf(qn, qsz, "%s%s/%s", K8S_CHART_QN_PREFIX, norm, name);
|
||||
if (w > 0 && (size_t)w < qsz) {
|
||||
char esc[CBM_SZ_2K];
|
||||
cbm_json_escape(esc, (int)sizeof(esc), norm);
|
||||
snprintf(props, psz, "{\"source\":\"helm\",\"external\":true,\"repository\":\"%s\"}",
|
||||
esc);
|
||||
return;
|
||||
}
|
||||
}
|
||||
snprintf(qn, qsz, "%s%s%s", project, K8S_HELM_DEP_MARK, name);
|
||||
snprintf(props, psz, "{\"source\":\"helm\",\"external\":true}");
|
||||
}
|
||||
|
||||
int64_t cbm_pipeline_external_chart(cbm_pipeline_ctx_t *ctx, const char *name, const char *url,
|
||||
const char *file) {
|
||||
if (!name) {
|
||||
return 0;
|
||||
}
|
||||
char qn[CBM_SZ_2K];
|
||||
char props[CBM_SZ_4K];
|
||||
k8s_chart_identity(ctx->project_name, name, url, qn, sizeof(qn), props, sizeof(props));
|
||||
const cbm_gbuf_node_t *known = cbm_gbuf_find_by_qn(ctx->gbuf, qn);
|
||||
if (known) {
|
||||
return known->id;
|
||||
}
|
||||
return cbm_gbuf_upsert_node(ctx->gbuf, "Chart", name, qn, file, SKIP_ONE, 0, props);
|
||||
}
|
||||
|
||||
bool cbm_pipeline_chart_is_external(const cbm_gbuf_node_t *node) {
|
||||
const char *qn = node ? node->qualified_name : NULL;
|
||||
return qn && (strncmp(qn, K8S_CHART_QN_PREFIX, K8S_SLEN(K8S_CHART_QN_PREFIX)) == 0 ||
|
||||
strstr(qn, K8S_HELM_DEP_MARK) != NULL);
|
||||
}
|
||||
|
||||
/* DEPENDS_ON from `chart_id` to the external Chart node of `name` in
|
||||
* `repository` (charts depending on the same published chart meet). */
|
||||
static bool k8s_external_dep(cbm_pipeline_ctx_t *ctx, int64_t chart_id, const char *name,
|
||||
const char *chart_rel, const char *props) {
|
||||
char dep_qn[CBM_SZ_512];
|
||||
snprintf(dep_qn, sizeof(dep_qn), "%s.__helm_dep__.%s", ctx->project_name, name);
|
||||
int64_t dep_id = cbm_gbuf_upsert_node(ctx->gbuf, "Chart", name, dep_qn, chart_rel, SKIP_ONE, 0,
|
||||
"{\"source\":\"helm\",\"external\":true}");
|
||||
const char *repository, const char *chart_rel, const char *props) {
|
||||
char dep_qn[CBM_SZ_2K];
|
||||
char node_props[CBM_SZ_4K];
|
||||
k8s_chart_identity(ctx->project_name, name, repository, dep_qn, sizeof(dep_qn), node_props,
|
||||
sizeof(node_props));
|
||||
int64_t dep_id =
|
||||
cbm_gbuf_upsert_node(ctx->gbuf, "Chart", name, dep_qn, chart_rel, SKIP_ONE, 0, node_props);
|
||||
if (dep_id <= 0) {
|
||||
return false;
|
||||
}
|
||||
@@ -2613,7 +2759,9 @@ static void k8s_link_local_deps(cbm_pipeline_ctx_t *ctx, k8s_local_deps_t *deps)
|
||||
target->id != d->chart_id) {
|
||||
cbm_gbuf_insert_edge(ctx->gbuf, d->chart_id, target->id, "DEPENDS_ON", d->props);
|
||||
} else if (d->chart_rel) {
|
||||
(void)k8s_external_dep(ctx, d->chart_id, d->name, d->chart_rel, d->props);
|
||||
/* a file:// directory this repository cannot resolve names no
|
||||
* publisher: the per-project external Chart */
|
||||
(void)k8s_external_dep(ctx, d->chart_id, d->name, NULL, d->chart_rel, d->props);
|
||||
}
|
||||
}
|
||||
k8s_local_deps_free(deps);
|
||||
@@ -2621,8 +2769,9 @@ static void k8s_link_local_deps(cbm_pipeline_ctx_t *ctx, k8s_local_deps_t *deps)
|
||||
|
||||
/* Emit a Chart node for a Chart.yaml and a DEPENDS_ON edge per declared
|
||||
* dependency (#338): to the chart in this repository for a resolvable
|
||||
* `file://` dependency (once all charts are read), otherwise to a shared
|
||||
* external Chart node per dependency name, the repository on the edge. */
|
||||
* `file://` dependency (once all charts are read), otherwise to the external
|
||||
* Chart node of the dependency's repository and name, the repository also on
|
||||
* the edge. */
|
||||
static void handle_helm_chart(cbm_pipeline_ctx_t *ctx, const char *rel_path, const char *source,
|
||||
k8s_local_deps_t *local_deps) {
|
||||
cbm_helm_chart_t hc;
|
||||
@@ -2656,7 +2805,7 @@ static void handle_helm_chart(cbm_pipeline_ctx_t *ctx, const char *rel_path, con
|
||||
continue;
|
||||
}
|
||||
char *props = dep->repository[0] ? k8s_dep_props(dep->repository) : NULL;
|
||||
dep_edges += k8s_external_dep(ctx, chart_id, dep->name, rel_path, props);
|
||||
dep_edges += k8s_external_dep(ctx, chart_id, dep->name, dep->repository, rel_path, props);
|
||||
cbm_free(CBM_MEM_CLASS_OTHER, props);
|
||||
}
|
||||
cbm_helm_chart_free(&hc);
|
||||
@@ -3439,7 +3588,7 @@ static void k8s_link_includes(cbm_pipeline_ctx_t *ctx, k8s_bundles_t *b,
|
||||
* outermost kustomizations below it -- their Kustomization node -- and the
|
||||
* manifests outside any, their File node; a chart dir: its Chart); a
|
||||
* HelmRelease -> its chart (a repository chart dir of a Git/Bucket source,
|
||||
* else the shared external Chart of that name). */
|
||||
* else the external Chart of that name). */
|
||||
|
||||
typedef struct {
|
||||
cbm_pipeline_ctx_t *ctx;
|
||||
@@ -3475,21 +3624,17 @@ static int64_t k8s_chart_node(k8s_dep_link_t *l, const char *dir) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* The shared external Chart of `name` (the one chart dependencies use). */
|
||||
static int64_t k8s_external_chart(k8s_dep_link_t *l, const char *name, const char *file) {
|
||||
char qn[CBM_SZ_512];
|
||||
snprintf(qn, sizeof(qn), "%s.__helm_dep__.%s", l->ctx->project_name, name);
|
||||
const cbm_gbuf_node_t *known = cbm_gbuf_find_by_qn(l->ctx->gbuf, qn);
|
||||
if (known) {
|
||||
return known->id;
|
||||
}
|
||||
return cbm_gbuf_upsert_node(l->ctx->gbuf, "Chart", name, qn, file, SKIP_ONE, 0,
|
||||
"{\"source\":\"helm\",\"external\":true}");
|
||||
/* The external Chart of `name` in repository `url` (NULL: unknown), the one
|
||||
* chart dependencies use. */
|
||||
static int64_t k8s_external_chart(k8s_dep_link_t *l, const char *name, const char *url,
|
||||
const char *file) {
|
||||
return cbm_pipeline_external_chart(l->ctx, name, url, file);
|
||||
}
|
||||
|
||||
/* A HelmRelease whose chartRef names OCIRepository `oci` (its USAGE edge
|
||||
* {"ref":"chartRef"}, resolved with the references) deploys the chart that
|
||||
* repository fetches: the shared external Chart of the URL's last segment. */
|
||||
* repository fetches: the external Chart of that OCI reference (its last
|
||||
* segment the chart, the rest its repository). */
|
||||
static void k8s_link_chart_refs(k8s_dep_link_t *l, const k8s_hr_t *oci) {
|
||||
const char *url = oci->chart;
|
||||
size_t end = strlen(url);
|
||||
@@ -3513,6 +3658,8 @@ static void k8s_link_chart_refs(k8s_dep_link_t *l, const k8s_hr_t *oci) {
|
||||
snprintf(props, sizeof(props),
|
||||
"{\"via\":\"helmrelease\",\"chart\":\"%s\",\"external\":true,\"url\":\"%s\"}",
|
||||
esc_name, esc_url);
|
||||
char repo[CBM_SZ_1K];
|
||||
const char *repo_url = cbm_chart_ref_repo(url, repo, sizeof(repo)) ? repo : NULL;
|
||||
const cbm_gbuf_edge_t **in = NULL;
|
||||
int n = 0;
|
||||
if (cbm_gbuf_find_edges_by_target_type(l->ctx->gbuf, oci->node, "USAGE", &in, &n) != 0) {
|
||||
@@ -3523,7 +3670,8 @@ static void k8s_link_chart_refs(k8s_dep_link_t *l, const k8s_hr_t *oci) {
|
||||
const char *p = in[i]->properties_json;
|
||||
const cbm_gbuf_node_t *hr = cbm_gbuf_find_by_id(l->ctx->gbuf, in[i]->source_id);
|
||||
if (p && strstr(p, "\"ref\":\"chartRef\"") && hr && hr->file_path) {
|
||||
k8s_deploys_edge(l, hr->id, k8s_external_chart(l, name, hr->file_path), props);
|
||||
k8s_deploys_edge(l, hr->id, k8s_external_chart(l, name, repo_url, hr->file_path),
|
||||
props);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3542,7 +3690,7 @@ static void k8s_prop_add(char *out, size_t out_sz, int *w, const char *key, cons
|
||||
* build: DEPLOYS {via: helmcharts, external, chart, url, version, release,
|
||||
* namespace, values, line} from the kustomization (its Kustomization
|
||||
* Resource, else its File -- the unit a Flux / Argo deploy reaches) to the
|
||||
* shared external Chart of that name. */
|
||||
* external Chart of that repository and name. */
|
||||
static void k8s_link_kcharts(k8s_dep_link_t *l, const k8s_facts_t *f) {
|
||||
for (int i = 0; i < f->kchart_n; i++) {
|
||||
const k8s_kchart_t *c = &f->kcharts[i];
|
||||
@@ -3563,7 +3711,7 @@ static void k8s_link_kcharts(k8s_dep_link_t *l, const k8s_facts_t *f) {
|
||||
continue;
|
||||
}
|
||||
snprintf(props + w, sizeof(props) - (size_t)w, ",\"line\":%d}", c->line);
|
||||
k8s_deploys_edge(l, src, k8s_external_chart(l, c->chart, c->kust), props);
|
||||
k8s_deploys_edge(l, src, k8s_external_chart(l, c->chart, c->url, c->kust), props);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4230,7 +4378,9 @@ static void k8s_link_deploys(cbm_pipeline_ctx_t *ctx, k8s_kusts_t *ks, k8s_bundl
|
||||
k8s_deploys_edge(&l, hr->node, k8s_chart_node(&l, dir), props);
|
||||
}
|
||||
} else if (src && src->file_path && !strchr(hr->chart, '/') && !strchr(hr->chart, '$')) {
|
||||
k8s_deploys_edge(&l, hr->node, k8s_external_chart(&l, hr->chart, src->file_path),
|
||||
/* the source's URL is not read (a HelmRepository's spec.url):
|
||||
* the per-project external Chart */
|
||||
k8s_deploys_edge(&l, hr->node, k8s_external_chart(&l, hr->chart, NULL, src->file_path),
|
||||
props);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1640,7 +1640,7 @@ static CBMHashTable *gc_chart_index(const cbm_gbuf_t *gb) {
|
||||
const char *qn = nodes[i]->qualified_name;
|
||||
const char *slash = fp ? strrchr(fp, '/') : NULL;
|
||||
const char *base = slash ? slash + SKIP_ONE : fp;
|
||||
if (!fp || !qn || strstr(qn, ".__helm_dep__.") ||
|
||||
if (!fp || !qn || cbm_pipeline_chart_is_external(nodes[i]) ||
|
||||
(strcmp(base, "Chart.yaml") != 0 && strcmp(base, "Chart.yml") != 0) || !slash) {
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -11,8 +11,8 @@
|
||||
* kubectl ... -k D the Kustomization of D
|
||||
* kustomize build D | ... | kubectl apply -f - (also kubectl kustomize D)
|
||||
* helm install|upgrade [NAME] CHART the Chart of a chart dir, else the
|
||||
* shared external Chart (repo/name,
|
||||
* oci://.../name)
|
||||
* external Chart of that name
|
||||
* (repo/name, oci://.../name)
|
||||
* helm template [NAME] CHART | ... | kubectl apply -f -
|
||||
* find D -name G -exec kubectl apply -f {} \; D's manifests matching G
|
||||
*
|
||||
@@ -216,7 +216,7 @@ static bool sh_index_build(sh_index_t *ix, cbm_pipeline_ctx_t *ctx) {
|
||||
const char *base = fp ? sh_basename(fp) : "";
|
||||
const char *qn = nodes[i]->qualified_name;
|
||||
if ((strcmp(base, "Chart.yaml") == 0 || strcmp(base, "Chart.yml") == 0) && qn &&
|
||||
!strstr(qn, ".__helm_dep__.")) {
|
||||
!cbm_pipeline_chart_is_external(nodes[i])) {
|
||||
cbm_ht_set(ix->charts, fp, (void *)(intptr_t)nodes[i]->id);
|
||||
}
|
||||
}
|
||||
@@ -3569,16 +3569,11 @@ static int64_t sh_source(cbm_pipeline_ctx_t *ctx, const sh_span_t *spans, int n,
|
||||
return fn ? fn->id : 0;
|
||||
}
|
||||
|
||||
/* The shared external Chart of `name` (the one chart dependencies use). */
|
||||
/* The external Chart of `name` a helm command names: the hit keeps no
|
||||
* repository URL (a `repo/name` alias, an oci:// reference's last segment),
|
||||
* so the per-project one. */
|
||||
static int64_t sh_external_chart(cbm_pipeline_ctx_t *ctx, const char *name, const char *file) {
|
||||
char qn[CBM_SZ_512];
|
||||
snprintf(qn, sizeof(qn), "%s.__helm_dep__.%s", ctx->project_name, name);
|
||||
const cbm_gbuf_node_t *known = cbm_gbuf_find_by_qn(ctx->gbuf, qn);
|
||||
if (known) {
|
||||
return known->id;
|
||||
}
|
||||
return cbm_gbuf_upsert_node(ctx->gbuf, "Chart", name, qn, file, SKIP_ONE, 0,
|
||||
"{\"source\":\"helm\",\"external\":true}");
|
||||
return cbm_pipeline_external_chart(ctx, name, NULL, file);
|
||||
}
|
||||
|
||||
/* A hit's CI context as JSON members (`,"ci":..,"job":..`); "" for none. */
|
||||
|
||||
@@ -7,8 +7,9 @@
|
||||
* chart = "./charts/x" / "${path.module}/charts/x" the Chart of that
|
||||
* directory, relative to
|
||||
* the module's directory
|
||||
* repository = "<url>", chart = "<name>" the shared external
|
||||
* chart = "oci://registry/.../<name>" Chart of that name
|
||||
* repository = "<url>", chart = "<name>" the external Chart of
|
||||
* chart = "oci://registry/.../<name>" that name (an oci://
|
||||
* reference: of its URL)
|
||||
* Values are evaluated from string literals and ${path.module}; any other
|
||||
* expression is unknown, and an unknown chart is no edge. Run at the end of
|
||||
* the K8s pass, over the same files.
|
||||
@@ -198,7 +199,9 @@ static CBMHashTable *tf_chart_index(cbm_pipeline_ctx_t *ctx) {
|
||||
const char *fp = nodes[i]->file_path;
|
||||
const char *qn = nodes[i]->qualified_name;
|
||||
const char *base = fp ? tf_basename(fp) : "";
|
||||
if (fp && qn && !strstr(qn, ".__helm_dep__.") &&
|
||||
/* an external chart a Chart.yaml depends on carries that file's
|
||||
* path too: only the chart it declares is the directory's */
|
||||
if (fp && qn && !cbm_pipeline_chart_is_external(nodes[i]) &&
|
||||
(strcmp(base, "Chart.yaml") == 0 || strcmp(base, "Chart.yml") == 0)) {
|
||||
cbm_ht_set(ix, fp, (void *)(intptr_t)nodes[i]->id);
|
||||
}
|
||||
@@ -220,18 +223,6 @@ static int64_t tf_local_chart(const tf_t *t, const char *dir) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* The shared external Chart of `name` (the one chart dependencies use). */
|
||||
static int64_t tf_external_chart(cbm_pipeline_ctx_t *ctx, const char *name, const char *file) {
|
||||
char qn[CBM_SZ_512];
|
||||
snprintf(qn, sizeof(qn), "%s.__helm_dep__.%s", ctx->project_name, name);
|
||||
const cbm_gbuf_node_t *known = cbm_gbuf_find_by_qn(ctx->gbuf, qn);
|
||||
if (known) {
|
||||
return known->id;
|
||||
}
|
||||
return cbm_gbuf_upsert_node(ctx->gbuf, "Chart", name, qn, file, SKIP_ONE, 0,
|
||||
"{\"source\":\"helm\",\"external\":true}");
|
||||
}
|
||||
|
||||
/* The node of block `resource.helm_release.<name>` in the file, by the QN
|
||||
* its definition got -- not among every block of that name: modules that
|
||||
* share a release name (`this`) would make each lookup scan all of them. */
|
||||
@@ -293,6 +284,9 @@ static void tf_helm_release(tf_t *t, TSNode block, const char *name) {
|
||||
int64_t src = tf_block_node(t, name);
|
||||
int64_t tgt = 0;
|
||||
bool external = false;
|
||||
/* the repository of an oci:// reference; "" when unknown (a
|
||||
* `repository =` value is not read) */
|
||||
char repo[CBM_SZ_1K] = "";
|
||||
char path[CBM_SZ_1K] = "";
|
||||
static const char OCI[] = "oci://";
|
||||
if (strncmp(chart, OCI, sizeof(OCI) - SKIP_ONE) == 0) {
|
||||
@@ -300,6 +294,10 @@ static void tf_helm_release(tf_t *t, TSNode block, const char *name) {
|
||||
if (tf_plain_name(base)) {
|
||||
snprintf(path, sizeof(path), "%s", base);
|
||||
external = true;
|
||||
/* the reference names its repository: all but the chart */
|
||||
if (!cbm_chart_ref_repo(chart, repo, sizeof(repo))) {
|
||||
repo[0] = '\0';
|
||||
}
|
||||
}
|
||||
} else if (has_repo) {
|
||||
if (tf_plain_name(chart)) {
|
||||
@@ -310,7 +308,7 @@ static void tf_helm_release(tf_t *t, TSNode block, const char *name) {
|
||||
tgt = tf_local_chart(t, path);
|
||||
}
|
||||
if (external) {
|
||||
tgt = tf_external_chart(t->ctx, path, t->file);
|
||||
tgt = cbm_pipeline_external_chart(t->ctx, path, repo[0] ? repo : NULL, t->file);
|
||||
}
|
||||
if (src <= 0 || tgt <= 0) {
|
||||
return;
|
||||
|
||||
@@ -1337,6 +1337,30 @@ bool cbm_repo_rel_join(const char *dir, const char *v, char *out, size_t sz);
|
||||
* digest dropped), made when new; 0 for scratch or no image. */
|
||||
int64_t cbm_pipeline_image_node(cbm_pipeline_ctx_t *ctx, const char *ref);
|
||||
|
||||
/* A chart repository URL as an identity (pass_k8s.c): `url` trimmed, its
|
||||
* scheme and host lower-cased, trailing `/` and `.git` dropped. False for no
|
||||
* URL: no scheme or host, a file:// directory, credentials in the authority,
|
||||
* a query or fragment (`?`, `#`: a token may ride there), or a byte no
|
||||
* stated URL holds (whitespace, quote, template, `$`). A
|
||||
* repository URL is never cut: oci://host/x with chart x is oci://host/x/x. */
|
||||
bool cbm_chart_repo_url(const char *url, char *out, size_t sz);
|
||||
|
||||
/* The repository of chart reference `ref` (a Flux OCIRepository url, a
|
||||
* Terraform `chart = "oci://..."`): all before its last segment, the chart.
|
||||
* False when none is left. */
|
||||
bool cbm_chart_ref_repo(const char *ref, char *out, size_t sz);
|
||||
|
||||
/* The external Chart node of chart `name` from repository `url` (pass_k8s.c),
|
||||
* made when new and owned by `file`: `__chart__<url>/<name>` {source,
|
||||
* external, repository} -- one QN in every project -- when `url` resolves
|
||||
* (cbm_chart_repo_url), else `<project>.__helm_dep__.<name>`. */
|
||||
int64_t cbm_pipeline_external_chart(cbm_pipeline_ctx_t *ctx, const char *name, const char *url,
|
||||
const char *file);
|
||||
|
||||
/* Is Chart node `node` an external chart (either identity) rather than one
|
||||
* a Chart.yaml of this repository declares? */
|
||||
bool cbm_pipeline_chart_is_external(const cbm_gbuf_node_t *node);
|
||||
|
||||
/* Environment variable `key` that `owner` sets (pass_docker.c): a K8s or
|
||||
* compose Resource for its container `container`, a Dockerfile Module for
|
||||
* its image (container NULL). Makes the container-scoped EnvVar
|
||||
@@ -1548,8 +1572,9 @@ int cbm_pipeline_build_fresh_semantic_manifest(cbm_pipeline_t *p, const char *pr
|
||||
* and roles its directives name, its tasks and handlers are Functions
|
||||
* {source:ansible}, a notify CALLS {via:notify} the handlers it names,
|
||||
* a kube / k8s task DEPLOYS {via:ansible} the template it applies
|
||||
* (pass_ansible.c). An index built before holds none of them, so it
|
||||
* rebuilds once on upgrade. */
|
||||
* (pass_ansible.c). An external Chart is keyed by its publisher URL.
|
||||
* An index built before holds none of them, so it rebuilds once on
|
||||
* upgrade. */
|
||||
enum { CBM_SEMANTIC_INDEX_VERSION = 9 };
|
||||
|
||||
typedef struct {
|
||||
|
||||
@@ -1812,7 +1812,7 @@ static char *gpf_deploys(const GpfFile *files, int nfiles) {
|
||||
"SELECT coalesce(group_concat(x, char(10)), '') FROM (SELECT "
|
||||
"CASE WHEN s.label IN ('Module','File') THEN '@' || s.file_path ELSE s.name END "
|
||||
"|| ' -> ' || CASE WHEN t.label='Chart' THEN 'chart:' || t.name || "
|
||||
"CASE WHEN instr(t.qualified_name,'__helm_dep__') THEN '(ext)' ELSE '' END "
|
||||
"CASE WHEN json_extract(t.properties,'$.external') THEN '(ext)' ELSE '' END "
|
||||
"ELSE t.label || ':' || t.file_path END "
|
||||
"|| ' (' || json_extract(e.properties,'$.via') || ')' AS x "
|
||||
"FROM edges e JOIN nodes s ON s.id=e.source_id JOIN nodes t ON t.id=e.target_id "
|
||||
@@ -4763,6 +4763,128 @@ TEST(probe_helm_unresolved_file_dependency_stays_external) {
|
||||
PASS();
|
||||
}
|
||||
|
||||
/* Every external Chart as "<name> <QN, the project as <p>> <repository or ->
|
||||
* in=<edges into it>", sorted, one per line. NULL on failure; the caller
|
||||
* frees. */
|
||||
static char *gpf_external_charts(const GpfFile *files, int nfiles) {
|
||||
GpfProj lp;
|
||||
cbm_store_t *store = gpf_index_files(&lp, files, nfiles);
|
||||
char *out = NULL;
|
||||
if (store) {
|
||||
sqlite3_stmt *st = NULL;
|
||||
const char *sql =
|
||||
"SELECT coalesce(group_concat(x, char(10)), '') FROM (SELECT n.name || ' ' || "
|
||||
"replace(n.qualified_name, ?1, '<p>') || ' ' || "
|
||||
"coalesce(json_extract(n.properties,'$.repository'), '-') || ' in=' || "
|
||||
"(SELECT count(*) FROM edges e WHERE e.target_id=n.id) AS x FROM nodes n "
|
||||
"WHERE n.project=?1 AND n.label='Chart' AND json_extract(n.properties,'$.external') "
|
||||
"ORDER BY x)";
|
||||
if (sqlite3_prepare_v2(cbm_store_get_db(store), sql, -1, &st, NULL) == SQLITE_OK) {
|
||||
sqlite3_bind_text(st, 1, lp.project, -1, SQLITE_STATIC);
|
||||
if (sqlite3_step(st) == SQLITE_ROW) {
|
||||
out = strdup((const char *)sqlite3_column_text(st, 0));
|
||||
}
|
||||
}
|
||||
sqlite3_finalize(st);
|
||||
}
|
||||
gpf_cleanup(&lp, store);
|
||||
return out;
|
||||
}
|
||||
|
||||
/* An external Chart is keyed by its publisher, __chart__<repository>/<name>,
|
||||
* the repository's scheme and host lower-cased, a trailing slash and `.git`
|
||||
* dropped: a Chart.yaml dependency, a kustomization, a Flux chartRef and a
|
||||
* Terraform oci:// reference naming one published chart meet in one node,
|
||||
* the same QN in every project. An oci:// repository is never cut, even when
|
||||
* it ends in the chart's name (karpenter); a chart reference is cut to its
|
||||
* repository. A repository that is no URL (an @alias, no scheme, file://,
|
||||
* credentials in the authority, a query or fragment a token may ride in, an
|
||||
* unexpanded ${...}) or that the pass does not read (a
|
||||
* Terraform `repository =`) keeps the per-project
|
||||
* <project>.__helm_dep__.<name>. The external charts a Chart.yaml depends on
|
||||
* carry its path, yet a helm_release, a helm command and a Go loader naming
|
||||
* that chart directory deploy the chart the directory declares. */
|
||||
TEST(probe_external_chart_identity_is_its_publisher) {
|
||||
static const GpfFile files[] = {
|
||||
{"charts/app/Chart.yaml",
|
||||
"apiVersion: v2\nname: app\nversion: 1.0.0\ndependencies:\n"
|
||||
" - name: redis\n repository: https://Charts.Example.com/stable/\n"
|
||||
" version: 1.x\n"
|
||||
" - name: common\n repository: \"@stable\"\n version: 1.x\n"
|
||||
" - name: nats\n repository: oci://registry.example.com/charts\n"
|
||||
" version: 1.x\n"
|
||||
" - name: vault\n repository: https://user@charts.example.com/private\n"
|
||||
" version: 1.x\n"},
|
||||
{"main.tf", "resource \"helm_release\" \"app\" {\n"
|
||||
" name = \"app\"\n chart = \"./charts/app\"\n}\n\n"
|
||||
"resource \"helm_release\" \"queue\" {\n"
|
||||
" name = \"queue\"\n"
|
||||
" chart = \"oci://registry.example.com/charts/nats\"\n}\n\n"
|
||||
"resource \"helm_release\" \"cache\" {\n"
|
||||
" name = \"cache\"\n"
|
||||
" repository = \"https://charts.example.com/stable\"\n"
|
||||
" chart = \"redis\"\n}\n"},
|
||||
{"deploy.sh", "#!/bin/sh\ndeploy_app() {\n helm upgrade --install app ./charts/app\n}\n"},
|
||||
{"go.mod", "module example.com/app\n\ngo 1.22\n"},
|
||||
{"install.go", "package main\n\n"
|
||||
"import \"helm.sh/helm/v3/pkg/chart/loader\"\n\n"
|
||||
"func installApp() {\n\t_, _ = loader.Load(\"charts/app\")\n}\n"},
|
||||
{"obs/kustomization.yaml",
|
||||
GPF_KUST "helmCharts:\n- name: redis\n repo: https://charts.example.com/stable.git\n"
|
||||
"- name: karpenter\n repo: oci://registry.example.com/karpenter\n"
|
||||
"- name: templated\n repo: https://charts.example.com/${ENV}\n"
|
||||
"- name: local\n repo: file://../charts\n"
|
||||
"- name: bare\n repo: charts.example.com/stable\n"
|
||||
"- name: tokened\n repo: https://charts.example.com/stable?token=x\n"
|
||||
"- name: fragment\n repo: https://charts.example.com/stable#x\n"},
|
||||
{"apps/web/kustomization.yaml",
|
||||
GPF_KUST "namespace: web\nresources:\n - ocirepository.yaml\n - helmrelease.yaml\n"},
|
||||
{"apps/web/ocirepository.yaml",
|
||||
"apiVersion: source.toolkit.fluxcd.io/v1\nkind: OCIRepository\nmetadata:\n"
|
||||
" name: web-chart\nspec:\n url: oci://registry.example.com/charts/nats\n"},
|
||||
{"apps/web/helmrelease.yaml",
|
||||
"apiVersion: helm.toolkit.fluxcd.io/v2\nkind: HelmRelease\nmetadata:\n name: web\n"
|
||||
"spec:\n chartRef:\n kind: OCIRepository\n name: web-chart\n"}};
|
||||
int n = (int)(sizeof(files) / sizeof(files[0]));
|
||||
char *got = gpf_external_charts(files, n);
|
||||
static const char want[] =
|
||||
"bare <p>.__helm_dep__.bare - in=1\n"
|
||||
"common <p>.__helm_dep__.common - in=1\n"
|
||||
"fragment <p>.__helm_dep__.fragment - in=1\n"
|
||||
"karpenter __chart__oci://registry.example.com/karpenter/karpenter "
|
||||
"oci://registry.example.com/karpenter in=1\n"
|
||||
"local <p>.__helm_dep__.local - in=1\n"
|
||||
"nats __chart__oci://registry.example.com/charts/nats oci://registry.example.com/charts "
|
||||
"in=3\n"
|
||||
"redis <p>.__helm_dep__.redis - in=1\n"
|
||||
"redis __chart__https://charts.example.com/stable/redis https://charts.example.com/stable "
|
||||
"in=2\n"
|
||||
"templated <p>.__helm_dep__.templated - in=1\n"
|
||||
"tokened <p>.__helm_dep__.tokened - in=1\n"
|
||||
"vault <p>.__helm_dep__.vault - in=1";
|
||||
if (!got || strcmp(got, want) != 0) {
|
||||
fprintf(stderr, "external charts got:\n%s\nwant:\n%s\n", got ? got : "(null)", want);
|
||||
free(got);
|
||||
FAIL("external chart identity");
|
||||
}
|
||||
free(got);
|
||||
GPF_ASSERT_DEPLOYS(files, n,
|
||||
"HelmRelease/web -> chart:nats(ext) (helmrelease)\n"
|
||||
"Kustomization -> chart:bare(ext) (helmcharts)\n"
|
||||
"Kustomization -> chart:fragment(ext) (helmcharts)\n"
|
||||
"Kustomization -> chart:karpenter(ext) (helmcharts)\n"
|
||||
"Kustomization -> chart:local(ext) (helmcharts)\n"
|
||||
"Kustomization -> chart:redis(ext) (helmcharts)\n"
|
||||
"Kustomization -> chart:templated(ext) (helmcharts)\n"
|
||||
"Kustomization -> chart:tokened(ext) (helmcharts)\n"
|
||||
"deploy_app -> chart:app (helm)\n"
|
||||
"installApp -> chart:app (helm-sdk)\n"
|
||||
"resource.helm_release.app -> chart:app (terraform)\n"
|
||||
"resource.helm_release.cache -> chart:redis(ext) (terraform)\n"
|
||||
"resource.helm_release.queue -> chart:nats(ext) (terraform)");
|
||||
PASS();
|
||||
}
|
||||
|
||||
/* A remote base is outside the repository. The import resolver's symbol-name
|
||||
* guess walked its URL segments and linked kubermatic's gcp addon to the
|
||||
* `deploy` function of hack/ci/deploy.sh. A kustomize entry is a path,
|
||||
@@ -5980,6 +6102,7 @@ SUITE(grammar_probe_f) {
|
||||
RUN_TEST(probe_k8s_manifest_mints_no_envelope_variables);
|
||||
RUN_TEST(probe_yaml_config_keeps_variables);
|
||||
RUN_TEST(probe_helm_unresolved_file_dependency_stays_external);
|
||||
RUN_TEST(probe_external_chart_identity_is_its_publisher);
|
||||
RUN_TEST(probe_kustomize_nested_sibling_entries);
|
||||
RUN_TEST(probe_kustomize_single_module);
|
||||
RUN_TEST(probe_kustomize_entries_after_commented_out_entry);
|
||||
|
||||
Reference in new issue
Block a user