From 150851cc058c8de11926508e75b5838b91c82896 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Wed, 14 Oct 2026 20:46:18 +0200 Subject: [PATCH 1/3] feat(infra): an Ansible kube / k8s task DEPLOYS the template it applies kubespray applies its cluster add-ons with its own `kube` module: a task runs kubectl apply on a manifest the play rendered from a template a few tasks earlier. Both tasks were Functions and the template a File, with nothing between them: no query reached the manifest a task applies. A kube (kubespray's library module) or k8s (kubernetes.core / community.kubernetes) task now DEPLOYS {via: ansible, module, line} the repository file its play rendered the applied manifest from: a template task's src (the role's templates/, else the task file's directory) or a copy task's (its files/; never a remote_src copy, which reads the host). The applied filename is traced through the play: a renderer whose dest is the very string; {{ item.item.K }} over "{{ R.results }}" -> each item of the renderer registering R; {{ item.K }} -> each of the task's own items through the renderer of that dest; a literal filename a looped renderer renders to; a k8s template: argument. A loop list is a literal, a set_fact of the file, or a list of the role's defaults / vars main file. Templates are found before copies to the same dest. remote_src is read the way Ansible's boolean() reads it (y / yes / on / 1 / true / t, any case): `remote_src: true` is a YAML boolean, not a string, and a string-only read had kept every remote copy as a renderer. The lists in a role's defaults / vars reach past the task file, so a change to one takes the full rebuild (closure_ansible_decline); without it, a list that names another template kept the edge to the old one. A file holding Ansible tasks that the repair would re-walk as a dependent -- a template it applies was deleted -- also takes the full rebuild (closure_ansible_dependent_decline): the repair walks it without its role's defaults, handlers and other task files, and drew none of its list-fed edges. Measured on 17 infrastructure repositories: 164 of 164 applied files (52 of kubespray's 62 kube / k8s tasks, 158 templates) against an independent reading, all in kubespray; nothing in the other 16; the rest of the graph unchanged (+164 DEPLOYS only); deterministic over three runs; the incremental re-index equals a full index for a template body edit, a deleted template (with and without a same-named fallback beside the task file), a task list edit, a copied manifest edit, a defaults list edit and a deleted vars file. Of the 10 tasks left out, 4 act on an object by name, one builds its definition with a template lookup, one applies a remote_src copy, and 4 name their file through a filter, a variable or two joined results lists. Signed-off-by: Martin Vogel --- README.md | 2 +- src/pipeline/pass_ansible.c | 671 +++++++++++++++++++++++++++- src/pipeline/pipeline_incremental.c | 92 +++- src/pipeline/pipeline_internal.h | 8 +- tests/test_grammar_probe_f.c | 112 +++++ tests/test_pipeline.c | 90 ++++ 6 files changed, 951 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index 128642d40..4b09dcb04 100644 --- a/README.md +++ b/README.md @@ -341,7 +341,7 @@ The install script placed beside the binary is **reported, not deleted**: uninst - `USAGE` with `via: terraform`: a Terraform / OpenTofu reference (`var.X`, `local.X`, `module.X`, `data.T.N`, `T.N`) from the top-level block holding it to the block of its module (the `.tf` / `.tofu` files of its directory) that declares it (a `locals` entry: its `locals` block); an entry of the block's `depends_on` list is a `DEPENDS_ON` instead. The edge keeps `ref` (`var`, `local`, `module`, `data`, `resource`) and `line`; a name only mentioned in string text is no reference. A module call with a local source (`module "net" { source = "./modules/net" }`) `IMPORTS` (`local_name`: the module's name) the `Folder` of that directory (the root directory: the `Project`), and `module.net.id` (also `module.net[0].id`, `module.net[*].id`) `USAGE`s (`ref: output`) the `output "id"` there, beside `module.net` - `OVERRIDE` with `via: tfvars`: a `.tfvars` assignment to a variable the module of its own directory declares, from the file's `Variable` to that `variable` block (the edge keeps `line`); a `.tfvars` file whose own directory declares none of its names sets the nearest ancestor module's that does, the edge marked `inferred` (such a file is applied by a `-var-file` the repository does not state) - Terragrunt (`via: terragrunt`): a unit (`terragrunt.hcl`), evaluated in its own context: an `include` `IMPORTS` (`local_name`) the file its `path` names; a `read_terragrunt_config` `IMPORTS` (`read`: the file holding the call) the file it names, from the block holding the call or, for a call in an included file, from the unit's `include` block that brings it in; a `dependency` `DEPENDS_ON` the other unit; a local `terraform` `source` `IMPORTS` its `Folder`. Paths may use `find_in_parent_folders`, `dirname` and `get_terragrunt_dir`; anything else is unknown and links nothing -- Ansible (`via: ansible`): a playbook or task file (a YAML file whose top level is a sequence) `IMPORTS` (`local_name`: the directive) what its directives name: `import_playbook` a playbook, `include_tasks` / `import_tasks` / `include` a task file (from its directory, else its role's `tasks/`), `roles:` items and `include_role` / `import_role` a role's `Folder` (`roles/` at or above its directory). A templated target links nothing. Each task and handler is a `Function` (named by its `name:`, else its module; `source: ansible`, `kind`, `module`) that its file `DEFINES`: in a playbook's plays, a role's `tasks/` and `handlers/` files, and the files those include; a block's tasks, not the block. A `notify` `CALLS` (`via: notify`) each handler of that name or `listen` topic in the task's own role (else its file) +- Ansible (`via: ansible`): a playbook or task file (a YAML file whose top level is a sequence) `IMPORTS` (`local_name`: the directive) what its directives name: `import_playbook` a playbook, `include_tasks` / `import_tasks` / `include` a task file (from its directory, else its role's `tasks/`), `roles:` items and `include_role` / `import_role` a role's `Folder` (`roles/` at or above its directory). A templated target links nothing. Each task and handler is a `Function` (named by its `name:`, else its module; `source: ansible`, `kind`, `module`) that its file `DEFINES`: in a playbook's plays, a role's `tasks/` and `handlers/` files, and the files those include; a block's tasks, not the block. A `notify` `CALLS` (`via: notify`) each handler of that name or `listen` topic in the task's own role (else its file). A `kube` (kubespray's module) or `kubernetes.core.k8s` task `DEPLOYS` (`via: ansible`, `module`) the repository file its play rendered the applied manifest from: the `template` (in the role's `templates/`) or `copy` (its `files/`) task whose `dest` is the task's `filename`, item by item through `register` / `with_items` / `loop` lists the play states (a literal list, a `set_fact`, the role's `defaults` / `vars`), or a `k8s` `template:` argument - `EMITS`, `LISTENS_ON` (channels) - `DATA_FLOWS` with arg-to-param mapping + field access chains - `SIMILAR_TO` (MinHash + LSH near-clone detection, Jaccard scored) diff --git a/src/pipeline/pass_ansible.c b/src/pipeline/pass_ansible.c index 4c1dff69d..6ccb2f7f4 100644 --- a/src/pipeline/pass_ansible.c +++ b/src/pipeline/pass_ansible.c @@ -34,6 +34,28 @@ * directory before its tasks/), else in its file (a play's handlers). The * incremental repair declines on any change to Ansible content * (closure_ansible_decline): a notify joins two files. + * + * PHASE 7 P7.3: applies. A kube (kubespray's library module) or k8s + * (kubernetes.core / community.kubernetes) task applies a manifest the play + * rendered: it DEPLOYS {via: ansible, module, line} the repository file the + * render read -- a template task's src (the role's templates/) or a copy + * task's (its files/; never remote_src), else the task file's directory: + * a k8s template: argument that template; + * filename == a renderer's dest (D) that renderer's src; + * {{ item.item.K }} over "{{ R.results }}" (L) + * the src of the renderer registering + * R, for each item of its loop; + * {{ item.K }} (S) for each of the task's own items, the + * src of the renderer of that dest; + * a literal filename (R) the src of each looped renderer item + * whose dest renders to it. + * A loop is a literal list, or "{{ V }}": a set_fact of the file, else the + * role's defaults/ or vars/ main.yml. Templates are searched before copies, + * each in file order. The incremental repair declines on a change to a + * role's defaults / vars main file (closure_ansible_decline), and when a + * task file would join it as a dependent -- a deleted template it applies -- + * since the repair would walk it without its role's other files + * (closure_ansible_dependent_decline). */ #include "foundation/constants.h" #include "foundation/compat.h" @@ -88,6 +110,20 @@ typedef struct { int note_cap; int tasks; int calls; + CBMHashTable *file_nodes; /* P7.3: File nodes by path (what a task applies) */ + CBMHashTable *by_rel; /* every input file's path -> its index + 1 */ + const cbm_file_info_t *files; + TSParser *parser; /* reads a role's defaults / vars */ + struct an_render *renders; /* per file: template / copy tasks */ + int render_n; + int render_cap; + struct an_apply *applies; /* per file: kube / k8s tasks */ + int apply_n; + int apply_cap; + struct an_fact *facts; /* per file: set_fact lists */ + int fact_n; + int fact_cap; + int deploys; } an_t; static bool an_type(TSNode n, const char *t) { @@ -143,6 +179,38 @@ static bool an_scalar(const an_t *a, TSNode n, char *out, size_t sz) { return out[0] != '\0'; } +/* Is `n` an Ansible boolean that holds true -- y / yes / on / 1 / true / t, + * any case (Ansible's boolean()), a YAML boolean or a string? an_scalar + * reads strings only: `remote_src: true` is a boolean_scalar. */ +static bool an_true(const an_t *a, TSNode n) { + static const char *const TRUE_WORDS[] = {"y", "yes", "on", "1", "true", "t"}; + while ((an_type(n, "flow_node") || an_type(n, "block_node") || an_type(n, "plain_scalar")) && + ts_node_named_child_count(n) == SKIP_ONE) { + n = ts_node_named_child(n, 0); + } + char v[CBM_SZ_16]; + if (an_type(n, "boolean_scalar") || an_type(n, "integer_scalar")) { + uint32_t b = ts_node_start_byte(n); + uint32_t e = ts_node_end_byte(n); + if (e - b + SKIP_ONE > sizeof(v)) { + return false; + } + memcpy(v, a->src + b, e - b); + v[e - b] = '\0'; + } else if (!an_scalar(a, n, v, sizeof(v))) { + return false; + } + for (char *c = v; *c; c++) { + *c = (char)(*c >= 'A' && *c <= 'Z' ? *c - 'A' + 'a' : *c); + } + for (size_t i = 0; i < sizeof(TRUE_WORDS) / sizeof(TRUE_WORDS[0]); i++) { + if (strcmp(v, TRUE_WORDS[i]) == 0) { + return true; + } + } + return false; +} + /* The key of a mapping pair, its text, the ansible.builtin. prefix dropped. */ static bool an_key(const an_t *a, TSNode pair, char *out, size_t sz) { static const char BUILTIN[] = "ansible.builtin."; @@ -586,6 +654,586 @@ static bool an_name_at(const an_t *a, TSNode v, TSNode list, uint32_t i, char *o return an_scalar(a, item, out, sz); } +/* ── Applies (P7.3) ───────────────────────────────────────────────── */ + +/* One item of a loop list: a scalar, or a mapping's scalar pairs. */ +typedef struct { + char *scalar; + char **keys; + char **vals; + int n; +} an_item_t; + +/* A loop (with_items / loop): its literal items, or its expression + * ("{{ X }}", "{{ R.results }}"). */ +typedef struct { + an_item_t *items; + int n; + char *expr; +} an_loop_t; + +/* A task rendering a file: template (src in the role's templates/) or copy + * of a repository file (its files/). */ +typedef struct an_render { + char *src; + char *dest; + char *reg; + an_loop_t loop; + bool copy; +} an_render_t; + +/* A task applying a manifest: kube / k8s, its filename (src) or template + * argument, its loop. */ +typedef struct an_apply { + int64_t task; + int line; + char *module; + char *fname; + char *tmpl; + an_loop_t loop; +} an_apply_t; + +/* A list a set_fact of the file names. */ +typedef struct an_fact { + char *name; + an_loop_t list; +} an_fact_t; + +static const char *const AN_APPLIERS[] = {"kube", "k8s", "kubernetes.core.k8s", + "community.kubernetes.k8s"}; + +static void *an_grow(void *items, int *cap, int count, size_t elem) { + if (count < *cap) { + return items; + } + int ncap = *cap ? *cap * PAIR_LEN : AN_GROW_MIN; + void *grown = cbm_realloc(CBM_MEM_CLASS_OTHER, items, (size_t)ncap * elem); + if (grown) { + *cap = ncap; + } + return grown; +} + +static char *an_dup(const char *s) { + return s ? cbm_mem_strdup(CBM_MEM_CLASS_OTHER, s) : NULL; +} + +/* A value node without its block_node / flow_node wrappers. */ +static TSNode an_bare(TSNode v) { + while ((an_type(v, "block_node") || an_type(v, "flow_node")) && + ts_node_named_child_count(v) == SKIP_ONE) { + v = ts_node_named_child(v, 0); + } + return v; +} + +/* A loop item: a scalar, or the scalar pairs of a block / flow mapping. */ +static void an_item_parse(const an_t *a, TSNode node, an_item_t *it) { + char buf[CBM_SZ_1K]; + *it = (an_item_t){0}; + if (an_scalar(a, node, buf, sizeof(buf))) { + it->scalar = an_dup(buf); + return; + } + TSNode m = an_bare(node); + if (!an_type(m, "block_mapping") && !an_type(m, "flow_mapping")) { + return; + } + int cap = 0; + int vcap = 0; + for (uint32_t i = 0; i < ts_node_named_child_count(m); i++) { + TSNode pair = ts_node_named_child(m, i); + char k[CBM_SZ_128]; + if (!an_type(pair, "block_mapping_pair") && !an_type(pair, "flow_pair")) { + continue; + } + if (!an_scalar(a, ts_node_child_by_field_name(pair, "key", sizeof("key") - SKIP_ONE), k, + sizeof(k)) || + !an_scalar(a, ts_node_child_by_field_name(pair, "value", sizeof("value") - SKIP_ONE), + buf, sizeof(buf))) { + continue; + } + char **keys = an_grow(it->keys, &cap, it->n, sizeof(char *)); + if (keys) { + it->keys = keys; + } + char **vals = keys ? an_grow(it->vals, &vcap, it->n, sizeof(char *)) : NULL; + if (!keys || !vals) { + return; + } + it->vals = vals; + it->keys[it->n] = an_dup(k); + it->vals[it->n++] = an_dup(buf); + } +} + +/* A loop value: an expression (a scalar holding {{ }}, or a list of exactly + * one -- with_items flattens one level), else a literal list of items. */ +static void an_loop_parse(const an_t *a, TSNode v, an_loop_t *out) { + char buf[CBM_SZ_1K]; + *out = (an_loop_t){0}; + if (an_scalar(a, v, buf, sizeof(buf))) { + out->expr = strstr(buf, "{{") ? an_dup(buf) : NULL; + return; + } + TSNode s = an_bare(v); + bool block = an_type(s, "block_sequence"); + if (!block && !an_type(s, "flow_sequence")) { + return; + } + int cap = 0; + for (uint32_t i = 0; i < ts_node_named_child_count(s); i++) { + TSNode item = ts_node_named_child(s, i); + if (block ? !an_type(item, "block_sequence_item") : !an_type(item, "flow_node")) { + continue; + } + if (block) { + item = ts_node_named_child(item, 0); + } + an_item_t *grown = an_grow(out->items, &cap, out->n, sizeof(an_item_t)); + if (!grown) { + return; + } + out->items = grown; + an_item_parse(a, item, &out->items[out->n++]); + } + if (out->n == SKIP_ONE && out->items[0].scalar && strstr(out->items[0].scalar, "{{")) { + out->expr = out->items[0].scalar; + out->items[0].scalar = NULL; + cbm_free(CBM_MEM_CLASS_OTHER, out->items); + out->items = NULL; + out->n = 0; + } +} + +static void an_loop_free(an_loop_t *l) { + for (int i = 0; i < l->n; i++) { + an_item_t *it = &l->items[i]; + for (int k = 0; k < it->n; k++) { + cbm_free(CBM_MEM_CLASS_OTHER, it->keys[k]); + cbm_free(CBM_MEM_CLASS_OTHER, it->vals[k]); + } + cbm_free(CBM_MEM_CLASS_OTHER, it->keys); + cbm_free(CBM_MEM_CLASS_OTHER, it->vals); + cbm_free(CBM_MEM_CLASS_OTHER, it->scalar); + } + cbm_free(CBM_MEM_CLASS_OTHER, l->items); + cbm_free(CBM_MEM_CLASS_OTHER, l->expr); + *l = (an_loop_t){0}; +} + +/* The task's loop: with_items, else loop. */ +static void an_task_loop(const an_t *a, TSNode m, an_loop_t *out) { + TSNode v = an_get(a, m, "with_items"); + an_loop_parse(a, ts_node_is_null(v) ? an_get(a, m, "loop") : v, out); +} + +/* What task `m` (Function `id`) renders, applies or sets, for an_link_applies. */ +static void an_collect(an_t *a, TSNode m, int64_t id) { + char k[CBM_SZ_128]; + char v[CBM_SZ_1K]; + for (uint32_t i = 0; i < ts_node_named_child_count(m); i++) { + TSNode pair = ts_node_named_child(m, i); + if (!an_key(a, pair, k, sizeof(k))) { + continue; + } + TSNode value = ts_node_child_by_field_name(pair, "value", sizeof("value") - SKIP_ONE); + bool copy = strcmp(k, "copy") == 0; + if (strcmp(k, "template") == 0 || copy) { + TSNode args = an_bare(value); + if (!an_type(args, "block_mapping") && !an_type(args, "flow_mapping")) { + continue; + } + if (copy && an_true(a, an_get(a, value, "remote_src"))) { + continue; + } + an_render_t *r = an_grow(a->renders, &a->render_cap, a->render_n, sizeof(*r)); + if (!r) { + continue; + } + a->renders = r; + r = &a->renders[a->render_n++]; + *r = (an_render_t){.copy = copy}; + r->src = an_scalar(a, an_get(a, value, "src"), v, sizeof(v)) ? an_dup(v) : NULL; + r->dest = an_scalar(a, an_get(a, value, "dest"), v, sizeof(v)) ? an_dup(v) : NULL; + r->reg = an_scalar(a, an_get(a, m, "register"), v, sizeof(v)) ? an_dup(v) : NULL; + an_task_loop(a, m, &r->loop); + } else if (strcmp(k, "set_fact") == 0) { + TSNode f = an_bare(value); + for (uint32_t j = 0; an_type(f, "block_mapping") && j < ts_node_named_child_count(f); + j++) { + TSNode fp = ts_node_named_child(f, j); + char name[CBM_SZ_128]; + TSNode fv = ts_node_child_by_field_name(fp, "value", sizeof("value") - SKIP_ONE); + TSNode seq = an_bare(fv); + an_fact_t *g = + (an_type(fp, "block_mapping_pair") && an_key(a, fp, name, sizeof(name)) && + (an_type(seq, "block_sequence") || an_type(seq, "flow_sequence"))) + ? an_grow(a->facts, &a->fact_cap, a->fact_n, sizeof(*g)) + : NULL; + if (g) { + a->facts = g; + a->facts[a->fact_n].name = an_dup(name); + an_loop_parse(a, fv, &a->facts[a->fact_n++].list); + } + } + } else { + for (size_t j = 0; j < sizeof(AN_APPLIERS) / sizeof(AN_APPLIERS[0]); j++) { + if (strcmp(k, AN_APPLIERS[j]) != 0) { + continue; + } + an_apply_t *p = an_grow(a->applies, &a->apply_cap, a->apply_n, sizeof(*p)); + if (!p) { + break; + } + a->applies = p; + p = &a->applies[a->apply_n++]; + *p = (an_apply_t){.task = id, + .line = (int)ts_node_start_point(m).row + SKIP_ONE, + .module = an_dup(k)}; + if (an_scalar(a, an_get(a, value, "template"), v, sizeof(v))) { + p->tmpl = an_dup(v); + } else if (an_scalar(a, an_get(a, value, "filename"), v, sizeof(v)) || + an_scalar(a, an_get(a, value, "src"), v, sizeof(v))) { + p->fname = an_dup(v); + } + an_task_loop(a, m, &p->loop); + break; + } + } + } +} + +/* The trimmed inside of a {{ }}, `n` bytes at `b`: "item" (key length 0) or + * "item.K", K an identifier (into key / kl); false for anything else. */ +static bool an_item_ref(const char *b, size_t n, const char **key, size_t *kl) { + static const char ITEM[] = "item"; + size_t il = sizeof(ITEM) - SKIP_ONE; + if (n < il || strncmp(b, ITEM, il) != 0) { + return false; + } + *key = b + il; + *kl = 0; + if (n == il) { + return true; + } + if (b[il] != '.' || n == il + SKIP_ONE) { + return false; + } + *key = b + il + SKIP_ONE; + *kl = n - il - SKIP_ONE; + for (size_t i = 0; i < *kl; i++) { + char c = (*key)[i]; + if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c == '_' || + (i > 0 && c >= '0' && c <= '9'))) { + return false; + } + } + return true; +} + +/* `pattern` with each {{ item }} / {{ item.K }} replaced from `it`; false + * when one cannot be, or -- `strict` -- when any other {{ }} is left. */ +static bool an_subst(const char *pattern, const an_item_t *it, bool strict, char *out, size_t sz) { + size_t w = 0; + for (const char *p = pattern; p && *p;) { + const char *end = p[0] == '{' && p[SKIP_ONE] == '{' ? strstr(p + PAIR_LEN, "}}") : NULL; + const char *piece = p; + size_t len = SKIP_ONE; + if (end) { + const char *b = p + PAIR_LEN; + while (*b == ' ' || *b == '\t') { + b++; + } + size_t n = (size_t)(end - b); /* b stops at end's '}' at the latest */ + while (n > 0 && (b[n - SKIP_ONE] == ' ' || b[n - SKIP_ONE] == '\t')) { + n--; + } + const char *key = NULL; + size_t kl = 0; + if (an_item_ref(b, n, &key, &kl)) { + piece = kl == 0 ? it->scalar : NULL; + for (int k = 0; kl > 0 && k < it->n && !piece; k++) { + if (strlen(it->keys[k]) == kl && strncmp(it->keys[k], key, kl) == 0) { + piece = it->vals[k]; + } + } + if (!piece) { + return false; + } + len = strlen(piece); + } else if (strict) { + return false; + } else { + len = (size_t)(end + PAIR_LEN - p); + } + p = end + PAIR_LEN; + } else { + p++; + } + if (w + len >= sz) { + return false; + } + memcpy(out + w, piece, len); + w += len; + } + out[w] = '\0'; + return true; +} + +/* The repository file a template / copy `src` names: in the role's + * templates/ (files/ for a copy), else the task file's directory. */ +static const cbm_gbuf_node_t *an_src_file(const an_t *a, const char *src, bool copy) { + char base[CBM_SZ_1K]; + char path[CBM_SZ_1K]; + if (!src || strstr(src, "{{")) { + return NULL; + } + if (a->role_dir[0]) { + snprintf(base, sizeof(base), "%s/%s", a->role_dir, copy ? "files" : "templates"); + const cbm_gbuf_node_t *n = cbm_repo_rel_join(base, src, path, sizeof(path)) + ? cbm_ht_get(a->file_nodes, path) + : NULL; + if (n) { + return n; + } + } + return cbm_repo_rel_join(a->dir, src, path, sizeof(path)) ? cbm_ht_get(a->file_nodes, path) + : NULL; +} + +/* A top-level list `name` of the role's defaults/ or vars/ main.yml, into + * `out`; false when none holds one. */ +static bool an_role_list(an_t *a, const char *name, an_loop_t *out) { + static const char *const FILES[] = {"defaults/main.yml", "defaults/main.yaml", "vars/main.yml", + "vars/main.yaml"}; + bool found = false; + for (size_t f = 0; a->role_dir[0] && !found && f < sizeof(FILES) / sizeof(FILES[0]); f++) { + char rel[CBM_SZ_1K]; + snprintf(rel, sizeof(rel), "%s/%s", a->role_dir, FILES[f]); + intptr_t ix = (intptr_t)cbm_ht_get(a->by_rel, rel); + size_t len = 0; + char *src = ix > 0 ? an_read(a->files[ix - SKIP_ONE].path, &len) : NULL; + TSTree *tree = src ? ts_parser_parse_string(a->parser, NULL, src, (uint32_t)len) : NULL; + TSNode top = + tree ? an_bare(an_child(an_child(ts_tree_root_node(tree), "document"), "block_node")) + : (TSNode){0}; + const char *saved = a->src; + a->src = src; + for (uint32_t i = 0; + an_type(top, "block_mapping") && !found && i < ts_node_named_child_count(top); i++) { + TSNode pair = ts_node_named_child(top, i); + char k[CBM_SZ_128]; + TSNode v = ts_node_child_by_field_name(pair, "value", sizeof("value") - SKIP_ONE); + TSNode seq = an_bare(v); + if (an_key(a, pair, k, sizeof(k)) && strcmp(k, name) == 0 && + (an_type(seq, "block_sequence") || an_type(seq, "flow_sequence"))) { + an_loop_parse(a, v, out); + found = true; + } + } + a->src = saved; + if (tree) { + ts_tree_delete(tree); + } + cbm_free(CBM_MEM_CLASS_OTHER, src); + } + return found; +} + +/* An expression "{{ X }}" with X an identifier, then `tail` ("" or + * ".results"): X into `out`. */ +static bool an_expr_name(const char *expr, const char *tail, char *out, size_t sz) { + const char *b = expr && expr[0] == '{' && expr[SKIP_ONE] == '{' ? expr + PAIR_LEN : NULL; + size_t n = 0; + while (b && (*b == ' ' || *b == '\t')) { + b++; + } + while (b && n + SKIP_ONE < sz && + ((*b >= 'a' && *b <= 'z') || (*b >= 'A' && *b <= 'Z') || *b == '_' || + (n > 0 && *b >= '0' && *b <= '9'))) { + out[n++] = *b++; + } + out[n] = '\0'; + size_t tl = strlen(tail); + if (!n || !b || strncmp(b, tail, tl) != 0) { + return false; + } + for (b += tl; *b == ' ' || *b == '\t'; b++) {} + return strcmp(b, "}}") == 0; +} + +/* "{{ R.results }}": R into `out`. */ +static bool an_results_of(const char *expr, char *out, size_t sz) { + return an_expr_name(expr, ".results", out, sz); +} + +/* Does `s` hold a {{ item.K }} reference (one key level)? */ +static bool an_has_item_key(const char *s) { + for (const char *p = s ? strstr(s, "{{") : NULL; p; p = strstr(p + PAIR_LEN, "{{")) { + const char *b = p + PAIR_LEN; + while (*b == ' ' || *b == '\t') { + b++; + } + if (strncmp(b, "item.", sizeof("item.") - SKIP_ONE) != 0) { + continue; + } + b += sizeof("item.") - SKIP_ONE; + const char *k = b; + while ((*b >= 'a' && *b <= 'z') || (*b >= 'A' && *b <= 'Z') || *b == '_') { + b++; + } + while (b > k && (*b == ' ' || *b == '\t')) { + b++; + } + if (b > k && b[0] == '}' && b[SKIP_ONE] == '}') { + return true; + } + } + return false; +} + +/* The items a loop goes over: its literal list, or the list "{{ V }}" names + * -- a set_fact of the file, else the role's defaults / vars. `scratch` + * holds a list read from the role (the caller frees it). */ +static const an_loop_t *an_items(an_t *a, const an_loop_t *l, an_loop_t *scratch) { + if (!l->expr) { + return l->items ? l : NULL; + } + char name[CBM_SZ_128]; + if (!an_expr_name(l->expr, "", name, sizeof(name))) { + return NULL; + } + for (int i = 0; i < a->fact_n; i++) { + if (strcmp(a->facts[i].name, name) == 0) { + return &a->facts[i].list; + } + } + return an_role_list(a, name, scratch) ? scratch : NULL; +} + +static void an_deploy(an_t *a, const an_apply_t *p, const cbm_gbuf_node_t *target) { + if (!target || p->task <= 0) { + return; + } + char em[CBM_SZ_256]; + char props[CBM_SZ_512]; + cbm_json_escape(em, (int)sizeof(em), p->module); + snprintf(props, sizeof(props), "{\"via\":\"ansible\",\"module\":\"%s\",\"line\":%d}", em, + p->line); + cbm_gbuf_insert_edge(a->ctx->gbuf, p->task, target->id, "DEPLOYS", props); + a->deploys++; +} + +/* The renderer `src` of each item of `items`, substituted (strict). */ +static int an_deploy_items(an_t *a, const an_apply_t *p, const an_render_t *r, + const an_loop_t *items) { + int made = 0; + char path[CBM_SZ_1K]; + for (int i = 0; r && items && i < items->n; i++) { + const cbm_gbuf_node_t *t = + r->src && an_subst(r->src, &items->items[i], true, path, sizeof(path)) + ? an_src_file(a, path, r->copy) + : NULL; + if (t) { + an_deploy(a, p, t); + made++; + } + } + return made; +} + +/* The first renderer (templates before copies, each in file order) whose + * `dest` -- or, with `reg`, whose register -- is the very string. */ +static const an_render_t *an_render_find(const an_t *a, const char *dest, const char *reg) { + for (int pass = 0; pass < PAIR_LEN; pass++) { + for (int i = 0; i < a->render_n; i++) { + const an_render_t *r = &a->renders[i]; + const char *v = reg ? r->reg : r->dest; + if (r->copy == (pass == SKIP_ONE) && v && strcmp(v, reg ? reg : dest) == 0) { + return r; + } + } + } + return NULL; +} + +/* kube / k8s -> DEPLOYS {via: ansible, module, line} the repository file it + * applies, traced to the template (or copied file) the play rendered it from: + * a k8s template: argument; a renderer whose dest is the filename (D); for + * {{ item.item.K }} over "{{ R.results }}", each item of the renderer + * registering R (L); for {{ item.K }}, each of the task's own items through + * the renderer of that dest (S); a literal filename a looped renderer's dest + * renders to (R). */ +static void an_link_applies(an_t *a) { + for (int i = 0; i < a->apply_n; i++) { + const an_apply_t *p = &a->applies[i]; + an_loop_t scratch = {0}; + int made = 0; + if (p->tmpl) { + an_deploy(a, p, an_src_file(a, p->tmpl, false)); + continue; + } + if (!p->fname) { + continue; + } + const an_render_t *r = an_render_find(a, p->fname, NULL); + const cbm_gbuf_node_t *d = r ? an_src_file(a, r->src, r->copy) : NULL; + if (d) { + an_deploy(a, p, d); + continue; + } + char reg[CBM_SZ_128]; + if (strstr(p->fname, "item.item") && an_results_of(p->loop.expr, reg, sizeof(reg))) { + const an_render_t *g = an_render_find(a, NULL, reg); + made = g ? an_deploy_items(a, p, g, an_items(a, &g->loop, &scratch)) : 0; + } + if (!made && an_has_item_key(p->fname) && (p->loop.expr || p->loop.items)) { + an_loop_free(&scratch); + made = an_deploy_items(a, p, r, an_items(a, &p->loop, &scratch)); + } + if (!made && !strstr(p->fname, "{{ item")) { + char path[CBM_SZ_1K]; + for (int j = 0; j < a->render_n; j++) { + const an_render_t *x = &a->renders[j]; + an_loop_free(&scratch); + const an_loop_t *items = an_items(a, &x->loop, &scratch); + for (int k = 0; items && x->dest && k < items->n; k++) { + if (an_subst(x->dest, &items->items[k], false, path, sizeof(path)) && + strcmp(path, p->fname) == 0 && x->src && + an_subst(x->src, &items->items[k], true, path, sizeof(path))) { + an_deploy(a, p, an_src_file(a, path, x->copy)); + } + } + } + } + an_loop_free(&scratch); + } +} + +/* The per-file render / apply / set_fact records, freed. */ +static void an_applies_reset(an_t *a) { + for (int i = 0; i < a->render_n; i++) { + cbm_free(CBM_MEM_CLASS_OTHER, a->renders[i].src); + cbm_free(CBM_MEM_CLASS_OTHER, a->renders[i].dest); + cbm_free(CBM_MEM_CLASS_OTHER, a->renders[i].reg); + an_loop_free(&a->renders[i].loop); + } + for (int i = 0; i < a->apply_n; i++) { + cbm_free(CBM_MEM_CLASS_OTHER, a->applies[i].module); + cbm_free(CBM_MEM_CLASS_OTHER, a->applies[i].fname); + cbm_free(CBM_MEM_CLASS_OTHER, a->applies[i].tmpl); + an_loop_free(&a->applies[i].loop); + } + for (int i = 0; i < a->fact_n; i++) { + cbm_free(CBM_MEM_CLASS_OTHER, a->facts[i].name); + an_loop_free(&a->facts[i].list); + } + a->render_n = 0; + a->apply_n = 0; + a->fact_n = 0; +} + static void an_index_handler(an_t *a, const char *name, int64_t id) { char key[CBM_SZ_2K]; if (a->role_dir[0]) { @@ -654,6 +1302,7 @@ static void an_task(an_t *a, TSNode m, bool handler) { } cbm_gbuf_insert_edge(a->ctx->gbuf, a->from, id, "DEFINES", "{}"); a->tasks++; + an_collect(a, m, id); TSNode list; char n[CBM_SZ_256]; if (handler) { @@ -926,6 +1575,13 @@ static void an_free(an_t *a) { cbm_ht_foreach(a->handlers, an_free_ids_entry, NULL); cbm_ht_free(a->handlers); } + an_applies_reset(a); + cbm_free(CBM_MEM_CLASS_OTHER, a->renders); + cbm_free(CBM_MEM_CLASS_OTHER, a->applies); + cbm_free(CBM_MEM_CLASS_OTHER, a->facts); + if (a->by_rel) { + cbm_ht_free(a->by_rel); + } } /* Parse `files[i]` when it is a YAML file whose top level is a sequence; @@ -974,9 +1630,16 @@ void cbm_pipeline_pass_ansible(cbm_pipeline_ctx_t *ctx, const cbm_file_info_t *f } cbm_free(CBM_MEM_CLASS_OTHER, src); } - /* pass 2: the tasks of the Ansible files, then their notifies */ + /* pass 2: the tasks of the Ansible files (what each applies), then their notifies */ an_mark_ansible(&a, &cands, files); a.handlers = cands.count > 0 ? cbm_ht_create(0) : NULL; + a.by_rel = a.handlers ? cbm_ht_create(0) : NULL; + for (int i = 0; a.by_rel && i < file_count; i++) { + cbm_ht_set(a.by_rel, files[i].rel_path, (void *)(intptr_t)(i + SKIP_ONE)); + } + a.files = files; + a.file_nodes = file_nodes; + a.parser = parser; for (int c = 0; a.handlers && c < cands.count; c++) { const cbm_file_info_t *f = &files[cands.items[c].file]; char *src = NULL; @@ -985,6 +1648,8 @@ void cbm_pipeline_pass_ansible(cbm_pipeline_ctx_t *ctx, const cbm_file_info_t *f if (fnode) { an_begin_file(&a, f->rel_path, fnode, src); an_file_tasks(&a, an_top(ts_tree_root_node(tree)), cands.items[c].playbook); + an_link_applies(&a); + an_applies_reset(&a); } if (tree) { ts_tree_delete(tree); @@ -1004,9 +1669,11 @@ void cbm_pipeline_pass_ansible(cbm_pipeline_ctx_t *ctx, const cbm_file_info_t *f char e[CBM_SZ_32]; char t[CBM_SZ_32]; char n[CBM_SZ_32]; + char d[CBM_SZ_32]; snprintf(e, sizeof(e), "%d", a.edges); snprintf(t, sizeof(t), "%d", a.tasks); snprintf(n, sizeof(n), "%d", a.calls); - cbm_log_info("pass.ansible", "imports", e, "tasks", t, "notifies", n); + snprintf(d, sizeof(d), "%d", a.deploys); + cbm_log_info("pass.ansible", "imports", e, "tasks", t, "notifies", n, "deploys", d); } } diff --git a/src/pipeline/pipeline_incremental.c b/src/pipeline/pipeline_incremental.c index 3098210ba..15add39b6 100644 --- a/src/pipeline/pipeline_incremental.c +++ b/src/pipeline/pipeline_incremental.c @@ -2539,31 +2539,65 @@ static const char *closure_terraform_decline(char **paths, int n) { return NULL; } +/* A role's defaults/ or vars/ main.yml (.yaml): the lists a kube task's loop + * reads (pass_ansible.c, an_role_list) -- a roles/ component before. */ +static bool closure_ansible_role_vars(const char *rel) { + static const char *const TAILS[] = {"/defaults/main.yml", "/defaults/main.yaml", + "/vars/main.yml", "/vars/main.yaml"}; + static const char ROLES[] = "roles/"; + size_t n = strlen(rel); + for (size_t k = 0; k < sizeof(TAILS) / sizeof(TAILS[0]); k++) { + size_t tl = strlen(TAILS[k]); + if (n <= tl || strcmp(rel + n - tl, TAILS[k]) != 0) { + continue; + } + const char *end = rel + n - tl; /* the role directory: rel up to end */ + for (const char *p = rel; (p = strstr(p, ROLES)) != NULL && p < end; p++) { + if ((p == rel || p[-SKIP_ONE] == '/') && p + sizeof(ROLES) - SKIP_ONE < end) { + return true; + } + } + } + return false; +} + +/* Does `rel` hold Ansible tasks (a Function {source: ansible})? -1 on a + * lookup error. */ +static int closure_holds_ansible_tasks(cbm_store_t *store, const char *project, const char *rel) { + cbm_node_t *nodes = NULL; + int count = 0; + if (cbm_store_find_nodes_by_file(store, project, rel, &nodes, &count) != CBM_STORE_OK) { + return -SKIP_ONE; + } + int held = 0; + for (int j = 0; j < count && !held; j++) { + held = nodes[j].label && strcmp(nodes[j].label, "Function") == 0 && + nodes[j].properties_json && + strstr(nodes[j].properties_json, "\"source\":\"ansible\"") != NULL; + } + cbm_store_free_nodes(nodes, count); + return held; +} + /* Ansible (pass_ansible.c): a task's notify CALLS the handlers of its role - * in another file, and whether a file is Ansible at all can hang on another - * file's include. The repair recomputes the changed files only, so a - * changed or deleted file that held Ansible tasks, or a changed YAML file - * whose top level is a sequence (it may be Ansible now), declines to the - * full rebuild. Returns the reason, or NULL. */ + * in another file, whether a file is Ansible at all can hang on another + * file's include, and what a kube task DEPLOYS on the lists of its role's + * defaults / vars. The repair recomputes the changed files only, so a + * changed or deleted file that held Ansible tasks, a changed YAML file whose + * top level is a sequence (it may be Ansible now), or a changed or deleted + * role defaults / vars main file declines to the full rebuild. Returns the + * reason, or NULL. */ static const char *closure_ansible_decline(cbm_store_t *store, const char *project, const char *repo_path, char **paths, int n_changed, int n_deleted) { for (int i = 0; i < n_changed + n_deleted; i++) { const char *rel = paths[i]; - cbm_node_t *nodes = NULL; - int count = 0; - if (cbm_store_find_nodes_by_file(store, project, rel, &nodes, &count) != CBM_STORE_OK) { - return "ansible_lookup_failed"; + if (closure_ansible_role_vars(rel)) { + return "ansible_role_vars_changed"; } - bool held = false; - for (int j = 0; j < count && !held; j++) { - held = nodes[j].label && strcmp(nodes[j].label, "Function") == 0 && - nodes[j].properties_json && - strstr(nodes[j].properties_json, "\"source\":\"ansible\""); - } - cbm_store_free_nodes(nodes, count); - if (held) { - return "ansible_tasks_changed"; + int held = closure_holds_ansible_tasks(store, project, rel); + if (held != 0) { + return held < 0 ? "ansible_lookup_failed" : "ansible_tasks_changed"; } const char *dot = strrchr(rel, '.'); if (i >= n_changed || !dot || (strcmp(dot, ".yaml") != 0 && strcmp(dot, ".yml") != 0)) { @@ -2595,6 +2629,24 @@ static const char *closure_ansible_decline(cbm_store_t *store, const char *proje return NULL; } +/* A file holding Ansible tasks that the repair would re-walk as a dependent + * (an edge of it reaches a changed or deleted file: a deleted template its + * task DEPLOYS) is walked without the files its edges are drawn from -- + * its role's defaults / vars, handlers, included task files -- which are + * outside the closure: it would lose those edges, or keep a dropped one's + * fallback undrawn. Declines to the full rebuild. Returns the reason, or + * NULL. */ +static const char *closure_ansible_dependent_decline(cbm_store_t *store, const char *project, + char **dependents, int count) { + for (int i = 0; i < count; i++) { + int held = closure_holds_ansible_tasks(store, project, dependents[i]); + if (held != 0) { + return held < 0 ? "ansible_lookup_failed" : "ansible_dependent"; + } + } + return NULL; +} + /* The env join (pass_envjoin.c) reads each image's files from its * Dockerfile's Module, stored with the ignore patterns of its context: an * ignore file changing, or a deletion of a file an image copies (which @@ -2992,6 +3044,10 @@ static int closure_try_plan(cbm_pipeline_t *p, cbm_store_t *store, const char *p decline = "doc_scope_changed"; goto done; } + decline = closure_ansible_dependent_decline(store, project, dependents, dependent_count); + if (decline) { + goto done; + } /* Closure = changed ∪ dependents. Every member must be in the current * discovery (a dependent outside it cannot be re-resolved). */ diff --git a/src/pipeline/pipeline_internal.h b/src/pipeline/pipeline_internal.h index d0069a30c..6a6ebfb11 100644 --- a/src/pipeline/pipeline_internal.h +++ b/src/pipeline/pipeline_internal.h @@ -1317,8 +1317,9 @@ char *cbm_k8s_resource_props(const char *qn, const char *effective_ns, const cha /* Ansible (pass_ansible.c): a playbook or task file IMPORTS the playbooks, * task files and roles its directives name; its tasks and handlers are - * Functions, a notify CALLS the handlers it names. Run at the end of the - * K8s pass, over the same files. */ + * Functions, a notify CALLS the handlers it names, a kube / k8s task + * DEPLOYS the template it applies. Run at the end of the K8s pass, over the + * same files. */ void cbm_pipeline_pass_ansible(cbm_pipeline_ctx_t *ctx, const cbm_file_info_t *files, int file_count); @@ -1545,7 +1546,8 @@ int cbm_pipeline_build_fresh_semantic_manifest(cbm_pipeline_t *p, const char *pr * (pass_terraform.c, extract_unified.c, extract_defs.c); an Ansible * playbook / task file IMPORTS {via:ansible} the playbooks, task files * and roles its directives name, its tasks and handlers are Functions - * {source:ansible} and a notify CALLS {via:notify} the handlers it names + * {source:ansible}, a notify CALLS {via:notify} the handlers it names, + * a kube / k8s task DEPLOYS {via:ansible} the template it applies * (pass_ansible.c). An index built before holds none of them, so it * rebuilds once on upgrade. */ enum { CBM_SEMANTIC_INDEX_VERSION = 9 }; diff --git a/tests/test_grammar_probe_f.c b/tests/test_grammar_probe_f.c index 8ee89ffba..843073bea 100644 --- a/tests/test_grammar_probe_f.c +++ b/tests/test_grammar_probe_f.c @@ -3484,6 +3484,117 @@ TEST(probe_ansible_tasks_are_functions) { PASS(); } +/* Every DEPLOYS an Ansible task draws, as " DEPLOYS []", + * then "deploys ". */ +static char *gpf_ansible_deploys(const GpfFile *files, int nfiles) { + GpfProj lp; + cbm_store_t *store = gpf_index_files(&lp, files, nfiles); + char *out = NULL; + if (store) { + sqlite3_stmt *st = NULL; + const char *sql = + "SELECT coalesce((SELECT group_concat(x, char(10)) FROM (SELECT s.name || ' DEPLOYS ' " + "|| t.file_path || ' [' || json_extract(e.properties,'$.module') || ']' AS x FROM " + "edges e JOIN nodes s ON s.id=e.source_id JOIN nodes t ON t.id=e.target_id WHERE " + "e.project=?1 AND e.type='DEPLOYS' AND json_extract(e.properties,'$.via')='ansible' " + "ORDER BY x)), '') || char(10) || 'deploys ' || (SELECT COUNT(*) FROM edges WHERE " + "project=?1 AND type='DEPLOYS' AND json_extract(properties,'$.via')='ansible')"; + if (sqlite3_prepare_v2(cbm_store_get_db(store), sql, -1, &st, NULL) == SQLITE_OK) { + sqlite3_bind_text(st, 1, lp.project, -1, SQLITE_STATIC); + if (sqlite3_step(st) == SQLITE_ROW) { + out = strdup((const char *)sqlite3_column_text(st, 0)); + } + } + sqlite3_finalize(st); + } + gpf_cleanup(&lp, store); + return out; +} + +/* A kube / k8s task DEPLOYS {via: ansible, module, line} the repository file + * it applies, traced to the template (or copied file) its play rendered it + * from: a renderer whose dest is the very filename; {{ item.item.K }} over + * "{{ R.results }}" -> each item of the renderer registering R (its list + * literal, a set_fact's or the role defaults'); {{ item.K }} -> each of the + * task's own items through the renderer of that dest; a literal filename a + * looped renderer renders to; a k8s template: argument. A template is found + * before a copy to the same dest. A remote copy (even of a name the role's + * files/ holds), a delete by name, apply nothing in the repository. */ +TEST(probe_ansible_kube_tasks_deploy_their_templates) { + static const GpfFile files[] = { + {"roles/app/tasks/main.yml", + "- name: Render manifests\n template:\n src: \"{{ item.file }}.j2\"\n" + " dest: \"{{ conf }}/{{ item.file }}\"\n with_items:\n - {name: a, file: a.yml}\n" + " - {name: b, file: b.yml}\n register: app_manifests\n" + "- name: Apply manifests\n kube:\n filename: \"{{ conf }}/{{ item.item.file }}\"\n" + " with_items:\n - \"{{ app_manifests.results }}\"\n" + "- name: Render single\n template:\n src: single.yml.j2\n" + " dest: \"{{ conf }}/single.yml\"\n" + "- name: Apply single\n kube:\n filename: \"{{ conf }}/single.yml\"\n" + "- name: Copy static\n copy:\n src: static.yml\n dest: \"{{ conf }}/static.yml\"\n" + "- name: Apply static\n kube:\n filename: \"{{ conf }}/static.yml\"\n" + "- name: Render from defaults\n template:\n src: \"{{ item.file }}.j2\"\n" + " dest: \"{{ conf }}/x/{{ item.file }}\"\n loop: \"{{ app_extra }}\"\n" + " register: extra_out\n" + "- name: Apply from defaults\n kube:\n filename: \"{{ conf }}/x/{{ item.item.file }}\"\n" + " loop: \"{{ extra_out.results }}\"\n" + "- name: Facts\n set_fact:\n sf_list:\n - {file: e.yml}\n" + "- name: Render from facts\n template:\n src: \"{{ item.file }}.j2\"\n" + " dest: \"{{ conf }}/f/{{ item.file }}\"\n with_items: \"{{ sf_list }}\"\n" + " register: sf_out\n" + "- name: Apply set_fact\n kube:\n filename: \"{{ conf }}/f/{{ item.item.file }}\"\n" + " with_items: \"{{ sf_out.results }}\"\n" + "- name: Pairs\n template:\n src: \"{{ item.name }}/{{ item.file }}.j2\"\n" + " dest: \"{{ conf }}/{{ item.name }}-{{ item.file }}\"\n with_items:\n" + " - {name: c, file: c.yml}\n" + "- name: Apply pairs\n kube:\n filename: \"{{ conf }}/{{ item.name }}-{{ item.file }}\"\n" + " loop:\n - {name: c, file: c.yml}\n" + "- name: Rendered literal\n template:\n src: \"{{ item.file }}.j2\"\n" + " dest: \"{{ conf }}/r/{{ item.file }}\"\n with_items:\n - {file: ipam.yml}\n" + "- name: Apply literal\n kube:\n filename: \"{{ conf }}/r/ipam.yml\"\n" + "- name: Copy first\n copy:\n src: first.yml\n dest: \"{{ conf }}/dup.yml\"\n" + "- name: Render dup\n template:\n src: dup.yml.j2\n dest: \"{{ conf }}/dup.yml\"\n" + "- name: Apply dup\n kube:\n filename: \"{{ conf }}/dup.yml\"\n" + "- name: Remote copy\n copy:\n src: static.yml\n dest: \"{{ conf }}/remote.yml\"\n" + " remote_src: true\n" + "- name: Apply remote\n kube:\n filename: \"{{ conf }}/remote.yml\"\n" + "- name: Delete by name\n kube:\n name: old\n resource: deploy\n state: absent\n" + "- name: k8s template\n kubernetes.core.k8s:\n template: tpl.yml.j2\n"}, + {"roles/app/defaults/main.yml", "app_extra:\n - file: d.yml\nother: 1\n"}, + {"roles/app/templates/a.yml.j2", "kind: ConfigMap\n"}, + {"roles/app/templates/b.yml.j2", "kind: ConfigMap\n"}, + {"roles/app/templates/c/c.yml.j2", "kind: ConfigMap\n"}, + {"roles/app/templates/d.yml.j2", "kind: ConfigMap\n"}, + {"roles/app/templates/e.yml.j2", "kind: ConfigMap\n"}, + {"roles/app/templates/ipam.yml.j2", "kind: ConfigMap\n"}, + {"roles/app/templates/single.yml.j2", "kind: ConfigMap\n"}, + {"roles/app/templates/tpl.yml.j2", "kind: ConfigMap\n"}, + {"roles/app/templates/dup.yml.j2", "kind: ConfigMap\n"}, + {"roles/app/files/first.yml", "kind: ConfigMap\n"}, + {"roles/app/files/static.yml", "kind: ConfigMap\n"}}; + int n = (int)(sizeof(files) / sizeof(files[0])); + char *got = gpf_ansible_deploys(files, n); + static const char want[] = + "Apply dup DEPLOYS roles/app/templates/dup.yml.j2 [kube]\n" + "Apply from defaults DEPLOYS roles/app/templates/d.yml.j2 [kube]\n" + "Apply literal DEPLOYS roles/app/templates/ipam.yml.j2 [kube]\n" + "Apply manifests DEPLOYS roles/app/templates/a.yml.j2 [kube]\n" + "Apply manifests DEPLOYS roles/app/templates/b.yml.j2 [kube]\n" + "Apply pairs DEPLOYS roles/app/templates/c/c.yml.j2 [kube]\n" + "Apply set_fact DEPLOYS roles/app/templates/e.yml.j2 [kube]\n" + "Apply single DEPLOYS roles/app/templates/single.yml.j2 [kube]\n" + "Apply static DEPLOYS roles/app/files/static.yml [kube]\n" + "k8s template DEPLOYS roles/app/templates/tpl.yml.j2 [kubernetes.core.k8s]\n" + "deploys 10"; + if (!got || strcmp(got, want) != 0) { + fprintf(stderr, "ansible deploys got:\n%s\nwant:\n%s\n", got ? got : "(null)", want); + free(got); + FAIL("ansible deploys"); + } + free(got); + PASS(); +} + /* Every script HTTP_CALLS edge as " ", then a line * "edgeless routes ": the Route nodes no edge touches. */ static char *gpf_script_http(const GpfFile *files, int nfiles) { @@ -5930,6 +6041,7 @@ SUITE(grammar_probe_f) { RUN_TEST(probe_terraform_kubernetes_objects_are_resources); RUN_TEST(probe_kustomize_name_prefix_makes_variants); RUN_TEST(probe_ansible_tasks_are_functions); + RUN_TEST(probe_ansible_kube_tasks_deploy_their_templates); RUN_TEST(probe_image_hub_splits_by_component); RUN_TEST(probe_gitops_objects_deploy_their_units); RUN_TEST(probe_shell_kubectl_deploys_what_it_applies); diff --git a/tests/test_pipeline.c b/tests/test_pipeline.c index a770e34e3..f17761fa5 100644 --- a/tests/test_pipeline.c +++ b/tests/test_pipeline.c @@ -1592,6 +1592,95 @@ TEST(pipeline_incremental_keeps_ansible_notifies_across_files) { PASS(); } +/* The files Ansible tasks DEPLOY (via ansible), ordered, one per line, into + * `out`; false on a query error. */ +static bool ansible_deploy_targets(cbm_store_t *s, const char *project, char *out, size_t sz) { + sqlite3_stmt *st = NULL; + bool ok = false; + if (sqlite3_prepare_v2(cbm_store_get_db(s), + "SELECT coalesce(group_concat(p, char(10)), '') FROM (SELECT " + "t.file_path AS p FROM edges e JOIN nodes t ON t.id=e.target_id WHERE " + "e.project=?1 AND e.type='DEPLOYS' AND " + "json_extract(e.properties,'$.via')='ansible' ORDER BY p)", + -1, &st, NULL) == SQLITE_OK && + sqlite3_bind_text(st, 1, project, -1, SQLITE_TRANSIENT) == SQLITE_OK && + sqlite3_step(st) == SQLITE_ROW) { + snprintf(out, sz, "%s", (const char *)sqlite3_column_text(st, 0)); + ok = true; + } + sqlite3_finalize(st); + return ok; +} + +/* A kube task DEPLOYS the template its play rendered, read through a list of + * the role's defaults/main.yml -- a file holding no task. Editing it (the + * list names another template) or deleting it must land on the full graph, + * and so must deleting the deployed template while a file of that name sits + * in the task file's directory (the render falls back to it). The repair + * did not re-walk the task file for the list edit -- it kept the edge to the + * template the list no longer names -- and re-walked it for the deletion as + * a dependent of the template, without the role's defaults: no edge at all. + * Both decline to the full rebuild (closure_ansible_decline: a role's + * defaults / vars; closure_ansible_dependent_decline: a task file joining + * the repair as a dependent). A template body edit keeps the repair and the + * edge. */ +TEST(pipeline_incremental_keeps_ansible_deploys_across_files) { + if (setup_test_repo() != 0) { + FAIL("failed to create temp dir"); + } + write_temp_file(g_tmpdir, "roles/app/tasks/main.yml", + "- name: Render\n template:\n src: \"{{ item.file }}.j2\"\n" + " dest: \"/etc/k/{{ item.file }}\"\n loop: \"{{ app_list }}\"\n" + " register: out\n" + "- name: Apply\n kube:\n filename: \"/etc/k/{{ item.item.file }}\"\n" + " loop: \"{{ out.results }}\"\n"); + write_temp_file(g_tmpdir, "roles/app/defaults/main.yml", "app_list:\n - {file: a.yml}\n"); + write_temp_file(g_tmpdir, "roles/app/templates/a.yml.j2", "a: 1\n"); + write_temp_file(g_tmpdir, "roles/app/templates/b.yml.j2", "b: 1\n"); + write_temp_file(g_tmpdir, "roles/app/tasks/b.yml.j2", "b: 2\n"); + char db_path[512]; + snprintf(db_path, sizeof(db_path), "%s/test_incr_deploys.db", g_tmpdir); + static const char *const want[] = { + "roles/app/templates/a.yml.j2", /* the full index */ + "roles/app/templates/a.yml.j2", /* a.yml.j2's body edited */ + "roles/app/templates/b.yml.j2", /* the defaults list names b.yml */ + "roles/app/tasks/b.yml.j2", /* templates/b.yml.j2 deleted */ + "", /* the defaults deleted */ + }; + char path[1024]; + for (int step = 0; step < (int)(sizeof(want) / sizeof(want[0])); step++) { + if (step == 1) { + snprintf(path, sizeof(path), "%s/roles/app/templates/a.yml.j2", g_tmpdir); + ASSERT_EQ(th_append_file(path, "a2: 2\n"), 0); + } else if (step == 2) { + write_temp_file(g_tmpdir, "roles/app/defaults/main.yml", + "app_list:\n - {file: b.yml}\n"); + } else if (step == 3) { + snprintf(path, sizeof(path), "%s/roles/app/templates/b.yml.j2", g_tmpdir); + ASSERT_EQ(th_unlink_force(path), 0); + } else if (step == 4) { + snprintf(path, sizeof(path), "%s/roles/app/defaults/main.yml", g_tmpdir); + ASSERT_EQ(th_unlink_force(path), 0); + } + cbm_pipeline_t *p = cbm_pipeline_new(g_tmpdir, db_path, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + ASSERT_EQ(cbm_pipeline_run(p), 0); + cbm_store_t *s = cbm_store_open_path(db_path); + ASSERT_NOT_NULL(s); + char got[2048]; + bool ok = ansible_deploy_targets(s, cbm_pipeline_project_name(p), got, sizeof(got)); + cbm_store_close(s); + cbm_pipeline_free(p); + ASSERT_TRUE(ok); + if (strcmp(got, want[step]) != 0) { + fprintf(stderr, "step %d deploys got [%s] want [%s]\n", step, got, want[step]); + } + ASSERT_STR_EQ(got, want[step]); + } + teardown_test_repo(); + PASS(); +} + /* Regression: a Terraform module's output read lives in one file * (infra/outputs.tf: module.net.vpc_id) and the module's source in another * (infra/main.tf). Pointing the source elsewhere edits main.tf alone; the @@ -23295,6 +23384,7 @@ SUITE(pipeline) { RUN_TEST(pipeline_incremental_drops_terraform_reads_of_a_moved_module); RUN_TEST(pipeline_incremental_keeps_ansible_includes_of_an_edited_file); RUN_TEST(pipeline_incremental_keeps_ansible_notifies_across_files); + RUN_TEST(pipeline_incremental_keeps_ansible_deploys_across_files); RUN_TEST(pipeline_field_named_like_method_keeps_its_reads_and_writes); RUN_TEST(pipeline_objectscript_export_preserves_calls_sequential_parallel); RUN_TEST(pipeline_objectscript_export_incremental_matches_full_relationships); From dec56c9c69c5a24119cd4740258a4bddbc760313 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Wed, 14 Oct 2026 21:06:34 +0200 Subject: [PATCH 2/3] feat(infra): an external chart is keyed by the URL that publishes it An external Chart was .__helm_dep__.: keyed by its name, inside one project. Two repositories naming the same published chart held two unrelated nodes, and a project naming two charts of one name from two publishers (kubermatic's cert-manager from charts.jetstack.io and from its own mirror) would have merged them into one. An external Chart whose repository URL is stated is now __chart__/, the same QN in every project, the URL in a `repository` property: scheme and host lower-cased, a trailing slash and `.git` dropped. A Chart.yaml dependency, a kustomize helmCharts entry, a Flux HelmRelease chartRef and a Terraform helm_release of an oci:// chart that name one published chart meet in one node. A chart reference (a Flux OCIRepository url, `chart = "oci://..."`) is cut to its repository; a repository URL never is, so repository oci://public.ecr.aws/karpenter with chart karpenter stays oci://public.ecr.aws/karpenter/karpenter. A repository that is no URL (an @alias, no scheme, file://, credentials in the authority, an unexpanded ${...}) or that the pass does not read (a Flux HelmRepository source, a Terraform `repository =`, a shell `repo/name`) keeps the per-project node: no identity is claimed for what was not stated. A query or fragment (`?token=...`) takes the per-project node too, so a token never becomes part of a QN or a property. One predicate now tells an external chart from a chart of the repository where Terraform, shell and the Go installer index the chart directories: an external chart a Chart.yaml depends on carries that file's path, and must never stand in for the chart the directory declares. Measured on 17 infrastructure repositories against an independent reading: 141 of 141 edges into external charts reach the identity their stated repository gives (114 by URL, 27 per project). The only differences to the previous index are the 79 external Chart nodes that moved to their URL identity, each with its repository property, and the 114 edges into them, re-pointed with type, source and properties unchanged. Every chart gate (Helm includes, helmCharts, Terraform, the Go installer, shell, CI, GitOps deploys) reads the same as before; three runs are identical; the incremental re-index equals a full index for a shared chart's owning file edited and deleted, a URL respelled, a dependency moved to another publisher and to an alias, a chartRef URL edit and a new Terraform oci:// release. The index rebuilds once on upgrade with the other version 9 changes. Signed-off-by: Martin Vogel --- README.md | 2 +- src/pipeline/pass_k8s.c | 204 +++++++++++++++++++++++++++---- src/pipeline/pass_operator.c | 2 +- src/pipeline/pass_shell.c | 19 ++- src/pipeline/pass_terraform.c | 30 +++-- src/pipeline/pipeline_internal.h | 29 ++++- tests/test_grammar_probe_f.c | 125 ++++++++++++++++++- 7 files changed, 351 insertions(+), 60 deletions(-) diff --git a/README.md b/README.md index 4b09dcb04..0f09953d4 100644 --- a/README.md +++ b/README.md @@ -333,7 +333,7 @@ The install script placed beside the binary is **reported, not deleted**: uninst - `INFRA_MAPS`: a Kubernetes Service to each workload whose pod labels carry every `key: value` of the Service's selector, within one namespace - `CONNECTS_TO`: a Kubernetes workload or compose service to the Service its containers' environment addresses (`PRODUCT_CATALOG_SERVICE_ADDR: productcatalogservice:3550`, a URL's host, `..svc...`, a bare name on an `*_ADDR` / `*_HOST` / `*_URL` / ... key; compose: the project's `.env` interpolated): a Kubernetes Service of the same bundle and namespace, or a compose service of the same file. The edge keeps `via: env`, `key` and `container`, never the value - `USAGE` with `ref: image`: a Kubernetes workload (any pod template: containers, initContainers, CronJobs, custom resources) or a docker compose service to the `Image` it runs -- `DEPLOYS`: what puts manifests, kustomizations or charts on a cluster, to what it puts there: a Flux Kustomization or Argo Application to the kustomizations / manifests / chart of its path, a Flux HelmRelease to its chart (`spec.chart` or `chartRef` → OCIRepository), a shell command or Make recipe (`kubectl apply -f/-k`, `kustomize build | kubectl apply`, `helm install/upgrade/template`), a Go installer loading a chart with the Helm SDK, a Terraform `helm_release`, a Terraform `kubernetes_*` resource the object it declares (a `Resource` like a manifest's, `ecosystem: terraform`, when its name is a literal), a kustomization inflating a chart (`helmCharts`; its values file an `IMPORTS`). A chart outside the repository is a shared external `Chart`; the edge says `via` (`flux`, `argo`, `helmrelease`, `kubectl`, `kustomize`, `helm`, `helm-sdk`, `terraform`, `helmcharts`, ...) +- `DEPLOYS`: what puts manifests, kustomizations or charts on a cluster, to what it puts there: a Flux Kustomization or Argo Application to the kustomizations / manifests / chart of its path, a Flux HelmRelease to its chart (`spec.chart` or `chartRef` → OCIRepository), a shell command or Make recipe (`kubectl apply -f/-k`, `kustomize build | kubectl apply`, `helm install/upgrade/template`), a Go installer loading a chart with the Helm SDK, a Terraform `helm_release`, a Terraform `kubernetes_*` resource the object it declares (a `Resource` like a manifest's, `ecosystem: terraform`, when its name is a literal), a kustomization inflating a chart (`helmCharts`; its values file an `IMPORTS`). A chart outside the repository is an external `Chart`, keyed by its publisher when its repository URL is stated (`__chart__/`, the same node in every project, the URL in `repository`), else one per project and name; the edge says `via` (`flux`, `argo`, `helmrelease`, `kubectl`, `kustomize`, `helm`, `helm-sdk`, `terraform`, `helmcharts`, ...) - `DEPENDS_ON`: a Helm chart to the charts it declares (the local chart for a resolvable `file://` repository, otherwise an external `Chart`; the edge keeps `repository`), and a `go.mod` / `requirements.txt` file to an external `Package`; a Dockerfile to the `Image`s it builds on (`via: from`, `copy-from`); an `Image` to the Dockerfile it is built from (`via: build`; `tool`: `skaffold`, `compose`, `docker-build` for `docker` / `podman` / `nerdctl` / `buildah` build commands in scripts, Make and CI); a compose service to the services its `depends_on` / `links` name - `CONFIGURES`: code reading an environment variable to the shared `EnvVar` of that name (`strategy: env_access`); a Kubernetes container, compose service or Dockerfile (`ENV` of the image) setting one to that container's own `EnvVar` (`strategy: env_set`, with `source`: `value`, `configMapKeyRef`, `secretKeyRef`, `fieldRef`, ..., `compose`, `dockerfile`; values are never stored); and that container's `EnvVar` to the code reading the name inside the files its image is built from (`strategy: env_read`, with `dockerfile`): the image's files are the sources its Dockerfile's `COPY` / `ADD` put into it, less the context's `.dockerignore` - `OVERRIDE` with `via: patch`: a Kustomize patch to each object of its kustomization's build it changes (child → base): a patch file's documents to the object of the same kind and name; an inline patch (from the kustomization) or a JSON6902 operation file to what its `target` selects (group, version, kind, name, namespace, label selector); a component's in the build of each kustomization including it. The edge keeps `kustomization`, `patch` and `line`; a patch file is an `IMPORTS` (`local_name: patches`) of its kustomization, never a member of the build diff --git a/src/pipeline/pass_k8s.c b/src/pipeline/pass_k8s.c index 67bbb37f8..1af48be43 100644 --- a/src/pipeline/pass_k8s.c +++ b/src/pipeline/pass_k8s.c @@ -26,6 +26,7 @@ #include "foundation/str_util.h" /* cbm_json_escape */ #include "pipeline/worker_pool.h" +#include #include #include #include @@ -2542,14 +2543,159 @@ static char *k8s_dep_props(const char *repository) { return props; } -/* DEPENDS_ON from `chart_id` to the shared external Chart node of `name` - * (one per name and project, so charts depending on the same chart meet). */ +/* ── External charts ─────────────────────────────────────────────── */ + +/* An external Chart is keyed by its publisher: `__chart__/`, the + * same QN in every project, so two repositories naming one published chart + * name one node. A chart whose repository does not resolve to a URL (a + * `@alias`, an empty or templated repository, a source the pass cannot read) + * keeps the per-project `.__helm_dep__.`: no identity is + * claimed for what was not stated. */ +static const char K8S_CHART_QN_PREFIX[] = "__chart__"; +static const char K8S_HELM_DEP_MARK[] = ".__helm_dep__."; + +/* A byte a stated repository URL may hold: no whitespace or control byte, + * no quote, template or shell expansion, escape or angle bracket. */ +static bool k8s_url_byte_ok(unsigned char c) { + return c > ' ' && c < 0x7f && !strchr("\"'`{}$<>\\", c); +} + +bool cbm_chart_repo_url(const char *url, char *out, size_t sz) { + if (!url || !out || sz == 0) { + return false; + } + while (*url == ' ' || *url == '\t') { + url++; + } + size_t n = strlen(url); + while (n > 0 && isspace((unsigned char)url[n - SKIP_ONE])) { + n--; + } + if (n == 0 || n + SKIP_ONE > sz || !isalpha((unsigned char)url[0])) { + return false; + } + for (size_t i = 0; i < n; i++) { + if (!k8s_url_byte_ok((unsigned char)url[i])) { + return false; + } + } + /* a query or fragment may carry a token (?token=...): like credentials + * in the authority, never put into a QN or a property */ + if (memchr(url, '?', n) || memchr(url, '#', n)) { + return false; + } + size_t s = 0; /* the scheme: [A-Za-z][A-Za-z0-9+.-]* then :// */ + while (s < n && + (isalnum((unsigned char)url[s]) || url[s] == '+' || url[s] == '-' || url[s] == '.')) { + s++; + } + if (s + K8S_SLEN("://") > n || strncmp(url + s, "://", K8S_SLEN("://")) != 0) { + return false; + } + size_t host = s + K8S_SLEN("://"); + size_t path = host; + while (path < n && url[path] != '/') { + path++; + } + /* no host, or credentials in the authority: never put into a QN */ + if (path == host || memchr(url + host, '@', path - host)) { + return false; + } + for (size_t i = 0; i < path; i++) { + out[i] = (char)tolower((unsigned char)url[i]); + } + memcpy(out + path, url + path, n - path); + out[n] = '\0'; + if (strncmp(out, "file://", K8S_SLEN("file://")) == 0) { + return false; /* a directory, not a publisher */ + } + size_t end = n; + while (end > path && out[end - SKIP_ONE] == '/') { + end--; + } + if (end - path >= K8S_SLEN(".git") && + strncmp(out + end - K8S_SLEN(".git"), ".git", K8S_SLEN(".git")) == 0) { + end -= K8S_SLEN(".git"); + while (end > path && out[end - SKIP_ONE] == '/') { + end--; + } + } + out[end] = '\0'; + return true; +} + +bool cbm_chart_ref_repo(const char *ref, char *out, size_t sz) { + if (!ref || !out || sz == 0) { + return false; + } + size_t end = strlen(ref); + while (end > 0 && ref[end - SKIP_ONE] == '/') { + end--; + } + while (end > 0 && ref[end - SKIP_ONE] != '/') { + end--; + } + while (end > 0 && ref[end - SKIP_ONE] == '/') { + end--; + } + if (end == 0 || end + SKIP_ONE > sz) { + return false; + } + memcpy(out, ref, end); + out[end] = '\0'; + return true; +} + +/* QN and node properties of the external Chart `name` (non-NULL) of + * repository `url` (NULL: unknown). */ +static void k8s_chart_identity(const char *project, const char *name, const char *url, char *qn, + size_t qsz, char *props, size_t psz) { + char norm[CBM_SZ_1K]; + if (name[0] && !strchr(name, '/') && cbm_chart_repo_url(url, norm, sizeof(norm))) { + int w = snprintf(qn, qsz, "%s%s/%s", K8S_CHART_QN_PREFIX, norm, name); + if (w > 0 && (size_t)w < qsz) { + char esc[CBM_SZ_2K]; + cbm_json_escape(esc, (int)sizeof(esc), norm); + snprintf(props, psz, "{\"source\":\"helm\",\"external\":true,\"repository\":\"%s\"}", + esc); + return; + } + } + snprintf(qn, qsz, "%s%s%s", project, K8S_HELM_DEP_MARK, name); + snprintf(props, psz, "{\"source\":\"helm\",\"external\":true}"); +} + +int64_t cbm_pipeline_external_chart(cbm_pipeline_ctx_t *ctx, const char *name, const char *url, + const char *file) { + if (!name) { + return 0; + } + char qn[CBM_SZ_2K]; + char props[CBM_SZ_4K]; + k8s_chart_identity(ctx->project_name, name, url, qn, sizeof(qn), props, sizeof(props)); + const cbm_gbuf_node_t *known = cbm_gbuf_find_by_qn(ctx->gbuf, qn); + if (known) { + return known->id; + } + return cbm_gbuf_upsert_node(ctx->gbuf, "Chart", name, qn, file, SKIP_ONE, 0, props); +} + +bool cbm_pipeline_chart_is_external(const cbm_gbuf_node_t *node) { + const char *qn = node ? node->qualified_name : NULL; + return qn && (strncmp(qn, K8S_CHART_QN_PREFIX, K8S_SLEN(K8S_CHART_QN_PREFIX)) == 0 || + strstr(qn, K8S_HELM_DEP_MARK) != NULL); +} + +/* DEPENDS_ON from `chart_id` to the external Chart node of `name` in + * `repository` (charts depending on the same published chart meet). */ static bool k8s_external_dep(cbm_pipeline_ctx_t *ctx, int64_t chart_id, const char *name, - const char *chart_rel, const char *props) { - char dep_qn[CBM_SZ_512]; - snprintf(dep_qn, sizeof(dep_qn), "%s.__helm_dep__.%s", ctx->project_name, name); - int64_t dep_id = cbm_gbuf_upsert_node(ctx->gbuf, "Chart", name, dep_qn, chart_rel, SKIP_ONE, 0, - "{\"source\":\"helm\",\"external\":true}"); + const char *repository, const char *chart_rel, const char *props) { + char dep_qn[CBM_SZ_2K]; + char node_props[CBM_SZ_4K]; + k8s_chart_identity(ctx->project_name, name, repository, dep_qn, sizeof(dep_qn), node_props, + sizeof(node_props)); + int64_t dep_id = + cbm_gbuf_upsert_node(ctx->gbuf, "Chart", name, dep_qn, chart_rel, SKIP_ONE, 0, node_props); if (dep_id <= 0) { return false; } @@ -2613,7 +2759,9 @@ static void k8s_link_local_deps(cbm_pipeline_ctx_t *ctx, k8s_local_deps_t *deps) target->id != d->chart_id) { cbm_gbuf_insert_edge(ctx->gbuf, d->chart_id, target->id, "DEPENDS_ON", d->props); } else if (d->chart_rel) { - (void)k8s_external_dep(ctx, d->chart_id, d->name, d->chart_rel, d->props); + /* a file:// directory this repository cannot resolve names no + * publisher: the per-project external Chart */ + (void)k8s_external_dep(ctx, d->chart_id, d->name, NULL, d->chart_rel, d->props); } } k8s_local_deps_free(deps); @@ -2621,8 +2769,9 @@ static void k8s_link_local_deps(cbm_pipeline_ctx_t *ctx, k8s_local_deps_t *deps) /* Emit a Chart node for a Chart.yaml and a DEPENDS_ON edge per declared * dependency (#338): to the chart in this repository for a resolvable - * `file://` dependency (once all charts are read), otherwise to a shared - * external Chart node per dependency name, the repository on the edge. */ + * `file://` dependency (once all charts are read), otherwise to the external + * Chart node of the dependency's repository and name, the repository also on + * the edge. */ static void handle_helm_chart(cbm_pipeline_ctx_t *ctx, const char *rel_path, const char *source, k8s_local_deps_t *local_deps) { cbm_helm_chart_t hc; @@ -2656,7 +2805,7 @@ static void handle_helm_chart(cbm_pipeline_ctx_t *ctx, const char *rel_path, con continue; } char *props = dep->repository[0] ? k8s_dep_props(dep->repository) : NULL; - dep_edges += k8s_external_dep(ctx, chart_id, dep->name, rel_path, props); + dep_edges += k8s_external_dep(ctx, chart_id, dep->name, dep->repository, rel_path, props); cbm_free(CBM_MEM_CLASS_OTHER, props); } cbm_helm_chart_free(&hc); @@ -3439,7 +3588,7 @@ static void k8s_link_includes(cbm_pipeline_ctx_t *ctx, k8s_bundles_t *b, * outermost kustomizations below it -- their Kustomization node -- and the * manifests outside any, their File node; a chart dir: its Chart); a * HelmRelease -> its chart (a repository chart dir of a Git/Bucket source, - * else the shared external Chart of that name). */ + * else the external Chart of that name). */ typedef struct { cbm_pipeline_ctx_t *ctx; @@ -3475,21 +3624,17 @@ static int64_t k8s_chart_node(k8s_dep_link_t *l, const char *dir) { return 0; } -/* The shared external Chart of `name` (the one chart dependencies use). */ -static int64_t k8s_external_chart(k8s_dep_link_t *l, const char *name, const char *file) { - char qn[CBM_SZ_512]; - snprintf(qn, sizeof(qn), "%s.__helm_dep__.%s", l->ctx->project_name, name); - const cbm_gbuf_node_t *known = cbm_gbuf_find_by_qn(l->ctx->gbuf, qn); - if (known) { - return known->id; - } - return cbm_gbuf_upsert_node(l->ctx->gbuf, "Chart", name, qn, file, SKIP_ONE, 0, - "{\"source\":\"helm\",\"external\":true}"); +/* The external Chart of `name` in repository `url` (NULL: unknown), the one + * chart dependencies use. */ +static int64_t k8s_external_chart(k8s_dep_link_t *l, const char *name, const char *url, + const char *file) { + return cbm_pipeline_external_chart(l->ctx, name, url, file); } /* A HelmRelease whose chartRef names OCIRepository `oci` (its USAGE edge * {"ref":"chartRef"}, resolved with the references) deploys the chart that - * repository fetches: the shared external Chart of the URL's last segment. */ + * repository fetches: the external Chart of that OCI reference (its last + * segment the chart, the rest its repository). */ static void k8s_link_chart_refs(k8s_dep_link_t *l, const k8s_hr_t *oci) { const char *url = oci->chart; size_t end = strlen(url); @@ -3513,6 +3658,8 @@ static void k8s_link_chart_refs(k8s_dep_link_t *l, const k8s_hr_t *oci) { snprintf(props, sizeof(props), "{\"via\":\"helmrelease\",\"chart\":\"%s\",\"external\":true,\"url\":\"%s\"}", esc_name, esc_url); + char repo[CBM_SZ_1K]; + const char *repo_url = cbm_chart_ref_repo(url, repo, sizeof(repo)) ? repo : NULL; const cbm_gbuf_edge_t **in = NULL; int n = 0; if (cbm_gbuf_find_edges_by_target_type(l->ctx->gbuf, oci->node, "USAGE", &in, &n) != 0) { @@ -3523,7 +3670,8 @@ static void k8s_link_chart_refs(k8s_dep_link_t *l, const k8s_hr_t *oci) { const char *p = in[i]->properties_json; const cbm_gbuf_node_t *hr = cbm_gbuf_find_by_id(l->ctx->gbuf, in[i]->source_id); if (p && strstr(p, "\"ref\":\"chartRef\"") && hr && hr->file_path) { - k8s_deploys_edge(l, hr->id, k8s_external_chart(l, name, hr->file_path), props); + k8s_deploys_edge(l, hr->id, k8s_external_chart(l, name, repo_url, hr->file_path), + props); } } } @@ -3542,7 +3690,7 @@ static void k8s_prop_add(char *out, size_t out_sz, int *w, const char *key, cons * build: DEPLOYS {via: helmcharts, external, chart, url, version, release, * namespace, values, line} from the kustomization (its Kustomization * Resource, else its File -- the unit a Flux / Argo deploy reaches) to the - * shared external Chart of that name. */ + * external Chart of that repository and name. */ static void k8s_link_kcharts(k8s_dep_link_t *l, const k8s_facts_t *f) { for (int i = 0; i < f->kchart_n; i++) { const k8s_kchart_t *c = &f->kcharts[i]; @@ -3563,7 +3711,7 @@ static void k8s_link_kcharts(k8s_dep_link_t *l, const k8s_facts_t *f) { continue; } snprintf(props + w, sizeof(props) - (size_t)w, ",\"line\":%d}", c->line); - k8s_deploys_edge(l, src, k8s_external_chart(l, c->chart, c->kust), props); + k8s_deploys_edge(l, src, k8s_external_chart(l, c->chart, c->url, c->kust), props); } } @@ -4230,7 +4378,9 @@ static void k8s_link_deploys(cbm_pipeline_ctx_t *ctx, k8s_kusts_t *ks, k8s_bundl k8s_deploys_edge(&l, hr->node, k8s_chart_node(&l, dir), props); } } else if (src && src->file_path && !strchr(hr->chart, '/') && !strchr(hr->chart, '$')) { - k8s_deploys_edge(&l, hr->node, k8s_external_chart(&l, hr->chart, src->file_path), + /* the source's URL is not read (a HelmRepository's spec.url): + * the per-project external Chart */ + k8s_deploys_edge(&l, hr->node, k8s_external_chart(&l, hr->chart, NULL, src->file_path), props); } } diff --git a/src/pipeline/pass_operator.c b/src/pipeline/pass_operator.c index 85fd106d8..239cacd85 100644 --- a/src/pipeline/pass_operator.c +++ b/src/pipeline/pass_operator.c @@ -1640,7 +1640,7 @@ static CBMHashTable *gc_chart_index(const cbm_gbuf_t *gb) { const char *qn = nodes[i]->qualified_name; const char *slash = fp ? strrchr(fp, '/') : NULL; const char *base = slash ? slash + SKIP_ONE : fp; - if (!fp || !qn || strstr(qn, ".__helm_dep__.") || + if (!fp || !qn || cbm_pipeline_chart_is_external(nodes[i]) || (strcmp(base, "Chart.yaml") != 0 && strcmp(base, "Chart.yml") != 0) || !slash) { continue; } diff --git a/src/pipeline/pass_shell.c b/src/pipeline/pass_shell.c index 5a25d4ad1..8e6f82ce9 100644 --- a/src/pipeline/pass_shell.c +++ b/src/pipeline/pass_shell.c @@ -11,8 +11,8 @@ * kubectl ... -k D the Kustomization of D * kustomize build D | ... | kubectl apply -f - (also kubectl kustomize D) * helm install|upgrade [NAME] CHART the Chart of a chart dir, else the - * shared external Chart (repo/name, - * oci://.../name) + * external Chart of that name + * (repo/name, oci://.../name) * helm template [NAME] CHART | ... | kubectl apply -f - * find D -name G -exec kubectl apply -f {} \; D's manifests matching G * @@ -216,7 +216,7 @@ static bool sh_index_build(sh_index_t *ix, cbm_pipeline_ctx_t *ctx) { const char *base = fp ? sh_basename(fp) : ""; const char *qn = nodes[i]->qualified_name; if ((strcmp(base, "Chart.yaml") == 0 || strcmp(base, "Chart.yml") == 0) && qn && - !strstr(qn, ".__helm_dep__.")) { + !cbm_pipeline_chart_is_external(nodes[i])) { cbm_ht_set(ix->charts, fp, (void *)(intptr_t)nodes[i]->id); } } @@ -3569,16 +3569,11 @@ static int64_t sh_source(cbm_pipeline_ctx_t *ctx, const sh_span_t *spans, int n, return fn ? fn->id : 0; } -/* The shared external Chart of `name` (the one chart dependencies use). */ +/* The external Chart of `name` a helm command names: the hit keeps no + * repository URL (a `repo/name` alias, an oci:// reference's last segment), + * so the per-project one. */ static int64_t sh_external_chart(cbm_pipeline_ctx_t *ctx, const char *name, const char *file) { - char qn[CBM_SZ_512]; - snprintf(qn, sizeof(qn), "%s.__helm_dep__.%s", ctx->project_name, name); - const cbm_gbuf_node_t *known = cbm_gbuf_find_by_qn(ctx->gbuf, qn); - if (known) { - return known->id; - } - return cbm_gbuf_upsert_node(ctx->gbuf, "Chart", name, qn, file, SKIP_ONE, 0, - "{\"source\":\"helm\",\"external\":true}"); + return cbm_pipeline_external_chart(ctx, name, NULL, file); } /* A hit's CI context as JSON members (`,"ci":..,"job":..`); "" for none. */ diff --git a/src/pipeline/pass_terraform.c b/src/pipeline/pass_terraform.c index 6dad7172b..6328311c1 100644 --- a/src/pipeline/pass_terraform.c +++ b/src/pipeline/pass_terraform.c @@ -7,8 +7,9 @@ * chart = "./charts/x" / "${path.module}/charts/x" the Chart of that * directory, relative to * the module's directory - * repository = "", chart = "" the shared external - * chart = "oci://registry/.../" Chart of that name + * repository = "", chart = "" the external Chart of + * chart = "oci://registry/.../" that name (an oci:// + * reference: of its URL) * Values are evaluated from string literals and ${path.module}; any other * expression is unknown, and an unknown chart is no edge. Run at the end of * the K8s pass, over the same files. @@ -198,7 +199,9 @@ static CBMHashTable *tf_chart_index(cbm_pipeline_ctx_t *ctx) { const char *fp = nodes[i]->file_path; const char *qn = nodes[i]->qualified_name; const char *base = fp ? tf_basename(fp) : ""; - if (fp && qn && !strstr(qn, ".__helm_dep__.") && + /* an external chart a Chart.yaml depends on carries that file's + * path too: only the chart it declares is the directory's */ + if (fp && qn && !cbm_pipeline_chart_is_external(nodes[i]) && (strcmp(base, "Chart.yaml") == 0 || strcmp(base, "Chart.yml") == 0)) { cbm_ht_set(ix, fp, (void *)(intptr_t)nodes[i]->id); } @@ -220,18 +223,6 @@ static int64_t tf_local_chart(const tf_t *t, const char *dir) { return 0; } -/* The shared external Chart of `name` (the one chart dependencies use). */ -static int64_t tf_external_chart(cbm_pipeline_ctx_t *ctx, const char *name, const char *file) { - char qn[CBM_SZ_512]; - snprintf(qn, sizeof(qn), "%s.__helm_dep__.%s", ctx->project_name, name); - const cbm_gbuf_node_t *known = cbm_gbuf_find_by_qn(ctx->gbuf, qn); - if (known) { - return known->id; - } - return cbm_gbuf_upsert_node(ctx->gbuf, "Chart", name, qn, file, SKIP_ONE, 0, - "{\"source\":\"helm\",\"external\":true}"); -} - /* The node of block `resource.helm_release.` in the file, by the QN * its definition got -- not among every block of that name: modules that * share a release name (`this`) would make each lookup scan all of them. */ @@ -293,6 +284,9 @@ static void tf_helm_release(tf_t *t, TSNode block, const char *name) { int64_t src = tf_block_node(t, name); int64_t tgt = 0; bool external = false; + /* the repository of an oci:// reference; "" when unknown (a + * `repository =` value is not read) */ + char repo[CBM_SZ_1K] = ""; char path[CBM_SZ_1K] = ""; static const char OCI[] = "oci://"; if (strncmp(chart, OCI, sizeof(OCI) - SKIP_ONE) == 0) { @@ -300,6 +294,10 @@ static void tf_helm_release(tf_t *t, TSNode block, const char *name) { if (tf_plain_name(base)) { snprintf(path, sizeof(path), "%s", base); external = true; + /* the reference names its repository: all but the chart */ + if (!cbm_chart_ref_repo(chart, repo, sizeof(repo))) { + repo[0] = '\0'; + } } } else if (has_repo) { if (tf_plain_name(chart)) { @@ -310,7 +308,7 @@ static void tf_helm_release(tf_t *t, TSNode block, const char *name) { tgt = tf_local_chart(t, path); } if (external) { - tgt = tf_external_chart(t->ctx, path, t->file); + tgt = cbm_pipeline_external_chart(t->ctx, path, repo[0] ? repo : NULL, t->file); } if (src <= 0 || tgt <= 0) { return; diff --git a/src/pipeline/pipeline_internal.h b/src/pipeline/pipeline_internal.h index 6a6ebfb11..5bd522266 100644 --- a/src/pipeline/pipeline_internal.h +++ b/src/pipeline/pipeline_internal.h @@ -1337,6 +1337,30 @@ bool cbm_repo_rel_join(const char *dir, const char *v, char *out, size_t sz); * digest dropped), made when new; 0 for scratch or no image. */ int64_t cbm_pipeline_image_node(cbm_pipeline_ctx_t *ctx, const char *ref); +/* A chart repository URL as an identity (pass_k8s.c): `url` trimmed, its + * scheme and host lower-cased, trailing `/` and `.git` dropped. False for no + * URL: no scheme or host, a file:// directory, credentials in the authority, + * a query or fragment (`?`, `#`: a token may ride there), or a byte no + * stated URL holds (whitespace, quote, template, `$`). A + * repository URL is never cut: oci://host/x with chart x is oci://host/x/x. */ +bool cbm_chart_repo_url(const char *url, char *out, size_t sz); + +/* The repository of chart reference `ref` (a Flux OCIRepository url, a + * Terraform `chart = "oci://..."`): all before its last segment, the chart. + * False when none is left. */ +bool cbm_chart_ref_repo(const char *ref, char *out, size_t sz); + +/* The external Chart node of chart `name` from repository `url` (pass_k8s.c), + * made when new and owned by `file`: `__chart__/` {source, + * external, repository} -- one QN in every project -- when `url` resolves + * (cbm_chart_repo_url), else `.__helm_dep__.`. */ +int64_t cbm_pipeline_external_chart(cbm_pipeline_ctx_t *ctx, const char *name, const char *url, + const char *file); + +/* Is Chart node `node` an external chart (either identity) rather than one + * a Chart.yaml of this repository declares? */ +bool cbm_pipeline_chart_is_external(const cbm_gbuf_node_t *node); + /* Environment variable `key` that `owner` sets (pass_docker.c): a K8s or * compose Resource for its container `container`, a Dockerfile Module for * its image (container NULL). Makes the container-scoped EnvVar @@ -1548,8 +1572,9 @@ int cbm_pipeline_build_fresh_semantic_manifest(cbm_pipeline_t *p, const char *pr * and roles its directives name, its tasks and handlers are Functions * {source:ansible}, a notify CALLS {via:notify} the handlers it names, * a kube / k8s task DEPLOYS {via:ansible} the template it applies - * (pass_ansible.c). An index built before holds none of them, so it - * rebuilds once on upgrade. */ + * (pass_ansible.c). An external Chart is keyed by its publisher URL. + * An index built before holds none of them, so it rebuilds once on + * upgrade. */ enum { CBM_SEMANTIC_INDEX_VERSION = 9 }; typedef struct { diff --git a/tests/test_grammar_probe_f.c b/tests/test_grammar_probe_f.c index 843073bea..6450ab5b5 100644 --- a/tests/test_grammar_probe_f.c +++ b/tests/test_grammar_probe_f.c @@ -1812,7 +1812,7 @@ static char *gpf_deploys(const GpfFile *files, int nfiles) { "SELECT coalesce(group_concat(x, char(10)), '') FROM (SELECT " "CASE WHEN s.label IN ('Module','File') THEN '@' || s.file_path ELSE s.name END " "|| ' -> ' || CASE WHEN t.label='Chart' THEN 'chart:' || t.name || " - "CASE WHEN instr(t.qualified_name,'__helm_dep__') THEN '(ext)' ELSE '' END " + "CASE WHEN json_extract(t.properties,'$.external') THEN '(ext)' ELSE '' END " "ELSE t.label || ':' || t.file_path END " "|| ' (' || json_extract(e.properties,'$.via') || ')' AS x " "FROM edges e JOIN nodes s ON s.id=e.source_id JOIN nodes t ON t.id=e.target_id " @@ -4763,6 +4763,128 @@ TEST(probe_helm_unresolved_file_dependency_stays_external) { PASS(); } +/* Every external Chart as " > + * in=", sorted, one per line. NULL on failure; the caller + * frees. */ +static char *gpf_external_charts(const GpfFile *files, int nfiles) { + GpfProj lp; + cbm_store_t *store = gpf_index_files(&lp, files, nfiles); + char *out = NULL; + if (store) { + sqlite3_stmt *st = NULL; + const char *sql = + "SELECT coalesce(group_concat(x, char(10)), '') FROM (SELECT n.name || ' ' || " + "replace(n.qualified_name, ?1, '

') || ' ' || " + "coalesce(json_extract(n.properties,'$.repository'), '-') || ' in=' || " + "(SELECT count(*) FROM edges e WHERE e.target_id=n.id) AS x FROM nodes n " + "WHERE n.project=?1 AND n.label='Chart' AND json_extract(n.properties,'$.external') " + "ORDER BY x)"; + if (sqlite3_prepare_v2(cbm_store_get_db(store), sql, -1, &st, NULL) == SQLITE_OK) { + sqlite3_bind_text(st, 1, lp.project, -1, SQLITE_STATIC); + if (sqlite3_step(st) == SQLITE_ROW) { + out = strdup((const char *)sqlite3_column_text(st, 0)); + } + } + sqlite3_finalize(st); + } + gpf_cleanup(&lp, store); + return out; +} + +/* An external Chart is keyed by its publisher, __chart__/, + * the repository's scheme and host lower-cased, a trailing slash and `.git` + * dropped: a Chart.yaml dependency, a kustomization, a Flux chartRef and a + * Terraform oci:// reference naming one published chart meet in one node, + * the same QN in every project. An oci:// repository is never cut, even when + * it ends in the chart's name (karpenter); a chart reference is cut to its + * repository. A repository that is no URL (an @alias, no scheme, file://, + * credentials in the authority, a query or fragment a token may ride in, an + * unexpanded ${...}) or that the pass does not read (a + * Terraform `repository =`) keeps the per-project + * .__helm_dep__.. The external charts a Chart.yaml depends on + * carry its path, yet a helm_release, a helm command and a Go loader naming + * that chart directory deploy the chart the directory declares. */ +TEST(probe_external_chart_identity_is_its_publisher) { + static const GpfFile files[] = { + {"charts/app/Chart.yaml", + "apiVersion: v2\nname: app\nversion: 1.0.0\ndependencies:\n" + " - name: redis\n repository: https://Charts.Example.com/stable/\n" + " version: 1.x\n" + " - name: common\n repository: \"@stable\"\n version: 1.x\n" + " - name: nats\n repository: oci://registry.example.com/charts\n" + " version: 1.x\n" + " - name: vault\n repository: https://user@charts.example.com/private\n" + " version: 1.x\n"}, + {"main.tf", "resource \"helm_release\" \"app\" {\n" + " name = \"app\"\n chart = \"./charts/app\"\n}\n\n" + "resource \"helm_release\" \"queue\" {\n" + " name = \"queue\"\n" + " chart = \"oci://registry.example.com/charts/nats\"\n}\n\n" + "resource \"helm_release\" \"cache\" {\n" + " name = \"cache\"\n" + " repository = \"https://charts.example.com/stable\"\n" + " chart = \"redis\"\n}\n"}, + {"deploy.sh", "#!/bin/sh\ndeploy_app() {\n helm upgrade --install app ./charts/app\n}\n"}, + {"go.mod", "module example.com/app\n\ngo 1.22\n"}, + {"install.go", "package main\n\n" + "import \"helm.sh/helm/v3/pkg/chart/loader\"\n\n" + "func installApp() {\n\t_, _ = loader.Load(\"charts/app\")\n}\n"}, + {"obs/kustomization.yaml", + GPF_KUST "helmCharts:\n- name: redis\n repo: https://charts.example.com/stable.git\n" + "- name: karpenter\n repo: oci://registry.example.com/karpenter\n" + "- name: templated\n repo: https://charts.example.com/${ENV}\n" + "- name: local\n repo: file://../charts\n" + "- name: bare\n repo: charts.example.com/stable\n" + "- name: tokened\n repo: https://charts.example.com/stable?token=x\n" + "- name: fragment\n repo: https://charts.example.com/stable#x\n"}, + {"apps/web/kustomization.yaml", + GPF_KUST "namespace: web\nresources:\n - ocirepository.yaml\n - helmrelease.yaml\n"}, + {"apps/web/ocirepository.yaml", + "apiVersion: source.toolkit.fluxcd.io/v1\nkind: OCIRepository\nmetadata:\n" + " name: web-chart\nspec:\n url: oci://registry.example.com/charts/nats\n"}, + {"apps/web/helmrelease.yaml", + "apiVersion: helm.toolkit.fluxcd.io/v2\nkind: HelmRelease\nmetadata:\n name: web\n" + "spec:\n chartRef:\n kind: OCIRepository\n name: web-chart\n"}}; + int n = (int)(sizeof(files) / sizeof(files[0])); + char *got = gpf_external_charts(files, n); + static const char want[] = + "bare

.__helm_dep__.bare - in=1\n" + "common

.__helm_dep__.common - in=1\n" + "fragment

.__helm_dep__.fragment - in=1\n" + "karpenter __chart__oci://registry.example.com/karpenter/karpenter " + "oci://registry.example.com/karpenter in=1\n" + "local

.__helm_dep__.local - in=1\n" + "nats __chart__oci://registry.example.com/charts/nats oci://registry.example.com/charts " + "in=3\n" + "redis

.__helm_dep__.redis - in=1\n" + "redis __chart__https://charts.example.com/stable/redis https://charts.example.com/stable " + "in=2\n" + "templated

.__helm_dep__.templated - in=1\n" + "tokened

.__helm_dep__.tokened - in=1\n" + "vault

.__helm_dep__.vault - in=1"; + if (!got || strcmp(got, want) != 0) { + fprintf(stderr, "external charts got:\n%s\nwant:\n%s\n", got ? got : "(null)", want); + free(got); + FAIL("external chart identity"); + } + free(got); + GPF_ASSERT_DEPLOYS(files, n, + "HelmRelease/web -> chart:nats(ext) (helmrelease)\n" + "Kustomization -> chart:bare(ext) (helmcharts)\n" + "Kustomization -> chart:fragment(ext) (helmcharts)\n" + "Kustomization -> chart:karpenter(ext) (helmcharts)\n" + "Kustomization -> chart:local(ext) (helmcharts)\n" + "Kustomization -> chart:redis(ext) (helmcharts)\n" + "Kustomization -> chart:templated(ext) (helmcharts)\n" + "Kustomization -> chart:tokened(ext) (helmcharts)\n" + "deploy_app -> chart:app (helm)\n" + "installApp -> chart:app (helm-sdk)\n" + "resource.helm_release.app -> chart:app (terraform)\n" + "resource.helm_release.cache -> chart:redis(ext) (terraform)\n" + "resource.helm_release.queue -> chart:nats(ext) (terraform)"); + PASS(); +} + /* A remote base is outside the repository. The import resolver's symbol-name * guess walked its URL segments and linked kubermatic's gcp addon to the * `deploy` function of hack/ci/deploy.sh. A kustomize entry is a path, @@ -5980,6 +6102,7 @@ SUITE(grammar_probe_f) { RUN_TEST(probe_k8s_manifest_mints_no_envelope_variables); RUN_TEST(probe_yaml_config_keeps_variables); RUN_TEST(probe_helm_unresolved_file_dependency_stays_external); + RUN_TEST(probe_external_chart_identity_is_its_publisher); RUN_TEST(probe_kustomize_nested_sibling_entries); RUN_TEST(probe_kustomize_single_module); RUN_TEST(probe_kustomize_entries_after_commented_out_entry); From eca1a004b4d3c5235f7bd50fa50872914cfb68fe Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Wed, 14 Oct 2026 22:09:37 +0200 Subject: [PATCH 3/3] feat(configlink): TOML and INI keys link by exact name in their component A config key linked to every code symbol whose name contained it. In YAML, JSON and HCL files that guess was retired in WP0.9; TOML and INI files kept it, and in 17 infrastructure repositories every one of their 72 key_symbol edges was a coincidence: `.gitleaks.toml useDefault` linked to TestConfig_UseDefault and three other symbols, a toml-merge tool's test data `config_path` to 46 `*ConfigPath` constants all over deckhouse, `grafana.ini admin_user` to user-cluster constants, a Rust Cargo.toml dependency to other services' go.mod requirements. A .toml, .ini, .cfg or .conf key now links like an infrastructure key: by exact name (camelCase and snake_case still normalize to one key) and only to code inside the config file's component, its directory or the one around a config/, conf/, settings/ or etc/ directory. .env and .properties keys keep substring matching; the tests that covered it on a TOML key now use a .properties key. Measured on the 17 repositories: key_symbol 95 -> 23, the 23 being the infrastructure edges WP0.9 reviewed, the 72 removed all TOML/INI and all coincidences, no other edge or node changed; deterministic over three runs; the incremental re-index equals a full index for INI, TOML, code and tfvars edits. Signed-off-by: Martin Vogel --- README.md | 1 + src/pipeline/pass_configlink.c | 19 ++++++---- tests/test_configlink.c | 68 +++++++++++++++++++++++++++++----- 3 files changed, 70 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 0f09953d4..27262a079 100644 --- a/README.md +++ b/README.md @@ -336,6 +336,7 @@ The install script placed beside the binary is **reported, not deleted**: uninst - `DEPLOYS`: what puts manifests, kustomizations or charts on a cluster, to what it puts there: a Flux Kustomization or Argo Application to the kustomizations / manifests / chart of its path, a Flux HelmRelease to its chart (`spec.chart` or `chartRef` → OCIRepository), a shell command or Make recipe (`kubectl apply -f/-k`, `kustomize build | kubectl apply`, `helm install/upgrade/template`), a Go installer loading a chart with the Helm SDK, a Terraform `helm_release`, a Terraform `kubernetes_*` resource the object it declares (a `Resource` like a manifest's, `ecosystem: terraform`, when its name is a literal), a kustomization inflating a chart (`helmCharts`; its values file an `IMPORTS`). A chart outside the repository is an external `Chart`, keyed by its publisher when its repository URL is stated (`__chart__/`, the same node in every project, the URL in `repository`), else one per project and name; the edge says `via` (`flux`, `argo`, `helmrelease`, `kubectl`, `kustomize`, `helm`, `helm-sdk`, `terraform`, `helmcharts`, ...) - `DEPENDS_ON`: a Helm chart to the charts it declares (the local chart for a resolvable `file://` repository, otherwise an external `Chart`; the edge keeps `repository`), and a `go.mod` / `requirements.txt` file to an external `Package`; a Dockerfile to the `Image`s it builds on (`via: from`, `copy-from`); an `Image` to the Dockerfile it is built from (`via: build`; `tool`: `skaffold`, `compose`, `docker-build` for `docker` / `podman` / `nerdctl` / `buildah` build commands in scripts, Make and CI); a compose service to the services its `depends_on` / `links` name - `CONFIGURES`: code reading an environment variable to the shared `EnvVar` of that name (`strategy: env_access`); a Kubernetes container, compose service or Dockerfile (`ENV` of the image) setting one to that container's own `EnvVar` (`strategy: env_set`, with `source`: `value`, `configMapKeyRef`, `secretKeyRef`, `fieldRef`, ..., `compose`, `dockerfile`; values are never stored); and that container's `EnvVar` to the code reading the name inside the files its image is built from (`strategy: env_read`, with `dockerfile`): the image's files are the sources its Dockerfile's `COPY` / `ADD` put into it, less the context's `.dockerignore` +- `CONFIGURES` with `strategy: key_symbol`: a code symbol to the config key it names (`config_key`). A YAML, JSON, HCL (Terraform, OpenTofu, tfvars), TOML or INI key links by exact name (`maxRetryCount` = `max_retry_count`) and only to code inside the config file's component: its directory, or the one around a `config/`, `configs/`, `conf/`, `settings/` or `etc/` directory. A `.env` or `.properties` key also links by substring (confidence 0.75) - `OVERRIDE` with `via: patch`: a Kustomize patch to each object of its kustomization's build it changes (child → base): a patch file's documents to the object of the same kind and name; an inline patch (from the kustomization) or a JSON6902 operation file to what its `target` selects (group, version, kind, name, namespace, label selector); a component's in the build of each kustomization including it. The edge keeps `kustomization`, `patch` and `line`; a patch file is an `IMPORTS` (`local_name: patches`) of its kustomization, never a member of the build - `OVERRIDE` with `via: namePrefix` / `nameSuffix`: an object a kustomization renames, from its variant: a `Resource` named with the deployed name (`capi-` + `controller-manager`; the prefixes of nested kustomizations outermost first, suffixes innermost first) in the namespace that build gives it, `ecosystem: kustomize`, which the kustomization's file `DEFINES`. CRDs, Namespaces and API services keep their names, as kustomize does. The edge keeps `kustomization` and `line` - `USAGE` with `via: terraform`: a Terraform / OpenTofu reference (`var.X`, `local.X`, `module.X`, `data.T.N`, `T.N`) from the top-level block holding it to the block of its module (the `.tf` / `.tofu` files of its directory) that declares it (a `locals` entry: its `locals` block); an entry of the block's `depends_on` list is a `DEPENDS_ON` instead. The edge keeps `ref` (`var`, `local`, `module`, `data`, `resource`) and `line`; a name only mentioned in string text is no reference. A module call with a local source (`module "net" { source = "./modules/net" }`) `IMPORTS` (`local_name`: the module's name) the `Folder` of that directory (the root directory: the `Project`), and `module.net.id` (also `module.net[0].id`, `module.net[*].id`) `USAGE`s (`ref: output`) the `output "id"` there, beside `module.net` diff --git a/src/pipeline/pass_configlink.c b/src/pipeline/pass_configlink.c index 5c1b41654..d4cdf3582 100644 --- a/src/pipeline/pass_configlink.c +++ b/src/pipeline/pass_configlink.c @@ -116,15 +116,18 @@ static const cbm_gbuf_node_t **canonical_node_copy(const cbm_gbuf_node_t *const return sorted; } -/* Infrastructure files: YAML, JSON and HCL (Terraform, OpenTofu, tfvars). A - * key_symbol pair with either side in one links by exact name and only inside - * the config file's component (user decision 2026-10-10): in 17 infra - * repositories 4 of 40 sampled key_symbol edges were right, all of them exact - * and within one directory tree; the rest were substring and cross-component - * coincidences. Other config formats keep exact + substring matching. */ +/* Files whose key_symbol pairs link by exact name and only inside the config + * file's component: infrastructure files -- YAML, JSON, HCL (Terraform, + * OpenTofu, tfvars) -- (user decision 2026-10-10: in 17 infra repositories 4 + * of 40 sampled key_symbol edges were right, all of them exact and within one + * directory tree; the rest were substring and cross-component coincidences), + * and TOML and INI files (user decision 2026-10-14: all 72 of their substring + * or cross-component edges in those repositories were coincidences, such as + * `.gitleaks.toml useDefault` <- TestConfig_UseDefault). Other config formats + * (.env, .properties) keep exact + substring matching. */ static bool cl_is_infra_path(const char *path) { - static const char *const exts[] = {".yaml", ".yml", ".json", ".tf", - ".hcl", ".tofu", ".tfvars", NULL}; + static const char *const exts[] = {".yaml", ".yml", ".json", ".tf", ".hcl", ".tofu", + ".tfvars", ".toml", ".ini", ".cfg", ".conf", NULL}; const char *dot = path ? strrchr(path, '.') : NULL; const char *slash = path ? strrchr(path, '/') : NULL; if (!dot || (slash && dot < slash)) { diff --git a/tests/test_configlink.c b/tests/test_configlink.c index c7248a8bc..e665e99de 100644 --- a/tests/test_configlink.c +++ b/tests/test_configlink.c @@ -85,7 +85,8 @@ static void rm_rf(const char *path) { /* ── Strategy 1: Config Key → Code Symbol ───────────────────────── */ /* Go: TestConfigKeySymbol_ExactMatch - * config.toml has max_connections, main.go has getMaxConnections() */ + * config.toml has max_connections, main.go has maxConnections(). (A TOML key + * links by exact name only, see configlink_key_symbol_toml_ini_exact_scoped.) */ TEST(configlink_key_symbol_exact_match) { cbm_gbuf_t *gb = cbm_gbuf_new("test", "/tmp/test"); @@ -94,9 +95,9 @@ TEST(configlink_key_symbol_exact_match) { "test.config.max_connections", "config.toml", 0, 0, NULL); ASSERT_GT(cfg_id, 0); - /* Code Function node: getMaxConnections from main.go */ - int64_t func_id = cbm_gbuf_upsert_node(gb, "Function", "getMaxConnections", - "test.main.getMaxConnections", "main.go", 0, 0, NULL); + /* Code Function node: maxConnections from main.go */ + int64_t func_id = cbm_gbuf_upsert_node(gb, "Function", "maxConnections", + "test.main.maxConnections", "main.go", 0, 0, NULL); ASSERT_GT(func_id, 0); /* Run configlink */ @@ -113,12 +114,14 @@ TEST(configlink_key_symbol_exact_match) { } /* Go: TestConfigKeySymbol_SubstringMatch - * config.toml has request_timeout, main.go has getRequestTimeoutSeconds() */ + * app.properties has request_timeout, main.go has getRequestTimeoutSeconds(). + * (Formats outside the exact + component rule -- .properties, .env -- keep + * substring matching.) */ TEST(configlink_key_symbol_substring_match) { cbm_gbuf_t *gb = cbm_gbuf_new("test", "/tmp/test"); - cbm_gbuf_upsert_node(gb, "Variable", "request_timeout", "test.config.request_timeout", - "config.toml", 0, 0, NULL); + cbm_gbuf_upsert_node(gb, "Variable", "request_timeout", "test.app.request_timeout", + "app.properties", 0, 0, NULL); cbm_gbuf_upsert_node(gb, "Function", "getRequestTimeoutSeconds", "test.main.getRequestTimeoutSeconds", "main.go", 0, 0, NULL); @@ -281,6 +284,48 @@ TEST(configlink_key_symbol_terraform_variable) { PASS(); } +/* TOML and INI keys (.toml, .ini, .cfg, .conf) link like infrastructure keys: + * by exact name and only inside the config file's component. In 17 infra + * repositories all 72 TOML/INI key_symbol edges were substring or + * cross-component coincidences (`.gitleaks.toml useDefault` <- + * TestConfig_UseDefault, `grafana.ini admin_user` <- GenAdminUser). Each + * format's key below reaches only svc-a's exact `requestTimeout`: not svc-a's + * substring `getRequestTimeoutSeconds`, not svc-b's exact `requestTimeout`. */ +TEST(configlink_key_symbol_toml_ini_exact_scoped) { + static const char *const files[] = {"svc-a/config.toml", "svc-a/settings.ini", + "svc-a/setup.cfg", "svc-a/app.conf"}; + enum { FILES = 4 }; + cbm_gbuf_t *gb = cbm_gbuf_new("test", "/tmp/test"); + int64_t cfg[FILES]; + for (int i = 0; i < FILES; i++) { + char qn[CBM_SZ_128]; + snprintf(qn, sizeof(qn), "test.cfg%d.request_timeout", i); + cfg[i] = cbm_gbuf_upsert_node(gb, "Variable", "request_timeout", qn, files[i], 0, 0, NULL); + } + int64_t inside = cbm_gbuf_upsert_node(gb, "Function", "requestTimeout", + "test.svc_a.main.requestTimeout", "svc-a/main.go", 0, 0, + NULL); + cbm_gbuf_upsert_node(gb, "Function", "getRequestTimeoutSeconds", + "test.svc_a.client.getRequestTimeoutSeconds", "svc-a/client.go", 0, 0, + NULL); + cbm_gbuf_upsert_node(gb, "Function", "requestTimeout", "test.svc_b.main.requestTimeout", + "svc-b/main.go", 0, 0, NULL); + run_configlink(gb, "test", NULL); + const cbm_gbuf_edge_t **edges = NULL; + int count = 0; + cbm_gbuf_find_edges_by_type(gb, "CONFIGURES", &edges, &count); + ASSERT_EQ(count, FILES); + for (int i = 0; i < FILES; i++) { + bool found = false; + for (int e = 0; e < count; e++) { + found = found || (edges[e]->source_id == inside && edges[e]->target_id == cfg[i]); + } + ASSERT_TRUE(found); + } + cbm_gbuf_free(gb); + PASS(); +} + /* No count of config keys decides what links: the collector stopped at 4,096. */ TEST(configlink_key_symbol_no_config_cap) { enum { KEYS = 4200 }; @@ -469,9 +514,11 @@ enum { CL_DET_CANDIDATES = 9000 }; /* > the 8192 code-entry cap */ static cbm_gbuf_t *build_oversized_corpus(bool reverse) { cbm_gbuf_t *gb = cbm_gbuf_new("test", "/tmp/test"); - /* One config key every candidate can match on: "max_connections". */ - cbm_gbuf_upsert_node(gb, "Variable", "max_connections", "test.config.max_connections", - "config.toml", 0, 0, NULL); + /* One config key every candidate can match on by substring: + * "max_connections", in a format that keeps substring matching (TOML and + * INI keys link by exact name only). */ + cbm_gbuf_upsert_node(gb, "Variable", "max_connections", "test.app.max_connections", + "app.properties", 0, 0, NULL); for (int k = 0; k < CL_DET_CANDIDATES; k++) { int i = reverse ? (CL_DET_CANDIDATES - 1 - k) : k; @@ -529,6 +576,7 @@ SUITE(configlink) { RUN_TEST(configlink_key_symbol_config_dir_component); RUN_TEST(configlink_key_symbol_no_config_cap); RUN_TEST(configlink_key_symbol_terraform_variable); + RUN_TEST(configlink_key_symbol_toml_ini_exact_scoped); /* Strategy 2: Dependency → Import */ RUN_TEST(configlink_dep_import_package_json);