The closure-delta incremental route scored importance off a graph that does
not contain the project's edges, and persisted the result.
cbm_delta_preseed fills that route's gbuf with PROXY nodes -- id, label, name,
qualified name, file path, and nothing else. Only the re-extracted files carry
real edges; every inbound edge from an unchanged file is snapshotted and
re-linked later, inside cbm_delta_patch. Scoring in run_postpasses therefore
read an in-degree of zero for symbols with hundreds of real callers, and
because the repaired files' rows are purged and re-inserted, cbm_delta_patch
wrote those zeros to disk. Measured on a fixture whose `helper` has 25
callers: full index 5.000000, after one closure-delta re-index 0.000000.
closure_try_plan runs first, so this is the preferred warm path, not an edge
case. The comment at the call site claimed a rehydrated whole-project buffer;
that is true only of the legacy-partial route, and it is now stated correctly.
The fix rescores in SQL over the staging store, immediately after
cbm_delta_patch has inserted the new nodes and re-linked the inbound edges --
the first moment on this route where the complete graph is queryable. Loading
the full graph into RAM instead would have given back the point of the delta
route.
Rescoring is project-wide, not limited to the changed files. Distinct-files-
per-name and in-degree are both GLOBAL inputs: a body-only edit in one file
can change the score of a symbol defined in a file that did not change. A
scoped rescore would leave those stale.
TWO ROUTES, ONE RULE. The scoring rule and the JSON write-back are not
reimplemented in SQL. cbm_pipeline_importance_score() and
cbm_pipeline_importance_set_prop() are now single definitions that both routes
call -- the SQL side reaches them through two SQLite user-defined functions,
so the same C code runs either way and the rule cannot drift. SQL contributes
only the three aggregates, each an indexed GROUP BY. What remains free is the
input GATHERING, and tests bind that directly: a full index and a closure-delta
incremental index converging on the same tree must produce identical scores for
every symbol. Both new tests assert the route was actually taken, so neither
can pass by silently running some other path.
Cost, measured on a 666 MB Java corpus: cold full index unchanged at 83 s
(the recompute does not run on that path); warm closure-delta re-index 20 s
without the recompute, 26 s with it. The delta route keeps its advantage over
a full reindex (26 s vs 83 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Score every Function/Method/Class node during indexing and persist the
result as a numeric "importance" key inside the node's existing
properties_json:
importance = sqrt(num_refs) * priv * generic * distinct * test_penalty
num_refs is the incoming CALLS + USAGE degree. The multipliers demote
private (leading-underscore) and generic names (defined in >= 5 distinct
files), promote distinctive identifiers (snake_case or camelCase, len >=
8), and demote test scaffolding -- symbols in a test file per the graph's
own cbm_is_test_path() classifier, or targets of an incoming TESTS edge.
Weighted degree only; a transitive (PageRank) refinement is deliberately
not built here until it can be shown to beat this on a fixed judgment
set.
No schema change and no CBM_INDEX_FORMAT_VERSION bump: the score lives in
the existing properties_json TEXT column, so no index is forced to
rebuild and indexes written by older builds simply lack the key.
Three failure modes this shape invites, all of them silent, are addressed
directly:
* Cost. The generic-name multiplier needs |{files a name is defined in}|.
Computing that per NODE with a pairwise file-path scan is O(k^3) in a
same-name group of size k -- ruinous on real corpora, where a single
Java name group reaches k > 4000. Here the count is computed once per
distinct NAME and memoized, and file paths are deduplicated through a
hash set, so total distinct-file work is linear in the graph. Measured
cold full index of a 666 MB Java corpus, same -O2 binary: 85 s with the
pass disabled, 80 s with this implementation, 481 s with the per-node
shape.
* Registration. PREDUMP_PASS_COUNT is now derived from the pass table
with sizeof instead of being hand-written. A hand-written count that
lags an appended entry silently skips the last-registered pass with
every test still green.
* Incremental. The incremental post-pass sequence runs the pass too, over
a graph rehydrated from the store whose nodes already carry the key.
The write-back overwrites an existing key in place instead of appending
a second one, which would otherwise corrupt properties_json into
{"importance":1.0,...,"importance":2.0}.
tests/test_importance.c covers the formula and binds all three failure
modes; the complexity suite gains a GATED counter asserting the pass's
same-name-group work stays linear (~2x, not ~4x) under a corpus doubling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: petercoxphoto <20230727+petercoxphoto@users.noreply.github.com>
Martin Vogel
