A Markdown section that embeds a diagram is about its candidate functions
more often than the Markdown curve says, by how much depends on the
diagram's kind. 553 candidates from such sections in 16 held-out
repositories, judged blind in three rounds:
0.20-0.30 0.30-0.40 0.40+ local extras
prose 0.24 0.28 0.45 0.275
structural 0.32 0.46 0.57 0.78 (9 items)
Structural is Mermaid sequence/class/ER/C4/architecture and PlantUML
C4/class/sequence/use-case/component; prose any other Mermaid or PlantUML
kind. A section takes the highest family of the Diagram nodes it defines;
its function candidates at or inside the doc's home and its local extras
carry the family's value, and the first band is now stored. Outside
functions, files, folders, other formats and sections without a diagram
keep the Markdown curve.
On kubernetes' apiserver folder the sequence-diagram section's
candidates move from 0.22/0.51 to 0.46/0.57, the flowchart sections gain
five first-band candidates at 0.24, other sections and the graph are
unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Doc -> code candidates now carry a p judged for their kind, their document
format and where the document sits.
Format (sample 6, 234 blind-judged candidates): reStructuredText, AsciiDoc and
PDF sections each get their own curve (reST stored from tfidf 0.30, AsciiDoc
from 0.20, PDF not measured and not stored). A candidate that repeats an
exact MENTIONS link of the same section is not stored.
Position (sample 7, 490 blind-judged candidates from four monorepos): every
document gets a home folder -- its folder; a docs folder documents its
parent; a folder without code hands over to the nearest one with code; the
repository root means the whole project. Markdown candidates outside the
document's home were right 6 of 90 and are never stored; inside it they are
stored from tfidf 0.30 (0.33 / 0.53); for documents of the whole project
samples 4 and 7 pooled give 0.22 / 0.51 from 0.30.
New kinds, all in doc_link_candidates (evidence gains "kind" and
"position"; no new edge type, no schema change):
- local: up to two home functions ranked below the top five (0.32);
- file: a whole file, its vector the sum of its functions' terms and its
definitions' names, config keys included, cut to 64 terms (0.30, from
tfidf 0.40);
- folder: a README or index document -> its home folder (0.87).
get_code_snippet on a section lists every kind with its label;
get_file_outline shows the sections about the file or a folder holding it
(cbm_store_doc_candidates_for_path).
cbm_sem_p_2dp rounded 0.325 and 0.525 down (their float values sit 1.2e-6
below the half); its epsilon is 1e-4 on the x100 scale.
Replayed on the sample repositories, the build stores exactly the rows the
judged curves select (950 rows, 0 differences).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A documentation section that explains a function rarely names it, so
the deterministic doc links cannot connect them. The semantic pass now
ranks, for every Markdown section that can be about code, the functions
whose TF-IDF terms best match the section's heading and body, over the
functions' own corpus (function-pair scores are untouched):
- gate: no candidates for release-number or project-meta headings
(changelog, licence, contributing, links, contents, install, ...),
bodies under 20 words, or bodies that are mostly links;
- targets: functions and methods, never tests (by path or name);
- stored: a section's best five with tfidf >= 0.20, each with p, the
probability that the section is about the function, from 274
blind-judged candidates in five held-out repositories (three reading
rounds): 0.36 below 0.30, 0.42 below 0.40, else 0.55. Below 0.20 the
judged share was 0.06, so nothing is stored there.
These are candidates for an agent to verify, not graph edges: at best
about half are right, far below the bar for a link. They live in a side
table, doc_link_candidates (section and function qualified names, rank,
score, p, evidence), created when a generation is published, so older
indexes read as "no candidates". Full and legacy-partial reindexes
replace a project's rows; the delta route keeps the previous rows (its
proxy buffer has no section text or function tokens to recompute
them), and a row whose node is gone is skipped when read.
get_code_snippet shows them: on a Section, "possibly_about" (its
candidate functions); on a function or method, "possibly_described_in"
(the sections), at most five, highest p then score first, with a
candidates_note saying what they are.
Measured on fastapi (6,144 gated sections): 24,293 rows, identical to
the generator's dump; index time within noise; the database grows by
the table (37 MB there, with a 148-character project prefix on every
qualified name).
Tests: semantic:sem_doc_calibrated_p_bands,
store_arch:doc_candidates_replace_and_get (RED when a replace keeps old
rows), pipeline:pipeline_doc_candidates_published_and_kept_by_delta
(RED with test functions admitted, the heading gate off, the publish
write skipped, or the delta route replacing rows),
mcp:snippet_doc_candidates_both_directions (RED with stale rows shown).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The semantic pass read every string property of a function (body text,
signature, return type, decorators, docstring) with a reader that
stopped at the first '"' and kept escapes raw: a value holding an
escaped quote lost everything after it, and "\nreturn" became the token
"nreturn". cbm_sem_json_str honours the escapes: \" \\ \/ keep their
character, \n \t \r \b \f and \uXXXX become a separator.
The tokens change, so admissions change: on 13 repositories 1,340 ->
1,387 admitted SEMANTICALLY_RELATED pairs, 614 of them unchanged. Of
the dropped pairs that earlier samples had judged, 52 of 89 were
related (0.58); of the kept ones, 73 of 92 (0.79). A pre-registered
blind sample of 100 newly admitted pairs from 11 held-out repositories
(three reading rounds) judged 65 related (0.65, Wilson 0.55-0.74), so
the admitted set is 0.72 precise against 0.69 before. The same index
run twice gives identical pairs and scores.
"p" is refit on the new admitted population (bands at the quartiles of
the admitted scores, pooled where a higher band judged lower; added and
kept pairs weighted by their share): score < 0.773: 0.63, else 0.76
(was 0.56 / 0.64 / 0.73 / 0.86 in four bands).
Tests: sem_json_str_honours_escapes (RED with escapes ignored),
sem_calibrated_p_bands.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
When a function has more candidate partners than its budget of ten
SEMANTICALLY_RELATED edges, admission took the pairs in discovery order:
the first ten won, whatever their scores. Admission is now best-first
(cbm_sem_admit_best_first: descending score, ties in the canonical
order, still a pure function of the sorted inputs). On a blind judged
sample from held-out repositories, the pairs first-come admitted over
budget were 0 of 17 related; the pairs best-first admits instead, 7 of
20. Where no budget binds nothing changes (django 66 edges identical;
fastapi 426 -> 430 edges, 222 changed).
Every SEMANTICALLY_RELATED edge now carries "p": the probability that a
reader judges the pair related, by score band, from 120 blind-judged
admitted pairs (score < 0.778: 0.56, < 0.806: 0.64, < 0.852: 0.73,
else 0.86). p is computed from the score as stored (three decimals),
so one shown score always has one p. Scores and the 0.75 threshold are
unchanged; a scoring change (TF-IDF alone, or weights fitted on the
first sample) was judged on the same sample and did not beat the
shipped score by the pre-registered margin.
Tests: sem_admit_best_first_keeps_the_best, sem_calibrated_p_bands,
pipeline_semantic_edges_carry_p; each is RED with its behavior
reverted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The TF-IDF signal keyed each term by the token's POSITION in the
function, so any two functions of similar length shared their first
terms whatever their words: 0.20 of every SEMANTICALLY_RELATED score
measured length, not vocabulary. Terms are now keyed by corpus token
(cbm_sem_tfidf_terms: one term per distinct word, weight term
frequency x idf, ascending token index, which the sparse cosine
merges on).
Measured on a blind judged sample: 222 function pairs from six
held-out repositories, stratified by score band, read in three rounds.
The share of admitted SEMANTICALLY_RELATED edges a reader judges
related rises from 0.713 to 0.742, with about 19% fewer edges (fastapi
507 -> 426, django 64 -> 66). The same sample shows that no score
threshold reaches 0.90 precision, so the documented "~95% at 0.75" did
not hold before this change either.
The harness that measured it ships with the fix, off by default:
CBM_SEM_PAIR_SIGNALS=<path> writes one line per scored pair (both
qualified names, the score, every signal value, whether the pair was
admitted, both locations); CBM_SEM_PAIR_SIGNALS_FLOOR also records
pairs below the threshold, never admitting them. cbm_sem_combined_score
is now cbm_sem_signal_values + cbm_sem_combine (same arithmetic, same
order). CBM_SEMANTIC_INDEX_VERSION 3 -> 4: indexes built before score
pairs differently.
Tests: sem_tfidf_terms_compare_vocabulary_not_positions,
sem_combine_weighs_signals,
pipeline_semantic_pair_signals_never_change_the_graph; each is RED with
its behavior reverted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every item here was measured on a real corpus before and after, not
reasoned about. Go corpus, event lane, first run -> now: 245.5 M
allocations -> 134.8 M, 190 GB cumulative -> 29.3 GB, 158 GB never
written -> 9.2 GB, 132 M churn pairs -> 35.7 M, 93 GB of allocator byte
work -> 0.84 GB. Wall clock: Go 40.7 s -> 19.3 s, linux kernel 239.6 s ->
194.6 s, and all ten LSP languages faster with none slower.
Graph output is unchanged, proven rather than assumed: per-class multiset
digests over the full kernel index (8,563,471 nodes identical, 18/20 node
classes and 18/24 edge classes byte-identical, the rest differing only in
the classes a same-binary re-run also perturbs through worker merge order).
The changes, each with its own reason:
arena an exact arena that is appended to later restarts growth at
8 KB instead of the 64 KB default — the cross-file pass adds a
few resolved calls to a compacted result, so the default block
was ~60 KB untouched per result, 0.5 GB of the Go worker peak.
cbm_arena_init_lazy defers the first block for arenas that are
usually never used at all.
scratch per-thread reuse instead of allocate-and-free per unit of
work: result_compact (8 MB kept), the cross-file pass (4 MB),
extract_defs' walk stack, and the sqlite index cells (a 1 MB
arena replacing 12.1 M malloc/free pairs on Go). Walk frames
moving into scratch is why pass_k8s now opens the keep window
explicitly: without it every YAML file opened a 512 KB scratch
on the main thread, +3 GB.
sqlite index cells are written in place — building the record and
copying it into the cell was a second allocation per entry,
16 M record buffers of pure churn on Go.
registry is_test_qn and common_prefix_len in one pass, per-bucket test
flags computed once instead of per candidate comparison, and
the reachability keys arena allocated lazily.
minhash the unique-trigram set is sized from the trigram count and
skips repeats, and LSH queries reuse one seen-set per worker
instead of allocating per query.
parallel lsp_surface rows and the k8s manifest prep now run across
workers (graph writes stay sequential and ordered), the file
node is hoisted out of register_and_link_def, and configlink
prepares dependency imports once with a length prefilter.
go_lsp one sealed stdlib registry shared by all workers, published
by CAS, instead of one per worker.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The semantic pass kept four dense 768-float vectors resident per function
(ri/api/type/deco = 12.3 KB per func, ~9.4 GB transient on the linux kernel)
solely so candidate pairs could be cosine-scored after LSH. Replace the
resident floats with 4-bit quantized codes (~0.5 KB each, ~6x): new
src/semantic/rotsq.{h,c} — rotated scalar quantization from the RaBitQ
family of algorithms (Gao et al., SIGMOD 2024/2025), implemented from the
papers in plain C11 (no Eigen/library dependency): a deterministic
XXH3-seeded +-1 diagonal followed by a Fast Walsh-Hadamard rotation
(768 -> 1024 padded) spreads vector mass so per-vector scalar quantization
is near-optimal; the inner product is recovered from the codes with the
exact SQ expansion (one u8 integer dot + four multiplies) — a deterministic
pure function of the codes. Named rotsq, not rabitq: this is the family
core, not the papers' exact codebook construction.
Producers build each dense vector in a worker-local buffer, encode, and
discard; the int8 qvec export is unchanged. LSH signatures are computed
from DEQUANTIZED codes against hyperplanes in the rotated basis — random
hyperplanes are basis-agnostic, so no dense originals are retained and the
phase structure is untouched.
Estimator quality gated by sem_rotsq_ip_error_bounds (seeded vectors:
mean error < 1%, max < 4% of unit scale, ASan-green). Effect on output:
multi-threaded determinism preserved (xfs: two runs byte-identical);
SEMANTICALLY_RELATED edges shift once as scores move within ~1% of the
0.75 threshold (xfs: ~130 -> 203; sampled new pairs are high quality —
alloc/free, read-verify and collapse/insert families).
Suites: semantic 33 (incl. the new estimator gate), simhash 24,
pipeline 216 — zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
Raul Acosta