The note on daemon-owned environment now carries the CLI rule from main.
The C test badge and counts come from a run of the suites on main:
9,388 tests in 189 suites.
Signed-off-by: Bernhard Jackiewicz <bj@techport.io>
The UI section now describes the Lens that is on main: the six
workspaces, the Architecture views, Galaxy and the local chat. Three GIFs
and six stills of this repository at 4d9592b5 replace the old screenshot
and the large WebP recordings. Statements that no longer match the UI
are corrected or removed, and the test counts come from main.
Signed-off-by: Bernhard Jackiewicz <bj@techport.io>
Normalize Windows cache paths when reporting runtime settings and resolve
Git executables before spawning history queries. Keep executable lookup
results per thread and cover the ownership contract deterministically.
Support Git 2.43 while denying history-query network transports, and close
the activity database on daemon startup failure so Windows can clean up.
Validation: 463 native macOS tests, 421 Windows tests, 44 MSan UI tests,
and focused canonical lint passed. Production-only revert checks confirm
the repaired failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Five suites (grammar_regression grammar_labels pipeline lang_contract
grammar_probe_e) were excluded from the MSan lane for "stack-overflow
under instrumentation". That was never deep recursion. MSan aborted while
REPORTING real uninitialized reads:
- every abort was on the main thread. MSan's "T<n>" is the OS thread id
and equalled the pid in all five x86-64 aborts (CI job 91868259558);
the old block read it as a worker-thread ordinal, so the worker-stack
knob, MSAN_ORIGINS and CBM_WORKERS=1 could never change anything;
- with the default slow unwinder (the image's MSan-instrumented
libunwind) a report exhausted the main thread's 256 MiB stack, and the
overflow handler then failed with "nested bug in the same thread";
- with fast_unwind_on_fatal=1 the same test prints the
use-of-uninitialized-value report instead (measured on one tree and
test: slow unwind aborts at the stack limit, fast unwind reports
wit/scanner.c:13).
The two reads are fixed in the previous commits of this series, so:
- MSAN_OPTIONS defaults to fast_unwind_on_fatal=1 (the build already
keeps frame pointers);
- MSAN_EXCLUDE is down to cli, whose install-path failure is a separate,
still-undiagnosed cause;
- the known-red block, the header (it still claimed CI runs without
exclusions) and the MSAN_ORIGINS and stack comments say what is true;
- the partial-lane warning and the exclude-name typo check lived in a
branch behind `elif true` that never ran. They now run in the
per-suite path, the check as a case match instead of a grep -q pipe
that pipefail can misreport;
- the comments in scripts/ci/msan-lane.sh and above the test-msan job
in _test.yml no longer say the five suites overflow.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The test filled 3675 commits' files and count but never their timestamp,
and cbm_compute_change_coupling reads each commit's timestamp to track a
pair's last co-change (pass_githistory.c:304). MemorySanitizer reports
the read; the stack array is now zero-initialized like every other
githistory test's commits. Production is unaffected: its commits come
from calloc and a zeroed parse state.
This was the read behind the pipeline suite's MSan "stack overflow"
(the x86-64 CI log of 2026-08-04 aborted in this test).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
tree_sitter_wit_external_scanner_create() is declared to return void *
but had an empty body. tree-sitter's runtime stores its result as the
scanner payload (parser.c ts_parser__external_scanner_create), so every
WIT parse used an indeterminate value, which C11 6.9.1p12 makes
undefined. The scanner never reads its payload, so ordinary builds were
unaffected, and the vendored build's -w hid -Wreturn-type.
MemorySanitizer reports it at wit/scanner.c:13 from grammar_labels,
grammar_probe_e, grammar_regression and lang_contract. Those four suites
were excluded from the MSan lane because the report itself crashed and
posed as a stack overflow (fixed in the msan.sh commit of this series).
Recorded in MANIFEST.md's local-patch table. Upstream
bytecodealliance/tree-sitter-wit still has the empty body at
059bfdd1af72, the last change to src/scanner.c.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Windows headers define far and near as empty macros, so
'const char *far = ...' became 'const char * = ...' on the CLANG64 leg.
The variable is far_src now.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Merge validated environment overrides into a child-owned environment on POSIX and Windows, preserving caller PATH lookup for executable names. Forward each ObjectScript export parse timeout to the extractor.
Cover inherited and replaced values, empty values, duplicate overrides, parent isolation, invalid names, executable lookup, and timeout forwarding.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Rename the reading workspace to CBM Lens Beta, distinguish Beta with an amber badge, and keep build diagnostics out of the header. Put architecture history beside the active context and record meaningful exploration steps.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up to #2590: three more waits that a timeout decided.
cli_activation_quiesce_does_not_wait_on_bootstrap_startup:
- Its participant child joined the version cohort with a 45 s deadline. It
now uses UINT64_MAX, which waits for locks and answers a conflicting
holder at once.
- The child watched for the installer's maintenance request for at most
120 s. It now watches until a definite event: the request, an UNSAFE/IO
answer, or the parent closing a new install-done pipe once its install
has returned. A failed install therefore ends the watch at once instead
of hanging, and a slow install is waited for instead of failing.
cli_install_into_host_namespace_still_drains_host_cohort:
- The "drained host stopped serving" probe was one status request with a
2 s budget, so a slow host that still served could pass as drained (a
false green). It now asks through the same mute-retry rule as the
positive probe: a drained host is unreachable or reports stopping at
once, and a slow host that still serves is asked again until it answers.
cli_scope_host_serving_within and CLI_SCOPE_HOST_DRAINED_PROBE_MS are
gone.
Checked by hand: with the install made to fail before it requests
maintenance (an unknown option), the test ends in 3 s through the
install-done pipe. Without the pipe and the old deadline it would hang;
with the deadline it took 120 s.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Markdown section that embeds a diagram is about its candidate functions
more often than the Markdown curve says, by how much depends on the
diagram's kind. 553 candidates from such sections in 16 held-out
repositories, judged blind in three rounds:
0.20-0.30 0.30-0.40 0.40+ local extras
prose 0.24 0.28 0.45 0.275
structural 0.32 0.46 0.57 0.78 (9 items)
Structural is Mermaid sequence/class/ER/C4/architecture and PlantUML
C4/class/sequence/use-case/component; prose any other Mermaid or PlantUML
kind. A section takes the highest family of the Diagram nodes it defines;
its function candidates at or inside the doc's home and its local extras
carry the family's value, and the first band is now stored. Outside
functions, files, folders, other formats and sections without a diagram
keep the Markdown curve.
On kubernetes' apiserver folder the sequence-diagram section's
candidates move from 0.22/0.51 to 0.46/0.57, the flowchart sections gain
five first-band candidates at 0.24, other sections and the graph are
unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up to #2589.
No timeout decides a cli_scope test any more:
- The status probe and the parent's client connect repeat an attempt that
reaches a live but mute endpoint (a loaded runner answering slowly). A
reply or a definite refusal decides. Each attempt keeps the API's finite
5 s budget. This replaces the 15 s status budget and the single 5 s
connect.
- The host child is reaped without a deadline (it used to get SIGKILL after
60 s).
- The host child joins its version cohort with UINT64_MAX, which waits for
locks but answers a conflicting holder at once (it used to have a 45 s
deadline).
A wedged child is left to the harness's per-suite ceiling.
cli_scope_fixture_finish restored HOME, CBM_CACHE_DIR and SHELL even when
start had failed before saving them, which unset them for every later test
in the suite. It now restores only what start saved.
New tests:
- cli_scope_fixture_finish_keeps_env_after_an_early_failure fails on main
at ASSERT(home_kept);
- cli_scope_retries_a_mute_endpoint_until_it_answers checks the retry rule
on scripted attempt outcomes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The held-out audit of the eight new languages is scored. Each tier is a
language and a reference form, read on a sample of 150 links (census below
150) from repositories never used to build or tune the resolvers. Three
blind reading rounds; bar: Wilson 95% lower bound >= 0.90 and point >= 0.95.
tier population judged correct Wilson low
go doc_link 5980 150 150 0.975
java link 23934 150 150 0.975
java linkplain 1372 150 150 0.975
java see 7711 150 150 0.975
java throws 650 150 150 0.975
java value 79 79 79 0.954
kotlin kdoc_link 3888 150 150 0.975
kotlin sample 279 150 150 0.975
kotlin see 369 150 150 0.975
kotlin throws 73 73 73 0.950
rust intra_doc 14119 150 149 0.963
rust link_def 1491 150 150 0.975
perl pod_L 2255 150 150 0.975
ts/js link 1222 150 150 0.975
php see 5171 150 150 0.975
c doxygen_ref 463 150 150 0.975
c see 509 150 145 0.924
These 17 families now ship: a resolved reference is a MENTIONS edge.
TS/JS {@linkcode} (10 held-out links) and @see (18) were all judged
correct, but are too few to reach the bar; they stay held, as do the
diagram and OpenAPI link families (3 links on 21 held-out repositories).
The Go ship-gate test turned its family off and restored only the held
families afterwards; with Go shipping it left Go off for the tests that
follow it. It restores the production gates first now.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Markdown Section's docstring is its body collapsed and cut at 500 bytes.
A diagram fence in that body put the format's syntax into it (`-->`,
`participant`, `classDiagram`, `subgraph`), which names no code, and pushed
the section's prose out of the cap. The doc -> code candidates are scored
on that text.
Each top-level diagram fence (mermaid, plantuml, puml, uml, dot, graphviz,
d2) is now replaced in the section's text by the labels its diagram parser
reads: the names of its elements, then its arrow labels, in source order.
The parse runs into a scratch result, so the file's own Diagram and
DiagramElement nodes are untouched. A section without such a fence keeps
its text byte for byte; other fences stay as written.
Measured before it was chosen, on held-out repositories (147 sections that
embed diagrams, 503 section/candidate pairs, three blind reading rounds):
fence as written 277 stored precision 0.45 42 sections with a correct candidate
labels (this) 310 stored precision 0.54 56 sections with a correct candidate
fence dropped 127 stored precision 0.54 29 sections with a correct candidate
Labels gave 20 sections a correct candidate that the fence as written did
not, and lost 6 (exact McNemar p = 0.009). Dropping the fence lost more
than it gained.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cli_install_skip_binary_unchanged_in_host_namespace_quiesces_nothing
failed once on 2026-10-09 at ASSERT(ready), with nothing else in the log.
cli_scope_fixture_start returned false silently from any of its setup
steps, and a 90 s deadline decided the wait for the host child's
readiness.
The fixture now records and prints the step that failed: the fixture
directory, the host directories, the runner's own image fingerprint, the
pipes, the fork, the host's start, or the client connect, with the OS error
where one applies. The host child prints which of its own steps failed
before it answers 'E'.
The readiness wait has no deadline any more, because no timeout may decide
a test. The child answers 'R' or 'E', or the pipe reaches EOF when it dies.
A child that wedges is left to the harness's per-suite ceiling.
New test: cli_scope_fixture_names_its_failing_setup_step.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On macOS the signed-candidate preparation folded every step into one
boolean, so each failure printed the same line: "signed update candidate
preparation failed". A full /tmp read exactly like a broken codesign. That
is what a cli test hit on 2026-10-09, while a multi-GB index was filling a
nearly full disk.
Each step now records why it failed, and the update prints, for example:
error: signed update candidate preparation failed while writing the
unsigned candidate: No space left on device
The staging functions in activation_transaction.c keep the errno of the
failing write (or sync, close, read) through their cleanup, so a caller can
name it. The non-macOS path adds the OS error to its existing "failed to
stage verified update" line (POSIX only; Windows reports through
GetLastError). A failed fingerprint of the staged file now gets its own
message instead of "...: activation transaction completed".
A test seam makes the next staging write fail with a chosen errno. The new
test cli_update_names_the_failing_staging_step_and_os_error fails without
the cli.c change and passes with it, and checks that the old executable
stays in place. Also checked on real input: with /tmp on a small volume
that fills between two updates, the update prints the line above.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The TS/JS, PHP, C and Perl doc-link layers keep a scope blob per file but
had no scope_delta hook, so any change of a file's declarations made the
incremental closure repair decline (doc_scope_changed) and the index
rebuild in full: removing a TypeScript function forced a full run.
cbm_doclinks_scope_delta_lines compares two blobs line by line in order,
as C# does: a dropped record that declares one name is repaired file by
file (the name is reported, so the files whose references named it are
re-resolved); anything new or changed, and every record others resolve
through, stays a full run. The records that declare one name: TS D
(a top-level declaration), PHP F and K (a function, a member), C D of a
function, variable, enum constant, field or macro and A (a Doxygen
anchor), Perl S (a sub).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Diagrams name the parts of a system and how they connect; an OpenAPI
document names its endpoints and their handlers. Both now enter the graph.
Diagrams:
- a Diagram node per diagram: a fenced block of a Markdown file (under its
Section), a directive of a reST file (`.. mermaid::`, `.. uml::`,
`.. graphviz::`, Sphinx `.. digraph::`), an Asciidoctor Diagram block of
an AsciiDoc file (`[plantuml]`, `[source,mermaid]`, ...), or a diagram
file of its own, named by its title, else by its file;
- a DiagramElement node per element, DEFINES from its Diagram, and a
DIAGRAM_ARROW edge per arrow (label, style, line);
- formats: Mermaid (flowchart, sequence, class, ER, state, C4,
architecture), PlantUML, Graphviz DOT, D2, draw.io (plain, compressed,
.drawio.svg) and Excalidraw (.excalidraw, .excalidraw.json,
.excalidraw.svg); .drawio.svg and .excalidraw.svg are no longer dropped
as images;
- an element's explicit link (Mermaid click/link, PlantUML [[url]], url of
and !include, draw.io and Excalidraw link, DOT URL/href, D2 link) is a
doc link read as a Markdown link, MENTIONS when it resolves.
OpenAPI:
- each operation of an OpenAPI or Swagger document (openapi.json and
swagger.json, now read with the JSON grammar but no node per key, and
any YAML or JSON file whose root says openapi/swagger) is a Route, named
by the same QN a route its handler registers in code gets
(cbm_route_canon_path, moved into the extraction library);
- its operationId is a doc link to the handler: a dotted one (Connexion,
with x-openapi-router-controller) as a qualified code name, a bare one
when exactly one function or method of the repository has that name.
The eight new link families are held until their held-out audit passes.
Read on real repositories, which shaped the rules: Excalidraw drawings put
text over shapes and draw arrows up to them without binding, so loose text
names the smallest box it sits in and a loose arrow end meets the smallest
named box it touches (the drawing's frame only at its border); arrows bound
to free text meet the text; D2 block strings and nested blocks; DOT line
breaks and sentence-long ids; PlantUML kinds of classes inside packages
and nested classes (`..+`, nested -> container, so they do not collide with
the container's association). Connexion's 24 specs: 160 operationIds, all
linked to their handlers.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
rst_body cut a reST section's text at 500 bytes by setting a stop value
past the end, then backing off from out[RST_BODY_MAX] while that byte
looked like a UTF-8 continuation byte. That byte was never written, so
where the text ended (499 or 500 bytes) depended on what the arena memory
held, and when a multibyte character did not fit, bytes never written
could end up in the text.
Only whole characters are written now, and the text ends at the last one
written. The same django tree indexed twice gave doc_link_candidates rows
that differed in about 250 of 11,300 (through the section text's TF-IDF);
with this change two runs are identical, and so are the 6,594 sections.
The new test pins the cut at the boundary: a word that fills the 500 bytes,
a two-byte character that would cross them, a word cut at a character.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The doc-section candidate pass (#2574) made a Linux kernel index 2.8x
slower: 207 s before #2574, 580 s after, on the same machine. Every
candidate of every section got a dot product that merged the section's up
to 1024 terms with the function's, and then every candidate was sorted for
a top 10.
- The section's vector is held densely per worker, indexed by term and
stamped per section, so a candidate's dot product walks only the
function's (or file's) own terms. The same products are added in the
same ascending term order, so every score is the same float.
- Only the candidates the results read are sorted: the top
SEM_DOC_TOP_K (and the local ones down to CBM_SEM_DOC_MIN_SCORE), the
top CBM_SEM_DOC_FILE_K files. cmp_doc_cand is a total order, so the
sorted prefix is the full sort's prefix.
Kernel (perf-bench/linux, M5 Pro, same plain build): 579.7 s -> 391.5 s,
nodes and edges unchanged. Exactness, measured directly: a build that also
ran the old merge and the old full sort found 0 score, 0 shared-term and
0 order differences over 5,003 django sections.
Candidate generation itself (a section's key terms, df <= funcs/20) is
unchanged here: narrowing it would change the stored rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A bare import specifier was always external. The TS resolver now asks the
pipeline what it already knows for import resolution, without indexing
package.json or tsconfig.json: the nearest tsconfig/jsconfig `paths` and
`baseUrl` (the alias collection; the first target, as TypeScript tries it
first), then the workspace package whose `name` the specifier is (the
package map), bound to the package's entry module (`exports`, `main`,
`module`) when the repository indexes it. Every other bare specifier, a
package's subpath included, stays external. Incremental runs stay exact:
a tsconfig change re-resolves the files under it, a package.json change is
a manifest change.
path_alias gets cbm_path_alias_resolve_into, which writes into the caller's
buffer; cbm_path_alias_resolve now wraps it (two raw allocations fewer, the
memory-core baseline lowered to match).
Also fixed: a declaration in an index file has no `index` in its QN (fqn.c
strips it from a symbol's QN, not from the module's), so a reference to one
missed its node; the resolver now names a file's declarations with that
prefix.
Proof (ship build, Apple M5 Pro / 48 GB): the TypeScript compiler
b465fdbfe175 unchanged (69 edges, 57 of the prototype's 61 pairs);
discord.js 4dc1acc6 unchanged (no doc reference names a workspace import;
its packages' entries are build output). Tests: a tsconfig alias, a
workspace package's entry and an index file's declaration each bind, each
red with its rule reverted; path_alias, pipeline, edge_imports, incremental
and all doc suites pass (1298).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
phpDocumentor reads `@see name()` as a function. Authors write it for the
documented class's own method: symfony has 97 such references that ended as
external rows (Command `@see setCode()`, ProgressBar `{@see resumeAll()}`),
and one that bound a global build-script function (ProcessHelper `@see
run()`, meaning its own run()). In the doc of a type or of one of its
members, a bare name() is now a method the type declares or inherits; a
`use function` import of the name still wins, and outside a type the
function rules stay. The ambiguity guard this replaces is gone.
Proof (ship build, Apple M5 Pro / 48 GB): symfony 9917e115 +91 edges
(external bare name() rows 97 -> 6), cakephp a75828db +6 (7 -> 1); a sample
of 14 new edges read against their lines: all correct, inherited methods
included (UploadedFile -> File::guessExtension).
Test: doc_mentions_php_current_class (the method wins over a global function
of the same name; the documented method itself is a self-mention), red with
the rule reverted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
PHP properties, class constants and enum cases have no graph node, so their
doc blocks were never read: cakephp has 36 @see references there, symfony
21. The PHP leg now uses the driver's loose_docs hook: a /** */ block right
above a property, constant or case is harvested with the type around it as
the source (as the field test's prototype attached it), and resolved like
any other reference written in that type (self:: is the type).
The driver marks as the file's only what a hook pushes with the file
stand-in (the C leg's prototype docs); a hook may push with a definition of
the file instead.
Proof (ship build, Apple M5 Pro / 48 GB, base 80e000a9):
- koel 04ed8beaec34: all 5 of the prototype's pairs bound alike (the fifth
sits in a class constant's doc).
- cakephp a75828db: 373 edge mentions + 44 rows = every one of the 417
non-URL references (was 381 of 417).
- symfony 9917e115: 459 + 219 = 678 of 681 (the other 3 sit in blocks before
code the census does not classify).
Tests: doc_mentions_php resolves a property's doc block to an edge from its
class; red with the hook's walk reverted, and red when the driver marks
every token of the file pass as the file's.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Doxygen comment in a .c or .h file (a block opening with slash-star-star
or slash-star-bang, a run of /// or //! lines) names code with `@ref name`
or `\ref name` (family doxygen_ref) and with the names an `@see`, `\see`,
`@sa` or `\sa` paragraph lists that have the shape of a symbol (`name`,
`name()`, `::name`, `Type::member`; family see). Code blocks, fences, code
spans, quoted labels and the word after \c, \p, \a, \b, \e, \em hold none; a
URL is the core's href family. A .h is parsed as C++, which does not change
the rule; other C++ and Objective-C files are not part of the leg.
C keeps no node for a function prototype, so the docs of a header's API
were never stored anywhere. The doc-link driver gets an optional loose_docs
hook, called after the definitions and the file's own doc: the C leg reads
every Doxygen block outside function bodies that no definition took, and
its references belong to the file (the File node is the source).
The scope ("c1") lists the definitions with nodes (function, variable,
type, enum constant, field, object-like and function-like macro) with their
qualified names, their linkage (static, or a type, constant or macro of a
.c file, is the file's own) and the parent of a constant or field, and the
pages, groups, anchors and sections the file's docs define.
doc_links_c.c resolves a name as Doxygen does, without imports:
- a page, group, anchor or section name is no code entity (graph_gap) and
never binds a symbol of the same name;
- then the documented file's own definitions (static ones too), its fields,
then what every file sees (external linkage, a header's types, constants
and macros); a static definition of another file is not visible;
- candidates count by qualified name (a prototype and its definition, #if
variants: one node); with `()` written a function or function-like macro
wins, then a definition wins over a macro;
- `Type::member`: a field of the struct or union Type, or a constant of the
enum Type;
- one candidate binds, several are ambiguous; a word of an @see paragraph
that names nothing and is a plain lowercase word is prose (Doxygen prints
it as text): no row; test code binds only from test code.
There is no scope_delta hook yet: a declaration or linkage change is a full
run.
The families are held (ships=false) until their held-out audit passes; the
suite ships them through the test seam.
Proof (ship build, Apple M5 Pro / 48 GB, base 7a321cac):
- xxhash (redis 4f20cb489344, a clean copy), the field test's 174 audited
links, 110 of which had hit a rename macro before C1: every harvested one
now names the right entity (169 edges, 2 self-mentions: XXH_NAMESPACE's
own doc); the other 3 are @copydoc, which this leg does not harvest.
- redis: doxygen_ref 74 edges (179 mentions), see 25 (30); rows: graph_gap
50, missing 6, ambiguous 3, unparseable 1. The prose rule took the 75
rows of words like "for" and "details" out.
- pico-sdk 079c6f39: doxygen_ref 488 (730), see 199 (219); rows: graph_gap
142 (groups and pages), missing 94, ambiguous 68 (one type per chip,
rp2040 and rp2350), test_only_target 28 (assembly functions with only a
C test version), unparseable 5.
- libusb abba63ed: doxygen_ref 256 (284), see 39 (44); rows: graph_gap 62,
missing 18, unparseable 2, ambiguous 2. 46 of the edges are
Type::member references.
- libevent d82464a2: see 176 (230); rows: missing 9.
- a random sample of 24 edges (8 per corpus, pico-sdk, libusb, libevent)
read against their source lines: all correct, among them a header
prototype's doc linking to the definition in the .c file.
Finding outside the leg: libevent's event_assign has no node (its return
type and name are split by an #if block), so its references are missing.
Tests (doc_mentions_c): tokens with their lines and the file-level source
of a prototype's doc; C++ files out, headers in; the scope blob; every
resolution rule; incremental equals full (a doc edit and a body edit repair
one file, a definition made static is a full run); 60 modules give the same
edges and rows with one worker and several. Each rule red with it reverted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A PHP doc block names code with `@see Ref` at the start of a tag line (a
description may follow) and with inline `{@see Ref}`, both family `see`.
doclink_php.c harvests them (code spans and fenced blocks hold none; @link,
@uses and the other tags are not harvested; a URL is the core's href family)
and writes each file's scope: namespace blocks, use imports, types with
their supertypes already qualified in their own file, members, functions.
The persisted scope drops lines and imports: other files read names only.
Declarations inside a top-level conditional count as top level
(`if (!function_exists('f')) { function f() {} }`), never a function body.
doc_links_php.c resolves a reference as a FQSEN with PHP's own name rules:
- a leading backslash is taken as written; otherwise the first segment is
replaced when it is the alias of a class `use` of the namespace block the
reference is written in, else the block's namespace is put in front
(phpDocumentor's resolver);
- beyond that tool, as PHP itself has them: an unqualified function name is
taken from a `use function` import, else from the current namespace, else
from the global namespace; and self/static/parent name the documented
type and its superclass;
- a bare `name()` that would bind a function but is written in a type that
declares or inherits a method of that name is ambiguous (the method or the
function; the reference does not say which);
- members are looked up on the type, its traits, its superclass and its
interfaces (depth first); type, function and method names compare without
case, properties and constants with it;
- properties, class constants and enum cases have no graph node: graph_gap;
so has a type or function the graph folds into another one of the same
name in its file (two namespace blocks), which the scope flags;
- a name the repository does not declare is external outside the
repository's namespace roots (built-ins, Composer dependencies), else
missing. Residual: a function a dependency declares in the current
namespace is invisible, so the fallback would bind a global function of
that name.
There is no scope_delta hook yet: a declaration change is a full run.
The family is held (ships=false) until its held-out audit passes; the suite
ships it through the test seam.
Proof (ship build, Apple M5 Pro / 48 GB, base 196ca71cda):
- koel 04ed8beaec34 vs the field test's prototype: 4 of its 5 pairs bound
alike; the fifth sits in a class constant's doc block, which has no node.
- cakephp a75828db: 342 edge mentions + 39 unresolved rows = every one of
the 381 non-URL references in function and type doc blocks; 36 more sit
in property and constant doc blocks (no node, not harvested).
- symfony 9917e115: 452 edge mentions (434 edges) + 205 rows = all 657; 24
in property, constant and other doc blocks.
- edges only through the global fallback: 1 (cakephp `@see debug()` in
Cake\Core, the global debug() helper: correct); one more on symfony was
wrong (`@see run()` meaning the class's own method bound a global script
function) and the method-name rule above makes it ambiguous. Through
self/static/parent: 20, all correct (12 + 5 self, 3 parent).
- a random sample of 24 edges (12 per corpus) read against their source
lines: all correct.
Tests (doc_mentions_php): tokens with their lines; the scope blob, its
persisted form, namespace blocks and conditional declarations; every
resolution rule; the global fallback and self/static/parent each in a test
of its own; incremental equals full (a doc edit and a body edit repair one
file, a removed function is a full run); 60 modules give the same edges and
rows with one worker and several. Each rule red with it reverted (global
fallback, self/static, parent, trait members, the folded-name gap, use
aliases, method names without case, the bare-name method lookup, the
namespace roots, code spans, the method-name guard, conditional
declarations).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
81264814 built the production allocator as mimalloc's release build
(-DMI_DEBUG=0). That bought a large speed-up, but it raised what macOS
charges an index worker, its phys_footprint:
| peak phys_footprint | v0.11.0 | main |
|---|---:|---:|
| TypeScript | 1.25 GB | 1.94 GB |
| dotnet/runtime | 4.5 GB | 12.7 GB |
The 2026-10-09 bench missed most of this, because it sampled RSS.
The difference is one compile-time branch in mimalloc's unix decommit.
With MI_DEBUG or MI_SECURE>2, a purged range is mprotect(PROT_NONE)'d
and flagged needs_recommit. A release build leaves it "committed".
When the arena hands such a range out again, it is re-accounted as a
whole with MADV_FREE_REUSE (arena.c, _mi_os_reuse). macOS then charges
every page of it to the footprint, touched or not.
On macOS, take the recommit path in every build. Linux and Windows keep
upstream's behaviour; there MADV_DONTNEED already drops RSS at once.
Measured on Apple arm64 (macOS 26.6), main ec4fbd3a vs this change, cold
full indexes interleaved with main (C#: 2 runs each, in the same session
as main):
| corpus | footprint before | footprint after | change | wall time |
|---|---:|---:|---:|---|
| TypeScript | 1.98 GB | 1.09 GB | -45 % | unchanged |
| kubernetes | 2.92 GB | 1.88 GB | -35 % | unchanged |
| dotnet/runtime | 12.6 GB | 4.9 GB | -61 % | unchanged |
| linux kernel | 16.0 GB | 13.5 GB | -16 % | 574 -> 559 s |
The graph is byte-identical: TypeScript, kubernetes and django give
the same nodes, edges and CALLS-set hash.
dotnet/runtime is now close to v0.11.0's 4.5 GB.
Measured but not taken:
- the whole debug build, which costs ~5 % wall time on C#;
- MI_STAT=2, MI_PADDING=1 or MI_ENCODE_FREELIST=1 alone, none of which
moves the footprint;
- MIMALLOC_PURGE_DELAY=0, which is already set.
Vendored file changed in place (precedent 1f674c80); its hash in
scripts/vendored-checksums.txt is updated and security-vendored.sh passes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A TypeScript, TSX or JavaScript doc comment (a block comment opening with a
slash and two stars) names code with {@link Ref}, {@linkcode Ref} (each with
an optional `|` label) and `@see Ref` at the start of a tag line that holds
no inline tag. doclink_ts.c harvests them (code spans and fenced blocks hold
none, {@linkplain} is not harvested, a hyphenated word after @see is prose, a
reference naming the declaration's own parameter is dropped, a URL is the
core's href family) and writes each file's scope: imports, exports, export *
barrels, top-level declarations, namespace bodies and functions or classes
declared inside a function body.
doc_links_ts.c resolves a reference as the TypeScript checker sees it from
the documented declaration, in one index over TS, TSX and JS files:
1. the namespaces around the reference, innermost first;
2. a function or class declared in the enclosing function body (the graph
gives it a file-level name: two of one name in one file share one node,
a graph gap);
3. the file's own top-level declarations;
4. its imports, followed through export { a as b }, export ... from and
export * barrels (cycle guard, depth bound; two barrels that answer
differently are ambiguous); a relative specifier resolves with the usual
extension and index rules, a bare one is external;
5. the standard library's global names: external;
6. the top-level declarations of script files (no import, no export): one
binds, several are ambiguous;
7. the members of the documented class or interface, `this.` included.
The other segments walk members or a module's exports. A member declared
without a node of its own is a graph gap; test code binds only from test
code. There is no scope_delta hook yet: an export change is a full run.
The families are held (ships=false) until their held-out audit passes; the
suite ships them through the test seam.
Proof: the TypeScript compiler (perf-bench/typescript b465fdbfe175), ship
build, against the field test's prototype records (cascade results left
out): 69 edges; 57 of the prototype's 61 pairs bound alike. The 4 others are
doc blocks without a definition of their own (on an `if` statement, a local
`var`, an enum member, a free-standing @typedef block), which the core does
not harvest. Of the 12 edges the prototype has no pair for, 4 are the same
targets the prototype resolved without recording a source node, and 8 bind
PropertyDescriptor to the repository's own src/lib/es5.d.ts, which the
prototype called an external library global (kept: the repository declares
it). Unresolved rows agree with the prototype's reasons apart from the
documented reason split (not_in_scope -> missing/ambiguous).
Known limit: package.json and tsconfig.json are not indexed (discovery
ignores them), so workspace packages and tsconfig `paths` stay external.
Tests (doc_mentions_ts): tokens with their lines; the scope blob and its
persisted form; every resolution rule, including a function-body
declaration that binds and two of one name that do not; incremental equals
full (a doc edit and a body edit repair one file, an export change is a full
run); 60 modules give the same edges and rows with one worker and several.
Each rule red with it reverted (the body declaration's visibility and its
line check, barrel ambiguity, the parameter rule, code spans, the library
globals).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
Bernhard Jackiewicz