mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-06 16:53:56 +00:00
a35671e962f1f2fb4e94deaf55e71cf05f6b3828
149
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
6f9374281f |
Merge main a6d0d6c9 into the strict shared resolver
Threads main's Go rel_path (#1910) through the proof-source and import-snapshot variants, keeps main's ADR sidecar so the rebuild no longer re-applies ADRs, and keeps the feature's removal of the import name fallback and JVM tail join, inside which main had added a generated-member filter. Main took semantic index version 12 for Markdown Section names, so the resolver generation becomes version 13. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b821d1506f |
Complete strict resolver proof bridges and type propagation
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ebdf9bafd5 |
test(mem): unpoison mimalloc output before the handler reads it
The MSan output handler checked msg[0] before marking the text initialized, so MSan reported the handler's own read (tests/test_main.c:2151). Unpoison first, then test for the empty replay. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
642fe3fe46 |
Complete proof bridges and migrate resolver quality contracts
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0f8b46570c |
test(mem): unpoison mimalloc output in the MSan test runner
The MSan lane compiles mimalloc without -fsanitize=memory on purpose, so MSan never sees mimalloc write its own stack. Its text output is the one place those stack bytes reach an MSan interceptor: a message formatted in a mimalloc stack buffer arrives at fputs still carrying the shadow an earlier instrumented frame left there (cbm_log's line_buf), and MSan reports initialized bytes as uninitialized. mimalloc 3.5.3 prints an arena-alignment fallback warning on the x86-64 MSan lane, which turned this into a use-of-uninitialized-value report in the mem suite (_mi_prim_out_stderr, 43 bytes: the per-thread prefix). Under __has_feature(memory_sanitizer) only, the test runner now registers a mimalloc output handler at the top of main that unpoisons the finished text and writes it to stderr. Registering replays mimalloc's delayed-output buffer, which its constructor already wrote to stderr before main, so the handler skips that replay instead of repeating it behind a stray newline. Production code and every non-MSan build are unchanged. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
891d6eb249 |
Require occurrence-bound evidence for cross-file edges
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
75f45ba248 |
Add shared semantic resolution with explicit proof admission
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
67ff40d5cb |
fix(kotlin): resolve calls through typed class properties (#1555)
The cross-file Kotlin walk resolved a call on a typed method parameter by type, but a call on a class property - a primary-constructor val, a body property or `this.prop` - fell to the name guess. The cross registry held each own-file class from its extraction defs, whose properties carry no type (constructor vals none at all), and the walk's typed AST registration of the same class was added behind that entry under the same QN, so every lookup returned the field-less one. Constructor injection is the common Kotlin dependency shape, so `repository.inserir(...)` on an injected port bound to whichever same-named method the guess found, including an implementation of another port (#1555). The walk now registers the file's own declarations into an overlay chained to the cross registry and checked first, indexed once the top-level declarations are collected. The sole-definer fallback walks the whole chain so its candidate set is unchanged. Typing more receivers exposed resolution gaps that the untyped receivers had hidden; each would otherwise turn a typed call into a wrong name guess: - a member of the receiver class wins over a same-named extension of the same file (Kotlin resolution order; the overlay holds both); - `x.hashCode()` / `toString()` / `equals()` on a receiver typed by another file: the per-file pass emits the kotlin.Any placeholder, the cross pass the declared override, and two targets at one occurrence are ambiguous to the call join. The per-file placeholder is dropped when the cross pass resolves the same occurrence; - the same members on a stdlib or JDK type (`count.toString()` on an Int) resolve to the kotlin.Any builtin instead of a seeded stdlib member that has no graph node; - a property READ on a typed receiver whose QN the graph spells differently (a nested class's property) never becomes a CALL_REFERENCE to a same-named function: the reference join falls back by name to non-callable targets only, which binds the property's own node; - a typed value without an `invoke` operator (`private val handshake: Handshake?`) no longer shadows the same-named member function; - inside a lambda whose typed receiver becomes `this` (`repo.let { .. }`), and inside a member extension, the enclosing class's members stay callable bare (the implicit-receiver chain); a call naming a class stays a constructor call. Fixes #1555 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7f6f567840 |
wip: Smart CI narrows PR CI for real (pre-rebase checkpoint)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
8b4d5e8b5b |
perf(store): in-memory adjacency layer for trace_path and degree filters
On a kernel-scale index (8.5M nodes, 16.9M edges) trace_path's recursive CTE and search_graph's per-row degree subqueries spend seconds in B-tree seeks. graph_mem reads the edges once, in one sequential scan of the covering UNIQUE index, into a typed CSR (node id -> out/in neighbours with interned edge types) and answers both from RAM: - store_bfs (shortest-path mode): BFS over the layer with the CTE's exact semantics (minimal hop per node, the start only via a cycle back to it, (hop, id) order, probe row, JOIN-skips for ids without a node row); node rows and edge collection still come from SQL. Trail mode is unchanged. - cbm_store_search with min/max_degree: the two correlated COUNT(*) subqueries become cbm_adj_degree(); filter, order, page and total are the same SQL. Layers live in a process-wide registry (the daemon closes its stores after every request), refcounted across request threads, allocated through the memory core (CBM_MEM_CLASS_STORE) and bounded: all layers stay under a quarter of the budget, idle ones are evicted oldest-first, and a layer that does not fit is not built (SQL path, never a partial layer). A layer is keyed by db path + store generation + the rollback-journal file state (the header change counter every rollback commit bumps), read in one read transaction with the scan; only read-only query stores of sealed DBs qualify. The file state matters: the cross-repo pass writes CROSS_* edges into other projects' published DBs in place without a new generation; such a DB is in WAL mode and keeps the SQL path until the next publish. Kernel, macOS, same index, fresh daemon: trace_path schedule 7.5 s -> 21 ms warm (1.23 s first call incl. a 1.2 s build), search_graph Function min_degree 50 4.2 s -> 170 ms warm (1.35 s first). Layer: 16,890,273 edges, 237 MB. Responses byte-identical to the baseline on django, redis and the kernel (tools, Cypher and extra BFS/degree cases). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
4d9592b5fb |
Merge pull request #2068 from DeusData/feat/codeatlas-web
CBM Lens Beta: graph exploration and architecture workspace |
||
|
Martin Vogel
|
0392068d49 |
Merge main into CBM Lens Beta
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
afeaca0217 |
Complete child-local environments and ObjectScript parse deadlines
Merge validated environment overrides into a child-owned environment on POSIX and Windows, preserving caller PATH lookup for executable names. Forward each ObjectScript export parse timeout to the extractor. Cover inherited and replaced values, empty values, duplicate overrides, parent isolation, invalid names, executable lookup, and timeout forwarding. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
f4a49feba1 |
feat(docs): diagrams and OpenAPI operations in the graph
Diagrams name the parts of a system and how they connect; an OpenAPI document names its endpoints and their handlers. Both now enter the graph. Diagrams: - a Diagram node per diagram: a fenced block of a Markdown file (under its Section), a directive of a reST file (`.. mermaid::`, `.. uml::`, `.. graphviz::`, Sphinx `.. digraph::`), an Asciidoctor Diagram block of an AsciiDoc file (`[plantuml]`, `[source,mermaid]`, ...), or a diagram file of its own, named by its title, else by its file; - a DiagramElement node per element, DEFINES from its Diagram, and a DIAGRAM_ARROW edge per arrow (label, style, line); - formats: Mermaid (flowchart, sequence, class, ER, state, C4, architecture), PlantUML, Graphviz DOT, D2, draw.io (plain, compressed, .drawio.svg) and Excalidraw (.excalidraw, .excalidraw.json, .excalidraw.svg); .drawio.svg and .excalidraw.svg are no longer dropped as images; - an element's explicit link (Mermaid click/link, PlantUML [[url]], url of and !include, draw.io and Excalidraw link, DOT URL/href, D2 link) is a doc link read as a Markdown link, MENTIONS when it resolves. OpenAPI: - each operation of an OpenAPI or Swagger document (openapi.json and swagger.json, now read with the JSON grammar but no node per key, and any YAML or JSON file whose root says openapi/swagger) is a Route, named by the same QN a route its handler registers in code gets (cbm_route_canon_path, moved into the extraction library); - its operationId is a doc link to the handler: a dotted one (Connexion, with x-openapi-router-controller) as a qualified code name, a bare one when exactly one function or method of the repository has that name. The eight new link families are held until their held-out audit passes. Read on real repositories, which shaped the rules: Excalidraw drawings put text over shapes and draw arrows up to them without binding, so loose text names the smallest box it sits in and a loose arrow end meets the smallest named box it touches (the drawing's frame only at its border); arrows bound to free text meet the text; D2 block strings and nested blocks; DOT line breaks and sentence-long ids; PlantUML kinds of classes inside packages and nested classes (`..+`, nested -> container, so they do not collide with the container's association). Connexion's 24 specs: 160 operationIds, all linked to their handlers. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
80e000a92a |
doc-links(c): Doxygen @ref and @see names become MENTIONS edges, held until their audit
A Doxygen comment in a .c or .h file (a block opening with slash-star-star
or slash-star-bang, a run of /// or //! lines) names code with `@ref name`
or `\ref name` (family doxygen_ref) and with the names an `@see`, `\see`,
`@sa` or `\sa` paragraph lists that have the shape of a symbol (`name`,
`name()`, `::name`, `Type::member`; family see). Code blocks, fences, code
spans, quoted labels and the word after \c, \p, \a, \b, \e, \em hold none; a
URL is the core's href family. A .h is parsed as C++, which does not change
the rule; other C++ and Objective-C files are not part of the leg.
C keeps no node for a function prototype, so the docs of a header's API
were never stored anywhere. The doc-link driver gets an optional loose_docs
hook, called after the definitions and the file's own doc: the C leg reads
every Doxygen block outside function bodies that no definition took, and
its references belong to the file (the File node is the source).
The scope ("c1") lists the definitions with nodes (function, variable,
type, enum constant, field, object-like and function-like macro) with their
qualified names, their linkage (static, or a type, constant or macro of a
.c file, is the file's own) and the parent of a constant or field, and the
pages, groups, anchors and sections the file's docs define.
doc_links_c.c resolves a name as Doxygen does, without imports:
- a page, group, anchor or section name is no code entity (graph_gap) and
never binds a symbol of the same name;
- then the documented file's own definitions (static ones too), its fields,
then what every file sees (external linkage, a header's types, constants
and macros); a static definition of another file is not visible;
- candidates count by qualified name (a prototype and its definition, #if
variants: one node); with `()` written a function or function-like macro
wins, then a definition wins over a macro;
- `Type::member`: a field of the struct or union Type, or a constant of the
enum Type;
- one candidate binds, several are ambiguous; a word of an @see paragraph
that names nothing and is a plain lowercase word is prose (Doxygen prints
it as text): no row; test code binds only from test code.
There is no scope_delta hook yet: a declaration or linkage change is a full
run.
The families are held (ships=false) until their held-out audit passes; the
suite ships them through the test seam.
Proof (ship build, Apple M5 Pro / 48 GB, base
|
||
|
Martin Vogel
|
7a321cac16 |
doc-links(php): PHPDoc @see references become MENTIONS edges, held until their audit
A PHP doc block names code with `@see Ref` at the start of a tag line (a
description may follow) and with inline `{@see Ref}`, both family `see`.
doclink_php.c harvests them (code spans and fenced blocks hold none; @link,
@uses and the other tags are not harvested; a URL is the core's href family)
and writes each file's scope: namespace blocks, use imports, types with
their supertypes already qualified in their own file, members, functions.
The persisted scope drops lines and imports: other files read names only.
Declarations inside a top-level conditional count as top level
(`if (!function_exists('f')) { function f() {} }`), never a function body.
doc_links_php.c resolves a reference as a FQSEN with PHP's own name rules:
- a leading backslash is taken as written; otherwise the first segment is
replaced when it is the alias of a class `use` of the namespace block the
reference is written in, else the block's namespace is put in front
(phpDocumentor's resolver);
- beyond that tool, as PHP itself has them: an unqualified function name is
taken from a `use function` import, else from the current namespace, else
from the global namespace; and self/static/parent name the documented
type and its superclass;
- a bare `name()` that would bind a function but is written in a type that
declares or inherits a method of that name is ambiguous (the method or the
function; the reference does not say which);
- members are looked up on the type, its traits, its superclass and its
interfaces (depth first); type, function and method names compare without
case, properties and constants with it;
- properties, class constants and enum cases have no graph node: graph_gap;
so has a type or function the graph folds into another one of the same
name in its file (two namespace blocks), which the scope flags;
- a name the repository does not declare is external outside the
repository's namespace roots (built-ins, Composer dependencies), else
missing. Residual: a function a dependency declares in the current
namespace is invisible, so the fallback would bind a global function of
that name.
There is no scope_delta hook yet: a declaration change is a full run.
The family is held (ships=false) until its held-out audit passes; the suite
ships it through the test seam.
Proof (ship build, Apple M5 Pro / 48 GB, base
|
||
|
Martin Vogel
|
196ca71cda |
doc-links(ts): JSDoc and TSDoc links become MENTIONS edges, held until their audit
A TypeScript, TSX or JavaScript doc comment (a block comment opening with a
slash and two stars) names code with {@link Ref}, {@linkcode Ref} (each with
an optional `|` label) and `@see Ref` at the start of a tag line that holds
no inline tag. doclink_ts.c harvests them (code spans and fenced blocks hold
none, {@linkplain} is not harvested, a hyphenated word after @see is prose, a
reference naming the declaration's own parameter is dropped, a URL is the
core's href family) and writes each file's scope: imports, exports, export *
barrels, top-level declarations, namespace bodies and functions or classes
declared inside a function body.
doc_links_ts.c resolves a reference as the TypeScript checker sees it from
the documented declaration, in one index over TS, TSX and JS files:
1. the namespaces around the reference, innermost first;
2. a function or class declared in the enclosing function body (the graph
gives it a file-level name: two of one name in one file share one node,
a graph gap);
3. the file's own top-level declarations;
4. its imports, followed through export { a as b }, export ... from and
export * barrels (cycle guard, depth bound; two barrels that answer
differently are ambiguous); a relative specifier resolves with the usual
extension and index rules, a bare one is external;
5. the standard library's global names: external;
6. the top-level declarations of script files (no import, no export): one
binds, several are ambiguous;
7. the members of the documented class or interface, `this.` included.
The other segments walk members or a module's exports. A member declared
without a node of its own is a graph gap; test code binds only from test
code. There is no scope_delta hook yet: an export change is a full run.
The families are held (ships=false) until their held-out audit passes; the
suite ships them through the test seam.
Proof: the TypeScript compiler (perf-bench/typescript b465fdbfe175), ship
build, against the field test's prototype records (cascade results left
out): 69 edges; 57 of the prototype's 61 pairs bound alike. The 4 others are
doc blocks without a definition of their own (on an `if` statement, a local
`var`, an enum member, a free-standing @typedef block), which the core does
not harvest. Of the 12 edges the prototype has no pair for, 4 are the same
targets the prototype resolved without recording a source node, and 8 bind
PropertyDescriptor to the repository's own src/lib/es5.d.ts, which the
prototype called an external library global (kept: the repository declares
it). Unresolved rows agree with the prototype's reasons apart from the
documented reason split (not_in_scope -> missing/ambiguous).
Known limit: package.json and tsconfig.json are not indexed (discovery
ignores them), so workspace packages and tsconfig `paths` stay external.
Tests (doc_mentions_ts): tokens with their lines; the scope blob and its
persisted form; every resolution rule, including a function-body
declaration that binds and two of one name that do not; incremental equals
full (a doc edit and a body edit repair one file, an export change is a full
run); 60 modules give the same edges and rows with one worker and several.
Each rule red with it reverted (the body declaration's visibility and its
line check, barrel ambiguity, the parameter rule, code spans, the library
globals).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
f4277ae783 |
doc-links(perl): POD L<> codes become MENTIONS edges, held until their audit
A sub's POD section and the module's own POD (the File node's doc) name code with L<> codes: L<Module::Name>, L<Module::Name/section>, L</section>, with a `text|` prefix or not. doclink_perl.c reads them the way perlpodspec and Pod::Simple do (verbatim paragraphs and data regions hold none, an L inside an L is no link, a URL is the core's href family) and writes each file's scope: its packages, its subs and its headings. Lines are exact: a sub's doc is matched against the file's sections, and the module doc's runs (extract_defs.c) are found again in the source in order; a module doc is POD from its start, so a run that begins with text is read too. doc_links_perl.c resolves them: - a module is the file that declares its package; of several, the one the library layout names (.../Name/Space.pm), else ambiguous. The target is the file's File node, as the field test's audited prototype has it (a Module node is missing where a same-named Folder holds its qualified name, lib/Mojo.pm next to lib/Mojo/). - L<Module/section>: the sub its heading names when the module's file defines that sub; any other section is a place in the module's documentation, the module. - L</section>: a sub the heading names; a heading that names no sub is the page itself (local, neither edge nor row); a section the page has no heading for is missing. - a module the repository does not declare (CPAN, the core, a .pod page: discovery keeps no .pod file) and a man page are external; test code (t/, xt/, *.t, and cbm_is_test_file, which has no Perl rule) binds only from test code. There is no scope_delta hook yet: a change to a file's packages, subs or headings re-resolves every Perl reference. The family is held (ships=false) until its held-out audit passes; the suite ships it through the test seam. Tests (doc_mentions_perl): tokens with their lines, file-level and sub-level, each read once; every resolution rule; incremental equals full (a module doc edit and an added module-level reference repair the file, a heading change is a full run); 60 modules give the same edges and rows with one worker and several. Each rule red with it reverted (the module doc's line mapping, a module run read from its start, the missing section, the library layout). Mojolicious at 93f591742a7d (the family shipped, a measurement build) against the field test's prototype records of its .pm/.pl files: 950 (source, target) pairs both have; the 197 only the prototype has are all references within one file (570), which the core makes no edge of; 8 only here come from the extensionless scripts the prototype did not read. Every prototype row has its row here: external 102, missing 1, and the 172 targets in .pod pages external by the ruling for .pod pages. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3af25b432f |
doc-links(rust): intra-doc links become MENTIONS edges, held until their audit
Rustdoc's intra-doc links ([`Name`], [path], [text](path), the kind
disambiguators struct@, fn@, macro@ ...) and links that take their destination
from a link reference definition, bound the way rustdoc resolves them: the
item's module scope, its use declarations, the crate root and the crates
Cargo.toml names. The leg (doclink_rs.c extracts the links and each file's
scope, a Cargo.toml its crate facts; doc_links_rs.c resolves them) is ported
onto the restacked core: family rows carry their edge type, resolve() takes
the per-file state (none here). Both families are held (ships=false) until
their held-out audit passes; the suite ships them through the test seam.
The graph moved under the leg: cfg-gated twins are one definition with its
variants on one node (
|
||
|
Martin Vogel
|
a1190afbb0 |
doc-links(java, kotlin): Javadoc and KDoc references become MENTIONS edges, held until their audit
Java's {@link}, {@linkplain}, @see, @throws and {@value}, and Kotlin's [Name]
links, @see, @throws and @sample, bound the way javadoc and Dokka search for a
member (one resolver for both languages, so a Kotlin doc can name a Java type
and back). The leg (doclink_jvm.c extracts the references and each file's
scope; doc_links_jvm.c resolves them) is ported onto the restacked core:
family rows carry their edge type, resolve() takes the per-file state (none
here). All nine families are held (ships=false) until their held-out audit
passes; the suite ships them through the test seam.
The graph moved under the leg since it was written: a Java field's node is
named by the field alone (no longer by its whole declarator, `MAX = 10`), and
a field named like a method of its class has its own node,
<Owner>.<name>#field, where the method used to take the name. The tests
follow: the expected field names lose the initializer, a changed
initializer is a local change repaired file by file (it renamed the node
before, which forced a full run), and a field that shares its accessor's name
is now an edge target and a source (doc_mentions_jvm_java_families_and_reasons,
doc_mentions_jvm_node_ownership), where both were graph_gap rows.
Measured with the leg's binary (1078036f) and this one on the same input:
- elasticsearch subset (libs/ + server/src/main at f3dc4b0bd3a0, 5,956 files):
no edge lost; 66 new edges and 7 edges moved onto a field's own #field node;
81 graph_gap rows gone, no new row, no reason changed. A seeded sample of 8
of the gone rows read against the source: every new edge is right (a field
named like its accessor, enum constants, a field's own doc as the source).
- Exposed (Kotlin, c419caaab61f): identical, 896 edges and 371 rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
0c7ca34419 |
doc-links(go): Go doc links become MENTIONS edges, held until their audit
Go doc comments name code as doc links ([Name], [Recv.Name], [pkg.Name], [pkg.Recv.Name], an import path for pkg), the way go/doc/comment reads them. The Go leg (doclink_go.c extracts them and a go.mod's module path as the scope; doc_links_go.c resolves them against the package's declarations and its imports) is ported onto the restacked core: the family row carries its edge type, and resolve() takes the per-file state the core now passes (Go keeps none). The family is held (ships=false) until its held-out audit passes: a resolved doc link is a below_bar_tier row until then. The suite ships it through the test seam, like the other held families; doc_mentions_go_tokens asserts the production default. Kubernetes at 8c078bc32e90 with the family shipped (a measurement build): 1,928 edges (1,115 exact, 813 unique) and 101 rows (external 61, graph_gap 22, missing 18), the same numbers the leg measured on the old core. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
92f5edfe32 |
doc-links(asciidoc): AsciiDoc sections linked to the code they include and name
AsciiDoc (.adoc, .asciidoc) is now indexed, without a grammar like PDF: a
Section per heading (`=` .. `======`, or `#`), the text up to the next
heading as its docstring, delimited blocks respected (listing, literal,
comment, passthrough, example, sidebar, quote, table, fences). Its
references become MENTIONS edges, via "asciidoc", from the section they are
written in:
include include::target[...] -- Asciidoctor's preprocessor directive,
read in every block but a comment block; the page's own
attributes, then the Antora component's and the playbook's,
substituted
attribute an {attribute} whose value is an Antora javadoc link
({javadoc-root}/<module>/<package path>/<Type>.html[#member])
binds the Java or Kotlin type, or its one member, that the
repository's main sources declare under that path
code_path, monospace spans naming a path or a qualified name (the
code_name Markdown code-span rules)
The rules are the field test's (H8 Asciidoctor/Antora harvester, audited
~100 % precise): an Antora resource ID ([component:][module:]family$path)
names <component>/modules/<module>/<family dir>/<path>, or the file a
collector scan contributes there (a scan of build output is generated: no
row); any other target is relative to the including file. `lines=` and
`tag=`/`tags=` regions bind by H8's rule: edge blank and comment lines
trimmed, the innermost enclosing definition, unless that is a type (or there
is none) and the definitions inside cover every other line (an annotation
right above one counts) -- then that definition; tag regions come from the
included file's own `tag::x[]` / `end::x[]` comments, read from that indexed
file; a tag the file lacks binds the file. One reference makes one edge: a
region holding several top-level definitions binds the file.
Antora settings travel as YAML scope blobs (antora.yml: component name,
asciidoc attributes, collector scans; antora-playbook*.yml: attributes),
read as text, so an incremental run resolves as a full one; a change to them
is GLOBAL. AsciiDoc joins the documents bound by lines: a body edit of an
included file re-resolves its pages.
Also: the Markdown resolver's module index no longer takes AsciiDoc or PDF
files for code modules.
Proof on junit-team/junit-framework at 6c8ca116 (H8's corpus and commit),
C against H8's recorded links: javadoc attributes 362/362 identical;
includes 292 identical, 60 bind the type enclosing the several definitions
H8 names, 7 regions with several top-level definitions bind the file. 317
sections, 668 edges, no include rows (9 generated partials, 1 file type the
indexer does not read); monospace spans 30/30 correct by hand. 8.3 s for the
whole index.
Tests: suite doc_links_adoc (structure and tokens, the Antora blob, the
pipeline: resource IDs through a collector scan, tag regions, a region with
its imports binding its class, lines=, javadoc attributes, spans; incremental
== full: a moved tag region, a changed component attribute). Revert check:
without H8's region rule and without AsciiDoc among the line-bound documents
exactly their assertions fail (observed).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
1b444c6b61 |
doc-links(rst): reStructuredText sections linked to the code Sphinx markup names
A reST document now gets a Section per title (the text up to the next title
as its docstring, its lines from the title to the next one) and MENTIONS
edges, via "rst", from each section to the code it names:
role :class:`pkg.Model`, :meth:`~x.Y.z`, :func:`.f`, :c:func:`f`,
and extlink roles whose URL is a repository blob path
(:source:`django/db/models/base.py`)
object .. py:class:: / .. method:: / .. function:: ... and their
C-domain twins
autodoc .. automodule:: / .. autoclass:: ...
include, .. include:: / .. kernel-include:: of a code file,
literalinclude .. literalinclude:: (its :lines: bind the innermost
definition holding them, kept as target_lines; its
:pyobject: the named definition)
kernel_doc .. kernel-doc:: path and its :identifiers: / :functions:
code_path, inline literals naming a path or a qualified name (the
code_name Markdown code-span rules)
Structure is the field test's line model (H3): titles by adornment style,
literal blocks after `::` and in code directives, comments, directive
arguments and options, Sphinx's module / currentmodule / class contexts,
CPython's str whitespace and \w. Text before the first title belongs to the
file. A Sphinx documentation set's conf.py is read as text, never run: its
source_suffix makes discovery index e.g. Django's `.txt` documents as reST in
that set only, its primary_domain decides undomained roles, its extlinks and
intersphinx names are known to the resolver.
Python names resolve the way Sphinx imports them: the longest prefix that is
a module's import name (named from its topmost package directory), the rest
by qualified name, a package's re-exports (`from .base import Model` in its
__init__.py) followed. The re-exports and the conf.py settings travel as
Python scope blobs (doclink_py.c), so an incremental run resolves as a full
one; one resolver serves reST and Python. Forms in Sphinx's order (T, C.T,
M.T, M.C.T; a leading `.` reverses it). EXACT when a form names a node,
otherwise UNIQUE by name with the field test's fixes: a bare member
(:meth:`save`, :attr:`x`) is never searched (37 of 40 such results were
wrong), an object directive's target lies in its declared module, a module
never answers a member's role (`:meth:`dispatch`` is not tests/dispatch),
test code is no target of a search but binds when the document writes its
name in full (`django.test.TestCase`). C identifiers are whole names: the one
C/C++ definition of that name and kind. Unresolved: missing when the name or
path is the repository's, ambiguous, test_only_target; another project's
name (stdlib, intersphinx) is no row.
reST architecture decision records: the ADR detector reads reST records
like Markdown ones (titles underlined with = or -): course-discovery's 34
decision records become ADR nodes.
Proof on Django's documentation (the field test's corpus and commit), C
against the frozen H3 harvester over the same graph, per (document,
section, target): 6,695 of H3's 7,238 links identical; H3-only 543 = links
C attributes to the document because a section title repeats in it (the
Markdown rule; the same document -> code link exists), deliberate refusals
(bare members, directive searches without a module) and H3's own audited
error class (router protocol methods -> ConnectionRouter, forms Field ->
model Field); C-only = the same attribution mirrored, plus 1,053 inline
literal links (sample of 30: 30 correct). 6,574 reST sections; 12 s for the
whole index.
Tests: suite doc_links_rst (structure and tokens, the scope blobs, the
pipeline's rules, source_suffix in discovery, reST ADRs, incremental ==
full). Revert check: without the module-kind guard and without the
source_suffix mapping exactly their assertions fail (observed). Two
expectations of the old behaviour now name the new one: the reST grammar
golden and probe have the document's Sections (they recorded "reST titles
are no Sections", the gap this closes).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
033ca372a8 |
doc-links(pdf): PDF pages as Sections, linked to the code their text names
A PDF is now indexed: its text layer is read by a dependency-free extractor
(internal/cbm/pdf, zlib only) and every page becomes a Section node ("page N",
the page's text as its docstring, property page), so search finds what a page
says and get_code_snippet shows a page's text, never the file's bytes. The code
a page names structurally becomes MENTIONS edges from that page, via "pdf":
pdf_path a repository path ("pkg/server/handler.go", "./x/y")
pdf_file a file name with a code extension ("models.py", "Pool.sol")
pdf_qn a qualified name ("app.models.User.save", "a::b", "x->y")
Bare identifiers never become edges (field test E6c: a type-target rule failed
its pre-registered bar), and only EXACT or UNIQUE resolutions link: the field
test's tiers (path-exact, component-aligned path-suffix, qn-exact, qn-suffix,
the latter also through a node's owner -- a Go method's QN has no receiver --
and with the same one-entity collapses). A mention split across a line break is
proposed as one reference and replaces its fragments only when it resolves.
Hygiene as measured: no links from PDFs under test or fixture directories (R1),
none to data files (R2), none for keyword arguments (R3); a test, mock or
fixture target is a test_only_target row (R4, with the test directories the
field test found missing). A path or name whose first component is not of this
repository is no row at all.
The extractor is a port of the field-tested prototype: xref tables and
streams, /Prev chains, repair near a wrong offset and reconstruction from the
object headers, object streams, Flate (with partial recovery), LZW, ASCII85,
ASCIIHex, RunLength and PNG/TIFF predictors, simple fonts (base encodings,
/Differences, built-in Type1 encodings, standard-14 widths, Type3), ToUnicode
and encoding CMaps, Type0/CID fonts with the embedded TrueType cmap read
backwards, Form XObjects, inline images skipped, word and line breaks from
glyph geometry, right-to-left runs reversed. Its geometry is bit-exact on every
platform (no FMA contraction; CPython's hypot). Untrusted input: every read is
bounds-checked, recursion is depth-limited, integers from the file saturate, a
decoded stream is cut at 512 MiB, and every lookup that could rescan the file
per object or per glyph is answered from an index built once. Not read:
encrypted documents (reported as such); scanned pages have no text layer.
The mention scanner is the field test's (idents.py) with CPython's semantics:
NFKC per code point plus canonical composition and the str predicates, from
generated Unicode 16.0 tables (pdf_unicode.c), and re's leftmost-first
matching.
Pipeline: CBM_LANG_PDF (no grammar; the registry ledger gets a DOCUMENT
capability, the call-node manifest exempts it like the other grammar-free
classification), extraction before any grammar lookup, a resolver hook
file_prepare (a file's state from all its references before any resolves).
Incremental: a proxy node carries parent_class where owner + name differs
from its QN (Go methods, class-body variables), so a re-resolved document binds
a Go method as a full build does.
Proof against the field-test prototype on its corpus (749 PDFs from 21
repositories, restored at the recorded commits, sha256-verified): text
byte-identical on all 1,492 pages of the 720 comparable PDFs (0.73 s vs 14.7 s);
mention tokens identical on all 960 page-text sets of E6, E6b and E6c (26,435
tokens); links identical to the frozen Python pipeline run over the same graph
on semantic-kernel 5/5, cosmos-sdk 30/30, Polly 11/11, mediastore3 3/3, dask
4/4. 44,070 seeded mutants (truncations, flips, nesting and keyword bombs, on
the originals and on uncompressed copies) under ASan and UBSan: 0 findings
after the first campaign's integer overflows were fixed as a class.
Tests: suite doc_links_pdf (layout, fonts and CMaps, object and xref streams,
xref repair and reconstruction, Form XObjects incl. a self-referencing one,
inline images, filters, hostile input, the scanner against the field test's
output, the pipeline's tiers, hygiene and rows, back navigation, the snippet,
incremental == full).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
afedff5176 |
doc-links(md): link Markdown sections to the code files and definitions they name
A Markdown document is documentation in itself: its sections now link to the code they reference explicitly, as MENTIONS edges (via "markdown") from the Section node (the File node for text before the first heading, and for a heading whose title repeats in the file, whose Section node it does not own). An agent can go from a document to the code and back through the graph: trace_path inbound with edge_types ["MENTIONS"] lists the sections that name a definition. Families (the "syntax" property; each with its ship gate): - link: the destination of an inline link or a link reference definition (no URL, no in-page anchor, no image); - path: a bare path in prose (a slash and a file extension); - code_path: a code span or HTML <code> span holding a path, a file name or a directory, with an optional line range (#L3-L9, :3-9) or member (path::Name); - code_name: a code span holding a qualified name (a.b, a::b, A#b, A\B, mod:attr). Resolution is guess-free, with the rules the documentation field tests measured: - a path is read relative to the document and, when it has a directory part, to the repository root; a file name alone only next to the document; - a line range binds the innermost code definition holding the whole range, else the file with "target_lines" on the edge; path::member the one definition of that name in the file; - a qualified name binds the one code definition whose qualifier chain ends with the written qualifier, else the one module or package so named; a match as written outranks one that needs case folding; a field binds only through its type, never through an instance; module:attr only in Python; product documentation never binds test code; - a bare name is never an edge. Unresolved references are rows in doc_link_unresolved: missing (the directory or the repository's own root is there, the name is not), ambiguous, test_only_target. A URL, an image, a URL route (/docs), a file of a type the indexer does not read, and code of other projects are no references to the repository and leave no row. Extraction scans the document's bytes once (doclink_md.c); every scan of a line is linear, or n log n where it sorts, using position lists that grow with the bytes they track, not with the line. Front matter, fenced code, HTML comments and MDX import/export lines are not scanned. The incremental route's node proxies now carry their line numbers (pipeline_delta.c), so a re-resolved document binds a line range exactly as a full build does; the Markdown resolver reads definitions from its own by-file list of the run's nodes, never from DEFINES edges, which an incremental run holds for re-extracted files only. Measured on four repositories against the main branch (which links file to file only, REFERENCES_FILE): fastapi 2,899 MENTIONS edges (2,225 REFERENCES_FILE before), cosmos-sdk 1,286 with 205 distinct definitions reached and 188 edges from 41 ADRs into code, dapr 184, curl 52. Link pairs agree with the field-test harvester (cosmos-sdk 408 of 409; the difference is a link of a document to itself, skipped on purpose). A hand check of 45 sampled qualified-name links found one wrong, fixed by the exact-case rule above. Tests: doc_links_md (extraction, repeated headings, the span classifier table, resolution with every reason, back navigation, incremental == full for a code edit, a document edit and an added name). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
fb8d14f6bc |
doc-links(cs): what the Linux leg showed: path-ordered binding, a test thread's caches, split suites
The first run of the doc-link suites on the Linux arm64 leg (gcc, ASan with LeakSanitizer) found three things macOS does not show: - The node pass bound files in the order the file system listed them, and the scratch table's cost depends on that order (the table is emptied per file and made anew only after a far larger file). doc_mentions_cs_scratch_table_work counted 102,912 steps on Linux against 26,048 on macOS for the same files. Files are now bound in path order: a file's bindings do not depend on the others, so only the cost changes, and it is the same on every system. The test itself had a second dependence: its 20,000-field file exceeded the extraction budget on that leg (5 s, defs=0), so the test ran without it; 4,000 fields were still cut there beside five other suites. It now uses 1,000 fields and 20 small files (3,008 steps against a bound of 4,160; 40,512 with the S13 fix reverted) and requires the large file to reach the index whole. - A test thread that extracted C# on its own small stack ended without freeing its parser and node-kind caches, as worker threads do: 40 KB reported leaked at exit. It frees them now. - The suite ran 1,014 s against the harness's 900 s per-suite wall clock (gcc on aarch64 makes ASan's stack-use-after-return check about 100x slower per call). Its 119 tests now run as doc_mentions (72), doc_mentions_cost (9, the work-counter tests) and doc_mentions_msbuild (38): about 230 s, 210 s and 480 s there. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ea9fe36b9e |
doc-links: doc-comment references become MENTIONS edges, C# first
A reference written in a doc comment is extracted with the definition it
documents (internal/cbm/doclink.c; the C# scanner doclink_cs.c) and
resolved per file in the pipeline (src/pipeline/doc_links.c,
doc_links_cs.c, doc_links_msbuild.c). A reference that names a definition
of the repository with certainty becomes a MENTIONS edge from the
documented definition (the File node for a file-level comment) with the
properties via, syntax, tier, line and count. Any other reference becomes
a row of the new doc_link_unresolved table with its reason: missing,
ambiguous, external, test_only_target, graph_gap, unparseable or
below_bar_tier. Nothing is guessed. index_status reports the layer in a
doc_links block.
C# binds a cref by the compiler's lookup rule: the first scope with the
name decides, a cref never binds an inherited member, System is outside
the repository. Assemblies come from project files, and MSBuild global
usings (<Using>, ImplicitUsings, Directory.Build.props/.targets, imports,
conditions) are evaluated from the project files' content as data the
indexer has already read; the layer opens no file. Two declarations of a
name in different assemblies make the reference ambiguous.
Each file's scope is stored line-free with its LSP surface ("dl"), so an
incremental run re-resolves the files a change can reach and falls back
to a full run for a change that can re-route references anywhere (a
project file, a global using, a deleted file).
The scanner and the resolver treat the repository as untrusted input:
bounded nesting, no input-driven recursion, work that grows linearly
(held by test seams, never by a clock), no silent limits, and a damaged
or foreign stored scope fails closed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
bf7ed74f7f |
fix(test): start the runner again by its image path, not argv[0]
The subprocess capture probes, the runner-filter tests and the git-facts diff probe start (or copy) the test runner itself, by argv[0]. On Windows that is the caller's spelling: the native python scheduler (run-test-wave.py) finds the runner by search and passes it without ".exe", and an exact spawn or copy of that path fails. Started that way, 12 subprocess tests failed and the runner-filter tests with them; started with ".exe", the same build passed. tf_runner_image() returns the module path on Windows (cbm_module_path_utf8) and argv[0] elsewhere; the three sites use it. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
42a9320200 |
graph: one identity per conditional definition, SPAWNS edges, no cross-language guesses
The test-impact engine selects tests by walking the graph, so it can only be as right as the graph. Three gaps made it wrong: One identity for a definition's mutually exclusive variants - A definition written once per #if/#else branch, per #[cfg], per platform file (foo_linux.c / foo_win.c, platform directories) or per Go build tag is ONE node. The graph buffer keeps every variant's span in a `variants` node property (definition labels only), so a change in any variant maps to the definition. - Extraction now sees definitions inside conditional blocks where it skipped them: C++/C#/ObjC class members under #if (one cursor walk, conditional block types per language in lang_specs), Elixir defs under if/unless, Fortran .F90/.F95/.F03/.F08, Rust #[cfg] twins (no more `name#cfg(...)` QNs that lost every call), C platform files and directories (non-static file-scope functions). - Haskell: the vendored scanner reads only an #if group's first branch. The grammar stays upstream; every further branch is parsed as its own same-length view (cpp_branch_views.c) and adds what the raw parse missed. haskell/bytestring: no new partial parses, +27 functions, IMPORTS 322 -> 329, 131 definitions with variant spans. - Incremental closure repair re-parses every file of such a definition when one of them changes (variant partners through DEFINES), so a repair equals a full index. No cross-language name guesses - A name-only resolution (unique_name, suffix_match, qualified_suffix, field_type_hint, and same_module across files that only share a path stem) never binds a symbol another language defines. - Bash imports are the files a script sources: every top-level command was read as an import of its name (`set -e` imported "set", bound to a TSX method in 137 scripts here). - This repository: cross-language CALLS 5,916 -> 5,091; the remainder are C/C++ pairs, builtins and same-stem module collisions (recorded). SPAWNS edges (per language, like the HTTP client patterns) - A call that starts a program (subprocess.run, exec.Command, child_process.spawn, posix_spawn, system, Process.Start, Command::new, ... about 30 languages, through the file's own imports) gets a SPAWNS edge to a Process node named after the program (literal first word, or <dynamic>) instead of a CALLS guess. No target resolution. Both resolvers ask at the same point (parity test). The engine follows the graph - Seeds map changed lines through every variant span; a test written once per #if branch is one test; when a change reaches a non-test `main`, the tests that spawn processes are reached too (the process runs main, and main reaches everything). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
867e5f883d |
Merge origin/main into the smart-CI branch
Ten files changed on both sides; every conflict was a union of independent additions: - workflows (_lint.yml, _test.yml): keep the branch's lane `if:` and main's pinned ubuntu-24.04 runner with its reason. - internal/cbm/cbm.h: main's callable-identity and module-doc fields plus the branch's configured-test fields. - Makefile.cbm: both new test files; prod_lsp_all.o keeps main's .build-config dependency and the branch's coverage flags. - scripts/test.sh: main's setup-scripts contract becomes Step 0y3, ahead of the branch's 0z series and the contracts-only exit. - src/foundation/subprocess.c: main's cbm_posix_close_fds_from (it closes the error descriptor too). - src/mcp/mcp.c: main's repository default base for detect_changes, with the early returns released through the branch's detect_release. - src/pipeline/lsp_resolve.h, tests/test_main.c: both includes, suites and start-up steps (main's git environment reset first). - scripts/memory-core-baseline.txt: recounted on the merged tree and lowered (mcp.c 773, config_toml_edit.c 73). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
36e5874525 |
feat(test-impact): team artifact bundles and admitted coverage
A main commit can now publish its test-impact bundle — the commit's frozen graph, its per-test coverage map and receipt.json binding both — and a PR selection given the bundle of its merge base, verified by the caller, builds its graph incrementally from the bundle's graph and narrows suites by the admitted map. Without a verified bundle nothing changes: the graph is built from scratch and every reached suite runs whole. Producer and consumer: - `codebase-memory-mcp test-impact publish|select` (in process, no daemon): publish builds the bundle of the checked-out commit, from the previous bundle's graph when that bundle verifies; select is the selection with optional --artifact/--artifact-commit/--artifact-verified. The MCP surface is unchanged. - receipt.json binds the exact imported graph (graph digest), the graph's content and topology, the map's three files and a compatibility digest (platform label + the bytes of test_impact.coverage.compatibility_paths). - A bundle is used only when the caller verified its source, it is the merge base's own, and its imported graph is the one the receipt names; the map is admitted by the existing receipt policy (source, ancestry, content, age, compatibility, metadata). Refusals become rejection reasons in the answer's receipt. - C_eff: a changed function the map knows contributes its own id; one it does not know (new, platform-only, a type or macro) contributes its nearest mapped callers on the static inbound closure; deleted names contribute every mapped function of that name. Graph: - the frozen build accepts a base candidate and repairs it through the incremental closure route; any doubt declines to the full build. Pinned to one worker like the full frozen build. - incremental closure repair skipped the interprocedural complexity pass: re-parsed functions lost transitive_loop_depth/recursive and callers kept stale values. One traversal now serves the in-memory pass and a new store-level recompute that runs before the importance rescore. - test-impact's frozen builds use a fixed display Git context: the per-run pin path and the commit ids made two builds of one tree differ and made every commit a semantic-input change, so repair always declined. - graph content and topology digests (store_content_digest.c): the exact graph digest binds one database (ids, db_uid, generation), never equal across builds; the content digest keys rows by what they say, the topology digest by what selection reads. Measured on this repository (frozen publish, macOS arm64): a two-file C change repairs in 18-20 s instead of 75-80 s with the topology of the full build; a docs-only change repairs with identical content; selections with and without the bundle are identical on all seven replayed pairs (two repaired incrementally, five declined to the full build). Recorded, not fixed: closure repair still differs from a full build in edge PROPERTIES (argument quoting and key order, CONFIGURES args) and in the corpus-wide SIMILAR_TO/SEMANTICALLY_RELATED edges; selection reads neither, so the topology digest is the incremental-equals-full check and the receipt carries both. Tooling: scripts/test-impact/coverage-merge.py builds a push's incremental coverage map (re-run suites verbatim, the rest carried with ids remapped onto the new image's function table), with its contract as test.sh Step 0z7. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c5abb64943 |
feat(test-impact): detect_changes scope:"tests" end to end
One request pins the change (merge base .. HEAD), builds the graph and the runnable test registry of exactly the HEAD snapshot, seeds the graph from the change, walks what depends on it and answers which tests to run (cbm.test_impact). Missing or doubtful evidence never narrows: it becomes a run-all answer that names why. Snapshot (classified ledger -> frozen graph -> registry): - one shared language classifier for discovery and pinned blobs (raw bytes on every platform); the classified ledger feeds the frozen build, which indexes exactly the listed files and records partial parses as per-file PARSE_GAP instead of refusing the graph; - the pinned tree skips symlinks and submodule links (counted), as discovery never indexes them; - the test model scopes uncertainty to the suites of its file (they run whole), reads line splices where they occur, treats a mid-file `#ifndef X` + `#define X` as an ordinary condition, and no longer reads a plain identifier named like a macro as an invocation. On this repository the registry is now narrowable and matches a text census exactly (8,365 cases, 141 runner suites). Seeding and reach (port of the measured replay, rule B): - hunk -> overlapping definitions, comment-only hunks dropped, pure deletions -> enclosing definition, no-definition hunks -> whole file, deleted names found by text, header changes propagated through the include closure and the headers' macros (pastes included), file-level escalation for unclassifiable header lines; - inbound fixpoint walk over CALLS, ASYNC_CALLS, USAGE, CALL_REFERENCE, READS (+ routes); cases, suites and the runner's entry points are sinks; a reached runner main runs everything (RUNNER_REACHED); - a parse gap whose text names a reached definition reaches the definition around it (or its file); - macro-registered tests run their suites whole; cases and suites map to graph nodes by their start line. The line patterns reproduce the reference regexes exactly on 1,383,850 lines of this repository and 37,167,926 kernel lines; on 24 replayed changes the engine selects at least what the reference selected (13 identical, the rest only conservative supersets on the same graph). Also: - configured test declarations that cannot be mapped degrade their file (no configured roles, a test_declarations diagnostic) instead of aborting the whole index; - the selector keeps other runners' suites out of the selection, and the result codec judges the model by narrowability like the selector; - a change to .codebase-memory.json runs everything (built-in rule); - .codebase-memory.json with the c-cbm preset and this repository's rules. Gate fixes (suites, lint-ci, Windows cross-compile): - the request runs on its own thread with a default-size stack (daemon tool calls run on 256 KiB threads, where the in-process build died); - a request-supplied config_path is no longer followed by an unlink of an uninitialized path buffer; - the inventory's frame-local reader no longer stays reachable from the caller's source after the filter call; - FILE_ID_INFO needs _WIN32_WINNT >= 0x0602; the pinned-tree header pins it before any system header (toolchains defaulting lower failed); - memory-core ratchet: the win_utf8 converters gain core-backed `_in(cls, ...)` forms over one implementation (the legacy names keep the C library for unmigrated callers), the Git facts module uses them, and the Windows spawn path releases everything through one failure exit; pipeline, extraction and config allocations that this work touched moved to the core in matched pairs; baselines tightened; - the branch's C files formatted with the project's clang-format. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
5a9f6e7637 |
Merge pull request #2494 from DeusData/fix/issue-1300-silent-worker-exit
fix(index): report clean worker exits without a response |
||
|
Martin Vogel
|
01fd6cb5e6 |
Merge pull request #2342 from DeusData/feat/callable-identity-plumbing
feat(identity): callable identity plumbing, no language enabled (#2061) |
||
|
Martin Vogel
|
a245618a64 |
wip(ci): checkpoint hybrid test-impact implementation
Preserve the current selector, frozen-input inventory, result codec, coverage collection, runner filtering, and isolated Windows venue work. Component-level regression proofs are recorded in the PR handover. The live classified provider, public execution integration, artifact admission, fresh oracle/shadow evidence, and full platform gates remain incomplete. This checkpoint does not enable or claim verified narrowing. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
6b25fff3a1 |
test(harness): coverage runner profiles children that leave through _exit
The profile runtime writes from an atexit handler, and the children that do the real work of a test never reach it: a fork that hosts a whole MCP server leaves through _exit(), and the re-exec'd index worker leaves through _Exit(), as the production worker does. Measured on mcp:index_bg_paths_route_through_supervisor_issue832 with the hooks as first written, neither of its two children left a profile and the test mapped to the 19 functions its parent ran. A test-impact map built on that would never select the test for a pipeline change. In the coverage build only (CBM_TEST_COVERAGE, POSIX), the runner image now defines _exit and _Exit to write the profile before calling the libc one, so every call in the image is covered: tests, runner roles and product code. A pthread_atfork child handler gives a forked child the profile name an exec'd child derives from LLVM_PROFILE_FILE, so it cannot write over the profile of its parent. The same test now leaves both child profiles and maps to 1530 functions, 1511 of them executed only in a child. A child that dies by signal still leaves no profile, and none of this is compiled into any other build: the ASan runner has no symbol from it. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3ecac479a6 |
test(harness): per-test selection and per-test coverage profiles
The test-impact selector has to hand the C runner individual tests, and the map behind it has to know what each test executes. The runner could do neither: argv selects whole suites. Selection. CBM_TEST_ONLY (comma-separated) and CBM_TEST_ONLY_FILE (one per line; blank lines and # comments allowed) take suite:test tokens, and are unioned when both are set. A suite no token names never executes its body. Inside a named suite only the named tests run; the rest are counted as deselected and the summary reports the count. Argv still selects suites and the tokens narrow inside that selection, so the parallel harness hands every per-suite process the same list with no change of its own. Tokens are strict: one that names a suite this build does not register, or a test its suite never ran, is printed and fails the run. An unreadable file, a malformed token, and a selection that names no test stop the run before any suite. Without a selection the output is byte-identical to before, and --list-suites ignores the selection. Coverage. TEST_COVERAGE=1 compiles clang source coverage into our own code (the one-shot sources, the LSP unity object, the preprocessor; vendored objects stay uninstrumented) and the runner's hooks into the test runner. `make test-runner-cov` and `make cbm-cov` build the instrumented runner and product binary into build/cov. With CBM_TEST_COVERAGE_DIR set the runner writes <suite>/<test>.parent.profraw for itself around each test, points LLVM_PROFILE_FILE at <suite>/<test>.%p.profraw for the children the test spawns, and sends what runs between tests to _setup.<n>.profraw. No other build compiles any of it. tests/test_harness_test_only.sh pins the selection contract against the built runner and runs as Step 2b of scripts/test.sh, before the suites. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
11867dfa74 |
feat(identity): callable identity plumbing, no language enabled (#2061)
Overloads that share a base QN (Java/C#/C++/Kotlin/Swift/Scala/ObjC) merge
into one graph node today, and that node collects the CALLS of every
overload. The agreed cure is one signature-qualified callable identity for
all overloading languages:
callable_qn := base_qn [<tparams>] (params) [cvref]
This change lands the plumbing with every language still at
CBM_CALLABLE_ID_NONE, so the graph is byte-identical; each language is
switched on later in its own change (with its index-format bump).
- internal/cbm/callable_sig.{c,h}: the per-language mode side table
(NONE|TYPED|LABELED_TYPED|LABELED|ARITY; kept out of CBMLangSpec like
cbm_string_dispatch_suffixes, since ~160 positional rows would otherwise
all need the member under -Wmissing-field-initializers), the shared
builder cbm_callable_sig[_mode] for the def and the call-scope QN, and
the inverse cbm_qn_callable_base_len[_named]. Suffix contract: no '.',
'::' or '->'; whitespace-normalized; comments/annotations/attributes
dropped; qualified types keep their last segment; '->' spelled '=>';
variadic '[]' (Java) / '~'; C++ top-level cv dropped, arrays decay,
cvref kept; <= 200 bytes, longer ones keep whole leading entries and end
in '#<fnv1a64 of the full suffix>)'. The builder is iterative (cursor
walk, bounded declarator loop) and self-checks against the inverse.
- The inverse is anchored to the node's bare name wherever the name is
known: real QNs already end in ')' without being callables (a Java field
whose name is its declarator `X = f()`, make `$(VAR)` targets), and only
the anchor makes the inverse exact on them.
- Leaf splitters strip a suffix first (no-op today): registry by-name key
(indexed by the bare name for a suffixed callable), cbm_lsp_bare_segment,
cbm_pipeline_qn_class_method_tail, pxc_last_component, pxc_qn_leaf,
go_index_visit_node, collect_prod_defs, sg_qn_prefix_len.
- New unused fields: CBMDefinition.qn_sig_off, CBMResolvedCall.callee_sig
(relocated by result compaction), CBMRegisteredFunc.sig, CBMLSPDef.sig.
The lsp_surface codec is unchanged (no format bump in this change).
- get_code_snippet and trace_path gain base-QN tiers
(cbm_store_find_nodes_by_qn_base) that find nothing until suffixes exist.
- store: cbm_store_find_nodes_by_qn_suffix built its LIKE pattern in a
512-byte buffer (a long suffix was truncated into a pattern that matched
nothing) and did not escape '_'/'%' (`my_func` also matched `myXfunc`).
The pattern is now heap-built, escaped, and matched with ESCAPE '\'.
Proof: full sorted node+edge+lsp_surface dumps are byte-identical before
and after on 7 corpora (this repo, elasticsearch libs, Exposed, dotnet
System.Linq, System.Collections, coreclr/gc, a C corpus).
Refs #2061
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
cf72f1a657 |
fix(tests): ignore an inherited git repository environment (#2003)
Git exports repository-local variables (GIT_DIR, GIT_INDEX_FILE,
GIT_WORK_TREE, ...) into hooks, and they take precedence over
`git -C <dir>`. Every fixture git command in the test runner therefore
acted on the caller's real repository whenever the suite ran with those
variables set: with GIT_DIR alone, fixture assertions failed and commits,
branches and linked worktrees were written into the caller's clone; with
the full hook set, fixture commits failed and git-dependent tests degraded
to "git not available" SKIPs. The pre-commit hook already clears the
variables (
|
||
|
Martin Vogel
|
f4ff01b10b |
Merge pull request #2410 from DeusData/fix/test-runner-isolate-runtime-dir
fix(test): isolate the C test runners' daemon runtime |
||
|
Martin Vogel
|
533eee07ec |
fix(test): isolate the C test runners' daemon runtime
Why: the test runner isolated HOME and CBM_CACHE_DIR but not
CBM_RUNTIME_DIR, and neither of the first two moves the daemon
rendezvous. Its key is a constant product hash (cbm_daemon_rendezvous_key,
"c888acc3ae367a1e"), and with no explicit parent and CBM_RUNTIME_DIR unset
cbm_daemon_bootstrap_endpoint_new falls back to the account-wide default
parent (/private/tmp on macOS, /tmp on Linux, LocalAppData on Windows).
Every endpoint a test resolved without an explicit parent was therefore
byte-identical to the developer's live daemon socket
(/private/tmp/cbm-daemon-<uid>/cbm-c888acc3ae367a1e.sock), whatever HOME
said.
The only in-runner product caller is CLI activation
(cli_activation_production_context_init, reached from cbm_cmd_install,
cbm_cmd_uninstall and update). Only a process-global seam kept it off the
default: SUITE(cli) sets that seam on entry and clears it on exit. On the
default endpoint, activation admits into the live daemon's cohort. When
the cohort is ours or busy, or the active cache fingerprint is unreadable,
activation prints "Stopping active CBM sessions" and drains them. That is
the same class of failure as an old binary's install, which stopped a
user daemon and six MCP sessions. One test outside SUITE(cli), or one
fixture that drops the variable, was enough to reach it.
daemon_bootstrap_runtime_dir_env_relocates_rendezvous did exactly that:
it restored by unsetting CBM_RUNTIME_DIR.
Fix:
- tests/test_main.c: tf_setup_cache_sentinel creates a fresh per-run
runtime parent (th_secure_runtime_parent_new "run", under /tmp on POSIX
and LocalAppData on Windows) and sets CBM_RUNTIME_DIR to it for the
whole run. It replaces any inherited value, the rule
scripts/test-runtime.sh applies to the shell harness. The existing
atexit cleanup removes the directory.
- src/daemon/bootstrap.c, CBM_ENABLE_TEST_SEAMS only: once a test runner
arms the guard, an endpoint request with no explicit parent and no
CBM_RUNTIME_DIR is refused with a stderr diagnostic instead of
resolving the default. No product binary arms it.
- tests/repro/repro_main.c arms the same guard. No reproduction needs the
rendezvous.
- tests/test_daemon_bootstrap.c: the relocation test restores the value
it found instead of unsetting it.
- tests/test_helpers.h: th_secure_runtime_base() names the anchor that
th_secure_runtime_parent_new already used.
Proof:
- Before, on origin/main
|
||
|
Martin Vogel
|
ea931e7647 |
feat(ui): refine graph exploration and add local agent configuration
Add focused galaxy tracing, edge filters, organic cluster layouts and cached graph exploration with compact architecture and impact summaries. Make the browser-local agent explain the current selection proactively, and add a Config panel with persisted daemon overrides, shared browser preferences and a reference for external configuration inputs. Validation: 182 focused frontend tests and TypeScript passed. Reused the recorded production build and 485 passing native checks on unchanged source. The pre-existing capped-layout loading-note test failure remains documented separately; the full local CI ladder was waived. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a1b7c05685 |
test(spill): make spill test verdicts independent of host free disk
extraction::extract_spill_round_trip_keeps_every_field and parallel::parallel_spill_mode_builds_the_same_graph failed on any machine with less than 10 GB free: cbm_result_spill_open refuses to spill onto a nearly full disk (mem.spill.refused reason=low_disk), so the store never opened. Three pipeline tests took the same refused-store path there and passed without spilling at all (backpressure_futile_nap_disengages, over_budget_after_futility_fails_whole_and_preserves_db, spill_resolves_namespace_imports_like_memory). A verdict must be a pure function of code, test, platform and seed, not of how full the disk is. Add a free-space pin to result_spill that exists only in test-seam builds (CBM_ENABLE_TEST_SEAMS). Production still asks the disk, and its 10 GB floor and half-of-free cap are unchanged. The test runner pins 64 GiB at the top of main, before worker re-exec dispatch, so every suite and re-exec'd index worker sees the same disk. A new test, extraction::extract_spill_refuses_below_the_free_disk_floor, pins the guard itself: one byte under the floor refuses, the floor opens. Verified with cbm_fs_free_bytes temporarily forced to 9 GB: both tests RED 3/3 before, green 3/3 after, RED again with the runner pin removed; the refusal test goes RED with the production guard disabled. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7acb5eac83 |
fix(index): name a worker that exits cleanly without a response (#1300)
The supervisor creates the worker's response file before spawning it. A worker that exited 0 without writing that file left it empty, and slurp_worker_response read the empty file as "" with status OK. The run then counted as a SUCCESS with an empty response (the daemon marked the job successful). The clean-exit branch of worker_terminal_log also deleted the worker log, the only record of where the run stopped. Only an oversized response was treated as a failure. - index_supervisor: an empty response file is WORKER_RESPONSE_READ_EMPTY. A clean exit with no readable response sets response_missing, keeps the worker log, records its path, and records the last structured event the log holds (text or JSON format, last 64 KiB) as last_phase. It is logged as index.supervisor.no_response with the log path and the last phase. The outcome stays CLEAN, so the disposition is FALLBACK: an explicit error, with no in-process fallback and no recovery re-run of a worker that did not crash. - mcp: cbm_mcp_index_worker_no_response_failure builds the error (reason "no_response", last_phase, worker_log, hint). It is used for the first attempt and for a recovery run. - daemon application: the job failure message says the worker exited cleanly without writing a response, and gives the last phase and the log to inspect. - pipeline_incremental: the legacy per-file purge loop now logs incremental.purge.start (changed/deleted counts) and incremental.purge.progress. Progress lines carry counts only, use a ceiling stride (at most ~20 lines plus the final one) and never log one line per file. Before this, nothing was logged between incremental.edge_snapshot and incremental.purge. Tests (RED before, green after, RED on revert): - index_supervisor_clean_exit_without_response_is_named_failure_issue1300 (a real worker subprocess logs a phase and exits 0 without a response) - daemon_application_clean_exit_without_response_is_named_failure_issue1300 - incr_purge_reports_bounded_progress_issue1300 - index_supervisor_clean_exit_without_response_response_issue1300 (the contract of the response builder) Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ff127ea44c | Merge branch 'main' into codex/fix-1648-sanitized-build-config | ||
  Clément DeustandClaude Sonnet 5
|
6ae6cd4c3f |
fix(pipeline): address doclinks review feedback
Replace raw malloc/free with cbm_alloc/cbm_free (CBM_MEM_CLASS_EXTRACT) and a growable CBM_DYN_ARRAY for matches. Remove the 4KiB line-copy cap; lines are split in place instead of copied. Remove the DOCLINK_MAX_REFS(256) cap; the ref vector now grows, is sorted by target id, then collapsed. Track reference kind (bare, relative, rooted) so /x never attempts a directory join and ./x never falls back to root. Report per-file skips via cbm_pipeline_add_file_error and cbm_log_warn, and check cbm_pipeline_check_cancel in the per-file loop. Add negative-control tests for all of the above, plus CRLF handling, fenced-code-block pinning, bare URLs, backtick-basename exactness, and ".." escaping. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Clément Deust <clement.deust@gmail.com> |
||
|
Clément DeustandClaude Fable 5
|
f3bdf11a03 |
feat(pipeline): link markdown file references into the graph
Markdown docs reference other repo files constantly (a coding-standards doc links the modules it governs, a README points at entry points) but none of that surfaced as edges, so fan-in queries were blind to documentation hubs. In an A/B retrieval benchmark I ran in August on a docs-heavy repo, the graph's top fan-in answer was off by roughly 17x: the most-referenced file was a standards doc with 173 references and zero inbound edges. New pre-dump pass, pass_doclinks.c, modeled on pass_configlink.c: three strategies emit REFERENCES_FILE edges between existing File nodes only (unresolvable targets are dropped, the pass never invents nodes): inline links, backtick paths, bare path mentions. Targets resolve against the referencing file's directory and the repo root; repeated references collapse into one edge carrying strategy, confidence and count. Registered in the pre-dump sequence after configlink and in the incremental post-passes; REFERENCES_FILE added to the skill's edge-type list and the structural/language contract tests. Tests mirror test_configlink.c (real files in a tmpdir, File nodes in a gbuf, run the pass, assert edges): inline link, backtick, bare mention, http/anchor ignored, anchor-suffixed file link, dedupe with count, relative-vs-root resolution, unresolvable-target guard, NULL repo_path skip. A companion change adds the same linking for shell files (source lines and script invocations); split out to keep each change reviewable. Validation: scripts/build.sh clean; focused serial runner (doclinks, configlink, pipeline, edge_structural, lang_contract) 340 passed under ASan/UBSan; cppcheck clean on the new files with the repo's flags. Signed-off-by: Clément Deust <clement.deust@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
 astandrik
|
95fa63441b |
test: report sanitizer build define directly
Signed-off-by: astandrik <astandrik@yandex-team.ru> |
||
|
astandrik
|
f50434eddf |
test: include Windows CRT headers directly
Signed-off-by: astandrik <astandrik@yandex-team.ru> |
||
|
astandrik
|
3ab2994f30 |
test: preserve LF in Windows build config output
Signed-off-by: astandrik <astandrik@yandex-team.ru> |