mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-04 10:48:20 +00:00
f4277ae783ded09d914930f7771ae8fdfeca0a2f
241
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
f4277ae783 |
doc-links(perl): POD L<> codes become MENTIONS edges, held until their audit
A sub's POD section and the module's own POD (the File node's doc) name code with L<> codes: L<Module::Name>, L<Module::Name/section>, L</section>, with a `text|` prefix or not. doclink_perl.c reads them the way perlpodspec and Pod::Simple do (verbatim paragraphs and data regions hold none, an L inside an L is no link, a URL is the core's href family) and writes each file's scope: its packages, its subs and its headings. Lines are exact: a sub's doc is matched against the file's sections, and the module doc's runs (extract_defs.c) are found again in the source in order; a module doc is POD from its start, so a run that begins with text is read too. doc_links_perl.c resolves them: - a module is the file that declares its package; of several, the one the library layout names (.../Name/Space.pm), else ambiguous. The target is the file's File node, as the field test's audited prototype has it (a Module node is missing where a same-named Folder holds its qualified name, lib/Mojo.pm next to lib/Mojo/). - L<Module/section>: the sub its heading names when the module's file defines that sub; any other section is a place in the module's documentation, the module. - L</section>: a sub the heading names; a heading that names no sub is the page itself (local, neither edge nor row); a section the page has no heading for is missing. - a module the repository does not declare (CPAN, the core, a .pod page: discovery keeps no .pod file) and a man page are external; test code (t/, xt/, *.t, and cbm_is_test_file, which has no Perl rule) binds only from test code. There is no scope_delta hook yet: a change to a file's packages, subs or headings re-resolves every Perl reference. The family is held (ships=false) until its held-out audit passes; the suite ships it through the test seam. Tests (doc_mentions_perl): tokens with their lines, file-level and sub-level, each read once; every resolution rule; incremental equals full (a module doc edit and an added module-level reference repair the file, a heading change is a full run); 60 modules give the same edges and rows with one worker and several. Each rule red with it reverted (the module doc's line mapping, a module run read from its start, the missing section, the library layout). Mojolicious at 93f591742a7d (the family shipped, a measurement build) against the field test's prototype records of its .pm/.pl files: 950 (source, target) pairs both have; the 197 only the prototype has are all references within one file (570), which the core makes no edge of; 8 only here come from the extensionless scripts the prototype did not read. Every prototype row has its row here: external 102, missing 1, and the 172 targets in .pod pages external by the ruling for .pod pages. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3af25b432f |
doc-links(rust): intra-doc links become MENTIONS edges, held until their audit
Rustdoc's intra-doc links ([`Name`], [path], [text](path), the kind
disambiguators struct@, fn@, macro@ ...) and links that take their destination
from a link reference definition, bound the way rustdoc resolves them: the
item's module scope, its use declarations, the crate root and the crates
Cargo.toml names. The leg (doclink_rs.c extracts the links and each file's
scope, a Cargo.toml its crate facts; doc_links_rs.c resolves them) is ported
onto the restacked core: family rows carry their edge type, resolve() takes
the per-file state (none here). Both families are held (ships=false) until
their held-out audit passes; the suite ships them through the test seam.
The graph moved under the leg: cfg-gated twins are one definition with its
variants on one node (
|
||
|
Martin Vogel
|
a1190afbb0 |
doc-links(java, kotlin): Javadoc and KDoc references become MENTIONS edges, held until their audit
Java's {@link}, {@linkplain}, @see, @throws and {@value}, and Kotlin's [Name]
links, @see, @throws and @sample, bound the way javadoc and Dokka search for a
member (one resolver for both languages, so a Kotlin doc can name a Java type
and back). The leg (doclink_jvm.c extracts the references and each file's
scope; doc_links_jvm.c resolves them) is ported onto the restacked core:
family rows carry their edge type, resolve() takes the per-file state (none
here). All nine families are held (ships=false) until their held-out audit
passes; the suite ships them through the test seam.
The graph moved under the leg since it was written: a Java field's node is
named by the field alone (no longer by its whole declarator, `MAX = 10`), and
a field named like a method of its class has its own node,
<Owner>.<name>#field, where the method used to take the name. The tests
follow: the expected field names lose the initializer, a changed
initializer is a local change repaired file by file (it renamed the node
before, which forced a full run), and a field that shares its accessor's name
is now an edge target and a source (doc_mentions_jvm_java_families_and_reasons,
doc_mentions_jvm_node_ownership), where both were graph_gap rows.
Measured with the leg's binary (1078036f) and this one on the same input:
- elasticsearch subset (libs/ + server/src/main at f3dc4b0bd3a0, 5,956 files):
no edge lost; 66 new edges and 7 edges moved onto a field's own #field node;
81 graph_gap rows gone, no new row, no reason changed. A seeded sample of 8
of the gone rows read against the source: every new edge is right (a field
named like its accessor, enum constants, a field's own doc as the source).
- Exposed (Kotlin, c419caaab61f): identical, 896 edges and 371 rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
0c7ca34419 |
doc-links(go): Go doc links become MENTIONS edges, held until their audit
Go doc comments name code as doc links ([Name], [Recv.Name], [pkg.Name], [pkg.Recv.Name], an import path for pkg), the way go/doc/comment reads them. The Go leg (doclink_go.c extracts them and a go.mod's module path as the scope; doc_links_go.c resolves them against the package's declarations and its imports) is ported onto the restacked core: the family row carries its edge type, and resolve() takes the per-file state the core now passes (Go keeps none). The family is held (ships=false) until its held-out audit passes: a resolved doc link is a below_bar_tier row until then. The suite ships it through the test seam, like the other held families; doc_mentions_go_tokens asserts the production default. Kubernetes at 8c078bc32e90 with the family shipped (a measurement build): 1,928 edges (1,115 exact, 813 unique) and 101 rows (external 61, graph_gap 22, missing 18), the same numbers the leg measured on the old core. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
92f5edfe32 |
doc-links(asciidoc): AsciiDoc sections linked to the code they include and name
AsciiDoc (.adoc, .asciidoc) is now indexed, without a grammar like PDF: a
Section per heading (`=` .. `======`, or `#`), the text up to the next
heading as its docstring, delimited blocks respected (listing, literal,
comment, passthrough, example, sidebar, quote, table, fences). Its
references become MENTIONS edges, via "asciidoc", from the section they are
written in:
include include::target[...] -- Asciidoctor's preprocessor directive,
read in every block but a comment block; the page's own
attributes, then the Antora component's and the playbook's,
substituted
attribute an {attribute} whose value is an Antora javadoc link
({javadoc-root}/<module>/<package path>/<Type>.html[#member])
binds the Java or Kotlin type, or its one member, that the
repository's main sources declare under that path
code_path, monospace spans naming a path or a qualified name (the
code_name Markdown code-span rules)
The rules are the field test's (H8 Asciidoctor/Antora harvester, audited
~100 % precise): an Antora resource ID ([component:][module:]family$path)
names <component>/modules/<module>/<family dir>/<path>, or the file a
collector scan contributes there (a scan of build output is generated: no
row); any other target is relative to the including file. `lines=` and
`tag=`/`tags=` regions bind by H8's rule: edge blank and comment lines
trimmed, the innermost enclosing definition, unless that is a type (or there
is none) and the definitions inside cover every other line (an annotation
right above one counts) -- then that definition; tag regions come from the
included file's own `tag::x[]` / `end::x[]` comments, read from that indexed
file; a tag the file lacks binds the file. One reference makes one edge: a
region holding several top-level definitions binds the file.
Antora settings travel as YAML scope blobs (antora.yml: component name,
asciidoc attributes, collector scans; antora-playbook*.yml: attributes),
read as text, so an incremental run resolves as a full one; a change to them
is GLOBAL. AsciiDoc joins the documents bound by lines: a body edit of an
included file re-resolves its pages.
Also: the Markdown resolver's module index no longer takes AsciiDoc or PDF
files for code modules.
Proof on junit-team/junit-framework at 6c8ca116 (H8's corpus and commit),
C against H8's recorded links: javadoc attributes 362/362 identical;
includes 292 identical, 60 bind the type enclosing the several definitions
H8 names, 7 regions with several top-level definitions bind the file. 317
sections, 668 edges, no include rows (9 generated partials, 1 file type the
indexer does not read); monospace spans 30/30 correct by hand. 8.3 s for the
whole index.
Tests: suite doc_links_adoc (structure and tokens, the Antora blob, the
pipeline: resource IDs through a collector scan, tag regions, a region with
its imports binding its class, lines=, javadoc attributes, spans; incremental
== full: a moved tag region, a changed component attribute). Revert check:
without H8's region rule and without AsciiDoc among the line-bound documents
exactly their assertions fail (observed).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
1b444c6b61 |
doc-links(rst): reStructuredText sections linked to the code Sphinx markup names
A reST document now gets a Section per title (the text up to the next title
as its docstring, its lines from the title to the next one) and MENTIONS
edges, via "rst", from each section to the code it names:
role :class:`pkg.Model`, :meth:`~x.Y.z`, :func:`.f`, :c:func:`f`,
and extlink roles whose URL is a repository blob path
(:source:`django/db/models/base.py`)
object .. py:class:: / .. method:: / .. function:: ... and their
C-domain twins
autodoc .. automodule:: / .. autoclass:: ...
include, .. include:: / .. kernel-include:: of a code file,
literalinclude .. literalinclude:: (its :lines: bind the innermost
definition holding them, kept as target_lines; its
:pyobject: the named definition)
kernel_doc .. kernel-doc:: path and its :identifiers: / :functions:
code_path, inline literals naming a path or a qualified name (the
code_name Markdown code-span rules)
Structure is the field test's line model (H3): titles by adornment style,
literal blocks after `::` and in code directives, comments, directive
arguments and options, Sphinx's module / currentmodule / class contexts,
CPython's str whitespace and \w. Text before the first title belongs to the
file. A Sphinx documentation set's conf.py is read as text, never run: its
source_suffix makes discovery index e.g. Django's `.txt` documents as reST in
that set only, its primary_domain decides undomained roles, its extlinks and
intersphinx names are known to the resolver.
Python names resolve the way Sphinx imports them: the longest prefix that is
a module's import name (named from its topmost package directory), the rest
by qualified name, a package's re-exports (`from .base import Model` in its
__init__.py) followed. The re-exports and the conf.py settings travel as
Python scope blobs (doclink_py.c), so an incremental run resolves as a full
one; one resolver serves reST and Python. Forms in Sphinx's order (T, C.T,
M.T, M.C.T; a leading `.` reverses it). EXACT when a form names a node,
otherwise UNIQUE by name with the field test's fixes: a bare member
(:meth:`save`, :attr:`x`) is never searched (37 of 40 such results were
wrong), an object directive's target lies in its declared module, a module
never answers a member's role (`:meth:`dispatch`` is not tests/dispatch),
test code is no target of a search but binds when the document writes its
name in full (`django.test.TestCase`). C identifiers are whole names: the one
C/C++ definition of that name and kind. Unresolved: missing when the name or
path is the repository's, ambiguous, test_only_target; another project's
name (stdlib, intersphinx) is no row.
reST architecture decision records: the ADR detector reads reST records
like Markdown ones (titles underlined with = or -): course-discovery's 34
decision records become ADR nodes.
Proof on Django's documentation (the field test's corpus and commit), C
against the frozen H3 harvester over the same graph, per (document,
section, target): 6,695 of H3's 7,238 links identical; H3-only 543 = links
C attributes to the document because a section title repeats in it (the
Markdown rule; the same document -> code link exists), deliberate refusals
(bare members, directive searches without a module) and H3's own audited
error class (router protocol methods -> ConnectionRouter, forms Field ->
model Field); C-only = the same attribution mirrored, plus 1,053 inline
literal links (sample of 30: 30 correct). 6,574 reST sections; 12 s for the
whole index.
Tests: suite doc_links_rst (structure and tokens, the scope blobs, the
pipeline's rules, source_suffix in discovery, reST ADRs, incremental ==
full). Revert check: without the module-kind guard and without the
source_suffix mapping exactly their assertions fail (observed). Two
expectations of the old behaviour now name the new one: the reST grammar
golden and probe have the document's Sections (they recorded "reST titles
are no Sections", the gap this closes).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
033ca372a8 |
doc-links(pdf): PDF pages as Sections, linked to the code their text names
A PDF is now indexed: its text layer is read by a dependency-free extractor
(internal/cbm/pdf, zlib only) and every page becomes a Section node ("page N",
the page's text as its docstring, property page), so search finds what a page
says and get_code_snippet shows a page's text, never the file's bytes. The code
a page names structurally becomes MENTIONS edges from that page, via "pdf":
pdf_path a repository path ("pkg/server/handler.go", "./x/y")
pdf_file a file name with a code extension ("models.py", "Pool.sol")
pdf_qn a qualified name ("app.models.User.save", "a::b", "x->y")
Bare identifiers never become edges (field test E6c: a type-target rule failed
its pre-registered bar), and only EXACT or UNIQUE resolutions link: the field
test's tiers (path-exact, component-aligned path-suffix, qn-exact, qn-suffix,
the latter also through a node's owner -- a Go method's QN has no receiver --
and with the same one-entity collapses). A mention split across a line break is
proposed as one reference and replaces its fragments only when it resolves.
Hygiene as measured: no links from PDFs under test or fixture directories (R1),
none to data files (R2), none for keyword arguments (R3); a test, mock or
fixture target is a test_only_target row (R4, with the test directories the
field test found missing). A path or name whose first component is not of this
repository is no row at all.
The extractor is a port of the field-tested prototype: xref tables and
streams, /Prev chains, repair near a wrong offset and reconstruction from the
object headers, object streams, Flate (with partial recovery), LZW, ASCII85,
ASCIIHex, RunLength and PNG/TIFF predictors, simple fonts (base encodings,
/Differences, built-in Type1 encodings, standard-14 widths, Type3), ToUnicode
and encoding CMaps, Type0/CID fonts with the embedded TrueType cmap read
backwards, Form XObjects, inline images skipped, word and line breaks from
glyph geometry, right-to-left runs reversed. Its geometry is bit-exact on every
platform (no FMA contraction; CPython's hypot). Untrusted input: every read is
bounds-checked, recursion is depth-limited, integers from the file saturate, a
decoded stream is cut at 512 MiB, and every lookup that could rescan the file
per object or per glyph is answered from an index built once. Not read:
encrypted documents (reported as such); scanned pages have no text layer.
The mention scanner is the field test's (idents.py) with CPython's semantics:
NFKC per code point plus canonical composition and the str predicates, from
generated Unicode 16.0 tables (pdf_unicode.c), and re's leftmost-first
matching.
Pipeline: CBM_LANG_PDF (no grammar; the registry ledger gets a DOCUMENT
capability, the call-node manifest exempts it like the other grammar-free
classification), extraction before any grammar lookup, a resolver hook
file_prepare (a file's state from all its references before any resolves).
Incremental: a proxy node carries parent_class where owner + name differs
from its QN (Go methods, class-body variables), so a re-resolved document binds
a Go method as a full build does.
Proof against the field-test prototype on its corpus (749 PDFs from 21
repositories, restored at the recorded commits, sha256-verified): text
byte-identical on all 1,492 pages of the 720 comparable PDFs (0.73 s vs 14.7 s);
mention tokens identical on all 960 page-text sets of E6, E6b and E6c (26,435
tokens); links identical to the frozen Python pipeline run over the same graph
on semantic-kernel 5/5, cosmos-sdk 30/30, Polly 11/11, mediastore3 3/3, dask
4/4. 44,070 seeded mutants (truncations, flips, nesting and keyword bombs, on
the originals and on uncompressed copies) under ASan and UBSan: 0 findings
after the first campaign's integer overflows were fixed as a class.
Tests: suite doc_links_pdf (layout, fonts and CMaps, object and xref streams,
xref repair and reconstruction, Form XObjects incl. a self-referencing one,
inline images, filters, hostile input, the scanner against the field test's
output, the pipeline's tiers, hygiene and rows, back navigation, the snippet,
incremental == full).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
6b0761160d |
doc-links(adr): architecture decision records as ADR nodes with their facts and SUPERSEDES
A Markdown file that is an architecture decision record now also defines an
ADR node (File DEFINES ADR): its name is the record's canonical id ("ADR-12";
another prefix as written, "DEC-5"; a dated record keeps its file stem), its
qualified name "<module>.__adr__", and its properties carry the record's
facts: adr_id, title, status (normalized: proposed, accepted, implemented,
superseded, deprecated, rejected, abandoned, draft, unknown), status_text,
date, deciders, superseded_by, and the decision itself as the docstring, so
search finds a record by what it decided. Its sections link to code like any
document's (MENTIONS, previous commit).
Detection is per file, with the field tests' rules: a file in a conventional
ADR directory (adr, adrs, decisions, decision-records, ...) whose name carries
an id or whose text has ADR structure; any adr-NNN file; a numbered or dated
file with ADR structure. Never README, index or template files without an id,
never test fixtures. Facts come from front matter, then header fields
(bullets, bold, tables), then the Status section; the date also from a
"Last updated" field, the change log's first date and the file name.
An ADR's own statement that it supersedes another ("Supersedes [ADR-3](...)",
"Supersedes ADR-3"; "Replaces ADR-3" where it opens a line, never "replaces"
as a verb of prose) becomes a SUPERSEDES edge between the two ADR nodes, by
link or by id. "Superseded by" is written in the old record about another
file; an edge from the other node would be owned by a file whose text does
not hold it, so it stays a fact of the old record (status, superseded_by).
The link layer gains a per-family edge type (doclink.h: the family list's
"edge" column), so a family can produce SUPERSEDES instead of MENTIONS; edges
are grouped by (type, source, target). Definitions gain a tail field for a
document's facts (extra_props: key, value pairs), carried through result
compaction and written by both property builders.
Measured against the field-test harvester on five repositories: detection
182 of 182 records, the same files; title 182/182; status 179/182 (cosmos
ADR-050 and its annexes say "Status: ARCHIVED", kept as status_text with
status unknown); date 174/182 (semantic-kernel's "date: {2024-05-02}" is read
as a date where the harvester dropped it). SUPERSEDES: semantic-kernel
0038 -> 0015, log4brains 20201016 -> 20200926, each checked against its
sentence. Markdown MENTIONS unchanged on fastapi, curl, dapr, cosmos-sdk.
Not covered: ADRs in reStructuredText, and ADR-log configuration files
(.adr-dir, log4brains) as a detection source.
Tests: doc_links_md_adr (detection and exclusions, facts from front matter
and from a Status section, SUPERSEDES by link and by id, no edge from prose,
the record's MENTIONS, incremental == full for a status edit and for an
added record named by id).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
afedff5176 |
doc-links(md): link Markdown sections to the code files and definitions they name
A Markdown document is documentation in itself: its sections now link to the code they reference explicitly, as MENTIONS edges (via "markdown") from the Section node (the File node for text before the first heading, and for a heading whose title repeats in the file, whose Section node it does not own). An agent can go from a document to the code and back through the graph: trace_path inbound with edge_types ["MENTIONS"] lists the sections that name a definition. Families (the "syntax" property; each with its ship gate): - link: the destination of an inline link or a link reference definition (no URL, no in-page anchor, no image); - path: a bare path in prose (a slash and a file extension); - code_path: a code span or HTML <code> span holding a path, a file name or a directory, with an optional line range (#L3-L9, :3-9) or member (path::Name); - code_name: a code span holding a qualified name (a.b, a::b, A#b, A\B, mod:attr). Resolution is guess-free, with the rules the documentation field tests measured: - a path is read relative to the document and, when it has a directory part, to the repository root; a file name alone only next to the document; - a line range binds the innermost code definition holding the whole range, else the file with "target_lines" on the edge; path::member the one definition of that name in the file; - a qualified name binds the one code definition whose qualifier chain ends with the written qualifier, else the one module or package so named; a match as written outranks one that needs case folding; a field binds only through its type, never through an instance; module:attr only in Python; product documentation never binds test code; - a bare name is never an edge. Unresolved references are rows in doc_link_unresolved: missing (the directory or the repository's own root is there, the name is not), ambiguous, test_only_target. A URL, an image, a URL route (/docs), a file of a type the indexer does not read, and code of other projects are no references to the repository and leave no row. Extraction scans the document's bytes once (doclink_md.c); every scan of a line is linear, or n log n where it sorts, using position lists that grow with the bytes they track, not with the line. Front matter, fenced code, HTML comments and MDX import/export lines are not scanned. The incremental route's node proxies now carry their line numbers (pipeline_delta.c), so a re-resolved document binds a line range exactly as a full build does; the Markdown resolver reads definitions from its own by-file list of the run's nodes, never from DEFINES edges, which an incremental run holds for re-extracted files only. Measured on four repositories against the main branch (which links file to file only, REFERENCES_FILE): fastapi 2,899 MENTIONS edges (2,225 REFERENCES_FILE before), cosmos-sdk 1,286 with 205 distinct definitions reached and 188 edges from 41 ADRs into code, dapr 184, curl 52. Link pairs agree with the field-test harvester (cosmos-sdk 408 of 409; the difference is a link of a document to itself, skipped on purpose). A hand check of 45 sampled qualified-name links found one wrong, fixed by the exact-case rule above. Tests: doc_links_md (extraction, repeated headings, the span classifier table, resolution with every reason, back navigation, incremental == full for a code edit, a document edit and an added name). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ea9fe36b9e |
doc-links: doc-comment references become MENTIONS edges, C# first
A reference written in a doc comment is extracted with the definition it
documents (internal/cbm/doclink.c; the C# scanner doclink_cs.c) and
resolved per file in the pipeline (src/pipeline/doc_links.c,
doc_links_cs.c, doc_links_msbuild.c). A reference that names a definition
of the repository with certainty becomes a MENTIONS edge from the
documented definition (the File node for a file-level comment) with the
properties via, syntax, tier, line and count. Any other reference becomes
a row of the new doc_link_unresolved table with its reason: missing,
ambiguous, external, test_only_target, graph_gap, unparseable or
below_bar_tier. Nothing is guessed. index_status reports the layer in a
doc_links block.
C# binds a cref by the compiler's lookup rule: the first scope with the
name decides, a cref never binds an inherited member, System is outside
the repository. Assemblies come from project files, and MSBuild global
usings (<Using>, ImplicitUsings, Directory.Build.props/.targets, imports,
conditions) are evaluated from the project files' content as data the
indexer has already read; the layer opens no file. Two declarations of a
name in different assemblies make the reference ambiguous.
Each file's scope is stored line-free with its LSP surface ("dl"), so an
incremental run re-resolves the files a change can reach and falls back
to a full run for a change that can re-route references anywhere (a
project file, a global using, a deleted file).
The scanner and the resolver treat the repository as untrusted input:
bounded nesting, no input-driven recursion, work that grows linearly
(held by test seams, never by a clock), no silent limits, and a damaged
or foreign stored scope fails closed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
42a9320200 |
graph: one identity per conditional definition, SPAWNS edges, no cross-language guesses
The test-impact engine selects tests by walking the graph, so it can only be as right as the graph. Three gaps made it wrong: One identity for a definition's mutually exclusive variants - A definition written once per #if/#else branch, per #[cfg], per platform file (foo_linux.c / foo_win.c, platform directories) or per Go build tag is ONE node. The graph buffer keeps every variant's span in a `variants` node property (definition labels only), so a change in any variant maps to the definition. - Extraction now sees definitions inside conditional blocks where it skipped them: C++/C#/ObjC class members under #if (one cursor walk, conditional block types per language in lang_specs), Elixir defs under if/unless, Fortran .F90/.F95/.F03/.F08, Rust #[cfg] twins (no more `name#cfg(...)` QNs that lost every call), C platform files and directories (non-static file-scope functions). - Haskell: the vendored scanner reads only an #if group's first branch. The grammar stays upstream; every further branch is parsed as its own same-length view (cpp_branch_views.c) and adds what the raw parse missed. haskell/bytestring: no new partial parses, +27 functions, IMPORTS 322 -> 329, 131 definitions with variant spans. - Incremental closure repair re-parses every file of such a definition when one of them changes (variant partners through DEFINES), so a repair equals a full index. No cross-language name guesses - A name-only resolution (unique_name, suffix_match, qualified_suffix, field_type_hint, and same_module across files that only share a path stem) never binds a symbol another language defines. - Bash imports are the files a script sources: every top-level command was read as an import of its name (`set -e` imported "set", bound to a TSX method in 137 scripts here). - This repository: cross-language CALLS 5,916 -> 5,091; the remainder are C/C++ pairs, builtins and same-stem module collisions (recorded). SPAWNS edges (per language, like the HTTP client patterns) - A call that starts a program (subprocess.run, exec.Command, child_process.spawn, posix_spawn, system, Process.Start, Command::new, ... about 30 languages, through the file's own imports) gets a SPAWNS edge to a Process node named after the program (literal first word, or <dynamic>) instead of a CALLS guess. No target resolution. Both resolvers ask at the same point (parity test). The engine follows the graph - Seeds map changed lines through every variant span; a test written once per #if branch is one test; when a change reaches a non-test `main`, the tests that spawn processes are reached too (the process runs main, and main reaches everything). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
867e5f883d |
Merge origin/main into the smart-CI branch
Ten files changed on both sides; every conflict was a union of independent additions: - workflows (_lint.yml, _test.yml): keep the branch's lane `if:` and main's pinned ubuntu-24.04 runner with its reason. - internal/cbm/cbm.h: main's callable-identity and module-doc fields plus the branch's configured-test fields. - Makefile.cbm: both new test files; prod_lsp_all.o keeps main's .build-config dependency and the branch's coverage flags. - scripts/test.sh: main's setup-scripts contract becomes Step 0y3, ahead of the branch's 0z series and the contracts-only exit. - src/foundation/subprocess.c: main's cbm_posix_close_fds_from (it closes the error descriptor too). - src/mcp/mcp.c: main's repository default base for detect_changes, with the early returns released through the branch's detect_release. - src/pipeline/lsp_resolve.h, tests/test_main.c: both includes, suites and start-up steps (main's git environment reset first). - scripts/memory-core-baseline.txt: recounted on the merged tree and lowered (mcp.c 773, config_toml_edit.c 73). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
36e5874525 |
feat(test-impact): team artifact bundles and admitted coverage
A main commit can now publish its test-impact bundle — the commit's frozen graph, its per-test coverage map and receipt.json binding both — and a PR selection given the bundle of its merge base, verified by the caller, builds its graph incrementally from the bundle's graph and narrows suites by the admitted map. Without a verified bundle nothing changes: the graph is built from scratch and every reached suite runs whole. Producer and consumer: - `codebase-memory-mcp test-impact publish|select` (in process, no daemon): publish builds the bundle of the checked-out commit, from the previous bundle's graph when that bundle verifies; select is the selection with optional --artifact/--artifact-commit/--artifact-verified. The MCP surface is unchanged. - receipt.json binds the exact imported graph (graph digest), the graph's content and topology, the map's three files and a compatibility digest (platform label + the bytes of test_impact.coverage.compatibility_paths). - A bundle is used only when the caller verified its source, it is the merge base's own, and its imported graph is the one the receipt names; the map is admitted by the existing receipt policy (source, ancestry, content, age, compatibility, metadata). Refusals become rejection reasons in the answer's receipt. - C_eff: a changed function the map knows contributes its own id; one it does not know (new, platform-only, a type or macro) contributes its nearest mapped callers on the static inbound closure; deleted names contribute every mapped function of that name. Graph: - the frozen build accepts a base candidate and repairs it through the incremental closure route; any doubt declines to the full build. Pinned to one worker like the full frozen build. - incremental closure repair skipped the interprocedural complexity pass: re-parsed functions lost transitive_loop_depth/recursive and callers kept stale values. One traversal now serves the in-memory pass and a new store-level recompute that runs before the importance rescore. - test-impact's frozen builds use a fixed display Git context: the per-run pin path and the commit ids made two builds of one tree differ and made every commit a semantic-input change, so repair always declined. - graph content and topology digests (store_content_digest.c): the exact graph digest binds one database (ids, db_uid, generation), never equal across builds; the content digest keys rows by what they say, the topology digest by what selection reads. Measured on this repository (frozen publish, macOS arm64): a two-file C change repairs in 18-20 s instead of 75-80 s with the topology of the full build; a docs-only change repairs with identical content; selections with and without the bundle are identical on all seven replayed pairs (two repaired incrementally, five declined to the full build). Recorded, not fixed: closure repair still differs from a full build in edge PROPERTIES (argument quoting and key order, CONFIGURES args) and in the corpus-wide SIMILAR_TO/SEMANTICALLY_RELATED edges; selection reads neither, so the topology digest is the incremental-equals-full check and the receipt carries both. Tooling: scripts/test-impact/coverage-merge.py builds a push's incremental coverage map (re-run suites verbatim, the rest carried with ids remapped onto the new image's function table), with its contract as test.sh Step 0z7. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c5abb64943 |
feat(test-impact): detect_changes scope:"tests" end to end
One request pins the change (merge base .. HEAD), builds the graph and the runnable test registry of exactly the HEAD snapshot, seeds the graph from the change, walks what depends on it and answers which tests to run (cbm.test_impact). Missing or doubtful evidence never narrows: it becomes a run-all answer that names why. Snapshot (classified ledger -> frozen graph -> registry): - one shared language classifier for discovery and pinned blobs (raw bytes on every platform); the classified ledger feeds the frozen build, which indexes exactly the listed files and records partial parses as per-file PARSE_GAP instead of refusing the graph; - the pinned tree skips symlinks and submodule links (counted), as discovery never indexes them; - the test model scopes uncertainty to the suites of its file (they run whole), reads line splices where they occur, treats a mid-file `#ifndef X` + `#define X` as an ordinary condition, and no longer reads a plain identifier named like a macro as an invocation. On this repository the registry is now narrowable and matches a text census exactly (8,365 cases, 141 runner suites). Seeding and reach (port of the measured replay, rule B): - hunk -> overlapping definitions, comment-only hunks dropped, pure deletions -> enclosing definition, no-definition hunks -> whole file, deleted names found by text, header changes propagated through the include closure and the headers' macros (pastes included), file-level escalation for unclassifiable header lines; - inbound fixpoint walk over CALLS, ASYNC_CALLS, USAGE, CALL_REFERENCE, READS (+ routes); cases, suites and the runner's entry points are sinks; a reached runner main runs everything (RUNNER_REACHED); - a parse gap whose text names a reached definition reaches the definition around it (or its file); - macro-registered tests run their suites whole; cases and suites map to graph nodes by their start line. The line patterns reproduce the reference regexes exactly on 1,383,850 lines of this repository and 37,167,926 kernel lines; on 24 replayed changes the engine selects at least what the reference selected (13 identical, the rest only conservative supersets on the same graph). Also: - configured test declarations that cannot be mapped degrade their file (no configured roles, a test_declarations diagnostic) instead of aborting the whole index; - the selector keeps other runners' suites out of the selection, and the result codec judges the model by narrowability like the selector; - a change to .codebase-memory.json runs everything (built-in rule); - .codebase-memory.json with the c-cbm preset and this repository's rules. Gate fixes (suites, lint-ci, Windows cross-compile): - the request runs on its own thread with a default-size stack (daemon tool calls run on 256 KiB threads, where the in-process build died); - a request-supplied config_path is no longer followed by an unlink of an uninitialized path buffer; - the inventory's frame-local reader no longer stays reachable from the caller's source after the filter call; - FILE_ID_INFO needs _WIN32_WINNT >= 0x0602; the pinned-tree header pins it before any system header (toolchains defaulting lower failed); - memory-core ratchet: the win_utf8 converters gain core-backed `_in(cls, ...)` forms over one implementation (the legacy names keep the C library for unmigrated callers), the Git facts module uses them, and the Windows spawn path releases everything through one failure exit; pipeline, extraction and config allocations that this work touched moved to the core in matched pairs; baselines tightened; - the branch's C files formatted with the project's clang-format. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
6644fc433d |
Merge pull request #2447 from DeusData/fix/issue-1146-laravel-withrouting-prefix
fix(routes): compose the Laravel 11+ withRouting api prefix into Route paths (#1146) |
||
|
Martin Vogel
|
01fd6cb5e6 |
Merge pull request #2342 from DeusData/feat/callable-identity-plumbing
feat(identity): callable identity plumbing, no language enabled (#2061) |
||
|
Martin Vogel
|
a245618a64 |
wip(ci): checkpoint hybrid test-impact implementation
Preserve the current selector, frozen-input inventory, result codec, coverage collection, runner filtering, and isolated Windows venue work. Component-level regression proofs are recorded in the PR handover. The live classified provider, public execution integration, artifact admission, fresh oracle/shadow evidence, and full platform gates remain incomplete. This checkpoint does not enable or claim verified narrowing. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7ddbe66542 |
fix(routes): compose the Laravel 11+ withRouting api prefix into Route paths (#1146)
Laravel 11+ registers its route files in bootstrap/app.php through
Application::configure()->withRouting(web: ..., api: ..., apiPrefix: ...)
and mounts every file passed as `api:` under `apiPrefix` (default 'api').
That prefix lives in a different file than the routes, so per-file
extraction -- which already composes the in-file Route::prefix()->group()
chain (#952) -- minted `GET /users/me` for a route Laravel serves at
`GET /api/users/me`. The Route qualified_name is the join key for
cross-repo HTTP matching, so real callers of /api/... never linked, and an
api route could silently merge with a same-path web route.
Fix: a small Laravel mount resolver, src/pipeline/laravel_routing.c. For
each PHP file that registers a path-shaped route, both route passes (the
sequential calls pass and the parallel resolver) look up the nearest
ancestor bootstrap/app.php once, parse it with the PHP grammar, and, when
its withRouting(api: ...) names that file, prepend the apiPrefix to every
Route minted from it. Only source facts are used: an `api:` entry must be a
__DIR__ / dirname(__DIR__) concatenation or a base_path() call over a
string literal (single path or array), and a given apiPrefix must be a
string literal. A non-literal apiPrefix, positional arguments or a
`using:` callback leave the paths untouched, the `web:` file is never
prefixed, and nothing is inferred from file names. The lookup keys only on
the repo path and the file path, so full, incremental and delta runs
compose the same identity.
Proof (origin/main
|
||
|
Martin Vogel
|
3ecac479a6 |
test(harness): per-test selection and per-test coverage profiles
The test-impact selector has to hand the C runner individual tests, and the map behind it has to know what each test executes. The runner could do neither: argv selects whole suites. Selection. CBM_TEST_ONLY (comma-separated) and CBM_TEST_ONLY_FILE (one per line; blank lines and # comments allowed) take suite:test tokens, and are unioned when both are set. A suite no token names never executes its body. Inside a named suite only the named tests run; the rest are counted as deselected and the summary reports the count. Argv still selects suites and the tokens narrow inside that selection, so the parallel harness hands every per-suite process the same list with no change of its own. Tokens are strict: one that names a suite this build does not register, or a test its suite never ran, is printed and fails the run. An unreadable file, a malformed token, and a selection that names no test stop the run before any suite. Without a selection the output is byte-identical to before, and --list-suites ignores the selection. Coverage. TEST_COVERAGE=1 compiles clang source coverage into our own code (the one-shot sources, the LSP unity object, the preprocessor; vendored objects stay uninstrumented) and the runner's hooks into the test runner. `make test-runner-cov` and `make cbm-cov` build the instrumented runner and product binary into build/cov. With CBM_TEST_COVERAGE_DIR set the runner writes <suite>/<test>.parent.profraw for itself around each test, points LLVM_PROFILE_FILE at <suite>/<test>.%p.profraw for the children the test spawns, and sends what runs between tests to _setup.<n>.profraw. No other build compiles any of it. tests/test_harness_test_only.sh pins the selection contract against the built runner and runs as Step 2b of scripts/test.sh, before the suites. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c61b3806bf |
Merge pull request #2449 from DeusData/fix/build-seams-and-cx-tmp
fix(build): make prod objects follow TEST_SEAMS; stop cx tmp leak |
||
|
Martin Vogel
|
11867dfa74 |
feat(identity): callable identity plumbing, no language enabled (#2061)
Overloads that share a base QN (Java/C#/C++/Kotlin/Swift/Scala/ObjC) merge
into one graph node today, and that node collects the CALLS of every
overload. The agreed cure is one signature-qualified callable identity for
all overloading languages:
callable_qn := base_qn [<tparams>] (params) [cvref]
This change lands the plumbing with every language still at
CBM_CALLABLE_ID_NONE, so the graph is byte-identical; each language is
switched on later in its own change (with its index-format bump).
- internal/cbm/callable_sig.{c,h}: the per-language mode side table
(NONE|TYPED|LABELED_TYPED|LABELED|ARITY; kept out of CBMLangSpec like
cbm_string_dispatch_suffixes, since ~160 positional rows would otherwise
all need the member under -Wmissing-field-initializers), the shared
builder cbm_callable_sig[_mode] for the def and the call-scope QN, and
the inverse cbm_qn_callable_base_len[_named]. Suffix contract: no '.',
'::' or '->'; whitespace-normalized; comments/annotations/attributes
dropped; qualified types keep their last segment; '->' spelled '=>';
variadic '[]' (Java) / '~'; C++ top-level cv dropped, arrays decay,
cvref kept; <= 200 bytes, longer ones keep whole leading entries and end
in '#<fnv1a64 of the full suffix>)'. The builder is iterative (cursor
walk, bounded declarator loop) and self-checks against the inverse.
- The inverse is anchored to the node's bare name wherever the name is
known: real QNs already end in ')' without being callables (a Java field
whose name is its declarator `X = f()`, make `$(VAR)` targets), and only
the anchor makes the inverse exact on them.
- Leaf splitters strip a suffix first (no-op today): registry by-name key
(indexed by the bare name for a suffixed callable), cbm_lsp_bare_segment,
cbm_pipeline_qn_class_method_tail, pxc_last_component, pxc_qn_leaf,
go_index_visit_node, collect_prod_defs, sg_qn_prefix_len.
- New unused fields: CBMDefinition.qn_sig_off, CBMResolvedCall.callee_sig
(relocated by result compaction), CBMRegisteredFunc.sig, CBMLSPDef.sig.
The lsp_surface codec is unchanged (no format bump in this change).
- get_code_snippet and trace_path gain base-QN tiers
(cbm_store_find_nodes_by_qn_base) that find nothing until suffixes exist.
- store: cbm_store_find_nodes_by_qn_suffix built its LIKE pattern in a
512-byte buffer (a long suffix was truncated into a pattern that matched
nothing) and did not escape '_'/'%' (`my_func` also matched `myXfunc`).
The pattern is now heap-built, escaped, and matched with ESCAPE '\'.
Proof: full sorted node+edge+lsp_surface dumps are byte-identical before
and after on 7 corpora (this repo, elasticsearch libs, Exposed, dotnet
System.Linq, System.Collections, coreclr/gc, a C corpus).
Refs #2061
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
94097f0727 |
Merge pull request #2341 from DeusData/fix/git-spawn-local-env
fix(git): clear git's repository-local env for cbm's own git children (#2003) |
||
|
Martin Vogel
|
4c466fc4f9 |
fix(build): make prod objects follow TEST_SEAMS; stop cx tmp leak
Two build/test-hygiene defects. 1. Prod vendored objects ignored TEST_SEAMS (and MEMWASTE flips) Why: CFLAGS_PROD carries $(TEST_SEAM_DEFINE), but the objects compiled once and linked into the product binary -- prod_lsp_all.o, the grammars, ts_runtime -- used GRAMMAR_CFLAGS without it. A TEST_SEAMS=1 binary therefore silently lacked the seam code inside lsp_all (ts_lsp.c budget probes), and as soon as cbm.c references a seam symbol defined in lsp_all -- the pending #1527 change adds cbm_lsp_work_steps in lsp/scope.c -- `make cbm TEST_SEAMS=1` fails to link, which breaks scripts/test.sh and the smoke/_test CI builds. Adding only the define is unsafe: none of these objects depended on $(BUILD_DIR)/.build-config, so flipping back to TEST_SEAMS=0 would link the stale seam-built lsp_all into a release binary. MEMWASTE had the same gap: its flags reach every vendored prod object through MEMWASTE_VENDOR_CFLAGS. Fix: GRAMMAR_CFLAGS carries $(TEST_SEAM_DEFINE) (as GRAMMAR_CFLAGS_TEST already carries the define), and every prod object whose flags take a BUILD_CONFIG_SIG variable depends on the .build-config stamp: prod_%.o, prod_ts_runtime.o, prod_lsp_all.o, prod_preprocessor.o, prod_lz4.o, prod_lz4hc.o, prod_zstd.o, prod_sqlite3.o, prod_tre.o. prod_mimalloc.o and the unixcoder blob take no config variable and stay off the stamp. Proof (macOS arm64, GNU Make 3.81): - #1527 tree, `make cbm TEST_SEAMS=1`: undefined _cbm_lsp_work_steps (3/3). With this Makefile it links and prod_lsp_all.o defines the symbol; dropping only the define brings the link error back. - main, TEST_SEAMS=1: prod_lsp_all.o had 0 seam symbols, now 1. - Flips 1 -> 0 -> 1 -> default each recompile the 169 stamped objects including prod_lsp_all.o; cbm_ts_lsp_test_budget_remaining in prod_lsp_all.o and in the binary is 1/0/1/0. A repeat build at the same config compiles and links nothing. The TEST_SEAMS=0 and default binaries are byte-identical to the pre-change release binary. - Revert-check: without prod_lsp_all.o's stamp prerequisite, a seams -> release flip keeps the seam object and the release binary carries the seam symbol (different sha256). - Release `make cbm` before/after: byte-identical binary (sha256 048e459dfbb4...), build shown reproducible from scratch. 2. The complexity suite leaked its temp corpora Why: cx_measure_pair and cx_measure_bigpkg_pair created /tmp/cbm_cx_base_*, cbm_cx_dbl_*, cbm_cxbig_a_* and cbm_cxbig_b_* and never removed them -- four dirs per run, more on failure retries. Fix: one cx_measure() helper builds both corpora, runs the pipeline on them and removes them with th_rmtree on success and on every failure path, touching only the dirs it created (the temp root is shared with concurrent runs). Only the metrics outlive the call; gates and counters are unchanged. Proof: /tmp listing diff around `build/c/test-runner complexity`: +4 dirs per run before (3/3), +0 after (3/3). Injected corpus-build failure, pipeline-run failure and assertion failure each leave +0. Suites: complexity 5/5 (0 leaked); TEST_SEAMS=1 build of test-runner and cbm, then complexity 5/5; make lint-ci passed. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c0a06b3d0a |
fix(sql): keep literal INSERT rows out of the SQL parse (#1735)
A mysqldump-style .sql file is a few CREATE TABLEs followed by megabytes of `INSERT ... VALUES (..),(..),...` rows. Tree-sitter built a full tree for every row (~70 bytes of tree per source byte; 0.26 s/MB with MySQL \' escapes, which the grammar does not know and error-recovers from) until the 5 s parse budget ran out, and the whole file, tables included, was skipped as "parse timeout". The reporter's database/sql/world.sql was dropped this way. The rows contribute nothing to the graph. A linear, quote- and comment-aware pre-scan (internal/cbm/sql_values.c) finds INSERT/REPLACE ... VALUES lists and excludes every tuple after the first that consists only of literals: strings (standard '' and MySQL \' escapes, X'..'/B'..'/N'.., _charset introducers), numbers, NULL, TRUE, FALSE, DEFAULT. A tuple holding a subquery, a function call, an identifier or a quoted identifier stays in the parse, and so does one whose string spans a raw newline. The excluded bytes go to ts_parser_set_included_ranges as the complement, so byte offsets and line/column positions of all kept text are unchanged. The ranges are cleared right after the parse because the parser is thread-local and reused. This is a syntax rule, not a work cap: the same file always gives the same ranges, and the parse budget is unchanged. Measured on generated mysqldump-shaped dumps, release build, same machine, index worker's own numbers (base = origin/main): dump extraction pipeline peak RSS graph 8 MB std '' 1310 -> 23 ms 1431 -> 139 ms 625 -> 37 MB identical 8 MB MySQL \' 2712 -> 24 ms 2827 -> 146 ms 474 -> 37 MB identical 32 MB MySQL \' 5145 -> 82 ms 5467 -> 396 ms 1132 -> 62 MB timeout -> indexed 59 MB MySQL \' 5130 -> 138 ms 5667 -> 674 ms 1157 -> 89 MB timeout -> indexed "identical" = every stored node and edge, Module end line included. For the 32/59 MB dumps the base run skipped the file as "parse timeout" (Module + 3 Tables + their DEFINES/USAGE missing). The fixed graph matches the 8 MB graph except for the Module's end line. With standard escapes the extraction and the stored graph are identical to the full parse. With MySQL \' escapes the full parse's error recovery swallows rows next to a misread escape, a subquery row among them, and the cut parse keeps those rows. So it loses nothing the full parse found outside the dropped rows and recovers what the full parse lost. The only things the full parse extracted INSIDE the dropped rows were junk usages (DEFAULT and _binary read as identifiers, fragments of misread strings) that name nothing. Tests (parse_coverage, pipeline; a CBM_TEST_SQL_FULL_PARSE_ON seam turns the exclusion off for one file so the same source can be compared both ways): - scanner cases: comments, strings, dollar quotes, ON DUPLICATE KEY UPDATE, VALUES outside INSERT, unclosed tuples, exact kept positions - extraction equals the full parse on a small dump (standard escapes); nothing lost outside dropped rows (MySQL escapes) - the subquery and function-call rows are still parsed - the tree does not grow when the row count doubles (a count, not a clock) - a 25 MB MySQL dump under the production budget is indexed, not "parse timeout" (asserts the outcome, not the time) - stored nodes and edges of a pipeline index equal the full parse's Refs #1735 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
40dad610ee |
fix(git): clear git's repository-local env for cbm's own git children (#2003)
Git exports repository-local variables (GIT_DIR, GIT_WORK_TREE,
GIT_INDEX_FILE, GIT_COMMON_DIR, ...) into hooks, and some editors export
them into their terminals. They take precedence over `git -C <dir>`, and
every git child cbm spawned inherited them. A cbm started from such an
environment therefore resolved every project against the CALLER's
repository: git_context reported a plain directory as is_git=true with
the caller's worktree root, branch and HEAD, the history pass attributed
the caller's commits to it, and the watcher / detect_changes /
artifact git calls read (and `config merge.ours.driver` could write) the
wrong repository.
Fix: one production helper, src/foundation/git_env.{c,h}, holds the
15-variable list `git rev-parse --local-env-vars` prints and builds the
child environment with exactly those removed -- a POSIX envp array or a
Windows CREATE_UNICODE_ENVIRONMENT block (case-insensitive names). It is
applied per spawn; cbm's own environment is never modified (no unsetenv
in the parent, which would race other threads):
- cbm_popen_git(cmd): read-mode popen for git command lines. POSIX spawns
/bin/sh -c via posix_spawn with the scrubbed envp (stdin /dev/null,
pipe ends close-on-exec, reaped by cbm_pclose); Windows passes the block
to the existing handle-isolated CreateProcessW spawn. Fails closed.
- cbm_proc_opts_t.strip_git_repo_env: cbm_subprocess_spawn passes the
scrubbed envp to posix_spawnp / the fork+exec child, or the block to
CreateProcessW. Fails closed.
- Every git spawn uses them: git_context, pass_githistory, artifact
(4 sites), the watcher's git runner, and detect_changes' git commands.
search_code's grep scan keeps the inherited environment.
Tests (RED before, green after, RED on revert of the helper and of the
call sites):
- git_context::git_context_ignores_inherited_repo_env -- GIT_DIR and the
hook set pointing at a decoy repo: a plain dir is is_git=false, a
fixture repo reports its own HEAD and branch, the decoy is untouched.
- git_context::githistory_ignores_inherited_repo_env -- the history pass
on a plain dir sees 0 commits (was the decoy's 4).
- subprocess::subprocess_strip_git_repo_env_is_per_child and
popen_git_strips_repo_env_and_reports_exit_status -- all 15 variables
are gone in the child, unrelated variables pass through, the parent
environment is unchanged, and cbm_pclose keeps pclose() status
semantics.
Proof (macOS arm64, GIT_DIR/GIT_WORK_TREE/GIT_INDEX_FILE/GIT_PREFIX
exported at a decoy clone, `cli index_status verbose=true`): a plain dir
went from is_git=true, worktree_root=<decoy>, branch=<decoy's> (v0.11.0)
to is_git=false; a fixture repo went from the decoy's root/branch/HEAD
to its own root and branch.
Refs #2003
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
  Clément DeustandClaude Fable 5
|
f3bdf11a03 |
feat(pipeline): link markdown file references into the graph
Markdown docs reference other repo files constantly (a coding-standards doc links the modules it governs, a README points at entry points) but none of that surfaced as edges, so fan-in queries were blind to documentation hubs. In an A/B retrieval benchmark I ran in August on a docs-heavy repo, the graph's top fan-in answer was off by roughly 17x: the most-referenced file was a standards doc with 173 references and zero inbound edges. New pre-dump pass, pass_doclinks.c, modeled on pass_configlink.c: three strategies emit REFERENCES_FILE edges between existing File nodes only (unresolvable targets are dropped, the pass never invents nodes): inline links, backtick paths, bare path mentions. Targets resolve against the referencing file's directory and the repo root; repeated references collapse into one edge carrying strategy, confidence and count. Registered in the pre-dump sequence after configlink and in the incremental post-passes; REFERENCES_FILE added to the skill's edge-type list and the structural/language contract tests. Tests mirror test_configlink.c (real files in a tmpdir, File nodes in a gbuf, run the pass, assert edges): inline link, backtick, bare mention, http/anchor ignored, anchor-suffixed file link, dedupe with count, relative-vs-root resolution, unresolvable-target guard, NULL repo_path skip. A companion change adds the same linking for shell files (source lines and script invocations); split out to keep each change reviewable. Validation: scripts/build.sh clean; focused serial runner (doclinks, configlink, pipeline, edge_structural, lang_contract) 340 passed under ASan/UBSan; cppcheck clean on the new files with the repo's flags. Signed-off-by: Clément Deust <clement.deust@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Martin Vogel
|
92abefa3f5 |
Merge pull request #1723 from liuchong/feat/index-resource-discovery
feat(index): add opt-in discovery resource limits |
||
|
Martin Vogel
|
6f290e11d0 |
build: make each grammar object depend on the sources it includes
A grammar unity object, grammar_<lang>.c, #includes its own vendored parser.c and, where the grammar has one, scanner.c. Those files were prerequisites of nothing. Editing a vendored scanner therefore left the object untouched and the edit silently absent from the binary. This is not hypothetical: the properties scanner fix in the previous commit appeared not to work at all. An A/B said 3 stalls against 2 - no effect - and the reason was that prod_grammar_properties.o predated the edit by five minutes and `nm` still showed the old `_reached_eof` symbol sitting in it. The measured "no effect" was a stale object, not a bad fix. The build already treats its other two unity objects this way and says why: LSP_UNITY_DEPS and TS_RUNTIME_DEPS both list the sources and headers they pull in, because this Makefile deliberately does not use compiler auto-dependency (-MMD) generation. Grammars were the gap, and MANIFEST.md's "Local source patches" table means patching a vendored scanner is an expected thing to do - so the trap was reachable by design. Secondary expansion is what lets one pattern rule name each grammar's own directory ($$* is the stem, grammar_properties -> properties). Applied to all three grammar object rules: production, test and tsan. Non-grammar objects match no directory and gain no prerequisite. Verified by touching internal/cbm/vendored/grammars/properties/scanner.c and watching exactly one object rebuild. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
81264814f1 |
feat(qa): memory + CPU waste sanitizer over the memory core
Answers a question the existing tooling could not: of everything we
allocate, how much is never used?
Lanes, all driven by scripts/memwaste.sh:
event every allocation observed through the memory core — size, age,
churn, and whether the bytes were ever written
access SanitizerCoverage load/store instrumentation, so "allocated and
never touched" is measured rather than inferred
scaling k vs 2k replicas through clang profile counters, to catch
super-linear work per file
padding allocator slack per size class
ratchet the numbers recorded per venue so a regression is visible
The event lane needed a foreign-free counter: on Windows the override sees
frees for blocks it never allocated (they come from the CRT), and counting
those as untracked tripped the soundness gate. mem_events now separates
them (14 daemon / 5 worker on the Go corpus).
Two portability fixes the lanes surfaced, both real:
- the enable decision must not be cached before `environ` exists. Under
-fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our
wrapper before glibc sets up environ, so the lane silently disabled
itself and produced no dump at all on Linux.
- the access map needs 48 bits: arm64 Linux hands out addresses above
2^47, which a 47-bit map dropped on the floor.
mem_profile is retired into this: it measured a strict subset and had its
own sampling story.
Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on
all three platforms report-only, and pr.yml/release.yml carry the wiring.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
7b77fd4834 |
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (
|
||
|
Martin Vogel
|
da3258f3c7 |
feat(mem): a central allocation core, a linter that enforces it, and phase attribution in the index pipeline
Memory in this project is now allocated through ONE core. This lands the
core, the gate that keeps new code on it, and the instrumentation that lets
an index run say which class of memory grew in which pass -- the three things
the 2026-09-13 audit found missing.
WHAT THE AUDIT FOUND
There was no centralized allocation layer. foundation/mem.h was policy and
measurement only (budget, RSS, over_budget, memory map, phase marks) with no
alloc/free wrapper, so the budget could be observed after the fact and never
enforced. Raw allocator calls in the tree: 4086 across 92 files. The
extraction engine had already adopted arenas (1301 call sites); the graph
buffer -- which holds the 35 GB peak of a kernel index -- used none,
allocating every 64 B node, 48 B edge and every name/qualified_name/
properties_json individually.
Two further facts made a self-accounting core the only workable design:
1. cbm_mem_map_collect() walks THIS thread's mimalloc heap only. Walking
the process-wide heap is a data race TSan caught on macOS, so it is
deliberately not done, and an 18-worker index attributes almost
nothing -- the rest lands in residual.
2. The mimalloc global override is ON for Linux/MinGW and permanently OFF
for macOS (two-level namespace: this binary's free becomes mi_free while
system libraries keep allocating from the system zone, and a pointer
crossing that boundary aborts). On macOS ordinary malloc is served by
the system allocator; the startup audit reports owned_classes=0/6. Any
accounting that assumes mimalloc owns the pointer is blind on an entire
platform.
THE CORE (src/foundation/mem_core.{h,c})
cbm_alloc / cbm_calloc / cbm_realloc / cbm_mem_strdup / cbm_free, each tagged
with an allocation class (gbuf_node, gbuf_edge, gbuf_string, gbuf_index,
extract, semantic, dump, store, other). Per-class live bytes, live blocks and
peak, kept in atomics -- thread-safe by construction, identical on every
platform, independent of which allocator serves malloc. libc semantics are
preserved exactly (alloc(0) is freeable, free(NULL) is a no-op, realloc(NULL)
allocates, a failed realloc leaves the block intact) so adoption is a
mechanical rename and never a behaviour change.
No per-allocation header: a {class,size} prefix costs 16 bytes on ~100M blocks
at kernel scale, 1.6 GB of overhead to measure a memory problem. Sizes come
from the platform usable-size query (malloc_size / malloc_usable_size /
_msize), which is correct under either allocator and free. Where no query
exists (BSD) the request size is charged, which understates -- the safe
direction for a diagnostic. A mismatched class on free clamps at zero rather
than wrapping, so a small drift can never masquerade as a colossal leak.
Arena-backed allocators report in bulk through cbm_mem_class_add_external /
remove_external so their memory appears in the same table; the extraction
engine must not be rewritten to per-object allocation, that would undo the
batching that keeps its allocation count low.
THE LINTER (scripts/lint-memory-core.py, wired into make lint-ci)
Raw malloc/calloc/realloc/free/strdup/strndup outside the core is a defect.
4086 sites cannot migrate at once, so the gate is a ratchet: a checked-in
baseline (scripts/memory-core-baseline.txt) records each file's count and
the build goes red when any file grows, or a new file appears with any.
Files only ever go down; the baseline line is lowered in the change that
migrates the file, and --strict fails a stale baseline. Comments and string
literals are stripped first, so prose that mentions malloc( cannot trip it --
the security audit already bit on exactly that with fork(. Proven both ways:
one appended malloc turns it red naming the file (83 -> 85), restoring turns
it green. Scans src/ and internal/cbm/: cli, mcp, daemon, store, pipeline.
PHASE ATTRIBUTION (src/pipeline/pipeline.c)
cbm_mem_phase_mark and the new cbm_mem_class_log now fire at pipeline.begin
and at every pass.timing site: parallel_extract, the six sequential passes,
the eight predump passes, tests and dump_and_persist, with peaks reset per
index. Both instruments existed in foundation/ and were wired only into MCP
request handling, never into the index pipeline -- which is where the memory
is. Locating the kernel peak this week required an external RSS sampler
because nothing in-process could say which phase it was in.
PRESSURE PRIMITIVES (system_info.c, mem.{c,h}, platform.h)
cbm_system_available_ram() -- macOS host_statistics64 (free + inactive +
purgeable), Linux MemAvailable, Windows GlobalMemoryStatusEx; 0 when unknown,
never cached -- and cbm_mem_system_under_pressure(), true below 12.5% of RAM
available and false when unknown so nothing ever aborts on a guess. Landed
UNWIRED: an earlier draft used them to let the over-budget latch press on
whenever the system was not under pressure, and that permitted the very
overshoot this work exists to end (Decision A's tests at
test_pipeline.c:13481 and :13583 went red under it, correctly). They are the
backstop for the next step, admission control, which keeps a run under budget
before an allocation rather than measuring it after.
mem suite 10 new tests green; pipeline + mem 339 passed 0 failed; lint-ci
clean including the new gate.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
 刘冲
|
21ae02154a |
feat(index): add opt-in discovery resource limits
Indexing accepts whatever a repository contains. A tree carrying a vendored monorepo, a generated dump, or a runaway build directory is discovered in full, and the first sign of trouble is a host under memory pressure with nothing that attributes it to indexing. Add two opt-in limits evaluated during discovery against accepted source files only: index_max_files and index_max_source_mb. Both default to off, so nothing changes until an operator sets one. Crossing a limit fails the whole attempt with a structured resource_limit_exceeded result naming the resource, the observed value and the limit; no partial graph is published, and an existing serving index keeps answering. Limits are read from the CLI-managed _config.db and are not MCP request arguments. A supervised parent replaces any caller-supplied policy before spawning its worker, and the worker rejects a missing or incomplete contract, so the CLI, the daemon and the supervised worker all enforce the same decision. Both keys reach an operator through the existing config get/set/list/reset with no new subcommand. `set` suppresses its own generic message for them because the policy writer names the precise reason -- so that writer speaks on every failure it can return, including a validated value whose write then fails on a database that cannot be written. Exiting non-zero in silence is not an acceptable answer from a CLI. The two shell regressions that hand-roll the supervisor's worker argv carry that contract as well. Without it the worker exits before either guard can observe anything, and the guard would go quietly vacuous. Signed-off-by: 刘冲 <mail@liuchong.dev> |
||
|
Martin Vogel
|
d40ed1a6bb |
Merge remote-tracking branch 'origin/main' into fix/1954-uninstall-symlinked-config
# Conflicts: # tests/test_cli.c |
||
|
Martin VogelandClaude Fable 5.1
|
e5aabdf86e |
fix(cli): follow user-owned symlinked config files in the editors
Every config editor (JSON-like, TOML, YAML, text) opens with O_NOFOLLOW and refuses symlinks, so a link planted at ~/.cursor/mcp.json cannot point a privileged writer at another file. Users keep exactly those files as symlinks into a dotfiles checkout; the blanket refusal made install and uninstall fail on them and, before the previous commit, turned uninstall into data loss (#1954). Decision C: a symlink the INVOKING user owns, resolving to a regular file the same user owns, is followed — as an explicit opt-in of the agent-configuration writers only. cbm_install_agent_configs and the uninstall activation register the user's configuration roots (HOME, XDG_CONFIG_HOME) for the duration of their work; a link whose own directory is outside every registered root, or any link seen by another editor caller, stays refused, so repository paths never follow. One shared helper (src/cli/config_edit_path.c) runs the race-safe sequence: lstat(link) → owner == invoking uid (a root process follows root-owned links only) → realpath → the target's parent directory opened as a dirfd (O_DIRECTORY|O_NOFOLLOW; owned by the invoking user or root, not writable by others unless sticky) → openat(dirfd, base, O_NOFOLLOW) → fstat: regular file, same owner, link count 1, and the same (dev, ino) as stat(realpath). Each editor reads through that validated descriptor, stages its temp file with openat(dirfd, …, O_CREAT|O_EXCL|O_NOFOLLOW), runs its pre-publish snapshot comparison against the resolved path, and commits with renameat on the pinned dirfd, so the link itself is never replaced. Every other link is refused with an observable reason (surfaced in the uninstall diagnostics): another owner, root over a non-root link, dangling, a directory or special target, a hard-linked or swapped target, a foreign or world-writable parent. Whether the target must also stay under a root is a pending user decision behind CBM_CONFIG_EDIT_TARGET_UNDER_ROOT (default: anywhere the user owns). Windows never follows reparse points. Tests: each editor suite gains a same-owner link edited in place under the opt-in (link preserved and still pointing at the same target, temp file staged beside the target, bytes identical to the same edit on a plain file, no stray temp files) and keeps its refusal cases; the JSON-like suite adds no-opt-in, outside-root, dangling, directory, identity-skew and world-writable-parent refusals. The foreign-owner and root-run refusals move the observer through a test seam (a test cannot create a foreign-owned link without root; a real root run demotes the link first). test_cli.c drives `uninstall -y` end to end through a symlinked ~/.cursor/mcp.json. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
fbe9eae11c | Merge branch 'main' into fix/1692-csharp-multi-attribute-decorates | ||
 Joshua RichterandClaude Opus 5
|
219b348263 |
test(coverage): pin the truncation marker, the partial ceiling and the grammar limit (#963)
Phase 5. Four test groups, each checked RED before it was kept. - The Studio Export range join puts ONE ",+<N>" marker at the end with the summed drop count. A marker left mid-string makes every reader stop there and silently lose the ranges after it. Reaching the join through the pipeline needs an export file with 256+ error regions across two <Class> elements, so it goes through a test seam, following the pattern already in this repo (CBM_COVERAGE_MARKER_TEST_API). - check_index_coverage emits every range in front of a marker, never turns the marker's digits into a range, and reports "truncated" from BOTH caps — the producer's and its own 256 limit. - test_index_resilience now has a ceiling beside its floor: exactly one of the two fixture files is flagged, the clean neighbour is absent, and the range does not cover the whole file. - The three _Thread_local forms are pinned as measured. Only the array form fails today; the plan's Phase 0 also listed the pointer form, and that is wrong on the grammar shipped now. Also fixes 13 clang-format violations the earlier commits on this branch left in cbm.c, mcp.c and pass_definitions.c. `make -f Makefile.cbm lint-format` would have failed CI. The changes are whitespace only — the two reflowed tool descriptions concatenate byte-identically, so no output moved. Full suite: 7735 passed, 28 failed, 7 skipped. The 28 are the pre-existing cli install/uninstall failures, identical at clean HEAD. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Joshua Richter <jrichter5781@gmail.com> |
||
|
Martin Vogel
|
683d4bc312 |
Merge branch 'main' into fix/1692-csharp-multi-attribute-decorates
Brings the branch onto current main (
|
||
 Martin VogelandPR9000
|
fece72b049 |
fix(build): put the whole GOT inside the RELRO window on ELF targets
Adds -Wl,-z,relro and -Wl,-z,now to ELF_HARDENING_FLAGS, and asserts the
resulting property on the produced artifact rather than on the compiler's
willingness to accept a flag.
The shipped Linux binaries measurably lacked the property. Built the way
.github/workflows/_build.yml builds them (STATIC=1, gcc 13.3 / GNU ld 2.42,
Ubuntu 24.04 aarch64), PT_GNU_RELRO ends at 0x11DA0000 in both cases:
without -z now: .got 0x11d9ec20+0x13c8, .got.plt 0x11d9ffe8+0x40
-> .got.plt ends at 0x11DA0028, 40 bytes PAST the window.
Those GOT slots stayed writable for the whole process
lifetime, in every Linux binary shipped to date.
with -z now: .got.plt folds into .got, the whole GOT sits inside the
window and is re-mapped read-only after startup.
Both flags are free at runtime for a static binary -- everything resolves at
link time, so there is no lazy binding left to pay for. -z relro is already
this toolchain's default and changes nothing here; it is named anyway so the
property stops depending on one distro's spec file, since the musl/portable
and glibc-floor images are different toolchains and a default is not a
guarantee.
A1c/A1d in scripts/ci/check-binary-composition.sh assert the OUTCOME on the
binary via readelf, for the same reason A1 exists beside the .note.GNU-stack
annotation: a flag the linker accepts is not evidence the artifact gained
anything. PR #1138 demonstrated that directly -- its -pie was silently
discarded under -static, and its -D_FORTIFY_SOURCE=2 silently DOWNGRADED
Ubuntu's default of 3, both while every compile probe passed.
The readers use awk rather than a short-circuiting reader on purpose: a
reader that exits at the first match leaves the upstream taking EPIPE, and
under `set -o pipefail` the satisfied case is reported as the failing one.
Deliberately NOT taken from #1138: -D_FORTIFY_SOURCE=2 (a downgrade on the
release platform), -fPIE/-pie (inert under STATIC=1), -fstack-protector-strong
(real but unmeasured cost, and #1138 injected it into SQLite, tree-sitter and
mimalloc -- the indexing hot loop), and the $(shell) probe machinery (~10
compiler spawns per Makefile parse, making the binary a function of the
builder's toolchain).
Co-authored-by: PR9000 <119280965+PR9000@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
 Martin Vogelandpetercoxphoto
|
e5f3aab02e |
feat(pipeline): index-time importance scoring (weighted degree)
Score every Function/Method/Class node during indexing and persist the
result as a numeric "importance" key inside the node's existing
properties_json:
importance = sqrt(num_refs) * priv * generic * distinct * test_penalty
num_refs is the incoming CALLS + USAGE degree. The multipliers demote
private (leading-underscore) and generic names (defined in >= 5 distinct
files), promote distinctive identifiers (snake_case or camelCase, len >=
8), and demote test scaffolding -- symbols in a test file per the graph's
own cbm_is_test_path() classifier, or targets of an incoming TESTS edge.
Weighted degree only; a transitive (PageRank) refinement is deliberately
not built here until it can be shown to beat this on a fixed judgment
set.
No schema change and no CBM_INDEX_FORMAT_VERSION bump: the score lives in
the existing properties_json TEXT column, so no index is forced to
rebuild and indexes written by older builds simply lack the key.
Three failure modes this shape invites, all of them silent, are addressed
directly:
* Cost. The generic-name multiplier needs |{files a name is defined in}|.
Computing that per NODE with a pairwise file-path scan is O(k^3) in a
same-name group of size k -- ruinous on real corpora, where a single
Java name group reaches k > 4000. Here the count is computed once per
distinct NAME and memoized, and file paths are deduplicated through a
hash set, so total distinct-file work is linear in the graph. Measured
cold full index of a 666 MB Java corpus, same -O2 binary: 85 s with the
pass disabled, 80 s with this implementation, 481 s with the per-node
shape.
* Registration. PREDUMP_PASS_COUNT is now derived from the pass table
with sizeof instead of being hand-written. A hand-written count that
lags an appended entry silently skips the last-registered pass with
every test still green.
* Incremental. The incremental post-pass sequence runs the pass too, over
a graph rehydrated from the store whose nodes already carry the key.
The write-back overwrites an existing key in place instead of appending
a second one, which would otherwise corrupt properties_json into
{"importance":1.0,...,"importance":2.0}.
tests/test_importance.c covers the formula and binds all three failure
modes; the complexity suite gains a GATED counter asserting the pass's
same-name-group work stays linear (~2x, not ~4x) under a corpus doubling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: petercoxphoto <20230727+petercoxphoto@users.noreply.github.com>
|
||
|
Martin Vogel
|
feb720809f |
Merge pull request #1104 from LA-10/fix/769-file-qn-keeps-extension
feat(store): index-format rebuild boundary for the File-QN change |
||
 LA-10
|
4d9da40b36 |
test(769): end-to-end sibling regression + index-format rebuild boundary Refs #769
Signed-off-by: LA-10 <leenshuhail1@gmail.com> |
||
 Thomas Dyar
|
ee126155ab |
feat(pipeline): add Ensemble production topology routing pass
Parses XData ProductionDefinition blocks to extract production topology
and emits Route/ASYNC_CALLS/HANDLES graph edges for Ensemble
interoperability routing.
Key design points:
- Emit Route nodes (qn: __route__ensemble__<prod>.<item>) with props
{broker, class, enabled, production} — visible in get_architecture
and search_graph at the same priority as HTTP routes
- Emit ASYNC_CALLS edges from source method/item to target Route node
via SendRequestSync literal scan and TargetConfigName(s) settings
- Emit HANDLES edges from each item's entry-point method to its Route
node, completing the code-to-topology link
- Fix scan_source_for_send_targets: scope search to calling method body
using brace-depth tracking to eliminate cross-method contamination
- Handle TargetConfigNames (plural, comma-separated) by emitting one
ASYNC_CALLS edge per token
- Segment-anchored class_name_matches() prevents prefix collisions
("Ens" not matching "Ens.BusinessService")
- Language-gate: skip entirely when no ObjectScript nodes present
Tests: pipeline_ensemble_routing_edges (Route nodes + ASYNC_CALLS +
HANDLES); pipeline_ensemble_routing_method_scoping (two methods, two
targets, no cross-contamination).
Fixes #467
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
|
||
 Martin Vogelandalexisperinger-ux
|
43ecc0988c |
feat(dbt): extract dbt model lineage from Jinja-templated SQL
A dbt model is an ordinary .sql file whose dependencies are written
{{ ref('other_model') }} or {{ source('group','table') }}, never as
literal table names. The SQL grammar cannot read those -- FROM {{ ref(
'x') }} is a parse error to it -- so the dependency structure of an
entire dbt project was invisible to the graph.
extract_dbt.c runs as a sub-extractor inside the normal indexing
pipeline. Per qualifying file it emits one Model definition (named by
the file stem, dbt's own model identity) plus one usage per ref()/
source() call; pass_usages resolves those into model -> relation
lineage edges like any other reference.
Model is a relation label alongside Table and View, which is what makes
the rest work without new plumbing: registry seeding, the central
relation veto, the incremental surface hash, search ranking and the
architecture queries all pick it up from cbm_label_is_relation. Sharing
one label class also means a model's source('raw','customers') resolves
onto a Table declared in a plain DDL migration in the same repository,
so dbt lineage and SQL lineage form one graph rather than two, while
the veto keeps model names out of every non-lineage consumer.
The pass is self-gating: SQL files only, and only those carrying a real
ref()/source() call. The dbt builtins are themselves the evidence, which
is cheaper than a dbt_project.yml lookup and more precise -- templated
SQL that is not dbt (an Airflow {{ ds }} parameter) produces no Model
node and no usages even inside a dbt repository.
{% macro %} definitions are deliberately excluded: the vendored
tree-sitter-jinja2 grammar has no node types for {% %} statements, so
they can only be recovered by a hand-written scanner that cannot see
comments or find {% endmacro %} for a correct span. Filed as follow-up
rather than approximated.
Tests cover the lineage, the last-string-argument semantics of both
builtins, the gate (against a plain-SQL control extraction), plain DDL
staying untouched, and an end-to-end pipeline case asserting model ->
model across files, model -> Table onto plain DDL, and cross-language
isolation. Disabling the pass reddens three of them; removing Model from
the relation set breaks lineage outright.
Implements the lineage half of #575.
Co-authored-by: alexisperinger-ux <alexis.peringer@iss-stoxx.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
 Jumaga2015
|
c4224cf3ef |
test: cover stacked C# attributes for #1692 (repro_issue1692)
A C# method carrying three stacked attributes ([Foo][Bar("x")][Baz]) must
end up with three DECORATES edges, one per bracket group. The fixture is
indexed through the real production pipeline (rh_index) and asserts
rh_count_edges(..., "DECORATES") == 3, so it exercises the MCP path rather
than the extractor in isolation.
Verified to actually reproduce: with find_jvm_modifiers() reverted to its
first-child lookup the suite goes red (decorates=1, expected 3), and green
again with the fix in place.
C#-only by design. Parsing the equivalent fixtures with the vendored
grammars shows `[A] [B] [C]` yields three sibling `attribute_list` nodes in
C#, while PHP's `#[A] #[B] #[C]` yields ONE `attribute_list` holding three
`attribute_group` children — the pre-fix lookup already found all of them,
so PHP could not hit this bug and a stacked-PHP fixture would pass either
way. Java/Kotlin/Swift use the `modifiers` wrapper, a single node that never
repeats. PHP's DECORATES path stays covered by mkc_c3_php8_attribute
(tests/test_matrix_known_classes.c).
Registered in repro_main.c and Makefile.cbm's TEST_REPRO_SRCS.
Signed-off-by: Jumaga2015 <jumaga2015@gmail.com>
|
||
|
Martin Vogel
|
0273250c33 |
Merge pull request #1310 from Yyunozor/fix/1264-hook-diff-aware-clang-tidy
fix(hooks): scope pre-commit clang-tidy to staged changes (#1264) |
||
|
Martin Vogel
|
51770de0b6 |
perf(ts): memoize expression-type evaluation per node
ts_eval_expr_type and ts_signature_for_call are mutually recursive: resolving a call evaluates its argument expressions once per lookup path (method dispatch + namespace fallback), and in tsc-compiled spread files the first argument is itself the next nested Object.assign(...) call — the same subtree re-evaluates once per enclosing level, 2^n total. The TS suite's objectSpreadRepeatedComplexity.js (3.6 KB, 5 nodes) measured 20.4 s; with the memo its eval cost is zero within measurement noise of a one-file control, and the microsoft/TypeScript corpus drops 37.5 -> ~24 s warm (nodes byte-identical, edges within the known scheduler jitter). Expression types are position-pure within a file pass (one node = one scope path; the per-file walk is single-threaded and deterministic), so one eval per node is the correct semantics, not a cache trade-off. The memo is a per-file, arena-backed, linear-probe table keyed on TSNode.id. Results produced under a depth-cap or budget bail are never stored: both bail sites bump a degradation counter, and a store only happens when the subtree completed clean — a degraded UNKNOWN can therefore never shadow a later full evaluation. The regression guard asserts work, not wall-clock: the nested-Object.assign shape must complete without exhausting the deterministic eval budget, read back through a new CBM_ENABLE_TEST_SEAMS accessor pair. The seam lives in the lsp_all unity object, so GRAMMAR_CFLAGS_TEST/TSAN now carry the seams define (test artifacts always have seams; prod never does). Verified RED without the memo (budget exhausted, suite 47.8 s) and green with it (3.2 s). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e700b6215f |
test(complexity): deterministic O(n^2) guard on work-counter ratios
Finding #1669 took an 11-corpus A/B across two release binaries. This suite makes that bug class fail a unit test in seconds, on every platform, from tiny corpora. Method: build k and 2k REPLICATED module copies, run the full in-process pipeline on both, assert counter RATIOS. Independent copies mean every extensive quantity — nodes, edges, Σ per-file registry defs — must grow linearly (ratio ~2). A files x corpus coupling makes per-file work itself grow with k and lands at ratio ~4. Ratios expose the exponent regardless of absolute scale, so 60-120 files suffice. Verdicts are pure functions of (code, input): gates ride ONLY on deterministic work counters and data-product counts, never on wall time. Throughput (nodes/s, edges/s) is information-only, written to private/benchmarks/complexity-<ts>.json (local, gitignored; skipped under CBM_SKIP_PERF where rates are meaningless). Two corpus shapes, both needed: - Independent modules (java/py/go/ts templates): catches cross-module contamination and dedup breakage. The #1669 bug is GREEN here — fully closed modules filter perfectly, which is exactly why it survived. - The growing shared package (bigpkg): one Java package whose file count scales with k — the real-repo shape (files concentrate in large packages). The JVM namespace filter branch makes per-file work track package size, so this corpus is the honest #1669 reproducer: ratio 4.00 RED on the pre-fix tree, 4.00 RED for a module-scoped overlay, 2.00 GREEN for the own-file overlay. It discriminated the correct fix design before the fix was written. Both legs of every pair exceed MIN_FILES_FOR_PARALLEL(50): below it the sequential path runs, which builds no shared registries and would be the wrong code path to gate (its per-file cost is bounded by the 50-file ceiling). Recorded but deliberately NOT gated, with reasons at the case: tail_candidates and fallback_rows are legitimately superlinear under replication until those scans are bounded, and measured ~1 ns/unit. Every ratio gate carries a non-vacuousness floor on the base counter so broken counter wiring fails loudly instead of green-washing (cbm_pxc_count_perfile_defs feeds the overlay path into the same counter the fallback path already used; wired for Java, extend to the TS overlay when touching ts_lsp). Dynamic coverage: languages iterate CBM_LANG_COUNT; embedded templates cover the LSP-hybrid languages, and tests/fixtures/complexity/<lang>/ dirs are auto-discovered so a new language joins the guard by dropping fixtures. Uncovered languages are listed in the report with the reason. The local report additionally carries per-language node/edge counts with ratios and a per-pass elapsed_ms table per run (captured via a TEE log sink during the in-process pipeline runs) — trend data for humans, still never a gate. Refs #1669. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
 Mauricio Offermann
|
c059396eb2 |
fix(build): give sanitized-build detection one spelling and a backstop
Four places ask "is this binary instrumented?" and each asked it differently.
One of them, the C# LSP bench, only recognised ASan, so TSan and MSan measured
an instrumented parse against the NATIVE 200ms budget. The other three carried
a hand-copied list of `__SANITIZE_*__` macros that nobody kept in sync.
That drift is what the TSan gap was made of. CFLAGS_TSAN never passed
SANITIZED_DEFINE, and the per-site conditions could not cover for it: they test
`__SANITIZE_THREAD__`, which is GCC's spelling. Clang — the compiler that leg
uses — announces thread instrumentation through `__has_feature(thread_sanitizer)`
only, and no site consulted it. The claim in
|
||
|
Martin Vogel
|
0a163d4f75 |
build(tsan): define CBM_SANITIZED_BUILD on the ThreadSanitizer leg
The previous commit widened the spawn-retry budget for sanitized builds, and it did nothing on TSan — the leg it was written for. `subprocess_run_spawn_failure` failed again on the very PR that was meant to fix it. CBM_SANITIZED_BUILD comes from SANITIZED_DEFINE, which keys off $(SANITIZE). TSan does not use that variable — it has its own TSAN_SANITIZE — and CFLAGS_TSAN never included SANITIZED_DEFINE. So the macro was undefined on that leg and every sanitized-budget branch compiled to its NATIVE value while running an instrumented, several-times-slower binary. The comment above SANITIZED_DEFINE already describes this exact failure for trap-UBSan: "the build system is the single source of truth for is this binary instrumented; compiler-specific probes miss clang's feature-check spelling and every non-ASan sanitizer". That lesson was recorded and the TSan leg was never wired up to it. Nor would compiler probes have saved this: clang spells thread instrumentation __has_feature(thread_sanitizer), not __SANITIZE_THREAD__. CFLAGS_TSAN now defines it unconditionally, which is honest — that flag set exists solely to build an instrumented binary. Checked the neighbours: MSan (scripts/msan.sh passes SANITIZE=) and the diag lane (passes SANITIZE= too) both already get the define. TSan was the only gap. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
93696aeb55 |
docs(security): publish the AV false-positive evidence; harden arm64 W^X
SECURITY.md gains an "Antivirus False Positives" section placed directly after the existing Verification block so it reuses those commands rather than restating them. It states the detection plainly, shows the measured evidence (the verdict inverts across architectures and link modes; identical macOS segment structure splits clean/flagged; entropy is 3.5-4.2 bits/byte, nowhere near packed), lists the cross-project precedent, and — deliberately — records that removing the embedded scripts and externalizing the assets did NOT move the detection. A negative result is still evidence, and publishing it is the point. It also documents the release policy, invites independent audit through the provenance/cosign/checksum path we already ship, and adds an `av-analysis` issue label route for anyone who finds something real. Tone is evidence-first: no vendor blame, and an explicit note that signing helps Windows but that no AV-honoured signing scheme exists for Linux ELF, so it is not a full answer. README carries a short pointer near Quick Start, where someone who just hit a Defender warning will actually look. Unrelated to AV, found while dissecting the artifacts: add -Wl,-z,separate-code. GNU ld enables it by default on x86-64 but not on aarch64, so arm64 emitted ONE R E PT_LOAD covering the whole image and mapped ~259 MB of tree-sitter parse tables executable at runtime. Section flags said A, not AX -- but the kernel applies segment permissions, so the section flags were never the control. The amd64 build of the same source mapped the same data R only. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |