mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-06 17:49:47 +00:00
a35671e962f1f2fb4e94deaf55e71cf05f6b3828
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
2fdcbfe288 |
doc-links(php): a property's, constant's or case's doc block belongs to its type
PHP properties, class constants and enum cases have no graph node, so their
doc blocks were never read: cakephp has 36 @see references there, symfony
21. The PHP leg now uses the driver's loose_docs hook: a /** */ block right
above a property, constant or case is harvested with the type around it as
the source (as the field test's prototype attached it), and resolved like
any other reference written in that type (self:: is the type).
The driver marks as the file's only what a hook pushes with the file
stand-in (the C leg's prototype docs); a hook may push with a definition of
the file instead.
Proof (ship build, Apple M5 Pro / 48 GB, base
|
||
|
Martin Vogel
|
80e000a92a |
doc-links(c): Doxygen @ref and @see names become MENTIONS edges, held until their audit
A Doxygen comment in a .c or .h file (a block opening with slash-star-star
or slash-star-bang, a run of /// or //! lines) names code with `@ref name`
or `\ref name` (family doxygen_ref) and with the names an `@see`, `\see`,
`@sa` or `\sa` paragraph lists that have the shape of a symbol (`name`,
`name()`, `::name`, `Type::member`; family see). Code blocks, fences, code
spans, quoted labels and the word after \c, \p, \a, \b, \e, \em hold none; a
URL is the core's href family. A .h is parsed as C++, which does not change
the rule; other C++ and Objective-C files are not part of the leg.
C keeps no node for a function prototype, so the docs of a header's API
were never stored anywhere. The doc-link driver gets an optional loose_docs
hook, called after the definitions and the file's own doc: the C leg reads
every Doxygen block outside function bodies that no definition took, and
its references belong to the file (the File node is the source).
The scope ("c1") lists the definitions with nodes (function, variable,
type, enum constant, field, object-like and function-like macro) with their
qualified names, their linkage (static, or a type, constant or macro of a
.c file, is the file's own) and the parent of a constant or field, and the
pages, groups, anchors and sections the file's docs define.
doc_links_c.c resolves a name as Doxygen does, without imports:
- a page, group, anchor or section name is no code entity (graph_gap) and
never binds a symbol of the same name;
- then the documented file's own definitions (static ones too), its fields,
then what every file sees (external linkage, a header's types, constants
and macros); a static definition of another file is not visible;
- candidates count by qualified name (a prototype and its definition, #if
variants: one node); with `()` written a function or function-like macro
wins, then a definition wins over a macro;
- `Type::member`: a field of the struct or union Type, or a constant of the
enum Type;
- one candidate binds, several are ambiguous; a word of an @see paragraph
that names nothing and is a plain lowercase word is prose (Doxygen prints
it as text): no row; test code binds only from test code.
There is no scope_delta hook yet: a declaration or linkage change is a full
run.
The families are held (ships=false) until their held-out audit passes; the
suite ships them through the test seam.
Proof (ship build, Apple M5 Pro / 48 GB, base
|
||
|
Martin Vogel
|
7a321cac16 |
doc-links(php): PHPDoc @see references become MENTIONS edges, held until their audit
A PHP doc block names code with `@see Ref` at the start of a tag line (a
description may follow) and with inline `{@see Ref}`, both family `see`.
doclink_php.c harvests them (code spans and fenced blocks hold none; @link,
@uses and the other tags are not harvested; a URL is the core's href family)
and writes each file's scope: namespace blocks, use imports, types with
their supertypes already qualified in their own file, members, functions.
The persisted scope drops lines and imports: other files read names only.
Declarations inside a top-level conditional count as top level
(`if (!function_exists('f')) { function f() {} }`), never a function body.
doc_links_php.c resolves a reference as a FQSEN with PHP's own name rules:
- a leading backslash is taken as written; otherwise the first segment is
replaced when it is the alias of a class `use` of the namespace block the
reference is written in, else the block's namespace is put in front
(phpDocumentor's resolver);
- beyond that tool, as PHP itself has them: an unqualified function name is
taken from a `use function` import, else from the current namespace, else
from the global namespace; and self/static/parent name the documented
type and its superclass;
- a bare `name()` that would bind a function but is written in a type that
declares or inherits a method of that name is ambiguous (the method or the
function; the reference does not say which);
- members are looked up on the type, its traits, its superclass and its
interfaces (depth first); type, function and method names compare without
case, properties and constants with it;
- properties, class constants and enum cases have no graph node: graph_gap;
so has a type or function the graph folds into another one of the same
name in its file (two namespace blocks), which the scope flags;
- a name the repository does not declare is external outside the
repository's namespace roots (built-ins, Composer dependencies), else
missing. Residual: a function a dependency declares in the current
namespace is invisible, so the fallback would bind a global function of
that name.
There is no scope_delta hook yet: a declaration change is a full run.
The family is held (ships=false) until its held-out audit passes; the suite
ships it through the test seam.
Proof (ship build, Apple M5 Pro / 48 GB, base
|
||
|
Martin Vogel
|
196ca71cda |
doc-links(ts): JSDoc and TSDoc links become MENTIONS edges, held until their audit
A TypeScript, TSX or JavaScript doc comment (a block comment opening with a
slash and two stars) names code with {@link Ref}, {@linkcode Ref} (each with
an optional `|` label) and `@see Ref` at the start of a tag line that holds
no inline tag. doclink_ts.c harvests them (code spans and fenced blocks hold
none, {@linkplain} is not harvested, a hyphenated word after @see is prose, a
reference naming the declaration's own parameter is dropped, a URL is the
core's href family) and writes each file's scope: imports, exports, export *
barrels, top-level declarations, namespace bodies and functions or classes
declared inside a function body.
doc_links_ts.c resolves a reference as the TypeScript checker sees it from
the documented declaration, in one index over TS, TSX and JS files:
1. the namespaces around the reference, innermost first;
2. a function or class declared in the enclosing function body (the graph
gives it a file-level name: two of one name in one file share one node,
a graph gap);
3. the file's own top-level declarations;
4. its imports, followed through export { a as b }, export ... from and
export * barrels (cycle guard, depth bound; two barrels that answer
differently are ambiguous); a relative specifier resolves with the usual
extension and index rules, a bare one is external;
5. the standard library's global names: external;
6. the top-level declarations of script files (no import, no export): one
binds, several are ambiguous;
7. the members of the documented class or interface, `this.` included.
The other segments walk members or a module's exports. A member declared
without a node of its own is a graph gap; test code binds only from test
code. There is no scope_delta hook yet: an export change is a full run.
The families are held (ships=false) until their held-out audit passes; the
suite ships them through the test seam.
Proof: the TypeScript compiler (perf-bench/typescript b465fdbfe175), ship
build, against the field test's prototype records (cascade results left
out): 69 edges; 57 of the prototype's 61 pairs bound alike. The 4 others are
doc blocks without a definition of their own (on an `if` statement, a local
`var`, an enum member, a free-standing @typedef block), which the core does
not harvest. Of the 12 edges the prototype has no pair for, 4 are the same
targets the prototype resolved without recording a source node, and 8 bind
PropertyDescriptor to the repository's own src/lib/es5.d.ts, which the
prototype called an external library global (kept: the repository declares
it). Unresolved rows agree with the prototype's reasons apart from the
documented reason split (not_in_scope -> missing/ambiguous).
Known limit: package.json and tsconfig.json are not indexed (discovery
ignores them), so workspace packages and tsconfig `paths` stay external.
Tests (doc_mentions_ts): tokens with their lines; the scope blob and its
persisted form; every resolution rule, including a function-body
declaration that binds and two of one name that do not; incremental equals
full (a doc edit and a body edit repair one file, an export change is a full
run); 60 modules give the same edges and rows with one worker and several.
Each rule red with it reverted (the body declaration's visibility and its
line check, barrel ambiguity, the parameter rule, code spans, the library
globals).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
f4277ae783 |
doc-links(perl): POD L<> codes become MENTIONS edges, held until their audit
A sub's POD section and the module's own POD (the File node's doc) name code with L<> codes: L<Module::Name>, L<Module::Name/section>, L</section>, with a `text|` prefix or not. doclink_perl.c reads them the way perlpodspec and Pod::Simple do (verbatim paragraphs and data regions hold none, an L inside an L is no link, a URL is the core's href family) and writes each file's scope: its packages, its subs and its headings. Lines are exact: a sub's doc is matched against the file's sections, and the module doc's runs (extract_defs.c) are found again in the source in order; a module doc is POD from its start, so a run that begins with text is read too. doc_links_perl.c resolves them: - a module is the file that declares its package; of several, the one the library layout names (.../Name/Space.pm), else ambiguous. The target is the file's File node, as the field test's audited prototype has it (a Module node is missing where a same-named Folder holds its qualified name, lib/Mojo.pm next to lib/Mojo/). - L<Module/section>: the sub its heading names when the module's file defines that sub; any other section is a place in the module's documentation, the module. - L</section>: a sub the heading names; a heading that names no sub is the page itself (local, neither edge nor row); a section the page has no heading for is missing. - a module the repository does not declare (CPAN, the core, a .pod page: discovery keeps no .pod file) and a man page are external; test code (t/, xt/, *.t, and cbm_is_test_file, which has no Perl rule) binds only from test code. There is no scope_delta hook yet: a change to a file's packages, subs or headings re-resolves every Perl reference. The family is held (ships=false) until its held-out audit passes; the suite ships it through the test seam. Tests (doc_mentions_perl): tokens with their lines, file-level and sub-level, each read once; every resolution rule; incremental equals full (a module doc edit and an added module-level reference repair the file, a heading change is a full run); 60 modules give the same edges and rows with one worker and several. Each rule red with it reverted (the module doc's line mapping, a module run read from its start, the missing section, the library layout). Mojolicious at 93f591742a7d (the family shipped, a measurement build) against the field test's prototype records of its .pm/.pl files: 950 (source, target) pairs both have; the 197 only the prototype has are all references within one file (570), which the core makes no edge of; 8 only here come from the extensionless scripts the prototype did not read. Every prototype row has its row here: external 102, missing 1, and the 172 targets in .pod pages external by the ruling for .pod pages. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3af25b432f |
doc-links(rust): intra-doc links become MENTIONS edges, held until their audit
Rustdoc's intra-doc links ([`Name`], [path], [text](path), the kind
disambiguators struct@, fn@, macro@ ...) and links that take their destination
from a link reference definition, bound the way rustdoc resolves them: the
item's module scope, its use declarations, the crate root and the crates
Cargo.toml names. The leg (doclink_rs.c extracts the links and each file's
scope, a Cargo.toml its crate facts; doc_links_rs.c resolves them) is ported
onto the restacked core: family rows carry their edge type, resolve() takes
the per-file state (none here). Both families are held (ships=false) until
their held-out audit passes; the suite ships them through the test seam.
The graph moved under the leg: cfg-gated twins are one definition with its
variants on one node (
|
||
|
Martin Vogel
|
a1190afbb0 |
doc-links(java, kotlin): Javadoc and KDoc references become MENTIONS edges, held until their audit
Java's {@link}, {@linkplain}, @see, @throws and {@value}, and Kotlin's [Name]
links, @see, @throws and @sample, bound the way javadoc and Dokka search for a
member (one resolver for both languages, so a Kotlin doc can name a Java type
and back). The leg (doclink_jvm.c extracts the references and each file's
scope; doc_links_jvm.c resolves them) is ported onto the restacked core:
family rows carry their edge type, resolve() takes the per-file state (none
here). All nine families are held (ships=false) until their held-out audit
passes; the suite ships them through the test seam.
The graph moved under the leg since it was written: a Java field's node is
named by the field alone (no longer by its whole declarator, `MAX = 10`), and
a field named like a method of its class has its own node,
<Owner>.<name>#field, where the method used to take the name. The tests
follow: the expected field names lose the initializer, a changed
initializer is a local change repaired file by file (it renamed the node
before, which forced a full run), and a field that shares its accessor's name
is now an edge target and a source (doc_mentions_jvm_java_families_and_reasons,
doc_mentions_jvm_node_ownership), where both were graph_gap rows.
Measured with the leg's binary (1078036f) and this one on the same input:
- elasticsearch subset (libs/ + server/src/main at f3dc4b0bd3a0, 5,956 files):
no edge lost; 66 new edges and 7 edges moved onto a field's own #field node;
81 graph_gap rows gone, no new row, no reason changed. A seeded sample of 8
of the gone rows read against the source: every new edge is right (a field
named like its accessor, enum constants, a field's own doc as the source).
- Exposed (Kotlin, c419caaab61f): identical, 896 edges and 371 rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
0c7ca34419 |
doc-links(go): Go doc links become MENTIONS edges, held until their audit
Go doc comments name code as doc links ([Name], [Recv.Name], [pkg.Name], [pkg.Recv.Name], an import path for pkg), the way go/doc/comment reads them. The Go leg (doclink_go.c extracts them and a go.mod's module path as the scope; doc_links_go.c resolves them against the package's declarations and its imports) is ported onto the restacked core: the family row carries its edge type, and resolve() takes the per-file state the core now passes (Go keeps none). The family is held (ships=false) until its held-out audit passes: a resolved doc link is a below_bar_tier row until then. The suite ships it through the test seam, like the other held families; doc_mentions_go_tokens asserts the production default. Kubernetes at 8c078bc32e90 with the family shipped (a measurement build): 1,928 edges (1,115 exact, 813 unique) and 101 rows (external 61, graph_gap 22, missing 18), the same numbers the leg measured on the old core. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
92f5edfe32 |
doc-links(asciidoc): AsciiDoc sections linked to the code they include and name
AsciiDoc (.adoc, .asciidoc) is now indexed, without a grammar like PDF: a
Section per heading (`=` .. `======`, or `#`), the text up to the next
heading as its docstring, delimited blocks respected (listing, literal,
comment, passthrough, example, sidebar, quote, table, fences). Its
references become MENTIONS edges, via "asciidoc", from the section they are
written in:
include include::target[...] -- Asciidoctor's preprocessor directive,
read in every block but a comment block; the page's own
attributes, then the Antora component's and the playbook's,
substituted
attribute an {attribute} whose value is an Antora javadoc link
({javadoc-root}/<module>/<package path>/<Type>.html[#member])
binds the Java or Kotlin type, or its one member, that the
repository's main sources declare under that path
code_path, monospace spans naming a path or a qualified name (the
code_name Markdown code-span rules)
The rules are the field test's (H8 Asciidoctor/Antora harvester, audited
~100 % precise): an Antora resource ID ([component:][module:]family$path)
names <component>/modules/<module>/<family dir>/<path>, or the file a
collector scan contributes there (a scan of build output is generated: no
row); any other target is relative to the including file. `lines=` and
`tag=`/`tags=` regions bind by H8's rule: edge blank and comment lines
trimmed, the innermost enclosing definition, unless that is a type (or there
is none) and the definitions inside cover every other line (an annotation
right above one counts) -- then that definition; tag regions come from the
included file's own `tag::x[]` / `end::x[]` comments, read from that indexed
file; a tag the file lacks binds the file. One reference makes one edge: a
region holding several top-level definitions binds the file.
Antora settings travel as YAML scope blobs (antora.yml: component name,
asciidoc attributes, collector scans; antora-playbook*.yml: attributes),
read as text, so an incremental run resolves as a full one; a change to them
is GLOBAL. AsciiDoc joins the documents bound by lines: a body edit of an
included file re-resolves its pages.
Also: the Markdown resolver's module index no longer takes AsciiDoc or PDF
files for code modules.
Proof on junit-team/junit-framework at 6c8ca116 (H8's corpus and commit),
C against H8's recorded links: javadoc attributes 362/362 identical;
includes 292 identical, 60 bind the type enclosing the several definitions
H8 names, 7 regions with several top-level definitions bind the file. 317
sections, 668 edges, no include rows (9 generated partials, 1 file type the
indexer does not read); monospace spans 30/30 correct by hand. 8.3 s for the
whole index.
Tests: suite doc_links_adoc (structure and tokens, the Antora blob, the
pipeline: resource IDs through a collector scan, tag regions, a region with
its imports binding its class, lines=, javadoc attributes, spans; incremental
== full: a moved tag region, a changed component attribute). Revert check:
without H8's region rule and without AsciiDoc among the line-bound documents
exactly their assertions fail (observed).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
1b444c6b61 |
doc-links(rst): reStructuredText sections linked to the code Sphinx markup names
A reST document now gets a Section per title (the text up to the next title
as its docstring, its lines from the title to the next one) and MENTIONS
edges, via "rst", from each section to the code it names:
role :class:`pkg.Model`, :meth:`~x.Y.z`, :func:`.f`, :c:func:`f`,
and extlink roles whose URL is a repository blob path
(:source:`django/db/models/base.py`)
object .. py:class:: / .. method:: / .. function:: ... and their
C-domain twins
autodoc .. automodule:: / .. autoclass:: ...
include, .. include:: / .. kernel-include:: of a code file,
literalinclude .. literalinclude:: (its :lines: bind the innermost
definition holding them, kept as target_lines; its
:pyobject: the named definition)
kernel_doc .. kernel-doc:: path and its :identifiers: / :functions:
code_path, inline literals naming a path or a qualified name (the
code_name Markdown code-span rules)
Structure is the field test's line model (H3): titles by adornment style,
literal blocks after `::` and in code directives, comments, directive
arguments and options, Sphinx's module / currentmodule / class contexts,
CPython's str whitespace and \w. Text before the first title belongs to the
file. A Sphinx documentation set's conf.py is read as text, never run: its
source_suffix makes discovery index e.g. Django's `.txt` documents as reST in
that set only, its primary_domain decides undomained roles, its extlinks and
intersphinx names are known to the resolver.
Python names resolve the way Sphinx imports them: the longest prefix that is
a module's import name (named from its topmost package directory), the rest
by qualified name, a package's re-exports (`from .base import Model` in its
__init__.py) followed. The re-exports and the conf.py settings travel as
Python scope blobs (doclink_py.c), so an incremental run resolves as a full
one; one resolver serves reST and Python. Forms in Sphinx's order (T, C.T,
M.T, M.C.T; a leading `.` reverses it). EXACT when a form names a node,
otherwise UNIQUE by name with the field test's fixes: a bare member
(:meth:`save`, :attr:`x`) is never searched (37 of 40 such results were
wrong), an object directive's target lies in its declared module, a module
never answers a member's role (`:meth:`dispatch`` is not tests/dispatch),
test code is no target of a search but binds when the document writes its
name in full (`django.test.TestCase`). C identifiers are whole names: the one
C/C++ definition of that name and kind. Unresolved: missing when the name or
path is the repository's, ambiguous, test_only_target; another project's
name (stdlib, intersphinx) is no row.
reST architecture decision records: the ADR detector reads reST records
like Markdown ones (titles underlined with = or -): course-discovery's 34
decision records become ADR nodes.
Proof on Django's documentation (the field test's corpus and commit), C
against the frozen H3 harvester over the same graph, per (document,
section, target): 6,695 of H3's 7,238 links identical; H3-only 543 = links
C attributes to the document because a section title repeats in it (the
Markdown rule; the same document -> code link exists), deliberate refusals
(bare members, directive searches without a module) and H3's own audited
error class (router protocol methods -> ConnectionRouter, forms Field ->
model Field); C-only = the same attribution mirrored, plus 1,053 inline
literal links (sample of 30: 30 correct). 6,574 reST sections; 12 s for the
whole index.
Tests: suite doc_links_rst (structure and tokens, the scope blobs, the
pipeline's rules, source_suffix in discovery, reST ADRs, incremental ==
full). Revert check: without the module-kind guard and without the
source_suffix mapping exactly their assertions fail (observed). Two
expectations of the old behaviour now name the new one: the reST grammar
golden and probe have the document's Sections (they recorded "reST titles
are no Sections", the gap this closes).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
6b0761160d |
doc-links(adr): architecture decision records as ADR nodes with their facts and SUPERSEDES
A Markdown file that is an architecture decision record now also defines an
ADR node (File DEFINES ADR): its name is the record's canonical id ("ADR-12";
another prefix as written, "DEC-5"; a dated record keeps its file stem), its
qualified name "<module>.__adr__", and its properties carry the record's
facts: adr_id, title, status (normalized: proposed, accepted, implemented,
superseded, deprecated, rejected, abandoned, draft, unknown), status_text,
date, deciders, superseded_by, and the decision itself as the docstring, so
search finds a record by what it decided. Its sections link to code like any
document's (MENTIONS, previous commit).
Detection is per file, with the field tests' rules: a file in a conventional
ADR directory (adr, adrs, decisions, decision-records, ...) whose name carries
an id or whose text has ADR structure; any adr-NNN file; a numbered or dated
file with ADR structure. Never README, index or template files without an id,
never test fixtures. Facts come from front matter, then header fields
(bullets, bold, tables), then the Status section; the date also from a
"Last updated" field, the change log's first date and the file name.
An ADR's own statement that it supersedes another ("Supersedes [ADR-3](...)",
"Supersedes ADR-3"; "Replaces ADR-3" where it opens a line, never "replaces"
as a verb of prose) becomes a SUPERSEDES edge between the two ADR nodes, by
link or by id. "Superseded by" is written in the old record about another
file; an edge from the other node would be owned by a file whose text does
not hold it, so it stays a fact of the old record (status, superseded_by).
The link layer gains a per-family edge type (doclink.h: the family list's
"edge" column), so a family can produce SUPERSEDES instead of MENTIONS; edges
are grouped by (type, source, target). Definitions gain a tail field for a
document's facts (extra_props: key, value pairs), carried through result
compaction and written by both property builders.
Measured against the field-test harvester on five repositories: detection
182 of 182 records, the same files; title 182/182; status 179/182 (cosmos
ADR-050 and its annexes say "Status: ARCHIVED", kept as status_text with
status unknown); date 174/182 (semantic-kernel's "date: {2024-05-02}" is read
as a date where the harvester dropped it). SUPERSEDES: semantic-kernel
0038 -> 0015, log4brains 20201016 -> 20200926, each checked against its
sentence. Markdown MENTIONS unchanged on fastapi, curl, dapr, cosmos-sdk.
Not covered: ADRs in reStructuredText, and ADR-log configuration files
(.adr-dir, log4brains) as a detection source.
Tests: doc_links_md_adr (detection and exclusions, facts from front matter
and from a Status section, SUPERSEDES by link and by id, no edge from prose,
the record's MENTIONS, incremental == full for a status edit and for an
added record named by id).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
afedff5176 |
doc-links(md): link Markdown sections to the code files and definitions they name
A Markdown document is documentation in itself: its sections now link to the code they reference explicitly, as MENTIONS edges (via "markdown") from the Section node (the File node for text before the first heading, and for a heading whose title repeats in the file, whose Section node it does not own). An agent can go from a document to the code and back through the graph: trace_path inbound with edge_types ["MENTIONS"] lists the sections that name a definition. Families (the "syntax" property; each with its ship gate): - link: the destination of an inline link or a link reference definition (no URL, no in-page anchor, no image); - path: a bare path in prose (a slash and a file extension); - code_path: a code span or HTML <code> span holding a path, a file name or a directory, with an optional line range (#L3-L9, :3-9) or member (path::Name); - code_name: a code span holding a qualified name (a.b, a::b, A#b, A\B, mod:attr). Resolution is guess-free, with the rules the documentation field tests measured: - a path is read relative to the document and, when it has a directory part, to the repository root; a file name alone only next to the document; - a line range binds the innermost code definition holding the whole range, else the file with "target_lines" on the edge; path::member the one definition of that name in the file; - a qualified name binds the one code definition whose qualifier chain ends with the written qualifier, else the one module or package so named; a match as written outranks one that needs case folding; a field binds only through its type, never through an instance; module:attr only in Python; product documentation never binds test code; - a bare name is never an edge. Unresolved references are rows in doc_link_unresolved: missing (the directory or the repository's own root is there, the name is not), ambiguous, test_only_target. A URL, an image, a URL route (/docs), a file of a type the indexer does not read, and code of other projects are no references to the repository and leave no row. Extraction scans the document's bytes once (doclink_md.c); every scan of a line is linear, or n log n where it sorts, using position lists that grow with the bytes they track, not with the line. Front matter, fenced code, HTML comments and MDX import/export lines are not scanned. The incremental route's node proxies now carry their line numbers (pipeline_delta.c), so a re-resolved document binds a line range exactly as a full build does; the Markdown resolver reads definitions from its own by-file list of the run's nodes, never from DEFINES edges, which an incremental run holds for re-extracted files only. Measured on four repositories against the main branch (which links file to file only, REFERENCES_FILE): fastapi 2,899 MENTIONS edges (2,225 REFERENCES_FILE before), cosmos-sdk 1,286 with 205 distinct definitions reached and 188 edges from 41 ADRs into code, dapr 184, curl 52. Link pairs agree with the field-test harvester (cosmos-sdk 408 of 409; the difference is a link of a document to itself, skipped on purpose). A hand check of 45 sampled qualified-name links found one wrong, fixed by the exact-case rule above. Tests: doc_links_md (extraction, repeated headings, the span classifier table, resolution with every reason, back navigation, incremental == full for a code edit, a document edit and an added name). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b3721128c1 |
doc-links(cs): a doc-line map that cannot grow fails the layer (R5)
cbm_doclink_note_doc_line returned silently when the map (or its first block) could not be allocated. The doc it was noting then keeps its definition's line, and a doc shared by the declarators of one field declaration is no longer marked as taken, so its references are parsed once per declarator -- while the layer reports ok. Both allocation failures now set the file's doc_links.failed, as every other lost piece of doc-link data does; the layer then says error. A test seam, CBM_DOCLINK_ALLOC_DOC_LINE, fails the map's growth. Test: alloc_failure_status gets the doc-line map as one more point (one error row). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
4b3f5a8bdf |
doc-links(cs): a shared doc comment counts once; lost doc-link data fails the layer (S4, S17)
S4: the declarators of one C# field declaration share one doc comment. Its references are taken once, from the first declarator (the Field definition that carries them, or the first variable where there is none); the other declarators take nothing from it, so tokens, resolutions and rows grow with the references, not with references x declarators. A shared doc that could not be collected whole is no doc of any declarator and is not looked up again per declarator. S17: memory that runs out while a file's doc-link data is extracted -- a doc span or doc text, a reference value, a token, the scope scan -- is recorded on the file's doc links (CBMDocLinkArray.failed), and the resolving half fails the layer for the run (status error). A MENTIONS edge that could not be stored fails the layer too and is not counted. Seams: the scope scan (CBM_DOCLINK_ALLOC_SCOPE) and the edge insert. Tests: cs_shared_doc_once, alloc_failure_status; the shared-doc tests of the checkpoint follow the once rule. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ea9fe36b9e |
doc-links: doc-comment references become MENTIONS edges, C# first
A reference written in a doc comment is extracted with the definition it
documents (internal/cbm/doclink.c; the C# scanner doclink_cs.c) and
resolved per file in the pipeline (src/pipeline/doc_links.c,
doc_links_cs.c, doc_links_msbuild.c). A reference that names a definition
of the repository with certainty becomes a MENTIONS edge from the
documented definition (the File node for a file-level comment) with the
properties via, syntax, tier, line and count. Any other reference becomes
a row of the new doc_link_unresolved table with its reason: missing,
ambiguous, external, test_only_target, graph_gap, unparseable or
below_bar_tier. Nothing is guessed. index_status reports the layer in a
doc_links block.
C# binds a cref by the compiler's lookup rule: the first scope with the
name decides, a cref never binds an inherited member, System is outside
the repository. Assemblies come from project files, and MSBuild global
usings (<Using>, ImplicitUsings, Directory.Build.props/.targets, imports,
conditions) are evaluated from the project files' content as data the
indexer has already read; the layer opens no file. Two declarations of a
name in different assemblies make the reference ambiguous.
Each file's scope is stored line-free with its LSP surface ("dl"), so an
incremental run re-resolves the files a change can reach and falls back
to a full run for a change that can re-route references anywhere (a
project file, a global using, a deleted file).
The scanner and the resolver treat the repository as untrusted input:
bounded nesting, no input-driven recursion, work that grows linearly
(held by test seams, never by a clock), no silent limits, and a damaged
or foreign stored scope fails closed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|