Two PDF doc-link tests named a Go method by its flat QN. With #1913 a
Go method's QN carries its receiver, so the lookups found no node.
The resolver is unchanged: for a Go method, rel(parent_class).name now
equals the QN, so the method binds through its QN, like a member of
any other language.
- pdf_links_pipeline: page 1 names `server.Handler.Serve`; the edge
target pkg.server.Serve becomes pkg.server.Handler.Serve. Same
assertions: pdf_qn syntax, unique tier.
- pdf_qn_method_through_owner, page 2: `time.Provider.Now` binds
pkg.time.Provider.Now (was pkg.time.Now), syntax pdf_qn.
- pdf_qn_method_through_owner, page 1: `time.Now` must not bind the
method. The target moves to pkg.time.Provider.Now too. Against the
old flat QN, no node exists any more, so the zero count would have
passed vacuously. The page 2 assertion proves that the node exists.
The ownerless-QN rule in doc_links_pdf.c stays: a templ method keeps
the flat package QN while it carries parent_class (extract_defs.c only
receiver-qualifies CBM_LANG_GO). Its comment no longer says that Go
methods have no receiver.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Supplementary held-out audits, each registered before its queries and run on
repositories chosen by an objective code-search walk, disjoint from every
earlier audit and development corpus (blind readers, a second reader on
every link not judged correct and on a seeded tenth of the rest; bar: Wilson
95 % lower bound >= 0.90 and point >= 0.95). A fix is judged on a NEW sample,
never on the one that found the error.
Ship now:
- Markdown code_path, refined to code spans with two or more path segments:
97 of 100 (lower bound 0.916). A file or one directory named alone in a
code span is its own family, bare_path, held: across the judged samples it
was 28 of 41 correct (`config.yaml` the reader's own file, `doc/` another
project's directory). A span without a word character (`../`, `./`) is no
path at all.
- reST literalinclude: 100 of 100; an open range (`:lines: 5-`, `-5`) now
reads the included file for its end and binds like a closed one.
- reST code_path: 99 of 100; its one error, a bare `../`, is fixed above.
- PDF qualified names: 97 of 99 after the owner rule: a definition whose
qualified name lacks its owner (a Go method under its package) matches
only owner-qualified text, so `time.Now` no longer binds a receiver's Now.
Fixed, still held:
- reST object directives: a function or class directive never binds a
Module, Folder or File (an example script's module took them); a new
sample is due.
- ADR supersedes: three censuses failed (37/38, 144/163, 82/87). Fixed from
them: a combined template label (`Supersedes / Depends on:`), a value of
"none" or "n/a" followed by records it extends, another record as the
subject (`ADR 0239 supersedes ...`, a table cell `0259 (supersedes ...)`),
a full stop after a number ending the target sentence while a
`[text](destination)` link stays one reference (adr-tools'
`Supersedes [3. Title](0003-...md)`). The family is split: a statement that
opens its line or field and names a record directly stays `supersedes`
(44 of 45 on the last census); the same words in running prose are
`supersedes_prose` (38 of 42), held. Both still held in this commit; a
fresh census of statements passed afterwards and is acted on next.
- An ADR log in a repository without code definitions lost every supersedes
link by file: the ADR table was filled after an early return. It is now
filled first.
Every fix has a test that fails with the fix reverted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The pre-registered held-out check of the PDF families: the C pipeline was
run on the eighteen repositories of the field test's two held-out PDF
audits (its links equal the frozen prototype pipeline's on the same
graph: 5,217 of 5,217, none only in one), and the audited prototype
links it makes take their labels (255 of 260 audited links). Bar per
family: Wilson 95 % lower bound >= 0.90 and point >= 0.95.
pdf_path 61 of 61 correct point 1.000 low 0.941 ships
pdf_file 110 of 114 point 0.965 low 0.913 ships
pdf_qn 77 of 80 point 0.963 low 0.896 does not ship
pdf_qn misses the bar: its references now become below_bar_tier rows
(the family's ships flag), as the pre-registration says, with no rule
tuned on the held-out data. Its three errors, all in one repository, are
names of an external API matched to a same-named local definition. The
pipeline test now
asserts the rows, and back navigation through the file a page names.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A PDF page's Section carried the whole page text as its docstring, which
is also what get_code_snippet returns for it: a crafted text layer
stored megabytes per node and handed them to an agent whole (D6 of the
document layer security review). Markdown, reStructuredText and AsciiDoc
sections keep a short docstring already.
A page's docstring is now its text up to 256 KiB, cut on a character
boundary and followed by a line that says where it was cut and how long
the text is. Pages hold a few kilobytes; the mentions are still read
from the whole text.
Test: pdf_page_doc_bounded: a 315 KB page keeps 256 KiB plus the note,
and a path written at its end is still a mention (the whole text is
stored with the bound reverted).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
For each line break that joins a name, the scanner listed the
page's candidates overlapping the joined span by reading every candidate of
the page: (joins x candidates), quadratic in a page's text, and the
geometry deciding the line breaks is the file's (D5 of the document
layer security review).
A join's span covers the end of one line and the start of the next, so
the emitted candidates are indexed by line once per page and a join
merges the two lines' lists: the same candidates in the same order, so
the same tokens.
Test: pdf_scan_join_linear: a page twice as long costs at most 2.5 x the
join work (seam counter; 717,600 vs 2,875,200 with the full scan
restored). The scanner's expected-token test is unchanged, and the PDF
links on three dev corpora (Polly, mediastore3, dask) equal the frozen
field-test pipeline's as before (0 links only in C, 0 only in Python).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Form XObject that drew the next one many times, level under level,
was read (draws ^ depth) times: a few kilobytes asked for an unbounded
reading, and the page text grew with every leaf (D2 of the document
layer security review). The forms' text cannot be read faithfully; it is
exponential in the file.
The first reading of each form in a document stays whole. Reading one
again (the same content drawn elsewhere: a logo, a header on every page)
is counted in decoded bytes; beyond 256 MiB of such re-readings the
document reads no form again, and each skip is counted
(form_rereads_cut), as a decoded stream above 512 MiB already is. Real
documents re-read small forms: the 749-PDF dev corpus extracts
byte-identically as before (720 of 720 documents, 1,492 of 1,492 pages;
flattened copies too). Reading each form only once per page was tried
first and changed one real page, so it was not taken.
Test: pdf_extract_form_ladder_bounded: seven levels of forms drawing the
next four times each, 64 KiB per form: re-readings are cut and the page
keeps its glyphs bounded (with the bound reverted nothing is cut).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Objects whose value began a token that never ended (a string without its
closing parenthesis) were each read to the end of the file, and the
cross-reference rebuild reads every object: n such objects cost n times
the file in time and in arena memory (D4 of the document layer security
review). The same held for scanned trailers and for the members of an
object stream.
The objects of a file do not overlap, so an indirect object's value is
now read up to the next object header (the header scan, built once per
document), a scanned trailer up to the next "trailer", and an object
stream member up to the next member. A stream's bytes are still placed by
its /Length. The total work is linear in the file; nothing is cut that a
well-formed file holds.
Test: pdf_extract_open_strings_linear: a thousand objects with open
strings and no cross-reference grow the arena and extraction classes by
at most 16 x the file + 1 MiB (67 MB for a 112 KB file with the window
reverted). The 749-PDF dev corpus extracts byte-identically as before
(720 of 720 documents, 1,492 of 1,492 pages), and its 720 flattened
copies give the same output before and after.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every classic section that named an /XRefStm parsed and decoded that
stream afresh, and each decoded copy was kept until the document closed:
thousands of tiny sections naming one highly compressible stream asked
for thousands of copies (D3 of the document layer security review).
A target already merged is skipped: a later (older) section naming it
again adds nothing, since every number it holds is in the table already.
A cross-reference stream's decoded bytes are freed once its rows are
read (a stream parsed there is no cached object), so distinct streams
cost one at a time. The filter pipeline of pdf_stream_decode is split
out for that (decode_filters); cached streams decode as before.
Test: pdf_extract_xrefstm_once: a thousand sections naming one stream
decode it once (xref_streams == 1; 1,000 with the skip reverted). The
749-PDF dev corpus extracts byte-identically as before (720 of 720
documents, 1,492 of 1,492 pages).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Loading a stream whose /Length referred to another stream loaded that
stream first, and its /Length the next one: one nested call per object.
Loading an object of an object stream whose /Length (or filter) lay in
another object stream opened that one inside the first, again one call
per stream. A crafted chain overflowed the stack (D1 of the document
layer security review).
A /Length is now read as a value only: an object that is a stream there
gives no length (the stream is then read up to its endstream, as for any
wrong length) and is marked, so a second reference costs nothing. The
objects that open an object stream are no objects of an object stream
(ISO 32000-1, 7.5.7); one that is, is refused and counted
(objstm_nested). Both rules leave the nesting constant; nothing is cut
by a limit.
Test: pdf_extract_object_chains extracts a 20,000-deep /Length chain and
a 3,000-deep object-stream chain on a 256 KiB thread stack (stack
overflow before; each half fails alone when its rule is reverted). The
749-PDF dev corpus extracts byte-identically to the prototype as before
(720 of 720 documents, 1,492 of 1,492 pages).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A PDF is now indexed: its text layer is read by a dependency-free extractor
(internal/cbm/pdf, zlib only) and every page becomes a Section node ("page N",
the page's text as its docstring, property page), so search finds what a page
says and get_code_snippet shows a page's text, never the file's bytes. The code
a page names structurally becomes MENTIONS edges from that page, via "pdf":
pdf_path a repository path ("pkg/server/handler.go", "./x/y")
pdf_file a file name with a code extension ("models.py", "Pool.sol")
pdf_qn a qualified name ("app.models.User.save", "a::b", "x->y")
Bare identifiers never become edges (field test E6c: a type-target rule failed
its pre-registered bar), and only EXACT or UNIQUE resolutions link: the field
test's tiers (path-exact, component-aligned path-suffix, qn-exact, qn-suffix,
the latter also through a node's owner -- a Go method's QN has no receiver --
and with the same one-entity collapses). A mention split across a line break is
proposed as one reference and replaces its fragments only when it resolves.
Hygiene as measured: no links from PDFs under test or fixture directories (R1),
none to data files (R2), none for keyword arguments (R3); a test, mock or
fixture target is a test_only_target row (R4, with the test directories the
field test found missing). A path or name whose first component is not of this
repository is no row at all.
The extractor is a port of the field-tested prototype: xref tables and
streams, /Prev chains, repair near a wrong offset and reconstruction from the
object headers, object streams, Flate (with partial recovery), LZW, ASCII85,
ASCIIHex, RunLength and PNG/TIFF predictors, simple fonts (base encodings,
/Differences, built-in Type1 encodings, standard-14 widths, Type3), ToUnicode
and encoding CMaps, Type0/CID fonts with the embedded TrueType cmap read
backwards, Form XObjects, inline images skipped, word and line breaks from
glyph geometry, right-to-left runs reversed. Its geometry is bit-exact on every
platform (no FMA contraction; CPython's hypot). Untrusted input: every read is
bounds-checked, recursion is depth-limited, integers from the file saturate, a
decoded stream is cut at 512 MiB, and every lookup that could rescan the file
per object or per glyph is answered from an index built once. Not read:
encrypted documents (reported as such); scanned pages have no text layer.
The mention scanner is the field test's (idents.py) with CPython's semantics:
NFKC per code point plus canonical composition and the str predicates, from
generated Unicode 16.0 tables (pdf_unicode.c), and re's leftmost-first
matching.
Pipeline: CBM_LANG_PDF (no grammar; the registry ledger gets a DOCUMENT
capability, the call-node manifest exempts it like the other grammar-free
classification), extraction before any grammar lookup, a resolver hook
file_prepare (a file's state from all its references before any resolves).
Incremental: a proxy node carries parent_class where owner + name differs
from its QN (Go methods, class-body variables), so a re-resolved document binds
a Go method as a full build does.
Proof against the field-test prototype on its corpus (749 PDFs from 21
repositories, restored at the recorded commits, sha256-verified): text
byte-identical on all 1,492 pages of the 720 comparable PDFs (0.73 s vs 14.7 s);
mention tokens identical on all 960 page-text sets of E6, E6b and E6c (26,435
tokens); links identical to the frozen Python pipeline run over the same graph
on semantic-kernel 5/5, cosmos-sdk 30/30, Polly 11/11, mediastore3 3/3, dask
4/4. 44,070 seeded mutants (truncations, flips, nesting and keyword bombs, on
the originals and on uncompressed copies) under ASan and UBSan: 0 findings
after the first campaign's integer overflows were fixed as a class.
Tests: suite doc_links_pdf (layout, fonts and CMaps, object and xref streams,
xref repair and reconstruction, Form XObjects incl. a self-referencing one,
inline images, filters, hostile input, the scanner against the field test's
output, the pipeline's tiers, hygiene and rows, back navigation, the snippet,
incremental == full).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel