The Linux arm64 production build (gcc -O2 -Werror) stopped on
maybe-uninitialized in the TrueType cmap readers and the cross-reference
stream rows: ttf_fmt4 and ttf_fmt12 ignored the results of rd16/rd32 after a
bounds check that makes every read succeed, and xref_rows read pairs[] that
only the default /Index branch fills. The ASan test build did not warn.
The reads now fail the map on a failed read (the prototype's short-slice
rule; never taken, the bounds check stands before them), and pairs[] starts
at zero where it is declared. Text output is unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Form XObject that drew the next one many times, level under level,
was read (draws ^ depth) times: a few kilobytes asked for an unbounded
reading, and the page text grew with every leaf (D2 of the document
layer security review). The forms' text cannot be read faithfully; it is
exponential in the file.
The first reading of each form in a document stays whole. Reading one
again (the same content drawn elsewhere: a logo, a header on every page)
is counted in decoded bytes; beyond 256 MiB of such re-readings the
document reads no form again, and each skip is counted
(form_rereads_cut), as a decoded stream above 512 MiB already is. Real
documents re-read small forms: the 749-PDF dev corpus extracts
byte-identically as before (720 of 720 documents, 1,492 of 1,492 pages;
flattened copies too). Reading each form only once per page was tried
first and changed one real page, so it was not taken.
Test: pdf_extract_form_ladder_bounded: seven levels of forms drawing the
next four times each, 64 KiB per form: re-readings are cut and the page
keeps its glyphs bounded (with the bound reverted nothing is cut).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Objects whose value began a token that never ended (a string without its
closing parenthesis) were each read to the end of the file, and the
cross-reference rebuild reads every object: n such objects cost n times
the file in time and in arena memory (D4 of the document layer security
review). The same held for scanned trailers and for the members of an
object stream.
The objects of a file do not overlap, so an indirect object's value is
now read up to the next object header (the header scan, built once per
document), a scanned trailer up to the next "trailer", and an object
stream member up to the next member. A stream's bytes are still placed by
its /Length. The total work is linear in the file; nothing is cut that a
well-formed file holds.
Test: pdf_extract_open_strings_linear: a thousand objects with open
strings and no cross-reference grow the arena and extraction classes by
at most 16 x the file + 1 MiB (67 MB for a 112 KB file with the window
reverted). The 749-PDF dev corpus extracts byte-identically as before
(720 of 720 documents, 1,492 of 1,492 pages), and its 720 flattened
copies give the same output before and after.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every classic section that named an /XRefStm parsed and decoded that
stream afresh, and each decoded copy was kept until the document closed:
thousands of tiny sections naming one highly compressible stream asked
for thousands of copies (D3 of the document layer security review).
A target already merged is skipped: a later (older) section naming it
again adds nothing, since every number it holds is in the table already.
A cross-reference stream's decoded bytes are freed once its rows are
read (a stream parsed there is no cached object), so distinct streams
cost one at a time. The filter pipeline of pdf_stream_decode is split
out for that (decode_filters); cached streams decode as before.
Test: pdf_extract_xrefstm_once: a thousand sections naming one stream
decode it once (xref_streams == 1; 1,000 with the skip reverted). The
749-PDF dev corpus extracts byte-identically as before (720 of 720
documents, 1,492 of 1,492 pages).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Loading a stream whose /Length referred to another stream loaded that
stream first, and its /Length the next one: one nested call per object.
Loading an object of an object stream whose /Length (or filter) lay in
another object stream opened that one inside the first, again one call
per stream. A crafted chain overflowed the stack (D1 of the document
layer security review).
A /Length is now read as a value only: an object that is a stream there
gives no length (the stream is then read up to its endstream, as for any
wrong length) and is marked, so a second reference costs nothing. The
objects that open an object stream are no objects of an object stream
(ISO 32000-1, 7.5.7); one that is, is refused and counted
(objstm_nested). Both rules leave the nesting constant; nothing is cut
by a limit.
Test: pdf_extract_object_chains extracts a 20,000-deep /Length chain and
a 3,000-deep object-stream chain on a 256 KiB thread stack (stack
overflow before; each half fails alone when its rule is reverted). The
749-PDF dev corpus extracts byte-identically to the prototype as before
(720 of 720 documents, 1,492 of 1,492 pages).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The PDF text layer carries data tables (D3). Their sources and notices are
now in THIRD_PARTY.md (shipped in every archive through
THIRD_PARTY_NOTICES.md) and in the SBOM:
- the Adobe Glyph List subset (BSD-3-Clause, Copyright 2002-2019 Adobe),
- the printable-ASCII advance widths of Helvetica and Times-Roman from
Adobe's core-14 font metrics (the AFM files are not distributed),
- the Unicode Character Database data of pdf_unicode.c (Unicode
License v3, full text included).
scripts/gen-pdf-unicode.py (stdlib CPython only) regenerates
pdf_unicode.c byte for byte with CPython 3.14.6 (Unicode 16.0.0);
pdf_tables.c is maintained data from here on. License texts were taken
from their sources (adobe-type-tools/agl-aglfn, unicode.org).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A PDF is now indexed: its text layer is read by a dependency-free extractor
(internal/cbm/pdf, zlib only) and every page becomes a Section node ("page N",
the page's text as its docstring, property page), so search finds what a page
says and get_code_snippet shows a page's text, never the file's bytes. The code
a page names structurally becomes MENTIONS edges from that page, via "pdf":
pdf_path a repository path ("pkg/server/handler.go", "./x/y")
pdf_file a file name with a code extension ("models.py", "Pool.sol")
pdf_qn a qualified name ("app.models.User.save", "a::b", "x->y")
Bare identifiers never become edges (field test E6c: a type-target rule failed
its pre-registered bar), and only EXACT or UNIQUE resolutions link: the field
test's tiers (path-exact, component-aligned path-suffix, qn-exact, qn-suffix,
the latter also through a node's owner -- a Go method's QN has no receiver --
and with the same one-entity collapses). A mention split across a line break is
proposed as one reference and replaces its fragments only when it resolves.
Hygiene as measured: no links from PDFs under test or fixture directories (R1),
none to data files (R2), none for keyword arguments (R3); a test, mock or
fixture target is a test_only_target row (R4, with the test directories the
field test found missing). A path or name whose first component is not of this
repository is no row at all.
The extractor is a port of the field-tested prototype: xref tables and
streams, /Prev chains, repair near a wrong offset and reconstruction from the
object headers, object streams, Flate (with partial recovery), LZW, ASCII85,
ASCIIHex, RunLength and PNG/TIFF predictors, simple fonts (base encodings,
/Differences, built-in Type1 encodings, standard-14 widths, Type3), ToUnicode
and encoding CMaps, Type0/CID fonts with the embedded TrueType cmap read
backwards, Form XObjects, inline images skipped, word and line breaks from
glyph geometry, right-to-left runs reversed. Its geometry is bit-exact on every
platform (no FMA contraction; CPython's hypot). Untrusted input: every read is
bounds-checked, recursion is depth-limited, integers from the file saturate, a
decoded stream is cut at 512 MiB, and every lookup that could rescan the file
per object or per glyph is answered from an index built once. Not read:
encrypted documents (reported as such); scanned pages have no text layer.
The mention scanner is the field test's (idents.py) with CPython's semantics:
NFKC per code point plus canonical composition and the str predicates, from
generated Unicode 16.0 tables (pdf_unicode.c), and re's leftmost-first
matching.
Pipeline: CBM_LANG_PDF (no grammar; the registry ledger gets a DOCUMENT
capability, the call-node manifest exempts it like the other grammar-free
classification), extraction before any grammar lookup, a resolver hook
file_prepare (a file's state from all its references before any resolves).
Incremental: a proxy node carries parent_class where owner + name differs
from its QN (Go methods, class-body variables), so a re-resolved document binds
a Go method as a full build does.
Proof against the field-test prototype on its corpus (749 PDFs from 21
repositories, restored at the recorded commits, sha256-verified): text
byte-identical on all 1,492 pages of the 720 comparable PDFs (0.73 s vs 14.7 s);
mention tokens identical on all 960 page-text sets of E6, E6b and E6c (26,435
tokens); links identical to the frozen Python pipeline run over the same graph
on semantic-kernel 5/5, cosmos-sdk 30/30, Polly 11/11, mediastore3 3/3, dask
4/4. 44,070 seeded mutants (truncations, flips, nesting and keyword bombs, on
the originals and on uncompressed copies) under ASan and UBSan: 0 findings
after the first campaign's integer overflows were fixed as a class.
Tests: suite doc_links_pdf (layout, fonts and CMaps, object and xref streams,
xref repair and reconstruction, Form XObjects incl. a self-referencing one,
inline images, filters, hostile input, the scanner against the field test's
output, the pipeline's tiers, hygiene and rows, back navigation, the snippet,
incremental == full).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel