nprasad2077/codebase-memory-mcp
1
0 Fork 0
mirror of https://github.com/DeusData/codebase-memory-mcp.git synced 2026-11-04 10:48:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

doc-links(pdf): PDF pages as Sections, linked to the code their text names

Browse Source 
A PDF is now indexed: its text layer is read by a dependency-free extractor
(internal/cbm/pdf, zlib only) and every page becomes a Section node ("page N",
the page's text as its docstring, property page), so search finds what a page
says and get_code_snippet shows a page's text, never the file's bytes. The code
a page names structurally becomes MENTIONS edges from that page, via "pdf":

  pdf_path   a repository path ("pkg/server/handler.go", "./x/y")
  pdf_file   a file name with a code extension ("models.py", "Pool.sol")
  pdf_qn     a qualified name ("app.models.User.save", "a::b", "x->y")

Bare identifiers never become edges (field test E6c: a type-target rule failed
its pre-registered bar), and only EXACT or UNIQUE resolutions link: the field
test's tiers (path-exact, component-aligned path-suffix, qn-exact, qn-suffix,
the latter also through a node's owner -- a Go method's QN has no receiver --
and with the same one-entity collapses). A mention split across a line break is
proposed as one reference and replaces its fragments only when it resolves.
Hygiene as measured: no links from PDFs under test or fixture directories (R1),
none to data files (R2), none for keyword arguments (R3); a test, mock or
fixture target is a test_only_target row (R4, with the test directories the
field test found missing). A path or name whose first component is not of this
repository is no row at all.

The extractor is a port of the field-tested prototype: xref tables and
streams, /Prev chains, repair near a wrong offset and reconstruction from the
object headers, object streams, Flate (with partial recovery), LZW, ASCII85,
ASCIIHex, RunLength and PNG/TIFF predictors, simple fonts (base encodings,
/Differences, built-in Type1 encodings, standard-14 widths, Type3), ToUnicode
and encoding CMaps, Type0/CID fonts with the embedded TrueType cmap read
backwards, Form XObjects, inline images skipped, word and line breaks from
glyph geometry, right-to-left runs reversed. Its geometry is bit-exact on every
platform (no FMA contraction; CPython's hypot). Untrusted input: every read is
bounds-checked, recursion is depth-limited, integers from the file saturate, a
decoded stream is cut at 512 MiB, and every lookup that could rescan the file
per object or per glyph is answered from an index built once. Not read:
encrypted documents (reported as such); scanned pages have no text layer.

The mention scanner is the field test's (idents.py) with CPython's semantics:
NFKC per code point plus canonical composition and the str predicates, from
generated Unicode 16.0 tables (pdf_unicode.c), and re's leftmost-first
matching.

Pipeline: CBM_LANG_PDF (no grammar; the registry ledger gets a DOCUMENT
capability, the call-node manifest exempts it like the other grammar-free
classification), extraction before any grammar lookup, a resolver hook
file_prepare (a file's state from all its references before any resolves).
Incremental: a proxy node carries parent_class where owner + name differs
from its QN (Go methods, class-body variables), so a re-resolved document binds
a Go method as a full build does.

Proof against the field-test prototype on its corpus (749 PDFs from 21
repositories, restored at the recorded commits, sha256-verified): text
byte-identical on all 1,492 pages of the 720 comparable PDFs (0.73 s vs 14.7 s);
mention tokens identical on all 960 page-text sets of E6, E6b and E6c (26,435
tokens); links identical to the frozen Python pipeline run over the same graph
on semantic-kernel 5/5, cosmos-sdk 30/30, Polly 11/11, mediastore3 3/3, dask
4/4. 44,070 seeded mutants (truncations, flips, nesting and keyword bombs, on
the originals and on uncompressed copies) under ASan and UBSan: 0 findings
after the first campaign's integer overflows were fixed as a class.

Tests: suite doc_links_pdf (layout, fonts and CMaps, object and xref streams,
xref repair and reconstruction, Form XObjects incl. a self-referencing one,
inline images, filters, hostile input, the scanner against the field test's
output, the pipeline's tiers, hygiene and rows, back navigation, the snippet,
incremental == full).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
Martin Vogel committed 2026-10-09 02:55:23 +02:00
1 parent 6b0761160d
commit 033ca372a8
25 files changed
+13942 -18
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Makefile.cbm
+8 -1
View File
@@ -377,6 +377,12 @@ EXTRACTION_SRCS = \
$(CBM_DIR)/doclink_cs.c \
$(CBM_DIR)/doclink_md.c \
$(CBM_DIR)/doc_adr.c \
$(CBM_DIR)/doc_pdf.c \
$(CBM_DIR)/pdf/pdf_obj.c \
$(CBM_DIR)/pdf/pdf_font.c \
$(CBM_DIR)/pdf/pdf_text.c \
$(CBM_DIR)/pdf/pdf_tables.c \
$(CBM_DIR)/pdf/pdf_unicode.c \
$(CBM_DIR)/sql_values.c \
$(CBM_DIR)/helpers.c \
$(CBM_DIR)/cpp_branch_views.c \
@@ -500,6 +506,7 @@ PIPELINE_SRCS = \
src/pipeline/doc_links.c \
src/pipeline/doc_links_cs.c \
src/pipeline/doc_links_md.c \
src/pipeline/doc_links_pdf.c \
src/pipeline/doc_links_msbuild.c \
src/pipeline/pass_route_nodes.c \
src/pipeline/pass_enrichment.c \
@@ -762,7 +769,7 @@ TEST_DISCOVER_SRCS = \
TEST_GRAPH_BUFFER_SRCS = tests/test_graph_buffer.c
TEST_PIPELINE_SRCS = tests/test_registry.c tests/test_pipeline.c tests/test_pipeline_conventions.c tests/test_pipeline_frozen.c tests/test_importance.c tests/test_cross_repo.c tests/test_fqn.c tests/test_route_canon.c tests/test_path_alias.c tests/test_configlink.c tests/test_doclinks.c tests/test_doc_mentions.c tests/test_doc_links_md.c tests/test_infrascan.c tests/test_worker_pool.c tests/test_parallel.c tests/test_index_resilience.c tests/test_index_format.c tests/test_call_reference_contract.c tests/repro/repro_call_scope_usages.c tests/repro/repro_call_argument_usages.c tests/repro/repro_reference_precision.c tests/repro/repro_lexical_binding_precision.c tests/repro/repro_call_argument_matrix_a.c tests/repro/repro_call_argument_matrix_b.c tests/repro/repro_call_node_behaviors.c tests/repro/repro_language_registry.c tests/repro/repro_call_node_manifest.c tests/repro/repro_lsp_ordered_signatures.c tests/repro/repro_lsp_ordered_local.c tests/repro/repro_ts_overload_return_chains.c tests/repro/repro_harness_cleanup.c tests/repro/repro_runner_filter.c
TEST_PIPELINE_SRCS = tests/test_registry.c tests/test_pipeline.c tests/test_pipeline_conventions.c tests/test_pipeline_frozen.c tests/test_importance.c tests/test_cross_repo.c tests/test_fqn.c tests/test_route_canon.c tests/test_path_alias.c tests/test_configlink.c tests/test_doclinks.c tests/test_doc_mentions.c tests/test_doc_links_md.c tests/test_doc_links_pdf.c tests/test_infrascan.c tests/test_worker_pool.c tests/test_parallel.c tests/test_index_resilience.c tests/test_index_format.c tests/test_call_reference_contract.c tests/repro/repro_call_scope_usages.c tests/repro/repro_call_argument_usages.c tests/repro/repro_reference_precision.c tests/repro/repro_lexical_binding_precision.c tests/repro/repro_call_argument_matrix_a.c tests/repro/repro_call_argument_matrix_b.c tests/repro/repro_call_node_behaviors.c tests/repro/repro_language_registry.c tests/repro/repro_call_node_manifest.c tests/repro/repro_lsp_ordered_signatures.c tests/repro/repro_lsp_ordered_local.c tests/repro/repro_ts_overload_return_chains.c tests/repro/repro_harness_cleanup.c tests/repro/repro_runner_filter.c
TEST_WATCHER_SRCS = tests/test_watcher.c
internal/cbm/cbm.c
+20
View File
@@ -2802,6 +2802,26 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
cbm_test_fault_inject(rel_path);
#endif
/* A PDF has no grammar: its text layer is read directly, a Section per
* page plus the pages' code mentions (doc_pdf.c). */
if (language == CBM_LANG_PDF) {
result->module_qn = cbm_fqn_module_source_lang(a, project, rel_path, language);
CBMExtractCtx pdf_ctx = {
.arena = a,
.scratch = scratch,
.result = result,
.source = source,
.source_len = source_len,
.language = language,
.project = project,
.rel_path = rel_path,
.module_qn = result->module_qn,
};
cbm_pdf_extract_document(&pdf_ctx);
cbm_index_mark_done(rel_path);
return result;
}
// Get language spec
const CBMLangSpec *spec = cbm_lang_spec(language);
if (!spec) {
internal/cbm/cbm.h
+1
View File
@@ -209,6 +209,7 @@ typedef enum {
CBM_LANG_ARKTS, // ArkTS (HarmonyOS/OpenHarmony .ets — TypeScript superset + ArkUI)
CBM_LANG_PLSQL, // Oracle PL/SQL
CBM_LANG_CHIALISP, // Chialisp (.clsp/.clib/.clinc — Chia smart-coin s-expression language)
CBM_LANG_PDF, // PDF documents: the text layer, read without a grammar (doc_pdf.c)
CBM_LANG_COUNT
} CBMLanguage;
internal/cbm/doc_pdf.c
+1306
View File
File diff suppressed because it is too large. Load diff
internal/cbm/doclink.h
+21
View File
@@ -77,6 +77,12 @@
X(CBM_DOCLINK_MD_CODE_NAME, CBM_LANG_MARKDOWN, "code_name", false, true, "MENTIONS") \
/* an ADR's own "supersedes" / "replaces" statement (doc_adr.c): ADR -> ADR */ \
X(CBM_DOCLINK_MD_SUPERSEDES, CBM_LANG_MARKDOWN, "supersedes", false, true, "SUPERSEDES") \
/* pdf (doc_pdf.c): structural mentions in a page's text; pdf_name (bare */ \
/* names) only decides which line fragments a cross-line join replaces */ \
X(CBM_DOCLINK_PDF_PATH, CBM_LANG_PDF, "pdf_path", false, true, "MENTIONS") \
X(CBM_DOCLINK_PDF_FILE, CBM_LANG_PDF, "pdf_file", false, true, "MENTIONS") \
X(CBM_DOCLINK_PDF_QN, CBM_LANG_PDF, "pdf_qn", false, true, "MENTIONS") \
X(CBM_DOCLINK_PDF_NAME, CBM_LANG_PDF, "pdf_name", false, false, "MENTIONS") \
/* any language (CBM_LANG_COUNT): a URL is never an edge */ \
X(CBM_DOCLINK_HREF, CBM_LANG_COUNT, "href", true, false, "MENTIONS")
@@ -98,6 +104,10 @@ enum {
* file-level doc -- a parser never sets it. A scan_file hook sets it on a
* reference written before the document's first section. */
CBM_DOCLINK_FLAG_FILE = 1,
/* A PDF mention that runs across a line break, proposed as one reference.
* `raw` holds its forms (separated by 0x1F) and, after 0x1E, the indices
* of the line-local tokens it replaces when it resolves (doc_pdf.c). */
CBM_DOCLINK_FLAG_JOIN = 2,
};
/* One link family: a line of CBM_DOCLINK_FAMILY_LIST. */
@@ -295,4 +305,15 @@ void cbm_adr_extract(CBMExtractCtx *ctx);
/* The qualified-name tail of an ADR node: "<module QN>" + this. */
#define CBM_ADR_QN_NAME "__adr__"
/* PDF documents (doc_pdf.c): a Section per page (name "page N", the page's
* text as docstring, property page) and the pages' structural code mentions
* as doc-link tokens. Runs instead of a grammar for CBM_LANG_PDF. */
void cbm_pdf_extract_document(CBMExtractCtx *ctx);
#if defined(CBM_ENABLE_TEST_SEAMS) && CBM_ENABLE_TEST_SEAMS
/* Test seam: scan one page's text as cbm_pdf_extract_document does (tokens
* into ctx->result->doc_links), without a PDF around it. */
void cbm_pdf_test_scan_page(CBMExtractCtx *ctx, const char *text, size_t len, uint32_t page,
const char *page_qn);
#endif
#endif /* CBM_DOCLINK_H */
internal/cbm/lang_specs.c
+5
View File
@@ -2713,6 +2713,11 @@ static const CBMLangSpec lang_specs[CBM_LANG_COUNT] = {
empty_types, NULL, NULL, tree_sitter_objectscript_routine,
NULL},
// CBM_LANG_PDF — PDF documents. No grammar row: extraction reads the text
// layer itself (cbm_pdf_extract_document, doc_pdf.c) before any grammar is
// looked up, so this language never reaches cbm_lang_spec()/cbm_ts_language().
// Left as a zero spec.
// CBM_LANG_OBJECTSCRIPT_EXPORT — Studio Export XML. No grammar row: the
// pipeline transcodes Export XML to UDL (iris_export_xml.c) and re-extracts
// each class as CBM_LANG_OBJECTSCRIPT_UDL, so this language never reaches
internal/cbm/pdf/pdf.h
+60
View File
@@ -0,0 +1,60 @@
/*
* pdf.h — the text layer of a PDF, page by page.
*
* Reads what a PDF says in text (content-stream text operators decoded through
* the document's fonts) and nothing else: no rendering, no images, no OCR, no
* JavaScript, actions or URIs, nothing fetched, nothing executed. Input is
* untrusted: every read is bounds-checked, every recursion is depth-limited,
* every decoder is guarded against decompression bombs.
*
* Not read: encrypted documents (the result says so). Scanned pages have no
* text layer and come back empty.
*/
#ifndef CBM_PDF_H
#define CBM_PDF_H
#include "../arena.h"
#include <stdbool.h>
#include <stddef.h>
typedef enum {
CBM_PDF_OK = 0,
CBM_PDF_NOT_PDF, /* no "%PDF-" within the first 1024 bytes */
CBM_PDF_PARSE_FAILURE, /* no page could be found */
CBM_PDF_ENCRYPTED, /* an Encrypt dictionary: the text is not read */
CBM_PDF_NOMEM,
} cbm_pdf_status_t;
typedef struct {
const char *text; /* UTF-8, NUL-terminated, no NUL inside */
size_t len;
int glyphs; /* glyphs shown on the page */
int unmapped; /* glyphs without a Unicode value (U+FFFD in the text) */
} cbm_pdf_page_t;
/* How the document was read: counts of the repairs and limits that applied. */
typedef struct {
int xref_reconstructed; /* cross-reference rebuilt from a scan of the file */
int xref_offset_fixed; /* an xref section found near a wrong offset */
int stream_length_recovered; /* a stream's /Length was wrong */
int flate_recovered; /* a damaged Flate stream decoded in part */
int stream_guard_hits; /* a decoded stream cut at 512 MiB */
int filters_unsupported; /* a stream with a filter this reader does not decode */
int op_errors; /* content operators that could not be applied */
int page_errors; /* pages whose content stopped early */
} cbm_pdf_counts_t;
typedef struct {
cbm_pdf_status_t status;
cbm_pdf_page_t *pages;
int npages;
cbm_pdf_counts_t counts;
} cbm_pdf_result_t;
/* Read the text layer of data[0..len). Everything the result points to is
* allocated from `out` and lives as long as it. Returns false only when memory
* ran out (status CBM_PDF_NOMEM). */
bool cbm_pdf_extract(const unsigned char *data, size_t len, CBMArena *out, cbm_pdf_result_t *res);
#endif /* CBM_PDF_H */
internal/cbm/pdf/pdf_font.c
+1978
View File
File diff suppressed because it is too large. Load diff
internal/cbm/pdf/pdf_internal.h
+321
View File
@@ -0,0 +1,321 @@
/*
* pdf_internal.h — the PDF text-layer extractor's internals (pdf_*.c).
*
* Stages, each in its own file:
* pdf_obj.c lexer, values, the document (xref tables and streams, /Prev
* chains, reconstruction when offsets are broken), indirect
* objects, object streams, stream filters
* pdf_font.c glyph names, ToUnicode and encoding CMaps, simple fonts
* (base encodings, /Differences, built-in Type1 encodings,
* standard-14 widths, Type3), Type0/CID fonts, the TrueType
* cmap fallback
* pdf_text.c content streams (graphics and text state, text operators,
* Form XObjects, inline images skipped), layout (word and line
* breaks from glyph geometry), pages, the driver
* pdf_tables.c generated static tables
*
* The reader is a port of the field-tested prototype (field test E6): the
* same tokens, repairs, font rules and layout thresholds, so its text is the
* text that was measured. Where the prototype's cost could grow faster than
* the input (a search per object, a table walk per glyph), this port answers
* from an index built once, with the same result.
*/
#ifndef CBM_PDF_INTERNAL_H
#define CBM_PDF_INTERNAL_H
#include "pdf.h"
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
/* Layout geometry must come out bit for bit the same on every platform (and
* the same as the measured prototype): no fused multiply-add. GCC in ISO C
* mode never contracts; clang does unless told. */
#if defined(__clang__)
#pragma STDC FP_CONTRACT OFF
#endif
/* ── Tables (pdf_tables.c) ───────────────────────────────────────── */
typedef struct {
const char *name;
const char *utf8;
} pdf_glyph_t;
enum { PDF_STD_WIDTHS = 95 }; /* codes 0x20..0x7E */
extern const pdf_glyph_t PDF_GLYPHS[];
extern const int PDF_GLYPH_COUNT;
extern const char *const PDF_ENC_STANDARD[256];
extern const char *const PDF_ENC_WINANSI[256];
extern const char *const PDF_ENC_MACROMAN[256];
extern const char *const PDF_ENC_PDFDOC[256];
extern const char *const PDF_ENC_SYMBOL[256];
extern const unsigned short PDF_WIDTH_HELVETICA[PDF_STD_WIDTHS];
extern const unsigned short PDF_WIDTH_TIMES[PDF_STD_WIDTHS];
extern const char *const PDF_STD14[];
/* ── Limits: structure, never content ────────────────────────────── */
enum {
PDF_MAX_NESTING = 256, /* arrays/dicts inside one value */
PDF_MAX_RESOLVE = 32, /* reference chains */
PDF_MAX_FORM_DEPTH = 16, /* Form XObjects inside Form XObjects */
PDF_MAX_PAGE_DEPTH = 64, /* page-tree depth */
PDF_MAX_PREV = 4096, /* xref sections along /Prev */
PDF_HEADER_WINDOW = 1024, /* "%PDF-" must start within this many bytes */
};
/* A decoded stream above this size is cut and counted: a decompression bomb is
* no text layer, and real documents stay far below it. */
#define PDF_MAX_STREAM_OUT ((size_t)512 << 20)
/* ── Integers from the file: saturated, never undefined ──────────── */
static inline int64_t pdf_sat_add(int64_t a, int64_t b) {
int64_t r;
if (__builtin_add_overflow(a, b, &r)) {
return b > 0 ? INT64_MAX : INT64_MIN;
}
return r;
}
/* Python's int() of a float: truncated toward zero; saturated here, NaN 0. */
static inline int64_t pdf_d2i(double x) {
if (x != x) {
return 0;
}
if (x >= 9223372036854775807.0) {
return INT64_MAX;
}
if (x <= -9223372036854775808.0) {
return INT64_MIN;
}
return (int64_t)x;
}
/* ── Values ──────────────────────────────────────────────────────── */
typedef enum {
PV_NULL = 0,
PV_BOOL,
PV_NUM,
PV_NAME,
PV_STR,
PV_ARR,
PV_DICT,
PV_REF,
PV_STREAM,
PV_KW,
} pdf_kind_t;
typedef struct pdf_val pdf_val_t;
typedef struct pdf_stream pdf_stream_t;
typedef struct {
const unsigned char *key; /* a name's bytes, NUL-terminated */
size_t klen;
pdf_val_t *val;
} pdf_kv_t;
struct pdf_val {
pdf_kind_t kind;
bool b;
bool is_int;
double num;
int64_t inum;
const unsigned char *s; /* PV_NAME, PV_STR, PV_KW (NUL-terminated copies) */
size_t n;
pdf_val_t **items; /* PV_ARR */
int count;
pdf_kv_t *kv; /* PV_DICT: sorted by key, one entry per key (the last written) */
int nkv;
int64_t ref_num; /* PV_REF */
int64_t ref_gen;
pdf_stream_t *stream; /* PV_STREAM */
};
struct pdf_stream {
pdf_val_t *dict;
const unsigned char *raw;
size_t raw_len;
bool done;
const unsigned char *decoded; /* NULL: nothing decodable (image, unsupported) */
size_t dec_len;
};
/* ── Lexer ───────────────────────────────────────────────────────── */
typedef enum {
PT_EOF = 0,
PT_NUM,
PT_NAME,
PT_STR,
PT_DS, /* << */
PT_DE, /* >> */
PT_AS, /* [ */
PT_AE, /* ] */
PT_BS, /* { */
PT_BE, /* } */
PT_KW,
} pdf_tok_kind_t;
typedef struct {
pdf_tok_kind_t kind;
bool is_int;
double num;
int64_t inum;
const unsigned char *s; /* NAME, STR: arena copies; KW: points into the data */
size_t n;
} pdf_tok_t;
typedef struct pdf_doc pdf_doc_t;
/* The next token of data[*pos..len); *pos moves past it. */
void pdf_lex(pdf_doc_t *d, const unsigned char *data, size_t len, size_t *pos, pdf_tok_t *t);
/* The value that starts with token t. NULL: out of memory or nested too deep. */
pdf_val_t *pdf_parse_value(pdf_doc_t *d, const unsigned char *data, size_t len, size_t *pos,
const pdf_tok_t *t, int depth);
/* An array operand of a content stream: no references, an operator ends it. */
pdf_val_t *pdf_parse_content_array(pdf_doc_t *d, const unsigned char *data, size_t len, size_t *pos,
int depth);
bool pdf_kw_is(const pdf_tok_t *t, const char *kw);
/* ── Integer maps (open addressing, owned by the document) ───────── */
typedef struct {
uint64_t *keys;
int64_t *vals;
uint32_t cap; /* a power of two, or 0 */
uint32_t n;
} pdf_map_t;
/* -1 when absent. */
int64_t pdf_map_get(const pdf_map_t *m, uint64_t key);
bool pdf_map_put(pdf_map_t *m, uint64_t key, int64_t val);
void pdf_map_free(pdf_map_t *m);
/* ── Document ────────────────────────────────────────────────────── */
typedef struct pdf_objstm pdf_objstm_t;
typedef struct {
int64_t num;
uint8_t type; /* 0 free, 1 at an offset, 2 inside an object stream */
int64_t a; /* type 1: offset; type 2: the object stream's number */
int64_t b; /* type 1: generation; type 2: index in the stream */
pdf_val_t *obj;
bool cached;
bool loading;
pdf_objstm_t *stm; /* this object as a decoded object stream */
} pdf_xent_t;
typedef struct {
pdf_xent_t *v;
int n;
int cap;
pdf_map_t by_num; /* object number -> index in v */
} pdf_xref_t;
/* One "N G obj" header found by scanning the whole file. */
typedef struct {
int64_t num;
int64_t gen;
size_t off; /* where N starts */
uint8_t num_digits;
uint8_t gen_digits;
bool canonical; /* both written without leading zeros */
bool reg_after; /* "obj" runs on into a regular character */
} pdf_hdr_t;
typedef struct {
int64_t num;
int idx;
} pdf_numidx_t;
typedef struct pdf_font pdf_font_t;
struct pdf_doc {
const unsigned char *data;
size_t len;
size_t hdr; /* offset of "%PDF-" */
CBMArena *a; /* the document's values, fonts, CMaps */
CBMArena *cur; /* where the lexer and the value parser allocate (a, or a
* content stream's operand arena) */
bool peek; /* lex without copying names or strings */
bool nomem; /* an allocation failed: the result is CBM_PDF_NOMEM */
pdf_xref_t xref;
pdf_val_t *trailer; /* a dict */
bool reconstructed;
/* every "N G obj" in file order, built on first need */
pdf_hdr_t *hdrs;
int nhdrs;
bool hdrs_built;
pdf_numidx_t *hdr_sorted; /* (num, index) sorted, for the per-object fallback */
/* blocks from the memory core, freed with the document */
void **owned;
int nowned;
int owned_cap;
/* an arena for token values that end with one CMap's parse */
CBMArena scratch_arena;
bool scratch_arena_live;
/* a scratch buffer for string and stream assembly */
unsigned char *scratch;
size_t scratch_cap;
/* fonts by reference (num << 20 | gen) or by inline dict address */
pdf_map_t fonts;
pdf_font_t **font_list;
int nfonts;
int font_cap;
cbm_pdf_counts_t *counts;
};
pdf_val_t *pdf_get(pdf_doc_t *d, int64_t num);
pdf_val_t *pdf_resolve(pdf_doc_t *d, pdf_val_t *v);
/* dict[key] as written (a reference stays a reference); NULL when absent, null,
* or the value is no dict. */
pdf_val_t *pdf_dget_raw(const pdf_val_t *dict, const char *key);
/* Whether dict has key at all (a null value counts). */
bool pdf_dhas(const pdf_val_t *dict, const char *key);
/* dict[key] as written, a null value included; NULL only when absent. */
pdf_val_t *pdf_dfind(const pdf_val_t *dict, const char *key);
pdf_val_t *pdf_dfind_n(const pdf_val_t *dict, const unsigned char *key, size_t n);
/* dict[key] resolved; NULL when absent or null. */
pdf_val_t *pdf_dget(pdf_doc_t *d, const pdf_val_t *dict, const char *key);
bool pdf_is_name(const pdf_val_t *v, const char *name);
bool pdf_is_num(const pdf_val_t *v);
/* A number's value; 0 for anything else. */
double pdf_num(const pdf_val_t *v);
/* The decoded data of a stream; NULL when it has none. */
const unsigned char *pdf_stream_decode(pdf_doc_t *d, pdf_stream_t *st, size_t *len);
bool pdf_doc_open(pdf_doc_t *d);
void pdf_doc_close(pdf_doc_t *d);
/* Memory: arena values, and big blocks the document owns. */
void *pdf_alloc(pdf_doc_t *d, size_t n);
void *pdf_calloc(pdf_doc_t *d, size_t n);
void *pdf_big(pdf_doc_t *d, size_t n); /* owned, freed with the document */
bool pdf_own(pdf_doc_t *d, void *block); /* hand a memory-core block to the document */
bool pdf_scratch(pdf_doc_t *d, size_t need); /* d->scratch holds at least `need` bytes */
/* ── Fonts (pdf_font.c) ──────────────────────────────────────────── */
typedef struct {
const char *utf8; /* NULL: unmapped; "" maps to nothing; may point into inl */
double width; /* advance in text-space units per unit of font size */
bool space; /* the single-byte code 32 */
char inl[16]; /* a short mapping made for this glyph */
} pdf_glyph_out_t;
/* The font resources[/Font][name] names; NULL when there is none. Cached. */
pdf_font_t *pdf_font_for(pdf_doc_t *d, pdf_val_t *resources, const pdf_val_t *name);
/* Decode shown bytes into glyphs, out[0..*n) (out holds at least len entries). */
void pdf_font_decode(pdf_doc_t *d, pdf_font_t *f, const unsigned char *s, size_t len,
pdf_glyph_out_t *out, size_t *n);
/* The text-space size of one em per unit of font size. */
double pdf_font_em(const pdf_font_t *f);
/* A glyph name's Unicode value as UTF-8 (arena or static); NULL when unknown. */
const char *pdf_glyph_to_utf8(pdf_doc_t *d, const unsigned char *name, size_t n);
#endif /* CBM_PDF_INTERNAL_H */
internal/cbm/pdf/pdf_obj.c
+2594
View File
File diff suppressed because it is too large. Load diff
internal/cbm/pdf/pdf_tables.c
+1619
View File
File diff suppressed because it is too large. Load diff
internal/cbm/pdf/pdf_text.c
+1040
View File
File diff suppressed because it is too large. Load diff
internal/cbm/pdf/pdf_unicode.c
+3124
View File
File diff suppressed because it is too large. Load diff
internal/cbm/pdf/pdf_unicode.h
+103
View File
@@ -0,0 +1,103 @@
/*
* pdf_unicode.h — the Unicode facts the PDF mention scanner needs, exactly
* as CPython answers them (pdf_unicode.c is generated from CPython's
* unicodedata; the field-tested scanner ran on CPython's str and re).
*/
#ifndef CBM_PDF_UNICODE_H
#define CBM_PDF_UNICODE_H
#include <stdbool.h>
#include <stdint.h>
typedef struct {
uint32_t cp;
uint16_t off; /* into PDF_NFKC_SEQ */
uint8_t len;
} pdf_nfkc_t;
typedef struct {
uint32_t first;
uint32_t second;
uint32_t composite;
} pdf_ucomp_t;
typedef struct {
uint32_t lo;
uint32_t hi;
} pdf_urange_t;
extern const pdf_nfkc_t PDF_NFKC[]; /* sorted by cp: NFKC of one code point where it differs */
extern const int PDF_NFKC_COUNT;
extern const uint32_t PDF_NFKC_SEQ[];
extern const pdf_ucomp_t PDF_UCOMP[]; /* sorted by (first, second): canonical compositions */
extern const int PDF_UCOMP_COUNT;
extern const pdf_urange_t PDF_U_SPACE[]; /* str.isspace */
extern const int PDF_U_SPACE_COUNT;
extern const pdf_urange_t PDF_U_WORD[]; /* re's \w: str.isalnum or '_' */
extern const int PDF_U_WORD_COUNT;
extern const pdf_urange_t PDF_U_ALPHA[]; /* str.isalpha */
extern const int PDF_U_ALPHA_COUNT;
extern const pdf_urange_t PDF_U_UPPER[]; /* one-character str.isupper */
extern const int PDF_U_UPPER_COUNT;
extern const pdf_urange_t PDF_U_LOWER[]; /* one-character str.islower */
extern const int PDF_U_LOWER_COUNT;
extern const pdf_urange_t PDF_U_TITLE[]; /* titlecase letters (neither upper nor lower) */
extern const int PDF_U_TITLE_COUNT;
static inline bool pdf_u_in(const pdf_urange_t *r, int n, uint32_t cp) {
int lo = 0;
int hi = n - 1;
while (lo <= hi) {
int mid = lo + (hi - lo) / 2;
if (r[mid].hi < cp) {
lo = mid + 1;
} else if (r[mid].lo > cp) {
hi = mid - 1;
} else {
return true;
}
}
return false;
}
static inline bool pdf_u_space(uint32_t c) {
if (c < 0x80) {
return c == ' ' || (c >= 0x09 && c <= 0x0D) || (c >= 0x1C && c <= 0x1F);
}
return pdf_u_in(PDF_U_SPACE, PDF_U_SPACE_COUNT, c);
}
static inline bool pdf_u_word(uint32_t c) {
if (c < 0x80) {
return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') ||
c == '_';
}
return pdf_u_in(PDF_U_WORD, PDF_U_WORD_COUNT, c);
}
static inline bool pdf_u_alpha(uint32_t c) {
if (c < 0x80) {
return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z');
}
return pdf_u_in(PDF_U_ALPHA, PDF_U_ALPHA_COUNT, c);
}
static inline bool pdf_u_upper(uint32_t c) {
if (c < 0x80) {
return c >= 'A' && c <= 'Z';
}
return pdf_u_in(PDF_U_UPPER, PDF_U_UPPER_COUNT, c);
}
static inline bool pdf_u_lower(uint32_t c) {
if (c < 0x80) {
return c >= 'a' && c <= 'z';
}
return pdf_u_in(PDF_U_LOWER, PDF_U_LOWER_COUNT, c);
}
static inline bool pdf_u_title(uint32_t c) {
return c >= 0x80 && pdf_u_in(PDF_U_TITLE, PDF_U_TITLE_COUNT, c);
}
#endif /* CBM_PDF_UNICODE_H */
src/discover/language.c
+2
View File
@@ -618,6 +618,7 @@ static const ext_entry_t EXT_TABLE[] = {
{".clsp", CBM_LANG_CHIALISP},
{".clib", CBM_LANG_CHIALISP},
{".clinc", CBM_LANG_CHIALISP},
{".pdf", CBM_LANG_PDF},
/* Slang */
{".slang", CBM_LANG_SLANG},
@@ -829,6 +830,7 @@ static const char *LANG_NAMES[CBM_LANG_COUNT] = {
[CBM_LANG_NIM] = "Nim",
[CBM_LANG_SCHEME] = "Scheme",
[CBM_LANG_CHIALISP] = "Chialisp",
[CBM_LANG_PDF] = "PDF",
[CBM_LANG_FENNEL] = "Fennel",
[CBM_LANG_FISH] = "Fish",
[CBM_LANG_AWK] = "AWK",
src/discover/userconfig.c
+1
View File
@@ -160,6 +160,7 @@ static const lang_name_entry_t LANG_NAME_TABLE[] = {
{"mojo", CBM_LANG_MOJO},
{"plsql", CBM_LANG_PLSQL},
{"chialisp", CBM_LANG_CHIALISP},
{"pdf", CBM_LANG_PDF},
{"lean", CBM_LANG_LEAN},
{"form", CBM_LANG_FORM},
{"magma", CBM_LANG_MAGMA},
src/mcp/mcp.c
+38 -2
View File
@@ -12535,6 +12535,36 @@ static char *resolve_path_owner(const char *path, char *derived) {
return owner;
}
/* A PDF is read as its text layer at index time (doc_pdf.c); its bytes are no
* source text. Its page Sections carry the page's text as their docstring. */
static bool snippet_is_pdf(const char *file_path) {
size_t n = file_path ? strlen(file_path) : 0;
return n >= 4 && file_path[n - 4] == '.' && (file_path[n - 3] | 0x20) == 'p' &&
(file_path[n - 2] | 0x20) == 'd' && (file_path[n - 1] | 0x20) == 'f';
}
static char *snippet_pdf_text(const cbm_node_t *node) {
if (!node->properties_json) {
return NULL;
}
yyjson_doc *pd = yyjson_read(node->properties_json, strlen(node->properties_json), 0);
if (!pd) {
return NULL;
}
yyjson_val *ds = yyjson_obj_get(yyjson_doc_get_root(pd), "docstring");
const char *text = yyjson_is_str(ds) ? yyjson_get_str(ds) : NULL;
char *out = NULL;
if (text && text[0]) {
/* the same builder read_file_lines returns its text from */
cbm_sb_t sb;
cbm_sb_init(&sb);
cbm_sb_append_n(&sb, text, strlen(text));
out = cbm_sb_finish(&sb);
}
yyjson_doc_free(pd);
return out;
}
static char *resolve_snippet_source(const char *root_path, const char *file_path, int start,
int end, char **out_abs_path) {
*out_abs_path = NULL;
@@ -12885,8 +12915,14 @@ static char *build_snippet_response(cbm_mcp_server_t *srv, cbm_node_t *node,
snippet_clipped = true;
}
char *abs_path = NULL;
char *source =
outline ? NULL : resolve_snippet_source(root_path, node->file_path, start, end, &abs_path);
char *source = NULL;
if (outline) {
source = NULL;
} else if (snippet_is_pdf(node->file_path)) {
source = snippet_pdf_text(node); /* never the file's bytes */
} else {
source = resolve_snippet_source(root_path, node->file_path, start, end, &abs_path);
}
yyjson_mut_doc *doc = yyjson_mut_doc_new(NULL);
yyjson_mut_val *root_obj = yyjson_mut_obj(doc);
src/pipeline/doc_links.c
+7 -1
View File
@@ -47,6 +47,7 @@ const char *cbm_doclink_reason_name(int reason) {
static const cbm_doclink_resolver_t *const DOCLINK_RESOLVERS[] = {
&cbm_doclink_cs_resolver,
&cbm_doclink_md_resolver,
&cbm_doclink_pdf_resolver,
};
enum { DOCLINK_RESOLVER_COUNT = sizeof(DOCLINK_RESOLVERS) / sizeof(DOCLINK_RESOLVERS[0]) };
@@ -462,7 +463,12 @@ void cbm_doclinks_resolve_file(cbm_doclinks_t *dl, int file_idx, const CBMFileRe
* file-level doc resolves */
const cbm_gbuf_node_t *file_node = NULL;
bool file_node_looked_up = false;
void *state = (R && index && R->file_begin) ? R->file_begin(index, file_idx) : NULL;
void *state = NULL;
if (R && index && R->file_prepare) {
state = R->file_prepare(index, file_idx, result->doc_links.items, n, graph);
} else if (R && index && R->file_begin) {
state = R->file_begin(index, file_idx);
}
for (int i = 0; i < n; i++) {
const CBMDocLink *link = &result->doc_links.items[i];
cbm_doclink_outcome_t out = {.kind = CBM_DOCLINK_UNRESOLVED,
src/pipeline/doc_links.h
+8
View File
@@ -198,6 +198,12 @@ typedef struct {
* of the file, released after its last. A NULL state (no hook, or memory
* ran out) changes the cost of resolving, never an outcome. */
void *(*file_begin)(const void *index, int run_file);
/* Optional, instead of file_begin: the file's state built from ALL its
* references before any is resolved (a PDF's cross-line joins decide
* which line fragments stand). `links` stays valid until file_end, and
* resolve's `link` points into it. */
void *(*file_prepare)(const void *index, int run_file, const CBMDocLink *links, int n,
const cbm_gbuf_t *graph);
void (*file_end)(void *state);
/* Resolve one reference of run file `run_file` (its own scope is in the
* index). Thread-safe: the index is read-only after build; `state` is the
@@ -234,6 +240,8 @@ typedef struct {
extern const cbm_doclink_resolver_t cbm_doclink_cs_resolver;
/* Markdown documents (doc_links_md.c): via "markdown". */
extern const cbm_doclink_resolver_t cbm_doclink_md_resolver;
/* PDF documents (doc_links_pdf.c): via "pdf". */
extern const cbm_doclink_resolver_t cbm_doclink_pdf_resolver;
#if defined(CBM_ENABLE_TEST_SEAMS) && CBM_ENABLE_TEST_SEAMS
/* Test seam (doc_links_cs.c): the scope levels and overloads the C# resolver
src/pipeline/doc_links_pdf.c
+807
View File
@@ -0,0 +1,807 @@
/*
* doc_links_pdf.c — PDF mentions -> MENTIONS edges: the resolving half.
*
* A PDF page (doc_pdf.c) names code structurally: a path or a qualified name.
* The tiers are the field-tested ones (E6, frozen resolver), EXACT or UNIQUE
* only:
* path-exact the mention is a File/Folder node's repository path
* path-suffix exactly one File/Folder whose path ends with the mention at a
* '/' boundary
* qn-exact exactly one node whose qualified name (project prefix off)
* is the mention
* qn-suffix exactly one node whose qualified name ends with the mention
* at a component boundary -- or whose owner-qualified name
* does (parent_class + name: a Go method's QN has no receiver)
* Two candidates that are one entity collapse first (a class and its
* constructors; duplicate nodes of one definition). A cross-line join is
* resolved before anything else in its file: when it resolves EXACT or
* UNIQUE it replaces the line fragments it was built from.
*
* Hygiene (the field test's R2 and R4): a reference that resolves to a data
* file (.json, .yaml, ...) is no code reference; one that resolves into a
* test, mock or fixture file is a test_only_target row.
*
* Unresolved: ambiguous (several entities); missing (nothing, but the
* mention's first component is a name or directory of this repository).
* Anything else names code outside the repository: LOCAL, no row.
*
* Targets exclude the labels the incremental route does not proxy (Macro,
* Comment, Section, Branch, Commit, Tag) and the field test's non-code labels
* (Project, Route, Channel, Decorator), so a re-resolved document binds what a
* full build binds.
*/
#include "pipeline/doc_links.h"
#include "doclink.h"
#include "foundation/arena.h"
#include "foundation/constants.h"
#include "foundation/hash_table.h"
#include "foundation/mem_core.h"
#include "graph_buffer/graph_buffer.h"
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
enum {
PDFR_TABLE_INIT = 1024,
PDFR_KEY_CAP = 1024, /* the longest mention form looked up */
PDFR_MAX_COMPS = 64,
PDFR_MAX_FORMS = 2,
PDFR_MEMO_INIT = 64,
};
static const char *const PDFR_EXCLUDED_LABELS[] = {
"Section", "Project", "Branch", "Route", "Channel", "Decorator", "Macro",
"Comment", "Commit", "Tag", "File", "Folder", NULL};
static const char *const PDFR_CONTAINERS[] = {"Class", "Struct", "Interface", "Enum",
"Trait", "Type", "Record", "Object",
"Protocol", "Union", NULL};
static const char *const PDFR_DATA_EXT[] = {".json", ".yaml", ".yml", ".toml",
".csv", ".tsv", ".lock", NULL};
/* R4: the frozen test-file rule, with the directories the field test found it
* missing (Foundry *.t.sol, test/, mocks/, testutil/, testing/, testdata/). */
static const char *const PDFR_TEST_DIRS[] = {"tests", "__tests__", "test", "mocks",
"testutil", "testing", "testdata", NULL};
static bool in_list(const char *s, const char *const *list) {
for (int i = 0; list[i]; i++) {
if (strcmp(s, list[i]) == 0) {
return true;
}
}
return false;
}
static bool ends_with(const char *s, const char *suf) {
size_t n = strlen(s);
size_t m = strlen(suf);
return n >= m && memcmp(s + n - m, suf, m) == 0;
}
static bool starts_with(const char *s, const char *pre) {
return strncmp(s, pre, strlen(pre)) == 0;
}
/* ── Index ───────────────────────────────────────────────────────── */
typedef struct pdfr_list {
const cbm_gbuf_node_t *node;
struct pdfr_list *next;
} pdfr_list_t;
typedef struct {
const char *project;
size_t project_len;
CBMArena arena;
CBMHashTable *paths; /* File/Folder repository path -> pdfr_list_t* */
CBMHashTable *basenames; /* last path component -> pdfr_list_t* */
CBMHashTable *dirs; /* every directory component of an indexed path (value 1) */
/* Code nodes whose qualified name does not end with their name (a Module
* is named by its file path), keyed by the QN's last component: the
* field test matched qualified names by their components, not by name. */
CBMHashTable *irregular;
bool failed;
} pdfr_index_t;
static void pdfr_destroy(void *index);
static bool list_add(pdfr_index_t *x, CBMHashTable *t, const char *key, const cbm_gbuf_node_t *n) {
pdfr_list_t *e = (pdfr_list_t *)cbm_arena_alloc(&x->arena, sizeof(*e));
if (!e) {
return false;
}
e->node = n;
e->next = (pdfr_list_t *)cbm_ht_get(t, key);
cbm_ht_set(t, key, e);
return true;
}
static bool index_paths(pdfr_index_t *x, const cbm_gbuf_t *graph, const char *label) {
const cbm_gbuf_node_t **nodes = NULL;
int count = 0;
if (cbm_gbuf_find_by_label(graph, label, &nodes, &count) != 0) {
return true;
}
for (int i = 0; i < count; i++) {
const char *fp = nodes[i]->file_path;
if (!fp || !fp[0] || fp[0] == '<') {
continue;
}
while (*fp == '/') {
fp++;
}
size_t n = strlen(fp);
while (n > 0 && fp[n - 1] == '/') {
n--;
}
char *p = cbm_arena_strndup(&x->arena, fp, n);
if (!p || !list_add(x, x->paths, p, nodes[i])) {
return false;
}
const char *slash = strrchr(p, '/');
if (!list_add(x, x->basenames, slash ? slash + 1 : p, nodes[i])) {
return false;
}
/* directory components, for "is this a name of the repository" */
const char *s = p;
for (const char *q = p;; q++) {
if (*q == '/' || *q == '\0') {
if (q > s) {
char *c = cbm_arena_strndup(&x->arena, s, (size_t)(q - s));
if (!c) {
return false;
}
cbm_ht_set(x->dirs, c, (void *)(uintptr_t)1);
}
if (!*q) {
break;
}
s = q + 1;
}
}
}
return true;
}
static bool code_node(const pdfr_index_t *x, const cbm_gbuf_node_t *n);
static const char *rel_qn(const pdfr_index_t *x, const char *qn);
static void index_irregular(const cbm_gbuf_node_t *n, void *ud) {
pdfr_index_t *x = (pdfr_index_t *)ud;
if (x->failed || !code_node(x, n)) {
return;
}
const char *rel = rel_qn(x, n->qualified_name);
const char *dot = strrchr(rel, '.');
const char *last = dot ? dot + 1 : rel;
if (n->name && strcmp(last, n->name) == 0) {
return;
}
if (!list_add(x, x->irregular, last, n)) {
x->failed = true;
}
}
static void *pdfr_build(const cbm_doclink_build_in_t *in) {
if (in->file_count == 0) {
return NULL; /* no PDF in the repository: nothing to resolve */
}
pdfr_index_t *x = (pdfr_index_t *)cbm_calloc(CBM_MEM_CLASS_OTHER, sizeof(*x));
if (!x) {
return NULL;
}
cbm_arena_init(&x->arena);
x->project = in->ctx ? in->ctx->project_name : NULL;
x->project_len = x->project ? strlen(x->project) : 0;
x->paths = cbm_ht_create_in(CBM_MEM_CLASS_HASH_TABLE, PDFR_TABLE_INIT);
x->basenames = cbm_ht_create_in(CBM_MEM_CLASS_HASH_TABLE, PDFR_TABLE_INIT);
x->dirs = cbm_ht_create_in(CBM_MEM_CLASS_HASH_TABLE, PDFR_TABLE_INIT);
x->irregular = cbm_ht_create_in(CBM_MEM_CLASS_HASH_TABLE, PDFR_TABLE_INIT);
if (!x->project || !x->paths || !x->basenames || !x->dirs || !x->irregular ||
!index_paths(x, in->graph, "File") || !index_paths(x, in->graph, "Folder")) {
pdfr_destroy(x);
return NULL;
}
cbm_gbuf_foreach_node(in->graph, index_irregular, x);
if (x->failed) {
pdfr_destroy(x);
return NULL;
}
return x;
}
static void pdfr_destroy(void *index) {
pdfr_index_t *x = (pdfr_index_t *)index;
if (!x) {
return;
}
cbm_ht_free(x->paths);
cbm_ht_free(x->basenames);
cbm_ht_free(x->dirs);
cbm_ht_free(x->irregular);
cbm_arena_destroy(&x->arena);
cbm_free(CBM_MEM_CLASS_OTHER, x);
}
/* ── Candidate sets ──────────────────────────────────────────────── */
typedef struct {
const cbm_gbuf_node_t **v;
int n;
int cap;
bool fail;
} nodeset_t;
static void ns_add(nodeset_t *s, const cbm_gbuf_node_t *n) {
for (int i = 0; i < s->n; i++) {
if (s->v[i] == n) {
return;
}
}
if (s->fail) {
return;
}
if (s->n == s->cap) {
int ncap = s->cap ? s->cap * 2 : 8;
const cbm_gbuf_node_t **g = (const cbm_gbuf_node_t **)cbm_realloc(
CBM_MEM_CLASS_OTHER, (void *)s->v, (size_t)ncap * sizeof(*g));
if (!g) {
s->fail = true;
return;
}
s->v = g;
s->cap = ncap;
}
s->v[s->n++] = n;
}
static void ns_free(nodeset_t *s) {
cbm_free(CBM_MEM_CLASS_OTHER, (void *)s->v);
memset(s, 0, sizeof(*s));
}
/* The qualified name without the project prefix. */
static const char *rel_qn(const pdfr_index_t *x, const char *qn) {
if (qn && x->project_len && strncmp(qn, x->project, x->project_len) == 0 &&
qn[x->project_len] == '.') {
return qn + x->project_len + 1;
}
return qn ? qn : "";
}
static bool code_node(const pdfr_index_t *x, const cbm_gbuf_node_t *n) {
if (!n->label || in_list(n->label, PDFR_EXCLUDED_LABELS) || !n->qualified_name ||
!n->qualified_name[0]) {
return false;
}
if (n->file_path && n->file_path[0] == '<') {
return false; /* builtins */
}
return strncmp(rel_qn(x, n->qualified_name), "__", 2) != 0; /* dependency stand-ins */
}
/* s is a component-aligned suffix of q ('.'-separated) */
static bool comp_suffix(const char *q, const char *s) {
size_t ql = strlen(q);
size_t sl = strlen(s);
if (sl > ql || memcmp(q + ql - sl, s, sl) != 0) {
return false;
}
return sl == ql || q[ql - sl - 1] == '.';
}
/* The owner-qualified name of a node: rel(parent_class) + "." + name, when the
* node has a parent_class (properties JSON). */
static bool owner_name(const pdfr_index_t *x, const cbm_gbuf_node_t *n, char *buf, size_t cap) {
const char *p = n->properties_json ? strstr(n->properties_json, "\"parent_class\":\"") : NULL;
if (!p) {
return false;
}
p += strlen("\"parent_class\":\"");
char pc[PDFR_KEY_CAP];
size_t k = 0;
while (*p && *p != '"' && k + 1 < sizeof(pc)) {
if (*p == '\\' && p[1]) {
p++;
}
pc[k++] = *p++;
}
pc[k] = '\0';
if (*p != '"') {
return false;
}
const char *prel = rel_qn(x, pc);
int w = snprintf(buf, cap, "%s.%s", prel, n->name ? n->name : "");
return w > 0 && (size_t)w < cap;
}
static void qn_candidate(const pdfr_index_t *x, const cbm_gbuf_node_t *n, const char *s,
nodeset_t *out, nodeset_t *exact) {
if (!code_node(x, n)) {
return;
}
const char *rel = rel_qn(x, n->qualified_name);
char alt[PDFR_KEY_CAP];
bool via_rel = comp_suffix(rel, s);
bool via_alt = !via_rel && owner_name(x, n, alt, sizeof(alt)) && strcmp(alt, rel) != 0 &&
comp_suffix(alt, s);
if (via_rel || via_alt) {
ns_add(out, n);
if (via_rel && strcmp(rel, s) == 0) {
ns_add(exact, n);
}
}
}
/* Nodes whose QN or owner-qualified name ends with the mention s (>= 2
* components). A node's name may itself hold dots, so every component suffix
* of s is tried as a name. */
static void qn_suffix_set(const pdfr_index_t *x, const cbm_gbuf_t *graph, const char *s,
nodeset_t *out, nodeset_t *exact) {
const char *starts[PDFR_MAX_COMPS];
int nc = 0;
starts[nc++] = s;
for (const char *q = s; *q && nc < PDFR_MAX_COMPS; q++) {
if (*q == '.') {
starts[nc++] = q + 1;
}
}
if (nc < 2) {
return;
}
for (int j = nc - 1; j >= 0; j--) {
const cbm_gbuf_node_t **nodes = NULL;
int count = 0;
if (cbm_gbuf_find_by_name(graph, starts[j], &nodes, &count) != 0) {
continue;
}
for (int i = 0; i < count; i++) {
qn_candidate(x, nodes[i], s, out, exact);
}
}
for (const pdfr_list_t *l = (const pdfr_list_t *)cbm_ht_get(x->irregular, starts[nc - 1]); l;
l = l->next) {
qn_candidate(x, l->node, s, out, exact);
}
}
static int node_id_cmp(const void *a, const void *b) {
const cbm_gbuf_node_t *x = *(const cbm_gbuf_node_t *const *)a;
const cbm_gbuf_node_t *y = *(const cbm_gbuf_node_t *const *)b;
return x->id < y->id ? -1 : x->id > y->id ? 1 : 0;
}
/* The field test's _collapse: one entity's several nodes become one. */
static const cbm_gbuf_node_t *collapse(const pdfr_index_t *x, nodeset_t *s) {
if (s->n == 1) {
return s->v[0];
}
if (s->n == 0) {
return NULL;
}
qsort((void *)s->v, (size_t)s->n, sizeof(*s->v), node_id_cmp);
const cbm_gbuf_node_t *c = NULL;
int containers = 0;
for (int i = 0; i < s->n; i++) {
if (s->v[i]->label && in_list(s->v[i]->label, PDFR_CONTAINERS)) {
c = s->v[i];
containers++;
}
}
if (containers == 1) {
const char *crel = rel_qn(x, c->qualified_name);
size_t cl = strlen(crel);
bool all = true;
for (int i = 0; i < s->n && all; i++) {
const cbm_gbuf_node_t *n = s->v[i];
if (n == c) {
continue;
}
const char *rel = rel_qn(x, n->qualified_name);
all = n->name && c->name && strcmp(n->name, c->name) == 0 &&
strncmp(rel, crel, cl) == 0 && rel[cl] == '.';
}
if (all) {
return c; /* a class and its constructors */
}
}
const cbm_gbuf_node_t *f = s->v[0];
for (int i = 1; i < s->n; i++) {
const cbm_gbuf_node_t *n = s->v[i];
if (strcmp(n->file_path ? n->file_path : "", f->file_path ? f->file_path : "") != 0 ||
n->start_line != f->start_line ||
strcmp(n->name ? n->name : "", f->name ? f->name : "") != 0) {
return NULL;
}
}
return f; /* duplicate nodes of one definition: the lowest id */
}
/* ── One mention form ────────────────────────────────────────────── */
typedef enum { RES_NONE = 0, RES_AMBIGUOUS, RES_HIT } res_kind_t;
typedef struct {
res_kind_t kind;
bool exact;
const cbm_gbuf_node_t *node;
} res_t;
static res_t outcome(const pdfr_index_t *x, nodeset_t *s, bool exact) {
res_t r = {RES_NONE, false, NULL};
if (s->n == 0) {
return r;
}
const cbm_gbuf_node_t *n = collapse(x, s);
if (n) {
r.kind = RES_HIT;
r.exact = exact;
r.node = n;
} else {
r.kind = RES_AMBIGUOUS;
}
return r;
}
static void list_to_set(const pdfr_list_t *l, nodeset_t *s) {
for (; l; l = l->next) {
ns_add(s, l->node);
}
}
static res_t resolve_form(const pdfr_index_t *x, const cbm_gbuf_t *graph, int syntax,
const char *form) {
res_t none = {RES_NONE, false, NULL};
if (syntax == CBM_DOCLINK_PDF_PATH || syntax == CBM_DOCLINK_PDF_FILE) {
const char *p = form;
while (*p == '/') {
p++;
}
char key[PDFR_KEY_CAP];
size_t n = strlen(p);
while (n > 0 && p[n - 1] == '/') {
n--;
}
if (n == 0 || n >= sizeof(key)) {
return none;
}
memcpy(key, p, n);
key[n] = '\0';
nodeset_t s = {0};
list_to_set((const pdfr_list_t *)cbm_ht_get(x->paths, key), &s);
if (s.n) {
res_t r = outcome(x, &s, true);
ns_free(&s);
return r;
}
const char *slash = strrchr(key, '/');
for (const pdfr_list_t *l =
(const pdfr_list_t *)cbm_ht_get(x->basenames, slash ? slash + 1 : key);
l; l = l->next) {
const char *fp = l->node->file_path;
while (*fp == '/') {
fp++;
}
size_t fl = strlen(fp);
while (fl > 0 && fp[fl - 1] == '/') {
fl--;
}
if (fl >= n && memcmp(fp + fl - n, key, n) == 0 && (fl == n || fp[fl - n - 1] == '/')) {
ns_add(&s, l->node);
}
}
if (s.n) {
res_t r = outcome(x, &s, false);
ns_free(&s);
return r;
}
ns_free(&s);
if (syntax == CBM_DOCLINK_PDF_PATH) {
return none;
}
}
if (syntax == CBM_DOCLINK_PDF_QN || syntax == CBM_DOCLINK_PDF_FILE) {
nodeset_t all = {0};
nodeset_t ex = {0};
qn_suffix_set(x, graph, form, &all, &ex);
res_t r = none;
if (ex.n == 1) {
r.kind = RES_HIT;
r.exact = true;
r.node = ex.v[0];
} else if (all.n) {
r = outcome(x, &all, false);
}
ns_free(&all);
ns_free(&ex);
return r;
}
if (syntax == CBM_DOCLINK_PDF_NAME) {
const cbm_gbuf_node_t **nodes = NULL;
int count = 0;
nodeset_t s = {0};
if (cbm_gbuf_find_by_name(graph, form, &nodes, &count) == 0) {
for (int i = 0; i < count; i++) {
if (code_node(x, nodes[i])) {
ns_add(&s, nodes[i]);
}
}
}
res_t r = outcome(x, &s, false);
ns_free(&s);
return r;
}
return none;
}
/* A join's forms: the first, plus the hyphen-kept alternative. One node hit
* by the forms is the outcome; two different nodes are ambiguous. */
static res_t resolve_join(const pdfr_index_t *x, const cbm_gbuf_t *graph, int syntax,
const char *raw) {
char buf[PDFR_KEY_CAP];
const char *end = strchr(raw, '\x1e');
size_t len = end ? (size_t)(end - raw) : strlen(raw);
if (len >= sizeof(buf)) {
res_t none = {RES_NONE, false, NULL};
return none;
}
memcpy(buf, raw, len);
buf[len] = '\0';
char *forms[PDFR_MAX_FORMS];
int nf = 0;
forms[nf++] = buf;
char *sep = strchr(buf, '\x1f');
if (sep) {
*sep = '\0';
forms[nf++] = sep + 1;
}
res_t first = resolve_form(x, graph, syntax, forms[0]);
if (nf == 1) {
return first;
}
res_t second = resolve_form(x, graph, syntax, forms[1]);
if (first.kind == RES_HIT && second.kind == RES_HIT) {
if (first.node == second.node) {
return first;
}
res_t amb = {RES_AMBIGUOUS, false, NULL};
return amb;
}
if (first.kind == RES_HIT) {
return first;
}
if (second.kind == RES_HIT) {
return second;
}
return first;
}
/* ── Hygiene ─────────────────────────────────────────────────────── */
static bool data_target(const char *fp) {
char low[PDFR_KEY_CAP];
size_t n = strlen(fp);
if (n >= sizeof(low)) {
return false;
}
for (size_t i = 0; i <= n; i++) {
low[i] = (char)(fp[i] >= 'A' && fp[i] <= 'Z' ? fp[i] + ('a' - 'A') : fp[i]);
}
for (int i = 0; PDFR_DATA_EXT[i]; i++) {
if (ends_with(low, PDFR_DATA_EXT[i])) {
return true;
}
}
return false;
}
static bool test_target(const char *fp) {
const char *base = strrchr(fp, '/');
base = base ? base + 1 : fp;
if ((starts_with(base, "test_") && ends_with(base, ".py")) || ends_with(base, "_test.py") ||
ends_with(base, "_test.go") || ends_with(base, "Test.cs") || ends_with(base, "Tests.cs") ||
ends_with(base, "Test.java") || ends_with(base, "Tests.java") ||
strcmp(base, "conftest.py") == 0 || ends_with(base, ".t.sol")) {
return true;
}
static const char *const js[] = {".test.js", ".test.jsx", ".test.ts", ".test.tsx", ".spec.js",
".spec.jsx", ".spec.ts", ".spec.tsx", NULL};
for (int i = 0; js[i]; i++) {
if (ends_with(base, js[i]) && strlen(base) > strlen(js[i])) {
return true;
}
}
const char *s = fp;
for (const char *q = fp; q < base; q++) {
if (*q == '/') {
char comp[PDFR_KEY_CAP];
size_t n = (size_t)(q - s);
if (n < sizeof(comp)) {
memcpy(comp, s, n);
comp[n] = '\0';
if (in_list(comp, PDFR_TEST_DIRS)) {
return true;
}
}
s = q + 1;
}
}
return false;
}
/* ── Per file ────────────────────────────────────────────────────── */
typedef struct {
const CBMDocLink *base;
int n;
uint8_t *suppressed; /* line-local tokens a resolved join replaces */
res_t *join; /* each join token's own outcome */
/* A page set repeats its paths and names (an audit names one file on
* every page): mention text -> 1 + the index of the token whose outcome
* res[] holds, reused for a token of the same family. */
CBMHashTable *memo;
res_t *res;
} pdfr_state_t;
static void pdfr_end(void *state);
static void *pdfr_prepare(const void *index, int run_file, const CBMDocLink *links, int n,
const cbm_gbuf_t *graph) {
(void)run_file;
const pdfr_index_t *x = (const pdfr_index_t *)index;
pdfr_state_t *st = (pdfr_state_t *)cbm_calloc(CBM_MEM_CLASS_OTHER, sizeof(*st));
if (!st) {
return NULL;
}
st->base = links;
st->n = n;
st->suppressed = (uint8_t *)cbm_calloc(CBM_MEM_CLASS_OTHER, (size_t)(n ? n : 1));
st->join = (res_t *)cbm_calloc(CBM_MEM_CLASS_OTHER, (size_t)(n ? n : 1) * sizeof(res_t));
st->res = (res_t *)cbm_calloc(CBM_MEM_CLASS_OTHER, (size_t)(n ? n : 1) * sizeof(res_t));
st->memo = cbm_ht_create_in(CBM_MEM_CLASS_HASH_TABLE, PDFR_MEMO_INIT);
if (!st->suppressed || !st->join || !st->res || !st->memo) {
pdfr_end(st);
return NULL;
}
for (int i = 0; i < n; i++) {
const CBMDocLink *l = &links[i];
if (!(l->flags & CBM_DOCLINK_FLAG_JOIN) || !l->raw) {
continue;
}
st->join[i] = resolve_join(x, graph, l->syntax, l->raw);
if (st->join[i].kind != RES_HIT) {
continue;
}
const char *f = strchr(l->raw, '\x1e');
while (f && *f) {
f++;
char *endp = NULL;
long k = strtol(f, &endp, 10);
if (endp == f) {
break;
}
if (k >= 0 && k < n) {
st->suppressed[k] = 1;
}
f = (*endp == ',') ? endp : NULL;
}
}
return st;
}
static void pdfr_end(void *state) {
pdfr_state_t *st = (pdfr_state_t *)state;
if (!st) {
return;
}
cbm_free(CBM_MEM_CLASS_OTHER, st->suppressed);
cbm_free(CBM_MEM_CLASS_OTHER, st->join);
cbm_free(CBM_MEM_CLASS_OTHER, st->res);
cbm_ht_free(st->memo);
cbm_free(CBM_MEM_CLASS_OTHER, st);
}
/* The mention's first component names something of this repository. */
static bool first_known(const pdfr_index_t *x, const cbm_gbuf_t *graph, int syntax,
const char *raw) {
char first[PDFR_KEY_CAP];
size_t n = 0;
const char *p = raw;
while (*p == '/') {
p++;
}
char sep = syntax == CBM_DOCLINK_PDF_PATH ? '/' : '.';
while (p[n] && p[n] != sep && n + 1 < sizeof(first)) {
first[n] = p[n];
n++;
}
first[n] = '\0';
if (!n) {
return false;
}
if (cbm_ht_get(x->dirs, first) || cbm_ht_get(x->basenames, first)) {
return true;
}
const cbm_gbuf_node_t **nodes = NULL;
int count = 0;
return cbm_gbuf_find_by_name(graph, first, &nodes, &count) == 0 && count > 0;
}
static void finish_hit(cbm_doclink_outcome_t *out, res_t r) {
const char *fp = r.node->file_path ? r.node->file_path : "";
if (data_target(fp)) {
out->kind = CBM_DOCLINK_LOCAL; /* R2: a data file, not code */
return;
}
if (test_target(fp)) {
out->kind = CBM_DOCLINK_UNRESOLVED; /* R4 */
out->reason = CBM_DOCLINK_REASON_TEST_ONLY;
return;
}
out->kind = CBM_DOCLINK_EDGE;
out->target = r.node;
out->exact = r.exact;
}
static void pdfr_resolve(const void *index, void *state, int run_file, const CBMDocLink *link,
const cbm_gbuf_t *graph, cbm_doclink_outcome_t *out) {
(void)run_file;
const pdfr_index_t *x = (const pdfr_index_t *)index;
pdfr_state_t *st = (pdfr_state_t *)state;
out->kind = CBM_DOCLINK_LOCAL;
if (!x || !link->raw) {
return;
}
int i = st ? (int)(link - st->base) : -1;
bool in_file = st && i >= 0 && i < st->n;
if (link->flags & CBM_DOCLINK_FLAG_JOIN) {
/* a join stands only when it resolved; a bare-name join never links */
if (in_file && st->join[i].kind == RES_HIT && link->syntax != CBM_DOCLINK_PDF_NAME) {
finish_hit(out, st->join[i]);
}
return;
}
if (in_file && st->suppressed[i]) {
return; /* a resolved join replaced this fragment */
}
if (link->syntax == CBM_DOCLINK_PDF_NAME) {
return;
}
res_t r;
uintptr_t seen = in_file ? (uintptr_t)cbm_ht_get(st->memo, link->raw) : 0;
if (seen && st->base[seen - 1].syntax == link->syntax) {
r = st->res[seen - 1];
} else {
r = resolve_form(x, graph, link->syntax, link->raw);
if (in_file && !seen) {
st->res[i] = r;
cbm_ht_set(st->memo, link->raw, (void *)(uintptr_t)(i + 1));
}
}
if (r.kind == RES_HIT) {
finish_hit(out, r);
} else if (r.kind == RES_AMBIGUOUS) {
out->kind = CBM_DOCLINK_UNRESOLVED;
out->reason = CBM_DOCLINK_REASON_AMBIGUOUS;
} else if (first_known(x, graph, link->syntax, link->raw)) {
out->kind = CBM_DOCLINK_UNRESOLVED;
out->reason = CBM_DOCLINK_REASON_MISSING;
}
}
const cbm_doclink_resolver_t cbm_doclink_pdf_resolver = {
.langs = {CBM_LANG_PDF},
.lang_count = 1,
.scope_tag = NULL,
.build = pdfr_build,
.destroy = pdfr_destroy,
.file_prepare = pdfr_prepare,
.file_end = pdfr_end,
.resolve = pdfr_resolve,
.via = "pdf",
};
src/pipeline/pipeline_delta.c
+63 -4
View File
@@ -37,6 +37,7 @@
#include <string.h>
#include "foundation/compat_fs.h"
#include "foundation/mem_core.h"
#include "foundation/log.h"
#include "sqlite3.h"
#include "store/store.h"
@@ -254,6 +255,50 @@ int cbm_delta_purge(cbm_store_t *store, const char *project, const char *const *
* guessing. The proxy load stays edge-free and property-free, which is
* where the old full load actually spent its time. */
/* `qn` and `parent` without the project prefix. */
static const char *delta_rel(const char *project, const char *qn) {
size_t pl = project ? strlen(project) : 0;
if (qn && pl && strncmp(qn, project, pl) == 0 && qn[pl] == '.') {
return qn + pl + 1;
}
return qn ? qn : "";
}
/* A proxy's properties carrying `parent` when owner + "." + name is not the
* node's own (relative) QN, merged with the http client keys if any; NULL
* when the owner adds nothing (CBM_MEM_CLASS_OTHER; the caller frees). */
static char *delta_owner_props(const char *project, const char *name, const char *qn,
const char *parent, const char *client_props) {
const char *rel = delta_rel(project, qn);
const char *prel = delta_rel(project, parent);
size_t pl = strlen(prel);
size_t nl = name ? strlen(name) : 0;
if (strlen(rel) == pl + 1 + nl && strncmp(rel, prel, pl) == 0 && rel[pl] == '.' &&
(nl == 0 || memcmp(rel + pl + 1, name, nl) == 0)) {
return NULL;
}
size_t cap = 2 * strlen(parent) + (client_props ? strlen(client_props) : 0) + 32;
char *out = (char *)cbm_alloc(CBM_MEM_CLASS_OTHER, cap);
if (!out) {
return NULL;
}
size_t k = 0;
k += (size_t)snprintf(out + k, cap - k, "{\"parent_class\":\"");
for (const char *p = parent; *p && k + 3 < cap; p++) {
if (*p == '"' || *p == '\\') {
out[k++] = '\\';
}
out[k++] = *p;
}
out[k++] = '"';
if (client_props && client_props[0] == '{' && client_props[1] && client_props[1] != '}') {
snprintf(out + k, cap - k, ",%s", client_props + 1);
} else {
snprintf(out + k, cap - k, "}");
}
return out;
}
int64_t cbm_delta_preseed(cbm_store_t *store, const char *project, cbm_gbuf_t *gbuf) {
sqlite3 *db = cbm_store_get_db(store);
if (!db) {
@@ -292,7 +337,13 @@ int64_t cbm_delta_preseed(cbm_store_t *store, const char *project, cbm_gbuf_t *g
* compose `api.get('/p')` against its unchanged wrapper exactly as a full
* build does. Only the two keys, only on the rare Module/Variable rows
* that carry them; every other proxy stays "{}". Proxies are never
* written back (cbm_delta_patch skips id <= max_db_id). */
* written back (cbm_delta_patch skips id <= max_db_id).
*
* The other: a node's parent_class where it changes the name the node
* goes by -- a Go method's QN has no receiver, a class-body variable's QN
* no class. A document naming `Order.Reset` binds such a node through
* its owner (doc_links_pdf.c), so a re-resolved document needs it too.
* Only where owner + name differs from the QN (delta_owner_props). */
sqlite3_stmt *stmt = NULL;
if (sqlite3_prepare_v2(
db,
@@ -301,7 +352,10 @@ int64_t cbm_delta_preseed(cbm_store_t *store, const char *project, cbm_gbuf_t *g
" AND instr(properties, '\"http_client\"') > 0"
" THEN json_object('http_client', json_extract(properties, '$.http_client'),"
" 'http_base_url', json_extract(properties, '$.http_base_url')) END,"
" start_line, end_line"
" start_line, end_line,"
" CASE WHEN label IN ('Method','Variable')"
" AND instr(properties, '\"parent_class\"') > 0"
" THEN json_extract(properties, '$.parent_class') END"
" FROM nodes"
" WHERE project = ?1 AND label NOT IN"
" ('Macro','Comment','Section','Branch','Commit','Tag')"
@@ -324,10 +378,15 @@ int64_t cbm_delta_preseed(cbm_store_t *store, const char *project, cbm_gbuf_t *g
* in a full build. */
int start_line = sqlite3_column_int(stmt, 6);
int end_line = sqlite3_column_int(stmt, 7);
const char *parent = (const char *)sqlite3_column_text(stmt, 8);
char *owner_props =
parent ? delta_owner_props(project, name, qn, parent, client_props) : NULL;
/* Pin the gbuf id to the database id: proxies ARE their rows. */
cbm_gbuf_set_next_id(gbuf, id);
int64_t got = cbm_gbuf_upsert_node(gbuf, label, name, qn, fp ? fp : "", start_line,
end_line, client_props ? client_props : "{}");
int64_t got =
cbm_gbuf_upsert_node(gbuf, label, name, qn, fp ? fp : "", start_line, end_line,
owner_props ? owner_props : (client_props ? client_props : "{}"));
cbm_free(CBM_MEM_CLASS_OTHER, owner_props);
if (got != id) {
/* A QN collision inside the preseed set would silently split
* identity between RAM and disk; the run cannot be trusted. */
tests/repro/repro_call_node_manifest.c
+5 -3
View File
@@ -547,9 +547,11 @@ TEST(repro_call_node_manifest_live_specs_contain_only_expected_primary_entries)
CBMLanguage language = (CBMLanguage)value;
const CBMLangSpec *spec = cbm_lang_spec(language);
/* The registry reproduction owns Nim's stale enum and Studio Export's
* transform-only, intentionally grammar-free classification. */
if (language == CBM_LANG_NIM || language == CBM_LANG_OBJECTSCRIPT_EXPORT) {
/* The registry reproduction owns Nim's stale enum and the
* intentionally grammar-free classifications: Studio Export's
* transform-only one and PDF's DOCUMENT one (doc_pdf.c). */
if (language == CBM_LANG_NIM || language == CBM_LANG_OBJECTSCRIPT_EXPORT ||
language == CBM_LANG_PDF) {
continue;
}
if (!spec || spec->language != language) {
tests/repro/repro_language_registry.c
+16 -7
View File
@@ -18,6 +18,7 @@ typedef enum {
CAP_NO_CALL,
CAP_TRANSFORM_ONLY,
CAP_UNSUPPORTED,
CAP_DOCUMENT,
CAP_COUNT
} LanguageCapability;
@@ -43,6 +44,9 @@ size_t repro_call_argument_matrix_b_copy_language_ids(CBMLanguage *language_ids,
"source is transformed into another registered language first"}
#define UNSUPPORTED(lang) \
[CBM_LANG_##lang] = {CAP_UNSUPPORTED, #lang, "enum retained without a registered grammar"}
#define DOCUMENT(lang) \
[CBM_LANG_##lang] = {CAP_DOCUMENT, #lang, \
"a document format read without a grammar; no code, no calls"}
/* Explicit ledger: no default initializer is intentional. A new enum remains
* CAP_UNSET until its call-metadata/reference-vocabulary behavior is audited. */
@@ -213,6 +217,7 @@ static const LanguageCapabilityEntry LANGUAGE_CAPABILITIES[CBM_LANG_COUNT] = {
CALL_WITH_REFERENCE_VOCAB(ARKTS),
CALL_WITHOUT_REFERENCE_VOCAB(PLSQL),
CALL_WITHOUT_REFERENCE_VOCAB(CHIALISP),
DOCUMENT(PDF),
};
#undef CALL_WITH_REFERENCE_VOCAB
@@ -220,6 +225,7 @@ static const LanguageCapabilityEntry LANGUAGE_CAPABILITIES[CBM_LANG_COUNT] = {
#undef NO_CALL
#undef TRANSFORM_ONLY
#undef UNSUPPORTED
#undef DOCUMENT
_Static_assert(sizeof(LANGUAGE_CAPABILITIES) / sizeof(LANGUAGE_CAPABILITIES[0]) == CBM_LANG_COUNT,
"language capability ledger must track CBM_LANG_COUNT");
@@ -271,7 +277,8 @@ TEST(repro_language_capability_ledger_covers_every_enum) {
}
counts[entry->capability]++;
if (entry->capability == CAP_UNSUPPORTED || entry->capability == CAP_TRANSFORM_ONLY) {
if (entry->capability == CAP_UNSUPPORTED || entry->capability == CAP_TRANSFORM_ONLY ||
entry->capability == CAP_DOCUMENT) {
if (spec) {
fprintf(stderr,
" [language-registry] lang=%s "
@@ -303,12 +310,14 @@ TEST(repro_language_capability_ledger_covers_every_enum) {
if (counts[CAP_CALL_WITH_REFERENCE_VOCAB] != 88 ||
counts[CAP_CALL_WITHOUT_REFERENCE_VOCAB] != 27 || counts[CAP_NO_CALL] != 49 ||
counts[CAP_TRANSFORM_ONLY] != 1 || counts[CAP_UNSUPPORTED] != 1) {
counts[CAP_TRANSFORM_ONLY] != 1 || counts[CAP_UNSUPPORTED] != 1 ||
counts[CAP_DOCUMENT] != 1) {
fprintf(stderr,
" [language-registry] invariant=capability_partition call_ref_vocab=%d ref_gap=%d "
"no_call=%d transform_only=%d unsupported=%d\n",
"no_call=%d transform_only=%d unsupported=%d document=%d\n",
counts[CAP_CALL_WITH_REFERENCE_VOCAB], counts[CAP_CALL_WITHOUT_REFERENCE_VOCAB],
counts[CAP_NO_CALL], counts[CAP_TRANSFORM_ONLY], counts[CAP_UNSUPPORTED]);
counts[CAP_NO_CALL], counts[CAP_TRANSFORM_ONLY], counts[CAP_UNSUPPORTED],
counts[CAP_DOCUMENT]);
failures++;
}
@@ -321,7 +330,7 @@ TEST(repro_call_argument_matrices_equal_call_capability_ledger) {
EXPECTED_MATRIX_A_ROWS = 68,
EXPECTED_MATRIX_B_ROWS = 49,
EXPECTED_CALL_CAPABLE_LANGUAGES = 115,
EXPECTED_NON_CALL_LANGUAGES = 51,
EXPECTED_NON_CALL_LANGUAGES = 52, /* 49 no-call + transform-only + unsupported + PDF */
EXPECTED_NON_CALL_DOMAIN_CONTROLS = 2,
};
CBMLanguage matrix_a_ids[CBM_LANG_COUNT];
@@ -432,11 +441,11 @@ TEST(repro_call_argument_matrices_equal_call_capability_ledger) {
}
if (entry->capability != CAP_NO_CALL && entry->capability != CAP_TRANSFORM_ONLY &&
entry->capability != CAP_UNSUPPORTED) {
entry->capability != CAP_UNSUPPORTED && entry->capability != CAP_DOCUMENT) {
fprintf(stderr,
" [language-registry] lang=%s id=%d "
"invariant=remaining_capability_explicit "
"expected=NO_CALL_or_TRANSFORM_ONLY_or_UNSUPPORTED "
"expected=NO_CALL_or_TRANSFORM_ONLY_or_UNSUPPORTED_or_DOCUMENT "
"actual=%d\n",
language_name, language_id, (int)entry->capability);
failures++;
tests/test_doc_links_pdf.c
+793
View File
@@ -0,0 +1,793 @@
/*
* test_doc_links_pdf.c — PDF documents -> Sections and MENTIONS edges.
*
* The text-layer extractor (internal/cbm/pdf: fonts, encodings, CMaps,
* cross-reference forms, filters, layout, hostile input), the mention scanner
* (doc_pdf.c, through its test seam: expected tokens are the field-test
* scanner's own output for the same text), the resolver (doc_links_pdf.c:
* tiers, owner-qualified Go methods, hygiene, unresolved reasons), the
* snippet of a page, and incremental == full. PDFs are built here, byte by
* byte, with their cross-reference offsets computed.
*/
#include "../src/foundation/compat.h"
#include "test_framework.h"
#include "test_helpers.h"
#include "test_doc_mentions_helpers.h"
#include "cbm.h"
#include "doclink.h"
#include "mcp/mcp.h"
#include "pdf/pdf.h"
#include "pipeline/pipeline.h"
#include "pipeline/pipeline_internal.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <zlib.h>
/* ── building PDFs ───────────────────────────────────────────────── */
typedef struct {
unsigned char *p;
size_t n;
size_t cap;
} tp_buf_t;
static void tp_put(tp_buf_t *b, const void *s, size_t n) {
if (b->n + n + 1 > b->cap) {
size_t cap = b->cap ? b->cap * 2 : 4096;
while (cap < b->n + n + 1) {
cap *= 2;
}
b->p = (unsigned char *)realloc(b->p, cap);
b->cap = cap;
}
memcpy(b->p + b->n, s, n);
b->n += n;
b->p[b->n] = 0;
}
static void tp_puts(tp_buf_t *b, const char *s) {
tp_put(b, s, strlen(s));
}
/* One object's body: a value, or a dict plus stream data. */
typedef struct {
const char *dict; /* the value, or the stream's dict without /Length */
const void *data; /* stream data, or NULL */
size_t len;
} tp_obj_t;
/* A PDF of objects 1..n (objs[i] is object i+1) with a classic xref table;
* `trailer_extra` is added to the trailer dict. Free with free(). */
static unsigned char *tp_pdf(const tp_obj_t *objs, int n, const char *trailer_extra,
size_t *out_len) {
tp_buf_t b = {0};
tp_puts(&b, "%PDF-1.7\n%\xe2\xe3\xcf\xd3\n");
size_t *off = (size_t *)calloc((size_t)n + 1, sizeof(size_t));
char line[256];
for (int i = 0; i < n; i++) {
off[i + 1] = b.n;
snprintf(line, sizeof(line), "%d 0 obj\n", i + 1);
tp_puts(&b, line);
if (objs[i].data) {
snprintf(line, sizeof(line), "<< /Length %zu ", objs[i].len);
tp_puts(&b, line);
tp_puts(&b, objs[i].dict);
tp_puts(&b, " >>\nstream\n");
tp_put(&b, objs[i].data, objs[i].len);
tp_puts(&b, "\nendstream");
} else {
tp_puts(&b, objs[i].dict);
}
tp_puts(&b, "\nendobj\n");
}
size_t xref = b.n;
snprintf(line, sizeof(line), "xref\n0 %d\n0000000000 65535 f \n", n + 1);
tp_puts(&b, line);
for (int i = 1; i <= n; i++) {
snprintf(line, sizeof(line), "%010zu 00000 n \n", off[i]);
tp_puts(&b, line);
}
snprintf(line, sizeof(line),
"trailer\n<< /Size %d /Root 1 0 R %s >>\nstartxref\n%zu\n%%%%EOF\n", n + 1,
trailer_extra ? trailer_extra : "", xref);
tp_puts(&b, line);
free(off);
*out_len = b.n;
return b.p;
}
static unsigned char *tp_flate(const void *src, size_t n, size_t *out_len) {
uLongf cap = compressBound((uLong)n);
unsigned char *out = (unsigned char *)malloc(cap);
compress(out, &cap, (const Bytef *)src, (uLong)n);
*out_len = cap;
return out;
}
#define TP_CATALOG "<< /Type /Catalog /Pages 2 0 R >>"
#define TP_HELV "<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>"
/* A one-page PDF whose content is `content` with font F1 = Helvetica. */
static unsigned char *tp_simple(const char *content, size_t *len) {
tp_obj_t o[5] = {
{TP_CATALOG, NULL, 0},
{"<< /Type /Pages /Kids [3 0 R] /Count 1 >>", NULL, 0},
{"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 4 "
"0 R >> >> /Contents 5 0 R >>",
NULL, 0},
{TP_HELV, NULL, 0},
{"", content, strlen(content)},
};
return tp_pdf(o, 5, NULL, len);
}
typedef struct {
CBMArena arena;
cbm_pdf_result_t r;
} tp_text_t;
static bool tp_extract(const unsigned char *pdf, size_t len, tp_text_t *t) {
cbm_arena_init(&t->arena);
return cbm_pdf_extract(pdf, len, &t->arena, &t->r);
}
/* ── extractor ───────────────────────────────────────────────────── */
TEST(pdf_extract_layout) {
/* inherited resources, two pages, a line break by Td, a word break by TJ
* kerning, a new text object further down */
const char *c1 = "BT /F1 12 Tf 72 700 Td (Hello src/auth/handler.go) Tj 0 -14 Td (second line) "
"Tj ET\nBT /F1 12 Tf 72 600 Td [(word)-1000(gap)] TJ ET";
const char *c2 = "BT /F1 12 Tf 72 700 Td (page two) Tj ET";
tp_obj_t o[7] = {
{TP_CATALOG, NULL, 0},
{"<< /Type /Pages /Kids [3 0 R 6 0 R] /Count 2 /Resources << /Font << /F1 4 0 R >> >> "
"/MediaBox [0 0 612 792] >>",
NULL, 0},
{"<< /Type /Page /Parent 2 0 R /Contents 5 0 R >>", NULL, 0},
{TP_HELV, NULL, 0},
{"", c1, strlen(c1)},
{"<< /Type /Page /Parent 2 0 R /Contents 7 0 R >>", NULL, 0},
{"", c2, strlen(c2)},
};
size_t len;
unsigned char *pdf = tp_pdf(o, 7, NULL, &len);
tp_text_t t;
ASSERT_TRUE(tp_extract(pdf, len, &t));
ASSERT_EQ(t.r.status, CBM_PDF_OK);
ASSERT_EQ(t.r.npages, 2);
ASSERT_STR_EQ(t.r.pages[0].text, "Hello src/auth/handler.go\nsecond line\nword gap");
ASSERT_STR_EQ(t.r.pages[1].text, "page two");
ASSERT_EQ(t.r.pages[0].unmapped, 0);
ASSERT_TRUE(t.r.pages[0].glyphs > 30);
cbm_arena_destroy(&t.arena);
free(pdf);
PASS();
}
TEST(pdf_extract_fonts) {
/* Type0 Identity-H through a ToUnicode CMap (bfchar, a ligature, a
* bfrange, an unmapped code), and a simple font with /Differences */
static const char cmap[] = "/CIDInit /ProcSet findresource begin 12 dict begin begincmap\n"
"1 begincodespacerange <0000> <FFFF> endcodespacerange\n"
"2 beginbfchar <0001> <0041> <0002> <00660069> endbfchar\n"
"1 beginbfrange <0003> <0005> <0061> endbfrange\n"
"endcmap CMapName currentdict /CMap defineresource pop end end\n";
const char *c = "BT /F1 10 Tf 10 700 Td <000100020003000400050006> Tj ET\n"
"BT /F2 10 Tf 10 600 Td (AB) Tj ET";
tp_obj_t o[8] = {
{TP_CATALOG, NULL, 0},
{"<< /Type /Pages /Kids [3 0 R] /Count 1 >>", NULL, 0},
{"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 4 0 "
"R /F2 8 0 R >> >> /Contents 5 0 R >>",
NULL, 0},
{"<< /Type /Font /Subtype /Type0 /BaseFont /ABCDEE+Foo /Encoding /Identity-H "
"/DescendantFonts [6 0 R] /ToUnicode 7 0 R >>",
NULL, 0},
{"", c, strlen(c)},
{"<< /Type /Font /Subtype /CIDFontType2 /BaseFont /Foo /DW 500 /W [1 [600 600] 3 5 "
"700] >>",
NULL, 0},
{"", cmap, sizeof(cmap) - 1},
{"<< /Type /Font /Subtype /Type1 /BaseFont /Times-Roman /Encoding << /BaseEncoding "
"/WinAnsiEncoding /Differences [65 /B /A] >> >>",
NULL, 0},
};
size_t len;
unsigned char *pdf = tp_pdf(o, 8, NULL, &len);
tp_text_t t;
ASSERT_TRUE(tp_extract(pdf, len, &t));
ASSERT_EQ(t.r.status, CBM_PDF_OK);
ASSERT_STR_EQ(t.r.pages[0].text, "Afiabc\xEF\xBF\xBD\nBA");
ASSERT_EQ(t.r.pages[0].unmapped, 1);
cbm_arena_destroy(&t.arena);
free(pdf);
PASS();
}
/* A PDF whose page and font live in a Flate object stream, indexed by a
* PNG-predicted xref stream, with no classic table at all. */
static unsigned char *tp_compressed(size_t *out_len) {
const char *content = "BT /F1 12 Tf 72 700 Td (from an object stream) Tj ET";
size_t clen;
unsigned char *cz = tp_flate(content, strlen(content), &clen);
/* object stream: objects 3 (page) and 4 (font) */
const char *o3 = "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << "
"/F1 4 0 R >> >> /Contents 5 0 R >>";
const char *o4 = TP_HELV;
char hdr[64];
snprintf(hdr, sizeof(hdr), "3 0 4 %zu ", strlen(o3) + 1);
tp_buf_t stm = {0};
tp_puts(&stm, hdr);
size_t first = stm.n;
tp_puts(&stm, o3);
tp_puts(&stm, " ");
tp_puts(&stm, o4);
size_t sz;
unsigned char *stmz = tp_flate(stm.p, stm.n, &sz);
tp_buf_t b = {0};
tp_puts(&b, "%PDF-1.7\n");
size_t off[8] = {0};
char line[256];
off[1] = b.n;
tp_puts(&b, "1 0 obj\n" TP_CATALOG "\nendobj\n");
off[2] = b.n;
tp_puts(&b, "2 0 obj\n<< /Type /Pages /Kids [3 0 R] /Count 1 >>\nendobj\n");
off[5] = b.n;
snprintf(line, sizeof(line), "5 0 obj\n<< /Length %zu /Filter /FlateDecode >>\nstream\n", clen);
tp_puts(&b, line);
tp_put(&b, cz, clen);
tp_puts(&b, "\nendstream\nendobj\n");
off[6] = b.n;
snprintf(line, sizeof(line),
"6 0 obj\n<< /Type /ObjStm /N 2 /First %zu /Length %zu /Filter /FlateDecode "
">>\nstream\n",
first, sz);
tp_puts(&b, line);
tp_put(&b, stmz, sz);
tp_puts(&b, "\nendstream\nendobj\n");
off[7] = b.n;
/* xref stream rows: W [1 2 1], PNG Up/None rows of 4 bytes + filter byte */
unsigned char rows[8][5];
for (int i = 0; i < 8; i++) {
unsigned type = (i == 0) ? 0 : (i == 3 || i == 4) ? 2 : 1;
unsigned f2 = (type == 1) ? (unsigned)off[i] : (type == 2) ? 6U : 0U;
unsigned f3 = (i == 4) ? 1U : 0U;
rows[i][0] = 0; /* PNG filter: None */
rows[i][1] = (unsigned char)type;
rows[i][2] = (unsigned char)(f2 >> 8);
rows[i][3] = (unsigned char)f2;
rows[i][4] = (unsigned char)f3;
}
size_t xz;
unsigned char *xrz = tp_flate(rows, sizeof(rows), &xz);
snprintf(line, sizeof(line),
"7 0 obj\n<< /Type /XRef /Size 8 /W [1 2 1] /Root 1 0 R /Length %zu /Filter "
"/FlateDecode /DecodeParms << /Predictor 12 /Columns 4 >> >>\nstream\n",
xz);
tp_puts(&b, line);
tp_put(&b, xrz, xz);
tp_puts(&b, "\nendstream\nendobj\n");
snprintf(line, sizeof(line), "startxref\n%zu\n%%%%EOF\n", off[7]);
tp_puts(&b, line);
free(cz);
free(stm.p);
free(stmz);
free(xrz);
*out_len = b.n;
return b.p;
}
TEST(pdf_extract_structure) {
size_t len;
unsigned char *pdf = tp_compressed(&len);
tp_text_t t;
ASSERT_TRUE(tp_extract(pdf, len, &t));
ASSERT_EQ(t.r.status, CBM_PDF_OK);
ASSERT_EQ(t.r.npages, 1);
ASSERT_STR_EQ(t.r.pages[0].text, "from an object stream");
ASSERT_EQ(t.r.counts.xref_reconstructed, 0);
cbm_arena_destroy(&t.arena);
free(pdf);
/* a wrong startxref: the xref section is found near it */
unsigned char *ok = tp_simple("BT /F1 12 Tf 72 700 Td (rebuilt) Tj ET", &len);
char *sx = strstr((char *)ok, "startxref\n");
ASSERT_NOT_NULL(sx);
sx[strlen("startxref\n")] = '9';
ASSERT_TRUE(tp_extract(ok, len, &t));
ASSERT_EQ(t.r.status, CBM_PDF_OK);
ASSERT_STR_EQ(t.r.pages[0].text, "rebuilt");
ASSERT_EQ(t.r.counts.xref_offset_fixed, 1);
ASSERT_EQ(t.r.counts.xref_reconstructed, 0);
cbm_arena_destroy(&t.arena);
/* no xref section at all: rebuilt from the object headers, same text */
char *xr = strstr((char *)ok, "xref\n0 ");
ASSERT_NOT_NULL(xr);
memcpy(xr, "XXXX", 4);
ASSERT_TRUE(tp_extract(ok, len, &t));
ASSERT_EQ(t.r.status, CBM_PDF_OK);
ASSERT_STR_EQ(t.r.pages[0].text, "rebuilt");
ASSERT_EQ(t.r.counts.xref_reconstructed, 1);
cbm_arena_destroy(&t.arena);
free(ok);
/* a Form XObject with its own resources, a self-referencing form, an
* inline image with binary data, content split over two streams */
const char *page_a = "q 1 0 0 1 100 500 cm /Fm1 Do Q /Fm2 Do\n";
const char *page_b = "BI /W 2 /H 2 /BPC 8 /CS /G ID \x01\xff(junk)\xfe EI\n"
"BT /F1 12 Tf 72 400 Td (after image) Tj ET";
const char *form = "BT /F1 12 Tf 0 0 Td (in form) Tj ET";
const char *self = "/Fm2 Do BT /F1 12 Tf 0 300 Td (self) Tj ET";
tp_obj_t o[9] = {
{TP_CATALOG, NULL, 0},
{"<< /Type /Pages /Kids [3 0 R] /Count 1 >>", NULL, 0},
{"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /XObject << /Fm1 6 "
"0 R /Fm2 9 0 R >> /Font << /F1 4 0 R >> >> /Contents [5 0 R 7 0 R] >>",
NULL, 0},
{TP_HELV, NULL, 0},
{"", page_a, strlen(page_a)},
{"/Type /XObject /Subtype /Form /BBox [0 0 200 200] /Resources << /Font << /F1 8 0 R "
">> >>",
form, strlen(form)},
{"", page_b, strlen(page_b)},
{TP_HELV, NULL, 0},
{"/Type /XObject /Subtype /Form /BBox [0 0 200 200]", self, strlen(self)},
};
unsigned char *fp = tp_pdf(o, 9, NULL, &len);
ASSERT_TRUE(tp_extract(fp, len, &t));
ASSERT_EQ(t.r.status, CBM_PDF_OK);
ASSERT_NOT_NULL(strstr(t.r.pages[0].text, "in form"));
ASSERT_NOT_NULL(strstr(t.r.pages[0].text, "self"));
ASSERT_NOT_NULL(strstr(t.r.pages[0].text, "after image"));
ASSERT_NULL(strstr(t.r.pages[0].text, "junk"));
cbm_arena_destroy(&t.arena);
free(fp);
PASS();
}
static void tp_ahx(tp_buf_t *b, const char *s) {
static const char hex[] = "0123456789abcdef";
for (; *s; s++) {
char h[3] = {hex[(unsigned char)*s >> 4], hex[(unsigned char)*s & 15], ' '};
tp_put(b, h, 3);
}
tp_puts(b, ">");
}
static void tp_a85(tp_buf_t *b, const unsigned char *s, size_t n) {
tp_puts(b, "<~");
for (size_t i = 0; i < n; i += 4) {
unsigned char g[4] = {0, 0, 0, 0};
size_t k = n - i < 4 ? n - i : 4;
memcpy(g, s + i, k);
uint32_t v = ((uint32_t)g[0] << 24) | ((uint32_t)g[1] << 16) | ((uint32_t)g[2] << 8) | g[3];
char out[5];
for (int j = 4; j >= 0; j--) {
out[j] = (char)('!' + v % 85);
v /= 85;
}
tp_put(b, out, k + 1);
}
tp_puts(b, "~>");
}
TEST(pdf_extract_filters_and_hostile) {
size_t len;
tp_text_t t;
const char *content = "BT /F1 12 Tf 72 700 Td (filtered text) Tj ET";
/* ASCIIHex, then ASCII85 over Flate */
tp_buf_t h = {0};
tp_ahx(&h, content);
size_t zl;
unsigned char *z = tp_flate(content, strlen(content), &zl);
tp_buf_t a = {0};
tp_a85(&a, z, zl);
const char *dicts[2] = {"/Filter /ASCIIHexDecode", "/Filter [/ASCII85Decode /FlateDecode]"};
const tp_buf_t *datas[2] = {&h, &a};
for (int i = 0; i < 2; i++) {
tp_obj_t o[5] = {
{TP_CATALOG, NULL, 0},
{"<< /Type /Pages /Kids [3 0 R] /Count 1 >>", NULL, 0},
{"<< /Type /Page /Parent 2 0 R /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 "
"R >>",
NULL, 0},
{TP_HELV, NULL, 0},
{dicts[i], datas[i]->p, datas[i]->n},
};
unsigned char *pdf = tp_pdf(o, 5, NULL, &len);
ASSERT_TRUE(tp_extract(pdf, len, &t));
ASSERT_EQ(t.r.status, CBM_PDF_OK);
ASSERT_STR_EQ(t.r.pages[0].text, "filtered text");
cbm_arena_destroy(&t.arena);
free(pdf);
}
free(h.p);
free(a.p);
free(z);
/* not a PDF; encrypted; truncated anywhere; nested a hundred thousand deep */
ASSERT_TRUE(tp_extract((const unsigned char *)"hello", 5, &t));
ASSERT_EQ(t.r.status, CBM_PDF_NOT_PDF);
cbm_arena_destroy(&t.arena);
tp_obj_t enc[5] = {
{TP_CATALOG, NULL, 0},
{"<< /Type /Pages /Kids [3 0 R] /Count 1 >>", NULL, 0},
{"<< /Type /Page /Parent 2 0 R /Contents 5 0 R >>", NULL, 0},
{"<< /Filter /Standard /V 1 /R 2 /O <00> /U <00> /P -4 >>", NULL, 0},
{"", "BT ET", 5},
};
unsigned char *pdf = tp_pdf(enc, 5, "/Encrypt 4 0 R", &len);
ASSERT_TRUE(tp_extract(pdf, len, &t));
ASSERT_EQ(t.r.status, CBM_PDF_ENCRYPTED);
ASSERT_EQ(t.r.npages, 0);
cbm_arena_destroy(&t.arena);
free(pdf);
pdf = tp_simple("BT /F1 12 Tf 72 700 Td (cut short) Tj ET", &len);
for (size_t cut = 0; cut < len; cut += 7) {
ASSERT_TRUE(tp_extract(pdf, cut, &t)); /* any status; never a crash */
cbm_arena_destroy(&t.arena);
}
free(pdf);
enum { DEEP = 100000 };
char *deep = (char *)malloc(DEEP * 2 + 64);
memset(deep, '[', DEEP);
strcpy(deep + DEEP, " BT /F1 12 Tf (x) Tj ET");
pdf = tp_simple(deep, &len);
ASSERT_TRUE(tp_extract(pdf, len, &t));
ASSERT_EQ(t.r.status, CBM_PDF_OK);
ASSERT_EQ(t.r.counts.page_errors, 1); /* the content stops at the nesting limit */
cbm_arena_destroy(&t.arena);
free(pdf);
free(deep);
/* a page tree that contains itself, a /Prev loop */
tp_obj_t loop[5] = {
{TP_CATALOG, NULL, 0},
{"<< /Type /Pages /Kids [2 0 R 3 0 R] /Count 1 >>", NULL, 0},
{"<< /Type /Page /Parent 2 0 R /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R "
">>",
NULL, 0},
{TP_HELV, NULL, 0},
{"", "BT /F1 12 Tf (loops) Tj ET", 26},
};
pdf = tp_pdf(loop, 5, "/Prev 9", &len);
char *sx = strstr((char *)pdf, "startxref\n");
char prev[32];
snprintf(prev, sizeof(prev), "/Prev %ld", strtol(sx + strlen("startxref\n"), NULL, 10));
char *pv = strstr((char *)pdf, "/Prev 9");
ASSERT_NOT_NULL(pv);
/* the same width: point /Prev at this very section (a loop) */
if (strlen(prev) <= strlen("/Prev 9") + 6) {
tp_buf_t nb = {0};
tp_put(&nb, pdf, (size_t)(pv - (char *)pdf));
tp_puts(&nb, prev);
tp_puts(&nb, pv + strlen("/Prev 9"));
free(pdf);
pdf = nb.p;
len = nb.n;
}
ASSERT_TRUE(tp_extract(pdf, len, &t));
ASSERT_EQ(t.r.status, CBM_PDF_OK);
ASSERT_EQ(t.r.npages, 1);
ASSERT_STR_EQ(t.r.pages[0].text, "loops");
cbm_arena_destroy(&t.arena);
free(pdf);
PASS();
}
/* ── scanner (expected: the field-test scanner on the same text) ─── */
TEST(pdf_scan_mentions) {
const char *page = "See src/auth/handler.go and ./pkg/x.py, http://example.com/a/b.py and "
"and/or 2024/01/02.\n"
"Call pkg.Config.Name(), Handler::run and obj->field, e.g. x.y and "
"example.com.\n"
"The file config.py and setup.cfg; f(a, opts.debug=True).\n"
"long/path/to/mod-\n"
"ule.py and Handler::\n"
"serve here\n"
"\xEF\xAC\x81le_utils.py ligature\n";
static const struct {
uint32_t line;
int syntax;
uint16_t flags;
const char *raw;
} want[] = {
{1, CBM_DOCLINK_PDF_PATH, 0, "src/auth/handler.go"},
{1, CBM_DOCLINK_PDF_PATH, 0, "pkg/x.py"},
{2, CBM_DOCLINK_PDF_QN, 0, "pkg.Config.Name"},
{2, CBM_DOCLINK_PDF_QN, 0, "Handler.run"},
{2, CBM_DOCLINK_PDF_QN, 0, "obj.field"},
{3, CBM_DOCLINK_PDF_FILE, 0, "config.py"},
{3, CBM_DOCLINK_PDF_FILE, 0, "setup.cfg"},
{4, CBM_DOCLINK_PDF_PATH, 0, "long/path/to/mod"},
{5, CBM_DOCLINK_PDF_FILE, 0, "ule.py"},
{7, CBM_DOCLINK_PDF_FILE, 0, "file_utils.py"},
{4, CBM_DOCLINK_PDF_PATH, CBM_DOCLINK_FLAG_JOIN,
"long/path/to/module.py\x1flong/path/to/mod-ule.py\x1e"
"7,8"},
{5, CBM_DOCLINK_PDF_QN, CBM_DOCLINK_FLAG_JOIN, "Handler.serve\x1e"},
};
CBMFileResult *r = (CBMFileResult *)calloc(1, sizeof(*r));
cbm_arena_init(&r->arena);
CBMExtractCtx ctx;
memset(&ctx, 0, sizeof(ctx));
ctx.arena = &r->arena;
ctx.result = r;
cbm_pdf_test_scan_page(&ctx, page, strlen(page), 1, "p.doc.page_1");
ASSERT_FALSE(r->doc_links.failed);
ASSERT_EQ(r->doc_links.count, (int)(sizeof(want) / sizeof(want[0])));
for (int i = 0; i < r->doc_links.count; i++) {
const CBMDocLink *l = &r->doc_links.items[i];
ASSERT_EQ(l->line, want[i].line);
ASSERT_EQ(l->syntax, want[i].syntax);
ASSERT_EQ(l->flags, want[i].flags);
ASSERT_STR_EQ(l->raw, want[i].raw);
ASSERT_EQ(l->def_line, 1);
ASSERT_STR_EQ(l->source_qn, "p.doc.page_1");
}
cbm_free_result(r);
PASS();
}
/* ── the pipeline ────────────────────────────────────────────────── */
/* A PDF whose pages each show the given lines (Helvetica, one Tj a line). */
static void tp_write_pdf(const char *path, const char *const *pages, int npages) {
int n = 3 + 2 * npages;
tp_obj_t *o = (tp_obj_t *)calloc((size_t)n, sizeof(*o));
char **bufs = (char **)calloc((size_t)npages * 2 + 1, sizeof(char *));
o[0].dict = TP_CATALOG;
tp_buf_t kids = {0};
tp_puts(&kids, "<< /Type /Pages /Count 1 /Kids [");
for (int i = 0; i < npages; i++) {
char k[32];
snprintf(k, sizeof(k), "%d 0 R ", 4 + 2 * i);
tp_puts(&kids, k);
}
tp_puts(&kids, "] >>");
o[1].dict = (const char *)kids.p;
o[2].dict = TP_HELV;
for (int i = 0; i < npages; i++) {
char *pg = (char *)malloc(256);
snprintf(pg, 256,
"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 "
"3 0 R >> >> /Contents %d 0 R >>",
5 + 2 * i);
bufs[2 * i] = pg;
o[3 + 2 * i].dict = pg;
tp_buf_t c = {0};
tp_puts(&c, "BT /F1 10 Tf 72 720 Td 12 TL\n");
for (const char *s = pages[i]; *s;) {
const char *e = strchr(s, '\n');
size_t k = e ? (size_t)(e - s) : strlen(s);
tp_puts(&c, "(");
tp_put(&c, s, k);
tp_puts(&c, ") Tj T*\n");
s += k + (e ? 1 : 0);
}
tp_puts(&c, "ET");
bufs[2 * i + 1] = (char *)c.p;
o[4 + 2 * i].dict = "";
o[4 + 2 * i].data = c.p;
o[4 + 2 * i].len = c.n;
}
size_t len;
unsigned char *pdf = tp_pdf(o, n, NULL, &len);
char dir[512];
snprintf(dir, sizeof(dir), "%s", path);
char *slash = strrchr(dir, '/');
if (slash) {
*slash = '\0';
cbm_mkdir_p(dir, 0755);
}
FILE *f = fopen(path, "wb");
fwrite(pdf, 1, len, f);
fclose(f);
free(pdf);
free(kids.p);
for (int i = 0; i < npages * 2; i++) {
free(bufs[i]);
}
free(bufs);
free(o);
}
static const char *const DESIGN_P1 =
"Design: pkg/server/handler.go serves via server.Handler.Serve\n"
"and app.models.User.save stores users; see\n"
"tests/test_api.py and config/settings.json.\n"
"Gone: pkg/server/gone.go and other/thing.go.";
static const char *const DESIGN_P2 =
"Factory server.NewHandler and models.py build it; app.models holds it.";
static void tp_write_repo(const char *repo, const char *p2) {
th_write_file(TH_PATH(repo, "pkg/server/handler.go"),
"package server\n"
"\n"
"type Handler struct{}\n"
"\n"
"func (h *Handler) Serve() {}\n"
"\n"
"func NewHandler() *Handler { return nil }\n");
th_write_file(TH_PATH(repo, "app/__init__.py"), "");
th_write_file(TH_PATH(repo, "app/models.py"), "class User:\n"
" def save(self):\n"
" return None\n");
th_write_file(TH_PATH(repo, "tests/test_api.py"), "def test_x():\n"
" return None\n");
th_write_file(TH_PATH(repo, "config/settings.json"), "{\"a\": 1}\n");
char path[1024];
snprintf(path, sizeof(path), "%s/docs/design.pdf", repo);
const char *pages[2] = {DESIGN_P1, p2};
tp_write_pdf(path, pages, 2);
snprintf(path, sizeof(path), "%s/tests/fixtures/sample.pdf", repo);
const char *fx[1] = {"Fixture names pkg/server/handler.go too."};
tp_write_pdf(path, fx, 1);
}
TEST(pdf_links_pipeline) {
char tmp[256];
snprintf(tmp, sizeof(tmp), "/tmp/cbm_dlpdf_XXXXXX");
ASSERT_NOT_NULL(cbm_mkdtemp(tmp));
char repo[512];
char db[512];
snprintf(repo, sizeof(repo), "%s/repo", tmp);
snprintf(db, sizeof(db), "%s/g.db", tmp);
tp_write_repo(repo, DESIGN_P2);
ASSERT_EQ(dm_index(repo, db, NULL), 0);
/* one Section per page, the page's text as its docstring */
ASSERT_EQ(dm_count(db, "SELECT COUNT(*) FROM nodes WHERE label='Section' AND "
"file_path='docs/design.pdf'"),
2);
ASSERT_EQ(dm_count(db, "SELECT COUNT(*) FROM nodes WHERE label='Section' AND name='page 1' "
"AND file_path='docs/design.pdf' AND properties LIKE "
"'%pkg/server/handler.go serves%'"),
1);
char props[512];
int n;
/* path-exact */
dm_edge(db, "docs.design.page_1", "pkg.server.handler.go.__file__", props, sizeof(props), &n);
ASSERT_EQ(n, 1);
ASSERT_NOT_NULL(strstr(props, "\"via\":\"pdf\""));
ASSERT_NOT_NULL(strstr(props, "\"syntax\":\"pdf_path\""));
ASSERT_NOT_NULL(strstr(props, "\"tier\":\"exact\""));
ASSERT_NOT_NULL(strstr(props, "\"line\":1"));
/* a Go method through its receiver (its QN has none) */
dm_edge(db, "docs.design.page_1", "pkg.server.Serve", props, sizeof(props), &n);
ASSERT_EQ(n, 1);
ASSERT_NOT_NULL(strstr(props, "\"tier\":\"unique\""));
/* qn-exact */
dm_edge(db, "docs.design.page_1", "app.models.User.save", props, sizeof(props), &n);
ASSERT_EQ(n, 1);
ASSERT_NOT_NULL(strstr(props, "\"tier\":\"exact\""));
ASSERT_NOT_NULL(strstr(props, "\"line\":2"));
/* page 2: a Go function, a file name */
dm_edge(db, "docs.design.page_2", "pkg.server.NewHandler", props, sizeof(props), &n);
ASSERT_EQ(n, 1);
dm_edge(db, "docs.design.page_2", "app.models.py.__file__", props, sizeof(props), &n);
ASSERT_EQ(n, 1);
ASSERT_NOT_NULL(strstr(props, "\"syntax\":\"pdf_file\""));
/* a module by its qualified name (a Module node is named by its path) */
ASSERT_EQ(dm_count(db, "SELECT COUNT(*) FROM edges e JOIN nodes s ON s.id=e.source_id JOIN "
"nodes t ON t.id=e.target_id WHERE e.type='MENTIONS' AND "
"t.label='Module' AND t.file_path='app/models.py' AND s.name='page 2' "
"AND e.properties LIKE '%\"tier\":\"exact\"%'"),
1);
/* hygiene: a test target is a row, a data file nothing; a fixture PDF links nothing */
char reason[64];
dm_row(db, "docs/design.pdf", "tests/test_api.py", reason, sizeof(reason), NULL, 0);
ASSERT_STR_EQ(reason, "test_only_target");
ASSERT_EQ(dm_count(db, "SELECT COUNT(*) FROM edges e JOIN nodes t ON t.id=e.target_id WHERE "
"e.type='MENTIONS' AND t.file_path='config/settings.json'"),
0);
ASSERT_EQ(dm_count(db, "SELECT COUNT(*) FROM doc_link_unresolved WHERE "
"raw='config/settings.json'"),
0);
ASSERT_EQ(dm_count(db, "SELECT COUNT(*) FROM edges e JOIN nodes s ON s.id=e.source_id WHERE "
"e.type='MENTIONS' AND s.file_path='tests/fixtures/sample.pdf'"),
0);
/* unresolved: a missing file under a known directory; another repo's path is no row */
dm_row(db, "docs/design.pdf", "pkg/server/gone.go", reason, sizeof(reason), NULL, 0);
ASSERT_STR_EQ(reason, "missing");
ASSERT_EQ(dm_count(db, "SELECT COUNT(*) FROM doc_link_unresolved WHERE raw='other/thing.go'"),
0);
/* back navigation: the method knows the page that names it */
ASSERT_EQ(dm_count(db, "SELECT COUNT(*) FROM edges e JOIN nodes s ON s.id=e.source_id JOIN "
"nodes t ON t.id=e.target_id WHERE e.type='MENTIONS' AND t.name='Serve' "
"AND s.name='page 1'"),
1);
th_cleanup(tmp);
PASS();
}
TEST(pdf_snippet_is_page_text) {
char tmp[256] = "/tmp/cbm_dlpdf_snip_XXXXXX";
ASSERT_NOT_NULL(cbm_mkdtemp(tmp));
char repo[400];
char cache[400];
char db[1024];
snprintf(repo, sizeof(repo), "%s/repo", tmp);
snprintf(cache, sizeof(cache), "%s/cache", tmp);
tp_write_repo(repo, DESIGN_P2);
cbm_mkdir_p(cache, 0700);
char *project = cbm_project_name_from_path(repo);
ASSERT_NOT_NULL(project);
const char *saved = getenv("CBM_CACHE_DIR");
char *saved_copy = saved ? strdup(saved) : NULL;
snprintf(db, sizeof(db), "%s/%s.db", cache, project);
cbm_setenv("CBM_CACHE_DIR", cache, 1);
ASSERT_EQ(dm_index(repo, db, NULL), 0);
cbm_mcp_server_t *srv = cbm_mcp_server_new(NULL);
ASSERT_NOT_NULL(srv);
char args[1024];
snprintf(args, sizeof(args),
"{\"project\":\"%s\",\"qualified_name\":\"%s.docs.design.page_2\"}", project, project);
char *resp = cbm_mcp_handle_tool(srv, "get_code_snippet", args);
char *text = dm_tool_text(resp);
ASSERT_NOT_NULL(text);
ASSERT_NOT_NULL(
strstr(text, "Factory server.NewHandler and models.py build it; app.models holds it."));
ASSERT_NULL(strstr(text, "%PDF"));
free(text);
free(resp);
cbm_mcp_server_free(srv);
if (saved_copy) {
cbm_setenv("CBM_CACHE_DIR", saved_copy, 1);
} else {
cbm_unsetenv("CBM_CACHE_DIR");
}
free(saved_copy);
free(project);
th_cleanup(tmp);
PASS();
}
TEST(pdf_links_incremental) {
char tmp[256];
snprintf(tmp, sizeof(tmp), "/tmp/cbm_dlpdf_inc_XXXXXX");
ASSERT_NOT_NULL(cbm_mkdtemp(tmp));
char repo[512];
char db[512];
char full_db[512];
snprintf(repo, sizeof(repo), "%s/repo", tmp);
snprintf(db, sizeof(db), "%s/inc.db", tmp);
snprintf(full_db, sizeof(full_db), "%s/full.db", tmp);
tp_write_repo(repo, DESIGN_P2);
ASSERT_EQ(dm_index(repo, db, NULL), 0);
/* a body edit of a linked code file: the document re-resolves */
th_write_file(TH_PATH(repo, "app/models.py"), "class User:\n"
" def save(self):\n"
" return 1\n");
ASSERT_EQ(dm_step(repo, db, full_db, "code body edit", CBM_INCREMENTAL_ROUTE_CLOSURE_REPAIR),
0);
/* the document changes while the Go file does not: its method binds
* through the owner a proxy carries */
char path[1024];
snprintf(path, sizeof(path), "%s/docs/design.pdf", repo);
const char *pages[2] = {DESIGN_P1, "Now only server.Handler.Serve and pkg/server/new.go."};
tp_write_pdf(path, pages, 2);
ASSERT_EQ(dm_step(repo, db, full_db, "document edit", CBM_INCREMENTAL_ROUTE_CLOSURE_REPAIR), 0);
/* the missing file appears: the row becomes an edge, as in a full index */
th_write_file(TH_PATH(repo, "pkg/server/gone.go"), "package server\n"
"\n"
"func Gone() {}\n");
ASSERT_EQ(dm_step(repo, db, full_db, "missing file added", CBM_INCREMENTAL_ROUTE_FORCED_FULL),
0);
th_cleanup(tmp);
PASS();
}
SUITE(doc_links_pdf) {
RUN_TEST(pdf_extract_layout);
RUN_TEST(pdf_extract_fonts);
RUN_TEST(pdf_extract_structure);
RUN_TEST(pdf_extract_filters_and_hostile);
RUN_TEST(pdf_scan_mentions);
RUN_TEST(pdf_links_pipeline);
RUN_TEST(pdf_snippet_is_page_text);
RUN_TEST(pdf_links_incremental);
}
tests/test_main.c
+2
View File
@@ -1822,6 +1822,7 @@ extern void suite_doc_mentions(void);
extern void suite_doc_mentions_cost(void);
extern void suite_doc_mentions_msbuild(void);
extern void suite_doc_links_md(void);
extern void suite_doc_links_pdf(void);
extern void suite_infrascan(void);
extern void suite_cli(void);
extern void suite_agent_clients(void);
@@ -2290,6 +2291,7 @@ int main(int argc, char **argv) {
RUN_SELECTED_SUITE(doc_mentions_cost);
RUN_SELECTED_SUITE(doc_mentions_msbuild);
RUN_SELECTED_SUITE(doc_links_md);
RUN_SELECTED_SUITE(doc_links_pdf);
/* Infrastructure scanning */
RUN_SELECTED_SUITE(infrascan);