doc-links(pdf): notices for the PDF tables; the Unicode table generator

The PDF text layer carries data tables (D3). Their sources and notices are now in THIRD_PARTY.md (shipped in every archive through THIRD_PARTY_NOTICES.md) and in the SBOM: - the Adobe Glyph List subset (BSD-3-Clause, Copyright 2002-2019 Adobe), - the printable-ASCII advance widths of Helvetica and Times-Roman from Adobe's core-14 font metrics (the AFM files are not distributed), - the Unicode Character Database data of pdf_unicode.c (Unicode License v3, full text included). scripts/gen-pdf-unicode.py (stdlib CPython only) regenerates pdf_unicode.c byte for byte with CPython 3.14.6 (Unicode 16.0.0); pdf_tables.c is maintained data from here on. License texts were taken from their sources (adobe-type-tools/agl-aglfn, unicode.org). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-04 10:48:20 +00:00 · 2026-10-09 02:55:23 +02:00
1 parent 92f5edfe32
commit c1da1924b2
@@ -138,6 +138,109 @@ Semantic vector search uses static token embeddings derived from the
 See `vendored/nomic/NOTICE` for the exact derivation procedure
 (per-token inference + int8 quantization via `scripts/extract_nomic_vectors.py`).

+## Embedded Character Data (PDF text layer)
+
+The PDF text-layer extractor (`internal/cbm/pdf/`) carries static tables
+derived from the following sources. No upstream source code is included.
+
+- **Adobe Glyph List** — the glyph-name subset in `internal/cbm/pdf/pdf_tables.c`
+  (the names of the standard Latin encodings and common symbols, mapped to
+  Unicode) follows [adobe-type-tools/agl-aglfn](https://github.com/adobe-type-tools/agl-aglfn)
+  `glyphlist.txt`.
+  - **License:** BSD-3-Clause
+  - **Copyright:** Copyright 2002-2019 Adobe (http://www.adobe.com/).
+- **Adobe core-14 font metrics** — the advance widths of the printable ASCII
+  range of Helvetica and Times-Roman in `pdf_tables.c` (used only for a
+  standard-14 font that omits its `/Widths`) are the values of Adobe's
+  core-14 AFM files. The AFM files themselves are not distributed.
+  - **Copyright:** Copyright (c) 1985, 1987, 1989, 1990, 1997 Adobe Systems
+    Incorporated. All Rights Reserved.
+- **Unicode Character Database** — `internal/cbm/pdf/pdf_unicode.c` (NFKC of
+  each code point, canonical compositions, character classes) is generated
+  by `scripts/gen-pdf-unicode.py` from CPython's `unicodedata` (Unicode
+  16.0.0).
+  - **License:** Unicode License v3 (text below)
+  - **Copyright:** Copyright © 1991-2026 Unicode, Inc.
+- The base encodings (Standard, WinAnsi, MacRoman, Symbol) are those of
+  ISO 32000-1, Annex D.
+
+Adobe Glyph List license (BSD-3-Clause):
+
+```
+Copyright 2002-2019 Adobe (http://www.adobe.com/).
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+Redistributions of source code must retain the above copyright notice,
+this list of conditions and the following disclaimer.
+
+Redistributions in binary form must reproduce the above copyright
+notice, this list of conditions and the following disclaimer in the
+documentation and/or other materials provided with the distribution.
+
+Neither the name of Adobe nor the names of its contributors may be
+used to endorse or promote products derived from this software without
+specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+```
+
+Unicode License v3:
+
+```
+UNICODE LICENSE V3
+
+COPYRIGHT AND PERMISSION NOTICE
+
+Copyright © 1991-2026 Unicode, Inc.
+
+NOTICE TO USER: Carefully read the following legal agreement. BY
+DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR
+SOFTWARE, YOU UNEQUIVOCALLY ACCEPT, AND AGREE TO BE BOUND BY, ALL OF THE
+TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE, DO NOT
+DOWNLOAD, INSTALL, COPY, DISTRIBUTE OR USE THE DATA FILES OR SOFTWARE.
+
+Permission is hereby granted, free of charge, to any person obtaining a
+copy of data files and any associated documentation (the "Data Files") or
+software and any associated documentation (the "Software") to deal in the
+Data Files or Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, and/or sell
+copies of the Data Files or Software, and to permit persons to whom the
+Data Files or Software are furnished to do so, provided that either (a)
+this copyright and permission notice appear with all copies of the Data
+Files or Software, or (b) this copyright and permission notice appear in
+associated Documentation.
+
+THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
+KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
+THIRD PARTY RIGHTS.
+
+IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE
+BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES,
+OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
+WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,
+ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA
+FILES OR SOFTWARE.
+
+Except as contained in this notice, the name of a copyright holder shall
+not be used in advertising or otherwise to promote the sale, use or other
+dealings in these Data Files or Software without prior written
+authorization of the copyright holder.
+```
+
 ## Hybrid LSP — Reference Language Servers

 The Hybrid LSP layer (`internal/cbm/lsp/`) is an original C implementation
@@ -4,8 +4,13 @@
 * Glyph names (the Adobe Glyph List subset every standard Latin encoding
 * uses, common TeX and symbol names, and the accented Latin letters),
 * the base encodings of ISO 32000-1 Annex D as code -> UTF-8, and
- * approximate standard-14 widths. Generated from the field-tested tables;
- * do not edit by hand.
+ * approximate standard-14 widths. Generated once from the field-tested
+ * prototype's tables; maintained data from here on (edit by hand, keep the
+ * sources). Sources and notices (THIRD_PARTY.md): the Adobe Glyph List,
+ * Copyright 2002-2019 Adobe, BSD-3-Clause; the printable-ASCII advance
+ * widths of Helvetica and Times-Roman from Adobe's core-14 font metrics,
+ * Copyright (c) 1985, 1987, 1989, 1990, 1997 Adobe Systems Incorporated;
+ * ISO 32000-1 Annex D for the encodings.
 */
 #include "pdf_internal.h"

@@ -5,7 +5,9 @@
 * CPython 3.14.6 answers: NFKC of each code point, the canonical compositions,
 * str.isspace, the re module's \w (str.isalnum or '_'), str.isalpha, and the
 * one-character str.isupper / str.islower / titlecase. Generated by
- * gen_pdf_unicode.py; do not edit by hand.
+ * scripts/gen-pdf-unicode.py; do not edit by hand. Data of the Unicode
+ * Character Database, (c) Unicode, Inc., Unicode License v3
+ * (THIRD_PARTY.md).
 */
 #include "pdf_unicode.h"

@@ -49,6 +49,9 @@ sbom = {
        {"SPDXID": "SPDXRef-Package-simplecpp", "name": "simplecpp", "versionInfo": "1.x", "licenseDeclared": "0BSD", "downloadLocation": "https://github.com/danmar/simplecpp", "filesAnalyzed": False},
        {"SPDXID": "SPDXRef-Package-verstable", "name": "verstable", "versionInfo": "2.2.1", "licenseDeclared": "MIT", "downloadLocation": "https://github.com/JacksonAllan/Verstable", "filesAnalyzed": False},
        {"SPDXID": "SPDXRef-Package-wyhash", "name": "wyhash", "versionInfo": "final-4.3", "licenseDeclared": "Unlicense", "downloadLocation": "https://github.com/wangyi-fudan/wyhash", "filesAnalyzed": False},
+        {"SPDXID": "SPDXRef-Package-adobe-glyph-list", "name": "adobe-glyph-list", "versionInfo": "2.0", "licenseDeclared": "BSD-3-Clause", "downloadLocation": "https://github.com/adobe-type-tools/agl-aglfn", "filesAnalyzed": False, "comment": "Glyph-name subset in internal/cbm/pdf/pdf_tables.c (PDF text layer); see THIRD_PARTY.md"},
+        {"SPDXID": "SPDXRef-Package-adobe-core14-metrics", "name": "adobe-core14-afm-metrics", "versionInfo": "1997", "licenseDeclared": "NOASSERTION", "downloadLocation": "NOASSERTION", "filesAnalyzed": False, "comment": "Printable-ASCII advance widths of Helvetica and Times-Roman in internal/cbm/pdf/pdf_tables.c; (c) 1985-1997 Adobe Systems Incorporated; the AFM files are not distributed; see THIRD_PARTY.md"},
+        {"SPDXID": "SPDXRef-Package-unicode-ucd", "name": "unicode-character-database", "versionInfo": "16.0.0", "licenseDeclared": "Unicode-3.0", "downloadLocation": "https://www.unicode.org/Public/16.0.0/ucd/", "filesAnalyzed": False, "comment": "NFKC, compositions and character classes in internal/cbm/pdf/pdf_unicode.c, generated by scripts/gen-pdf-unicode.py; see THIRD_PARTY.md"},
        {"SPDXID": "SPDXRef-Package-nomic-embed-code", "name": "nomic-embed-code-token-embeddings", "versionInfo": "1.0", "licenseDeclared": "Apache-2.0", "downloadLocation": "https://huggingface.co/nomic-ai/nomic-embed-code", "filesAnalyzed": False, "comment": "Derived int8 token embeddings; see vendored/nomic/NOTICE"},
        {"SPDXID": "SPDXRef-Package-tree-sitter-grammars", "name": "tree-sitter-grammars-aggregate", "versionInfo": f"{n_grammars}-grammars", "licenseDeclared": "MIT", "downloadLocation": "NOASSERTION", "filesAnalyzed": False, "comment": f"Aggregate of {n_grammars} vendored tree-sitter grammars, compiled statically. Predominantly MIT; non-MIT families present include CC0-1.0 (clojure, fennel), Apache-2.0 (elixir, erlang, gleam, hcl, ini, jinja2, just, pkl, sway, wit), ISC (pine, templ), Unlicense (fish); first-party grammars (c) DeusData, MIT. This summary is non-exhaustive; the authoritative complete per-grammar license list is internal/cbm/vendored/grammars/MANIFEST.md, and full license texts ship in THIRD_PARTY_NOTICES.md inside each archive."},
    ],
@@ -0,0 +1,103 @@
+#!/usr/bin/env python3
+"""Generate internal/cbm/pdf/pdf_unicode.c: the Unicode facts the PDF mention
+scanner needs, exactly as CPython sees them (the field test's scanner ran on
+CPython's str and re).
+
+usage: scripts/gen-pdf-unicode.py internal/cbm/pdf/pdf_unicode.c  (then clang-format -i it)
+The output depends on the CPython version (its Unicode data): the committed
+tables were made with CPython 3.14.6 (Unicode 16.0.0).
+"""
+import sys
+import unicodedata
+
+MAXCP = 0x110000
+
+
+def is_surrogate(c):
+    return 0xD800 <= c <= 0xDFFF
+
+
+def ranges(pred):
+    out = []
+    start = None
+    for c in range(MAXCP):
+        ok = not is_surrogate(c) and pred(chr(c))
+        if ok and start is None:
+            start = c
+        elif not ok and start is not None:
+            out.append((start, c - 1))
+            start = None
+    if start is not None:
+        out.append((start, MAXCP - 1))
+    return out
+
+
+def emit_ranges(f, name, rs):
+    f.write("const pdf_urange_t %s[] = {\n" % name)
+    for i in range(0, len(rs), 4):
+        f.write("    " + " ".join("{0x%X, 0x%X}," % r for r in rs[i:i + 4]) + "\n")
+    f.write("};\nconst int %s_COUNT = (int)(sizeof(%s) / sizeof(%s[0]));\n\n" % (name, name, name))
+
+
+def main():
+    out = sys.argv[1]
+    # per-character NFKC
+    nfkc = []
+    seq = []
+    for c in range(MAXCP):
+        if is_surrogate(c):
+            continue
+        ch = chr(c)
+        m = unicodedata.normalize("NFKC", ch)
+        if m != ch:
+            nfkc.append((c, len(seq), len(m)))
+            seq.extend(ord(x) for x in m)
+    # canonical composition pairs: a + b -> c when NFC composes them
+    comp = []
+    for c in range(MAXCP):
+        if is_surrogate(c):
+            continue
+        d = unicodedata.decomposition(chr(c))
+        if not d or d.startswith("<"):
+            continue
+        parts = [int(x, 16) for x in d.split()]
+        if len(parts) != 2:
+            continue
+        a, b = parts
+        if unicodedata.normalize("NFC", chr(a) + chr(b)) == chr(c):
+            comp.append((a, b, c))
+    comp.sort()
+    with open(out, "w", encoding="utf-8") as f:
+        f.write("/*\n * pdf_unicode.c -- Unicode %s facts for the PDF mention scanner (generated).\n"
+                " *\n * The field test's scanner ran on CPython's str and re; these tables are what\n"
+                " * CPython %s answers: NFKC of each code point, the canonical compositions,\n"
+                " * str.isspace, the re module's \\w (str.isalnum or '_'), str.isalpha, and the\n"
+                " * one-character str.isupper / str.islower / titlecase. Generated by\n"
+                " * scripts/gen-pdf-unicode.py; do not edit by hand. Data of the Unicode\n"
+                " * Character Database, (c) Unicode, Inc., Unicode License v3\n"
+                " * (THIRD_PARTY.md).\n */\n"
+                % (unicodedata.unidata_version, sys.version.split()[0]))
+        f.write("#include \"pdf_unicode.h\"\n\n")
+        f.write("const pdf_nfkc_t PDF_NFKC[] = {\n")
+        for i in range(0, len(nfkc), 4):
+            f.write("    " + " ".join("{0x%X, %d, %d}," % e for e in nfkc[i:i + 4]) + "\n")
+        f.write("};\nconst int PDF_NFKC_COUNT = (int)(sizeof(PDF_NFKC) / sizeof(PDF_NFKC[0]));\n\n")
+        f.write("const uint32_t PDF_NFKC_SEQ[] = {\n")
+        for i in range(0, len(seq), 10):
+            f.write("    " + " ".join("0x%X," % x for x in seq[i:i + 10]) + "\n")
+        f.write("};\n\n")
+        f.write("const pdf_ucomp_t PDF_UCOMP[] = {\n")
+        for i in range(0, len(comp), 3):
+            f.write("    " + " ".join("{0x%X, 0x%X, 0x%X}," % e for e in comp[i:i + 3]) + "\n")
+        f.write("};\nconst int PDF_UCOMP_COUNT = (int)(sizeof(PDF_UCOMP) / sizeof(PDF_UCOMP[0]));\n\n")
+        emit_ranges(f, "PDF_U_SPACE", ranges(lambda c: c.isspace()))
+        emit_ranges(f, "PDF_U_WORD", ranges(lambda c: c.isalnum() or c == "_"))
+        emit_ranges(f, "PDF_U_ALPHA", ranges(lambda c: c.isalpha()))
+        emit_ranges(f, "PDF_U_UPPER", ranges(lambda c: c.isupper()))
+        emit_ranges(f, "PDF_U_LOWER", ranges(lambda c: c.islower()))
+        emit_ranges(f, "PDF_U_TITLE", ranges(lambda c: c.istitle() and not c.isupper()))
+    print("nfkc", len(nfkc), "seq", len(seq), "comp", len(comp))
+
+
+if __name__ == "__main__":
+    main()