From 032ebf5327e4b2f258f0b644f3b967ee38deeead Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Thu, 27 Aug 2026 00:02:04 +0200 Subject: [PATCH] fix(vendor): ship arkts upstream MIT byte-identical and register the fork The vendored LICENSE carried tree-sitter-typescript's MIT text plus an appended derivative notice. That is not what MIT asks of a derivative -- it asks that upstream's notice travel with the code -- and the appended text meant the provenance audit could not byte-verify the file against upstream, so arkts landed as NO-MANIFEST-ENTRY and failed license-gate. The vendored LICENSE is now byte-identical to upstream's, and arkts is registered in the audit's FORKS map, which is defined for exactly this case: a self-maintained fork whose vendored LICENSE must match its upstream. That is a routing entry, not an exemption -- check_upstream still byte-verifies against the real upstream repository, so the fact is checked rather than asserted. Audit now reports arkts as IDENTICAL. Our copyright for the ArkTS additions is not lost, it moves to where our SOURCE is: the grammar.js header (which now states all three copyrights explicitly) and THIRD_PARTY.md, which ships inside the release archives. The edit to scripts/audit-license-provenance.py is a maintainer-only enforcement-surface change, made deliberately and stated here rather than slipped in; a contribution may never supply its own audit entry. No generated parser or source changes -- comments, licence text, manifest prose and one routing entry only. Signed-off-by: Martin Vogel --- internal/cbm/vendored/grammars/MANIFEST.md | 4 ++-- internal/cbm/vendored/grammars/arkts/LICENSE | 14 -------------- scripts/audit-license-provenance.py | 7 +++++++ tools/tree-sitter-arkts/grammar.js | 10 ++++++++++ 4 files changed, 19 insertions(+), 16 deletions(-) diff --git a/internal/cbm/vendored/grammars/MANIFEST.md b/internal/cbm/vendored/grammars/MANIFEST.md index b4e9d75bf..27ada7451 100644 --- a/internal/cbm/vendored/grammars/MANIFEST.md +++ b/internal/cbm/vendored/grammars/MANIFEST.md @@ -15,7 +15,7 @@ The grammars were originally vendored as bare `parser.c`+`scanner.c` with **no r - `verdict`: VERIFIED-BOTH = our source matches *both* registries; VERIFIED-NVIM/HELIX = matches one; registry-disagreement = registries name a different repo (listed separately); `vendor-maintained` = the language vendor's own grammar, not in nvim/Helix. - **objectscript_udl / objectscript_routine** (added 2026-06-24): vendored from [intersystems/tree-sitter-objectscript](https://github.com/intersystems/tree-sitter-objectscript) @ `a7ffcdf` — MIT, the InterSystems-official grammars (a niche vendor language, hence `vendor-maintained`, not in nvim-treesitter/Helix). **Re-vendor note:** each `scanner.c`'s upstream `#include "../../common/scanner.h"` is repointed to a per-directory `objectscript_common.h` (a verbatim copy of upstream `common/scanner.h`), because this repo's shared `vendored/common/scanner.h` belongs to the cfml/fsharp grammars and differs. The generated `parser.c`/`scanner.c` are otherwise byte-for-byte upstream — on re-vendor, re-apply only that single include rename. **Local modification (2026-07-16):** in `objectscript_common.h`, two loop counters `uint8_t i` were widened to `int i` (the `reverse_marker` scan and the `html_marker_buffer` reversal) to clear CodeQL `cpp/comparison-with-wider-type` — a false positive in practice (both lengths are hard-bounded by `MARKER_BUFFER_MAX_LEN = 30`, so `uint8_t` could never wrap), fixed for cleanliness. On re-vendor, re-apply this widening too (or upstream it at intersystems/tree-sitter-objectscript). - **mojo** (added 2026-07-01): vendored from [lsh/tree-sitter-mojo](https://github.com/lsh/tree-sitter-mojo) @ `33193a99afe6` — MIT, ABI 15. Helix tracks `lsh/tree-sitter-mojo` as its Mojo grammar source, but the Helix-pinned commit (`3d7c53b8038f`) no longer resolves in the upstream repository after a force-push, so this vendor uses current upstream `main` rather than the stale registry SHA. Security review covered only the vendored C surface (`parser.c`, `scanner.c`, `tree_sitter/*.h`) plus upstream license/provenance metadata; no package manager hooks, workflow files, prompt/agent instruction files, or generated lockfiles were vendored. -- **arkts** (added 2026-08-26): **first-party derivative** — a fork of [tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)'s `typescript` dialect pinned @ `75b3874edb2d` (v0.23.2, the same commit our vendored typescript/tsx come from), on the tree-sitter-javascript base @ `3a837b6f3658` (v0.23.1, the version upstream's own package-lock pins), extended with ArkTS/ArkUI syntax (`@Component struct` declarations, UI-DSL trailing-closure calls with post-block attribute chains, decorated function declarations, `import lazy`, `@Extend`/`@Styles` leading-dot attribute chains, anonymous `stateStyles` style blocks). Grammar source + corpus tests live in `tools/tree-sitter-arkts/`; regenerate with `npx tree-sitter-cli@0.25.10 generate` (ABI 15). **External scanner:** `scanner.c` is a symbol-rename trampoline; `_common_scanner.h` is byte-identical (same SHA-256) to the reviewed `typescript/_common_scanner.h` already shipped. LICENSE = tree-sitter-typescript's MIT text verbatim (© 2017 Max Brunsfeld) + a derivative-work notice adding the tree-sitter-javascript attribution (© 2014 Max Brunsfeld) and the DeusData copyright for the ArkTS additions. +- **arkts** (added 2026-08-26): **first-party derivative** — a fork of [tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)'s `typescript` dialect pinned @ `75b3874edb2d` (v0.23.2, the same commit our vendored typescript/tsx come from), on the tree-sitter-javascript base @ `3a837b6f3658` (v0.23.1, the version upstream's own package-lock pins), extended with ArkTS/ArkUI syntax (`@Component struct` declarations, UI-DSL trailing-closure calls with post-block attribute chains, decorated function declarations, `import lazy`, `@Extend`/`@Styles` leading-dot attribute chains, anonymous `stateStyles` style blocks). Grammar source + corpus tests live in `tools/tree-sitter-arkts/`; regenerate with `npx tree-sitter-cli@0.25.10 generate` (ABI 15). **External scanner:** `scanner.c` is a symbol-rename trampoline; `_common_scanner.h` is byte-identical (same SHA-256) to the reviewed `typescript/_common_scanner.h` already shipped. **LICENSE:** tree-sitter-typescript's MIT text **verbatim and byte-identical** (© 2017 Max Brunsfeld) — which is what MIT requires of a derivative work, and what lets the provenance audit byte-verify it against upstream rather than take a note on trust. The fork is registered in the audit's `FORKS` map. Our own copyright for the ArkTS additions, and the tree-sitter-javascript attribution (© 2014 Max Brunsfeld), live with the SOURCE in `tools/tree-sitter-arkts/grammar.js` and in `THIRD_PARTY.md`, which is what ships in the release archives. > ⚠️ **Pinned commit = the revision nvim-treesitter/Helix vendor** (battle-tested, canonical source), not bleeding-edge HEAD. When re-vendoring, update the pinned commit here. @@ -235,7 +235,7 @@ upstream author (correctly retained). The table now records the true origin. | grammar | cur ABI | original upstream | license | |---|:---:|---|---| -| arkts | 15 | **first-party derivative**: forked from tree-sitter/tree-sitter-typescript @ `75b3874edb2d` (v0.23.2; javascript base @ `3a837b6f3658` v0.23.1); grammar source in `tools/tree-sitter-arkts/` | MIT, (c) 2017 Max Brunsfeld (verbatim) + (c) 2026 DeusData additions — see summary note | +| arkts | 15 | **self-maintained fork** of tree-sitter/tree-sitter-typescript @ `75b3874edb2d` (v0.23.2; javascript base @ `3a837b6f3658` v0.23.1); grammar source in `tools/tree-sitter-arkts/` | MIT, (c) 2017 Max Brunsfeld — byte-identical to upstream; ArkTS additions (c) 2026 DeusData, see grammar.js + THIRD_PARTY.md | | assembly | 14 | RubixDev/tree-sitter-assembly (**repo deleted from GitHub** — our retained MIT copy, (c) 2023 RubixDev, is the surviving grant) | MIT | | cfml | 15 | cfmleditor/tree-sitter-cfml | MIT, (c) 2014 Gareth Edwards & Gavin Baumanis — byte-identical | | cfscript | 15 | cfmleditor/tree-sitter-cfml | MIT, same — byte-identical | diff --git a/internal/cbm/vendored/grammars/arkts/LICENSE b/internal/cbm/vendored/grammars/arkts/LICENSE index abb887a2d..aa9f858db 100644 --- a/internal/cbm/vendored/grammars/arkts/LICENSE +++ b/internal/cbm/vendored/grammars/arkts/LICENSE @@ -19,17 +19,3 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - ---- - -This ArkTS grammar is a derivative work of tree-sitter-typescript -(https://github.com/tree-sitter/tree-sitter-typescript, MIT license above, -forked at commit 75b3874edb2dc714fb1fd77a32013d0f8699989f / v0.23.2), which -itself builds on tree-sitter-javascript -(https://github.com/tree-sitter/tree-sitter-javascript, The MIT License (MIT), -Copyright (c) 2014 Max Brunsfeld; included at commit -3a837b6f3658ca3618f2022f8707e29739c91364 / v0.23.1). - -Copyright (c) 2026 DeusData — ArkTS additions (struct component declarations, -UI-DSL trailing-closure calls, decorated function declarations, lazy imports, -@Extend/@Styles attribute chains), licensed under the same MIT terms. diff --git a/scripts/audit-license-provenance.py b/scripts/audit-license-provenance.py index 3fc97429d..edbdbfdb9 100755 --- a/scripts/audit-license-provenance.py +++ b/scripts/audit-license-provenance.py @@ -22,6 +22,13 @@ GRAMMARS = os.path.join(ROOT, "internal/cbm/vendored/grammars") FIRST_PARTY = {"cobol", "form", "janet", "magma", "protobuf", "wolfram"} FORKS = { # self-maintained forks: vendored LICENSE must match the original upstream + # arkts: our own ArkTS fork of tree-sitter-typescript. The vendored LICENSE is + # upstream's MIT verbatim, which is what MIT requires of a derivative; our own + # copyright and the fork's provenance live where our SOURCE is (grammar.js + # header, MANIFEST.md, THIRD_PARTY.md) rather than inside upstream's notice. + # This is a routing entry, NOT an exemption -- check_upstream still byte-verifies + # against the real upstream repository. + "arkts": "tree-sitter/tree-sitter-typescript", "cfml": "cfmleditor/tree-sitter-cfml", "cfscript": "cfmleditor/tree-sitter-cfml", "dotenv": "pnx/tree-sitter-dotenv", diff --git a/tools/tree-sitter-arkts/grammar.js b/tools/tree-sitter-arkts/grammar.js index 99a6f7f33..c2788ac98 100644 --- a/tools/tree-sitter-arkts/grammar.js +++ b/tools/tree-sitter-arkts/grammar.js @@ -2,6 +2,16 @@ * @file ArkTS (HarmonyOS/OpenHarmony .ets) grammar for tree-sitter * @license MIT * + * Copyright (c) 2017 Max Brunsfeld (tree-sitter-typescript, forked below) + * Copyright (c) 2014 Max Brunsfeld (tree-sitter-javascript base) + * Copyright (c) 2026 DeusData (ArkTS/ArkUI additions in this file) + * + * The VENDORED parser ships upstream's MIT LICENSE byte-identical, because that + * is what MIT requires of a derivative and what lets the provenance audit verify + * it against upstream instead of trusting a note. The copyright for our own + * additions is stated HERE, with the source, and in THIRD_PARTY.md, which ships + * inside the release archives. + * * First-party DERIVATIVE grammar: a fork of tree-sitter-typescript's * `typescript` dialect (no JSX), extended with the ArkTS/ArkUI syntax that * plain TypeScript cannot parse.