Merge pull request #1619 from DeusData/fix/vt-notes-shipped-only

fix(ci): release notes report the shipped binary, not every candidate

README.md

@@ -794,7 +794,7 @@ internal/cbm/         Vendored tree-sitter grammars (158 languages) + AST extrac
 
 Every release is verified through a multi-layer pipeline before publication:
 
-- **VirusTotal** — all 24 executable candidates (unstripped, debug-stripped and stripped) across the eight release products are scanned before smoke/soak (clean is preferred and only the single-Microsoft `!ml` tolerance documented in [SECURITY.md](SECURITY.md#our-release-policy) may pass; the number of engines that returned a decisive result is recorded as evidence but is VirusTotal fleet availability, not a pass condition). The selected executable is packaged without changing its SHA-256, and release notes link every candidate result and the selection evidence. Every distinct object extracted from the shipped containers — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — is then scanned as well, so the full published surface is covered, not just the executables.
+- **VirusTotal** — all 24 executable candidates (unstripped, debug-stripped and stripped) across the eight release products are scanned before smoke/soak (clean is preferred and only the single-Microsoft `!ml` tolerance documented in [SECURITY.md](SECURITY.md#our-release-policy) may pass; the number of engines that returned a decisive result is recorded as evidence but is VirusTotal fleet availability, not a pass condition). The selected executable is packaged without changing its SHA-256, release notes link the verdict for the exact bytes shipped, and the full per-candidate evidence is published alongside the release as TSVs for anyone auditing the selection. Every distinct object extracted from the shipped containers — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — is then scanned as well, so the full published surface is covered, not just the executables.
 - **SLSA Level 3** — cryptographic build provenance generated by the trusted GitHub Actions build workflow; verify with `gh attestation verify <file> --repo DeusData/codebase-memory-mcp --signer-workflow DeusData/codebase-memory-mcp/.github/workflows/_build.yml`
 - **Sigstore cosign** — keyless signatures on all artifacts; bundles included in every release
 - **SHA-256 checksums** — `checksums.txt` published with every release; verified by both install scripts before extraction

SECURITY.md

@@ -126,7 +126,7 @@ Releases are created as **drafts** (invisible to users) and only published after
 2. **Sigstore cosign signing** — keyless digital signatures verifiable by anyone
 3. **SBOM** — Software Bill of Materials (SPDX) listing all vendored dependencies
 4. **SHA-256 checksums** — published with every release
-5. **VirusTotal scanning** — three behaviourally identical executable candidates — unstripped, debug-stripped and stripped — are derived from one linker output for every release product and scanned before smoke/soak. Zero malicious and zero suspicious is preferred; the only tolerated result is exactly one Microsoft malicious label ending in `!ml`, which is disclosed. The selected executable is packaged without changing its SHA-256. After packaging, every distinct object extracted from the 14 shipped containers is scanned under the same policy — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — so the covered surface is everything we publish, not only the executables. Release notes link every candidate result and the selection evidence.
+5. **VirusTotal scanning** — three behaviourally identical executable candidates — unstripped, debug-stripped and stripped — are derived from one linker output for every release product and scanned before smoke/soak. Zero malicious and zero suspicious is preferred; the only tolerated result is exactly one Microsoft malicious label ending in `!ml`, which is disclosed. The selected executable is packaged without changing its SHA-256. After packaging, every distinct object extracted from the 14 shipped containers is scanned under the same policy — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — so the covered surface is everything we publish, not only the executables. Release notes link the verdict for the exact bytes shipped; the per-candidate results and the selection table are published as release assets for independent audit.
 6. **OpenSSF Scorecard** — repository security health score
 
 Scope of the SLSA claim: this is a build provenance claim for release

scripts/ci/append-vt-notes.sh

@@ -211,40 +211,52 @@ asset_base = (
 )
 
 
-def verdict_cell(target: str, variant: str) -> str:
-    result = results_by_path[candidate_by_pair[target, variant]["scan_path"]]
+# Release notes report the SHIPPED binary and nothing else.
+#
+# Several candidates per product are scanned so the selector has an alternative
+# when an opaque classifier flags one of them, but a reader installing cbm cares
+# about the bytes they receive, not about the ones we discarded. The rejected
+# candidates' verdicts stay in the published evidence TSVs for anyone auditing
+# the selection, and they matter to US in development as a signal; they are
+# noise in a changelog.
+def shipped_result(target: str) -> dict:
+    selection = selection_by_target[target]
+    variant = selection["selected_variant"]
+    return results_by_path[candidate_by_pair[target, variant]["scan_path"]]
+
+
+def verdict_cell(target: str) -> str:
+    result = shipped_result(target)
     label = "clean" if result["policy_classification"] == "clean" else "Microsoft `!ml`"
     return f"[{label}]({result['virustotal_url']})"
 
 
-ml_count = sum(
-    result["policy_classification"] == "microsoft-ml" for result in results
-)
-engine_counts = [int(result["completed_engines"]) for result in results]
+shipped = [shipped_result(target) for target in TARGETS]
+ml_count = sum(result["policy_classification"] == "microsoft-ml" for result in shipped)
+engine_counts = [int(result["completed_engines"]) for result in shipped]
 engine_range = str(min(engine_counts)) if min(engine_counts) == max(engine_counts) else f"{min(engine_counts)}–{max(engine_counts)}"
 section = [
     START,
     "## Security Verification",
     "",
     (
-        f"Before smoke and soak testing, VirusTotal completed **{len(TARGETS) * len(VARIANTS)} executable scans**: "
-        "stripped and unstripped candidates for each of the **8 release products**. "
-        f"Every scan had at least 50 decisive engines (observed range: {engine_range})."
+        f"Every binary published below was scanned by VirusTotal before smoke and soak "
+        f"testing, and the verdict for the exact shipped bytes is linked per product "
+        f"(decisive engines: {engine_range})."
     ),
     (
-        "All candidates were clean."
+        "Every shipped binary was clean."
         if ml_count == 0
-        else f"**{ml_count} candidate(s)** had only the documented single Microsoft machine-learning `!ml` result; no other decisive engine reported malicious or suspicious."
+        else f"**{ml_count} shipped binary/binaries** carried only the documented single Microsoft machine-learning `!ml` result; no other decisive engine reported malicious or suspicious."
     ),
     "",
-    "| Product | Stripped candidate | Unstripped candidate | Shipped |",
-    "|---|---|---|---|",
+    "| Product | Shipped binary | VirusTotal verdict |",
+    "|---|---|---|",
 ]
 for target in TARGETS:
     selection = selection_by_target[target]
     section.append(
-        f"| `{target}` | {verdict_cell(target, 'stripped')} | "
-        f"{verdict_cell(target, 'unstripped')} | `{selection['selected_variant']}` (`{selection['selected_sha256']}`) |"
+        f"| `{target}` | `{selection['selected_sha256']}` | {verdict_cell(target)} |"
     )
 section.extend(
     [

tests/test_vt_release_notes_contract.sh

@@ -65,6 +65,12 @@ classes = {(target, variant): "clean" for target in targets for variant in varia
 classes["linux-amd64", "stripped"] = "microsoft-ml"
 classes["darwin-arm64", "stripped"] = "microsoft-ml"
 classes["darwin-arm64", "unstripped"] = "microsoft-ml"
+# windows-amd64 draws the tolerated verdict on ALL THREE candidates, so the
+# SHIPPED binary carries it and the notes must disclose exactly that one. The
+# two targets above keep a clean sibling, so their flagged candidates are
+# rejected and must NOT appear in the notes at all.
+for _variant in variants:
+    classes["windows-amd64", _variant] = "microsoft-ml"
 candidates = []
 results = []
 by_pair = {}
@@ -190,12 +196,18 @@ run_notes "$FIX/current.md" "$FIX/first.md"
 grep -q 'Intro text.' "$FIX/first.md" || fail "content before section was lost"
 grep -q 'Outro text.' "$FIX/first.md" || fail "content after section was lost"
 ! grep -q 'stale data' "$FIX/first.md" || fail "stale section was appended"
-grep -q '24 executable scans' "$FIX/first.md" || fail "candidate scan scope missing"
-grep -q '8 release products' "$FIX/first.md" || fail "product scope missing"
-grep -q '3 candidate(s).*Microsoft' "$FIX/first.md" || fail "tolerated findings not disclosed"
+# Release notes report ONLY the shipped bytes. Rejected candidates stay in the
+# published evidence TSVs; they are development signal, not changelog content.
+grep -q 'verdict for the exact shipped bytes is linked per product' "$FIX/first.md" || \
+  fail "shipped-binary scan scope missing"
+[ "$(grep -c '^| `' "$FIX/first.md")" = "8" ] || fail "product scope missing: expected one row per release product"
+grep -q '1 shipped binary.*Microsoft' "$FIX/first.md" || fail "tolerated finding on a shipped binary not disclosed"
+grep -q 'candidate(s)' "$FIX/first.md" && fail "notes must not count rejected candidates"
 grep -q 'no other decisive engine reported malicious or suspicious' "$FIX/first.md" || fail "non-decisive engines are overstated as clean"
 ! grep -q 'every other engine was clean' "$FIX/first.md" || fail "non-decisive engines are incorrectly described as clean"
-grep -q '| Product | Stripped candidate | Unstripped candidate | Shipped |' "$FIX/first.md" || fail "candidate table missing"
+grep -q '| Product | Shipped binary | VirusTotal verdict |' "$FIX/first.md" || fail "shipped-binary table missing"
+grep -q 'Stripped candidate' "$FIX/first.md" && \
+  fail "release notes must not enumerate rejected candidates"
 grep -q '| `linux-amd64` |' "$FIX/first.md" || fail "Linux product row missing"
 grep -q '| `windows-arm64` |' "$FIX/first.md" || fail "Windows product row missing"
 grep -q 'archive containers were not redundantly submitted' "$FIX/first.md" || fail "no-rescan boundary missing"
@@ -228,4 +240,4 @@ if run_notes "$FIX/reversed.md" "$FIX/reversed-capture.md"; then
   fail "reversed verification markers must fail closed"
 fi
 
-echo 'PASS: release notes reuse all 24 candidate verdicts, disclose selection, and avoid a duplicate scan'
+echo 'PASS: release notes report the shipped binary only, disclose selection, and avoid a duplicate scan'