From 7b4533b67dc4b19981a695dfb8b75cae5e458635 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Fri, 14 Aug 2026 00:21:31 +0200 Subject: [PATCH] fix(ci): finish the three-candidate wiring and restore full-surface VT scanning MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things, both found by the dry-run and by sweeping what it exposed. 1. The three-candidate change was incomplete. The dry-run failed at stage-release-candidates.py, which carries its OWN copy of the transform validation that the previous commit only fixed in the selector: stage-release-candidates: candidate linux-amd64/debug-stripped has invalid transform: 'strip-debug' Sweeping for that assumption found it in six places, not two: stage-release-candidates.py, select-release-candidates.py, verify-release-selection.py, append-vt-notes.sh and two contract-test fixtures — as hardcoded 16s, `len(TARGETS) * 2`, two-entry VARIANTS tuples and two-key truth tables. All are now derived from len(VARIANTS). Field prefixes needed care: the variant NAME keeps its hyphen because it is the on-disk directory, while the evidence columns use underscores, so `debug-stripped` reads `debug_stripped_sha256`. Every lookup now goes through an explicit FIELD_KEY map instead of interpolating the variant directly. The selection contract test now covers the truth table EXHAUSTIVELY: three variants x two tolerated classifications is exactly eight combinations, and there are exactly eight targets, so every case is exercised once. 2. Full-surface VirusTotal scanning is restored. This PR had moved scanning upstream to the candidates and deleted the post-package pass, which silently narrowed coverage from everything we ship to executables only. The 42 runtime files across the 14 containers — install.sh, install.ps1, LICENSE, THIRD_PARTY_NOTICES.md, the MCPB manifest.json and the unpacked UI assets — were still extracted, structurally verified and strings-audited, but no longer scanned at all. install.sh and install.ps1 are the highest-consequence non-executable bytes we publish; users pipe them straight into a shell. The verify job scans every extracted object again, under the same policy. Re-submitting the selected executables alongside them is close to free because VirusTotal is content-addressed and answers for identical bytes from its own record — the same property that made the analysis-id equality check untenable two commits ago. The gate-chain contract asserted the opposite ("duplicate post-package VirusTotal path remains"). That assertion is inverted: the pass is required, and it is not a duplicate, since it covers a strictly larger set. README and SECURITY.md updated from "archive containers are checksummed rather than redundantly rescanned" to state the full covered surface. All five release/VT contract tests pass. Signed-off-by: Martin Vogel --- .github/workflows/release.yml | 54 +++++++++++++++ README.md | 2 +- SECURITY.md | 2 +- scripts/ci/append-vt-notes.sh | 42 +++++++----- scripts/ci/select-release-candidates.py | 2 +- scripts/ci/stage-release-candidates.py | 6 +- scripts/ci/verify-release-selection.py | 48 +++++++++----- tests/test_release_gate_chain_contract.sh | 18 +++-- tests/test_vt_candidate_selection_contract.sh | 66 +++++++++++-------- tests/test_vt_release_notes_contract.sh | 49 ++++++++------ 10 files changed, 199 insertions(+), 90 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 3545d8810..2e87ee7a6 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -298,6 +298,60 @@ jobs: scripts/security-strings.sh "$f" done + # Full-surface VirusTotal pass over the SHIPPED containers. + # + # The candidate scan upstream covers executables only. Everything else we + # actually ship — install.sh, install.ps1, LICENSE, THIRD_PARTY_NOTICES.md, + # the MCPB manifest.json and the unpacked UI assets, 42 runtime files + # across 14 containers — is extracted here and gets scanned here. Dropping + # it would have narrowed the promise made in README/SECURITY.md, and + # install.sh and install.ps1 are the highest-consequence non-executable + # bytes we publish: users pipe them straight into a shell. + # + # The selected executables are re-submitted too. That is deliberate and + # nearly free: VirusTotal is content-addressed, so identical bytes return + # the analysis it already holds instead of re-running 70+ engines — the + # same property that made the analysis-id equality check untenable. + - name: VirusTotal scan of every extracted release object + uses: crazy-max/ghaction-virustotal@936d8c5c00afe97d3d9a1af26d017cfdf26800a2 # v5.0.0 + id: virustotal + with: + vt_api_key: ${{ secrets.VIRUS_TOTAL_SCANNER_API_KEY }} + files: binaries/objects/* + request_rate: 4 + + # Same policy as the candidate gate: a single Microsoft `!ml` is tolerated + # and disclosed, anything else blocks. Engine count is evidence, not a gate. + - name: Wait for VirusTotal results + env: + VT_API_KEY: ${{ secrets.VIRUS_TOTAL_SCANNER_API_KEY }} + VT_ANALYSIS: ${{ steps.virustotal.outputs.analysis }} + VT_EXPECTED_SCAN_SET: binaries/scan-set.tsv + VT_ASSOCIATIONS: binaries/associations.tsv + VT_RESULTS_PATH: binaries/vt-results.tsv + MIN_ENGINES: 50 + run: scripts/ci/check-virustotal.sh + + - name: Preserve extracted-object VirusTotal evidence + if: ${{ always() }} + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: virustotal-evidence-release-${{ github.run_id }} + path: | + binaries/associations.tsv + binaries/scan-set.tsv + binaries/vt-results.tsv + if-no-files-found: warn + + - name: Publish durable public VirusTotal evidence + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + VERSION: ${{ inputs.version }} + VT_EXPECTED_SCAN_SET: binaries/scan-set.tsv + VT_ASSOCIATIONS: binaries/associations.tsv + VT_RESULTS_PATH: binaries/vt-results.tsv + run: bash scripts/ci/publish-vt-evidence.sh + # Candidate bytes were scanned before smoke/soak and the verifier above # proved every packaged executable has the selected candidate SHA-256. # Re-submitting the same executable bytes here would add wait time but no diff --git a/README.md b/README.md index b36b72f20..0e78ccabc 100644 --- a/README.md +++ b/README.md @@ -794,7 +794,7 @@ internal/cbm/ Vendored tree-sitter grammars (158 languages) + AST extrac Every release is verified through a multi-layer pipeline before publication: -- **VirusTotal** — all 24 executable candidates (unstripped, debug-stripped and stripped) across the eight release products are scanned before smoke/soak (clean is preferred and only the single-Microsoft `!ml` tolerance documented in [SECURITY.md](SECURITY.md#our-release-policy) may pass; the number of engines that returned a decisive result is recorded as evidence but is VirusTotal fleet availability, not a pass condition). The selected executable is packaged without changing its SHA-256, and release notes link every candidate result and the selection evidence. Archive containers are checksummed, not redundantly resubmitted. +- **VirusTotal** — all 24 executable candidates (unstripped, debug-stripped and stripped) across the eight release products are scanned before smoke/soak (clean is preferred and only the single-Microsoft `!ml` tolerance documented in [SECURITY.md](SECURITY.md#our-release-policy) may pass; the number of engines that returned a decisive result is recorded as evidence but is VirusTotal fleet availability, not a pass condition). The selected executable is packaged without changing its SHA-256, and release notes link every candidate result and the selection evidence. Every distinct object extracted from the shipped containers — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — is then scanned as well, so the full published surface is covered, not just the executables. - **SLSA Level 3** — cryptographic build provenance generated by the trusted GitHub Actions build workflow; verify with `gh attestation verify --repo DeusData/codebase-memory-mcp --signer-workflow DeusData/codebase-memory-mcp/.github/workflows/_build.yml` - **Sigstore cosign** — keyless signatures on all artifacts; bundles included in every release - **SHA-256 checksums** — `checksums.txt` published with every release; verified by both install scripts before extraction diff --git a/SECURITY.md b/SECURITY.md index 308c1586e..678a280ec 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -126,7 +126,7 @@ Releases are created as **drafts** (invisible to users) and only published after 2. **Sigstore cosign signing** — keyless digital signatures verifiable by anyone 3. **SBOM** — Software Bill of Materials (SPDX) listing all vendored dependencies 4. **SHA-256 checksums** — published with every release -5. **VirusTotal scanning** — three behaviourally identical executable candidates — unstripped, debug-stripped and stripped — are derived from one linker output for every release product and scanned before smoke/soak. Zero malicious and zero suspicious is preferred; the only tolerated result is exactly one Microsoft malicious label ending in `!ml`, which is disclosed. The selected executable is packaged without changing its SHA-256; archive containers are checksummed rather than redundantly rescanned. Release notes link every candidate result and the selection evidence. +5. **VirusTotal scanning** — three behaviourally identical executable candidates — unstripped, debug-stripped and stripped — are derived from one linker output for every release product and scanned before smoke/soak. Zero malicious and zero suspicious is preferred; the only tolerated result is exactly one Microsoft malicious label ending in `!ml`, which is disclosed. The selected executable is packaged without changing its SHA-256. After packaging, every distinct object extracted from the 14 shipped containers is scanned under the same policy — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — so the covered surface is everything we publish, not only the executables. Release notes link every candidate result and the selection evidence. 6. **OpenSSF Scorecard** — repository security health score Scope of the SLSA claim: this is a build provenance claim for release diff --git a/scripts/ci/append-vt-notes.sh b/scripts/ci/append-vt-notes.sh index 757654b8d..e9aa62ac5 100755 --- a/scripts/ci/append-vt-notes.sh +++ b/scripts/ci/append-vt-notes.sh @@ -38,7 +38,8 @@ TARGETS = ( "windows-amd64", "windows-arm64", ) -VARIANTS = ("unstripped", "stripped") +VARIANTS = ("unstripped", "debug-stripped", "stripped") +FIELD_KEY = {variant: variant.replace("-", "_") for variant in VARIANTS} CANDIDATE_FIELDS = ( "target", "variant", "relative_path", "source_sha256", "pre_sign_sha256", "sha256", "size", "format", "architecture", "linkage", "transform", @@ -54,7 +55,11 @@ SELECTION_FIELDS = ( "target", "selected_variant", "selected_path", "selected_sha256", "selected_size", "decision", "unstripped_sha256", "unstripped_scan_path", "unstripped_classification", "unstripped_analysis_id", - "unstripped_virustotal_url", "stripped_sha256", "stripped_scan_path", + "unstripped_virustotal_url", + "debug_stripped_sha256", "debug_stripped_scan_path", + "debug_stripped_classification", "debug_stripped_analysis_id", + "debug_stripped_virustotal_url", + "stripped_sha256", "stripped_scan_path", "stripped_classification", "stripped_analysis_id", "stripped_virustotal_url", ) SHA256 = re.compile(r"[0-9a-f]{64}\Z") @@ -110,20 +115,22 @@ selection_meta, selections = read_tsv( selection_path, "cbm-release-selection-v1", SELECTION_FIELDS ) expected_pairs = [(target, variant) for target in TARGETS for variant in VARIANTS] -if candidate_meta != {"targets": "8", "candidates": "16"}: - fail("candidate metadata does not bind the 8-target/16-candidate matrix") +if candidate_meta != {"targets": str(len(TARGETS)), "candidates": str(len(TARGETS) * len(VARIANTS))}: + fail("candidate metadata does not bind the canonical target/variant matrix") if [(row["target"], row["variant"]) for row in candidates] != expected_pairs: fail("candidate rows are not the canonical target/variant matrix") -if result_meta.get("scan_objects") != "16" or result_meta.get("associations") != "16": - fail("VirusTotal results do not cover all 16 candidates") -if selection_meta != {"policy": "virustotal-v2", "targets": "8", "candidates": "16"}: +expected_scans = str(len(TARGETS) * len(VARIANTS)) +if result_meta.get("scan_objects") != expected_scans or result_meta.get("associations") != expected_scans: + fail("VirusTotal results do not cover every candidate") +if selection_meta != {"policy": "virustotal-v2", "targets": str(len(TARGETS)), "candidates": str(len(TARGETS) * len(VARIANTS))}: fail("selection was not produced by the scanned release policy") if [row["target"] for row in selections] != list(TARGETS): fail("selection rows are not the canonical target matrix") candidate_by_pair = {(row["target"], row["variant"]): row for row in candidates} candidate_by_path = {row["scan_path"]: row for row in candidates} -if len(candidate_by_pair) != 16 or len(candidate_by_path) != 16: +expected_candidates = len(TARGETS) * len(VARIANTS) +if len(candidate_by_pair) != expected_candidates or len(candidate_by_path) != expected_candidates: fail("candidate evidence contains duplicates") results_by_path: dict[str, dict[str, str]] = {} @@ -175,9 +182,10 @@ for selection in selections: target = selection["target"] pair = {variant: candidate_by_pair[target, variant] for variant in VARIANTS} pair_results = {variant: results_by_path[pair[variant]["scan_path"]] for variant in VARIANTS} - stripped_class = pair_results["stripped"]["policy_classification"] - unstripped_class = pair_results["unstripped"]["policy_classification"] - expected_variant = "stripped" if stripped_class == "clean" or unstripped_class != "clean" else "unstripped" + classes = {v: pair_results[v]["policy_classification"] for v in VARIANTS} + order = ("stripped", "debug-stripped", "unstripped") + clean = [v for v in order if classes[v] == "clean"] + expected_variant = clean[0] if clean else "stripped" selected = pair[expected_variant] if ( selection["selected_variant"] != expected_variant @@ -188,11 +196,11 @@ for selection in selections: for variant in VARIANTS: result = pair_results[variant] if ( - selection[f"{variant}_sha256"] != pair[variant]["sha256"] - or selection[f"{variant}_scan_path"] != pair[variant]["scan_path"] - or selection[f"{variant}_classification"] != result["policy_classification"] - or selection[f"{variant}_analysis_id"] != result["analysis_id"] - or selection[f"{variant}_virustotal_url"] != result["virustotal_url"] + selection[f"{FIELD_KEY[variant]}_sha256"] != pair[variant]["sha256"] + or selection[f"{FIELD_KEY[variant]}_scan_path"] != pair[variant]["scan_path"] + or selection[f"{FIELD_KEY[variant]}_classification"] != result["policy_classification"] + or selection[f"{FIELD_KEY[variant]}_analysis_id"] != result["analysis_id"] + or selection[f"{FIELD_KEY[variant]}_virustotal_url"] != result["virustotal_url"] ): fail(f"selection evidence is not bound to both candidate verdicts: {target}") selection_by_target[target] = selection @@ -219,7 +227,7 @@ section = [ "## Security Verification", "", ( - "Before smoke and soak testing, VirusTotal completed **16 executable scans**: " + f"Before smoke and soak testing, VirusTotal completed **{len(TARGETS) * len(VARIANTS)} executable scans**: " "stripped and unstripped candidates for each of the **8 release products**. " f"Every scan had at least 50 decisive engines (observed range: {engine_range})." ), diff --git a/scripts/ci/select-release-candidates.py b/scripts/ci/select-release-candidates.py index 94ab81e25..89cdb41df 100755 --- a/scripts/ci/select-release-candidates.py +++ b/scripts/ci/select-release-candidates.py @@ -253,7 +253,7 @@ def load_candidates( marker="cbm-release-candidates-v1", fields=CANDIDATE_FIELDS, ) - if metadata != {"targets": str(len(TARGETS)), "candidates": str(len(TARGETS) * 2)}: + if metadata != {"targets": str(len(TARGETS)), "candidates": str(len(TARGETS) * len(VARIANTS))}: raise ContractError("candidate metadata does not bind the exact canonical matrix") expected_pairs = [(target, variant) for target in TARGETS for variant in VARIANTS] if [(row["target"], row["variant"]) for row in rows] != expected_pairs: diff --git a/scripts/ci/stage-release-candidates.py b/scripts/ci/stage-release-candidates.py index 3a4fda27f..e5d153ad2 100755 --- a/scripts/ci/stage-release-candidates.py +++ b/scripts/ci/stage-release-candidates.py @@ -194,7 +194,11 @@ def validate_row_shape(row: dict[str, str], *, target: str, variant: str) -> pat "format": file_format, "architecture": architecture, "linkage": linkage, - "transform": "copy" if variant == "unstripped" else "strip", + "transform": { + "unstripped": "copy", + "debug-stripped": "strip-debug", + "stripped": "strip", + }[variant], "signature": "adhoc-verified" if target.startswith("darwin-") else "not-applicable", "pair_verification": "same-linker-output-v1", } diff --git a/scripts/ci/verify-release-selection.py b/scripts/ci/verify-release-selection.py index f8269c703..be7733deb 100755 --- a/scripts/ci/verify-release-selection.py +++ b/scripts/ci/verify-release-selection.py @@ -31,7 +31,8 @@ TARGETS = ( "windows-amd64", "windows-arm64", ) -VARIANTS = ("unstripped", "stripped") +VARIANTS = ("unstripped", "debug-stripped", "stripped") +FIELD_KEY = {variant: variant.replace("-", "_") for variant in VARIANTS} SELECTION_FIELDS = ( "target", "selected_variant", @@ -44,6 +45,11 @@ SELECTION_FIELDS = ( "unstripped_classification", "unstripped_analysis_id", "unstripped_virustotal_url", + "debug_stripped_sha256", + "debug_stripped_scan_path", + "debug_stripped_classification", + "debug_stripped_analysis_id", + "debug_stripped_virustotal_url", "stripped_sha256", "stripped_scan_path", "stripped_classification", @@ -163,7 +169,7 @@ def load_selection( target = row["target"] binary = "codebase-memory-mcp.exe" if target.startswith("windows-") else "codebase-memory-mcp" classifications = { - variant: row[f"{variant}_classification"] for variant in VARIANTS + variant: row[f"{FIELD_KEY[variant]}_classification"] for variant in VARIANTS } if metadata["policy"] == "virustotal-v2": if any( @@ -172,27 +178,33 @@ def load_selection( ): raise ContractError(f"release selection contains a hard classification: {target}") for variant in VARIANTS: - sha256 = row[f"{variant}_sha256"] - analysis_id = row[f"{variant}_analysis_id"] + sha256 = row[f"{FIELD_KEY[variant]}_sha256"] + analysis_id = row[f"{FIELD_KEY[variant]}_analysis_id"] expected_url = f"https://www.virustotal.com/gui/file/{sha256}/detection" if ANALYSIS_ID_RE.fullmatch(analysis_id) is None: raise ContractError(f"release selection lacks VT analysis evidence: {target}") if analysis_id in analysis_ids: raise ContractError("release selection reuses one VT analysis id") analysis_ids.add(analysis_id) - if row[f"{variant}_virustotal_url"] != expected_url: + if row[f"{FIELD_KEY[variant]}_virustotal_url"] != expected_url: raise ContractError(f"release selection has unbound VT URL: {target}") - stripped_class = classifications["stripped"] - unstripped_class = classifications["unstripped"] - if stripped_class == "clean": - expected_variant = "stripped" - expected_decision = "stripped-preferred" - elif unstripped_class == "clean": - expected_variant = "unstripped" - expected_decision = "unstripped-clean-after-stripped-microsoft-ml" + # Re-derived here independently of the selector: smallest artifact + # first, first CLEAN one wins, and only if every candidate drew the + # tolerated verdict do we ship the smallest flagged one. + order = ("stripped", "debug-stripped", "unstripped") + clean = [v for v in order if classifications[v] == "clean"] + if clean: + expected_variant = clean[0] + if expected_variant == "stripped": + expected_decision = "stripped-preferred" + else: + others = "-".join( + f"{v}:{classifications[v]}" for v in order if v != expected_variant + ) + expected_decision = f"{expected_variant}-clean-after-{others}" else: expected_variant = "stripped" - expected_decision = "stripped-both-microsoft-ml" + expected_decision = "stripped-all-candidates-microsoft-ml" else: if any( classification != "unscanned-dry-run" @@ -200,7 +212,7 @@ def load_selection( ): raise ContractError(f"dry-run selection disguises unscanned evidence: {target}") if any( - row[f"{variant}_{suffix}"] + row[f"{FIELD_KEY[variant]}_{suffix}"] for variant in VARIANTS for suffix in ("analysis_id", "virustotal_url") ): @@ -217,14 +229,14 @@ def load_selection( raise ContractError(f"release selection has unsafe selected path: {target}") if ( SHA256_RE.fullmatch(row["selected_sha256"]) is None - or row["selected_sha256"] != row[f"{selected_variant}_sha256"] + or row["selected_sha256"] != row[f"{FIELD_KEY[selected_variant]}_sha256"] or not row["selected_size"].isdigit() or not 0 < int(row["selected_size"]) <= MAX_MEMBER_BYTES ): raise ContractError(f"release selection has invalid selected content: {target}") for variant in VARIANTS: - sha256 = row[f"{variant}_sha256"] - scan_path = pathlib.PurePosixPath(row[f"{variant}_scan_path"]) + sha256 = row[f"{FIELD_KEY[variant]}_sha256"] + scan_path = pathlib.PurePosixPath(row[f"{FIELD_KEY[variant]}_scan_path"]) if ( SHA256_RE.fullmatch(sha256) is None or scan_path.as_posix() != f"objects/{sha256}" diff --git a/tests/test_release_gate_chain_contract.sh b/tests/test_release_gate_chain_contract.sh index bac3e7f1f..3bb70b08b 100755 --- a/tests/test_release_gate_chain_contract.sh +++ b/tests/test_release_gate_chain_contract.sh @@ -291,11 +291,21 @@ for token in ("--pattern 'release-selection.tsv'", failures.append( f"release verify: final draft-byte selection binding is missing token: {token}") -for forbidden in ("crazy-max/ghaction-virustotal", "scripts/ci/check-virustotal.sh", - "scripts/ci/publish-vt-evidence.sh"): - if forbidden in verify_body: +# The post-package scan is REQUIRED, not forbidden. It is not a duplicate of the +# candidate scan: that one covers executables only, while this one covers every +# distinct object extracted from the 14 shipped containers - install.sh, +# install.ps1, LICENSE, THIRD_PARTY_NOTICES.md, the MCPB manifest.json and the +# unpacked UI assets. Those are the bytes users pipe into a shell, and dropping +# them would narrow the promise made in README.md and SECURITY.md. +# +# Re-submitting the selected executables alongside them is close to free: +# VirusTotal is content-addressed and returns the analysis it already holds for +# identical bytes. +for token in ("crazy-max/ghaction-virustotal", "scripts/ci/check-virustotal.sh", + "scripts/ci/publish-vt-evidence.sh", "files: binaries/objects/*"): + if token not in verify_body: failures.append( - f"release verify: duplicate post-package VirusTotal path remains: {forbidden}") + f"release verify: full-surface VirusTotal pass is missing: {token}") for token in ("release-candidates.tsv", "virustotal-candidate-results.tsv", "release-selection.tsv", "scripts/ci/append-vt-notes.sh"): if token not in verify_body: diff --git a/tests/test_vt_candidate_selection_contract.sh b/tests/test_vt_candidate_selection_contract.sh index 5f573cee5..0a2cf96c9 100755 --- a/tests/test_vt_candidate_selection_contract.sh +++ b/tests/test_vt_candidate_selection_contract.sh @@ -41,7 +41,8 @@ TARGETS = ( "linux-arm64-portable", "darwin-amd64", "darwin-arm64", "windows-amd64", "windows-arm64", ) -VARIANTS = ("unstripped", "stripped") +VARIANTS = ("unstripped", "debug-stripped", "stripped") +ORDER = ("stripped", "debug-stripped", "unstripped") FIELDS = ( "target", "variant", "relative_path", "source_sha256", "pre_sign_sha256", "sha256", "size", "format", "architecture", "linkage", "transform", @@ -106,7 +107,11 @@ def write_tsv(path: pathlib.Path, marker: str, metadata: dict[str, object], def candidate_bytes(target: str, variant: str) -> bytes: linked = f"codebase-memory-mcp:{target}:same-linker-output\n".encode() - suffix = b"debug-and-local-symbols\n" if variant == "unstripped" else b"stripped\n" + suffix = { + "unstripped": b"debug-and-local-symbols\n", + "debug-stripped": b"local-symbols-only\n", + "stripped": b"stripped\n", + }[variant] return linked + suffix @@ -137,7 +142,7 @@ def make_artifacts(directory: pathlib.Path, *, reverse: bool = False) -> dict[tu "format": "pe" if target.startswith("windows-") else "macho" if target.startswith("darwin-") else "elf", "architecture": target.split("-")[1], "linkage": "portable" if target.endswith("-portable") else "dynamic" if target.startswith("linux-") else "native", - "transform": "copy" if variant == "unstripped" else "strip", + "transform": {"unstripped": "copy", "debug-stripped": "strip-debug", "stripped": "strip"}[variant], "signature": "adhoc-verified" if target.startswith("darwin-") else "not-applicable", "strip_tool": "contract-strip", "strip_version": "1.0", @@ -152,7 +157,7 @@ def make_artifacts(directory: pathlib.Path, *, reverse: bool = False) -> dict[tu def invoke_stage(source: pathlib.Path, output: pathlib.Path) -> subprocess.CompletedProcess[str]: - return run(stage_tool, source, output, "--expect-targets", 8, "--expect-candidates", 16) + return run(stage_tool, source, output, "--expect-targets", 8, "--expect-candidates", 24) artifacts = fix / "artifacts" @@ -166,7 +171,7 @@ require(sorted(path.name for path in staged.iterdir()) == cmeta, candidates = manifest(staged / "candidates.tsv") require(cmeta["marker"] == "# cbm-release-candidates-v1", "candidate marker changed") -require(cmeta.get("targets") == "8" and cmeta.get("candidates") == "16", "candidate counts not bound") +require(cmeta.get("targets") == "8" and cmeta.get("candidates") == "24", "candidate counts not bound") require([(row["target"], row["variant"]) for row in candidates] == [(target, variant) for target in TARGETS for variant in VARIANTS], "candidate rows are not in canonical target/variant order") @@ -254,19 +259,33 @@ def result_rows(classifications: dict[tuple[str, str], str], *, shuffle: bool = def write_results(path: pathlib.Path, rows: list[dict[str, object]]) -> None: write_tsv(path, "cbm-virustotal-results-v2", { - "scan_objects": 16, "associations": 16, "min_engines_policy": 50, + "scan_objects": 24, "associations": 24, "min_engines_policy": 50, "min_completed_engines": 71, "max_completed_engines": 71, }, RESULT_FIELDS, rows) -# Keys are (stripped, unstripped), matching the policy's preferred-candidate -# order and making the asymmetric fallback case unambiguous. -TRUTH = { - ("clean", "clean"): "stripped", - ("clean", "microsoft-ml"): "stripped", - ("microsoft-ml", "clean"): "unstripped", - ("microsoft-ml", "microsoft-ml"): "stripped", -} +# Keys are (stripped, debug-stripped, unstripped) in the policy's preferred +# order. Three variants x two tolerated classifications is exactly eight +# combinations, so the eight targets below cover the table EXHAUSTIVELY. +TRUTH_KEYS = [ + (a, b, c) + for a in ("clean", "microsoft-ml") + for b in ("clean", "microsoft-ml") + for c in ("clean", "microsoft-ml") +] + + +def expected_choice(key: tuple[str, str, str]) -> tuple[str, str]: + """Return (selected_variant, decision) for a (stripped, debug, unstripped) key.""" + by_variant = dict(zip(ORDER, key)) + clean = [v for v in ORDER if by_variant[v] == "clean"] + if not clean: + return "stripped", "stripped-all-candidates-microsoft-ml" + chosen = clean[0] + if chosen == "stripped": + return "stripped", "stripped-preferred" + others = "-".join(f"{v}:{by_variant[v]}" for v in ORDER if v != chosen) + return chosen, f"{chosen}-clean-after-{others}" def selection_rows(directory: pathlib.Path) -> tuple[dict[str, str], list[dict[str, str]]]: @@ -277,11 +296,13 @@ def selection_rows(directory: pathlib.Path) -> tuple[dict[str, str], list[dict[s # repeat once, proving exact decision behavior and both result bindings. classes: dict[tuple[str, str], str] = {} expected: dict[str, str] = {} -truth_keys = list(TRUTH) +expected_decisions: dict[str, str] = {} +require(len(TRUTH_KEYS) == len(TARGETS), "truth table must map one combination per target") for index, target in enumerate(TARGETS): - pair = truth_keys[index % len(truth_keys)] - classes[target, "stripped"], classes[target, "unstripped"] = pair - expected[target] = TRUTH[pair] + key = TRUTH_KEYS[index] + for variant, classification in zip(ORDER, key): + classes[target, variant] = classification + expected[target], expected_decisions[target] = expected_choice(key) results_file = fix / "vt-results.tsv" write_results(results_file, result_rows(classes, shuffle=True)) selected = fix / "selected" @@ -296,14 +317,7 @@ require([row["target"] for row in selections] == list(TARGETS), "selection order for row in selections: target = row["target"] require(row["selected_variant"] == expected[target], f"truth-table decision wrong for {target}") - target_pair = (classes[target, "stripped"], classes[target, "unstripped"]) - expected_decision = ( - "unstripped-clean-after-stripped-microsoft-ml" - if expected[target] == "unstripped" - else "stripped-both-microsoft-ml" - if target_pair == ("microsoft-ml", "microsoft-ml") - else "stripped-preferred" - ) + expected_decision = expected_decisions[target] require(row["decision"] == expected_decision, f"truth-table decision reason wrong for {target}: {row['decision']!r}") chosen = next(c for c in candidates if c["target"] == target and c["variant"] == expected[target]) diff --git a/tests/test_vt_release_notes_contract.sh b/tests/test_vt_release_notes_contract.sh index 4b3dccc7a..8ced3ceae 100644 --- a/tests/test_vt_release_notes_contract.sh +++ b/tests/test_vt_release_notes_contract.sh @@ -21,7 +21,8 @@ targets = ( "linux-arm64-portable", "darwin-amd64", "darwin-arm64", "windows-amd64", "windows-arm64", ) -variants = ("unstripped", "stripped") +variants = ("unstripped", "debug-stripped", "stripped") +field_key = {v: v.replace("-", "_") for v in variants} candidate_fields = ( "target", "variant", "relative_path", "source_sha256", "pre_sign_sha256", "sha256", "size", "format", "architecture", "linkage", "transform", @@ -37,7 +38,11 @@ selection_fields = ( "target", "selected_variant", "selected_path", "selected_sha256", "selected_size", "decision", "unstripped_sha256", "unstripped_scan_path", "unstripped_classification", "unstripped_analysis_id", - "unstripped_virustotal_url", "stripped_sha256", "stripped_scan_path", + "unstripped_virustotal_url", + "debug_stripped_sha256", "debug_stripped_scan_path", + "debug_stripped_classification", "debug_stripped_analysis_id", + "debug_stripped_virustotal_url", + "stripped_sha256", "stripped_scan_path", "stripped_classification", "stripped_analysis_id", "stripped_virustotal_url", ) @@ -74,7 +79,7 @@ for target in targets: "size": "1234", "format": "pe" if target.startswith("windows-") else "macho" if target.startswith("darwin-") else "elf", "architecture": target.split("-")[1], "linkage": "portable" if target.endswith("-portable") else "dynamic" if target.startswith("linux-") else "native", - "transform": "copy" if variant == "unstripped" else "strip", + "transform": {"unstripped": "copy", "debug-stripped": "strip-debug", "stripped": "strip"}[variant], "signature": "adhoc-verified" if target.startswith("darwin-") else "not-applicable", "strip_tool": "strip", "strip_version": "contract", "pair_verification": "same-linker-output-v1", "scan_path": f"objects/{sha}", @@ -98,15 +103,17 @@ for target in targets: result_by_pair = {(row["target"], row["variant"]): result for row, result in zip(candidates, results)} selections = [] for target in targets: - stripped_class = classes[target, "stripped"] - unstripped_class = classes[target, "unstripped"] - chosen = "stripped" if stripped_class == "clean" or unstripped_class != "clean" else "unstripped" + order = ("stripped", "debug-stripped", "unstripped") + clean = [v for v in order if classes[target, v] == "clean"] + chosen = clean[0] if clean else "stripped" selected = by_pair[target, chosen] - decision = ( - "unstripped-clean-after-stripped-microsoft-ml" if chosen == "unstripped" - else "stripped-both-microsoft-ml" if stripped_class == unstripped_class == "microsoft-ml" - else "stripped-preferred" - ) + if not clean: + decision = "stripped-all-candidates-microsoft-ml" + elif chosen == "stripped": + decision = "stripped-preferred" + else: + others = "-".join(f"{v}:{classes[target, v]}" for v in order if v != chosen) + decision = f"{chosen}-clean-after-{others}" row = { "target": target, "selected_variant": chosen, "selected_path": f"selected/{target}/" + ("codebase-memory-mcp.exe" if target.startswith("windows-") else "codebase-memory-mcp"), @@ -117,21 +124,21 @@ for target in targets: candidate = by_pair[target, variant] result = result_by_pair[target, variant] row.update({ - f"{variant}_sha256": candidate["sha256"], - f"{variant}_scan_path": candidate["scan_path"], - f"{variant}_classification": result["policy_classification"], - f"{variant}_analysis_id": result["analysis_id"], - f"{variant}_virustotal_url": result["virustotal_url"], + f"{field_key[variant]}_sha256": candidate["sha256"], + f"{field_key[variant]}_scan_path": candidate["scan_path"], + f"{field_key[variant]}_classification": result["policy_classification"], + f"{field_key[variant]}_analysis_id": result["analysis_id"], + f"{field_key[variant]}_virustotal_url": result["virustotal_url"], }) selections.append(row) write(root / "release-candidates.tsv", "cbm-release-candidates-v1", - {"targets": 8, "candidates": 16}, candidate_fields, candidates) + {"targets": 8, "candidates": 24}, candidate_fields, candidates) write(root / "virustotal-candidate-results.tsv", "cbm-virustotal-results-v2", - {"scan_objects": 16, "associations": 16, "min_engines_policy": 50, + {"scan_objects": 24, "associations": 24, "min_engines_policy": 50, "min_completed_engines": 61, "max_completed_engines": 61}, result_fields, results) write(root / "release-selection.tsv", "cbm-release-selection-v1", - {"policy": "virustotal-v2", "targets": 8, "candidates": 16}, + {"policy": "virustotal-v2", "targets": 8, "candidates": 24}, selection_fields, selections) PY @@ -183,7 +190,7 @@ run_notes "$FIX/current.md" "$FIX/first.md" grep -q 'Intro text.' "$FIX/first.md" || fail "content before section was lost" grep -q 'Outro text.' "$FIX/first.md" || fail "content after section was lost" ! grep -q 'stale data' "$FIX/first.md" || fail "stale section was appended" -grep -q '16 executable scans' "$FIX/first.md" || fail "candidate scan scope missing" +grep -q '24 executable scans' "$FIX/first.md" || fail "candidate scan scope missing" grep -q '8 release products' "$FIX/first.md" || fail "product scope missing" grep -q '3 candidate(s).*Microsoft' "$FIX/first.md" || fail "tolerated findings not disclosed" grep -q 'no other decisive engine reported malicious or suspicious' "$FIX/first.md" || fail "non-decisive engines are overstated as clean" @@ -221,4 +228,4 @@ if run_notes "$FIX/reversed.md" "$FIX/reversed-capture.md"; then fail "reversed verification markers must fail closed" fi -echo 'PASS: release notes reuse all 16 candidate verdicts, disclose selection, and avoid a duplicate scan' +echo 'PASS: release notes reuse all 24 candidate verdicts, disclose selection, and avoid a duplicate scan'