[test](variant) refine google benchmark test (#67551)

be/benchmark/benchmark_main.cpp

@@ -34,7 +34,6 @@
 #include "benchmark_hll_merge.hpp"
 #include "benchmark_hybrid_set.hpp"
 #include "benchmark_json_extract.hpp"
-#include "benchmark_variant_segment.hpp"
 #include "benchmark_zone_map_index.hpp"
 #include "binary_cast_benchmark.hpp"
 #include "common/config.h"
@@ -51,6 +50,7 @@
 #include "runtime/memory/mem_tracker_limiter.h"
 #include "runtime/memory/thread_mem_tracker_mgr.h"
 #include "runtime/thread_context.h"
+#include "variant/benchmark_variant_segment.hpp"
 
 namespace doris { // change if need
 

be/benchmark/variant/BASELINE.md

@@ -0,0 +1,90 @@
+# Recorded baseline: 2026-09-17
+
+A full run of both suites (`^BM_Variant`, every registered case, default 1,000,000 rows) was
+executed three times back to back on a shared build host to record a baseline and to measure
+run-to-run noise from other jobs on the same machine.
+
+## How it was run
+
+```shell
+source custom_env.sh
+DORIS_BENCHMARK_CPU=16 bash be/benchmark/variant/run_variant_segment_benchmark.sh \
+    be/output/lib/benchmark_test <result_dir> '^BM_Variant'
+```
+
+- Commit: `adfb3451eaeafb1c3fbb4e437cf2931cdb74a57b` (this PR branch, merged with master
+  2026-09-15; working tree otherwise clean — `build.sh` differs only because of local worktree
+  tooling, unrelated to this code)
+- Build: `RELEASE`, `./build.sh --be --benchmark`
+- Host: shared 192-thread build machine, 2x Intel(R) Xeon(R) Platinum 8457C (48 cores/socket, 2
+  threads/core), pinned to **one** logical CPU (`DORIS_BENCHMARK_CPU=16` → `taskset -c 16`). All
+  `Writers8` threads inherit that single-CPU mask, so they interleave rather than run
+  simultaneously; this run cannot demonstrate cross-core contention (see the note under
+  `BM_VariantSparseImport` results below)
+- Rows: default (1,000,000)
+- Each round runs all 35 `BM_VariantCumulativeCompaction` cases (7 scenarios x 5 samples) and all
+  20 `BM_VariantSparseImport` cases (4 scenarios x 5 samples) — 55 cases/round, 165 total, 0
+  `error_occurred`.
+
+## Machine-load impact
+
+This is a **shared, heavily loaded** host, not an isolated benchmark machine. `/proc/loadavg`
+around each round (192 logical CPUs):
+
+| Round | load before | load after |
+|---|---|---|
+| 1 | 129.6 | 85.3 |
+| 2 | 97.3 | 122.2 (peaked at 349 during the run, per round 3's "before") |
+| 3 | 122.2 | 55.9 |
+
+Round 2 ran while background load spiked well above the other two rounds, and it measured
+20-100%+ slower across almost every scenario, e.g.:
+
+- `Doc16/WholeVariant`: ~16-18s (rounds 1, 3) vs. ~37s (round 2)
+- `MixedTypes/Writers1`: ~10.8 CPU-s/1M rows (rounds 1, 3) vs. ~24.6 (round 2)
+
+Rounds 1 and 3 agree with each other within roughly 20%. **Treat single-sample numbers on a shared
+host as unreliable**; compare medians across several rounds, and prefer an idle/pinned host when
+the comparison matters (e.g. deciding whether a change is a regression).
+
+## Results: `BM_VariantCumulativeCompaction` (`real_time`, ms; lower is faster)
+
+| Scenario | Round 1 median | Round 2 median | Round 3 median | All 3 rounds median (n=15) | Rounds 1+3 median (n=10, lower load) |
+|---|---:|---:|---:|---:|---:|
+| Sparse16/FlatPhysicalColumns | 7,830 | 9,668 | 6,257 | 8,177 | 7,305 |
+| Sparse16/MixedPhysicalPlacement | 7,577 | 10,364 | 6,824 | 7,577 | 7,216 |
+| Sparse16/WholeVariant | 16,076 | 16,648 | 15,877 | 16,123 | 15,977 |
+| Full/WholeVariant | 16,762 | 18,868 | 16,042 | 16,762 | 16,154 |
+| Doc16/DocBuckets | 18,847 | 32,732 | 19,069 | 26,066 | 18,958 |
+| Doc16/DocBucketsMaterialized | 30,530 | 43,819 | 22,102 | 30,530 | 22,930 |
+| Doc16/WholeVariant | 17,591 | 37,482 | 16,179 | 19,411 | 17,200 |
+
+## Results: `BM_VariantSparseImport` (`cpu_s_per_1m_rows`; lower is faster)
+
+| Scenario | Round 1 median | Round 2 median | Round 3 median | All 3 rounds median (n=15) | Rounds 1+3 median (n=10, lower load) |
+|---|---:|---:|---:|---:|---:|
+| MixedTypes/Writers1 | 10.831 | 24.594 | 10.880 | 11.151 | 10.855 |
+| NoArrays/Writers1 | 9.006 | 18.820 | 11.637 | 11.637 | 9.094 |
+| MixedTypes/Writers8 | 15.543 | 24.367 | 13.791 | 15.543 | 14.378 |
+| NoArrays/Writers8 | 10.810 | 13.407 | 10.617 | 10.816 | 10.671 |
+
+`MixedTypes/Writers8` costing more than `MixedTypes/Writers1` per-writer CPU time, and both
+`NoArrays` variants sitting below their `MixedTypes` counterparts, is a valid same-scenario
+comparison: `cpu_s_per_1m_rows` sums each writer's own thread-CPU time regardless of how they were
+scheduled, so it stands on its own. **This run cannot, however, be used as evidence of cross-core
+contention.** This run pinned all 8 `Writers8` threads to a single logical CPU (`taskset -c 16`),
+so they were interleaved, never running simultaneously — the kind of shared-reference/cache-line
+contention this benchmark is meant to expose only shows up across distinct cores. Attributing the
+Writers1-vs-Writers8 gap to that contention would require rerunning with the threads bound to at
+least 8 distinct physical CPUs (and recording the topology), which this baseline does not do.
+
+## Caveats
+
+- These absolute numbers are specific to this shared host, this build, and this moment's
+  contention; do not port them to a different machine or compare across a build-type change.
+- The "rounds 1+3" column is a best-effort lower-noise reference (excludes the visibly load-spiked
+  round 2), not a clean isolated-host measurement. Even rounds 1 and 3 individually show real
+  variance (e.g. `Doc16/DocBucketsMaterialized` min/max across all 15 samples was 21.4s/97.6s).
+- Full `raw.json` / `environment.txt` / `stdout.txt` / `stderr.txt` / `source.diff` for each round
+  are not committed (regenerate with the command above); this file records the derived medians
+  only.

be/benchmark/variant/README.md

@@ -0,0 +1,100 @@
+# Variant benchmarks
+
+Two Google Benchmark suites live here:
+
+- `BM_VariantCumulativeCompaction`: a source-version-neutral, high-level cumulative compaction
+  workload. It writes 10 overlapping rowsets of synthetic Variant rows, runs a real
+  `Compaction`/`VerticalSegmentWriter` merge over them, and validates the merged output's semantics
+  and physical layout (materialized/sparse/doc-bucket subcolumn counts) before reporting timing.
+  It has no Variant-version selector: the same workload should be comparable commit-to-commit as
+  the underlying storage format evolves.
+- `BM_VariantSparseImport`: an import workload for the segment writer's Variant shredder, which the
+  compaction workload does not reach because it only writes flat BIGINT objects. Each row carries a
+  Poisson(20) subset (clipped to [5, 60]) of 2,000 possible keys, with a key's value kind fixed by
+  its index (50% BIGINT, 25% short strings, 10% doubles, 5% booleans, 5% BIGINT arrays of 1-4
+  elements, 5% `{"x": BIGINT, "y": string}` objects) — matching the dataset used for end-to-end JSON
+  and Parquet VARIANT import measurements. Input generation and `parse_to_variant` are untimed; the
+  timed region is `VerticalSegmentWriter` init/append/finalize, which shreds every value through the
+  Variant path builders.
+
+The files are:
+
+- `benchmark_variant_segment.hpp`: both workloads' scenario definitions, synthetic data generation,
+  correctness checks, and Google Benchmark registration.
+- `run_variant_segment_benchmark.sh`: wraps a benchmark run with an `environment.txt` snapshot
+  (git commit, dirty diff, thirdparty fingerprints, host/CPU/memory info) so results can be tied to
+  the exact code and machine that produced them, plus JSON output for later parsing.
+- `README.md`: this file.
+- `BASELINE.md`: a recorded baseline run with reproduction steps, for commit-to-commit comparison.
+
+## Build
+
+Performance results must come from a Release build:
+
+```shell
+./build.sh --be --benchmark
+```
+
+This installs `be/output/lib/benchmark_test`.
+
+## List and run cases
+
+```shell
+be/output/lib/benchmark_test --benchmark_list_tests | grep '^BM_Variant'
+```
+
+`BM_VariantCumulativeCompaction` registers 7 scenarios x 5 samples = 35 cases:
+
+| Scenario | Layout | Notes |
+|---|---|---|
+| `Sparse16/FlatPhysicalColumns` | 16-bucket sparse | subcolumns compacted, flat workload |
+| `Sparse16/MixedPhysicalPlacement` | 16-bucket sparse | subcolumns compacted, mixed physical placement workload |
+| `Sparse16/WholeVariant` | 16-bucket sparse | whole-Variant column compacted (no subcolumn compaction) |
+| `Full/WholeVariant` | fully materialized | whole-Variant column compacted |
+| `Doc16/DocBuckets` | 16-bucket Doc Mode | doc-bucket subcolumns compacted |
+| `Doc16/DocBucketsMaterialized` | 16-bucket Doc Mode | doc buckets compacted and materialized |
+| `Doc16/WholeVariant` | 16-bucket Doc Mode | whole-Variant column compacted |
+
+`BM_VariantSparseImport` registers 4 scenarios x 5 samples = 20 cases: `MixedTypes` vs. `NoArrays`
+(arrays replaced with BIGINT, isolating per-array type inference) crossed with `Writers1` vs.
+`Writers8` (1 vs. 8 concurrent `VerticalSegmentWriter`s, matching parallel memtable flushes of
+several tablets — this contends process-wide shared state such as static `DataTypePtr` reference
+counts the way a real import does).
+
+Run the full suite and capture an environment-tagged, reproducible result:
+
+```shell
+source custom_env.sh   # sets JAVA_HOME
+bash be/benchmark/variant/run_variant_segment_benchmark.sh \
+    be/output/lib/benchmark_test <result_dir> '^BM_Variant'
+```
+
+`<result_dir>/raw.json` has the Google Benchmark JSON; `<result_dir>/environment.txt` has the
+commit, working-tree diff, thirdparty fingerprints, and host info needed to interpret it. Both
+suites read `DORIS_VARIANT_BENCHMARK_ROWS` (default 1,000,000) for the row count and
+`DORIS_VARIANT_BENCHMARK_ROOT` (default `/tmp`) for scratch space; `DORIS_BENCHMARK_CPU` pins the
+run with `taskset` when set.
+
+To run only one suite, pass its own filter as the third argument, e.g.
+`'^BM_VariantSparseImport/'`.
+
+## Interpret the result
+
+- `BM_VariantCumulativeCompaction` cases are `->UseManualTime()` with a single iteration per
+  registered case (`sample1`..`sample5` are separate named cases, not Google Benchmark
+  repetitions); use `real_time` in milliseconds. Lower is faster.
+- `BM_VariantSparseImport` reports `cpu_s_per_1m_rows` (also `append_ns_per_row`,
+  `finalize_ns_per_row`, `segment_bytes_per_row`, `input_json_bytes_per_row`) alongside
+  `real_time`/`cpu_time`. `cpu_s_per_1m_rows` is thread CPU time summed across writers, so
+  `Writers8` is expected to cost more than `Writers1` even with no regression; compare `Writers8` to
+  `Writers8` and `Writers1` to `Writers1` across commits, not to each other.
+- Both suites validate output before returning: a compaction case checks the merged segment's
+  subcolumn layout and rewritten values, and an import case checks each writer's row/segment count.
+  A failing case aborts benchmark execution (`error_occurred` in the JSON) rather than silently
+  reporting bad numbers.
+- This is a workload-level benchmark against real `Compaction`/`VerticalSegmentWriter` code, not a
+  microbenchmark of an isolated function — treat regressions as signals to investigate with a
+  profiler, not as a precise attribution of which code path changed.
+- On a shared/busy host, run-to-run noise can be large (see `BASELINE.md` for a measured example).
+  Prefer the median of several samples, and re-run before concluding a change regressed or improved
+  performance.

be/benchmark/variant/run_variant_segment_benchmark.sh

@@ -26,7 +26,7 @@ fi
 
 benchmark_binary=$1
 result_dir=$2
-benchmark_filter=${3:-'^BM_Variant(IngestToSegment|ReadWholeColumn|ReadExactPath|ScanAndRewriteSegment)/'}
+benchmark_filter=${3:-'^BM_VariantCumulativeCompaction/'}
 script_path=$(realpath "${BASH_SOURCE[0]}")
 repo_root=$(git -C "$(dirname "${script_path}")" rev-parse --show-toplevel)
 be_config=${repo_root}/conf/be.conf
@@ -57,25 +57,39 @@ mkdir -p "${result_dir}"
     echo "repo_root=${repo_root}"
     echo "invocation_pwd=${PWD}"
     echo "git_head=$(git -C "${repo_root}" rev-parse HEAD)"
+    echo "git_parent=$(git -C "${repo_root}" rev-parse HEAD^)"
+    echo "git_describe=$(git -C "${repo_root}" describe --always --dirty)"
     git -C "${repo_root}" status --short --branch
     echo "benchmark_binary=${benchmark_binary}"
     sha256sum "${benchmark_binary}"
+    echo "benchmark_binary_verified_against_source=no (this hash only identifies the binary; it is not independently checked against git_head/source.diff/thirdparty fingerprints below -- retain this exact binary alongside this environment.txt for a trustworthy comparison)"
     echo "JAVA_HOME=${JAVA_HOME}"
     echo "runtime_LD_LIBRARY_PATH=${runtime_library_path}"
     LD_LIBRARY_PATH="${runtime_library_path}" ldd "${benchmark_binary}"
     echo "be_config=${be_config}"
     sha256sum "${be_config}"
-    echo "variant_storage_parse_mode_config:"
-    grep -nE '^[[:space:]]*variant_storage_parse_mode[[:space:]]*=' "${be_config}" ||
-        echo "<compiled default>"
+    echo "benchmark_source=be/benchmark/variant/benchmark_variant_segment.hpp"
+    sha256sum "${repo_root}/be/benchmark/variant/benchmark_variant_segment.hpp"
     echo "thirdparty_installed=${thirdparty_installed_path}"
     while IFS= read -r fingerprint; do
         echo "$(basename "${fingerprint}")=$(< "${fingerprint}")"
     done < <(find "${thirdparty_installed_path}" -maxdepth 1 -type f \
         -name '*-build-fingerprint.txt' | sort)
-    if [[ -f "${repo_root}/be/build_Release/CMakeCache.txt" ]]; then
-        sha256sum "${repo_root}/be/build_Release/CMakeCache.txt"
-        grep -n '^CMAKE_BUILD_TYPE:' "${repo_root}/be/build_Release/CMakeCache.txt"
+    # ./build.sh --benchmark without an explicit BUILD_TYPE can name the build directory either
+    # build_RELEASE (BUILD_TYPE=RELEASE, as this script's own instructions set) or build_Release
+    # (build.sh's internal default casing); probe both so the documented bare invocation is still
+    # covered.
+    cmake_cache_found=0
+    for cmake_build_dir in "${repo_root}/be/build_RELEASE" "${repo_root}/be/build_Release"; do
+        if [[ -f "${cmake_build_dir}/CMakeCache.txt" ]]; then
+            sha256sum "${cmake_build_dir}/CMakeCache.txt"
+            grep -n '^CMAKE_BUILD_TYPE:' "${cmake_build_dir}/CMakeCache.txt"
+            cmake_cache_found=1
+            break
+        fi
+    done
+    if [[ "${cmake_cache_found}" -eq 0 ]]; then
+        echo "cmake_cache=not found under be/build_RELEASE or be/build_Release"
     fi
     echo "benchmark_root=${benchmark_root}"
     df -h "${benchmark_root}"
@@ -87,7 +101,6 @@ mkdir -p "${result_dir}"
     lscpu
     echo "load_before=$(< /proc/loadavg)"
     echo "rows=${DORIS_VARIANT_BENCHMARK_ROWS:-1000000}"
-    echo "rows_per_segment=${DORIS_VARIANT_BENCHMARK_ROWS_PER_SEGMENT:-1000000}"
     echo "filter=${benchmark_filter}"
     echo "cpu=${DORIS_BENCHMARK_CPU:-unbound}"
 } >"${result_dir}/environment.txt"