HappenLee
f40c80dbf8
### What problem does this PR solve? Issue Number: None The existing SSB, TPCH and TPCDS tools require generating local data files and running separate creation, import, statistics and benchmark commands. Add one entry point per suite, backed by a shared runner: ```bash tools/ssb-tools/bin/run-ssb.sh -s 1 -d ssb_sf1 tools/tpch-tools/bin/run-tpch.sh -s 1 -d tpch_sf1 tools/tpcds-tools/bin/run-tpcds.sh -s 1 -d tpcds_sf1 ``` Rows are generated on demand through Doris's Trino connector and imported with `INSERT INTO ... SELECT ...`, without intermediate data files or a separate Trino server. SSB uses the included native-dbgen plugin. TPCH/TPCDS use complete upstream Trino 435 plugin distributions, pinned by SHA-256 and downloaded during operator installation. Plugins must be installed on every FE and BE before use. The runner executes the original scale-specific table DDL and query SQL: - SSB prepares its five base tables and supports the original `lineorder_flat` workflow through `--mode ssb|flat|both`. Both suites have 13 query files. - TPCH prepares its eight base tables and `revenue0` view and runs 22 query files. - TPCDS prepares its 24 base tables and runs 103 scale-specific query files, including the second statements of templates 14, 23, 24 and 39. Full statistics are collected synchronously before query timing. Every query runs three times with SQL and BE query result caches disabled. `--queries-only` repeats measurements against already prepared data. Each run preserves SQL results, errors, preparation logs, settings and CSV timings in a fresh directory. Parallel TPCH/TPCDS tables default to `max(10, SCALE)` generator splits: 10/100/1000/10000 for SF1/100/1000/10000. `--splits COUNT` overrides that count; invalid values fail before connecting. TPCH region/nation and TPCDS tables below the pinned generator's one-million-record parallelization threshold use one split, including with an override. Small tables share a single-split catalog; `--splits 1` reuses the main catalog. Per-table effective counts are logged. Preparation stderr is saved in `prepare.log`, while failures still print the log directory to the terminal. SSB preserves the old generator's ten lineorder partitions, enables its existing 64-bit path and fixes SSB random stream usage-counter bounds. Each SSB scan owns and closes its generator process. Imports explicitly project target column names, including TPCH's reordered lineitem columns and Trino 435's `p_response_targe` spelling for TPCDS promotion. TPCH uses standard prefixed column names and DECIMAL values. A fresh database prevents duplicate appends on retries. INSERTs use strict mode with zero filtered rows and must return a metadata `status` of `VISIBLE`. SQL text or labels containing `COMMITTED` cannot cause a false failure; missing or non-visible status stops preparation. Query errors stop the suite and are not recorded as successful timings.
5.3 KiB
Run SSB with one command
The Trino SSB connector generates rows while Doris reads them. It uses the same
ssb-dbgen distribution as the existing tools, with stdout connected to the scan
through a bounded pipe. There are no intermediate .tbl files, Stream Load
requests, or separate Trino servers.
Install the generator plugin once
Build on Linux with the same CPU architecture as the Doris nodes. The build needs JDK 17, Maven, GCC, Make, curl, patch, and sha256sum:
./trino-ssb/build.sh
Copy the resulting entire directory trino-ssb/target/ssb into the configured
Trino plugin directory on every FE and BE. For current Doris installations:
cp -r trino-ssb/target/ssb /path/to/fe/plugins/trino_plugins/
cp -r trino-ssb/target/ssb /path/to/be/plugins/trino_plugins/
Start or restart these nodes after installing the plugin. It consists of a JAR,
the native dbgen executable, and its distribution file. The executable must
remain executable. No paths or executable commands are accepted as catalog properties.
Create, import, and benchmark
Set the Doris connection in conf/doris-cluster.conf, then run:
./bin/run-ssb.sh -s 1 -d ssb_sf1
This creates a new database and its ssb_gen_ssb_sf1 catalog, creates the five
SSB tables, and imports them using INSERT INTO ... SELECT ... from the catalog's
sf1 schema. It then builds lineorder_flat, collects statistics synchronously,
and runs all 13 SSB queries and all 13 SSB Flat queries three times each.
The supported scale factors are 1, 100, and 1000. Use --mode ssb or --mode flat
to select one benchmark; the default is both.
Each run writes a new results/<timestamp>-<pid>/ directory with:
result.csv: first-run, two repeated-run, and best repeated-run times in milliseconds.ssb/andssb-flat/: each query's output and errors.prepare.logandenvironment.txt: import diagnostics and Doris settings.
Timing includes the mysql client connection and result transfer, as in the old
scripts. The cold column means the first execution; the script does not flush
data caches. SQL result caching and BE query caching are disabled for every measured
session. A query or import error stops the run with a nonzero exit status.
Preparation requires a new database, so rerunning a failed or completed load cannot append duplicate data. Tables and the catalog are preserved for inspection. To rerun only the queries against completed data:
./bin/run-ssb.sh -d ssb_sf1 --queries-only
Use -c /path/to/cluster.conf for a separate connection configuration, and
--result-dir /path/to/new-directory for a custom output directory. Existing
result directories are never overwritten.
Generator consistency and tests
The connector keeps the old tools' default of ten lineorder generator partitions.
Each dimension uses one generator. Changing the original dbgen partition count
changes some random streams, so partition count is intentionally fixed here.
The date generator runs in UTC on every BE. The native build enables dbgen's
existing 64-bit code path for large scale factors and avoids an out-of-bounds
usage-counter update for its SSB-only random streams. Neither change replaces
the data-generation algorithm.
The existing benchmark SQL is preserved. In particular, SSB Q1.3 filters the
generator's d_weeknuminyear, whereas SSB Flat Q1.3 uses weekofyear(); these week
definitions differ, so their aggregate results are not expected to match.
The plugin build runs its Java tests, including full SF1 checksums of all five tables, handle serialization, projected columns, large keys, producer failures, and early cursor closure. Run script orchestration tests with:
python3 -m unittest discover -s ../benchmark/tests -v
The shared runner also supports TPCH and TPCDS.
Original file-based workflow
These scripts are used to make ssb and ssb flat test.
The ssb flat data comes from ssb tables by way of 'INSERT INTO ... SELECT ...'.
ssb test, follow the steps below:
1. build ssb dbgen tool.
./bin/build-ssb-dbgen.sh
2. generate ssb data. use -h for more infomations.
./bin/gen-ssb-data.sh -s 1
3. create ssb tables. modify conf/doris-cluster.conf to specify Doris cluster info, then run script below.
./bin/create-ssb-tables.sh -s 1
4. load ssb data. use -h for help.
./bin/load-ssb-data.sh
5. run ssb queries.
./bin/run-ssb-queries.sh
ssb flat test, follow the steps below:
1. prepare ssb data, which means ssb test steps 1 to 4 have been done.
2. run ssb flat queries.
./bin/run-ssb-flat-queries.sh