mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
HappenLee
f40c80dbf8
### What problem does this PR solve? Issue Number: None The existing SSB, TPCH and TPCDS tools require generating local data files and running separate creation, import, statistics and benchmark commands. Add one entry point per suite, backed by a shared runner: ```bash tools/ssb-tools/bin/run-ssb.sh -s 1 -d ssb_sf1 tools/tpch-tools/bin/run-tpch.sh -s 1 -d tpch_sf1 tools/tpcds-tools/bin/run-tpcds.sh -s 1 -d tpcds_sf1 ``` Rows are generated on demand through Doris's Trino connector and imported with `INSERT INTO ... SELECT ...`, without intermediate data files or a separate Trino server. SSB uses the included native-dbgen plugin. TPCH/TPCDS use complete upstream Trino 435 plugin distributions, pinned by SHA-256 and downloaded during operator installation. Plugins must be installed on every FE and BE before use. The runner executes the original scale-specific table DDL and query SQL: - SSB prepares its five base tables and supports the original `lineorder_flat` workflow through `--mode ssb|flat|both`. Both suites have 13 query files. - TPCH prepares its eight base tables and `revenue0` view and runs 22 query files. - TPCDS prepares its 24 base tables and runs 103 scale-specific query files, including the second statements of templates 14, 23, 24 and 39. Full statistics are collected synchronously before query timing. Every query runs three times with SQL and BE query result caches disabled. `--queries-only` repeats measurements against already prepared data. Each run preserves SQL results, errors, preparation logs, settings and CSV timings in a fresh directory. Parallel TPCH/TPCDS tables default to `max(10, SCALE)` generator splits: 10/100/1000/10000 for SF1/100/1000/10000. `--splits COUNT` overrides that count; invalid values fail before connecting. TPCH region/nation and TPCDS tables below the pinned generator's one-million-record parallelization threshold use one split, including with an override. Small tables share a single-split catalog; `--splits 1` reuses the main catalog. Per-table effective counts are logged. Preparation stderr is saved in `prepare.log`, while failures still print the log directory to the terminal. SSB preserves the old generator's ten lineorder partitions, enables its existing 64-bit path and fixes SSB random stream usage-counter bounds. Each SSB scan owns and closes its generator process. Imports explicitly project target column names, including TPCH's reordered lineitem columns and Trino 435's `p_response_targe` spelling for TPCDS promotion. TPCH uses standard prefixed column names and DECIMAL values. A fresh database prevents duplicate appends on retries. INSERTs use strict mode with zero filtered rows and must return a metadata `status` of `VISIBLE`. SQL text or labels containing `COMMITTED` cannot cause a false failure; missing or non-visible status stops preparation. Query errors stop the suite and are not recorded as successful timings.
Run TPCH with one command
Install the Trino TPCH generator on every FE and BE using the
shared installation instructions,
then configure conf/doris-cluster.conf and run:
./bin/run-tpch.sh -s 1 -d tpch_sf1
This streams generated rows directly into Doris, creates the existing benchmark
tables, collects statistics, and records three executions per query. It supports
SF1, SF100, SF1000, and SF10000. Preparation requires a new database.
Use --queries-only to measure existing data again, with the same scale factor.
See shared workflow options, result files, and timing semantics. TPCH uses the original scale-specific table DDL and multi-table queries. Install the plugin before running the command.
Original file-based workflow
Usage
These scripts are used to make tpc-h test. follow the steps below:
1. build tpc-h dbgen tool.
./bin/build-tpch-dbgen.sh
If the build failed in dbgen tools' compilation, update your GCC version or change download link used by wget in build-tpch-dbgen.sh from ".../TPC-H_Tools_v3.0.0new.zip" to ".../TPC-H_Tools_v3.0.0.zip"
2. generate tpc-h data. use -h for more infomations.
./bin/gen-tpch-data.sh -s 1
3. create tpc-h tables. modify conf/doris-cluster.conf to specify doris info, then run script below.
./bin/create-tpch-tables.sh -s 1
4. load tpc-h data. use -h for help.
./bin/load-tpch-data.sh
5. run tpc-h queries.
./bin/run-tpch-queries.sh -s 1
NOTICE: At present, Doris's query optimizer and statistical information functions are not complete, so we rewrite some queries in TPC-H to adapt to Doris' execution framework, but it does not affect the correctness of the results. The rewritten SQL is marked with "Modified" in the corresponding .sql file.
A new query optimizer will be released in subsequent releases.
Currently, differnt scales use the same suite of query sqls.