nprasad2077/doris

0 Fork 0

mirror of https://github.com/apache/doris.git synced 2026-11-04 07:31:17 +00:00

Files

T

HappenLee f40c80dbf8 [improvement](tools) Run SSB, TPCH and TPCDS benchmarks with one command (#68278 )

### What problem does this PR solve?

Issue Number: None

The existing SSB, TPCH and TPCDS tools require generating local data
files and
running separate creation, import, statistics and benchmark commands.
Add one
entry point per suite, backed by a shared runner:

```bash
tools/ssb-tools/bin/run-ssb.sh -s 1 -d ssb_sf1
tools/tpch-tools/bin/run-tpch.sh -s 1 -d tpch_sf1
tools/tpcds-tools/bin/run-tpcds.sh -s 1 -d tpcds_sf1
```

Rows are generated on demand through Doris's Trino connector and
imported with
`INSERT INTO ... SELECT ...`, without intermediate data files or a
separate
Trino server. SSB uses the included native-dbgen plugin. TPCH/TPCDS use
complete
upstream Trino 435 plugin distributions, pinned by SHA-256 and
downloaded during
operator installation. Plugins must be installed on every FE and BE
before use.

The runner executes the original scale-specific table DDL and query SQL:
- SSB prepares its five base tables and supports the original
`lineorder_flat`
workflow through `--mode ssb|flat|both`. Both suites have 13 query
files.
- TPCH prepares its eight base tables and `revenue0` view and runs 22
query files.
- TPCDS prepares its 24 base tables and runs 103 scale-specific query
files,
  including the second statements of templates 14, 23, 24 and 39.

Full statistics are collected synchronously before query timing. Every
query
runs three times with SQL and BE query result caches disabled.
`--queries-only`
repeats measurements against already prepared data. Each run preserves
SQL
results, errors, preparation logs, settings and CSV timings in a fresh
directory.

Parallel TPCH/TPCDS tables default to `max(10, SCALE)` generator splits:
10/100/1000/10000 for SF1/100/1000/10000. `--splits COUNT` overrides
that count;
invalid values fail before connecting. TPCH region/nation and TPCDS
tables below
the pinned generator's one-million-record parallelization threshold use
one
split, including with an override. Small tables share a single-split
catalog;
`--splits 1` reuses the main catalog. Per-table effective counts are
logged.
Preparation stderr is saved in `prepare.log`, while failures still print
the
log directory to the terminal. SSB preserves the old generator's ten
lineorder partitions, enables its existing 64-bit path and fixes SSB
random
stream usage-counter bounds. Each SSB scan owns and closes its generator
process.

Imports explicitly project target column names, including TPCH's
reordered
lineitem columns and Trino 435's `p_response_targe` spelling for TPCDS
promotion.
TPCH uses standard prefixed column names and DECIMAL values. A fresh
database
prevents duplicate appends on retries. INSERTs use strict mode with zero
filtered
rows and must return a metadata `status` of `VISIBLE`. SQL text or
labels
containing `COMMITTED` cannot cause a false failure; missing or
non-visible
status stops preparation. Query errors stop the suite and are not
recorded as
successful timings.

2026-10-10 18:03:42 +08:00

2.7 KiB

Raw Permalink Blame History

Run TPCH with one command

Install the Trino TPCH generator on every FE and BE using the shared installation instructions, then configure conf/doris-cluster.conf and run:

./bin/run-tpch.sh -s 1 -d tpch_sf1

This streams generated rows directly into Doris, creates the existing benchmark tables, collects statistics, and records three executions per query. It supports SF1, SF100, SF1000, and SF10000. Preparation requires a new database. Use --queries-only to measure existing data again, with the same scale factor.

See shared workflow options, result files, and timing semantics. TPCH uses the original scale-specific table DDL and multi-table queries. Install the plugin before running the command.

Original file-based workflow

Usage

These scripts are used to make tpc-h test. follow the steps below:

1. build tpc-h dbgen tool.

./bin/build-tpch-dbgen.sh

If the build failed in dbgen tools' compilation, update your GCC version or change download link used by wget in build-tpch-dbgen.sh from ".../TPC-H_Tools_v3.0.0new.zip" to ".../TPC-H_Tools_v3.0.0.zip"

2. generate tpc-h data. use -h for more infomations.

./bin/gen-tpch-data.sh -s 1

3. create tpc-h tables. modify `conf/doris-cluster.conf` to specify doris info, then run script below.

./bin/create-tpch-tables.sh -s 1

4. load tpc-h data. use -h for help.

./bin/load-tpch-data.sh

5. run tpc-h queries.

./bin/run-tpch-queries.sh -s 1

NOTICE: At present, Doris's query optimizer and statistical information functions are not complete, so we rewrite some queries in TPC-H to adapt to Doris' execution framework, but it does not affect the correctness of the results. The rewritten SQL is marked with "Modified" in the corresponding .sql file.

A new query optimizer will be released in subsequent releases.

Currently, differnt scales use the same suite of query sqls.

2.7 KiB Raw Permalink Blame History