mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Jack
14fde1c1dc
### What problem does this PR solve?
Issue Number: None
Related PR: #67180, #67538, #67918
Problem Summary:
MATCH and SEARCH each had their own Boolean, phrase, term-expansion and
scoring executors, written separately for CLucene (V1/V2) and SNII
indexes. Every query optimization had to be implemented once per
executor, the formats drifted apart in behavior, and cross-field queries
could mix document IDs from different physical partitions.
This PR runs both on one engine:
- **One logical form.** MATCH (`query/logical`) and SEARCH
(`exprs/function/search`) lower their queries to the same logical nodes.
- **One query engine.** Boolean logic (AND/OR/NOT, minimum_should_match,
WAND), term expansion (prefix, wildcard, regexp), exact and sloppy
phrases and BM25 scoring are implemented once, independent of the index
format.
- **Formats behind an SPI.** CLucene and SNII only provide dictionaries,
postings, positions, norms, statistics and batched I/O through
`query/spi`. The old per-format executors are removed.
- **Layered source tree.** `be/src/storage/index/` is reorganized by
layer, and `build-support/check-header-deps.py` enforces the allowed
include directions.
Index formats and writers are unchanged, and existing indexes remain
readable. FE is unchanged.
#### Source layout
```text
be/src/storage/index/
├── index_*.h/.cpp shared index files: file reader/writer, compound reader, storage formats
├── ann/ bloom_filter/ zone_map/
└── inverted/
├── analysis/ analyzers, tokenizers, char and token filters, normalizers, gram schemes
├── common/ descriptors, caches, statistics, profiles, single flight
├── reader/ text index reader, index selection, collection statistics
├── query/ format-independent query engine
│ ├── logical/ MATCH lowering and logical nodes (gram/ builds gram queries)
│ ├── engine/ Query/Weight/Scorer: boolean, term, phrase, expansion, WAND, union, collect
│ ├── exec/ doc-id sets and sinks, term waves, chained conjunctions
│ ├── phrase/ exact and sloppy phrase matchers
│ ├── scoring/ BM25
│ └── spi/ the source interface each format implements
├── clucene/ CLucene (V1/V2) reader, searcher, writer, compaction
│ └── spi/ CLucene index source and postings cursor
└── snii/ SNII: format/, encoding/, io/, bkd/, reader/, writer/, compaction/, query/
be/src/exprs/
├── vsearch.h/.cpp VSearchExpr, the SEARCH expression node
└── function/search/
├── function_search.* SEARCH entry: Boolean assembly, DSL result cache, NULL handling
├── search_lowering.* SEARCH DSL to logical nodes; binds each field to an index
├── function_search_leaf.* compiles each leaf into a query on its bound index
├── search_predicate_collector.* collects scored leaves for BM25 statistics
└── variant_inverted_index_search.* SEARCH on VARIANT subcolumns and NESTED groups
```
#### Performance
The 18 production SQL cases (V2 and SNII: point, range, absent-term and
phrase queries) were compared with master on fresh dual-BE pairs: 8
pairs, 16 samples per case and binary, alternating order. Every case is
within the 5% limit for both `InvertedIndexFilterTime` and client wall
time; the largest ratios are 1.032 and 1.038. SNII phrase cases take
25-60% less FilterTime. TPC-H, TPC-DS and ClickBench are within noise.
<details><summary>Per-case median ratios (PR / master)</summary>
Run on 2026-10-14 against master 7ac0eb2fdb on a busy shared host
(median 90% CPU), which widens the intervals; all 4,608 result checks
pass.
| Case | FilterTime | Client wall |
| --- | ---: | ---: |
| `v2/long/early/full` | 1.030 | 1.024 |
| `v2/long/early/range` | 1.018 | 0.999 |
| `v2/long/absent/full` | 1.032 | 1.014 |
| `v2/long/absent/range` | 1.018 | 0.995 |
| `v2/long/mid/full` | 1.006 | 1.038 |
| `v2/long/mid/range` | 1.013 | 1.012 |
| `v2/short/tiny` | 0.965 | 1.018 |
| `v2/search/scored_phrase` | 1.010 | 0.990 |
| `v2/search/phrase_2` | 0.954 | 0.951 |
| `snii/long/early/full` | 0.950 | 0.994 |
| `snii/long/early/range` | 0.961 | 1.022 |
| `snii/long/absent/full` | 0.728 | 0.847 |
| `snii/long/absent/range` | 0.891 | 0.983 |
| `snii/long/mid/full` | 0.821 | 0.924 |
| `snii/long/mid/range` | 0.975 | 1.007 |
| `snii/short/tiny` | 1.020 | 0.985 |
| `snii/search/scored_phrase` | 0.404 | 0.971 |
| `snii/search/phrase_2` | 0.746 | 0.986 |
</details>
BE compile benchmark (build.sh --compile-bench)
A timing framework for analyzing and optimizing the BE C++ build speed. It answers two questions:
- Which build phase is slow (gensrc / submodules / cmake configure / compile)?
- Which translation unit / header / template is slow, and why?
Usage
# One cold, cache-free benchmark build of the BE, then print + save a report
./build.sh --compile-bench
# Additionally collect clang -ftime-trace data (per-header / per-template costs)
COMPILE_BENCH_TRACE=ON ./build.sh --compile-bench
# The usual knobs still apply
BUILD_TYPE=Release ./build.sh --compile-bench -j 16
Results are stored per run in be/compile-bench-results/<UTC timestamp>/:
| file | content |
|---|---|
report.txt |
the human-readable report (also printed at the end of the run) |
summary.json |
machine-readable summary, used by report.py compare |
meta.tsv |
commit, toolchain, -j, PCH on/off, ... of the run |
phases.tsv |
raw phase timestamps |
compile_log.jsonl |
one record per compiler/linker invocation (wall/user/sys/rss) |
ninja_log.txt |
copy of the build dir's .ninja_log |
Compare two runs (e.g. before/after an optimization attempt):
python3 build-support/compile-bench/report.py compare \
be/compile-bench-results/<old_run> be/compile-bench-results/<new_run>
Re-generate a report from raw data (e.g. after an interrupted run, using the still-existing build dir):
python3 build-support/compile-bench/report.py be/compile-bench-results/<run> \
--build-dir be/build_Release_compile_bench --build-status interrupted
How caches are kept out of the measurement
Benchmark numbers are only comparable when every run does the same cold work:
- ccache is fully disabled: the timing wrapper replaces ccache as
CMAKE_<LANG>_COMPILER_LAUNCHER, andCCACHE_DISABLE=1is exported as a belt-and-braces measure for anything else that might call ccache. - A dedicated build dir
be/build_<Type>_compile_benchis deleted and recreated on every run: no incremental objects, no CMake cache, no stale ninja state. Your normalbe/build_<Type>dir is left untouched, so day-to-day incremental builds are not harmed by benchmarking. - BE-only scope: FE, cloud, java extensions, cdc client, UI and output
packaging are forcibly skipped; the run ends right after the C++ build and
the report, before
install. - PCH (
ENABLE_PCH) stays at its normal default on purpose — the PCH is generated inside the fresh build dir each run, so it is cold work, and it is part of the real build being optimized. Override withENABLE_PCH=OFFto compare with/without PCH.
What the report contains
- Phases: gensrc, contrib submodules, datasketches install, cmake configure, build — with durations and share of total wall time.
- Build summary: TU count, sum of per-TU wall/cpu time, effective parallelism, slowest TU, most memory-hungry TU, link times with peak RSS.
- Top N slowest translation units with wall/user/sys/maxrss per file.
- Wall time by directory (top-level and second-level under
be/src) — tells you which module to attack first. - Last finishers from
.ninja_log— the critical-path tail the whole build waits on (typically the monster TUs and the final link). - With
COMPILE_BENCH_TRACE=ON(clang only): top headers by inclusive parse time, top template instantiations, and a frontend vs backend split for the slowest TUs — this is what tells you why a file is slow (header/include cost vs template instantiation vs codegen).
Implementation notes
cc-timing-wrapper.pysits in the compiler-launcher slot,fork/execs the real compiler with inherited stdio, and records wall time plusrusage(user/sys/maxrss) per invocation intocompile_log.jsonl. It is a strict pass-through: exit codes and diagnostics are unchanged, and withoutDORIS_COMPILE_BENCH_DIRset itexecs straight through.- Linker launchers (
CMAKE_<LANG>_LINKER_LAUNCHER) need CMake >= 3.21; with an older CMake, link times still show up via the ninja log, just without RSS. -ftime-traceJSON files land next to the object files in the (transient) bench build dir and are aggregated immediately at the end of the run; expect roughly 1-2 GB of temporary JSON for a full BE build. Header/template times are inclusive (nested includes count into their parents), so they rank hotspots but do not add up to wall time.- Timings from a
make(non-ninja) build lose only the "last finishers" section; per-TU data comes from the wrapper and is generator-independent.