mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Jack
14fde1c1dc
### What problem does this PR solve?
Issue Number: None
Related PR: #67180, #67538, #67918
Problem Summary:
MATCH and SEARCH each had their own Boolean, phrase, term-expansion and
scoring executors, written separately for CLucene (V1/V2) and SNII
indexes. Every query optimization had to be implemented once per
executor, the formats drifted apart in behavior, and cross-field queries
could mix document IDs from different physical partitions.
This PR runs both on one engine:
- **One logical form.** MATCH (`query/logical`) and SEARCH
(`exprs/function/search`) lower their queries to the same logical nodes.
- **One query engine.** Boolean logic (AND/OR/NOT, minimum_should_match,
WAND), term expansion (prefix, wildcard, regexp), exact and sloppy
phrases and BM25 scoring are implemented once, independent of the index
format.
- **Formats behind an SPI.** CLucene and SNII only provide dictionaries,
postings, positions, norms, statistics and batched I/O through
`query/spi`. The old per-format executors are removed.
- **Layered source tree.** `be/src/storage/index/` is reorganized by
layer, and `build-support/check-header-deps.py` enforces the allowed
include directions.
Index formats and writers are unchanged, and existing indexes remain
readable. FE is unchanged.
#### Source layout
```text
be/src/storage/index/
├── index_*.h/.cpp shared index files: file reader/writer, compound reader, storage formats
├── ann/ bloom_filter/ zone_map/
└── inverted/
├── analysis/ analyzers, tokenizers, char and token filters, normalizers, gram schemes
├── common/ descriptors, caches, statistics, profiles, single flight
├── reader/ text index reader, index selection, collection statistics
├── query/ format-independent query engine
│ ├── logical/ MATCH lowering and logical nodes (gram/ builds gram queries)
│ ├── engine/ Query/Weight/Scorer: boolean, term, phrase, expansion, WAND, union, collect
│ ├── exec/ doc-id sets and sinks, term waves, chained conjunctions
│ ├── phrase/ exact and sloppy phrase matchers
│ ├── scoring/ BM25
│ └── spi/ the source interface each format implements
├── clucene/ CLucene (V1/V2) reader, searcher, writer, compaction
│ └── spi/ CLucene index source and postings cursor
└── snii/ SNII: format/, encoding/, io/, bkd/, reader/, writer/, compaction/, query/
be/src/exprs/
├── vsearch.h/.cpp VSearchExpr, the SEARCH expression node
└── function/search/
├── function_search.* SEARCH entry: Boolean assembly, DSL result cache, NULL handling
├── search_lowering.* SEARCH DSL to logical nodes; binds each field to an index
├── function_search_leaf.* compiles each leaf into a query on its bound index
├── search_predicate_collector.* collects scored leaves for BM25 statistics
└── variant_inverted_index_search.* SEARCH on VARIANT subcolumns and NESTED groups
```
#### Performance
The 18 production SQL cases (V2 and SNII: point, range, absent-term and
phrase queries) were compared with master on fresh dual-BE pairs: 8
pairs, 16 samples per case and binary, alternating order. Every case is
within the 5% limit for both `InvertedIndexFilterTime` and client wall
time; the largest ratios are 1.032 and 1.038. SNII phrase cases take
25-60% less FilterTime. TPC-H, TPC-DS and ClickBench are within noise.
<details><summary>Per-case median ratios (PR / master)</summary>
Run on 2026-10-14 against master 7ac0eb2fdb on a busy shared host
(median 90% CPU), which widens the intervals; all 4,608 result checks
pass.
| Case | FilterTime | Client wall |
| --- | ---: | ---: |
| `v2/long/early/full` | 1.030 | 1.024 |
| `v2/long/early/range` | 1.018 | 0.999 |
| `v2/long/absent/full` | 1.032 | 1.014 |
| `v2/long/absent/range` | 1.018 | 0.995 |
| `v2/long/mid/full` | 1.006 | 1.038 |
| `v2/long/mid/range` | 1.013 | 1.012 |
| `v2/short/tiny` | 0.965 | 1.018 |
| `v2/search/scored_phrase` | 1.010 | 0.990 |
| `v2/search/phrase_2` | 0.954 | 0.951 |
| `snii/long/early/full` | 0.950 | 0.994 |
| `snii/long/early/range` | 0.961 | 1.022 |
| `snii/long/absent/full` | 0.728 | 0.847 |
| `snii/long/absent/range` | 0.891 | 0.983 |
| `snii/long/mid/full` | 0.821 | 0.924 |
| `snii/long/mid/range` | 0.975 | 1.007 |
| `snii/short/tiny` | 1.020 | 0.985 |
| `snii/search/scored_phrase` | 0.404 | 0.971 |
| `snii/search/phrase_2` | 0.746 | 0.986 |
</details>
13 lines
357 B
Plaintext
13 lines
357 B
Plaintext
# Please add code that you don’t want to be formatted by clang here
|
||
|
||
be/src/apache-orc/*
|
||
be/src/clucene/*
|
||
be/src/gutil/*
|
||
be/src/glibc-compatibility/*
|
||
be/src/util/sse2neo.h
|
||
be/src/util/sse2neon.h
|
||
be/src/util/mustache/mustache.h
|
||
be/src/util/mustache/mustache.cc
|
||
be/src/storage/index/inverted/analysis/analyzer/kuromoji/dict/darts.h
|
||
cloud/src/common/defer.h
|