nprasad2077/doris
1
0 Fork 0
mirror of https://github.com/apache/doris.git synced 2026-11-04 07:31:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
master
doris/.github/CODEOWNERS
T
Jack 14fde1c1dc [refactor](be) Unify MATCH and SEARCH execution across inverted index formats (#68758) 
### What problem does this PR solve?

Issue Number: None

Related PR: #67180, #67538, #67918

Problem Summary:

MATCH and SEARCH each had their own Boolean, phrase, term-expansion and
scoring executors, written separately for CLucene (V1/V2) and SNII
indexes. Every query optimization had to be implemented once per
executor, the formats drifted apart in behavior, and cross-field queries
could mix document IDs from different physical partitions.

This PR runs both on one engine:

- **One logical form.** MATCH (`query/logical`) and SEARCH
(`exprs/function/search`) lower their queries to the same logical nodes.
- **One query engine.** Boolean logic (AND/OR/NOT, minimum_should_match,
WAND), term expansion (prefix, wildcard, regexp), exact and sloppy
phrases and BM25 scoring are implemented once, independent of the index
format.
- **Formats behind an SPI.** CLucene and SNII only provide dictionaries,
postings, positions, norms, statistics and batched I/O through
`query/spi`. The old per-format executors are removed.
- **Layered source tree.** `be/src/storage/index/` is reorganized by
layer, and `build-support/check-header-deps.py` enforces the allowed
include directions.

Index formats and writers are unchanged, and existing indexes remain
readable. FE is unchanged.

#### Source layout

```text
be/src/storage/index/
├── index_*.h/.cpp        shared index files: file reader/writer, compound reader, storage formats
├── ann/  bloom_filter/  zone_map/
└── inverted/
    ├── analysis/         analyzers, tokenizers, char and token filters, normalizers, gram schemes
    ├── common/           descriptors, caches, statistics, profiles, single flight
    ├── reader/           text index reader, index selection, collection statistics
    ├── query/            format-independent query engine
    │   ├── logical/      MATCH lowering and logical nodes (gram/ builds gram queries)
    │   ├── engine/       Query/Weight/Scorer: boolean, term, phrase, expansion, WAND, union, collect
    │   ├── exec/         doc-id sets and sinks, term waves, chained conjunctions
    │   ├── phrase/       exact and sloppy phrase matchers
    │   ├── scoring/      BM25
    │   └── spi/          the source interface each format implements
    ├── clucene/          CLucene (V1/V2) reader, searcher, writer, compaction
    │   └── spi/          CLucene index source and postings cursor
    └── snii/             SNII: format/, encoding/, io/, bkd/, reader/, writer/, compaction/, query/

be/src/exprs/
├── vsearch.h/.cpp                     VSearchExpr, the SEARCH expression node
└── function/search/
    ├── function_search.*              SEARCH entry: Boolean assembly, DSL result cache, NULL handling
    ├── search_lowering.*              SEARCH DSL to logical nodes; binds each field to an index
    ├── function_search_leaf.*         compiles each leaf into a query on its bound index
    ├── search_predicate_collector.*   collects scored leaves for BM25 statistics
    └── variant_inverted_index_search.*  SEARCH on VARIANT subcolumns and NESTED groups
```

#### Performance

The 18 production SQL cases (V2 and SNII: point, range, absent-term and
phrase queries) were compared with master on fresh dual-BE pairs: 8
pairs, 16 samples per case and binary, alternating order. Every case is
within the 5% limit for both `InvertedIndexFilterTime` and client wall
time; the largest ratios are 1.032 and 1.038. SNII phrase cases take
25-60% less FilterTime. TPC-H, TPC-DS and ClickBench are within noise.

<details><summary>Per-case median ratios (PR / master)</summary>

Run on 2026-10-14 against master 7ac0eb2fdb on a busy shared host
(median 90% CPU), which widens the intervals; all 4,608 result checks
pass.

| Case | FilterTime | Client wall |
| --- | ---: | ---: |
| `v2/long/early/full` | 1.030 | 1.024 |
| `v2/long/early/range` | 1.018 | 0.999 |
| `v2/long/absent/full` | 1.032 | 1.014 |
| `v2/long/absent/range` | 1.018 | 0.995 |
| `v2/long/mid/full` | 1.006 | 1.038 |
| `v2/long/mid/range` | 1.013 | 1.012 |
| `v2/short/tiny` | 0.965 | 1.018 |
| `v2/search/scored_phrase` | 1.010 | 0.990 |
| `v2/search/phrase_2` | 0.954 | 0.951 |
| `snii/long/early/full` | 0.950 | 0.994 |
| `snii/long/early/range` | 0.961 | 1.022 |
| `snii/long/absent/full` | 0.728 | 0.847 |
| `snii/long/absent/range` | 0.891 | 0.983 |
| `snii/long/mid/full` | 0.821 | 0.924 |
| `snii/long/mid/range` | 0.975 | 1.007 |
| `snii/short/tiny` | 1.020 | 0.985 |
| `snii/search/scored_phrase` | 0.404 | 0.971 |
| `snii/search/phrase_2` | 0.746 | 0.986 |

</details>
2026-10-16 10:22:35 +08:00

170 lines
15 KiB
Plaintext
Raw Permalink Blame History

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# for more syntax help and usage example
# https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners#codeowners-syntax
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java @dataroaring @CalvinKirs @morningman
**/pom.xml @CalvinKirs @morningman
fe/fe-common/src/main/java/org/apache/doris/common/FeMetaVersion.java @dataroaring @morningman @yiguolei @xiaokang
fe/fe-core/src/main/java/org/apache/doris/fs @CalvinKirs @924060929
fe/fe-core/src/main/java/org/apache/doris/fsv2 @CalvinKirs
be/src/olap/rowset/segment_v2/variant @eldenmoon @csun5285
be/src/storage/index/inverted/ @airborne12 @eldenmoon @csun5285
be/src/exprs/function/search/ @airborne12 @eldenmoon @csun5285
be/src/exprs/function/match.cpp @airborne12 @eldenmoon @csun5285
be/src/exprs/function/match.h @airborne12 @eldenmoon @csun5285
be/src/exprs/vsearch.h @airborne12 @eldenmoon @csun5285
be/src/exprs/vsearch.cpp @airborne12 @eldenmoon @csun5285
be/src/cloud @liaoxin01 @luwei16 @gavinchou
be/src/io @liaoxin01 @gavinchou @morningman @Gabriel39
be/src/io/cache @liaoxin01 @gavinchou @morningman @Gabriel39
be/src/io/fs @liaoxin01 @gavinchou @morningman @Gabriel39
be/src/io/tools @liaoxin01 @gavinchou
be/src/format_v2 @Gabriel39 @yiguolei
be/src/load @liaoxin01 @gavinchou
be/src/load/channel @liaoxin01 @gavinchou
be/src/load/delta_writer @liaoxin01 @gavinchou
be/src/load/group_commit @liaoxin01 @gavinchou
be/src/load/memtable @liaoxin01 @gavinchou
be/src/load/routine_load @liaoxin01 @gavinchou
be/src/load/stream_load @liaoxin01 @gavinchou
be/src/storage @gavinchou @yiguolei
be/src/storage/olap_common.h @gavinchou @yiguolei
be/src/storage/cache @liaoxin01 @gavinchou
be/src/storage/compaction @luwei16 @gavinchou
be/src/storage/delete @luwei16 @gavinchou
be/src/storage/index @airborne12 @eldenmoon @csun5285
be/src/storage/iterator @yiguolei @gavinchou
be/src/storage/predicate @yiguolei @gavinchou
be/src/storage/rowset @gavinchou @yiguolei
be/src/storage/schema_change @luwei16 @gavinchou
be/src/storage/segment @yiguolei @liaoxin01 @gavinchou @eldenmoon @csun5285 @airborne12
be/src/storage/snapshot @deardeng @luwei16 @gavinchou
be/src/storage/tablet @gavinchou @yiguolei
be/src/storage/task @deardeng @gavinchou
be/src/storage/txn @liaoxin01 @gavinchou
be/test/cloud @luwei16 @liaoxin01 @gavinchou
be/test/io @liaoxin01 @gavinchou @morningman @Gabriel39
be/test/io/cache @liaoxin01 @gavinchou @morningman @Gabriel39
be/test/io/client @gavinchou
be/test/io/fs @luwei16 @liaoxin01 @gavinchou @morningman @Gabriel39
be/test/load @liaoxin01 @gavinchou
be/test/load/channel @liaoxin01 @gavinchou
be/test/load/delta_writer @liaoxin01 @gavinchou
be/test/load/memtable @liaoxin01 @gavinchou
be/test/olap @gavinchou
be/test/olap/rowset @gavinchou @yiguolei
be/test/storage @gavinchou @yiguolei
be/test/storage/cache @liaoxin01 @gavinchou
be/test/storage/compaction @luwei16 @gavinchou
be/test/storage/delete @gavinchou
be/test/storage/index @airborne12 @eldenmoon @csun5285
be/test/storage/iterator @yiguolei @gavinchou
be/test/storage/predicate @yiguolei @gavinchou
be/test/storage/rowset @gavinchou @yiguolei
be/test/storage/schema_change @luwei16 @gavinchou
be/test/storage/segment @yiguolei @gavinchou @eldenmoon @csun5285 @airborne12
be/test/storage/snapshot @deardeng @luwei16 @gavinchou
be/test/storage/tablet @gavinchou @yiguolei
be/test/storage/txn @liaoxin01 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/alter @luwei16 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/backup @luwei16 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/binlog @luwei16 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/clone @deardeng @luwei16 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud @liaoxin01 @luwei16 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/alter @luwei16 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/backup @luwei16 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog @morningman @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/common @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/datasource @morningman @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/load @liaoxin01 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/master @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/persist @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/qe @morrySnow @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/rpc @luwei16 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/snapshot @luwei16 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/stage @liaoxin01 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/storage @luwei16 @liaoxin01 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/system @deardeng @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cloud/transaction @mymeiyi @liaoxin01 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/cooldown @liaoxin01 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/connector @924060929 @CalvinKirs
fe/fe-core/src/main/java/org/apache/doris/datasource @924060929 @CalvinKirs
fe/fe-core/src/main/java/org/apache/doris/fs @CalvinKirs @924060929 @morningman @liaoxin01 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/ha @mymeiyi @gavinchou
fe/fe-core/src/main/java/org/apache/doris/journal @mymeiyi @gavinchou
fe/fe-core/src/main/java/org/apache/doris/load @liaoxin01 @gavinchou
fe/fe-core/src/main/java/org/apache/doris/master @mymeiyi @deardeng @gavinchou
fe/fe-core/src/main/java/org/apache/doris/mtmv @morrySnow @yujun777
fe/fe-core/src/main/java/org/apache/doris/nereids @morrySnow @924060929 @englefly @starocean999
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression @yujun777
fe/fe-core/src/main/java/org/apache/doris/persist @mymeiyi @gavinchou
fe/fe-core/src/main/java/org/apache/doris/persist/gson @mymeiyi @gavinchou
fe/fe-core/src/main/java/org/apache/doris/persist/meta @mymeiyi @gavinchou
fe/fe-core/src/main/java/org/apache/doris/qe @morrySnow @924060929
fe/fe-core/src/main/java/org/apache/doris/spi @924060929 @CalvinKirs
fe/fe-core/src/main/java/org/apache/doris/statistics @morrySnow @englefly @yujun777
fe/fe-core/src/test/java/org/apache/doris/alter @luwei16 @gavinchou
fe/fe-core/src/test/java/org/apache/doris/backup @luwei16 @gavinchou
fe/fe-core/src/test/java/org/apache/doris/binlog @luwei16 @gavinchou
fe/fe-core/src/test/java/org/apache/doris/clone @deardeng @gavinchou
fe/fe-core/src/test/java/org/apache/doris/cloud @gavinchou
fe/fe-core/src/test/java/org/apache/doris/cloud/backup @luwei16 @gavinchou
fe/fe-core/src/test/java/org/apache/doris/cloud/cache @liaoxin01 @gavinchou
fe/fe-core/src/test/java/org/apache/doris/cloud/catalog @morningman @gavinchou
fe/fe-core/src/test/java/org/apache/doris/cloud/common @liaoxin01 @luwei16 @gavinchou
fe/fe-core/src/test/java/org/apache/doris/cloud/load @liaoxin01 @gavinchou
fe/fe-core/src/test/java/org/apache/doris/cloud/master @deardeng @gavinchou
fe/fe-core/src/test/java/org/apache/doris/cloud/rpc @luwei16 @gavinchou
fe/fe-core/src/test/java/org/apache/doris/cloud/stage @liaoxin01 @gavinchou
fe/fe-core/src/test/java/org/apache/doris/cloud/storage @liaoxin01 @gavinchou
fe/fe-core/src/test/java/org/apache/doris/cloud/system @deardeng @gavinchou
fe/fe-core/src/test/java/org/apache/doris/cloud/transaction @liaoxin01 @gavinchou
fe/fe-core/src/test/java/org/apache/doris/connector @924060929 @CalvinKirs
fe/fe-core/src/test/java/org/apache/doris/datasource @924060929 @CalvinKirs
fe/fe-core/src/test/java/org/apache/doris/fs @CalvinKirs @924060929 @morningman @gavinchou @Gabriel39
fe/fe-core/src/test/java/org/apache/doris/journal @mymeiyi @gavinchou
fe/fe-core/src/test/java/org/apache/doris/load @liaoxin01 @gavinchou
fe/fe-core/src/test/java/org/apache/doris/mtmv @morrySnow @yujun777
fe/fe-core/src/test/java/org/apache/doris/nereids @morrySnow @924060929 @englefly @starocean999
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression @yujun777
fe/fe-connector @924060929 @CalvinKirs
fe/fe-filesystem @CalvinKirs @924060929 @morningman @gavinchou
fe/fe-sql-parser/src/main/antlr4 @morrySnow @starocean999 @924060929 @englefly
cloud @gavinchou
cloud/cmake @liaoxin01 @luwei16 @gavinchou
cloud/conf @liaoxin01 @luwei16 @gavinchou
cloud/script @gavinchou @luwei16
cloud/src @gavinchou
cloud/src/common @liaoxin01 @luwei16 @gavinchou
cloud/src/gen-cpp @liaoxin01 @luwei16 @gavinchou
cloud/src/meta-service @liaoxin01 @luwei16 @gavinchou
cloud/src/meta-store @luwei16 @gavinchou
cloud/src/rate-limiter @luwei16 @gavinchou
cloud/src/recycler @liaoxin01 @gavinchou @wyxxxcat
cloud/src/resource-manager @deardeng @gavinchou
cloud/src/snapshot @luwei16 @gavinchou
cloud/test @liaoxin01 @luwei16 @gavinchou
gensrc/proto/cloud.proto @luwei16 @gavinchou
gensrc/proto/file_cache.proto @liaoxin01 @gavinchou
gensrc/proto/olap_file.proto @yiguolei @liaoxin01 @gavinchou
gensrc/proto/segment_v2.proto @yiguolei @liaoxin01 @gavinchou @eldenmoon @csun5285 @airborne12
.github/workflows/ @hello-stephen @morningman @yiguolei
thirdparty/ @yiguolei @morningman
Reference in new issue View Git Blame Copy Permalink