mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
[improvement](inverted index) Short-circuit pushed-down conjuncts once the row bitmap is empty (#67171)
### What problem does this PR solve?
Problem Summary:
On log-search workloads a query typically carries several pushed-down
conjuncts, e.g. two cheap untokenized MATCH terms plus one expensive
`msg MATCH_PHRASE_PREFIX '...'`:
```sql
SELECT * FROM logs
WHERE _ctime_ >= ... AND _ctime_ <= ...
AND _namespace_ MATCH 'ns' -- untokenized term, ~free
AND _pod_name_ MATCH 'pod-xyz' -- untokenized term, ~free
AND msg MATCH_PHRASE_PREFIX '...' -- whole-segment postings + positions
ORDER BY _ctime_ DESC LIMIT 1000
```
`SegmentIterator` evaluated ALL pushed-down conjuncts against the index
first and only intersected their bitmaps into `_row_bitmap` afterwards,
so the phrase conjunct paid its full whole-segment postings/positions
cost even when an earlier selective conjunct had already emptied the
candidate bitmap. Reproduced on a production log table: one query
returning 0 rows burned 11,291 CPU seconds and 124 GiB of local index
reads per physical SQL, virtually all of it inside phrase evaluation
(profile: 2,657 segments, InvertedIndexSearcherSearchExecTime = 99.7% of
scan cost, ScanRows = 0).
The empty-bitmap short circuit already exists on the neighbouring paths:
column predicates (`continue_apply` in `_apply_inverted_index`) and
compound AND inside a single expression (`VCompoundPred` COMPOUND_AND
early exit). But top-level AND conjuncts are flattened into separate
expr contexts and take `_apply_index_expr`, which had neither
progressive intersection nor a short circuit, so the most common query
shape never benefited.
Fix: intersect each consumed index result into `_row_bitmap` as soon as
it is produced, and stop evaluating further conjuncts once the bitmap is
empty. A skipped conjunct stays pushed down, so the row-level path keeps
its exact semantics over the now-empty candidate set (zero rows read,
zero cost). Consumed conjuncts are erased from
`_common_expr_ctxs_push_down` only after the ANN range-search pass,
which iterates the same list. A new profile counter
`InvertedIndexConjunctsShortCircuited` reports how many conjuncts were
skipped.
Behavior notes:
- Index results are now intersected before the ANN range-search pass, so
ANN range search executes on the already-narrowed bitmap. AND semantics
are commutative, and a smaller candidate set is what its small-candidate
fallback is designed for.
- Condition cache digest semantics unchanged: a short-circuited (thus
still pushed-down) conjunct keeps the digest cleared, so partial index
results are never cached as full coverage.
### Release note
Inverted index: once earlier pushed-down conjuncts empty the candidate
row bitmap, remaining conjuncts (e.g. expensive MATCH_PHRASE_PREFIX) are
no longer evaluated against the index.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Index results are intersected progressively (before the
ANN pass); short-circuited conjuncts fall back to the row-level path
over an empty candidate set. Same query results, strictly less index
work. -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---
Review round 2 (automated review, addressed in the third commit):
1. Dead work after the short circuit is now bypassed on an empty
candidate
bitmap: virtual-column MATCH projections skip their whole-segment index
evaluation, the ANN range-search pass breaks out before
loading/searching,
and ANN TopN falls back at entry (all three previously ran even with
zero
surviving rows when the small-candidate fallback thresholds are
disabled).
2. The consumed-list clear no longer drops the proven all-false
condition-cache entry: `_index_conjuncts_proved_empty` keeps the digest
alive for the exhausted case (the empty result is correct for the full
conjunction), while a counter-case test pins the normal fully-consumed
path to its existing digest-zeroing behavior.
This commit is contained in:
7 files changed
+405
-1
No files matched your search
@@ -290,6 +290,8 @@ Status OlapScanLocalState::_init_profile() {
|
||||
_segment_profile, "InvertedIndexSearcherCacheMiss", TUnit::UNIT, 1);
|
||||
_inverted_index_downgrade_count_counter =
|
||||
ADD_COUNTER_WITH_LEVEL(_segment_profile, "InvertedIndexDowngradeCount", TUnit::UNIT, 1);
|
||||
_inverted_index_conjuncts_short_circuited_counter = ADD_COUNTER_WITH_LEVEL(
|
||||
_segment_profile, "InvertedIndexConjunctsShortCircuited", TUnit::UNIT, 1);
|
||||
_inverted_index_analyzer_timer =
|
||||
ADD_TIMER_WITH_LEVEL(_segment_profile, "InvertedIndexAnalyzerTime", 1);
|
||||
_inverted_index_lookup_timer =
|
||||
|
||||
@@ -265,6 +265,7 @@ private:
|
||||
RuntimeProfile::Counter* _inverted_index_searcher_cache_hit_counter = nullptr;
|
||||
RuntimeProfile::Counter* _inverted_index_searcher_cache_miss_counter = nullptr;
|
||||
RuntimeProfile::Counter* _inverted_index_downgrade_count_counter = nullptr;
|
||||
RuntimeProfile::Counter* _inverted_index_conjuncts_short_circuited_counter = nullptr;
|
||||
RuntimeProfile::Counter* _inverted_index_analyzer_timer = nullptr;
|
||||
RuntimeProfile::Counter* _inverted_index_lookup_timer = nullptr;
|
||||
|
||||
|
||||
@@ -881,6 +881,8 @@ void OlapScanner::_collect_profile_before_close() {
|
||||
stats.inverted_index_searcher_cache_miss);
|
||||
COUNTER_UPDATE(local_state->_inverted_index_downgrade_count_counter,
|
||||
stats.inverted_index_downgrade_count);
|
||||
COUNTER_UPDATE(local_state->_inverted_index_conjuncts_short_circuited_counter,
|
||||
stats.inverted_index_conjuncts_short_circuited);
|
||||
COUNTER_UPDATE(local_state->_inverted_index_analyzer_timer,
|
||||
stats.inverted_index_analyzer_timer);
|
||||
COUNTER_UPDATE(local_state->_inverted_index_lookup_timer, stats.inverted_index_lookup_timer);
|
||||
|
||||
@@ -325,6 +325,9 @@ struct OlapReaderStatistics {
|
||||
int64_t inverted_index_searcher_cache_hit = 0;
|
||||
int64_t inverted_index_searcher_cache_miss = 0;
|
||||
int64_t inverted_index_downgrade_count = 0;
|
||||
// Pushed-down conjuncts skipped (never index-evaluated) because the row
|
||||
// bitmap was already empty when their turn came.
|
||||
int64_t inverted_index_conjuncts_short_circuited = 0;
|
||||
int64_t inverted_index_analyzer_timer = 0;
|
||||
int64_t inverted_index_lookup_timer = 0;
|
||||
// See snii_query_stats.h: one field here instead of one per SNII counter.
|
||||
|
||||
@@ -858,7 +858,9 @@ Status SegmentIterator::_get_row_ranges_by_column_conditions() {
|
||||
}
|
||||
}
|
||||
_opts.condition_cache_digest =
|
||||
_common_expr_ctxs_push_down.empty() ? 0 : _opts.condition_cache_digest;
|
||||
_common_expr_ctxs_push_down.empty() && !_index_conjuncts_proved_empty
|
||||
? 0
|
||||
: _opts.condition_cache_digest;
|
||||
_opts.stats->rows_inverted_index_filtered += (input_rows - _row_bitmap.cardinality());
|
||||
for (uint32_t cid = 0; cid < _schema->num_read_columns(); ++cid) {
|
||||
bool result_true = _check_all_conditions_passed_inverted_index_for_column(cid);
|
||||
@@ -925,6 +927,13 @@ Status SegmentIterator::_apply_ann_topn_predicate() {
|
||||
if (_ann_topn_runtime == nullptr) {
|
||||
return Status::OK();
|
||||
}
|
||||
if (_row_bitmap.isEmpty()) {
|
||||
// Zero candidates: nothing for TopN to select, and the residual
|
||||
// conjuncts that used to veto this path may have been consumed by the
|
||||
// proved-empty short circuit. Fall back before touching the ANN index
|
||||
// (the small-candidate fallback may be disabled by its thresholds).
|
||||
return Status::OK();
|
||||
}
|
||||
|
||||
VLOG_DEBUG << fmt::format("Try apply ann topn: {}", _ann_topn_runtime->debug_string());
|
||||
// AnnTopNRuntime keeps VSlotRef::column_id(), which is the read-schema ordinal.
|
||||
@@ -1237,7 +1246,23 @@ Status SegmentIterator::_apply_index_expr() {
|
||||
!_opts.runtime_state->query_options().__isset.enable_ann_index_result_cache ||
|
||||
_opts.runtime_state->query_options().enable_ann_index_result_cache;
|
||||
|
||||
// Intersect each consumed index result into _row_bitmap right away so a
|
||||
// selective conjunct short-circuits the remaining (potentially expensive,
|
||||
// e.g. MATCH_PHRASE_PREFIX) ones. A skipped conjunct stays pushed down and
|
||||
// keeps its semantics on the row-level path, which then sees zero rows.
|
||||
// Consumed conjuncts are erased only after the ANN pass below, which
|
||||
// iterates the same list.
|
||||
std::vector<const VExprContext*> consumed_by_index;
|
||||
bool bitmap_exhausted = false;
|
||||
size_t considered_conjuncts = 0;
|
||||
for (const auto& expr_ctx : _common_expr_ctxs_push_down) {
|
||||
if (_row_bitmap.isEmpty()) {
|
||||
_opts.stats->inverted_index_conjuncts_short_circuited +=
|
||||
_common_expr_ctxs_push_down.size() - considered_conjuncts;
|
||||
bitmap_exhausted = true;
|
||||
break;
|
||||
}
|
||||
++considered_conjuncts;
|
||||
if (Status st = expr_ctx->evaluate_inverted_index(num_rows()); !st.ok()) {
|
||||
if (_downgrade_without_index(st) || st.code() == ErrorCode::NOT_IMPLEMENTED_ERROR) {
|
||||
continue;
|
||||
@@ -1249,12 +1274,25 @@ Status SegmentIterator::_apply_index_expr() {
|
||||
return st;
|
||||
}
|
||||
}
|
||||
if (expr_ctx->all_expr_inverted_index_evaluated()) {
|
||||
const auto* result = expr_ctx->get_index_context()->get_index_result_for_expr(
|
||||
expr_ctx->root().get());
|
||||
if (result != nullptr) {
|
||||
_row_bitmap &= *result->get_data_bitmap();
|
||||
consumed_by_index.push_back(expr_ctx.get());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Evaluate inverted index for virtual column MATCH expressions (projections).
|
||||
// Unlike common exprs which filter rows, these only compute index result bitmaps
|
||||
// for later materialization via fast_execute().
|
||||
for (auto& [cid, expr_ctx] : _virtual_column_exprs) {
|
||||
if (_row_bitmap.isEmpty()) {
|
||||
// Zero surviving rows: the projection column is never materialized,
|
||||
// so its whole-segment index evaluation would be pure waste.
|
||||
break;
|
||||
}
|
||||
if (expr_ctx->get_index_context() == nullptr) {
|
||||
continue;
|
||||
}
|
||||
@@ -1272,6 +1310,13 @@ Status SegmentIterator::_apply_index_expr() {
|
||||
|
||||
// Apply ann range search
|
||||
for (const auto& expr_ctx : _common_expr_ctxs_push_down) {
|
||||
if (_row_bitmap.isEmpty()) {
|
||||
// A range search intersects into the bitmap; with zero candidates
|
||||
// it cannot add rows, so loading and searching the ANN index
|
||||
// (bypassing the small-candidate fallback when its thresholds are
|
||||
// disabled) would be pure waste.
|
||||
break;
|
||||
}
|
||||
segment_v2::AnnIndexStats ann_index_stats;
|
||||
size_t origin_rows = _row_bitmap.cardinality();
|
||||
bool ann_range_search_executed = false;
|
||||
@@ -1302,6 +1347,22 @@ Status SegmentIterator::_apply_index_expr() {
|
||||
_opts.stats->ann_index_range_cache_hits += ann_index_stats.range_cache_hits.value();
|
||||
}
|
||||
|
||||
if (bitmap_exhausted) {
|
||||
// Zero surviving rows satisfy every remaining conjunct, so the whole
|
||||
// list is consumed -- mirroring the column-predicate short circuit.
|
||||
// This keeps the "all conditions consumed by the index" contract, and
|
||||
// _index_conjuncts_proved_empty keeps the condition-cache digest
|
||||
// alive: the all-false result is correct for the full conjunction, so
|
||||
// clearing the list must not degrade it to "nothing left to cache".
|
||||
_index_conjuncts_proved_empty = true;
|
||||
_common_expr_ctxs_push_down.clear();
|
||||
} else if (!consumed_by_index.empty()) {
|
||||
std::erase_if(_common_expr_ctxs_push_down, [&](const VExprContextSPtr& ctx) {
|
||||
return std::find(consumed_by_index.begin(), consumed_by_index.end(), ctx.get()) !=
|
||||
consumed_by_index.end();
|
||||
});
|
||||
}
|
||||
|
||||
return Status::OK();
|
||||
}
|
||||
|
||||
|
||||
@@ -484,6 +484,14 @@ private:
|
||||
bool _count_fastpath_hit = false;
|
||||
bool _count_emit_shortcut = false;
|
||||
uint64_t _count_emit_rows_remaining = 0;
|
||||
|
||||
// An indexed conjunct prefix emptied _row_bitmap, proving the WHOLE
|
||||
// pushed-down conjunction false. Set by the _apply_index_expr short
|
||||
// circuit when it consumes (clears) the remaining conjuncts, and read
|
||||
// where an empty conjunct list would otherwise zero the condition-cache
|
||||
// digest: the all-false result stays valid for the full conjunction, so
|
||||
// it must remain cacheable.
|
||||
bool _index_conjuncts_proved_empty = false;
|
||||
// Batch size for shortcut emission: VStatisticsIterator's
|
||||
// MAX_ROW_SIZE_IN_COUNT, the largest default-rows block shape already
|
||||
// proven through every consumer above the segment iterator by the plain
|
||||
|
||||
@@ -0,0 +1,327 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
// White-box tests for progressive intersection and short-circuit of pushed-down
|
||||
// conjuncts in SegmentIterator::_apply_index_expr. Once an earlier conjunct's
|
||||
// index result empties _row_bitmap, the remaining (potentially expensive, e.g.
|
||||
// MATCH_PHRASE_PREFIX) conjuncts must not be evaluated against the index at
|
||||
// all; skipped conjuncts stay pushed down so the row-level path keeps their
|
||||
// semantics over the (now empty) candidate set. Uses the established
|
||||
// `#define private public` convention of segment_iterator_limit_opt_test.cpp.
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
#include <vector>
|
||||
|
||||
#include "common/status.h"
|
||||
#include "core/data_type/data_type_number.h"
|
||||
#include "exprs/vexpr.h"
|
||||
#include "exprs/vexpr_context.h"
|
||||
#include "runtime/runtime_state.h"
|
||||
#include "storage/index/ann/ann_topn_runtime.h"
|
||||
#include "storage/index/inverted/inverted_index_reader.h"
|
||||
#include "storage/olap_common.h"
|
||||
#include "storage/tablet/tablet_schema.h"
|
||||
|
||||
#if defined(__clang__)
|
||||
#pragma clang diagnostic push
|
||||
#pragma clang diagnostic ignored "-Wkeyword-macro"
|
||||
#endif
|
||||
#define private public
|
||||
#define protected public
|
||||
#include "storage/segment/segment.h"
|
||||
#include "storage/segment/segment_iterator.h"
|
||||
#undef private
|
||||
#undef protected
|
||||
#if defined(__clang__)
|
||||
#pragma clang diagnostic pop
|
||||
#endif
|
||||
|
||||
namespace doris::segment_v2 {
|
||||
|
||||
namespace {
|
||||
|
||||
// A test VExpr that counts evaluations and registers a fixed-cardinality index
|
||||
// result bitmap, mimicking how real match exprs publish results.
|
||||
class BitmapEvalExpr : public VExpr {
|
||||
public:
|
||||
explicit BitmapEvalExpr(std::vector<uint32_t> rows) : _rows(std::move(rows)) {
|
||||
_data_type = std::make_shared<DataTypeUInt8>();
|
||||
}
|
||||
|
||||
const std::string& expr_name() const override {
|
||||
static const std::string kName = "BitmapEvalExpr";
|
||||
return kName;
|
||||
}
|
||||
|
||||
Status execute(VExprContext*, Block*, int*) const override { return Status::OK(); }
|
||||
|
||||
Status execute_column_impl(VExprContext* context, const Block* block, const Selector* selector,
|
||||
size_t count, ColumnPtr& result_column) const override {
|
||||
return Status::OK();
|
||||
}
|
||||
|
||||
Status evaluate_inverted_index(VExprContext* context, uint32_t segment_num_rows) override {
|
||||
++_eval_count;
|
||||
auto data = std::make_shared<roaring::Roaring>();
|
||||
for (uint32_t row : _rows) {
|
||||
data->add(row);
|
||||
}
|
||||
InvertedIndexResultBitmap result(std::move(data), std::make_shared<roaring::Roaring>());
|
||||
context->get_index_context()->set_index_result_for_expr(this, std::move(result));
|
||||
return Status::OK();
|
||||
}
|
||||
|
||||
Status evaluate_ann_range_search(
|
||||
const segment_v2::AnnRangeSearchRuntime&,
|
||||
const std::vector<std::unique_ptr<segment_v2::IndexIterator>>&,
|
||||
const std::vector<std::unique_ptr<segment_v2::ColumnIterator>>&, size_t,
|
||||
roaring::Roaring&, segment_v2::AnnIndexStats&, bool,
|
||||
AnnRangeSearchEvaluationResult& result) override {
|
||||
++_ann_eval_count;
|
||||
result.executed = false;
|
||||
return Status::OK();
|
||||
}
|
||||
|
||||
int eval_count() const { return _eval_count; }
|
||||
int ann_eval_count() const { return _ann_eval_count; }
|
||||
|
||||
private:
|
||||
std::vector<uint32_t> _rows;
|
||||
int _eval_count = 0;
|
||||
int _ann_eval_count = 0;
|
||||
};
|
||||
|
||||
TabletSchemaSPtr make_tablet_schema() {
|
||||
TabletSchemaPB schema_pb;
|
||||
schema_pb.set_keys_type(KeysType::DUP_KEYS);
|
||||
auto* col = schema_pb.add_column();
|
||||
col->set_unique_id(0);
|
||||
col->set_name("k0");
|
||||
col->set_type("INT");
|
||||
col->set_is_key(true);
|
||||
col->set_is_nullable(false);
|
||||
auto tablet_schema = std::make_shared<TabletSchema>();
|
||||
tablet_schema->init_from_pb(schema_pb);
|
||||
return tablet_schema;
|
||||
}
|
||||
|
||||
// Minimal Segment stub: _apply_index_expr only consults num_rows().
|
||||
std::shared_ptr<Segment> make_stub_segment(uint32_t num_rows,
|
||||
const TabletSchemaSPtr& tablet_schema) {
|
||||
auto seg = std::make_shared<Segment>(0, RowsetId(), tablet_schema, InvertedIndexFileInfo());
|
||||
seg->_num_rows = num_rows;
|
||||
return seg;
|
||||
}
|
||||
|
||||
// VExprContext with a BitmapEvalExpr root and an index context to publish into.
|
||||
VExprContextSPtr make_bitmap_ctx(const std::shared_ptr<BitmapEvalExpr>& expr) {
|
||||
auto ctx = std::make_shared<VExprContext>(expr);
|
||||
std::vector<std::unique_ptr<IndexIterator>> index_iters;
|
||||
std::vector<IndexFieldNameAndTypePair> storage_types;
|
||||
std::unordered_map<ColumnId, std::unordered_map<const VExpr*, bool>> status_map;
|
||||
ColumnIteratorOptions column_iter_opts;
|
||||
auto index_ctx = std::make_shared<IndexExecContext>(index_iters, storage_types, status_map,
|
||||
nullptr, nullptr, column_iter_opts);
|
||||
ctx->set_index_context(index_ctx);
|
||||
return ctx;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
class SegmentIteratorConjunctShortCircuitTest : public testing::Test {
|
||||
protected:
|
||||
void SetUp() override {
|
||||
_tablet_schema = make_tablet_schema();
|
||||
_segment = make_stub_segment(100, _tablet_schema);
|
||||
_read_schema = std::make_shared<ReadSchema>(_tablet_schema->columns());
|
||||
_iter = std::make_unique<SegmentIterator>(_segment, _read_schema);
|
||||
|
||||
TQueryOptions query_options;
|
||||
query_options.__set_enable_fallback_on_missing_inverted_index(true);
|
||||
_runtime_state.set_query_options(query_options);
|
||||
|
||||
_iter->_opts.runtime_state = &_runtime_state;
|
||||
_iter->_opts.stats = &_stats;
|
||||
}
|
||||
|
||||
std::shared_ptr<Segment> _segment;
|
||||
std::shared_ptr<TabletSchema> _tablet_schema;
|
||||
ReadSchemaSPtr _read_schema;
|
||||
std::unique_ptr<SegmentIterator> _iter;
|
||||
RuntimeState _runtime_state;
|
||||
OlapReaderStatistics _stats;
|
||||
};
|
||||
|
||||
// Once an earlier conjunct's index result empties the row bitmap, remaining
|
||||
// conjuncts must not be evaluated against the index. With zero surviving
|
||||
// rows every remaining conjunct is trivially satisfied, so the whole
|
||||
// pushed-down list is consumed -- mirroring the column-predicate path and
|
||||
// keeping both the "all conditions consumed" contract (debug point
|
||||
// segment_iterator.apply_inverted_index) and the condition-cache digest
|
||||
// intact.
|
||||
TEST_F(SegmentIteratorConjunctShortCircuitTest, empty_bitmap_short_circuits_remaining) {
|
||||
_iter->_row_bitmap.addRange(0, 100);
|
||||
|
||||
auto empty_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {});
|
||||
auto expensive_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {0, 1, 2, 3, 4});
|
||||
auto empty_ctx = make_bitmap_ctx(empty_expr);
|
||||
auto expensive_ctx = make_bitmap_ctx(expensive_expr);
|
||||
_iter->_common_expr_ctxs_push_down = {empty_ctx, expensive_ctx};
|
||||
|
||||
ASSERT_TRUE(_iter->_apply_index_expr().ok());
|
||||
|
||||
// The empty result is intersected as soon as it is produced ...
|
||||
EXPECT_TRUE(_iter->_row_bitmap.isEmpty());
|
||||
// ... so the expensive conjunct is never evaluated against the index ...
|
||||
EXPECT_EQ(empty_expr->eval_count(), 1);
|
||||
EXPECT_EQ(expensive_expr->eval_count(), 0);
|
||||
// ... and the whole list is consumed: zero surviving rows satisfy every
|
||||
// remaining conjunct, exactly like the column-predicate short circuit.
|
||||
EXPECT_TRUE(_iter->_common_expr_ctxs_push_down.empty());
|
||||
// The skip is visible in reader statistics (profile:
|
||||
// InvertedIndexConjunctsShortCircuited).
|
||||
EXPECT_EQ(_stats.inverted_index_conjuncts_short_circuited, 1);
|
||||
}
|
||||
|
||||
// Once the candidate bitmap is empty, virtual-column MATCH projections must
|
||||
// not run their whole-segment index evaluation either: with zero surviving
|
||||
// rows their result column is never materialized, so the postings walk is
|
||||
// pure waste (SELECT msg MATCH_PHRASE_PREFIX '...' WHERE ns MATCH 'no-hit').
|
||||
TEST_F(SegmentIteratorConjunctShortCircuitTest, empty_bitmap_skips_virtual_column_projections) {
|
||||
_iter->_row_bitmap.addRange(0, 100);
|
||||
|
||||
auto empty_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {});
|
||||
auto projection_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {1, 2});
|
||||
_iter->_common_expr_ctxs_push_down = {make_bitmap_ctx(empty_expr)};
|
||||
_iter->_virtual_column_exprs[0] = make_bitmap_ctx(projection_expr);
|
||||
|
||||
ASSERT_TRUE(_iter->_apply_index_expr().ok());
|
||||
|
||||
EXPECT_TRUE(_iter->_row_bitmap.isEmpty());
|
||||
EXPECT_EQ(projection_expr->eval_count(), 0);
|
||||
}
|
||||
|
||||
// The ANN range-search pass iterates the same conjunct list; on an empty
|
||||
// bitmap a range search cannot produce rows, so with the small-candidate
|
||||
// fallback thresholds disabled it must still not load or search the index.
|
||||
TEST_F(SegmentIteratorConjunctShortCircuitTest, empty_bitmap_skips_ann_range_search) {
|
||||
_iter->_row_bitmap.addRange(0, 100);
|
||||
|
||||
auto empty_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {});
|
||||
auto ann_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {1, 2});
|
||||
// The short circuit keeps the second conjunct away from the inverted-index
|
||||
// loop; only the ANN pass below it iterates the (uncleared) list.
|
||||
_iter->_common_expr_ctxs_push_down = {make_bitmap_ctx(empty_expr), make_bitmap_ctx(ann_expr)};
|
||||
|
||||
ASSERT_TRUE(_iter->_apply_index_expr().ok());
|
||||
|
||||
EXPECT_TRUE(_iter->_row_bitmap.isEmpty());
|
||||
EXPECT_EQ(ann_expr->ann_eval_count(), 0);
|
||||
}
|
||||
|
||||
// An indexed prefix that empties the bitmap proves the WHOLE conjunction
|
||||
// false, which is exactly what the condition cache wants to remember. The
|
||||
// consumed-list clear must therefore keep the digest alive (an empty result
|
||||
// is correct for the full conjunction) instead of letting the empty list
|
||||
// zero it, which would silently drop the all-false cache entry that repeated
|
||||
// scans relied on before the short circuit existed.
|
||||
TEST_F(SegmentIteratorConjunctShortCircuitTest, exhausted_prefix_preserves_condition_cache_digest) {
|
||||
_iter->_row_bitmap.addRange(0, 100);
|
||||
_iter->_opts.condition_cache_digest = 12345;
|
||||
TQueryOptions query_options;
|
||||
query_options.__set_enable_fallback_on_missing_inverted_index(true);
|
||||
query_options.__set_enable_inverted_index_query(true);
|
||||
_runtime_state.set_query_options(query_options);
|
||||
|
||||
auto empty_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {});
|
||||
auto residual_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {0, 1});
|
||||
_iter->_common_expr_ctxs_push_down = {make_bitmap_ctx(empty_expr),
|
||||
make_bitmap_ctx(residual_expr)};
|
||||
|
||||
ASSERT_TRUE(_iter->_get_row_ranges_by_column_conditions().ok());
|
||||
|
||||
EXPECT_TRUE(_iter->_row_bitmap.isEmpty());
|
||||
EXPECT_TRUE(_iter->_common_expr_ctxs_push_down.empty());
|
||||
EXPECT_EQ(_iter->_opts.condition_cache_digest, 12345)
|
||||
<< "the all-false result is valid for the full conjunction and must stay cacheable";
|
||||
}
|
||||
|
||||
// Counter-case guarding against an over-wide digest fix: when every conjunct
|
||||
// is consumed normally (no short circuit, surviving rows remain), the digest
|
||||
// still zeroes exactly as before -- the preserved digest is only for the
|
||||
// proved-empty conjunction.
|
||||
TEST_F(SegmentIteratorConjunctShortCircuitTest, fully_consumed_conjuncts_still_zero_digest) {
|
||||
_iter->_row_bitmap.addRange(0, 100);
|
||||
_iter->_opts.condition_cache_digest = 12345;
|
||||
TQueryOptions query_options;
|
||||
query_options.__set_enable_fallback_on_missing_inverted_index(true);
|
||||
query_options.__set_enable_inverted_index_query(true);
|
||||
_runtime_state.set_query_options(query_options);
|
||||
|
||||
auto first_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {1, 2, 3});
|
||||
auto second_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {2, 3, 4});
|
||||
_iter->_common_expr_ctxs_push_down = {make_bitmap_ctx(first_expr),
|
||||
make_bitmap_ctx(second_expr)};
|
||||
|
||||
ASSERT_TRUE(_iter->_get_row_ranges_by_column_conditions().ok());
|
||||
|
||||
EXPECT_EQ(_iter->_row_bitmap.cardinality(), 2);
|
||||
EXPECT_TRUE(_iter->_common_expr_ctxs_push_down.empty());
|
||||
EXPECT_EQ(_iter->_opts.condition_cache_digest, 0);
|
||||
}
|
||||
|
||||
// ANN TopN falls back before touching the index once the candidate bitmap is
|
||||
// empty: there is nothing to select, and the veto that residual conjuncts
|
||||
// used to provide is gone after the consumed-list clear. The fixture leaves
|
||||
// _index_iterators empty, so reaching past the guard would index out of
|
||||
// bounds -- passing proves the guard runs before any ANN state is touched.
|
||||
TEST_F(SegmentIteratorConjunctShortCircuitTest, empty_bitmap_skips_ann_topn) {
|
||||
auto order_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {});
|
||||
_iter->_ann_topn_runtime =
|
||||
std::make_shared<AnnTopNRuntime>(true, 10, make_bitmap_ctx(order_expr));
|
||||
|
||||
ASSERT_TRUE(_iter->_apply_ann_topn_predicate().ok());
|
||||
}
|
||||
|
||||
// A conjunct whose index result does not empty the bitmap must not stop
|
||||
// evaluation of the following conjuncts; consumed results are intersected
|
||||
// progressively.
|
||||
TEST_F(SegmentIteratorConjunctShortCircuitTest, progressive_intersection_keeps_going) {
|
||||
_iter->_row_bitmap.addRange(0, 100);
|
||||
|
||||
auto first_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {1, 2, 3, 50, 60});
|
||||
auto second_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {2, 3, 4});
|
||||
auto first_ctx = make_bitmap_ctx(first_expr);
|
||||
auto second_ctx = make_bitmap_ctx(second_expr);
|
||||
_iter->_common_expr_ctxs_push_down = {first_ctx, second_ctx};
|
||||
|
||||
ASSERT_TRUE(_iter->_apply_index_expr().ok());
|
||||
|
||||
EXPECT_EQ(first_expr->eval_count(), 1);
|
||||
EXPECT_EQ(second_expr->eval_count(), 1);
|
||||
// {1,2,3,50,60} intersected with {2,3,4} is {2,3}.
|
||||
EXPECT_EQ(_iter->_row_bitmap.cardinality(), 2);
|
||||
EXPECT_TRUE(_iter->_row_bitmap.contains(2));
|
||||
EXPECT_TRUE(_iter->_row_bitmap.contains(3));
|
||||
// Both conjuncts were fully consumed by the index; nothing was skipped.
|
||||
EXPECT_TRUE(_iter->_common_expr_ctxs_push_down.empty());
|
||||
EXPECT_EQ(_stats.inverted_index_conjuncts_short_circuited, 0);
|
||||
}
|
||||
|
||||
} // namespace doris::segment_v2
|
||||
Reference in new issue
Block a user