[improvement](inverted index) Short-circuit pushed-down conjuncts once the row bitmap is empty (#67171)

### What problem does this PR solve?

Problem Summary:

On log-search workloads a query typically carries several pushed-down
conjuncts, e.g. two cheap untokenized MATCH terms plus one expensive
`msg MATCH_PHRASE_PREFIX '...'`:

```sql
SELECT * FROM logs
WHERE _ctime_ >= ... AND _ctime_ <= ...
  AND _namespace_ MATCH 'ns'            -- untokenized term, ~free
  AND _pod_name_ MATCH 'pod-xyz'        -- untokenized term, ~free
  AND msg MATCH_PHRASE_PREFIX '...'     -- whole-segment postings + positions
ORDER BY _ctime_ DESC LIMIT 1000
```

`SegmentIterator` evaluated ALL pushed-down conjuncts against the index
first and only intersected their bitmaps into `_row_bitmap` afterwards,
so the phrase conjunct paid its full whole-segment postings/positions
cost even when an earlier selective conjunct had already emptied the
candidate bitmap. Reproduced on a production log table: one query
returning 0 rows burned 11,291 CPU seconds and 124 GiB of local index
reads per physical SQL, virtually all of it inside phrase evaluation
(profile: 2,657 segments, InvertedIndexSearcherSearchExecTime = 99.7% of
scan cost, ScanRows = 0).

The empty-bitmap short circuit already exists on the neighbouring paths:
column predicates (`continue_apply` in `_apply_inverted_index`) and
compound AND inside a single expression (`VCompoundPred` COMPOUND_AND
early exit). But top-level AND conjuncts are flattened into separate
expr contexts and take `_apply_index_expr`, which had neither
progressive intersection nor a short circuit, so the most common query
shape never benefited.

Fix: intersect each consumed index result into `_row_bitmap` as soon as
it is produced, and stop evaluating further conjuncts once the bitmap is
empty. A skipped conjunct stays pushed down, so the row-level path keeps
its exact semantics over the now-empty candidate set (zero rows read,
zero cost). Consumed conjuncts are erased from
`_common_expr_ctxs_push_down` only after the ANN range-search pass,
which iterates the same list. A new profile counter
`InvertedIndexConjunctsShortCircuited` reports how many conjuncts were
skipped.

Behavior notes:
- Index results are now intersected before the ANN range-search pass, so
ANN range search executes on the already-narrowed bitmap. AND semantics
are commutative, and a smaller candidate set is what its small-candidate
fallback is designed for.
- Condition cache digest semantics unchanged: a short-circuited (thus
still pushed-down) conjunct keeps the digest cleared, so partial index
results are never cached as full coverage.

### Release note

Inverted index: once earlier pushed-down conjuncts empty the candidate
row bitmap, remaining conjuncts (e.g. expensive MATCH_PHRASE_PREFIX) are
no longer evaluated against the index.

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
- [x] Yes. <!-- Index results are intersected progressively (before the
ANN pass); short-circuited conjuncts fall back to the row-level path
over an empty candidate set. Same query results, strictly less index
work. -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

---

Review round 2 (automated review, addressed in the third commit):

1. Dead work after the short circuit is now bypassed on an empty
candidate
bitmap: virtual-column MATCH projections skip their whole-segment index
evaluation, the ANN range-search pass breaks out before
loading/searching,
and ANN TopN falls back at entry (all three previously ran even with
zero
surviving rows when the small-candidate fallback thresholds are
disabled).
2. The consumed-list clear no longer drops the proven all-false
condition-cache entry: `_index_conjuncts_proved_empty` keeps the digest
alive for the exhausted case (the empty result is correct for the full
conjunction), while a counter-case test pins the normal fully-consumed
   path to its existing digest-zeroing behavior.
This commit is contained in:
Jack authored and GitHub committed 2026-08-31 09:13:42 +08:00
1 parent c800f2aa61
commit 40ff995f89
7 files changed
+405 -1

No files matched your search

@@ -290,6 +290,8 @@ Status OlapScanLocalState::_init_profile() {
_segment_profile, "InvertedIndexSearcherCacheMiss", TUnit::UNIT, 1);
_inverted_index_downgrade_count_counter =
ADD_COUNTER_WITH_LEVEL(_segment_profile, "InvertedIndexDowngradeCount", TUnit::UNIT, 1);
_inverted_index_conjuncts_short_circuited_counter = ADD_COUNTER_WITH_LEVEL(
_segment_profile, "InvertedIndexConjunctsShortCircuited", TUnit::UNIT, 1);
_inverted_index_analyzer_timer =
ADD_TIMER_WITH_LEVEL(_segment_profile, "InvertedIndexAnalyzerTime", 1);
_inverted_index_lookup_timer =
@@ -265,6 +265,7 @@ private:
RuntimeProfile::Counter* _inverted_index_searcher_cache_hit_counter = nullptr;
RuntimeProfile::Counter* _inverted_index_searcher_cache_miss_counter = nullptr;
RuntimeProfile::Counter* _inverted_index_downgrade_count_counter = nullptr;
RuntimeProfile::Counter* _inverted_index_conjuncts_short_circuited_counter = nullptr;
RuntimeProfile::Counter* _inverted_index_analyzer_timer = nullptr;
RuntimeProfile::Counter* _inverted_index_lookup_timer = nullptr;
+2
View File
@@ -881,6 +881,8 @@ void OlapScanner::_collect_profile_before_close() {
stats.inverted_index_searcher_cache_miss);
COUNTER_UPDATE(local_state->_inverted_index_downgrade_count_counter,
stats.inverted_index_downgrade_count);
COUNTER_UPDATE(local_state->_inverted_index_conjuncts_short_circuited_counter,
stats.inverted_index_conjuncts_short_circuited);
COUNTER_UPDATE(local_state->_inverted_index_analyzer_timer,
stats.inverted_index_analyzer_timer);
COUNTER_UPDATE(local_state->_inverted_index_lookup_timer, stats.inverted_index_lookup_timer);
+3
View File
@@ -325,6 +325,9 @@ struct OlapReaderStatistics {
int64_t inverted_index_searcher_cache_hit = 0;
int64_t inverted_index_searcher_cache_miss = 0;
int64_t inverted_index_downgrade_count = 0;
// Pushed-down conjuncts skipped (never index-evaluated) because the row
// bitmap was already empty when their turn came.
int64_t inverted_index_conjuncts_short_circuited = 0;
int64_t inverted_index_analyzer_timer = 0;
int64_t inverted_index_lookup_timer = 0;
// See snii_query_stats.h: one field here instead of one per SNII counter.
+62 -1
View File
@@ -858,7 +858,9 @@ Status SegmentIterator::_get_row_ranges_by_column_conditions() {
}
}
_opts.condition_cache_digest =
_common_expr_ctxs_push_down.empty() ? 0 : _opts.condition_cache_digest;
_common_expr_ctxs_push_down.empty() && !_index_conjuncts_proved_empty
? 0
: _opts.condition_cache_digest;
_opts.stats->rows_inverted_index_filtered += (input_rows - _row_bitmap.cardinality());
for (uint32_t cid = 0; cid < _schema->num_read_columns(); ++cid) {
bool result_true = _check_all_conditions_passed_inverted_index_for_column(cid);
@@ -925,6 +927,13 @@ Status SegmentIterator::_apply_ann_topn_predicate() {
if (_ann_topn_runtime == nullptr) {
return Status::OK();
}
if (_row_bitmap.isEmpty()) {
// Zero candidates: nothing for TopN to select, and the residual
// conjuncts that used to veto this path may have been consumed by the
// proved-empty short circuit. Fall back before touching the ANN index
// (the small-candidate fallback may be disabled by its thresholds).
return Status::OK();
}
VLOG_DEBUG << fmt::format("Try apply ann topn: {}", _ann_topn_runtime->debug_string());
// AnnTopNRuntime keeps VSlotRef::column_id(), which is the read-schema ordinal.
@@ -1237,7 +1246,23 @@ Status SegmentIterator::_apply_index_expr() {
!_opts.runtime_state->query_options().__isset.enable_ann_index_result_cache ||
_opts.runtime_state->query_options().enable_ann_index_result_cache;
// Intersect each consumed index result into _row_bitmap right away so a
// selective conjunct short-circuits the remaining (potentially expensive,
// e.g. MATCH_PHRASE_PREFIX) ones. A skipped conjunct stays pushed down and
// keeps its semantics on the row-level path, which then sees zero rows.
// Consumed conjuncts are erased only after the ANN pass below, which
// iterates the same list.
std::vector<const VExprContext*> consumed_by_index;
bool bitmap_exhausted = false;
size_t considered_conjuncts = 0;
for (const auto& expr_ctx : _common_expr_ctxs_push_down) {
if (_row_bitmap.isEmpty()) {
_opts.stats->inverted_index_conjuncts_short_circuited +=
_common_expr_ctxs_push_down.size() - considered_conjuncts;
bitmap_exhausted = true;
break;
}
++considered_conjuncts;
if (Status st = expr_ctx->evaluate_inverted_index(num_rows()); !st.ok()) {
if (_downgrade_without_index(st) || st.code() == ErrorCode::NOT_IMPLEMENTED_ERROR) {
continue;
@@ -1249,12 +1274,25 @@ Status SegmentIterator::_apply_index_expr() {
return st;
}
}
if (expr_ctx->all_expr_inverted_index_evaluated()) {
const auto* result = expr_ctx->get_index_context()->get_index_result_for_expr(
expr_ctx->root().get());
if (result != nullptr) {
_row_bitmap &= *result->get_data_bitmap();
consumed_by_index.push_back(expr_ctx.get());
}
}
}
// Evaluate inverted index for virtual column MATCH expressions (projections).
// Unlike common exprs which filter rows, these only compute index result bitmaps
// for later materialization via fast_execute().
for (auto& [cid, expr_ctx] : _virtual_column_exprs) {
if (_row_bitmap.isEmpty()) {
// Zero surviving rows: the projection column is never materialized,
// so its whole-segment index evaluation would be pure waste.
break;
}
if (expr_ctx->get_index_context() == nullptr) {
continue;
}
@@ -1272,6 +1310,13 @@ Status SegmentIterator::_apply_index_expr() {
// Apply ann range search
for (const auto& expr_ctx : _common_expr_ctxs_push_down) {
if (_row_bitmap.isEmpty()) {
// A range search intersects into the bitmap; with zero candidates
// it cannot add rows, so loading and searching the ANN index
// (bypassing the small-candidate fallback when its thresholds are
// disabled) would be pure waste.
break;
}
segment_v2::AnnIndexStats ann_index_stats;
size_t origin_rows = _row_bitmap.cardinality();
bool ann_range_search_executed = false;
@@ -1302,6 +1347,22 @@ Status SegmentIterator::_apply_index_expr() {
_opts.stats->ann_index_range_cache_hits += ann_index_stats.range_cache_hits.value();
}
if (bitmap_exhausted) {
// Zero surviving rows satisfy every remaining conjunct, so the whole
// list is consumed -- mirroring the column-predicate short circuit.
// This keeps the "all conditions consumed by the index" contract, and
// _index_conjuncts_proved_empty keeps the condition-cache digest
// alive: the all-false result is correct for the full conjunction, so
// clearing the list must not degrade it to "nothing left to cache".
_index_conjuncts_proved_empty = true;
_common_expr_ctxs_push_down.clear();
} else if (!consumed_by_index.empty()) {
std::erase_if(_common_expr_ctxs_push_down, [&](const VExprContextSPtr& ctx) {
return std::find(consumed_by_index.begin(), consumed_by_index.end(), ctx.get()) !=
consumed_by_index.end();
});
}
return Status::OK();
}
@@ -484,6 +484,14 @@ private:
bool _count_fastpath_hit = false;
bool _count_emit_shortcut = false;
uint64_t _count_emit_rows_remaining = 0;
// An indexed conjunct prefix emptied _row_bitmap, proving the WHOLE
// pushed-down conjunction false. Set by the _apply_index_expr short
// circuit when it consumes (clears) the remaining conjuncts, and read
// where an empty conjunct list would otherwise zero the condition-cache
// digest: the all-false result stays valid for the full conjunction, so
// it must remain cacheable.
bool _index_conjuncts_proved_empty = false;
// Batch size for shortcut emission: VStatisticsIterator's
// MAX_ROW_SIZE_IN_COUNT, the largest default-rows block shape already
// proven through every consumer above the segment iterator by the plain
@@ -0,0 +1,327 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
// White-box tests for progressive intersection and short-circuit of pushed-down
// conjuncts in SegmentIterator::_apply_index_expr. Once an earlier conjunct's
// index result empties _row_bitmap, the remaining (potentially expensive, e.g.
// MATCH_PHRASE_PREFIX) conjuncts must not be evaluated against the index at
// all; skipped conjuncts stay pushed down so the row-level path keeps their
// semantics over the (now empty) candidate set. Uses the established
// `#define private public` convention of segment_iterator_limit_opt_test.cpp.
#include <gtest/gtest.h>
#include <cstdint>
#include <memory>
#include <vector>
#include "common/status.h"
#include "core/data_type/data_type_number.h"
#include "exprs/vexpr.h"
#include "exprs/vexpr_context.h"
#include "runtime/runtime_state.h"
#include "storage/index/ann/ann_topn_runtime.h"
#include "storage/index/inverted/inverted_index_reader.h"
#include "storage/olap_common.h"
#include "storage/tablet/tablet_schema.h"
#if defined(__clang__)
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wkeyword-macro"
#endif
#define private public
#define protected public
#include "storage/segment/segment.h"
#include "storage/segment/segment_iterator.h"
#undef private
#undef protected
#if defined(__clang__)
#pragma clang diagnostic pop
#endif
namespace doris::segment_v2 {
namespace {
// A test VExpr that counts evaluations and registers a fixed-cardinality index
// result bitmap, mimicking how real match exprs publish results.
class BitmapEvalExpr : public VExpr {
public:
explicit BitmapEvalExpr(std::vector<uint32_t> rows) : _rows(std::move(rows)) {
_data_type = std::make_shared<DataTypeUInt8>();
}
const std::string& expr_name() const override {
static const std::string kName = "BitmapEvalExpr";
return kName;
}
Status execute(VExprContext*, Block*, int*) const override { return Status::OK(); }
Status execute_column_impl(VExprContext* context, const Block* block, const Selector* selector,
size_t count, ColumnPtr& result_column) const override {
return Status::OK();
}
Status evaluate_inverted_index(VExprContext* context, uint32_t segment_num_rows) override {
++_eval_count;
auto data = std::make_shared<roaring::Roaring>();
for (uint32_t row : _rows) {
data->add(row);
}
InvertedIndexResultBitmap result(std::move(data), std::make_shared<roaring::Roaring>());
context->get_index_context()->set_index_result_for_expr(this, std::move(result));
return Status::OK();
}
Status evaluate_ann_range_search(
const segment_v2::AnnRangeSearchRuntime&,
const std::vector<std::unique_ptr<segment_v2::IndexIterator>>&,
const std::vector<std::unique_ptr<segment_v2::ColumnIterator>>&, size_t,
roaring::Roaring&, segment_v2::AnnIndexStats&, bool,
AnnRangeSearchEvaluationResult& result) override {
++_ann_eval_count;
result.executed = false;
return Status::OK();
}
int eval_count() const { return _eval_count; }
int ann_eval_count() const { return _ann_eval_count; }
private:
std::vector<uint32_t> _rows;
int _eval_count = 0;
int _ann_eval_count = 0;
};
TabletSchemaSPtr make_tablet_schema() {
TabletSchemaPB schema_pb;
schema_pb.set_keys_type(KeysType::DUP_KEYS);
auto* col = schema_pb.add_column();
col->set_unique_id(0);
col->set_name("k0");
col->set_type("INT");
col->set_is_key(true);
col->set_is_nullable(false);
auto tablet_schema = std::make_shared<TabletSchema>();
tablet_schema->init_from_pb(schema_pb);
return tablet_schema;
}
// Minimal Segment stub: _apply_index_expr only consults num_rows().
std::shared_ptr<Segment> make_stub_segment(uint32_t num_rows,
const TabletSchemaSPtr& tablet_schema) {
auto seg = std::make_shared<Segment>(0, RowsetId(), tablet_schema, InvertedIndexFileInfo());
seg->_num_rows = num_rows;
return seg;
}
// VExprContext with a BitmapEvalExpr root and an index context to publish into.
VExprContextSPtr make_bitmap_ctx(const std::shared_ptr<BitmapEvalExpr>& expr) {
auto ctx = std::make_shared<VExprContext>(expr);
std::vector<std::unique_ptr<IndexIterator>> index_iters;
std::vector<IndexFieldNameAndTypePair> storage_types;
std::unordered_map<ColumnId, std::unordered_map<const VExpr*, bool>> status_map;
ColumnIteratorOptions column_iter_opts;
auto index_ctx = std::make_shared<IndexExecContext>(index_iters, storage_types, status_map,
nullptr, nullptr, column_iter_opts);
ctx->set_index_context(index_ctx);
return ctx;
}
} // namespace
class SegmentIteratorConjunctShortCircuitTest : public testing::Test {
protected:
void SetUp() override {
_tablet_schema = make_tablet_schema();
_segment = make_stub_segment(100, _tablet_schema);
_read_schema = std::make_shared<ReadSchema>(_tablet_schema->columns());
_iter = std::make_unique<SegmentIterator>(_segment, _read_schema);
TQueryOptions query_options;
query_options.__set_enable_fallback_on_missing_inverted_index(true);
_runtime_state.set_query_options(query_options);
_iter->_opts.runtime_state = &_runtime_state;
_iter->_opts.stats = &_stats;
}
std::shared_ptr<Segment> _segment;
std::shared_ptr<TabletSchema> _tablet_schema;
ReadSchemaSPtr _read_schema;
std::unique_ptr<SegmentIterator> _iter;
RuntimeState _runtime_state;
OlapReaderStatistics _stats;
};
// Once an earlier conjunct's index result empties the row bitmap, remaining
// conjuncts must not be evaluated against the index. With zero surviving
// rows every remaining conjunct is trivially satisfied, so the whole
// pushed-down list is consumed -- mirroring the column-predicate path and
// keeping both the "all conditions consumed" contract (debug point
// segment_iterator.apply_inverted_index) and the condition-cache digest
// intact.
TEST_F(SegmentIteratorConjunctShortCircuitTest, empty_bitmap_short_circuits_remaining) {
_iter->_row_bitmap.addRange(0, 100);
auto empty_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {});
auto expensive_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {0, 1, 2, 3, 4});
auto empty_ctx = make_bitmap_ctx(empty_expr);
auto expensive_ctx = make_bitmap_ctx(expensive_expr);
_iter->_common_expr_ctxs_push_down = {empty_ctx, expensive_ctx};
ASSERT_TRUE(_iter->_apply_index_expr().ok());
// The empty result is intersected as soon as it is produced ...
EXPECT_TRUE(_iter->_row_bitmap.isEmpty());
// ... so the expensive conjunct is never evaluated against the index ...
EXPECT_EQ(empty_expr->eval_count(), 1);
EXPECT_EQ(expensive_expr->eval_count(), 0);
// ... and the whole list is consumed: zero surviving rows satisfy every
// remaining conjunct, exactly like the column-predicate short circuit.
EXPECT_TRUE(_iter->_common_expr_ctxs_push_down.empty());
// The skip is visible in reader statistics (profile:
// InvertedIndexConjunctsShortCircuited).
EXPECT_EQ(_stats.inverted_index_conjuncts_short_circuited, 1);
}
// Once the candidate bitmap is empty, virtual-column MATCH projections must
// not run their whole-segment index evaluation either: with zero surviving
// rows their result column is never materialized, so the postings walk is
// pure waste (SELECT msg MATCH_PHRASE_PREFIX '...' WHERE ns MATCH 'no-hit').
TEST_F(SegmentIteratorConjunctShortCircuitTest, empty_bitmap_skips_virtual_column_projections) {
_iter->_row_bitmap.addRange(0, 100);
auto empty_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {});
auto projection_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {1, 2});
_iter->_common_expr_ctxs_push_down = {make_bitmap_ctx(empty_expr)};
_iter->_virtual_column_exprs[0] = make_bitmap_ctx(projection_expr);
ASSERT_TRUE(_iter->_apply_index_expr().ok());
EXPECT_TRUE(_iter->_row_bitmap.isEmpty());
EXPECT_EQ(projection_expr->eval_count(), 0);
}
// The ANN range-search pass iterates the same conjunct list; on an empty
// bitmap a range search cannot produce rows, so with the small-candidate
// fallback thresholds disabled it must still not load or search the index.
TEST_F(SegmentIteratorConjunctShortCircuitTest, empty_bitmap_skips_ann_range_search) {
_iter->_row_bitmap.addRange(0, 100);
auto empty_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {});
auto ann_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {1, 2});
// The short circuit keeps the second conjunct away from the inverted-index
// loop; only the ANN pass below it iterates the (uncleared) list.
_iter->_common_expr_ctxs_push_down = {make_bitmap_ctx(empty_expr), make_bitmap_ctx(ann_expr)};
ASSERT_TRUE(_iter->_apply_index_expr().ok());
EXPECT_TRUE(_iter->_row_bitmap.isEmpty());
EXPECT_EQ(ann_expr->ann_eval_count(), 0);
}
// An indexed prefix that empties the bitmap proves the WHOLE conjunction
// false, which is exactly what the condition cache wants to remember. The
// consumed-list clear must therefore keep the digest alive (an empty result
// is correct for the full conjunction) instead of letting the empty list
// zero it, which would silently drop the all-false cache entry that repeated
// scans relied on before the short circuit existed.
TEST_F(SegmentIteratorConjunctShortCircuitTest, exhausted_prefix_preserves_condition_cache_digest) {
_iter->_row_bitmap.addRange(0, 100);
_iter->_opts.condition_cache_digest = 12345;
TQueryOptions query_options;
query_options.__set_enable_fallback_on_missing_inverted_index(true);
query_options.__set_enable_inverted_index_query(true);
_runtime_state.set_query_options(query_options);
auto empty_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {});
auto residual_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {0, 1});
_iter->_common_expr_ctxs_push_down = {make_bitmap_ctx(empty_expr),
make_bitmap_ctx(residual_expr)};
ASSERT_TRUE(_iter->_get_row_ranges_by_column_conditions().ok());
EXPECT_TRUE(_iter->_row_bitmap.isEmpty());
EXPECT_TRUE(_iter->_common_expr_ctxs_push_down.empty());
EXPECT_EQ(_iter->_opts.condition_cache_digest, 12345)
<< "the all-false result is valid for the full conjunction and must stay cacheable";
}
// Counter-case guarding against an over-wide digest fix: when every conjunct
// is consumed normally (no short circuit, surviving rows remain), the digest
// still zeroes exactly as before -- the preserved digest is only for the
// proved-empty conjunction.
TEST_F(SegmentIteratorConjunctShortCircuitTest, fully_consumed_conjuncts_still_zero_digest) {
_iter->_row_bitmap.addRange(0, 100);
_iter->_opts.condition_cache_digest = 12345;
TQueryOptions query_options;
query_options.__set_enable_fallback_on_missing_inverted_index(true);
query_options.__set_enable_inverted_index_query(true);
_runtime_state.set_query_options(query_options);
auto first_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {1, 2, 3});
auto second_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {2, 3, 4});
_iter->_common_expr_ctxs_push_down = {make_bitmap_ctx(first_expr),
make_bitmap_ctx(second_expr)};
ASSERT_TRUE(_iter->_get_row_ranges_by_column_conditions().ok());
EXPECT_EQ(_iter->_row_bitmap.cardinality(), 2);
EXPECT_TRUE(_iter->_common_expr_ctxs_push_down.empty());
EXPECT_EQ(_iter->_opts.condition_cache_digest, 0);
}
// ANN TopN falls back before touching the index once the candidate bitmap is
// empty: there is nothing to select, and the veto that residual conjuncts
// used to provide is gone after the consumed-list clear. The fixture leaves
// _index_iterators empty, so reaching past the guard would index out of
// bounds -- passing proves the guard runs before any ANN state is touched.
TEST_F(SegmentIteratorConjunctShortCircuitTest, empty_bitmap_skips_ann_topn) {
auto order_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {});
_iter->_ann_topn_runtime =
std::make_shared<AnnTopNRuntime>(true, 10, make_bitmap_ctx(order_expr));
ASSERT_TRUE(_iter->_apply_ann_topn_predicate().ok());
}
// A conjunct whose index result does not empty the bitmap must not stop
// evaluation of the following conjuncts; consumed results are intersected
// progressively.
TEST_F(SegmentIteratorConjunctShortCircuitTest, progressive_intersection_keeps_going) {
_iter->_row_bitmap.addRange(0, 100);
auto first_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {1, 2, 3, 50, 60});
auto second_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {2, 3, 4});
auto first_ctx = make_bitmap_ctx(first_expr);
auto second_ctx = make_bitmap_ctx(second_expr);
_iter->_common_expr_ctxs_push_down = {first_ctx, second_ctx};
ASSERT_TRUE(_iter->_apply_index_expr().ok());
EXPECT_EQ(first_expr->eval_count(), 1);
EXPECT_EQ(second_expr->eval_count(), 1);
// {1,2,3,50,60} intersected with {2,3,4} is {2,3}.
EXPECT_EQ(_iter->_row_bitmap.cardinality(), 2);
EXPECT_TRUE(_iter->_row_bitmap.contains(2));
EXPECT_TRUE(_iter->_row_bitmap.contains(3));
// Both conjuncts were fully consumed by the index; nothing was skipped.
EXPECT_TRUE(_iter->_common_expr_ctxs_push_down.empty());
EXPECT_EQ(_stats.inverted_index_conjuncts_short_circuited, 0);
}
} // namespace doris::segment_v2