mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Gabriel
c9d28771d8
### What problem does this PR solve? Lance profiles do not expose Session cache-miss observation windows, scalar-index task grouping cannot compare runtime costs, and Foyer memory capacity is tied to the data-cache read block size. This PR targets **branch-4.2** and adds: - `LanceSessionIndexCacheMisses` and `LanceSessionMetadataCacheMisses`, with availability and scope information. They report BE-wide Session deltas observed during a reader's dataset-open windows, including opening. Concurrent readers can overlap; these are not query-exclusive counts and must not be summed as an exact query total. - The startup BE option `enable_lance_scalar_index_auto_selection` (default `true`). FE keeps coverage-based task grouping and does not estimate selectivity or add a session variable. In BE, lance-c compares current BTree page min/max and null-count metadata for segments covering the entire task. Estimated work is matching pages times rows per page; a broad estimate can select a scoped full scan. Serialized lookup bytes, decoded buffers and probe work are bounded. Unknown estimates retain the existing segment path. The current native API cannot bound variable-width expansion, so string/binary costs remain unknown: retain such a grouping index or skip such an alternative before opening its lookup. Bitmap/LabelList remain supported on their existing path and are not cost-ranked; vector/FTS semantics are unchanged. - `lance_data_cache_memory_capacity_bytes`. Default `0` preserves the previous one-configured-block memory tier, including nondefault block sizes. Explicit capacities configure Foyer independently of block and disk capacities; negative values are rejected before unsigned conversion. The native selector and decoded-memory fix are now merged upstream in [lance-format/lance-c#98](https://github.com/lance-format/lance-c/pull/98). This PR pins upstream main commit `a51ab47a9c814a746cb1714c99fad93dd3a188b8`, refreshes the source archive checksum, and removes the local runtime-selection patch together with its application/checksum handling. The pre-existing Foyer patch is unchanged and applies with zero fuzz. Runtime decisions are exposed through Profile counters. Usage, cost-model limits and cache-counter scope are documented in `docs/lance-cache-and-index-selection.md`. ### Release note Add Lance Session cache-miss diagnostics, bounded BE runtime BTree selection controlled by a BE configuration, and independently configurable Foyer memory capacity. ### Validation - On the newly downloaded upstream archive plus the unchanged Foyer patch, `cargo test --offline --locked` passed all 515 tests, including 82 unit tests and 385 C API tests. The normal command ignores three C/C++ executable compilation tests. Coverage includes the compressed long-string allocation regression, decoded-budget limits, competing indices, scan fallback, fragment scope, NULL/deletion/stable-ID behavior and LIMIT/OFFSET. - C and C++ API sources passed syntax-only compilation against the newly patched headers. - Verified archive MD5, clean zero-fuzz patch application, cached no-op, stale-checksum source replacement, and equality of the resulting tested source. Selector/scanner/segment sources match merged upstream exactly. - All nine affected C++ files passed the repository formatting runner with clang-format 16.0.6. Shell syntax and staged whitespace checks passed. - Existing FE validation passed 74 focused tests and FE Checkstyle; this dependency synchronization changes no FE code. - Updated the array-predicate regression profile checks to account for explicit runtime cost-selected scans while retaining request/search/scan accounting, exact candidate counts and strict LabelList coverage. The actual assertion block passes six valid profile paths and rejects five invalid profiles; the previous assertion reproduces the observed failure. The focused native runtime-selection test and full-suite Groovy 4.0.19 parsing pass. Full external regression execution remains for CI. - Fixed Session counter publication for early LIMIT: active dataset-open windows are sampled before Thrift/protobuf profile copies, with synchronized delta accounting shared with close. All 14 focused C++ tests pass with ASAN/UBSAN and leak detection enabled, including a real Lance native read held in its statistics callback while another worker completes LIMIT 1. Disabling the snapshot hook reproduces both missing-window regressions (native saved misses 9 versus at least 19). The full reader and test sources pass syntax checks, and all 15 affected PR C++ files pass clang-format 16.0.6. - Full Doris BE build and unit-suite validation remain for CI and have not been verified locally. ### Check List (For Author) - Test - [x] Unit Test - [x] Manual test: native source/patch lifecycle and header compatibility checks above. - [ ] Regression test - Behavior changed: - [x] Yes. BE runtime BTree selection is enabled by default and can be disabled. Foyer's default memory behavior is preserved. - Does this need documentation? - [x] Yes. Usage and native memory limits are documented. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label