mirror of
https://github.com/apache/doris.git
synced 2026-11-04 15:39:47 +00:00
Gabriel
97f7f957a8
### What problem does this PR solve? File Scanner V2 could only use row-level Parquet dictionary filtering efficiently for a narrow set of predicates and projected INT columns. Range predicates and other typed dictionaries could still pay per-entry generic expression evaluation, generic SerDe insertion, or row-sized predicate-column materialization. ### What is changed? - Enable exact typed dictionary evaluation for `=`, `!=`, `<`, `<=`, `>`, and `>=`, including symmetric literal-on-left comparisons. - Accept both `RLE_DICTIONARY` and legacy `PLAIN_DICTIONARY` data-page encodings for supported primitive columns. - Build numeric dictionary bitmaps over contiguous typed INT, BIGINT, FLOAT, and DOUBLE dictionary values, evaluating each conjunct once per dictionary generation. - Build string equality and range bitmaps by comparing dictionary slices directly, without per-entry `Field` construction or generic dictionary expression dispatch. - Decode selected dictionary IDs at the native page-reader layer and apply the per-entry bitmap without materializing a complete predicate value column. - Write all supported fixed-width survivors directly from the filter loop into the target column. - Gather string survivors with pre-sized character/offset buffers and one copy per selected dictionary value. - Add direct-path profile counters and focused INT32/BIGINT/BYTE_ARRAY dictionary microbench scenarios. ### Verification - ASAN focused tests: 14/14 passed, covering typed numeric/string range filters, literal-on-left normalization, fixed-width fused gather, compact string gather, and benchmark scenario registration. - Related ASAN suite: 487 applicable tests passed across Parquet, File Scanner V2, SerDe, column mapping, JSON, WAL, and remote reader coverage. Three unrelated Flight tests could not bind their fixed localhost endpoint because it was already occupied by a pre-existing service. - `git diff --check` passed. ### Microbenchmark: upstream master vs this PR Compared upstream master `7809a73814e` directly with this PR at `943c0b94ffc`. Both binaries use the same Release compiler options, benchmark harness, fixtures, and fixed CPU. Each binary received three warmups; measurements used interleaved master/PR ordering and 20 samples per scenario. Values below are median CPU time. `raw_rows`, `selected_rows`, and fixture sizes were identical for every pair. | Dictionary type | Selectivity | Projection | Master | This PR | Change | |---|---:|---|---:|---:|---:| | INT32 | 10% | predicate only | 1,332,627 ns | 329,241 ns | -75.3% | | INT32 | 50% | predicate only | 1,344,746 ns | 338,433 ns | -74.8% | | INT32 | 10% | predicate projected | 1,686,162 ns | 1,000,973 ns | -40.6% | | INT32 | 50% | predicate projected | 1,713,684 ns | 1,041,358 ns | -39.2% | | BIGINT | 10% | predicate only | 1,053,504 ns | 329,214 ns | -68.8% | | BIGINT | 50% | predicate only | 1,053,577 ns | 340,627 ns | -67.7% | | BIGINT | 10% | predicate projected | 1,367,541 ns | 996,056 ns | -27.2% | | BIGINT | 50% | predicate projected | 1,438,891 ns | 1,053,813 ns | -26.8% | | BYTE_ARRAY | 10% | predicate only | 2,007,277 ns | 386,762 ns | -80.7% | | BYTE_ARRAY | 50% | predicate only | 2,017,380 ns | 398,810 ns | -80.2% | | BYTE_ARRAY | 10% | predicate projected | 2,437,263 ns | 1,194,903 ns | -51.0% | | BYTE_ARRAY | 50% | predicate projected | 2,541,460 ns | 1,323,767 ns | -47.9% | All twelve direct master-to-PR scenarios improved. Predicate-only scans avoid row-sized materialization and generic per-entry dispatch; projected scans additionally avoid generic survivor insertion.