mirror of
https://github.com/apache/doris.git
synced 2026-11-04 15:39:47 +00:00
Gabriel
49da8785c8
### What problem does this PR solve?
File scans maintained both the legacy scanner/readers and the V2 reader
stack. Route query scans, schema inference, file loads and external
row-ID fetches through FileScannerV2, migrate standalone Push and
Iceberg delete-file consumers, and remove the retired implementation and
tests.
- Keep enable_file_scanner_v2 as a deprecated no-op, with the existing
optional Thrift field emitted/defaulted to true.
- Preserve ORC session/adaptive batch caps, stripe dictionary rejection,
lazy predicate-column reuse and sequential CSV/TEXT/JSON S3 prefetch.
- Filter matching ORC dictionary IDs before string expansion in lazy,
eager and predicate-only scans. Retain residual predicates, NULL
semantics, pruning fences, physical column identity and physical row
positions.
- Restore ORC serialized-tail caching using the existing V1
FileMetaCache key policy, including its file-size fallback when mtime is
absent. Keep V2 payload types separate. ORC footer counters have
format-qualified names and publish once across repeated close; native
Parquet cache identity remains unchanged.
- Bound standalone position-delete reads to the native Parquet batch
limit. Broker source projections follow validated src_slot_ids order,
and malformed ORC dictionary offsets return CORRUPTION.
- Retain eight active V2 position-delete system-table regressions and
the operator shared-cache/shard-limit tests in maintained suites.
- Preserve all six Hive nested STRUCT/ARRAY/MAP access-path scenarios in
V2 across ORC/Parquet and name/position mapping, including separate
predicate-column IDs and complete-versus-projected row comparisons.
- Preserve active-file error context and the upstream Arrow column/type
context without changing original validation status codes.
Related PR: #69668
The supported rolling-upgrade direction remains BE first, then FE.
Old-FE/new-BE requests without semantic COUNT arguments retain their
ordinary-read fallback. New-FE/old-BE nullable COUNT is outside that
documented upgrade direction; the deprecated scanner option does not
provide a legacy routing gate.
The branch resolves the modify/delete conflict with master at
339e087b11. The old Arrow reader stays deleted, and its upstream
error-context fix is preserved in the V2 reader.
### Release note
File scans always use FileScannerV2. enable_file_scanner_v2 is
deprecated and has no effect. Unsupported operations fail in their
reader.
### Validation
- 565 selected BE ASAN tests passed on the rebased code: 190 ORC, 75
Parquet, 118 Iceberg reader, 21 delete-file helper, 28 scanner, 43 CSV,
43 Arrow load, 18 load-reader, 11 JSON load, 11 position-delete
system-table, 4 metadata-cache and 3 JSON-prefetch tests.
- The Hive follow-up passed 182 related BE ASAN tests, including 24
nested-path matrix cases. Selected values, parent null maps and
ARRAY/MAP offsets match full scans; KEYS-only MAP expectations retain
the existing V2 value-column fallback.
- The new 102401-row position-delete case failed before the batch-cap
fix and now checks every visited position.
- The selective dictionary case reproduced about 256 MiB of
rejected-string expansion before the fix. It now bounds materialized
strings and checks returned values and physical condition-cache granules
with lazy materialization enabled, disabled, and no follower column.
- Warm ORC footer opens perform zero physical pread calls. Versioned
keys, the legacy versionless file-size fallback, disabled caching and
repeated close are covered. Both ORC/Parquet profile initialization
orders retain independent footer counters.
- Arrow Date64 validation retains INVALID_ARGUMENT and reports the input
column and types.
- The cache/profile follow-up passed 311 related BE ASAN tests (278
reader/cache/profile and 33 scanner/operator/push tests). Two valid
operator cache tests were preserved from the retired fixture.
- FE core and all five affected connector modules pass Checkstyle.
clang-format 16 passes on all 60 affected surviving C/C++ files. Build
hygiene and changed-line clang-tidy pass.
External-service regression suites and full CI are not fresh results
from this update. No end-to-end performance benchmark is claimed. CHAR
dictionary pruning remains outside the restored STRING/VARCHAR path.
### Check List (For Author)
- [x] Unit Test
- [ ] External-service regression test on the updated head
- [x] Documentation synchronized
### Check List (For Reviewer who merge this PR)
- [ ] Confirm release note
- [ ] Confirm tests and documentation
- [ ] Add branch pick label