mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Gabriel
035a843648
### What problem does this PR solve? Parquet output relies on implicit page-index and data-page-size defaults, so users cannot explicitly omit indexes or tune the target page size. Add two validated session variables shared by OUTFILE, EXPORT, Hive and Iceberg Parquet writers: - `enable_parquet_write_page_index`: write ColumnIndex/OffsetIndex metadata; defaults to `true`, preserving the current master behavior. - `parquet_write_data_page_size`: target uncompressed data-page size in bytes; defaults to `1048576`, accepts positive 32-bit integers, and applies with indexes enabled or disabled. Keep the existing 128 KiB target cap for Hadoop-framed LZ4. Other codecs use the requested target. Row-group size, dictionary-page size, and data-page/format versions are unchanged. Optional query fields preserve existing defaults when omitted by older FEs. ### Release note Add session controls for Parquet page-index output and target data-page size. ### Validation - 16 BE ASAN tests passed, including real file metadata/readback, index on/off, dictionary on/off, null/all-null columns, Parquet versions, Data Page V1/V2, page offset bounds, Hadoop-framed LZ4 limits, and Hive/Iceberg partition writers. - 9 FE tests passed for defaults, variable forwarding, Thrift serialization, valid bounds, and invalid input. - The new tests failed for the missing settings before implementation. - FE Checkstyle, clang-format 16, build hygiene, and clang-tidy passed. - No cluster SQL run or performance benchmark was performed locally; CI validation is pending. ### Check List (For Author) - Test: - [x] Unit Test - Behavior changed: - [x] Yes. Explicit controls are available; current defaults are preserved. - Does this need documentation? - [x] Yes. English and Chinese documentation: https://github.com/apache/doris-website/pull/4581 ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label