mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Gabriel
035a843648
### What problem does this PR solve? Parquet output relies on implicit page-index and data-page-size defaults, so users cannot explicitly omit indexes or tune the target page size. Add two validated session variables shared by OUTFILE, EXPORT, Hive and Iceberg Parquet writers: - `enable_parquet_write_page_index`: write ColumnIndex/OffsetIndex metadata; defaults to `true`, preserving the current master behavior. - `parquet_write_data_page_size`: target uncompressed data-page size in bytes; defaults to `1048576`, accepts positive 32-bit integers, and applies with indexes enabled or disabled. Keep the existing 128 KiB target cap for Hadoop-framed LZ4. Other codecs use the requested target. Row-group size, dictionary-page size, and data-page/format versions are unchanged. Optional query fields preserve existing defaults when omitted by older FEs. ### Release note Add session controls for Parquet page-index output and target data-page size. ### Validation - 16 BE ASAN tests passed, including real file metadata/readback, index on/off, dictionary on/off, null/all-null columns, Parquet versions, Data Page V1/V2, page offset bounds, Hadoop-framed LZ4 limits, and Hive/Iceberg partition writers. - 9 FE tests passed for defaults, variable forwarding, Thrift serialization, valid bounds, and invalid input. - The new tests failed for the missing settings before implementation. - FE Checkstyle, clang-format 16, build hygiene, and clang-tidy passed. - No cluster SQL run or performance benchmark was performed locally; CI validation is pending. ### Check List (For Author) - Test: - [x] Unit Test - Behavior changed: - [x] Yes. Explicit controls are available; current defaults are preserved. - Does this need documentation? - [x] Yes. English and Chinese documentation: https://github.com/apache/doris-website/pull/4581 ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # fe-common This module is used to store some common classes of other modules. # spark-dpp This module is Spark DPP program, used for Spark Load function. Depends: fe-common # fe-core This module is the main process module of FE. Depends: fe-common, spark-dpp