mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Gabriel
91ace4fb7d
## Summary - Forward-port the Parquet nullable selection optimization from branch-4.1 to the latest master. - Forward-port Iceberg Variant reads to master's plugin-driven connector implementation. - Preserve File Scanner V2 safety checks, mixed-version backend gating, typed-leaf projection, and delete-only MERGE behavior. - Resolve the latest-master benchmark and deferred-column layout conflicts while retaining coverage from both sides. - Adapt the catalog Hadoop-property concurrency fix to master's storage adapter architecture by atomically publishing an immutable shared snapshot. - Expand Iceberg Variant reader coverage for primitive types, nested containers, multi-file scans, row groups, delete vectors, equality/position deletes, and lazy materialization. - Preserve append atomicity for nullable, STRUCT, ARRAY, and MAP destinations when lazy Variant fallback discovers corrupt input. - Cache Variant schema presence so ordinary Parquet scans avoid Variant-specific planning and statistics work. - Forward-port compound Parquet Page Index pruning, combining AND ranges by intersection and OR ranges by union while preserving residual evaluation. - Keep compound predicate pruning behind master's metadata-pruning safety fence. - Forward-port the hardened projected shredded Variant lifecycle fixes, preserving projected state across exchange, TopN, truncation, mixed file layouts, and scanner-profile destruction. - Add native Paimon reads for schema-matched unannotated Variant carriers without misclassifying ordinary structs. - Apply the external-table guardrail stabilization by removing nondeterministic scanner-distribution assertions while retaining deterministic parallel correctness coverage. - Fix late runtime-filter refreshes so predicate/non-predicate reclassification is accepted only when physical slots and projections remain unchanged. - Preserve Paimon nested timestamp semantics with Parquet's independent predicate and deferred-output projections after rebasing. - Map Paimon Variant schemas to the execution-only VariantV2 carrier so both JNI and native scan plans remain queryable in Nereids. - Align the Iceberg nested-path EXPLAIN assertion and Paimon six-column DESC golden with master's plugin-driven metadata format. - Forward-port Iceberg schema-evolution and nested partition-write hardening, including nested defaults and rollback-safe append behavior. - Preserve Doris floating-point equality semantics for Parquet pruning across NaN values and signed zero. - Safely localize nested Parquet Bloom probes while preserving predicate order, schema-validation barriers, and conservative fallback. ## Original pull requests - https://github.com/apache/doris/pull/66397 - https://github.com/apache/doris/pull/66302 - https://github.com/apache/doris/pull/66392 - https://github.com/apache/doris/pull/66421 - https://github.com/apache/doris/pull/66441 - https://github.com/apache/doris/pull/66412 - https://github.com/apache/doris/pull/66503 - https://github.com/apache/doris/pull/66509 - https://github.com/apache/doris/pull/66529 - https://github.com/apache/doris/pull/66470 - https://github.com/apache/doris/pull/66471 ## Verification - FE reactor build, Checkstyle, and targeted tests: 277 tests passed for the Parquet and Iceberg forward ports. - `CatalogPropertyTest`: 2 tests passed for atomic publication and snapshot immutability. - Final FE Checkstyle passed for all 74 reactor modules after rebasing onto master at `a82564ced5`. - Final post-rebase FE reactor compilation and targeted tests passed: 104 tests across `ConnectorPluginSurfaceTest`, `IcebergWritePlanProviderTest`, `RequestPropertyDeriverTest`, and `PluginDrivenTableSinkTest`. - Repository clang-format 16 verification passed for all 85 affected C/C++ source and header files. - BE ASAN targeted tests: 408 tests from 29 suites passed for the earlier forward ports. - Focused BE ASAN compound Page Index tests: 2 tests from 2 suites passed. - Final focused BE ASAN verification after rebasing: 225 tests from 18 suites passed, including late runtime-filter refresh, nested timestamp semantics, Variant, Parquet, and Paimon coverage. - Clang-format 16 dry run passed for all 85 C/C++ source and header files changed by this PR. - `PaimonTypeMappingReadTest`: 3 tests passed with Maven build cache disabled. - `ConnectorColumnConverterTest#testComputeVariantCarrierConversion`: 1 test passed with Maven build cache disabled. - Full FE Checkstyle passed for all 74 reactor modules after the Paimon Variant mapping fix. - Regression framework build and tests passed: 3 tests, 0 failures; the Paimon DESC golden was also validated as six columns with an empty Extra field. - `git diff --check` passed for the Groovy source changes; the generated Paimon `.out` rows retain the required trailing delimiter for an empty Extra field. - Full FE Checkstyle passed for all 74 reactor modules after applying #66529. - Targeted FE Iceberg planner, metadata, manifest-cache, and statement-scope tests passed after applying #66529. - BE ASAN floating-point pruning verification passed: 58 tests from 6 suites after applying #66470. - Final BE ASAN nested-pruning verification passed: 356 tests from 6 suites after applying #66471 and the master API adaptations. - Clang-format 16 verification passed for all 47 C/C++ source and header files changed by #66529, #66470, and #66471.