mirror of
https://github.com/apache/doris.git
synced 2026-11-04 15:39:47 +00:00
Gabriel
84e5e6cd52
### What problem does this PR solve? Forward-port the requested changes whose missing behavior is still present on master: complete the missing Variant writer hooks from #68381, then retain the ports of #68614, #68667 and #68859 in branch-4.1 merge order. | Source PR | Evidence on master before this change | Action | | --- | --- | --- | | #68381 | Master already contains the main converter refactor through #68396, but lacks the Variant-specific Paimon/Iceberg SerDe overrides. Native Flight dispatch intercepts Paimon binary structs. | Complete the missing hooks and preserve Paimon `value,metadata` versus Flight `metadata,value` storage contracts, including nested columns. | | #68614 | Arrow ARRAY/MAP/STRUCT child fields lose Doris logical type markers; JSON/VARIANT markers are absent. | Preserve recursive logical metadata and use protocol-specific schema converters. | | #68667 | Flight maps Variant V2 to UTF8 instead of native binary values. | Return `arrow.parquet.variant` with `struct<metadata: binary not null, value: binary not null>` storage; preserve SQL NULL, physical scalar types and row-local dictionaries. | | #68852 | The obsolete external-table queries are already removed. All 276 active query tags match the 276 expected output blocks in execution order. | Excluded; no expected-output changes. | | #68856 | The preload module is already removed. The Hudi plugin resolves Parquet readers and format structures consistently to 1.18.0. | Excluded; no dependency changes. | | #68859 | Block metadata omits VARBINARY length, and deserialization constructs `DataTypeVarbinary()` with an unspecified length. | Preserve declared lengths, including nested and nullable columns, using optional protobuf field 13. | Adapt the ports to master's existing architecture: retain native UUID and nanosecond/timestamp mappings, reuse the shared FE Arrow mapper, and wrap extension storage correctly for nested Variant arrays. Master already uses Variant V2, so the removed legacy Variant switch and SerDe are not restored. This does not add the branch-4.1-only Iceberg table Variant feature or prepared-statement schema machinery. ADBC recognizes the native Variant extension, validates and decodes its bytes, and retains Variant sub-column access; the scanner permits that carrier only for ADBC among Arrow readers. The Flight integration tests and Python samples cover native Variant and recursive metadata. GetTables and ADBC expectations follow the native schema. The Iceberg static-null partition unit-test fixture uses master's existing Thrift field ID 19; production Thrift definitions are unchanged. The Flink regression explicitly casts Variant to STRING for a connector that declares that column as STRING. ### Release note Preserve nested logical type metadata in Arrow Flight SQL results, return Variant V2 as native Arrow Variant binary, preserve native Variant values and sub-column access through ADBC, and retain declared VARBINARY lengths across block serialization. Clients requiring Variant JSON text should explicitly cast it to STRING. ### Validation - FE targeted unit tests: 25 passed for the follow-up (ADBC mapping, native Variant and Flight schemas); the initial port also passed 64 targeted tests including UUID and recursive mappings. - FE Checkstyle: passed. - BE targeted ASAN unit tests: 408 passed, covering Arrow, Variant, ADBC, FileScannerV2, UUID, VARBINARY and the static-null Iceberg fixture. - Changed-line clang-tidy, clang-format 16 and build hygiene checks: passed. - Groovy regression scripts compiled; Python samples passed syntax checks. - Checked modified Thrift/protobuf field IDs against master and branch-3.1, branch-4.0, branch-4.1 and branch-4.2. - Confirmed the ES baseline query/output alignment and Hudi Parquet dependency tree before excluding those ports. - Built FE and ASAN BE with the repository build script. On a local cluster, `test_adbc_type_semantic`, `test_get_tables_schema` and `test_flight_native_variant` passed (3 suites, 0 failures, 0 skips). The ADBC `.out` was generated by the regression framework and then checked by a normal rerun. - Full Paimon and Flink integration suites require the external test environment and were not run locally. Paimon writer and nested-container paths passed BE tests. - Four muted P0 failures also recur on unrelated PR builds: two optimizer plan-shape mismatches and two object-store upload timeouts. They are not included as fixed by this port. ### Check List (For Author) - Test - [x] Unit tests - [x] Regression tests added/updated - Behavior changed: - [x] Yes, native Variant Flight output and preserved type metadata. - Does this need documentation? - [x] Yes, Python Flight sample documentation updated.