mirror of
https://github.com/apache/doris.git
synced 2026-11-04 15:39:47 +00:00
Gabriel
a01cc89378
### What problem does this PR solve? Related PR: #68532. Supersedes #68603. Preserve external binary, UUID, and timestamp semantics across connector schemas, JDBC transport, file readers, and external writes. The tables below list the type mappings changed by this PR, including mappings that previously depended on an opt-in property. ### Behavior changes - The binary mappings listed below now expose `VARBINARY` and preserve raw bytes, including invalid UTF-8, embedded NULs, and trailing zeros. File TVFs also apply this rule to nested binary fields. `DESCRIBE` and query results therefore expose binary types and hexadecimal values instead of decoded text. - External logical UUID types map to native Doris `UUID`, including nested fields, independently of binary mapping options. Native UUID inputs and canonical/compact UUID text can be written to Iceberg UUID columns; invalid text fails explicitly. UUID file reads preserve network byte order and can load directly into native Doris UUID columns. - Timestamps representing an instant map to `TIMESTAMPTZ`; timestamps representing local wall-clock fields map to `DATETIMEV2`. Instant semantics also apply to MySQL `TIMESTAMP` and ClickHouse `DateTime`/`DateTime64` without an explicit timezone in the type name. `TIMESTAMPTZ` preserves the instant, not the source timezone identifier or original offset spelling. - Catalog properties `enable.mapping.varbinary` / `enable.mapping.timestamp_tz` and file TVF properties `enable_mapping_varbinary` / `enable_mapping_timestamp_tz` remain accepted but no longer select the mappings below. Absent, `false`, and `true` settings produce the same types. Recursive mappings apply to supported ARRAY/MAP/STRUCT fields. - JDBC reads and writes preserve UTC instants and supported fractional precision independently of JVM/session timezone. PostgreSQL infinite/out-of-range `timestamptz` values become SQL NULL, with nullable metadata propagated even for source NOT NULL columns. MySQL retains its configured zero-date handling. Predicates mixing instant and wall-clock timestamps stay local when remote evaluation would change their meaning. - Binary partition values retain byte identity and distinguish SQL NULL from literal text. Iceberg UUID writes preserve UUID semantics through ARRAY/MAP/STRUCT leaves, including map keys and values; fixed-width binary writes validate the actual byte count. Paimon legacy ORC routing uses the split's owning branch and historical schema so instant columns use the appropriate reader. - MySQL query TVF wrapping distinguishes `--1` arithmetic from comments and honors the remote connection’s `NO_BACKSLASH_ESCAPES` mode. SQL Server UUID range predicates stay local because its GUID ordering differs from Doris. Iceberg UUID value predicates also stay local because file bounds can otherwise prune matching UUID rows. - Ordinary views retain binary types and declared VARBINARY bounds. Native-table materialization remains subject to Doris storage restrictions and needs explicit conversion where the target cannot store VARBINARY. Unsupported VARBINARY comparisons/functions remain unsupported; FE validation prevents implicit text coercion for the guarded operations. CDC JSON ingestion retains its existing transport conversions. ### Changed read mappings: external system → Doris “Before” describes the previous default (mapping options absent/false). Where an opt-in mapping already existed, this PR makes that mapping unconditional. `p` denotes the effective fractional precision supported by the mapper, capped at 6 (HMS uses the catalog timestamp precision); fixed precisions are shown explicitly. `n` denotes the supported declared byte bound; unqualified `VARBINARY` uses Doris's default maximum bound. Unchanged source-type mappings are omitted. | External system / access path | External type | Before | After | | --- | --- | --- | --- | | Hive / HMS | `BINARY` | `STRING` | `VARBINARY` | | Hive / HMS | `TIMESTAMP WITH LOCAL TIME ZONE` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | Iceberg | `binary` | `STRING` | `VARBINARY` | | Iceberg | `fixed(n)` | `CHAR(n)` | `VARBINARY(n)` | | Iceberg | `uuid` | `STRING` | `UUID` | | Iceberg | `timestamptz` (`shouldAdjustToUTC=true`) | `DATETIMEV2(6)` | `TIMESTAMPTZ(6)` | | Paimon | `BINARY(n)`, `VARBINARY(n)` / `BYTES` | `STRING` | `VARBINARY(n)` / `VARBINARY` | | Paimon | `TIMESTAMP WITH LOCAL TIME ZONE` / `TIMESTAMP_LTZ(p)` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | Fluss | `BINARY(n)`, `BYTES` | `STRING` | `VARBINARY(n)`, `VARBINARY` respectively | | Fluss | `TIMESTAMP_LTZ(p)` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | Hudi / Avro logical schema | Logical `uuid` on `string` | `STRING` | `UUID` | | Hudi / Avro logical schema | `timestamp-millis`, `timestamp-micros` | `DATETIMEV2(3)`, `DATETIMEV2(6)` | `TIMESTAMPTZ(3)`, `TIMESTAMPTZ(6)` respectively | | Hudi / Avro logical schema | `local-timestamp-millis`, `local-timestamp-micros` | `BIGINT` | `DATETIMEV2(3)`, `DATETIMEV2(6)` respectively | | MaxCompute | `TIMESTAMP` | `DATETIMEV2(6)` | `TIMESTAMPTZ(6)` | | Trino Connector | `UUID` | Unsupported by the type mapper | `UUID` | | Trino Connector | `VARBINARY` | `STRING` | `VARBINARY` | | Trino Connector | `TIMESTAMP(p) WITH TIME ZONE` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | MySQL / MariaDB JDBC | `BINARY`, `VARBINARY`, `TINYBLOB`, `BLOB`, `MEDIUMBLOB`, `LONGBLOB` | `STRING` | `VARBINARY` with the supported JDBC-reported byte bound | | MySQL / MariaDB JDBC | `TIMESTAMP(p)` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | PostgreSQL JDBC | `uuid`, including array elements | `STRING` | `UUID` | | PostgreSQL JDBC | `bytea` | `STRING` | `VARBINARY` with the supported JDBC-reported byte bound | | PostgreSQL JDBC | `timestamptz` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | Oracle JDBC | `BLOB` | `STRING` | `VARBINARY` with the supported JDBC-reported byte bound | | Oracle JDBC | `TIMESTAMP(p) WITH LOCAL TIME ZONE` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | Oracle JDBC | `TIMESTAMP(p) WITH TIME ZONE` | `STRING` in the connector; unsupported in the legacy client | `TIMESTAMPTZ(p)` | | SQL Server JDBC | `uniqueidentifier` | `STRING` | `UUID` | | SQL Server JDBC | `binary`, `varbinary`, `image` | `STRING` | `VARBINARY` with the supported JDBC-reported byte bound | | SQL Server JDBC | `datetimeoffset(p)` | `STRING` | `TIMESTAMPTZ(p)` | | DB2 JDBC | `BLOB`, `BINARY`, `VARBINARY` | `STRING` | `VARBINARY` with the supported JDBC-reported byte bound | | GBase JDBC | JDBC `BINARY`, `VARBINARY`, `LONGVARBINARY`, `BLOB` | `STRING` | `VARBINARY` with the supported JDBC-reported byte bound | | ClickHouse JDBC | `DateTime`, `DateTime('zone')` | `DATETIMEV2(0)` | `TIMESTAMPTZ(0)` | | ClickHouse JDBC | `DateTime64(p[, 'zone'])` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | Trino / Presto JDBC | `uuid`, including array elements | Unsupported by the type mapper | `UUID` | | Trino / Presto JDBC | `TIMESTAMP(p) WITH TIME ZONE`, `TIMESTAMP WITH TIME ZONE` | `DATETIMEV2` / `STRING`, or a metadata precision-parsing failure depending on spelling/path | `TIMESTAMPTZ(p)`; precision defaults to 6 when absent | | Trino / Presto JDBC connector | Unparameterized `TIMESTAMP` metadata | `STRING` | `DATETIMEV2(6)` | | Parquet file TVFs | Raw `BYTE_ARRAY`, including binary fallback for legacy ENUM/BSON annotations | `STRING` | `VARBINARY` | | Parquet file TVFs | UUID logical annotation on `FIXED_LEN_BYTE_ARRAY(16)` | `STRING` during legacy schema inference; scanner V2 already supported native UUID; opt-in binary mapping exposed `VARBINARY(16)` | `UUID` consistently | | Parquet file TVFs | Logical `TIMESTAMP` with `isAdjustedToUTC=true` | `DATETIMEV2(3/6)` | `TIMESTAMPTZ(3/6)` for millis / micros or nanos | | ORC file TVFs | `BINARY` | `STRING` | `VARBINARY` | | ORC file TVFs | `BINARY` with `iceberg.binary-type=UUID` | `STRING` | `UUID` | | ORC file TVFs | `TIMESTAMP_INSTANT` | `DATETIMEV2(6)` | `TIMESTAMPTZ(6)` | OceanBase JDBC delegates to the MySQL or Oracle mapper according to its database mode and inherits the corresponding changed mappings above. The file TVF rows apply to schema inference through supported file sources such as S3, HDFS, LOCAL, and HTTP. Existing unzoned timestamp mappings (for example MySQL `DATETIME`, PostgreSQL `timestamp`, Iceberg `timestamp`, and Paimon/Fluss `TIMESTAMP`) remain `DATETIMEV2` and are omitted from the table. Precision beyond microseconds is truncated. Physical binary storage with a recognized STRING/UTF8/DECIMAL annotation retains that logical type. ### Changed external schema mappings: Doris → external system | External target | Doris type | Before | After | | --- | --- | --- | --- | | Iceberg | `UUID`, including nested fields | Unsupported by the reverse schema mapper | Iceberg `uuid` | | Hive / HMS | `VARBINARY` | Unsupported by the reverse schema mapper | Hive `binary`, including nested fields | | Paimon | `DATETIMEV2(p)` | Paimon `TIMESTAMP(6)`; source precision discarded | Paimon `TIMESTAMP(p)` | | Paimon | `TIMESTAMPTZ(p)` | Unsupported by the reverse schema mapper | Paimon `TIMESTAMP_LTZ(p)` | | MaxCompute | `TIMESTAMPTZ` | Unsupported by the reverse schema mapper | MaxCompute `TIMESTAMP`, with UTC microsecond write transport | ### Release note The external binary, UUID, and timestamp mappings listed above are determined by source semantics. The former mapping options no longer opt out of VARBINARY or TIMESTAMPTZ. Applications reading these columns may observe changed schemas, binary result formatting, and timestamp semantics. External UUIDs retain native UUID semantics rather than mapping to string or binary. ### Validation - FE and Java plugin package build passed with 339 targeted unit tests, plus 59 JDBC planning/binding tests. FE Checkstyle passed. - 202 BE unit tests passed, covering UUID byte order and nullability, Parquet/ORC data and equality-delete combinations, missing-column defaults, ORC writes, and UUID partitions. Changed C++ lines passed clang-tidy; clang-format 16 and build hygiene checks passed. - 13 local regression suites passed: seven UUID file suites, PostgreSQL native UUID and all-types suites, MySQL query TVFs under both SQL modes, Iceberg nested UUID writes, cross-format reads/deletes, and initial defaults. Result files are generated by actual execution and compared in a separate run. - Iceberg write coverage includes canonical/compact text, native UUID INSERT SELECT, identity partitions, ARRAY/MAP/STRUCT nesting, NULL and empty containers, invalid inputs, and volatile map expressions whose key/value pairing must be retained. A non-pushable reference verifies UUID equality predicates do not lose rows to file pruning. - SQL Server, Trino, and Hudi UUID paths have unit coverage; their external integration suites were not run in this update. - Previous revision validation also covered the affected HDFS, Paimon, and HTTP TVF suites. ### Check List (For Author) - [x] Unit and regression coverage included. - [x] User-visible behavior and changed type mappings described. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note. - [ ] Confirm test cases. - [ ] Confirm document. - [ ] Add branch pick label if needed.