mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Gabriel
0e741f9162
### What problem does this PR solve? Related issue: #67366 Related PR: #67449 On branch-4.1, Arrow Flight SQL returns `0000-01-02` for a stored DATE of `0000-01-01`. Doris follows the MySQL calendar, which excludes year zero's leap day, while Arrow DATE uses the proleptic Gregorian calendar. Subtracting the Unix epoch day number directly shifts dates in `0000-01-01` through `0000-02-28` by one day, including elements of `ARRAY<DATE>`. Adapt the DATE portion of #67449 to branch-4.1, with credit to its author. Convert between the calendars at Arrow and ORC write boundaries and Arrow/default Parquet/ORC read boundaries. Use the same conversion for ORC statistics so pushed-down MIN/MAX agrees with row decoding. Reject unrepresentable external dates and ignore undefined payloads in null Arrow slots. The two DATE regression suites and their expected results are copied unchanged from #67449. Add its DATE unit coverage plus an explicit null-payload test and an exhaustive comparison against the civil calendar for all 3,652,424 representable dates. Also align Iceberg DATE day/bucket partition transforms with file ordinals. For ORC/Parquet, metadata is usable only when its raw DATE range proves that every ordinal is representable, including the interior of the range. Unsafe ranges fall back to decoding: strict reads retain their errors, non-strict Parquet reads retain conversion-generated NULLs, and COUNT uses materialized values when physical null counts are insufficient. ORC uses a metadata-only SARG to check projected DATE leaves per stripe: safe excluded stripes retain SARG pruning, while unsafe stripes remain readable with SDK row-group pruning disabled. ORC COUNT validates every DATE leaf of its argument in each selected stripe, and strict Parquet COUNT validates projected DATE descendants before reading only levels. Scope: DATE, including its Iceberg partition transforms. This does not include DATETIME changes or the independent legacy file scanner conversions used when `enable_file_scanner_v2=false`. V1 reader changes and mixed-version/rolling-upgrade compatibility gates are explicitly outside this PR's scope. ### Validation - Baseline: 9 of the original 10 DATE unit tests fail; the unchanged Flight regression reproduces the one-day shift. - Fixed: 250 unit tests pass under ASAN, including the complete ORC reader suite, DATE SerDe exhaustive calendar check, Iceberg partition transforms, Parquet statistics, DATE scans and COUNT tests. These compile the modified branch-4.1 sources against cached test dependencies. - The unchanged `test_date_year_zero` Flight regression passes locally, including nullable and array values. - `test_outfile_date_year_zero` passes for Parquet and ORC using the original rows and golden assertions, with S3 transport replaced by local outfile/TVF only for the local run. The committed suite retains S3 transport for CI. - Local integration runs use the same fix rebuilt into an existing ASAN/UBSAN BE build at `5bcb996a5ff`, with an existing FE. A full branch-4.1 build and the original S3 run remain for CI. - clang-format 16 passes for all affected C++ files. Source whitespace checks pass; generated golden files retain their conventional trailing blank line from #67449. - Review coverage: new tests reproduce partition mismatches, invalid-interior MIN/MAX bypass, ORC SARG/Parquet footer/page pruning inconsistencies, and Parquet IS NULL/COUNT discrepancies. All five reproducing tests fail before the review fixes. - Added a DATE-only Spark/Iceberg partitioned write/read regression, including selective day/bucket predicates and partition metadata comparison. Its Groovy compilation passes; execution requires the external CI environment. - Follow-up review: scalar/nested ORC COUNT and strict nested Parquet COUNT reproductions fail before the guards and pass afterward. ARRAY, STRUCT, MAP, all-null values, safe ranges, non-strict Parquet, and unprojected DATE siblings are covered. - External ORC regression: retain the original LZ4 file as a BE fixture and verify that ordinal `-719530` raises a data-quality error. Replace the obsolete `1900-01-01` golden result with an explicit error assertion, restoring the scanner setting afterward. The fixture test and Groovy compilation pass locally. The complete group6 suite is also validated locally with HDFS transport replaced by LOCAL TVF; the committed suite retains HDFS for CI. - Pruning review: both new tests fail before the fixes and pass afterward. Partial ORC STRUCT projection retains SARG when the invalid DATE sibling is not read; selecting that DATE still preserves its error. Multi-page Parquet tests retain the unsafe DATE page while pruning safe pages for IS NULL and range predicates, through both single-slot and multi-slot OR evaluators. - Mixed-stripe ORC coverage: the original file-wide guard prunes zero safe stripes; the fix prunes both safe excluded stripes and still reports the invalid DATE in the retained stripe. The active SDK options outlive their reader, and the cached SARG evaluator is discarded before opening an unsafe range. - External group6 coverage: the original `non_vec_orc_scanner.orc` fixture encodes `20191111` as an invalid epoch-day ordinal. Replace its obsolete `1900-01-01` golden result with an explicit data-quality error assertion and keep the original bytes as a small BE fixture; projecting other columns still succeeds. - Final local integration after the stripe fix: all three suites pass with no failures or skips (complete group6: 41 fixtures, 39 golden-result assertions and two expected errors; Flight year zero; Parquet/ORC outfile year zero). These use the incremental ASAN/UBSAN build and local transport substitutions described above. - Metadata safety review: the malformed pruned-stripe index and truncated nested DATE statistics reproduce SDK bounds assertions before the fix. Both regressions pass afterward; the DATE preflight now adds no file read calls or bytes compared with the same excluded stripe without DATE projection. - Historical-file migration: an explicit offline tool converts only confirmed legacy Doris DATE ordinals, including nested values. It publishes a separate output atomically and refuses overwrites. Tests cover year-zero boundaries, NULLs, hidden null-container payloads, nested ORC/Parquet round trips, schema metadata, timestamp precision, and failed-write cleanup. - Final follow-up validation: all 250 ASAN tests and 10 migration-tool tests pass. The final incremental production BE passes the three existing local suites plus six legacy migration/read/export/read checks, covering both ORC and Parquet. Original HDFS/S3 transport and the full branch build remain for CI as described above. ### Release note Fix year-zero DATE values being returned one day late through Arrow Flight SQL, align DATE encoding and decoding in the default Parquet/ORC paths, and keep Iceberg DATE partition values consistent with file contents. Existing files written with the former year-zero encoding cannot be distinguished automatically from correctly encoded files. Before switching their consumers to the corrected reader, use the explicit [legacy DATE migration tool and procedure](https://github.com/apache/doris/blob/082429b00f2392f8e8eabcee6e6a5a94a403e7aa/tools/legacy_date_migration/README.md) on files confirmed to use the old Doris encoding, or re-export from authoritative original data. The tool writes new ORC/Parquet files without modifying the source. Existing Iceberg files must be migrated through a new table or a supported table-format transaction that recomputes partition metadata; do not replace files in existing snapshots. The proleptic-only `0000-02-29` remains unrepresentable as a Doris DATE. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - Behavior changed: - [x] Yes, external DATE ordinals use the correct calendar; internal DATE arithmetic is unchanged. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label