Mingyu Chen (Rayner)
e4c2dc261a
### What problem does this PR solve?
Issue Number: None
Related PR: #69581 #68516
Problem Summary:
**In short.** An Arrow timestamp whose civil year is outside 0000..9999
is read into DATETIMEV2 or TIMESTAMPTZ as a different, valid-looking
value: timestamp(us) 2005980768000000000, which is 65537-01-01, reads as
a non-null 0001-01-01. This PR checks the year before the value is
narrowed into Doris' 16-bit year field, as the ORC path already does,
and fails the read otherwise. The review of #69581 found it on the Lance
path; the code is shared by every Arrow reader.
**Background.** `DataTypeDateTimeV2SerDe::read_column_from_arrow` and
`DataTypeTimeStampTzSerDe::read_column_from_arrow` decode Arrow
timestamps for ADBC, Arrow stream load, paimon-rust, remote Doris and,
once #69581 lands, Lance, nested leaves included. Both convert epoch
seconds with `DateV2Value::from_unixtime()`, which passes cctz's 64-bit
year to `unchecked_set_time(uint16_t year, ...)` ("there's no overflow
check since it's hot path"). #68516 added a check before this narrowing
to the ORC path (`orc_timestamp_to_datetime()`).
**The problem, and what it cost.**
| Arrow timestamp(us) | DATETIMEV2 before | TIMESTAMPTZ before |
|---|---|---|
| 253402300800000000 (10000-01-01) | stored as year 10000, outside the
type's range | error |
| 2005980768000000000 (65537-01-01) | 0001-01-01 00:00:00 | 0001-01-01
00:00:00+00:00 |
| a year before 0000 | wraps to 655xx or back into 0000..9999 | error,
or a wrapped valid year |
In the wrapped cases the read succeeds with a wrong value, and nothing
in the result tells the user.
**How this PR fixes it.**
- DATETIMEV2: convert to civil time in the value's time zone (UTC for a
timezone-naive Arrow timestamp, the session time zone for a zoned one)
and reject a year outside 0000..9999 before packing, like
`orc_timestamp_to_datetime()`. After packing, `is_valid_date()` is
checked too: cctz counts year 0 as a leap year and Doris does not
(`is_leap(0)` is false), so the civil date 0000-02-29 passes the year
check, and before this PR it was stored as an invalid date.
- TIMESTAMPTZ is stored in UTC, so
`append_timestamptz_from_utc_epoch_micros` checks the instant against
0000-01-01 00:00:00 UTC .. 9999-12-31 23:59:59.999999 UTC before
converting. Its runtime `is_valid_date()` check after packing stays, for
the same leap day: timestamp(us) -62162121600000000 (0000-02-29 UTC) is
inside the range but packs into an invalid value. The range message now
says 0000-9999, the range `is_valid_date()` accepts.
- Both fail the read with DataQualityError, as the other range errors of
these readers do. NULL slots are still skipped before conversion.
- Not changed: the Parquet INT64 / INT96 timestamp paths validate the
UTC range before packing (`parquet_timestamp_micros()`,
`parquet_int96_timestamp_micros()`), so they could not wrap. Arrow reads
into the deprecated DATE / DATETIME (v1) types go through
`VecDateTimeValue::from_unixtime()` with the same unchecked narrowing;
they are left as they are. `orc_timestamp_to_datetime()` checks only the
year, so an ORC timestamp on 0000-02-29 is still stored as an invalid
date; it is not changed here.
**Merge order: after #69581.** A NULL STRUCT, LIST or MAP row may hide
any child value, an out-of-range timestamp included, because Arrow
leaves the payload under a NULL parent unspecified. #69581 masks those
child slots in the shared nested SerDes
(`mask_arrow_children_hidden_by_null_parents`, df95b41a33) before any
leaf SerDe reads them. Without it, the checks here fail a batch whose
visible values are valid: a STRUCT with rows `[NULL, {ts: 0}]` whose
NULL row carries 2005980768000000000 reads with DataQualityError on this
PR alone, and reads both rows on a tree with both PRs.
**Results.**
| Arrow timestamp(us) | DATETIMEV2 after | TIMESTAMPTZ after |
|---|---|---|
| 253402300799999999 (9999-12-31 23:59:59.999999) | read | read |
| -62167219200000000 (0000-01-01) | read | read |
| -62162121600000000 (0000-02-29, not a Doris date) | DataQualityError |
DataQualityError |
| 10000-01-01, 65537-01-01, -62167219200000001 (year -1) |
DataQualityError | DataQualityError |
| zoned, at +08:00: 253402271999999999 / 253402272000000000 | 9999-12-31
23:59:59.999999 / DataQualityError | |
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt