daidai
61906ce9a1
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
ORC `Int128` statistics already contain two's-complement bits, but the
Iceberg ORC writer negates negative values again before encoding them. A
file containing `[-100, -1]` consequently reports bounds `[100, 1]`, so
a negative predicate can prune a file that contains matching rows.
ORC footer statistics also trim trailing fractional zeros. For example,
`-1.00` is returned as `value=-1, scale=0`; a `DECIMAL(...,2)` manifest
needs the unscaled value `-100`. Preserving the sign alone therefore
leaves incorrect bounds.
Restore each bound to the declared column scale with the existing ORC
scaling utility, then serialize its original signed bits using minimal
big-endian encoding. Preserve necessary sign prefixes such as `128 -> 00
80` and `-129 -> FF 7F`.
The fix applies to newly collected bounds. Existing incorrect metrics
require recomputation or data-file rewriting.
### Release note
Correct decimal bounds emitted by Iceberg ORC writes to prevent
incorrect predicate pruning for negative values and values with trailing
fractional zeros.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Added BE tests cover 29 encoding cases, real ORC statistics at
precisions 1/9/18/38, scale restoration through scale 38, nullable
columns, and all-null columns. The external regression checks manifest
pruning for negative and mixed decimal data across decimal32/64/128.
Local validation:
- An isolated codec harness built from the production function passed
all 29 encoding cases.
- Five real ORC files passed independent Iceberg 1.12.0 manifest
write/read and predicate-pruning probes.
- clang-format 16, build hygiene, changed-line clang-tidy, and Groovy
4.0.19 syntax checks passed. Static analysis used real ASAN UT
compilation commands with PCH expanded as headers.
- The full local BE-UT build was stopped during the monolithic
dependency build before tests executed.
- Regenerated `test_iceberg_write_stats2.out` with the standard
regression runner after restoring decimal scales. Only `sql_4/5/7/9`
changed, and the decimal bounds match both stored rows and the Parquet
metrics.
- Re-ran `test_iceberg_write_stats2` and
`test_iceberg_orc_decimal_bounds` locally using this PR's CI FE/BE
artifacts: 2 suites passed, 0 failed, 0 skipped.
- Behavior changed:
- [ ] No.
- [x] Yes. Newly written ORC decimal bounds preserve the sign and the
column scale.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt