Gabriel
4bde8d0fb2
### What problem does this PR solve? Issue Number: close #69926 Problem Summary: Hive JSON queries that only need partition columns can return zero rows even when the selected files contain records. For example, `SELECT * ... WHERE dt = 'p1'` returns data while `SELECT COUNT(*) ... WHERE dt = 'p1'` returns zero. The V2 JSON reader derived its row count from the output block, which has no columns after physical field pruning. Writer-backed reader tests reproduce the same row-count loss for empty WAL projections. Track accepted JSON records independently of materialized columns, including batch progress, array expansion, error rollback, and ignore-malformed NULL rows. Preserve JSON-path record selection for empty projections, and let the existing TableReader materialize partition values using the returned row count. For WAL, save the deserialized source row count before materialization transfers column ownership, then pass it through the existing filter pipeline. This preserves the row count when no file columns are requested. The empty-projection reproduction is at the reader interface; production replay reachability of an empty projection remains unverified. Add ten JSON BE tests covering empty file schemas and projections, arrays, JSON roots/paths, malformed input, batch boundaries, EOF, and actual TableReader partition materialization. Add eight writer-backed WAL cases covering empty, single-column, and reordered full projections across multiple blocks, with and without schema discovery, plus repeated EOF reads. Add a WAL SQL regression that forces two original async-load transactions to abort, then verifies row counts, NULLs, aggregates, filters, and actual values after replay. Add a Hive SQL regression using the existing partitioned JSON fixture for COUNT, partition projections, grouping, and file-column filters with external COUNT pushdown enabled and disabled. ### Release note Fix incorrect zero counts and missing rows when querying only partition columns of Hive JSON tables. Preserve logical row counts for empty WAL reader projections. ### Check List (For Author) - Test - [x] Unit Test: 166 ASAN tests passed, including JSON reader, TableReader, WAL V2 reader, WAL reader/writer, and CSV/Text empty-projection controls. Before the JSON fix, nine of ten new tests failed; before the WAL fix, both empty-projection cases failed while six projection controls passed. - [x] WAL SQL regression: `test_group_commit_wal_row_count` passed twice on isolated Doris 4.1.5 with scanner V2 enabled. Generated the `.out` file with `run-regression-test.sh`, then reran for result comparison. Execution on master is pending CI. - [x] Hive SQL regression: added `test_hive_json_partition_only`; Groovy compilation passed. Full execution against the external Hive fixture is pending CI. - Behavior changed: - [x] Yes. Empty JSON and WAL file projections preserve logical row counts. Hive JSON partition-only queries produce the correct rows and aggregates. - Does this need documentation? - [x] No.
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt