zhaorongshengandzhaorongsheng
bdb90856e6
## What problem does this PR solve? Push eligible Hive partition equality/IN predicates into bounded HMS filtering without eagerly materializing every partition. Preserve conservative fallback for unsupported predicates, types, identifiers and unavailable views. This update addresses the current review findings: - Reuse filtered handles without losing stronger backend residual predicates. - Prepare complete partition views for MV creation and eligible refresh/rewrite consumers. - Distinguish unavailable coverage from native/all coverage during MV compensation. - Fence ROOT/ADMIN selective subsets against schema and connector changes. - Restrict filtered, unfiltered and empty refresh reads to their materialized pin. - Preserve connector-retained schema through deferred pin hydration. - Keep INSERT preloading outside the outer target lock; retry schema or target replacement before binding with refreshed table caches. - Avoid pre-binding constant folding of unbound CAST predicates. - Handle reserved Hive partition keys `in` and `between` through local fallback. - Display unknown full partition totals as `?` without enumerating solely for EXPLAIN. - Resolve the latest target integration conflict by preserving both partition-pruning and write-only-default capabilities and advancing the connector API to 28.0. ## Release note Improve selective Hive partition planning while preserving schema, snapshot and materialized-view consistency. Selective EXPLAIN output may show an unknown full partition total as `?`. ## Validation - Published head: `ae2f21cbb30469408970cfb81709d8bbe27a8998`. - Target snapshot integrated without rewriting published history: `0976b2cd5020607205969bf7986dcb551b6a7325`. - Before the final target integration, canonical `./run-fe-ut.sh --coverage --run ...` runs verify 39 selected classes: 945 tests, zero failures/errors, one existing disabled test. - After the final target-replacement correction, a fresh targeted canonical run passes 33 tests, zero failures/errors/skips, and 89 actual zero-violation Maven Checkstyle executions. - After the final target integration, fresh canonical reports verify 36 classes and 642 tests, zero failures/errors/skips. This includes SPI's own full suite, Hive/Iceberg, view/MVCC, planning and partition compensation, INSERT/defaults, MTMV task fixtures and SQL-cache version tests. - The first integration run passes 641 tests but exits 1 when the final mock-FE class hits an occupied ephemeral edit-log port before executing its test. The isolated single-method rerun passes the remaining test, exits 0 with `BUILD SUCCESS`, and performs 89 actual zero-violation Checkstyle executions. These results are aggregated from fresh reports, not a claim that the all-class reactor succeeded in one attempt. - Coverage includes connector/HMS/SPI, MVCC schema and partition views, sync/stream/batch handoffs, MV compensation, policy/replan boundaries, INSERT lock windows and real target replacement in both execution and planning-only modes. Previously failing CAST/shape classes are included. - Connector-import, metadata-funnel, build-hygiene and authored whitespace checks pass. - Local mock-FE port conflicts required targeted reruns. A temporary test-only non-ephemeral port selection was used and restored before committing. - Standard local `./build.sh --fe` packaging remains blocked before Maven by missing genuine native thirdparty prerequisites. No successful full packaging, live HMS/BE regression or complete refresh-scheduler run is claimed. - Extended review rounds were stopped at the author's request. All independently confirmed remaining findings are repaired and tested; no pipeline PASS receipt or new remote `/review` request is issued by this update. --------- Co-authored-by: zhaorongsheng <zhaorongsheng@corp.netease.com>
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt