Mingyu Chen (Rayner)
1f71a66565
### What problem does this PR solve? Issue Number: None Related PR: #69299, #69488 Problem Summary: #### In short An Iceberg table stored in HMS and read through an HMS catalog (`'type'='hms'`) loses every capability it inherits from the embedded Iceberg connector in any statement that pins the table at a snapshot. Since #69299 that is every statement that reads it: it gets no Top-N lazy materialization and no nested column pruning, and `DEFAULT(column)` in the conditions of `UPDATE` and `MERGE` silently matches nothing. This PR makes the snapshot overload of `HiveConnectorMetadata#getTableSchema` pass the inherited capabilities on, as the latest-schema overload already does. It is a master-only fix (branch-4.x has no connector SPI), left over from #69488 (review comment 4355508100). #### Background fe-core resolves a table-scoped capability (`PluginDrivenExternalTable.hasCapability`) from the catalog's connector or from the per-table set in the table's schema. It never asks the embedded connector that built the schema of an iceberg-on-HMS table. So `HiveConnectorMetadata` copies the capabilities that fe-core resolves per table (`SIBLING_INHERITABLE_CAPABILITIES`: auto and sampled analyze, Top-N lazy materialization, nested column pruning, field-id access paths, storage predicate pruning, nested column DDL and write-only defaults) from the owning connector into the per-table set of each schema it delegates. Iceberg declares seven of the eight; it does not declare sampled analyze. It did this only in `getTableSchema(session, handle)`. `getTableSchema(session, handle, snapshot)` returned the sibling's schema unchanged. fe-core uses that overload for explicit time travel. Since #69299 it also uses it for the latest-snapshot pin of every statement that reads an Iceberg table: the pin retains its schema (`retainSchema(true)`), `PluginDrivenMvccExternalTable` builds the schema at that pin, and `getSchemaCacheValue()`, where `hasCapability` reads the per-table set, returns it for the rest of the statement. #### The problem, and what it cost Reproduced on a local cluster with a standalone Hive 3.1.2 metastore. Two identical format-version 3 tables `(id INT, c INT, s STRUCT<a:INT, b:STRING>)` get two rows, then `ADD COLUMN d INT DEFAULT 42`, `RENAME COLUMN s.a TO a2` and a third row with `d = 7`. The same statements then run against one table through the Iceberg catalog and against the other through the HMS catalog. | Statement | Iceberg catalog | HMS catalog, master | HMS catalog, this PR | |---|---|---|---| | `EXPLAIN SELECT * FROM t ORDER BY id LIMIT 1`, also with `FOR VERSION AS OF` | `VMaterializeNode` | no lazy materialization | `VMaterializeNode` | | `EXPLAIN SELECT struct_element(s, 'a2') FROM t` | `all access paths: [s.a2]` | no pruning | `all access paths: [s.a2]` | | `UPDATE t SET c = 100 WHERE d = DEFAULT(d)` | updates rows 0 and 1 | updates nothing | updates rows 0 and 1 | | `MERGE INTO t USING (SELECT 0 AS id) s ON t.id = s.id AND t.d = DEFAULT(t.d) WHEN MATCHED THEN UPDATE SET c = 300` | updates row 0 | updates nothing | updates row 0 | | `DELETE FROM t WHERE d = DEFAULT(d)` | fails: `cannot be used in a DELETE` | deletes nothing | fails the same way | | `SELECT DEFAULT(d) FROM t` | fails: `must reference a column of the table that this statement writes` | returns NULL | fails the same way | The `UPDATE`, `MERGE` and `DELETE` rows fail silently: the statement succeeds and changes nothing. Reads of the renamed field return the right values in all three columns: on master the HMS catalog does not prune the struct at all, and with this PR it prunes it with field-id access paths, as the Iceberg catalog does. `INSERT ... VALUES (DEFAULT(d))` and `INSERT ... SELECT DEFAULT(d)` already wrote 42 through both catalogs. #### How this PR fixes it Both overloads now go through one private helper, `delegatedTableSchema`. It resolves the owning sibling once, asks it for the schema (at the pin, for the snapshot overload), and passes the result through the existing `reflectSiblingCapabilities`. The snapshot reaches the sibling unchanged. A hudi-on-HMS table is unaffected: the Hudi connector declares no capabilities, so its schema comes back as it was. A plain Hive table never reaches this branch. #### Results Through the HMS catalog, every statement in the table above now plans and answers as it does through the Iceberg catalog.
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt