mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Mingyu Chen (Rayner)andGabriel
1adeb3a680
### What problem does this PR solve? Issue Number: None Related PR: #68631, #68768 Problem Summary: **In short.** This ports branch-4.1 #68631 (also on branch-4.2) to master. A Flight SQL `CreatePreparedStatement` now returns the analyzed result schema of the statement, `GetSchema` is implemented for both statements and prepared statements, and a prepared execution is checked against the schema that Prepare advertised. Before this PR, Prepare always returned a fixed one-column placeholder, and `GetSchema` answered UNIMPLEMENTED. The analysis runs in the FE only and does not plan or execute the query. Original author: @Gabriel39 **Background.** Flight SQL clients learn the shape of a result before they fetch it. The Arrow Flight SQL JDBC driver prepares every statement and builds `ResultSetMetaData` from the dataset schema that `CreatePreparedStatement` returns. ADBC exposes the same schema as `execute_schema`, and `GetSchema` is the RPC that asks for it without running anything. **The problem, and what it cost.** On master: | | master today | with this PR | |---|---|---| | `CreatePreparedStatement` dataset schema | always `ResultMeta: Utf8`, whatever the statement | the statement's columns, names and Arrow types as BE / FE will send them; statements whose schema cannot be told before they run, and results with an untyped NULL column, keep the placeholder (see below) | | `GetSchema` for a statement or a prepared statement | UNIMPLEMENTED | the analyzed schema, or UNIMPLEMENTED for a statement that keeps the placeholder | | Invalid SQL | fails only at execution | fails at Prepare with INVALID_ARGUMENT | | A query whose privilege check fails | fails only at execution | still fails only at execution: Prepare keeps the placeholder, because only the check on the executed plan decides (see below), and GetSchema answers UNIMPLEMENTED without a schema | | Prepared handle used after the session's catalog / database changed, or after the result schema changed (e.g. another session's DDL) | runs the unqualified SQL in whatever namespace the session has now, with whatever columns come back | refused with NOT_FOUND "prepare again", and the result left by that execution is dropped | | Prepared handle presented by another session | resolved by the identity written in the handle | must start with the caller's own identity (from #68768) | | TIMEV2 / VARBINARY columns in GetTables and Prepare schemas | `Null`, while BE sends `float64` / `binary` | `Float64` / `Binary`, as BE sends them | A client that trusts the advertised schema saw one string column for every query, and could not ask for a schema at all without executing. **How this PR fixes it.** - `arrowflight/FlightSqlQuerySchema` analyzes the statement on a clone of the session variables: a USE / SWITCH prefix is resolved in a temporary namespace, the analysis `StatementContext` is closed on the same thread, and the connector statement scope it may have opened is closed explicitly. Each field gets the same `doris_type` markers and native Variant layout the execution side uses (`FlightSqlSchemaHelper.withDorisTypeMetadata` / `nativeVariantField`), and TIMESTAMPTZ carries the canonical session time zone that BE stamps on its batches. - Privileges are checked on the analyzed plan after the CTEs that execution inlines have been inlined (`PullUpCteAnchor`, `CTEInline`, as the rewriter does before its own check). A CTE used once is therefore checked only on the columns the query reads, and a CTE used more than once on every column it selects, which is also how execution checks them. The rewriter normalizes the plan further before it checks privileges. For example, it drops an ORDER BY under a derived table, so the analyzed plan can read a column that the executed plan does not. When the analyzed plan is denied, the statement keeps the placeholder and execution decides. No schema is disclosed in either case. - FE-side commands whose result header does not depend on running them now report that header without running: HELP, SHOW SNAPSHOT, SHOW CREATE TABLE (one header selector shared with `doRun`, including master's stream, plugin-view and system-table cases), SHOW PROC `/` and `/current_queries`, SHOW CONFIG, and EXPLAIN PLAN PROCESS (Rule / Before / After). - `ShowStoragePolicyCommand.getMetaData()` returned the four-column USING header for every form, while plain SHOW STORAGE POLICY answers the seven columns of `StoragePolicy.STORAGE_META_DATA`. It now returns the header of the form that runs. This also corrects the column definitions that MySQL `COM_STMT_PREPARE` reports for the statement. No other SHOW command declares a header that differs from the one it returns. - WARM UP SELECT keeps the placeholder. It answers OK instead of its statistics whenever planning proves its source empty: a predicate that folds to false, no partition left after pruning, or partitions without data. - A result with an untyped NULL column keeps the placeholder too, at any depth (`SELECT NULL`, `SELECT id, NULL FROM t`, `ARRAY(NULL)`). `DorisArrowTypeMapping` gives NULL_TYPE Arrow `Null`, but BE builds such a column as a `UInt8` flagged as a NULL literal and sends it as nullable `Bool`, so an advertised `Null` would expire the handle at every execution. A typed NULL (`CAST(NULL AS VARCHAR(20))`) is described as its type. - The prepared query kept in `FlightProtocolAdapter` records the SQL, the namespace and the advertised schema. A prepared execution re-analyzes the statement and compares the result with the advertised schema before it runs, and compares the schema it actually produced before it publishes a DoGet ticket. It is the session's next request whether or not these checks pass: before them, it drops what the previous request left, as a statement's execution does, while Prepare and GetSchema are metadata requests and leave it. On a mismatch, the handle is removed, the execution is cancelled, and the deferred coordinator, the channel result and the endpoints are dropped. A handle whose statement can no longer be described when it is re-analyzed, for example after a column grant was revoked, expires the same way, and the statement prepared again keeps the placeholder. A USE / SWITCH inside the statement rebinds only that handle. - `DorisArrowTypeMapping` maps TIMEV2 to `Float64` and VARBINARY to `Binary`, the types BE sends. TIMESTAMPTZ keeps the fixed `"UTC"` in GetTables, and AGG_STATE keeps `Null` there. - `FeServiceClient` rejects a remote Doris metadata timeout whose milliseconds do not fit in an int, instead of letting it wrap around. Where master differs from branch-4.x: - branch-4.x answers UNIMPLEMENTED at Prepare for statements whose schema it cannot tell before they run: DESC, SHOW DATA / PARTITIONS / QUERY STATS / PYTHON PACKAGES, other PROC nodes, COPY INTO, WARM UP CLUSTER, multi-statement requests that do not start with USE / SWITCH, `?` placeholders, and result types the FE cannot describe. Python ADBC falls back to executing without a prepared statement, but the Arrow Flight SQL JDBC driver prepares every statement and has no fallback, so on master these statements would fail over JDBC. `test_arrow_flight_session_lifecycle` runs DESC and `SET @multi = 2; SELECT ...` through JDBC. Here, these statements keep today's placeholder schema, and their handles are not checked at execution. - branch-4.x checks privileges on the analyzed plan as it is, so a CTE used once is checked on every column it selects, and any denial fails Prepare. Here, CTEs are inlined first, and a remaining denial keeps the placeholder. - branch-4.x advertises the statistics header for WARM UP SELECT and then rejects, as a schema change, an execution whose source turned out empty. Here, WARM UP SELECT keeps the placeholder. - branch-4.x advertises Arrow `Null` for an untyped NULL column, which BE sends as `Bool`, so every prepared execution of such a query is rejected as a schema change. Here, such a result keeps the placeholder. - The analysis and the checks run under the session's command lock (`FlightProtocolAdapter.runCommand` / `callCommand`) instead of `synchronized (connectContext)`, and the namespace is saved and restored with `getNameSpaceContext()` / `restoreNameSpaceContext()`. - EXPLAIN PLAN PROCESS has a result over Flight on master, so it gets a static header instead of UNIMPLEMENTED. PLAN REPLAYER and SHOW CREATE PROCEDURE do not exist on master and are not carried over. - SWITCH resolves the default database through the connector provider (`Env.defaultDatabaseOnUse`, now shared with `Env.changeCatalog`) instead of `EsExternalCatalog.DEFAULT_DB`. - TIMESTAMPTZ fields carry the canonical session time zone that BE stamps, where branch-4.x uses the raw session value. - Only the two handle checks from #68768 that do not depend on parameter binding are taken: the caller-prefix check and rebinding only the handle that ran the USE / SWITCH. Parameter binding itself is not part of this PR. - `GetTableTypes`, the other half of #68631, is implemented by #67899. Cost: every Prepare, every prepared execution and every `GetSchema` runs one more FE analysis of the statement. **Results.** On a local cluster built from this branch, every suite under `arrow_flight_sql_p0` passes: the new `test_schema_only` and `test_get_tables_schema`, plus the seven suites that run through Flight JDBC, `test_arrow_flight_session_lifecycle` among them. `datatype_p0/uuid/test_uuid_arrow_flight` passes too. `test_schema_only` checks the advertised schema of each kind of statement against what its execution actually returns. That covers queries, the side-effect-free frontend-side commands executed as prepared statements (SHOW STORAGE POLICY among them), a WARM UP SELECT on an empty table (executed in cloud mode, the only mode that warms up internal tables), and a `group_concat_state` AGG_STATE, which BE sends as Utf8. Co-authored-by: Gabriel <liwenqiang@selectdb.com>
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt