mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
lihangyu
3bdb43830f
Problem Summary: This PR adds an experimental, compute-only
`ColumnVariantV2` execution path. It uses Apache Parquet Variant
Encoding v1 for encoded values and canonicalizes logical values for
hashing and arena serialization. The path is selected by the
experimental FE session variable `enable_variant_v2`, which is `false`
by default.
When the session variable is enabled, FE assigns an execution-only
Variant V2 type marker to compute results produced by
`parse_to_variant`, `try_parse_to_variant `, and explicit `CAST(... AS
VARIANT)`. The marker is also applied recursively to Variant elements
inside `ARRAY`, `MAP`, and `STRUCT` cast targets. It is forwarded
through Thrift/Protobuf descriptors so BE creates `DataTypeVariantV2`
and the matching `ColumnVariantV2` physical column.
The marker is transient and is not persisted in table metadata. This PR
does not change Variant table creation, load, segment storage,
readers/writers, statistics, or compaction. Those paths continue to use
the legacy/native `ColumnVariant`. Focused compaction coverage verifies
that compute V2 sessions do not affect native Variant storage.
### ColumnVariantV2 behavior
#### Physical states
`ColumnVariantV2` has two whole-column physical states:
- **Encoded (E):** values use Parquet Variant Encoding v1.
- **Typed scalar (T):** the whole column uses one Doris scalar type plus
an internal Variant-null map.
A column does not mix E and T row by row. SQL `NULL` remains outside the
Variant value and is distinct from Variant/JSON `null`.
#### Enable ColumnVariantV2
Enable it for the current FE session:
```sql
SET enable_variant_v2 = true;
```
The default is `false`. There is no mutable BE config for this feature.
V2 selection is carried by the expression result type, so enabling it in
one session does not change the physical Variant type used by storage or
compaction work in other sessions.
#### Construct and access Variant values
Use `parse_to_variant()` to parse JSON text:
```sql
SELECT parse_to_variant('{"id": 1, "items": [true, null, "x"]}');
SELECT parse_to_variant('42');
SELECT parse_to_variant('"a JSON string"');
```
`try_parse_to_variant()` returns SQL `NULL` for Variant validation
failures. `CAST(string AS VARIANT)` creates a typed Variant string; it
does not parse JSON.
Object keys and zero-based array indexes can be accessed and then
explicitly cast:
```sql
SELECT CAST(parse_to_variant('{"object":{"k":1}}')['object']['k'] AS INT);
SELECT CAST(element_at(parse_to_variant('[10,20,null]'), 0) AS INT);
```
#### Canonical hash and arena serialization
Hashing and arena serialization use the same canonical logical-value
representation for encoded and typed columns. This makes semantically
equivalent values produce the same serialized hash key even when their
physical representations differ.
Canonical rules include:
- equivalent integral numeric representations normalize together;
- decimal trailing zeros normalize;
- `+0`, `-0`, and integral zero normalize together;
- object key order does not affect the canonical value, while array
element order does;
- Variant null is distinct from SQL `NULL`;
- invalid external encodings and broken internal invariants return
errors.
The canonical hash/serialization path supports:
- `GROUP BY`;
- `DISTINCT` and `COUNT(DISTINCT ...)`;
- `INTERSECT`, `EXCEPT`, and `UNION DISTINCT`;
- serialized hash keys used by aggregation and set operators.
#### Comparison predicates and join keys
Root Variant `ComparisonPredicate` expressions are not supported in this
phase, including `=`, `!=`, `<=>`, and ordering comparisons. Cast both
sides to a concrete scalar type first:
```sql
-- Supported
SELECT CAST(v1 AS STRING) = CAST(v2 AS STRING);
-- Rejected
SELECT v1 = v2;
```
Variant expressions are also not supported as join keys. This PR does
not add Variant Hash Join, Variant join runtime filters,
equality-specific column dispatch, or join-properties compatibility
rules.
#### Supported compute operators
| Area | Supported behavior |
| --- | --- |
| Parse and output | `parse_to_variant`, `try_parse_to_variant `,
Variant-to-string/JSON output |
| Cast | Scalar SQL types to Variant, nullable Variant-to-scalar
conversion, element-wise array conversion |
| Canonical hash | `GROUP BY`, `DISTINCT`, `COUNT(DISTINCT)`,
`INTERSECT`, `EXCEPT`, `UNION DISTINCT` |
| Aggregation | `COUNT(*)`, `COUNT(variant)`, grouping keys, string
coercion for `group_concat` |
| Conditional expressions | `IF`, `CASE`, `IFNULL`, `COALESCE`, eager
and short-circuit execution |
| Nested containers | Variant values inside `ARRAY`, `MAP`, and `STRUCT`
|
| Table functions | `explode_variant_array`, multi-input explode,
`explode`/`explode_outer` for `ARRAY<VARIANT>` |
| Execution plumbing | nullable columns, constants, unions, repeats,
exchange and arena serialization |
#### Explode coverage
`explode_variant_array()` emits array elements as Variant. Coverage
includes primitive values, canonical numeric equivalents, strings,
booleans, Variant null, objects, nested arrays, empty arrays,
scalar/non-array inputs, SQL `NULL`, grouped output, multiple arrays,
and outer explode semantics.
### Current limitations
- The feature is disabled by default and controlled by an experimental
FE session variable.
- Variant table creation, loading, segment storage, readers/writers,
statistics, and compaction remain on the legacy/native Variant path.
- Root Variant comparison predicates and Variant join keys are
unsupported.
- Sort, TopN, and root-Variant window partition/order keys are
unsupported. Window aggregate changes are excluded.
- `MIN(variant)` and `MAX(variant)` remain unsupported.
- E/T is a whole-column state; there is no row-level physical-state
mixing.
- Direct adapters between legacy Variant and compute-only Variant V2 are
not included.
- The PR does not add V1/V2 rolling-upgrade or mixed-version
compatibility handling.
The V2 regression suites remain tagged `nonConcurrent`, but correctness
no longer depends on process-wide config isolation or suite execution
order: V2 selection is session-scoped and native Variant compaction
always uses the storage type.
### Release note
Adds an experimental, default-disabled, session-scoped `ColumnVariantV2`
compute path with canonical hashing and arena serialization for
grouping, distinct aggregation, and set operations, plus parsing, casts,
conditional expressions, nested containers, and explode support. Root
Variant comparisons, Variant join keys, and Variant storage behavior are
not supported by this PR.
### Check List (For Author)
- Test:
- targeted FE analyzer/type-propagation UT;
- targeted BE Variant type, parse/cast, canonical, sort-state, and
serialization UT;
- `./build.sh --fe`;
- `BUILD_TYPE=ASAN ./build.sh --be`;
- `variant_compute_v2`, `variant_parse_functions`,
`test_variant_equality_contexts`, `test_variant_count_distinct`, and
`test_variant_ordering_comparison_error` regressions;
- native Variant cumulative compaction for DUPLICATE, UNIQUE, and
AGGREGATE tables via `test_compaction_variant`.
- Behavior changed: Yes (replaces the process-wide BE switch with a
session-scoped execution type marker while retaining canonical
hash/serialization for grouping, distinct aggregation, and set
operators).
- Does this need documentation: No
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt