### What problem does this PR solve?
Issue Number: close # (none)
Problem Summary:
`variant_p0/schema_change/test_double_write_when_schema_change` and its
`v2` sibling are flaky:
`qt_two_phase_off` / `qt_two_phase_on` intermittently fail with
```
Check tag 'two_phase_off' failed, line 6, TINYTEXT result mismatch.
Expect cell is: 123111
But real is : 123111.0
```
or the opposite direction (`expect 123111.0, real 123111`).
The suite inserts the string literal `"123111.0"` into `change_column`
while the column is being
altered from `double` to `text`, and then compares the whole row:
* rows written while the column is still `double` are stored as
`123111.0` and are converted by
the `double -> text` cast, which prints `123111`;
* rows written after the column has become `text` keep the literal
`123111.0`.
How many rows fall into each window depends on the schema change timing
(the test inserts one row
per loop iteration while `SHOW ALTER TABLE COLUMN` reports `RUNNING`),
so no single expected file
can match reliably.
This is a test issue rather than a product issue: both values are
correct for the phase in which
the row was written.
### What is changed?
Use an integer valued literal (`"123111"`) so the `double -> text`
conversion and the direct
`text` write both produce `123111`, and update the
`variant_p0/schema_change` expected file
accordingly. The `v2` expected file already contains `123111` only.
`branch-4.1` and `branch-4.2` carry the same case and would need the
same pick.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] No.
### What problem does this PR solve?
Related PR: #68758
SEARCH declares a non-nullable result, but UNKNOWN leaves and root-only
NULL masking can omit rows under NOT or change membership across scoring
and cached execution. Evaluate the complete backend query as a
two-valued match set: ordinary terms do not match NULL or proven-absent
fields; AND/OR combine match sets; NOT complements them. Ordinary SQL
expressions and MATCH retain their existing three-valued semantics.
Filter physical NULL postings at SEARCH leaf boundaries with the
existing `NullMaskedQuery`, including NULL ARRAY values with retained
nested payloads. Keep cached result bitmaps independently owned and
preserve source-managed document IDs across collectors.
This change uses generic execution for masked terms. It does not add
`excluded_rows` to term queries, weights, scorers, or grouped reads.
Grouped term opening and term-specific block-max WAND are deferred for
these masked leaves; no performance improvement or benchmark result is
claimed.
A missing iterator does not prove that a Variant path is absent. When a
historical rowset contains the path but has no executable index after
CREATE INDEX, fail explicitly instead of returning incomplete results.
Tests write actual objects with `parse_to_variant`, check their type and
extracted values, then distinguish unavailable historical indexes, newly
indexed rows, and genuinely absent paths. Index backfill and scan
fallback are outside this change. The historical-rowset fixture uses a
non-tokenized index with `enable_add_index_for_new_data=true`, so Cloud
and local mode both leave the historical rowsets unindexed.
**Independent validation:** this PR contains no FE or protocol changes
and was built and tested with FE source identical to master. Backend
field-existence semantics are tested directly through the existing
`TSearchClause` interface, including empty arrays, physical NULL
postings, numeric fields, absent paths, cache, scoring, and multiple
readers. SQL suites use clauses already emitted by master FE. There is
no companion-PR prerequisite or required merge order.
### What problem does this PR solve?
Issue Number: CIR-22037 (related: apache/doris#66298)
VARIANT compaction can fail forever with `E-3113 string column length is
too large`.
`ColumnStr<uint32_t>` rejects more han 4 GiB of uncompressed chars,
while vertical compaction
plans output segments from **compressed** bytes. High-compression-ratio
VARIANT data reaches the
uint32 offset limit long before the planned segment reaches
`vertical_compaction_max_segment_size`.
The writer side made this deterministic: the VARIANT V2 writer buffered
a whole output segment in
memory before flushing it.
Reproduced on master with 1.2M rows x 4 KiB of VARIANT payload (4.9 GiB
uncompressed, ~8 MiB on
disk) and `vertical_compaction_max_segment_size=256MiB`:
```
[E-3113] string column length is too large: total_length=4295296512,
limit=4294967295, element_number=1044576, rows=1043584
VariantV2ColumnWriter::append -> ColumnStr<unsigned int>::insert_range_from
```
### What changed?
1. **Stream the root-only VARIANT writer**
(`variant/v2/variant_column_writer.*`).
The root column writer is created at `init()` and every input batch is
appended to it, so the
writer never retains a segment-sized `ColumnString` (`_root_jsonb` is
gone). `finalize()` only
records the row count; `estimate_buffer_size()` reports the current page
instead of the whole
segment.
2. **Bound the writers that still buffer** (`compaction/compaction.*`,
`common/config.*`).
When the compaction schema still selects a segment-buffering writer -
the shredder (no
extracted subcolumns, e.g.
`enable_vertical_compact_variant_subcolumns=false`) or
`VariantSubcolumnWriter` (extracted VARIANT subcolumns) - clamp the
per-segment row budget
using the input segments' VARIANT `raw_data_bytes` and
`variant_compaction_max_segment_uncompressed_bytes` (2 GiB default, 0
disables).
`raw_data_bytes` is the pre-encoding size the writer buffers;
`uncompressed_data_bytes` is the
encoded page size and can be orders of magnitude smaller for
dictionary-encoded strings
(measured 142,560 vs 4,944,000,000 for the same segment). The streaming
path is not clamped, so
its segment layout does not change.
Depends on #68444, which makes variant writers report
`raw/compressed/uncompressed` data page
bytes in the segment footer (first commit is that change, kept here so
the branch is testable).
### Tests
-
`VariantColumnWriterReaderTest.test_variant_root_writer_streams_per_batch`
(new):
writes 4,000 x 4 KiB root scalars in 1,000-row batches and asserts the
buffered bytes stay below
two batches. Red/green verified locally with `doris_be_test`:
- red (buffering writer): `actual: 16428000 vs 8192000` -> FAILED
- green (streamed writer): `[ OK ] ... (156 ms)` -> PASSED
- Variant writer data page byte counters equal the bytes appended to the
file right after
`write_data()` (ordinary, extracted subcolumn, doc compact writer).
- `information_schema.column_data_sizes` reports VARIANT
compressed/uncompressed/raw bytes
(regression suite `test_query_sys_column_data_sizes_variant`).
- Manual end-to-end on a single-BE cluster (1.2M x 4 KiB):
- default path: **1 segment, no E-3113**, checksum unchanged
(count=1200000,
sum(len)=4927200000, sum(crc32)=2041942614000000), BE RSS peak +114 MiB
(was +5 GiB).
- `enable_vertical_compact_variant_subcolumns=false` (shredder): clamp
1,200,001 -> 521,233
rows per segment, **3 segments, no E-3113**, same checksum.
- **Sparse columns**: `variant_max_subcolumns_count=1` with a 4 KiB
`z_big` path that is not
materialized (sparse), extract off. Without the budget: E-3113 at
4,294,970,386 bytes
(`serialize_column_to_jsonb_vector`). With the budget: clamp 1,200,001
-> 517,465 rows per
segment, 3 segments, no E-3113, checksum identical to a fresh load
(count=1200000, sum(len)=4981200000, sum(crc32)=252636363600000).
### Remaining
- Non-vertical compaction does not use the segment row budget.
- Segments written before the footer raw-byte metadata existed (raw = 0)
cannot be estimated; the
streamed root path still covers them, the shredder path does not.
- Doc mode relies on the raw-byte budget.
### Release note
VARIANT compaction no longer buffers a whole output segment in one
`ColumnString`.
Issue Number: DORIS-29778
Problem Summary:
Test fixtures still assume the behavior that #68447 removed.
1.
`regression-test/suites/variant_p0/test_variant_json_functions.groovy`
builds its
Variant string row with `parse_to_variant('abc')`, which relied on the
old permissive
fallback that stored text that is not a JSON document as a Variant
string. Since
#68447, `parse_to_variant` rejects such input, so the INSERT fails and
the suite
stops before any assertion runs:
```
errCode = 2, detailMessage = [INVALID_ARGUMENT]Parse json document failed at row 0,
error: [INVALID_ARGUMENT]Failed to parse JSON as Variant
```
Parse the equivalent JSON string document `"abc"` instead: it produces
the same
Variant string value, so the remaining assertions still pass and the
expected
result file is unchanged.
2. BE unit tests still pass the removed
`JsonToVariantOptions::check_duplicate_json_path`
field, so the BE UT build fails to compile:
```
core/data_type_serde/data_type_variant_v2_serde_jsonb_test.cpp:115:32: error:
no matching constructor for initialization of 'JsonStringToVariantEncoder'
format/transformer/vparquet_writer_test.cpp:198:32: error:
no matching constructor for initialization of 'JsonStringToVariantEncoder'
```
Remove the field from the six affected initializers in
`data_type_serde_arrow_test.cpp`,
`data_type_variant_v2_serde_jsonb_test.cpp`,
`variant_v2_execution_test.cpp` and
`format/transformer/vparquet_writer_test.cpp`
(the last two were added by #69194) (DORIS-29778).
3. `external_table_p0/paimon/write/test_paimon_write_variant_errors`
expects
`try_parse_to_variant('not-json')` to be stored as the Variant string
`not-json`,
while it now returns NULL, so the row 21 check fails:
```
ExpectRow: [21, false, null, not-json]
RealRow : [21, true, null, null]
```
### What problem does this PR solve?
Related PR: #69077 (branch-4.1 backport)
Problem Summary: CREATE TABLE rejected top-level VARIANT column names
containing a dot, but ALTER TABLE ADD COLUMN could persist the same
invalid schema. Move the restriction into shared column-definition
validation so CREATE, single/batch ADD, and MODIFY enforce it
consistently. Reject RENAME to a dotted VARIANT name before changing
metadata, while preserving edit-log replay and nested-field semantics.
Update both existing CREATE regression controls to match the shared
VARIANT naming error, and assert the CREATE error in the FE SQL unit
test. The previous generic CREATE-only expectation caused
sql_create_time_range_table to fail even though the DDL was correctly
rejected.
### Release note
Reject ALTER TABLE operations that introduce a top-level VARIANT column
name containing a dot.
### Check List (For Author)
- Test:
- FE SQL unit test: `./run-fe-ut.sh --run
org.apache.doris.alter.VariantColumnNameTest` — passed (1 test, 0
failures/errors; full wrapper exit 0 after FE packaging).
- Master FE build and Checkstyle: `DISABLE_BUILD_UI=ON ./build.sh --fe
--exclude-obs-dependencies` — passed; exclude OBS to avoid dependency
repository validation stalls.
- On both master and branch-4.1, run `sql_create_time_range_table` under
`ddl_p0` and `test_variant_column_name` under `variant_p0/schema_change`
with `run-regression-test.sh`: all four focused suite runs passed, 0
failures/skips. Each runtime used its branch's built FE and a prebuilt
ASAN BE (326d74e175b); no BE source changed.
- Branch-4.1 FE Checkstyle and `git diff --check`: passed.
- Behavior changed: Yes; reject previously accepted invalid VARIANT
column names. Dotted names of other types remain supported.
- Does this need documentation: Yes;
https://github.com/apache/doris-website/pull/4307
Metadata / compatibility review: reject invalid names before catalog
mutation; RENAME retains its write lock and replay behavior. Both local
and cloud tables use the shared FE validation; no protocol or metadata
format changes.
### What problem does this PR solve?
Issue Number: None
Related PR: #67551
Problem Summary:
Variant V2 import spends most of its BE CPU in the segment writer's
Variant shredder. Importing 1M rows of sparse JSON (about 20 of 2,000
keys per row: BIGINT, string, double, boolean, BIGINT arrays, and small
objects) into a default `VARIANT` table through a `local()` TVF used
49.94 BE CPU seconds, and 70.1% of the BE samples were in
`VariantShredder::append`. Two defects in the ARRAY path of
`VariantPathBuilder` caused most of it.
1. **Array type reuse never matched.** `infer_type()` tries to reuse the
path's existing `DataTypeArray` when the element type is unchanged.
`DataTypeArray` always wraps its element in `Nullable`, but inferred
element types are never nullable, so `Nullable(BIGINT).equals(BIGINT)`
failed for every array value. Each value then went through
`path_least_common_type()` and `get_least_supertype_jsonb()`, which
builds temporary `DataTypes` vectors, strips `Nullable`, and allocates a
new `DataTypeNullable` only to arrive at the same type. The first commit
compares the unwrapped element type.
2. **Static data types were copied per value.** Memtable flushes of
several tablets shred concurrently. For every ARRAY value the builder
copied process-wide static data types by value: `infer_type()` returned
each element's static type, and `append_array()` and
`value_is_representable()` unwrapped the element with
`remove_nullable()`. Each copy is an atomic reference-count update on a
control block shared by all flush threads. With eight concurrent segment
writers, `perf annotate` put most of `infer_type()`'s own samples, a
third of `value_is_representable()`'s, and a fifth of `append_value()`'s
on lock-prefixed reference-count instructions. The second commit returns
scalar element types by reference, resolves a common element type only
when elements differ, and borrows the unwrapped array element.
Inferred and promoted types are unchanged in both commits.
Results, RELEASE build, one single-node cluster, binaries swapped
between runs in two interleaved rounds, medians of 6 imports each:
| Build | BE CPU s | Wall s | Rows per BE CPU s | Shredder share of BE
CPU |
|---|---|---|---|---|
| master | 49.94 | 8.41 | 20,026 | 70.1% |
| + array type reuse | 38.19 | 7.18 | 26,185 | 51.1% |
| + borrowed static types | 24.71 | 5.61 | 40,461 | 22.4% |
The imported data checksum is identical across all 18 imports.
`get_least_supertype_jsonb()` dropped from 21.6% of BE CPU to not
sampled.
`BM_VariantSparseImport` from #67551, thread CPU seconds per 1M rows,
medians of 5 samples:
| Scenario | master | + array type reuse | + borrowed static types |
|---|---|---|---|
| MixedTypes, 8 concurrent writers | 25.66 | 18.81 | 11.23 |
| MixedTypes, 1 writer | 12.09 | 10.56 | 10.84 |
| NoArrays, 8 concurrent writers | 8.62 | 8.74 | 8.93 |
With both commits the eight-writer cost matches the single-writer cost.
The single-writer and NoArrays rows vary by about 10% between runs on
the shared test host.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Validated on base 17ac3d9275f; this branch is rebased onto a newer
master whose two additional commits do not touch these files.
- Unit test:
`VariantPathBuilderTest.*:VariantShredderTest.*:VariantColumnWriterReaderTest.*`,
78 passed and 2 skipped (skipped on master too), including the new
`VariantPathBuilderTest.ArrayPathReusesElementTypeAcrossRows`.
- Regression test on a RELEASE cluster with each commit: `variant_p0`
suites `regression_test_variant`, `regression_test_variant_types`,
`test_variant_array_subscript`,
`regression_test_variant_array_with_predicate`,
`test_variant_array_function`, `variant_compute_v2`,
`regression_test_variant_multi_var`,
`regression_test_variant_predefine_schema` (10 suite files), all passed.
- Manual test: the 1M-row import and `BM_VariantSparseImport`
comparisons above; `build-support/check-format.sh` and
`build-support/check-build-hygiene.sh` pass. clang-tidy on the changed
test file reports nothing; on `variant_path_builder.cpp` the clang
static analyzer crashes on unchanged `__int128` code, and without
`clang-analyzer-*` the only diagnostic is the existing cognitive
complexity of the unchanged `VariantPathBuilder::append`.
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
### What problem does this PR solve?
Issue Number: None
Related PR: #66858
Problem Summary: After #66858 enabled Variant V2 by default, the V1 JSON
parser in `be/src/util/json/json_parser.{h,cpp}` (`JSONDataParser`,
`ParseConfig`, `ParseResult` and the
`getValueAsField`/`writeValueAsJsonb` helpers) has no production caller;
only tests still referenced it.
This PR removes:
- the parser itself (`be/src/util/json/json_parser.{h,cpp}`), including
its `extern template class JSONDataParser<SimdJSONParser>` declaration
and the matching explicit instantiation;
- its dedicated test `be/test/core/jsonb/json_parser_test.cpp`, which
covered nothing else;
- the stale `#include "util/json/json_parser.h"` lines left in
`segment_creator.cpp`, `function_variant_parse.cpp`, `variant_util.cpp`,
`cast_to_decimal128_perf.cpp` and `index_storage_test_util.h`;
- the never-read `IndexBatch::parse_to` test field (of type
`ParseConfig::ParseTo`) and its single assignment in
`index_storage_variant_debug_point_test.cpp`.
`be/test/util/variant/variant_field_test.cpp` contained one case
comparing the V1 parser against the V2 encoder; only its V2
`encode_json` assertions are kept, renamed to
`VariantFieldTest.EncodeJsonStableSubset`.
`util/json/simd_json_parser.h` and `util/json/path_in_data.{h,cpp}` are
kept, as both still have production users.
Note: `be/benchmark/benchmark_variant_segment.hpp` also references
`ParseConfig`, but its V1 branch already fails to compile on master (it
calls `ColumnVariant`, `variant_util::parse_json_to_variant` and
`variant_util::select_storage_variant_parse_target`, none of which exist
any more). It is built only with the BE benchmark target and is left for
a separate fix.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `build-support/check-build-hygiene.sh` passed
- ASAN BE build (`./build.sh --be`) passed
- `./run-be-ut.sh --run --filter='VariantFieldTest.*:IndexStorage*'`
passed: 83 passed, 2 pre-existing skips
- Behavior changed: No
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
### What problem does this PR solve?
Problem Summary:
An inverted index on a VARIANT column is copied to every extracted
subcolumn, so a segment holds one
copy of it per path. Since #53980 every analyzed index calls
`setOmitNorms(false)`, and CLucene writes
norms densely: 12 header bytes plus **one byte per segment row for every
indexed field**, including
the rows that hold no value for that field. For a VARIANT column that
cost is multiplied by the
number of paths.
On a real table (~1400 paths, 7 non-BKD indexes per path) a single
base-compacted segment held 9891
`.nrm` files of 4,055,884 bytes each: **37.36 GiB of norms out of a
38.47 GiB `.idx`, 97.1%**.
Segments written before the upgrade had no `.nrm` at all, and the
non-norms bytes had not grown.
Reproduced end to end on 1M rows / 1000 paths / 11 `field_pattern`
indexes, by loading with 3.1 and
then upgrading the same data in place to 4.1:
| | index size after full compaction | `.nrm` |
| --- | --- | --- |
| 3.1 | 99.6 MiB | none |
| 4.1, same data | 934.1 MiB | 875 files x 1,000,012 B = 834.5 MiB |
**9.4x**, and the norms are the whole difference. A/B on the same data:
norms on 457.3 MiB vs norms
off 99.6 MiB (**4.6x**), with identical `MATCH` row counts and sums.
#### What this PR changes
This PR gives operators a way to reclaim that space without changing
what anyone gets by default,
and applies it the same way to the CLucene formats (V2/V3) and to SNII.
1. **A `norms` index property.** An analyzed index writes norms unless
it sets `"norms" = "false"`,
on a VARIANT path exactly as on an ordinary column. A subcolumn copy
inherits the properties of
the index it came from, so the property on a whole-column VARIANT index
also settles every one of
its subcolumn copies.
2. **A mutable BE config `inverted_index_skip_norms_for_variant`, off by
default.** Turning it on
drops norms for every index on a VARIANT path (an index declared with a
`field_pattern`, or the
copy an extracted subcolumn inherits, which carries the path as its
index suffix), whatever that
index's property says, so a cluster hit by the size regression can
recover the space without
rewriting its index definitions.
3. **One policy function.** `should_write_index_norms(const
TabletIndex&)` holds the rule, and the
three places that decide whether norms get written all call it: the
CLucene writer, the SNII
writer, and SNII direct compaction. SNII direct compaction rebuilds
norms from the merged postings
rather than copying them; the other SNII compaction path rebuilds the
index through the SNII
writer.
4. **BM25 scoring needs norms from every segment it reads, in both
formats.** `MATCH` filtering is
unaffected; a query that computes `score()` on an analyzed index fails
with
`INVERTED_INDEX_NOT_SUPPORTED` when any segment of the field has no
norms.
- SNII already worked this way (`resolve_snii_scoring_segment`); its
message now names the
property and the config instead of suggesting that compaction will bring
the norms back.
- CLucene did not. It keeps a field's token count in the `.nrm` header,
and a row without norms
reads the fake norm `encodeNorm(0)`, i.e. length 0. With no segment
carrying norms, `avgdl` was
0 and `score()` came out NaN. With only some segments carrying norms, as
while the config is
being turned on, `avgdl` stayed positive and the rows without norms
ranked as the shortest
possible documents, above otherwise identical rows.
`CollectionStatistics` now refuses such a
segment before any score is computed. An index that is not analyzed
never writes norms and is
collected as before, so `SEARCH` scoring over keyword fields is
unchanged.
#### Compatibility
- Defaults are unchanged: norms are still written everywhere unless
someone opts out, and an index
that has norms scores exactly as before.
- No storage format change.
- Turning the config on, or setting `"norms" = "false"`, is for indexes
that are never ranked with
`score()`. For such an index, `score()` fails as soon as one segment
without norms exists.
- A table upgraded from a version that did not write norms (before 4.0)
now fails `score()` on its
analyzed indexes until compaction has rewritten the old segments, where
it used to return NaN or
skewed scores.
- Change the property or the config only after every BE runs this
version.
### Release note
Added the `norms` inverted index property and the BE config
`inverted_index_skip_norms_for_variant`
(off by default, mutable), which drops BM25 norms for inverted indexes
on VARIANT paths and can cut
the index size of a VARIANT column with many indexed paths by several
times. Both apply to the V2,
V3 and SNII index storage formats. A query that computes `score()` on an
analyzed index now fails
when a segment it reads has no norms, instead of returning NaN or
inconsistent scores.
### Check List (For Author)
- Test: Unit Test / Regression test
- `be/test/storage/segment/inverted_index_writer_test.cpp`:
`NormsFollowIndexNormsProperty`
- `be/test/storage/index/snii_writer_test.cpp`:
`SniiWriterNorms.WritesNormsFollowSharedNormsPolicy`
-
`be/test/storage/index/snii/compaction/snii_compaction_eligibility_test.cpp`:
`DestinationWritesNormsFollowSharedNormsPolicy`
-
`be/test/storage/index/inverted/similarity/collection_statistics_test.cpp`:
`LegacyV3RejectsSegmentWrittenWithoutNorms` (alone and next to a segment
with norms),
`LegacyV3KeywordIndexWithoutNormsIsStillCollected`, and the existing
SNII rejection tests
-
`fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java`
- Regression: `inverted_index_p0/test_variant_subcolumn_index_norms`
(checks the `.nrm` files)
and `inverted_index_p0/storage_format/test_storage_format_snii_norms`
(reads norms off the
scores): `MATCH` still filters, `score()` fails without norms, including
on a mixed-generation
table. Both are nonConcurrent, since they flip the BE config. Also
rerun:
`test_bm25_score_variant`, `test_omit_norms`,
`test_search_score_topn_predicates`,
`test_search_score_cache`, `test_search_score_topn_delete_predicate`,
`test_storage_format_snii`
- Behavior changed: No for defaults. `score()` on an analyzed index with
a segment that has no norms
now fails instead of returning NaN or skewed scores.
- Does this need documentation: Yes
(https://github.com/apache/doris-website/pull/4146)
ColumnVariantV2 needs a reproducible component benchmark that exercises
real segment write/read paths, scales beyond one segment, distinguishes
materialized and sparse exact-path reads, and measures the current
production compaction pipeline without inventing a benchmark-only V2
compaction mode.
### What is changed?
- Keep 1,000,000 rows as the checked-in default and add
environment-controlled total rows and rows per segment. Input is
generated in 4K batches, while parse/append/finalize remain in the timed
region.
- Add multi-segment ingestion for `Sparse16`, `Doc16`, and `Full`
layouts.
- Add exact-path read cases corresponding to the storage-scan portion of
a literal `SELECT v['path']` after subpath pruning:
- `Materialized100pct`: materialized `LEAF` route.
- `Sparse6pct`: sparse-bucket `BINARY_EXTRACT` route with one hit per 17
rows.
- Add a scan-and-rewrite pipeline benchmark. It queries key plus whole
Variant from the same
prevalidated V1-written source Segment as `ColumnVariant` or
`ColumnVariantV2`, then passes that
queried Block directly to a new `SegmentWriter` and reports every
pipeline stage separately.
- Add a current-production cumulative-compaction benchmark with ten
overlapping rowsets written by Variant V2. It calls public
`CumulativeCompaction::execute_compact()`, disables and verifies the
ordered-link shortcut, and validates the physically rewritten output.
- Keep correctness checks outside timing: full row/layout/route checks,
bounded whole-value canonical samples, and full exact-path checksums/hit
counts.
- Do not use whole-column read performance as a gate in this round.
The compaction benchmark intentionally does **not** label the current
compactor as V2. `TabletColumn::variant_is_v2` is a transient read
marker and is not persisted through schema protobuf copies, so the
benchmark records `persisted_compaction_variant_v2=0` instead of
creating a misleading V1/V2 pair.
### Release benchmark evidence
- Default 1M ingestion, five samples per representation:
| Layout | V1 median (ms) | V2 median (ms) | V2/V1 |
|---|---:|---:|---:|
| Sparse16 | 9,212.349 | 8,202.836 | 0.8904 |
| Doc16 | 7,917.702 | 8,286.466 | 1.0466 |
| Full | 10,231.216 | 9,195.444 | 0.8988 |
- Final exact-path 1M, five samples per representation:
| Route | V1 median (ms) | V2 median (ms) | V2/V1 |
|---|---:|---:|---:|
| Materialized `LEAF` | 3.630 | 2.403 | 0.6619 |
| Sparse `BINARY_EXTRACT` | 28.254 | 27.006 | 0.9558 |
#### Scan and rewrite: 1M rows
The source is the same V1-written physical Segment for both sides. The
only switched dimension is
the requested query representation and therefore the column handed to
the destination writer.
Every group contains five V1 and five V2 samples in fixed ABBA order.
Lower is better.
| Layout | Representation | Total median (ms) | Read median (ms) |
Append median (ms) | Finalize median (ms) | Total CV |
|---|---|---:|---:|---:|---:|---:|
| Sparse16 | V1 | 8,429.347 | 3,347.014 | 1,580.744 | 3,501.255 | 0.554%
|
| Sparse16 | V2 | 11,275.605 | 5,869.060 | 2,005.063 | 3,401.644 |
0.543% |
| Doc16 | V1 | 7,035.376 | 2,719.646 | 699.392 | 3,610.837 | 0.238% |
| Doc16 | V2 | 11,568.249 | 6,106.403 | 1,977.956 | 3,485.382 | 0.095% |
| Full | V1 | 8,023.795 | 2,010.926 | 1,166.440 | 4,804.266 | 0.212% |
| Full | V2 | 13,154.337 | 4,567.923 | 4,201.698 | 4,400.651 | 0.799% |
V2/V1 total latency is `1.3377` for Sparse16, `1.6443` for Doc16, and
`1.6394`
for Full. The gap is in the query/read and append stages; destination
finalize is comparable or
slightly faster for V2 (`0.92x`-`0.97x`). Destination segment size and
physical layout match for V1
and V2 in every layout.
Related PR: #62917
Problem Summary: The legacy `__DORIS_ROWID_COL__` ResultSink two-phase
fetch path remained after TopN lazy materialization moved to
`__DORIS_GLOBAL_ROWID_COL__` and `multiget_data_v2`. This PR removes the
unused FE and BE execution, rowset-retention, and row-at-a-time storage
read code.
The old protobuf and Thrift definitions retain their original field IDs
and are marked deprecated. Deprecated `multiget_data` calls now return
`NotSupported`.
### What problem does this PR solve?
Related PRs: #66204, #66668
Variant V2 ingestion repeatedly performed work that could be reused:
- Object layout and dictionary metadata were decoded again while
iterating children.
- Stable scalar paths repeated type inference and least-common-type
calculation.
- Narrow encoded integers caused unnecessary type promotion and column
rewrites.
- Sparse-cell serialization recreated SerDe objects.
- Finalization retained a `{plan, value}` pair for every sparse/doc
cell.
Typed STRING values were also converted row by row when cast back to
STRING, even though the
physical payload already had the requested representation.
This PR keeps only the optimizations that showed meaningful value. It
does not include native
shredded execution or any API dependency from #66705.
### What is changed?
#### Ingestion and finalization
- Add `VariantRef::ObjectView` to parse and validate an object's
physical layout and dictionary
size once while iterating its children.
- Cache observed parent/field-to-path transitions during one append
call. The earlier
dictionary-sized dense root cache was removed because its measured
benefit did not justify
the additional allocation and special-case logic; root and nested paths
now share the same
existing hash-cache representation.
- Add a strict stable-scalar fast path that skips repeated inference/LCS
work only when the
established physical type cannot change. Widening, arrays, decimal
range/scale changes, and
conflicts keep the original slow path and JSONB fallback.
- Infer encoded INT8/INT16/INT32/INT64 values as BIGINT. This matches
the V1 JSON parser,
avoids integer narrowing, and eliminates promotion churn between encoded
integer widths.
- Cache the per-path binary SerDe and invalidate it when a path is
initialized, promoted, or
converted.
- Transpose sparse/doc cells in bounded chunks. Per-path cursors recover
value indexes, so the
implementation stores only a compact plan id per cell instead of a
`{plan, value}` pair.
#### Typed STRING cast
- Reuse the existing `ColumnString` payload when a typed STRING-family
Variant is cast to
STRING.
- Merge forced SQL nulls without copying the nested payload.
- Preserve Variant-null semantics: a visible inner null still follows
the original path and
stringifies as literal `null`.
### Correctness coverage
- Object-view iteration and corruption boundaries.
- Root/nested path reuse and bounded transpose chunk boundaries.
- Stable scalar fast-path boundaries, widening, decimal conversion, and
conflict fallback.
- V1/V2 mixed integer/double inference and BIGINT normalization.
- Binary SerDe cache invalidation after promotion/conversion.
- Typed STRING identity, forced-null merging, payload reuse, and visible
Variant nulls.
### Performance validation
#### GH Archive ingestion
Input: 168 gzip files, 2,533,666 rows. Five alternating V1/V2 samples
with
`load_parallelism=8`:
| Version | Median wall time | Throughput normalized by 8 load slots |
|---|---:|---:|
| V1 | 21 s | 15,081 rows/s/slot |
| V2 | 19 s | 16,669 rows/s/slot |
V2 throughput is 10.53% higher.
The single-scanner boundary remains slower for V2 (V1 43 s, V2 55 s).
Profiling shows that
V2 performs Variant conversion in the scanner while V1 defers most
parsing to flush. Removing
the scanner's intermediate JSON serialization/reparse needs a separate
reader-to-builder
interface and is intentionally outside this PR.
#### GH Archive scalar queries
ARRAY/JSON-dependent statements were excluded, leaving 67 scalar SQL
statements. Three
paired V1/V2 runs produced identical output hashes for all statements:
| Version | Median suite wall time | Median CPU time | Single-core
throughput |
|---|---:|---:|---:|
| V1 | 8.990 s | 36.12 core-s | 1.855 queries/core-s |
| V2 | 8.101 s | 32.61 core-s | 2.055 queries/core-s |
V2 single-core throughput is 10.76% higher and wall throughput is 10.98%
higher. No scalar
SQL regressed by 10% or more; the largest observed regression was 7.88%,
within the 15%
acceptance threshold.
Excluded as ARRAY/JSON-related: `sql01.sql`, `sql02.sql`, `sql03.sql`,
`sql07.sql`,
`variant_issue_labels_explode.sql`, `variant_payload_shape_by_type.sql`,
`variant_push_commit_message_terms.sql`, and
`variant_push_commits_explode.sql`.
#### Google Benchmark from #66668
Release benchmark pinned to physical CPU 4, 1,000,000 rows, five ABBA
samples per version and
layout. All 30 cases passed. Throughput is normalized from CPU time to
one core:
| Layout | V1 median CPU | V2 median CPU | V1 throughput | V2 throughput
| V2 vs V1 |
|---|---:|---:|---:|---:|---:|
| Sparse16 | 7,864.076 ms | 7,463.093 ms | 127,161 rows/core-s | 133,993
rows/core-s | +5.37% |
| Doc16 | 6,673.045 ms | 7,545.014 ms | 149,857 rows/core-s | 132,538
rows/core-s | -11.56% |
| Full | 7,721.605 ms | 7,058.033 ms | 129,507 rows/core-s | 141,683
rows/core-s | +9.40% |
All layouts are within the 15% acceptance threshold. The two whole-read
helpers that require
#66705 were excluded; they are not used by the ingestion benchmark
filter.
### Validation
- Clang-format 16 and `git diff --check`: passed.
- Targeted ASAN BE tests: 27/27 passed across `VariantPathBuilderTest`,
`VariantShredderTest`, and `VariantValueTest`.
- `CastVariantV2FromTest`: 22/22 passed, including the STRING identity
and null-semantics
cases.
- Release BE and benchmark targets built with default build concurrency
and no explicit `-j`.
- Independent source review: APPROVE; P0=0, P1=0, P2=0.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (GH Archive and Release Google Benchmark)
- Behavior changed:
- [x] No user-visible behavior change; encoded integer inference now
matches V1 BIGINT
semantics.
- Does this need documentation?
- [x] No.
### Release note
None
Related PR: #65561
Problem Summary:
This PR adds the ColumnVariantV2 segment read and write paths while
preserving the existing V1 behavior.
For reads, eligible scans assemble legacy Variant segment data into
ColumnVariantV2 through plan/type metadata. Typed, sparse, document, and
root values retain their native representation, and the reader covers
RowID, hierarchical multi-batch, cache, malformed-input, and legacy
scalar cases.
For writes, `Config.enable_variant_v2` is propagated through
`VariantType.toThrift()` to select the V2 storage type. V1 keeps the
existing STRING-to-Variant sink path, while V2 inputs are explicitly
parsed into ColumnVariantV2. Nereids load planning injects
`try_parse_to_variant` only when V2 is enabled. The change deliberately
removes the V2-to-V1 cast bridge because a statement is not expected to
mix the two physical representations.
The segment layer keeps version-native SQL NULL and JSON null behavior
without adding conversion state or per-row normalization. Complex and
nested arrays that cannot use the same typed layout as V1 fall back to
JSONB, which also makes V1 and V2 `DESC` output consistent.
V2-only parsing and compute coverage is isolated under `variant_p0/v2`
and dynamically enables the FE config. It covers casts, aggregation,
DISTINCT, UNION/INTERSECT/EXCEPT, conditionals, arrays, explode, INSERT
SELECT, and segment round trips. Shared suites keep V1 string literals
and use explicit `parse_to_variant` for V2.
### Release note
Support ColumnVariantV2 segment reads and writes when
`enable_variant_v2` is enabled. The legacy V1 path remains available
when the config is disabled.
Related PR: #60847, #63660
Problem Summary:
OlapScanner materializes a requested Variant path as a scan-schema
column. For example, data.items.message gets its own SlotRef and scan
column position. VariantColumnReader then resolves either direct
subcolumn index metadata or metadata inherited from data, and
SegmentIterator constructs the runtime iterator for that child column
with the complete stored Variant path.
VSearch duplicated that storage responsibility. When the child iterator
was absent, it resolved the tablet ColumnId for data, borrowed the
parent iterator, and synthesized a stored field name. This mixed scan
column positions with tablet ColumnIds and could bypass the path and
physical index selection already performed by the storage layer,
including NestedGroup selection.
This PR removes the expression-level parent rebinding. VSearch now
consumes only the iterator attached to the SlotRef scan column. If that
iterator is absent, the field remains an empty index input. It also
removes the three fallback-only IndexExecContext APIs and separates
SlotRef collection into a small helper that names the scan column index
explicitly.
A focused unit test constructs the exact boundary case: the data parent
iterator exists, while the data.items.message child iterator does not.
The test verifies that SEARCH does not mark or execute the parent
iterator.
### Release note
Variant SEARCH no longer borrows a parent Variant iterator when the
materialized child iterator is absent.
Related PR: #61190
Problem Summary: Root VARIANT MATCH predicates were rejected only when
the Match expression was a top-level filter conjunct. A Match nested
under OR could bypass frontend analysis and reach backend execution,
where it failed with an unrelated runtime error. Recursively collect
Match expressions from each filter expression and apply the existing
operand and root Variant validation to every Match node. Add focused FE
unit and regression coverage for a mixed text/root Variant OR predicate.
### Release note
Reject MATCH predicates on VARIANT root columns even when nested in
compound filter expressions.
### Check List (For Author)
- Test: Regression test / Unit Test
- `./run-regression-test.sh --run -d search -s
test_disable_root_variant_match` (failed before the fix and passed after
it)
- `./run-fe-ut.sh --run
org.apache.doris.nereids.rules.rewrite.CheckMatchExpressionTest`
- `./build.sh --fe`
- Behavior changed: Yes. Nested root VARIANT MATCH predicates now fail
during FE analysis, consistent with direct predicates.
- Does this need documentation: No
Related PR: #63088
Problem Summary:
Array elements are nullable after #63088. `get_base_type_of_array()`
therefore returns `Nullable<Variant>` for the NestedGroup physical type,
but `is_nested_group_type()` directly casts that wrapper to
`DataTypeVariant`. The false negative bypasses the NestedGroup-specific
type-conflict policy during Variant merge/compaction.
This PR unwraps the nullable base type only when the input has an Array
dimension. It recognizes `Array<Nullable<Variant>>` as NestedGroup while
preserving the existing classifications of scalar `Variant` and
`Nullable<Variant>`.
Problem Summary: This PR adds an experimental, compute-only
`ColumnVariantV2` execution path. It uses Apache Parquet Variant
Encoding v1 for encoded values and canonicalizes logical values for
hashing and arena serialization. The path is selected by the
experimental FE session variable `enable_variant_v2`, which is `false`
by default.
When the session variable is enabled, FE assigns an execution-only
Variant V2 type marker to compute results produced by
`parse_to_variant`, `try_parse_to_variant `, and explicit `CAST(... AS
VARIANT)`. The marker is also applied recursively to Variant elements
inside `ARRAY`, `MAP`, and `STRUCT` cast targets. It is forwarded
through Thrift/Protobuf descriptors so BE creates `DataTypeVariantV2`
and the matching `ColumnVariantV2` physical column.
The marker is transient and is not persisted in table metadata. This PR
does not change Variant table creation, load, segment storage,
readers/writers, statistics, or compaction. Those paths continue to use
the legacy/native `ColumnVariant`. Focused compaction coverage verifies
that compute V2 sessions do not affect native Variant storage.
### ColumnVariantV2 behavior
#### Physical states
`ColumnVariantV2` has two whole-column physical states:
- **Encoded (E):** values use Parquet Variant Encoding v1.
- **Typed scalar (T):** the whole column uses one Doris scalar type plus
an internal Variant-null map.
A column does not mix E and T row by row. SQL `NULL` remains outside the
Variant value and is distinct from Variant/JSON `null`.
#### Enable ColumnVariantV2
Enable it for the current FE session:
```sql
SET enable_variant_v2 = true;
```
The default is `false`. There is no mutable BE config for this feature.
V2 selection is carried by the expression result type, so enabling it in
one session does not change the physical Variant type used by storage or
compaction work in other sessions.
#### Construct and access Variant values
Use `parse_to_variant()` to parse JSON text:
```sql
SELECT parse_to_variant('{"id": 1, "items": [true, null, "x"]}');
SELECT parse_to_variant('42');
SELECT parse_to_variant('"a JSON string"');
```
`try_parse_to_variant()` returns SQL `NULL` for Variant validation
failures. `CAST(string AS VARIANT)` creates a typed Variant string; it
does not parse JSON.
Object keys and zero-based array indexes can be accessed and then
explicitly cast:
```sql
SELECT CAST(parse_to_variant('{"object":{"k":1}}')['object']['k'] AS INT);
SELECT CAST(element_at(parse_to_variant('[10,20,null]'), 0) AS INT);
```
#### Canonical hash and arena serialization
Hashing and arena serialization use the same canonical logical-value
representation for encoded and typed columns. This makes semantically
equivalent values produce the same serialized hash key even when their
physical representations differ.
Canonical rules include:
- equivalent integral numeric representations normalize together;
- decimal trailing zeros normalize;
- `+0`, `-0`, and integral zero normalize together;
- object key order does not affect the canonical value, while array
element order does;
- Variant null is distinct from SQL `NULL`;
- invalid external encodings and broken internal invariants return
errors.
The canonical hash/serialization path supports:
- `GROUP BY`;
- `DISTINCT` and `COUNT(DISTINCT ...)`;
- `INTERSECT`, `EXCEPT`, and `UNION DISTINCT`;
- serialized hash keys used by aggregation and set operators.
#### Comparison predicates and join keys
Root Variant `ComparisonPredicate` expressions are not supported in this
phase, including `=`, `!=`, `<=>`, and ordering comparisons. Cast both
sides to a concrete scalar type first:
```sql
-- Supported
SELECT CAST(v1 AS STRING) = CAST(v2 AS STRING);
-- Rejected
SELECT v1 = v2;
```
Variant expressions are also not supported as join keys. This PR does
not add Variant Hash Join, Variant join runtime filters,
equality-specific column dispatch, or join-properties compatibility
rules.
#### Supported compute operators
| Area | Supported behavior |
| --- | --- |
| Parse and output | `parse_to_variant`, `try_parse_to_variant `,
Variant-to-string/JSON output |
| Cast | Scalar SQL types to Variant, nullable Variant-to-scalar
conversion, element-wise array conversion |
| Canonical hash | `GROUP BY`, `DISTINCT`, `COUNT(DISTINCT)`,
`INTERSECT`, `EXCEPT`, `UNION DISTINCT` |
| Aggregation | `COUNT(*)`, `COUNT(variant)`, grouping keys, string
coercion for `group_concat` |
| Conditional expressions | `IF`, `CASE`, `IFNULL`, `COALESCE`, eager
and short-circuit execution |
| Nested containers | Variant values inside `ARRAY`, `MAP`, and `STRUCT`
|
| Table functions | `explode_variant_array`, multi-input explode,
`explode`/`explode_outer` for `ARRAY<VARIANT>` |
| Execution plumbing | nullable columns, constants, unions, repeats,
exchange and arena serialization |
#### Explode coverage
`explode_variant_array()` emits array elements as Variant. Coverage
includes primitive values, canonical numeric equivalents, strings,
booleans, Variant null, objects, nested arrays, empty arrays,
scalar/non-array inputs, SQL `NULL`, grouped output, multiple arrays,
and outer explode semantics.
### Current limitations
- The feature is disabled by default and controlled by an experimental
FE session variable.
- Variant table creation, loading, segment storage, readers/writers,
statistics, and compaction remain on the legacy/native Variant path.
- Root Variant comparison predicates and Variant join keys are
unsupported.
- Sort, TopN, and root-Variant window partition/order keys are
unsupported. Window aggregate changes are excluded.
- `MIN(variant)` and `MAX(variant)` remain unsupported.
- E/T is a whole-column state; there is no row-level physical-state
mixing.
- Direct adapters between legacy Variant and compute-only Variant V2 are
not included.
- The PR does not add V1/V2 rolling-upgrade or mixed-version
compatibility handling.
The V2 regression suites remain tagged `nonConcurrent`, but correctness
no longer depends on process-wide config isolation or suite execution
order: V2 selection is session-scoped and native Variant compaction
always uses the storage type.
### Release note
Adds an experimental, default-disabled, session-scoped `ColumnVariantV2`
compute path with canonical hashing and arena serialization for
grouping, distinct aggregation, and set operations, plus parsing, casts,
conditional expressions, nested containers, and explode support. Root
Variant comparisons, Variant join keys, and Variant storage behavior are
not supported by this PR.
### Check List (For Author)
- Test:
- targeted FE analyzer/type-propagation UT;
- targeted BE Variant type, parse/cast, canonical, sort-state, and
serialization UT;
- `./build.sh --fe`;
- `BUILD_TYPE=ASAN ./build.sh --be`;
- `variant_compute_v2`, `variant_parse_functions`,
`test_variant_equality_contexts`, `test_variant_count_distinct`, and
`test_variant_ordering_comparison_error` regressions;
- native Variant cumulative compaction for DUPLICATE, UNIQUE, and
AGGREGATE tables via `test_compaction_variant`.
- Behavior changed: Yes (replaces the process-wide BE switch with a
session-scoped execution type marker while retaining canonical
hash/serialization for grouping, distinct aggregation, and set
operators).
- Does this need documentation: No
A distributed ordered query using TopN lazy
materialization could return an array root such as [] instead of the
object fields stored in a VARIANT row. Root visibility already
considered dense subcolumns and document snapshots but ignored sparse
subcolumns, so a non-null root incorrectly won during serialization.
Check the current row sparse offsets before treating the root as
visible. Before the fix the reproduced row returned []; after the fix it
returns {"n":7,"word":"hot"}.
## Proposed changes
- Distinguish the variant root from an empty JSON key in compaction
schema paths. The root stays `PathInData()`; the empty key is
materialized with an explicit empty path part, for example `v.` with
parts `["v", ""]`.
- Stop forcing empty-key paths into sparse columns. Empty keys now
follow the normal path-stat top-N selection rules.
- When path stats select the empty key but rowset type metadata only
exposes the variant root, append a VARIANT empty-key subcolumn fallback
so the key is not dropped.
- Add BE coverage for root vs empty-key path representation and keep
regression coverage for bucketized sparse compaction.
## Root cause
Temporary logs confirmed that the empty JSON key is present in path
stats as `""`, including stats aggregated from sparse metadata. The loss
happens when compaction schema construction treats the data-type path
`PathInData()` as the variant root and skips it via `path.empty()`, so
the selected empty key never gets an extracted subcolumn. Representing
the empty key as an explicit empty path part avoids that root collision.
Build a reusable BE index-storage unit-test fixture and
migrate the Variant `variant_p0` debugpoint/index/compaction coverage
that belongs at storage-reader/writer level into focused unit tests.
This PR intentionally keeps the newly-covered product bugs unfixed: the
tests remain as reproducers for Variant sparse stats, Variant
field_pattern index build/drop, patched schema compaction with missing
physical columns, and COUNT_ON_INDEX no-need-read-data behavior.
The migrated coverage includes array inverted-index writer debugpoint
checks, Variant path index lifecycle and compaction coverage,
field-pattern typed index coverage for cumulative/full compaction,
common expression pushdown for `IS NULL` / `IS NOT NULL`, and
`array_contains` common-expression index filtering. Variant and
non-Variant coverage is split into independent test files, with shared
rowset write/read/reload/compact/index-probe helpers kept in
`index_storage_test_util`. The remaining non-test code is limited to
index probe observability needed by the tests.
Follow-up update: refine the probe observability/test framework so unit
tests assert `OlapReaderStatistics::index_probe_events` directly instead
of relying on profile timer snapshots or `last_read_index_id()`. The row
batch model is now index-neutral, synthetic reader-schema injection is
named explicitly, and expr pushdown records structured
source/path/index-id probe events.
The `bloom_filter_must_filter_data` profile/debugpoint portion in
`variant_p0/predefine/test_types_with_indexes_profile.groovy` remains
regression/bloom-filter scope rather than inverted-index storage fixture
scope. Existing BE tests cover bloom-filter reader/writer primitives
separately; this PR migrates the inverted-index storage paths. The Cloud
schema-change transaction-conflict debugpoint remains regression-scope
because it validates cloud schema-change job conflict handling rather
than BE rowset/index storage behavior.
## Summary
- add a BE CMake option for Variant NestedGroup extension modules
- let the Storage target replace the default provider with external
extension sources when the option is enabled
- let BE unit tests include matching external extension test sources
when the option is enabled
- keep the change limited to existing build files shared by the public
and private trees
This PR exposes `variant_enable_nested_group` as a public VARIANT
property and wires the related configuration through parser/type
serialization.
Main changes:
- Allow `variant_enable_nested_group` in VARIANT predefined fields.
- Disable doc mode and sparse-column related options when NestedGroup is
enabled.
- Serialize `variant_enable_nested_group` in `VariantType#toSql`.
- Add `variant_nested_group_max_depth` config and make the default
NestedGroup write provider explicitly return not-supported status when
the write path is unavailable.
- Update FE/BE tests for parser behavior, type serialization, and
disabled NestedGroup write-path handling.
Problem Summary: Inserting JSONB-producing expressions, such as
`json_object` or a `JSON` column, into a `VARIANT` column could
materialize the root JSONB through the generic JSONB-to-string cast
path. That path unescaped string contents before variant parsing, so
JSON objects containing quoted strings could be parsed incorrectly and
subpath extraction failed.
### What is changed?
Convert scalar JSONB variant roots directly back to JSON text with
`JsonbToJson::jsonb_to_json_string` before parsing into `ColumnVariant`,
instead of using the generic JSONB-to-string cast path. This preserves
embedded quotes and backslashes in object string values and lets variant
path extraction work correctly.
The regression covers:
- `INSERT INTO variant SELECT json_object(...)` with `Willie "The Lion"
Smith`, verifying stored object text and path extraction.
- `INSERT INTO variant SELECT json_column` with escaped quotes, a
Windows-style path, nested object, and array, verifying `ok`, `msg`,
`path`, `nested.x`, and array size.
The test intentionally avoids asserting `variant_type(ch)` because the
physical variant layout can be reported either as a materialized
subcolumn shape or as `__DORIS_VARIANT_DOC_VALUE__` map while preserving
the same user-visible object and path extraction behavior
Problem Summary: `COUNT(DISTINCT variant_subcolumn)` could reach BE hash
key selection and fail with a vague `INTERNAL_ERROR` when the argument
was `VARIANT`. This PR rejects `VARIANT` arguments during FE aggregate
analysis, including grouped distinct aggregate plans before
`DistinctAggregateRewriter` can split the distinct aggregate.
`COUNT(DISTINCT ...)` on `VARIANT` arguments now reports a clear
unsupported-type error instead of a BE internal error. Cast `VARIANT`
expressions to `STRING` or another supported scalar type before using
`COUNT DISTINCT`.
- Test: Unit Test / Build / Regression Test
- `./run-fe-ut.sh --run
org.apache.doris.nereids.trees.expressions.functions.agg.CountTest`
- `./build.sh --fe`
- `./run-regression-test.sh --run --conf tmp/regression-conf.auto.groovy
-d correctness_p0 -s test_array_order_by`
- `./run-regression-test.sh --run --conf tmp/regression-conf.auto.groovy
-d correctness_p0 -s test_array_string_order_by`
- `./run-regression-test.sh --run --conf tmp/regression-conf.auto.groovy
-d variant_p0 -s test_variant_count_distinct`
- `git diff --check`
- Behavior changed: Yes. `COUNT DISTINCT` on `VARIANT` now fails during
analysis with a clearer error instead of a BE `INTERNAL_ERROR`.
- Does this need documentation: No
## Proposed changes
This PR addresses Variant import memory for sparse dynamic keys.
- Parse plain dynamic non-doc Variant JSON into doc-value KV during
storage parse instead of eagerly expanding every path into parse-time
subcolumns.
- Keep the eager subcolumn parse path for cases that still depend on
parse-time path/type metadata: nested group, deprecated flatten nested,
predefined typed paths, and parent inverted index columns.
- For legacy multi-subcolumn `ColumnVariant` blocks that already reach
the segment writer, append them into a doc-value intermediate when the
writer buffer is still root-only. This avoids copying thousands of
sparse subcolumns into the writer append buffer.
- Stream large doc-value materialization sets one path at a time when
selected materialized doc paths exceed 64, instead of holding all
materialized sparse subcolumns in memory.
- Gate serialized Variant doc-value block payloads by BE exec version,
so WAL blocks written with `be_exec_version=9` replay correctly.
- Add focused Release-gated BE UT/perf coverage. The perf tests stay
skipped by default unless `DORIS_RUN_VARIANT_WRITE_PERF_TEST=1` is set.
## Problem
The downloaded WAL first block is `be_exec_version=9`. It deserializes
as a root-only Variant column:
```text
source_rows=9234 source_subcolumns=1 source_sparse_entries=0 source_doc_value_entries=0
```
So this real WAL does not reproduce the 1000-subcolumn writer-buffer
shape directly. It did expose the old-version doc-value serialization
compatibility issue, which is fixed by the version gate.
Full first-block Release memory result:
```text
cir20431_wal_variant_memory rows=9234 source_rows=9234 source_subcolumns=1 source_sparse_entries=0 source_doc_value_entries=0 source_allocated=67452928 legacy_append_allocated=67649536 doc_value_append_allocated=67518464 doc_vs_legacy=0.998062
```
## Release Writer Perf
Release BE UT build (`BUILD_TYPE_UT=Release`, `-O3 -DNDEBUG`). Best of 3
measured runs after 1 warmup. JSON generation and JSON-to-Variant parse
are excluded; the measured window covers conversion, append, finish,
data/index writes, and file close.
```text
variant_write_perf case=sparse_keys rows=8192 paths_per_row=32 max_subcolumns=2 legacy_us=156634 kv_us=28341 kv_vs_legacy=0.180938 legacy_input_allocated=148013056 kv_input_allocated=10747904 legacy_append_buffer_allocated=148013056 optimized_append_buffer_bytes=6346316 kv_append_buffer_bytes=6346316 legacy_footer_columns=4 kv_footer_columns=4 legacy_materialized=2 kv_materialized=2 legacy_sparse=1 kv_sparse=1 legacy_doc_value=0 kv_doc_value=0 legacy_file_size=245659 kv_file_size=254370
variant_write_perf case=dense_keys rows=8192 paths_per_row=32 max_subcolumns=32 legacy_us=25043 kv_us=17450 kv_vs_legacy=0.696802 legacy_input_allocated=4980736 kv_input_allocated=10747904 legacy_append_buffer_allocated=4980736 optimized_append_buffer_bytes=5816320 kv_append_buffer_bytes=5816320 legacy_footer_columns=34 kv_footer_columns=34 legacy_materialized=32 kv_materialized=32 legacy_sparse=1 kv_sparse=1 legacy_doc_value=0 kv_doc_value=0 legacy_file_size=12963 kv_file_size=12963
```
Interpretation:
- Sparse-key shape: writer append buffer drops from `148013056` bytes to
`6346316` bytes, and write time is `0.181x` legacy.
- Dense-key shape: all 32 paths are materialized, so memory is roughly
comparable (`4980736` bytes vs `5816320` bytes) and write time is
`0.697x` legacy in this Release run.
- Both cases report `legacy_doc_value=0 kv_doc_value=0`, with identical
footer column counts in each case. The non-doc path does not persist
both doc-value and materialized subcolumns; doc-value is used as an
intermediate before writer-side materialization/sparse writing.
## Testing
Current head: `e27bdc09407033191d7d9770dda1ed60d2bb55ef`
- `clang-format` on modified C++ files before commit.
- `git diff --check`
- `env BUILD_TYPE_UT=Release DORIS_RUN_VARIANT_WRITE_PERF_TEST=1
DORIS_CIR20431_WAL_FILE=/tmp/cir20431_wal/walbak/1_1778071896877_16715020621810688_group_commit_b94fdc3cd7568b18_30cfcd56d2cfeb8f
DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20
PATH=/mnt/disk1/claude-max/ldb_toolchain20/bin:$PATH ./run-be-ut.sh
--run
--filter='VariantColumnWriterReaderTest.test_legacy_subcolumn_append_as_doc_value_buffer:VariantColumnWriterReaderTest.test_storage_parse_kv_write_materialized_and_sparse:VariantColumnWriterReaderTest.test_cir20431_wal_doc_value_buffer_memory:VariantColumnWriterReaderTest.test_storage_parse_kv_write_perf'`
- `env BUILD_TYPE_UT=Release
DORIS_CIR20431_WAL_FILE=/tmp/cir20431_wal/walbak/1_1778071896877_16715020621810688_group_commit_b94fdc3cd7568b18_30cfcd56d2cfeb8f
DORIS_CIR20431_WAL_ROWS=9234
DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20
PATH=/mnt/disk1/claude-max/ldb_toolchain20/bin:$PATH ./run-be-ut.sh
--run
--filter='VariantColumnWriterReaderTest.test_cir20431_wal_doc_value_buffer_memory'`
The Release test script used its default parallelism (`PARALLEL -- 39`);
no manual `-j` was passed.
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Queries that evaluate VARIANT expressions after local
exchange can share input blocks across downstream pipeline tasks.
Variant casts and Variant serialization finalized source columns
in-place, so one consumer could mutate a shared input column while
another consumer still expected the original column shape and row count.
This made local-shuffle query results unstable and could trigger later
operators to observe changed Variant column contents or sizes. This
change confines the fix to Variant handling by using private finalized
Variant copies for cast and serialization paths instead of mutating the
source column.
### Release note
None
### Check List (For Author)
- Test:
- PATH=/tmp/doris-clang-format-bin:$PATH build-support/clang-format.sh
- git diff --check HEAD^
- ./build.sh --be
- ./run-be-ut.sh --run
--filter='ColumnVariantTest.serialize_does_not_finalize_source_column:ColumnVariantTest.block_serialize_does_not_finalize_source_column:FunctionVariantCast.CastFromVariantDoesNotFinalizeSourceColumn:FunctionVariantCast.CastFromVariant'
- Manual test: with column_nullable.cpp and column_nullable_test.cpp
reverted from this PR, local 1FE+4BE first local-shuffle repro passed
16x100 concurrent executions on BE version doris-0.0.0-5078f25a971fc
- Manual test: with column_nullable.cpp and column_nullable_test.cpp
reverted from this PR, local 1FE+4BE second local-shuffle repro matched
local-off baseline for 100 iterations on BE version
doris-0.0.0-5078f25a971fc
- Behavior changed: No
- Does this need documentation: No
Problem Summary: `DataTypeVariantSerDe::write_column_to_arrow` always
cast the Arrow builder to `arrow::StringBuilder`. During Parquet OUTFILE
export, the Arrow block converter can switch utf8 columns to
`large_utf8` when a batch is large, which gives variant serialization an
`arrow::LargeStringBuilder` and crashes BE on the bad cast.
This patch handles both `arrow::StringBuilder` and
`arrow::LargeStringBuilder` for VARIANT Arrow serialization and adds a
BE UT that reproduces the LargeStringBuilder path.
### What problem does this PR solve?
Fix variant subpath pruning for projections where the top-level
expression is an array subscript or `element_at` over a variant subpath.
The planner could leave the outer subscript on the original variant
access chain after pruning, which made valid 1-based array subscripts
return `NULL`.
The original array-of-objects repro depends on nested-group variant
semantics, so the regression in this PR uses a plain `VARIANT` array
leaf without nested group. Since that query result is already correct on
current master, the regression asserts the verbose plan instead: the
scan uses `subColPath=[items, type]` and the final array subscript is
applied to the pruned variant slot.
### Check List
- [x] Added regression test
- [x] Added FE planner unit test
### Tests
- `./run-regression-test.sh --run --conf tmp/regression-conf.auto.groovy
-d variant_p0 -s test_variant_array_subscript`
- `./run-fe-ut.sh --run
org.apache.doris.nereids.rules.rewrite.VariantPruningLogicTest` passed
earlier on the same FE code; rerun after this regression-only amend was
blocked by system pid/thread exhaustion before test execution.
- Guard `CachedRemoteFileReader::read_at_impl` against nullable
`IOContext`.
- Pass `NativeReader` `_io_ctx` through header and block reads.
- Add BE unit coverage for reading through `CachedRemoteFileReader`
without an explicit `IOContext`.
MATCH predicates fail for VARIANT dot subcolumn access
such as `cast(msg.trace_id as string)`, while the equivalent bracket
access `msg['trace_id']` works. Dot access can leave an `Alias` around
the pruned subcolumn slot, and `CheckMatchExpression` rejected the
aliased slot.
Fix MATCH predicates on VARIANT dot subcolumn access such as
`msg.trace_id` so they are accepted like equivalent bracket subcolumn
access.
## What Problem Does This PR Solve?
This PR fixes Variant inverted-index search binding for scalar Variant
paths and nested Variant paths
Before this change, Variant search had several correctness gaps around
how a logical search field was bound to the physical segment/index
structures:
- Variant subcolumns could be resolved as logical fields, but the search
path did not consistently bind them to the actual materialized subcolumn
iterator or inherited parent Variant inverted index.
- Nested Variant search needed to evaluate leaf predicates on the active
nested group, then map matching child documents back to the parent/root
document scope. Without that mapping, nested predicates could match the
wrong document scope or fail to use the intended nested-group reader
state.
- Missing or unmaterialized Variant subcolumns should behave as an empty
leaf result for that field instead of accidentally falling back to an
unrelated reader.
- `BitSetQuery` treated an empty truth bitmap as an empty scorer even
when a null bitmap was present, which dropped null-bitmap semantics
needed by Variant search predicates.
## What Changed
- Split Variant-specific inverted-index search support out of
`function_search` into `variant_inverted_index_search.{h,cpp}` so the
generic search function is smaller and Variant binding logic is
isolated.
- Extended `FieldReaderResolver` to track whether a field is actually
bound, missing in the segment, or handled through a direct Variant index
reader.
- Added Variant nested search evaluation that maps nested leaf query
results through the active nested group chain before returning
parent/root matches.
- Added leaf-query mapping hooks so nested Variant leaves can be wrapped
with the correct document mapping query.
- Updated Variant subcolumn index discovery to consider direct subcolumn
indexes and inherited parent Variant indexes using the logical path,
relative path, index suffix, and field pattern.
- Preserved null bitmap behavior in `BitSetQuery`/`BitSetWeight` when
the truth bitmap is empty but the null bitmap is not.
- Added focused diagnostics for Variant search binding/index iterator
selection through existing debug/stat paths to make future binding
failures easier to inspect without changing query results.
## Testing
Ran focused Variant-related BE unit tests:
```bash
./run-be-ut.sh --run --filter='*Variant*:FunctionSearchTest.TestBuildLeafQueryDirectUnknownClauseUsesLeafMapper:FunctionSearchNestedTest.*:BitSetQueryTest.EmptyTruthBitmapPreservesNullBitmap'
```
Stabilize the variant predefine hirachinal regression
output by serializing `v['c']` through JSON instead of relying on
variant string formatting, which can differ in insignificant array
whitespace.
- NestedGroup compaction targets normally do not carry extracted
subcolumns, so preserving existing target subcolumns can miss regular
paths that sit beside NG arrays.
- Collect Variant extended info for NG roots, keep NG physical children
owned by the NestedGroup writer, and materialize only regular non-NG
paths through the existing data-types helper. Typed paths continue to
use get_compaction_typed_columns(), while doc-mode and normal non-NG
Variant flows keep their existing ordering.
- Also record materialized regular paths in sub_path_set and update
schema util coverage for the new target-schema contract.
Problem Summary: Flat-leaf compaction/checksum mode should keep root
variant reads on the root-flat path and should not fall back to
doc-value root reconstruction. Avoid probing an empty leaf path, keep
explicit doc bucket columns on DOC_COMPACT, and update the doc-bucket
unit test to assert that root uses VariantRootColumnIterator in
flat-leaf mode.
Add NestedGroup path metadata through Variant storage and query
planning.
- Write indexed NestedGroup paths into Variant subcolumn metadata so
top-array nested fields keep positional nulls.
- Teach ColumnVariant and DataTypeVariant to carry nested-group typed
paths through sparse/subcolumn IO, casting, element access, and
compaction.
- Align FE Variant type SQL, property analysis, and expression-to-thrift
translation with NestedGroup path planning.
- Regenerate nested_in_top_array expected output for the NestedGroup
positional layout.
- Add a block fast path for explode-like table functions to expand
contiguous nested ranges without per-row get_value()
- Fall back to the original row-wise path when the nested range is
non-contiguous or unsupported
- Add/extend unit tests to cover fast path and fallback cases
Test: ./run-be-ut.sh --run
--filter='TableFunctionOperatorTest.*:UnnestTest.*'
Performance :
Simple Array: 10.1s -> 5.1s
Nested Array: 100.2s -> 20.1s
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
- cap doc-value path reserve and reuse exact unique-path counts when
available
- release doc write plans before doc-value writing to avoid overlapping
peaks
- eagerly finish and flush compact-writer subcolumns, and make
finish/write_data idempotent
- avoid merge/sort when bucket = 1
- add unit coverage for expected_unique_paths and compact-writer
roundtrip/idempotence
The blue line is the improved version
<img width="1578" height="570" alt="image"
src="https://github.com/user-attachments/assets/eb501829-627c-45e8-85a1-93c63d7148de"
/>
Problem Summary:
Nested search predicates were parsed inconsistently across code paths.
Queries inside `NESTED(path, ...)` had to repeat the full nested prefix,
unsupported nested forms were validated late, and normalized field
bindings could diverge from the field paths pushed down to thrift.
This change centralizes nested field path construction and normalizes
child predicates against the active nested path during parsing. It
applies the same validation rules in standard and lucene modes, rejects
unsupported nested forms earlier, and keeps normalized field bindings
aligned with generated thrift search params. The added FE tests cover
standard mode, lucene mode, invalid nested syntax, and thrift
serialization of normalized nested fields.
Normalize FE handling of nested search predicates for Variant search
DSL. Fields inside `NESTED(path, ...)` must now be written relative to
the nested path, and unsupported forms such as absolute nested field
references, bare queries, nested `NESTED(...)`, and non-top-level
`NESTED` clauses now fail with explicit syntax errors.
- Test: Not run in this session (message-only amend; the code change
adds FE test coverage)
- Behavior changed: Yes (nested predicates now require relative field
references inside `NESTED(path, ...)`)
- Does this need documentation: No
Build a NestedGroup streaming write plan from input rowset metadata and
use it to initialize root, regular subcolumn and nested-group writers
during compaction, so Variant data can be written chunk by chunk instead
of being fully materialized first.
Refactor shared subcolumn writer helpers, allow the NestedGroup provider
to consume prebuilt groups or a streaming plan, and skip sparse/doc
path-stat collection for nested-group variant columns.
Add tests for streaming plan construction and array subcolumn writes
across multiple batches.
When Nereids generates multi-layer projections (e.g., for expressions
like `IFNULL(a, 0) + IFNULL(b, 0)`), the intermediate projection layers
reference slots from intermediate tuples. In short-circuit (point query)
execution, ShortCircuitQueryContext serializes the projection list and
sends it to BE. However, BE evaluates projections directly on scan
output and has no knowledge of intermediate tuples, causing incorrect
results or crashes when the projection references intermediate slot IDs.
This fix adds PlanNode.getPointQueryProjectList() which flattens all
intermediate projection layers into a single-layer projection by
substituting intermediate SlotRefs with their source expressions via
ExprSubstitutionMap. ShortCircuitQueryContext now calls
getPointQueryProjectList() instead of getProjectList() to ensure BE
receives fully resolved expressions.
VARIANT root columns do not have inverted indexes built on them
directly;
only typed sub-columns extracted via `response['field']` carry per-field
indexes. Issuing `response MATCH 'xxx'` on a root VARIANT slot silently
falls through to a full scan without any index acceleration, returning
unexpected results or errors.
Add a check in `CheckMatchExpression` that inspects the resolved
SlotReference after unwinding any Cast chain. If the slot is a
VARIANT type and has no sub-column path, the planner now throws an
AnalysisException with a clear message suggesting the correct
`column['field'] MATCH 'xxx'` syntax.
Refactor `isSlotOrCastChainOnSlot` → `getSlotFromSlotOrCastChain` to
return the actual SlotReference (or null) so the caller can inspect the
slot's type and sub-column path without a second traversal.
Add unit tests (CheckMatchExpressionTest) covering:
- Root VARIANT slot → rejected
- CAST(root VARIANT) → rejected
- VARIANT sub-column slot → allowed
Add regression test (test_disable_root_variant_match) that creates a
VARIANT column with a MATCH_NAME inverted index, verifies root MATCH is
rejected with the expected error, and confirms sub-column MATCH still
returns correct results.
Point queries on unique key tables with row store incorrectly used the
default timezone (Asia/Shanghai) instead of the session timezone when
evaluating timezone-sensitive functions like from_unixtime(). This was
because PTabletKeyLookupRequest did not carry timezone information, and
the RuntimeState created in Reusable::init() always used the default
timezone.
The fix adds a time_zone field to PTabletKeyLookupRequest, sets it from
the session variable in FE PointQueryExecutor, and applies it to the
RuntimeState in BE PointQueryExecutor::init().
Changes:
- gensrc/proto: Add time_zone field to PTabletKeyLookupRequest
- FE: Send session timezone in PointQueryExecutor.getNextInternal()
- BE: Add RuntimeState::set_timezone() and use it in
PointQueryExecutor::init()
- Add regression test for point query timezone handling
# NESTED clause
Introduce NestedGroup support for tracking nested array structures in
variant columns, including storage layer definitions, core data
structures, read/write path integration, FE pruning rule enhancements,
and NESTED clause in search DSL.
# IDL definitions:
Add NestedGroupInfoPB, NestedOffsetsIndexPB in segment_v2.proto
Add ColumnPathInfo fields for nested group metadata
Add NESTED clause type and nested_path field in Exprs.thrift
Core data structures (BE):
nested_group_path.h: Path utilities for nested group identification
nested_group_provider.h/cpp: Provider interface with default no-op
implementation
Offsets mapping index and offset tracking utilities for nested arrays
Variant reader/writer integration (BE):
# Integrate NestedGroup-aware read path in variant_column_reader
Integrate NestedGroup-aware write path in variant_column_writer_impl
Add segment_iterator and segment_creator integration
Enhance path_in_data, variant_util, json_parser for nested structures
Add variant_nested_group_max_depth config
Update variant_statistics for nested group tracking
FE pruning and rewrite rules:
# Add AccessPathExpressionCollector for access path collection
Refactor NestedColumnPruning and VariantSubPathPruning logic
Enhance ExpressionAnalyzer and expression context for pruning support
Add NestedGroup array support in Explode/ExplodeOuter
Search DSL NESTED clause:
# Add NESTED token and grammar rules in SearchLexer/Parser.g4
Implement NESTED clause parsing in SearchDslParser
Implement NESTED function evaluation in function_search and vsearch
Tests:
nested_group_path_test, nested_group_path_filter_test,
nested_group_provider_test
function_search_nested_test, vsearch_expr_test updates
PruneNestedColumnTest, VariantPruningLogicTest, SearchDslParserTest (FE)
4 new regression test suites for variant pruning
### Bug
INSERT INTO SELECT with LEFT JOIN on CTE that casts variant subcolumns
(e.g., cast(content['val'] as decimal)) returns all NULLs for the
right-side
CTE fields, while a direct SELECT produces correct results.
### Root cause
In INSERT context, strict mode is enabled. When cast_from_variant_impl
clones the FunctionContext, it inherits strict mode. The variant root
column may
contain null/empty JSONB entries for rows where the subcolumn doesn't
exist
(mixed-schema variant data). In strict mode, these null entries cause
the ENTIRE
cast to fail and return all NULLs.
### Fix
Explicitly disable strict mode in the cloned FunctionContext inside
cast_from_variant_impl, since this is an internal type conversion within
variant
processing, not user-provided INSERT data validation.
### Tests:
- BE unit test: CastFromVariantStrictModeRegression (variant→Int32,
string→Int32 with strict mode enabled)
- Regression test: test_variant_cast_strict_mode (INSERT INTO SELECT
with LEFT JOIN on CTE with variant subcolumn casts, verifying results
match SELECT)
Improve VARIANT doc-mode ingestion and compaction performance, and add a
write/compaction benchmark UT.
Key changes:
- Add config `enable_variant_doc_compaction_sparse_write` (default:
true) to
enable sparse writing for materialized subcolumns during doc-mode
compaction.
- Optimize doc value column generation: deduplicate paths, sort paths
once, and
serialize values directly into the binary column to reduce
allocations/copies.
- Optimize bucket sharding/stats collection with phmap and cached
path->bucket
mapping; remove per-row doc value sorting in writer.
- Refactor VariantDocCompactWriter finalize flow with an explicit write
plan;
skip invalid array types and fix doc materialization offset handling.
- Update variant-related UTs and add VariantDocModeCompactionTest to
benchmark
segment import and doc-mode compaction.
UT Testing:
- BE UT: VariantColumnWriterReaderTest.*, VariantUtilTest.*,
ColumnVariantTest.*
Benchmark Testing(Release compile):
- BE UT (Release-only, slow): VariantDocModeCompactionTest.*
<img width="902" height="778" alt="image"
src="https://github.com/user-attachments/assets/08fb9341-2620-484d-9cf9-f3a64c95d4bd"
/>
support export doris native format file
```
// Doris Native format file-level constants.
//
// File layout (byte stream):
//
// +-------------------------------+---------------------------+---------------------------+ ...
// | File header | Data block #0 | Data block #1 | ...
// +-------------------------------+---------------------------+---------------------------+ ...
//
// File header (12 bytes total):
// - [0..7] : magic bytes "DORISN1\0" (DORIS_NATIVE_MAGIC)
// - [8..11] : uint32_t format_version (DORIS_NATIVE_FORMAT_VERSION, little-endian)
//
// Each data block i:
// - uint64_t block_size : length in bytes of serialized PBlock (little-endian)
// - uint8_t[block_size] : PBlock protobuf payload produced by Block::serialize()
//
// NativeReader:
// - Detects the optional file header by checking the first 8 bytes against DORIS_NATIVE_MAGIC.
// - If the header is present, it skips 12 bytes and then starts reading blocks as
// [uint64_t block_size][PBlock bytes]...
// - If the header is absent (legacy files), it starts reading blocks from offset 0.
//
// VNativeTransformer:
// - Writes the header once in open(), then appends each block in write() as
// [uint64_t block_size][PBlock bytes]...
//
// These constants are shared between writer, reader and tests to keep the on-disk
// format definition in a single place.
// Header layout:
// [magic bytes "DORISN1\0"][uint32_t format_version]
```
docs: https://github.com/apache/doris-website/pull/3190
- Replace the per-row seek + next_batch(1) loop in
MapFileColumnIterator::read_by_rowids with batched offset reads via
OffsetFileColumnIterator::read_by_rowids to derive key/value ranges for
the requested rowids.
- Compute per-row map sizes from offset[rowid] and offset[rowid+1],
using the page-tail next_array_item_ordinal sentinel for the last row
when rowid+1 is out of bounds.
- Skip key/value decoding for null rows by consulting a pre-fetched null
map, and add a safety check to reject non-nullable destination columns
when the underlying map reader is nullable.
- Reuse a small peek column in
OffsetFileColumnIterator::_peek_one_offset to avoid repeated temporary
column allocations when reading page sentinels.
- Add a unit test (MapReadByRowidsSkipReadingResizesDestination) to
verify that read_by_rowids honors the SKIP_READING flag and only resizes
the destination column without touching sub-iterators.
- Improve performance from ~19s to ~0.1s in the worst-case access
pattern, and
from ~6s to ~3s in the normal case.
Reduce `SegmentFooterPB.columns` bloat caused by Variant subcolumns
and enable true on-demand meta loading.
- Introduce a general external column meta (CMO) layout that can be
reused for non-Variant columns.
- Add `storage_format = V3` as “V2 + external column meta” to safely
experiment in fuzzy tests.
## BE: External column meta (CMO) layout
Segment tail layout (not to scale):
```text
+------------------------------------------------------------------+
| ... data pages ... |
| |
| Column Meta Region (per top-level col, col_id ascending) |
| [Meta[col0]] [Meta[col1]] ... [Meta[colN-1]] |
| |
| Footer (SegmentFooterPB) |
| col_meta_region_start : uint64 |
| column_meta_entries : repeated ColumnMetaEntryPB |
| - unique_id : int32 |
| - length : uint32 |
| num_columns : uint32 |
+------------------------------------------------------------------+
```
- Introduce `ExternalColMetaUtil` with:
- `parse_external_meta_pointers()` to read `col_meta_region_start`,
`column_meta_offsets_start`, `num_columns` from footer and validate
them.
- `parse_uid_to_colid_map()` to build a `uid -> col_id` map based on
`top_level_column_uids[]`.
- `is_valid_meta_slice()` to sanity-check `(pos, size)` against the
external meta region with overflow guards.
- `read_col_meta()` to read a single `ColumnMetaPB` by `col_id` from
external meta region, using index cache lane.
- `write_external_column_meta()` to:
- externalize Variant subcolumns first (via `VariantExtMetaWriter`),
then
- write a contiguous Column Meta Region, then
- write CMO entries, then
- fill footer pointers and `top_level_column_uids`, finally clear
`footer.columns()` to enable external-first loading.
- Wire this layout into `SegmentWriter` / `Segment` / `ColumnReader` so
that:
- top-level column metas can be loaded lazily via CMO,
- old inline metas are no longer required once external meta is enabled.
## BE: Variant external meta (path-based subcolumn meta)
Variant subcolumn meta layout per root `uid`:
```text
variant_meta_keys.<uid> : IndexedColumn<VARCHAR> (path)
- value index: ON (fast seek_at_or_after by path)
- ordinal index: OFF
- encoding: PREFIX_ENCODING
footer.file_meta_datas:
- key = "variant_meta_keys.<uid>" , value = IndexedColumnMetaPB bytes
```
- Introduce `VariantExtMetaWriter` to preprocess
`SegmentFooterPB.columns`:
- For each column:
- If no `column_path_info` → keep as top-level column.
- If path relative to root is empty → treat as root Variant column and
keep.
- If relative path contains `__DORIS_VARIANT_SPARSE__` (including
bucketized sparse cols like `.b{i}`) → collect as sparse subcolumns to
be embedded into the root Variant’s `children_columns`.
- Otherwise (non-sparse subcolumn) → serialize `ColumnMetaPB` and buffer
as `(relative_path, meta_bytes)` under the root’s `uid`.
- For each root `uid`, sort `(relative_path, meta_bytes)` by path and:
- write keys (paths) into `variant_meta_keys.<uid>`,
- write values (meta bytes) into `variant_meta_values.<uid>`,
- persist the two `IndexedColumnMetaPB` as `file_meta_datas` entries.
- Replace `footer.columns()` with:
- only top-level columns (including Variant roots),
- where each root has its sparse subcolumns merged into
`children_columns`.
- Add `VariantExternalMetaReader` to read externalized Variant subcolumn
metas:
- `init_from_footer()`:
- locate `variant_meta_keys.<uid>` / `variant_meta_values.<uid>` from
`footer.file_meta_datas()`,
- support legacy unsuffixed keys (`variant_meta_keys` /
`variant_meta_values`) as a fallback,
- open two `IndexedColumnReader`s and load their indexes.
- `lookup_meta_by_path(rel_path)`:
- seek on key column via value index,
- use ordinal to read the matching value (meta bytes),
- parse `ColumnMetaPB`.
- `load_all()` / `load_all_once()`:
- iterate the whole value column, parse each `ColumnMetaPB`,
- reconstruct `SubcolumnColumnMetaInfo` tree by relative path,
- accumulate `VariantStatistics` (e.g. `none_null_size`) when available.
- `has_prefix(prefix)`:
- perform `seek_at_or_after(prefix)` on keys and check
`starts_with(prefix)` to answer “does any subcolumn with this path
prefix exist?”.
- Update Variant `ColumnReader` / schema utilities to:
- load root meta via CMO,
- obtain non-sparse subcolumn metas via `VariantExternalMetaReader`,
- obtain sparse/bucketized subcolumn metas from root’s
`children_columns`,
- keep compatibility with existing Variant formats while enabling V3
behavior when external meta is present.
## FE: V3 storage format and fuzz testing hooks
- In `PropertyAnalyzer`, extend `storage_format` parsing:
- reject `"V1"` as deprecated and keep `"V2"` as default.
- accept `"V3"` and map it to `TStorageFormat.V3` (semantically “V2 +
external column meta support”).
- In `Config`:
- add `random_use_v3_storage_format` (mutable, masterOnly) to allow FE
to:
- in fuzzy / regression tests, randomly choose some OLAP tables to use
`storage_format = V3`,
- increase coverage of V3 + ext-meta paths without impacting
user-specified formats.
- In `CreateTableInfo`:
- when engine is `OLAP` and `Config.random_use_v3_storage_format ==
true`,
- if user does not explicitly set `storage_format`,
- randomly set `storage_format = "V3"` for some tables and log this
choice for observability.
## Compatibility & safety
- Existing segments without external meta remain readable:
- CMO pointers validation will fail gracefully and readers fall back to
inline footer metas.
- Variant external meta reader stays in “unavailable” state when no
`variant_meta_*` entries are present.
- All external meta slices are validated via `is_valid_meta_slice()`
with overflow checks and region bounds.
- Footer write order ensures crash-consistency: no visible CMO pointers
in footer are written before the underlying regions are fully flushed.
## Testing
- Add BE unit tests:
- `external_col_meta_util_test` to exercise pointer parsing, CMO range
validation and read-back correctness.
- extend `variant_column_writer_reader_test` to cover V3 external meta
for Variant, including sparse columns and path resolution.
- extend `schema_util_rowset_test` to ensure schema evolution and
rowset-level utilities understand external meta layout.
- Add regression tests under `variant_p0`:
- `ext_meta/test_storage_format_v2_1` and multiple
`test_variant_external_meta_*` suites (concurrent, edge cases,
integration, with sparse),
- `query_subcolumns` cases to verify subcolumn discovery, prefix checks
and query correctness on V3 tables.
- Wire FE regression configs (`Config.groovy`, load scripts) to
optionally enable `random_use_v3_storage_format` so fuzzy runs
continuously hit V3 + ext-meta paths.
This commit modified the caching logic for the Segment footer.
Previously, the Segment footer was cached in the `DATA_PAGE` cache. This
was incorrect as the footer is metadata, not data. This commit changes
it to use the more appropriate `INDEX_PAGE` cache.
Additionally, the calculation for `_meta_mem_usage` has been adjusted
for better accuracy. The footer size is no longer included in the
calculation since it's now managed by the `StoragePageCache`. The memory
estimation for column readers is also updated to use
`config::max_segment_partial_column_cache_size` for a more realistic
value.
lihangyu
