mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
917b20be83878e98f94e3ebf7efccc80346e3e50
30859
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 yiguolei
|
917b20be83 | add 4.1.5 release (#68884) 4.1.5-release 4.1.5-rc03 | ||
 Gabriel
|
1b4d357e00 |
[fix](regression) Stabilize Paimon Rust IN filter checks on branch-4.1 (#68899)
### What problem does this PR solve? The Paimon Rust reader regression expects its small integer IN runtime filter to reach the reader, but inherits `runtime_filter_max_in_num`. Fuzzy sessions can set this limit to zero, disabling the filter before it reaches Rust even with `runtime_filter_wait_infinitely=true`. The query remains correct while the positive pushdown assertion fails. Set the limit to 1024 before the pushdown checks. The existing large-set checks still use 70000, and the existing `finally` block restores the original limit. All result and profile assertions remain intact. Target: `branch-4.1`. ### Validation Ran the complete `test_paimon_rust_reader_eq_for_null` suite against a local three-BE branch-4.1 CI build with Spark 4.0, Paimon 1.4.2 and MinIO: - Original suite, normal limit: passed. - Original suite, initial `runtime_filter_max_in_num=0`: failed at the integer IN pushdown assertion; the probe reported input/applied counters of 0/0. - Patched suite, the same initial limit of 0: passed (1 suite, 0 failures, 0 skipped); the probe reported 1/1 and the suite restored the original limit to 0. - `git diff --check`: passed. ### Release note None. Regression-test-only change. ### Check List (For Author) - [x] Regression test - [x] No product behavior change - [x] No documentation update needed |
||
 Oliveira
|
71b924ac59 |
[fix](regression) stabilize external timestamp expectations on 4.1 (#68864)
### What problem does this PR solve?
Jira: DORIS-29598
External regression goldens on branch-4.1 still expect wall-clock or
string output for columns now mapped to TIMESTAMPTZ. Update OceanBase
TIMESTAMP, SQL Server DATETIMEOFFSET (including its CTAS type and
session-zone conversion), MaxCompute TIMESTAMP, and Hudi timestamp
goldens. Preserve DATETIME and TIMESTAMP_NTZ expectations. Set the
session timezone explicitly where the goldens depend on Asia/Shanghai.
HTTP Parquet TVF queries sql08/sql09/sql10 now specify
`hive.parquet.time-zone=Asia/Shanghai` to make their existing
expectations deterministic.
```diff
- instant columns: old LocalDateTime/string expectations
+ instant columns: TIMESTAMPTZ precision and session-zone expectations
wall-clock columns: existing expectations retained
```
### Validation
- Before: branch-4.1 external CI build 215484 reported timestamp output
mismatches in the five affected external suites.
- After: local `test_http_tvf` (`-g p2 -d external_table_p0/tvf -s
test_http_tvf`) passed: 1 suite, 0 failures, 0 skips, including
sql08/sql09/sql10.
- Local SQL verified SQL Server's source `2023-06-25
14:30:45.123456-07:00` renders as `2023-06-26 05:30:45.123456+08:00`,
and CTAS preserves `timestamptz(6)`.
- `git diff --check` passed. Local Doris revision: `6ab06fceef9`; this
is not an exact branch-4.1 end-to-end result.
- JDBC/Hudi fixtures and MaxCompute cloud prerequisites are unavailable
in the personal environment. Their full-suite runtime validation remains
pending community CI; their goldens were reviewed against branch-4.1
mappings and the existing CI mismatch evidence.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test (HTTP TVF locally; external suite coverage pending
CI)
- [x] Manual test (local timestamp conversion and CTAS type probe
described above)
- Behavior changed:
- [x] No production behavior change; regression expectations and
timezone setup only.
- Does this need documentation?
- [x] No.
### Merge Danger
Two-way door; regression-only changes can be reverted. Blast radius:
regression.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
|
||
 Dongyang Li
|
7f5cd6a455 |
[fix](regression) Keep HDFS array export out of ordinary P0 runs (#68838)
### What problem does this PR solve?
`test_array_export` requires HDFS for its OUTFILE and EXPORT round
trips, but its `export,external` groups contain no explicit priority.
`SuiteScript.suite()` therefore adds `p0`, so ordinary P0 runs select it
even when no test HDFS service is provisioned and fail with `Connection
refused`.
Add the explicit `p2` priority, following existing external export
suites, while retaining `export,external`. The suite is no longer
selected by `-g p0` and remains selected by the External pipeline's `-g
external`. All test statements and assertions are unchanged.
### Release note
None
### Check List (For Author)
- Test:
- [x] No need to test or manual test. Explain why:
- [x] Other reason: Group metadata only. Reviewed the priority injection
and group filtering in `SuiteScript.suite()` and
`RegressionTest.filterGroups()`; `git diff --check` passed. No new
end-to-end run was triggered. Before this change, the existing
branch-4.1 External build 215484 recorded this exact suite as SUCCESS,
with both HDFS OUTFILE and EXPORT file downloads present in its
execution log; that build still failed on other suites.
- Behavior changed:
- [x] Yes. The suite is excluded from ordinary P0 group selection and
retained in External (and explicit p2/export) selection.
- Does this need documentation?
- [x] No.
|
||
 GabrielandMingyu Chen
|
4a54c47c4e |
[fix](regression) Backport external and Kerberos fixes to branch-4.1 (#68150, #66433) (#68869)
Backport #68150 and #66433 to branch-4.1. Trino startup/query failures and unfinished HDFS Broker Load jobs could appear as empty-result mismatches. Kerberized HDFS clients lacked the DataNode SASL setting, and the lightweight Kerberos metastore lacked the Paimon fixture. Iceberg metadata write failures could expose only a self-suppression wrapper. This backport adds bounded readiness/load checks with failure diagnostics, configures HDFS transfer protection, restores the Paimon fixture before the Kerberos readiness marker, and reports the root cause of failed Iceberg metadata commits. Branch adaptation: branch-4.1 uses `IcebergMetadataOps`, so the root-cause changes are applied to its create-table, schema-update, and branch/tag commit paths. Its partition-spec update helper already reports root causes. The newer connector implementation is not introduced. Validation: - FE Checkstyle passes: `mvn -pl fe-core -am checkstyle:check`. - All six changed suites compile with Groovy 4.0.19. - Eight local control-flow scenarios pass for Trino readiness/query errors and Broker Load completion/cancellation/timeouts. - Shell syntax, Kerberos settings isolation, both container role/switch combinations, env/config rendering, and rendered XML checks pass. - An isolated harness compiles the actual adapted create-table/schema-commit methods with stubbed dependencies: the original masks the storage failure, while the backport preserves the root message, operation context, and exception cause. - `git diff --check` passes. Full FE build and external integration runs remain pending CI. Release note: Improved external regression setup and diagnostics; Iceberg metadata commit errors now expose the underlying cause. Co-authored-by: Mingyu Chen (Rayner) <morningman.cmy@gmail.com> |
||
|
Gabriel
|
707c39170b |
[fix](type) Preserve VARBINARY length in serialized blocks on branch-4.1 (#68859)
### What problem does this PR solve?
A `VARBINARY(16)` column becomes an unspecified-length `VARBINARY` after
a Block serialization round trip. Iceberg FIXED/UUID writers then reject
it with `Iceberg fixed width does not match Doris VARBINARY length:
expected 16, got -1`, including when inserting omitted columns with
write defaults.
`PTypeDesc` already preserves the declared length, but Blocks use
`PColumnMeta`, which had no length field. The receiver consequently
reconstructed every VARBINARY column with the default length of -1.
Add an optional `varbinary_length` field to `PColumnMeta`, serialize it
from `DataTypeVarbinary`, and restore it in `DataTypeFactory`. The field
defaults to -1 so metadata from older senders retains its existing
behavior. Nested and nullable types reuse the same metadata path.
Iceberg width validation remains unchanged.
### Release note
Preserve VARBINARY declared lengths across Block serialization to
prevent Iceberg FIXED/UUID write failures.
### Check List (For Author)
- Test
- [x] Unit Test
- Behavior changed:
- [x] Yes. Serialized Blocks retain VARBINARY lengths, including nested
fields.
- Does this need documentation?
- [x] No.
### Validation
New tests cover:
- Full compressed Block and protobuf wire round trips for lengths -1,
16, and 32, with nullable and non-nullable columns, NULL values, and
embedded zero bytes.
- Nested array/map/struct metadata retaining independent binary lengths.
- Legacy metadata without the new field restoring length -1.
Before the fix, the Block and nested metadata tests fail with lengths
restored as -1; the legacy test passes.
After the fix, all 17 `DataTypeVarbinaryTest.*` tests pass in an ASAN
test binary. The focused build regenerated `data.proto` and rebuilt the
affected type/Block implementations and test source, reusing existing
dependency objects.
Also verified old/new protobuf wire compatibility and passed
clang-format 16 checks on all changed C++ files.
Full distributed Iceberg regression remains for CI. Old senders still
omit lengths, so the fix does not recover information already lost by an
older BE.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
|
||
 Mryange
|
bf963dfc5f |
[fix](expr) Fix DATETIMEV2 array range tail omission (#68855)
### What problem does this PR solve? Issue Number: N/A Related PR: #68566 Problem Summary: Backport the DATETIMEV2 fix from #68566 to branch-4.1. Date-based array_range and sequence omitted valid tail values when less than one complete interval remained. Root cause: datetime_diff was used as the loop condition. Compare directly with the exclusive end, enforce the array size limit, and stop safely when the successor exceeds the date range. TIMESTAMP_NS changes are excluded because branch-4.1 does not support this type. Add DATETIMEV2 regression coverage and update existing expected results. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test - [ ] No need to test or manual test Regression cases are included. Build and tests have not been run for this backport. clang-format 16 and git diff --check passed. - Behavior changed: - [ ] No. - [x] Yes. DATETIMEV2 ranges include all valid values below the exclusive end. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label4.1.5-rc02 |
||
 Mr.Vitvara Varavithya
|
896a722a60 |
branch-4.1: [fix](parquet) Restore column ownership after interrupted reads #66780 (#68840)
### What problem does this PR solve? Issue Number: None Related PR: #66780 Problem Summary: Cherry-pick of #66780 to branch-4.1. When a Parquet scalar column is read directly into the destination column (`read_directly_into_dst_logical_column()`), `read_column_data` moves the caller's only `ColumnPtr` into `resolved_column`; if any read step returns before `convert()` moves it back (for example when the scan is stopped), the caller is left with a null column and the BE crashes. The fix restores ownership with a `Defer` on every early return, exactly as on master. Conflict resolution: on branch-4.1 `read_column_data` ends with `return _converter->convert(...)` directly, so the last hunk becomes `return convert_column();` (master has `RETURN_IF_ERROR(convert_column()); … return Status::OK();`) — same behaviour. The unit test from #66780 is carried over as a standalone file, `parquet_scalar_column_reader_test.cpp`: the file it extends on master (`parquet_column_chunk_reader_test.cpp`) and the lazy-dictionary fixtures it builds on (#66000) are not on branch-4.1, so the test brings the minimal in-memory column-chunk fixture it needs. ### Release note Fix a BE crash when a Parquet read into a nested scalar column is interrupted before the column is converted. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test: `ParquetScalarColumnReaderTest.NestedReadRestoresColumnWhenStopped` (`be/test/format/parquet/parquet_scalar_column_reader_test.cpp`) - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label |
||
 zhangstar333
|
bcea9bbba5 |
[opt](lance) add prefilter option for filter execution control (#68813)
### What problem does this PR solve? Problem Summary: doc: https://github.com/apache/doris-website/pull/4203 add prefilter:bool to control filter exectue; prefilter default is false, so will do ann search firstly, then execute filter. prefilter turn true, will execute filter firstly, then do ANN search. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> |
||
 Calvin Kirs
|
3622455da6 |
branch-4.1: [fix](auth) Add missing privilege checks for several Nereids commands- #66218 (#68857)
### What problem does this PR solve? Backport #66218 to `branch-4.1` to enforce missing privileges for Nereids commands. Adapt constraint checks to 4.1's table-owned metadata and reject changed cascade targets before mutation. ### Release note Fix missing authorization for constraint, dictionary, recycle-bin, encryption-key and cloud administration commands. ### Check List (For Author) - Test: 33 focused FE unit tests and Checkstyle passed. Ported regression suites were not run locally. - Behavior changed: Yes. Users without the required privileges receive an access-denied error. - Does this need documentation: No (backport). |
||
|
Gabriel
|
79bc8db752 |
[fix](hudi) Align preloaded Parquet format classes on branch-4.1 (#68856)
### What problem does this PR solve? Hudi JNI scans on branch-4.1 fail with `NoSuchMethodError: org.apache.parquet.format.Statistics.isSetNan_count()` after the Parquet 1.18.0 upgrade (#68557). The BE preload assembly unpacks both the explicitly pinned `parquet-format:2.12.0` and `parquet-hadoop-bundle:1.18.0`. The former supplies an older `Statistics` class, while the latter supplies a reader that calls a method absent from that class. The scanner inherits these incompatible classes from its parent classloader. Remove the redundant standalone format dependency so the format classes and readers come from the same Parquet bundle. Keep the existing Parquet version and classloader behavior. ### Release note Fix Hudi JNI scan failures caused by incompatible Parquet classes in the BE preload jar. ### Check List (For Author) - Test - [x] Manual test (details below) - Behavior changed: - [x] Yes. Hudi JNI scans can convert Parquet statistics without the missing-method error. - Does this need documentation? - [x] No. ### Validation - Assembled the unmodified preload module and reproduced the missing-method error by converting INT64 statistics. - Cleanly reassembled the modified module with `mvn -pl be-java-extensions/preload-extensions clean assembly:single@make-assembly-deps` from `fe/`. - Ran the same statistics conversion against the rebuilt jar, with Hadoop Common and JTS Core runtime dependencies on the classpath; it passed. Both `Statistics` and `ParquetMetadataConverter` loaded from the rebuilt preload jar. - Compared all 1,795 `org/apache/parquet/**/*.class` entries from the bundle against the rebuilt assembly; all matched byte for byte. - Passed `mvn -pl be-java-extensions/preload-extensions -am checkstyle:check` and checked the reactor dependency tree. The statistics conversion probe uses: ```java Statistics stats = new Statistics(); stats.setNull_count(0); ParquetMetadataConverter.fromParquetStatistics( "parquet-mr version 1.18.0", stats, PrimitiveTypeName.INT64); ``` Full Hudi integration tests were not run locally. Existing Hudi catalog, incremental, time-travel, snapshot, and schema-evolution regression suites cover the affected JNI path and should be rerun in CI. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label |
||
|
Gabriel
|
8bd2fec41d |
[fix](regression) Restore missing ES query results on branch-4.1 (#68852)
### What problem does this PR solve? Related PR: #66425 `test_es_query` fails with `Missing output block for tag 'sql01'` because its expected output lost all 18 external-table query blocks in each of the two scroll-mode passes during an output refresh. Restore the 36 missing blocks verbatim from the generated baseline before commit `78c1626259`, in their original positions. The query SQL is unchanged, and all existing catalog results, including their updated floating-point formatting, are preserved. Add a comment documenting that both query groups need results for each scroll mode. ### Release note None ### Check List (For Author) - Test: - [x] Manual test: compiled the suite with Groovy 4.0.19 and exercised the regression framework's actual `OutputUtils.iterator` using all query tags in execution order for both scroll modes. Before the fix it fails at `sql01`; after the fix it parses all 312 blocks and 1,002 rows. - [x] Verified restored blocks match the historical generated baseline byte for byte and all 156 query SQL statements are unchanged. - [x] `git diff --check` - Full Doris/Elasticsearch integration execution was not run locally. - Behavior changed: No product behavior change; restores regression result checking. - Does this need documentation: No. |
||
|
Gabriel
|
cd673b8020 |
[feature](lance) Support distance bounds in vector_search (#68815)
### What problem does this PR solve? Issue Number: Closes #66493 `vector_search()` cannot currently restrict results by distance. This adds optional `distance_lower_bound` (inclusive) and `distance_upper_bound` (exclusive) parameters for single-vector Radius/Range Search on Lance tables. The FE validates finite FLOAT bounds and their order, includes the range in EXPLAIN, and carries it through the external-search request. The BE applies the bounds through the existing pinned lance-c API after configuring the query vector, covering indexed and Flat Search splits, including appended data outside index coverage. The existing external-search schema version remains unchanged at 1. Results retain the existing table columns and FLOAT `_distance`, prefilter semantics, offset, and Top-K cap. Bounds can be bound through prepared-statement parameters and are revalidated on each execution. Multi-vector distance bounds are rejected. Bounded index search is limited to explicitly identified IVF_FLAT segments with both bounds present. Quantized or unknown index types and single-sided ranges fall back to exact fragment scans, reported as `DISTANCE_RANGE_FALLBACK` in EXPLAIN. This avoids pre-refinement quantization errors and the pinned index implementation’s finite sentinels for absent bounds, at the cost of additional scanning. Unbounded queries retain their index selection; indexed IVF_FLAT recall still depends on probed partitions. Standalone BM25 is already delivered; the distributed BM25 statistics follow-up remains tracked separately in #67435. ### Release note Lance `vector_search()` supports optional lower and upper distance bounds for single-vector radius and range queries. ### Validation - BE: rebuilt the current branch and passed all 68 `LanceTableReader*` tests under ASAN, including exact SQ threshold crossing, the finite FLT_MAX DOT endpoint, prefilter variants, and existing reader coverage. - FE: 76 tests passed across `VectorSearchTableValuedFunctionTest`, `LancePreparedSearchTest`, `LancePreparedStatementTest`, and `LanceScanNodeTest`, including repeated bound binding, invalid parameters, all index subtypes, and indexed/flat split selection. - Native reproduction: the pinned indexed paths drop both the SQ threshold-crossing row and the finite FLT_MAX DOT endpoint; exact Flat Search returns both. Added self-contained fixtures and reader coverage for these cases. - Native execution: 10 cases passed against the pinned lance-c revision, covering indexed/Flat Search, appended unindexed data, inclusive/exclusive bounds, empty results, prefiltering, and offset. - FE Checkstyle and clang-format 16 checks passed. - Added SQL regression coverage for range results, projected stored columns, filtering, offset, invalid bounds, and EXPLAIN. Its Groovy syntax compilation passed; the SQL suite has not yet run against a live Doris cluster and requires External Regression CI. ### Check List (For Author) - Test - [x] Regression test added (execution pending CI) - [x] Unit Test - [x] Manual native execution test - Behavior changed: - [x] Yes: optional distance bounds on single-vector searches; existing unbounded queries retain their behavior. - Does this need documentation? - [x] Yes: bilingual 4.x Lance Catalog documentation: https://github.com/apache/doris-website/pull/4200. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label |
||
 linrrarity
|
303dd0e52c |
branch-4.1: [fix](function) Fix overflow in quarter arithmetic (#68410) (#68764)
pick: https://github.com/apache/doris/pull/68410 |
||
  kidandu70b3
|
74d81085cd |
branch-4.1: [refactor](lance) Remove the durable Lance index job framework (#66497) (#68781)
Per the updated direction in [#66497 (comment)](https://github.com/apache/doris/issues/66497#issuecomment-6053021847), Lance index mutations move to synchronous SQL with bounded worker execution. A durable asynchronous job framework is no longer part of the initial implementation, so this change reverts it explicitly rather than leaving it disabled or maintaining two execution paths. ### Removed - the durable Lance index job state machine, manager, same-name fences, unresolved-job quotas, and replay infrastructure introduced by #67235 - the job-specific parts of #67630: durable job creation and JobId results, `SHOW LANCE INDEX JOB(S)` with its grammar and plan wiring, and the catalog DDL guards tied to unresolved jobs - the Flight SQL JobId result-schema special case for Lance index DDL, so Flight advertises the same generic DDL status schema the statement now returns - the three `lance_index_job_max_unresolved_*` quota configuration keys, the job-specific tests, the now-unreferenced job-locator resolver pair, and the orphaned dataset-locator helper ### Preserved - read-only `SHOW INDEX` (#66637) and the `lance_index_entries` inspection TVF (#66671) - the statically validated DDL surface (#67201); `enable_lance_index_mutation` remains the mutation gate - the privilege, schema, parameter, snapshot, and IF-condition validation from #67630, rewired as pure admission preflight; a mutation that passes the whole preflight now ends in the shared not-supported rejection until the synchronous implementation lands - the schema-contract and name-normalization helpers, relocated out of the removed `job` package next to their remaining callers; the replay-path target guard keeps slimmed direct coverage ### Journal and image compatibility `enable_lance_index_mutation` has defaulted to `false` since #67630 merged, and no released build exposes it enabled, so no deployment can hold persisted job records unless someone manually enabled an undocumented, gated feature on an unreleased build. The edit-log operation `OP_LANCE_INDEX_JOB_UPSERT` (= 500) keeps its number as a tombstone and is never reused; its old Text-delimited JSON payload is consumed and ignored by a minimal discard branch in the journal reader, so edit logs from such a build stay readable without restoring any framework code. The image chunk the old manager wrote is skipped through the existing deprecated-module list, so images from previous builds still load. ### Evidence Focused FE suites (`mvn test -pl fe-common,fe-core -am -Dtest='*Lance*,DorisFlightSqlSchemaTest'`): 453 tests run, 0 failures, 0 errors, 6 skipped (known environment skips), checkstyle included, BUILD SUCCESS — the Flight schema suite is now in the filter (33 tests green, with the Lance DDL cases rewritten against the generic `StatusResult` schema) and a slimmed direct test covers the retained replay-path target guard (`hasLanceIdentityKeyChange` and its admission-epoch advance). Net change: 62 files, +400/−8922. Co-authored-by: u70b3 <u70b3@users.noreply.github.com> |
||
  github-actions[bot]andRefrain
|
81db30569f |
branch-4.1: [fix](load) Avoid false quorum failures with distributed commit reports #68275 (#68818)
Cherry-picked from #68275 Co-authored-by: Refrain <refrain520cc@gmail.com> |
||
|
Calvin Kirs
|
4413477105 |
[fix](fe) Fix HTTP system forwarding on follower FE (#68827)
https://github.com/apache/doris/pull/68823 |
||
 
|
0152b043b2 |
branch-4.1: [fix](function) Reject a non-constant seed in array_shuffle #68438 (#68832)
Cherry-picked from #68438 Co-authored-by: Chenyang Sun <sunchenyang@selectdb.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
844dfd223b |
branch-4.1: [fix](function) Sum integers exactly in array_avg #68632 (#68831)
Cherry-picked from #68632 Co-authored-by: Chenyang Sun <sunchenyang@selectdb.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
 github-actions[bot]andyujun
|
8c82d0257f |
branch-4.1: [fix](regression-test) Make test_analyze independent of the SQL cache #68829 (#68843)
Cherry-picked from #68829 Co-authored-by: yujun <yujun@selectdb.com> |
||
 HappenLee
|
4926a55b6a |
[fix](be) Fix null-safe string joins in 4.1 (backport #65975) (#68809)
### What problem does this PR solve? Issue Number: None Related PR: #65975 Problem Summary: Backport #65975 to `branch-4.1`. A single-column string hash join using `<=>` can fail to match two NULL keys when expression evaluation leaves different bytes in their nested string columns. For example, joining `CONCAT(COALESCE(p.ch, ''), CAST(p.v % 5 AS STRING)) <=> c.s` should match a NULL expression result with a NULL `c.s`, but previously could miss that row depending on block composition. Normalize NULL keys to an empty `StringRef` in the serialization shared by build and probe. Real empty strings remain in ordinary hash buckets, separate from the dedicated NULL bucket. Also avoid passing default empty refs to `memcmp`, and reject unsupported `<=>` ON predicates in ASOF JOIN. Resolve the parser conflict by retaining the 4.1 UNNEST handling and changing only the ASOF equality check. Resolve the BE test conflict by adding the original NULL normalization test without importing unrelated master-only iterator tests. Adapt the existing ASOF regression to its `nereids_p0` location on 4.1. ### Release note Fix incorrect results of single-column null-safe string joins when NULL keys come from expressions. ASOF JOIN rejects unsupported null-safe equality ON predicates. ### Check List (For Author) - Test: - BE formatting: repository clang-format/check-format scripts passed with clang-format 16.0.6. - Regression: preserve the upstream expression-NULL, mixed-block, NULL-versus-empty-string, and ASOF rejection suites. The generated expected output is copied unchanged from the original commit; local cluster execution has not been run. - FE style: `mvn checkstyle:check -pl fe-core` passed (0 violations). - FE Unit Test: `NereidsParserTest#testParseAsofJoinRejectNullSafeEquality+testParseJoin` passed locally (2 tests, 0 failures/errors). - BE Unit Test: the original normalization test is preserved. Local ASAN UT execution is blocked before compilation because the prebuilt dependency bundle lacks `libpaimon_c.a`; no BE UT execution success is claimed. The missing simdutf library was built with the repository script before retrying. - CI: `run buildall` requested for this backport. - Behavior changed: Yes, correct NULL string key matching and reject unsupported ASOF ON predicates. - Does this need documentation: No. |
||
|
Dongyang LiandMingyu Chen
|
e1b3386a71 |
branch-4.1: [fix](expr) Canonicalize logical OR results (#68798)
### What problem does this PR solve? Backport #68308 / #68401 to branch-4.1. The repair is present on branch-4.1.4, but branch-4.1 at |
||
 daidai
|
fbd27a6813 |
branch-4.1: [fix](test)(dynamic-partition) fix some unstable test cases and dynamic-partition logic (#63551) (#68820)
pick #63551 |
||
 github-actions[bot]andJerry Hu
|
e2075895a1 |
branch-4.1: [fix](window) Recompute sliding frames for floating-point sum/avg #68113 (#68736)
Cherry-picked from #68113 Co-authored-by: Jerry Hu <hushenggang@selectdb.com> |
||
 github-actions[bot]andYixuan Wang
|
8cd9bf9264 |
branch-4.1: [fix](cloud) Delete versioned tablet indexes and metadata atomically #68322 (#68817)
Cherry-picked from #68322 Co-authored-by: Yixuan Wang <wangyixuan@selectdb.com> |
||
|
Gabriel
|
878a05d149 |
[fix](parquet) Handle empty filter pages and offset index fallback (#68780)
### What problem does this PR solve? Parquet V2 scans can fail when an indexed page contributes no predicate rows: its `NONE` execution kind conflicts with the preceding page. Parsing a selected page can also discard a stale OffsetIndex after callers have already calculated its read range, leading to incorrect projected rows or EOF accounting. The fix ignores zero-row fragments when merging predicate execution kinds, reconciles selected-page ranges in ordinary/fixed-width/dictionary reads, and preserves physical-value counts and nested row positions across safe sequential fallback. ### Read cost and fallback behavior - Keep indexed skips lazy after the initial data-page reconciliation. Unselected pages do not require header reads, page-cache lookups, payload reads, decompression, or repetition-level scans solely for validation. - Do not drain the remaining repetition levels of a partially consumed indexed V1 data page when advancing. Continue to check selected page headers, row boundaries, and naturally exhausted level streams. - Track whether indexed navigation skipped an unread header or an unfinished nested V1 row span. If a later accessed header invalidates the index, reject that unsafe sequential fallback instead of reusing uncertain row positions or an incomplete physical-value prefix. Safe fallback still succeeds. - Retain the metadata-only byte-coverage check before publishing optional page indexes. It adds no file I/O; gaps and uncovered suffixes still conservatively disable the index pair, including auxiliary gaps or writer padding. This preserves the normal index trust boundary: structurally accepted indexes describe unread pages. The PR does not eagerly verify every index entry against physical headers/levels and does not claim to detect every forged row span or coordinated index/header inconsistency. Comprehensive corrupt-index validation is outside this fix's scope. Changes remain confined to `format_v2` and its tests. The legacy reader under `be/src/format` is unchanged. ### Release note Fix Parquet V2 scans with empty predicate page fragments and selected-page OffsetIndex fallback while preserving lazy indexed page skipping. Reject unsafe sequential fallback after an unverified indexed prefix. ### Check List (For Author) - [x] Unit tests - [x] No new configuration or documentation requirement - Reproduced both eager-read regressions before the implementation change: a three-page sparse seek made 4 or 5 stream reads rather than 3, and advancing a header-only/partially consumed V1 page loaded or drained its levels. - All **747** related BE ASAN tests passed with `*Parquet*:FileScannerV2Test.*:TableReaderTest.*`. - Sparse-read tests cover flat ordinary/fixed-width/dictionary paths, nested value/levels-only paths, all 16 four-page V1/V2 layouts, and cache hits/misses. Native 3-page and 64-page seeks parse exactly two headers and perform exactly three stream reads, independent of the skipped page count. - Retained physical-row result comparisons across mixed page versions, nulls, repeated values, dictionary encoding, batch sizes and all nonempty subsets of a seven-row fixture. Fallback cases assert exact success or the explicitly required corruption error. - Complete Arrow-written file regressions retain coverage for empty predicate fragments, selected-page fallback, incomplete index pairs and pruning behavior. - clang-format 16 on all eight affected C++ files and `git diff --check` passed. These are deterministic work-count and correctness checks, not remote-storage throughput benchmarks. |
||
|
Gabriel
|
872d4e01f7 |
[fix](regression) Bound large Parquet map output batches (#68808)
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
The large-string-map Parquet regression fixture contains two 1 GiB
string keys in a column chunk larger than 2 GiB, with dictionary and
PLAIN data pages. Materializing both rows into one output column
requires a 4 GiB character buffer and can exceed the BE process memory
limit during ASAN regression runs.
Read this query with `batch_size=1` and disable aggregate pushdown using
a statement-local hint. Both rows still go through full Map and string
decoding, while each output batch contains only one large key. The query
and its expected result remain unchanged. This retains the oversized
Parquet column-chunk and mixed-encoding coverage; it no longer exercises
accumulating both keys in one output ColumnString.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test
- Compiled the modified suite with Groovy 4.0.19 successfully.
- `git diff --check` passed.
- Reviewed the batch-size validation and propagation to the native
Parquet reader on branch-4.1.
- The HDFS regression was not run locally: the original CI HDFS fixture
environment is not configured. ASAN end-to-end validation remains for
CI.
- Behavior changed:
- [x] No. Production code is unchanged; the existing test uses
statement-local execution settings.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
|
||
 Gavin Chou
|
7dd24f917e |
[improvement](metrics) Unify build version metric labels #68655 (#68656)
### What problem does this PR solve? Issue Number: None Related PR: #68655 Problem Summary: Backport #68655 to `branch-4.1`. FE already exposes its build identity with `version`, `major`, `minor`, `patch`, `hotfix`, and `short_hash` labels. BE and Cloud expose the same numeric version value without those labels, even though the decimal concatenation is ambiguous for some versions and does not include RC or revision information. This change adds the same fixed label schema to the existing BE and Cloud metrics while preserving all three metric names and their numeric values. The version metrics still produce one series per process, so the fixed build labels do not introduce runtime-dependent cardinality. The regression case verifies that FE, BE, and Cloud expose identical build metadata and values. ### Release note Expose consistent build-version labels on FE, BE, and Cloud version metrics. ### Check List (For Author) - Test: Added regression coverage; ran `build-support/check-format.sh`, `git diff --check`, and a focused C++ syntax check for the labeled bvar construction. - Behavior changed: Yes. Existing BE and Cloud version metric series now include fixed build-version labels; metric names and numeric values are unchanged. - Does this need documentation: No. |
||
|
Gabriel
|
bd4ce9fbc0 |
[fix](flight-sql) Support prepared query parameter binding on branch-4.1 (#68768)
### What problem does this PR solve?
Flight SQL prepared queries containing `?` fail because Prepare rejects
placeholders and `acceptPutPreparedStatementQuery` returns
`UNIMPLEMENTED`. For example, ADBC cannot bind an integer to `SELECT
CAST(? AS BIGINT)` or execute a range predicate with two bound values.
This change implements single-row typed parameter binding on branch-4.1.
Prepare analyzes explicit casts and comparison columns to advertise
parameter types and a best-effort result schema. DoPut converts Arrow
values into detached Nereids literals, and execution binds those
literals to a fresh statement context. Result schemas are reanalyzed
after binding. Values are never interpolated into SQL text.
Arrow JDBC receives typed parameter metadata and a result schema before
binding, so it can construct parameter vectors and classify SELECT
correctly. Types that cannot be inferred remain unknown; JDBC callers
should use an explicit CAST for queries such as `SELECT ?`.
Timezone-bearing timestamp uploads preserve their UTC instants as
TIMESTAMPTZ literals; timestamps without a timezone retain wall-clock
semantics. Unsupported parameter vector types are not advertised as
bindable. Invalid SQL returns `INVALID_ARGUMENT` consistently through
Prepare and GetSchema.
Failed rebinds invalidate previous values. Handle ownership, concurrent
closure, upload versions, and per-binding/session memory limits protect
retained state. Multi-row uploads and unsupported types fail explicitly.
Queries requiring forwarding to the master FE reject bound parameters
because the forwarding protocol cannot carry typed bindings.
The PR also removes an obsolete duplicate JWT version property,
preserving the effective dependency version. The target branch already
contains the dependency declaration cleanup.
### Release note
Support single-row scalar parameter binding for Flight SQL prepared
queries, including repeated execution and NULL values.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test
- Behavior changed:
- [x] Yes. Parameterized Flight SQL queries can bind and execute
supported scalar values.
- Does this need documentation?
- [x] No. This implements the existing prepared-query protocol.
Validation:
- Full FE Checkstyle passed (`mvn -B clean checkstyle:check`).
- Focused compilation of all 11 changed Java source/test files passed
with `javac --release 8` using existing FE dependency artifacts. Java
8-compatible Guava copies preserve immutable parameter snapshots.
- 23 new tests and 33 existing Flight SQL schema tests passed, including
invalid-SQL status checks over Flight RPC and connection recovery.
- The new regression suite passed against a local FE service test
harness and a real BE, including actual JDBC `PreparedStatement`
execution, result fetching, rebinding, NULL, mixed types, range
predicates, and recovery after invalid uploads. TIMESTAMPTZ predicates
are checked under UTC, Asia/Shanghai and America/New_York, including a
daylight-saving transition.
- ADBC 1.11.0 with PyArrow 23.0.1 passed 18 concurrent connections and
540 parameterized queries, with result verification plus NULL and
connection-reuse checks. A further 72 timestamp result checks cover all
four Arrow units, three session timezones, two input timezone
annotations, negative epochs and NULL values.
The local integration harness uses one FE service and one real BE; a
full multi-FE cluster run and full FE build are left to CI.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
|
||
 zy-kkk
|
da03ee7824 |
[feature](lance) Search historical versions, tags and branches with vector_search and full_text_search (#68707)
### What problem does this PR solve? Issue Number: close #68625 Related PR: #68453 (time travel, tags, branches and managed versioning for Lance table scans) Problem Summary: `vector_search()` and `full_text_search()` always searched the latest version of `main`, and their `table` argument rejected `FOR VERSION AS OF`, `@tag` and `@branch`. A search could not be reproduced against a released tag, compared across versions, or run on an experimental branch. Both TVFs now take four optional properties, with the semantics of the table syntax from #68453: | Property | Value | Table syntax | Combines with | |-|-|-|-| | `version` | positive integer | `FOR VERSION AS OF N` | `branch` | | `timestamp` | `yyyy-MM-dd HH:mm:ss[.SSS]`, session time zone | `FOR TIME AS OF` | `branch` | | `tag` | tag name | `@tag(name)` | none | | `branch` | branch name, `main` is the main chain | `@branch(name)` | `version` or `timestamp` | ```sql SELECT id, _distance FROM vector_search( "table" = "lance_catalog.db.documents", "column" = "embedding", "query_vector" = "[0.1, 0.2, 0.3]", "version" = "3") ORDER BY _distance; SELECT id, _score FROM full_text_search( "table" = "lance_catalog.db.documents", "column" = "content", "query" = "database", "branch" = "dev", "timestamp" = "2026-09-20 12:00:00") ORDER BY _score DESC; ``` How it works: - **FE.** The TVF parses the properties into a `LanceRefSelector` and loads its metadata through the same `LanceCatalogClient.readTableSnapshot` as table scans, so storage-versioned and namespace-managed tables follow the #68453 rules unchanged. Schema, search column and field id, vector type and dimension, fragments and index segments all come from the selected snapshot, and planning never reopens the dataset. The splits carry the branch directory URI and the version, so no thrift change is needed. - **Each execution resolves again.** A bound TVF is never cached, prepared statements re-analyze on every `EXECUTE`, and TVF statements skip the SQL cache, so the latest version, a future time, a branch head and a moved tag are resolved per execution and stay fixed within its plan. `CREATE TABLE AS SELECT` plans its query twice (schema, then rows), so there a moving selector can resolve twice, as it already can for a table scan with an explicit selector; pinning `version` gives an exact copy. This is documented. - **No fallback.** A snapshot that cannot be selected, or whose index files are missing, is an error; nothing falls back to the latest version. - **EXPLAIN** of a search node now shows `lanceCatalogType`, `lanceManagedVersioning` and `lanceBranch` next to `lanceVersion`, as table scans do. Planner errors that name "dataset version N" also name the branch, since branch versions overlap main's. Decisions worth calling out: - `version` takes only a number. `FOR VERSION AS OF 'x'` names a tag because SQL has one clause for both; the TVF has a `tag` property, and Lance allows tag names made of digits, which `FOR VERSION AS OF` cannot reach. - An empty selector value is an error rather than "not set", so an empty client-side variable cannot silently search the latest version of `main`. - The selector properties must be constants. Accepting `?` would leave PREPARE without a snapshot to take the result schema from. `full_text_search` now rejects `?` in any property: until now a placeholder reached it as the literal text "?", so `'query'=?` searched for "?". - Each search relation resolves its own snapshot, as each explicit selector already does for table scans. `vector_search(t) JOIN t` can read two versions if a commit lands in between; pinning both with `version` / `FOR VERSION AS OF` avoids that. This is documented. - A managed table's branch is opened by a URI Doris joins (`<table>/tree/<branch>`), which Lance does not validate on that path, so Doris now rejects names that could leave the table's `tree/` directory or change that URI: empty segments, `..`, `\`, and ASCII characters other than letters, digits, `.`, `-`, `_` and `/`, so `../other` and a percent-encoded dot segment are rejected before the branch is opened. Outside ASCII only whitespace and control characters are rejected, so no name Lance accepts is rejected, whichever Unicode version the JDK and Lance's Rust use. Other tables check a branch out through Lance, which validates the name itself. BE changes: - A search split or a two-phase row fetch without a positive version is rejected instead of opening the latest version. - Errors from opening the dataset, reading batches, preparing the FTS query context and the two-phase row fetch name the snapshot: "... at Lance dataset version N at <uri>", with the query string and any password removed from the URI. Missing index files surface at execution because Lance loads indexes lazily. - Fragment ids from the FE must exist in the opened snapshot. lance-c silently skips unknown fragment ids, which would drop rows without an error. - The profile records `LanceDatasetVersion` and `LanceDatasetUri`, the snapshot an execution actually read. EXPLAIN cannot show this for moving selectors because it resolves again. - **Behavior change for every vector search:** a split without index segments now always runs flat (`use_index=false`). The FE plans such splits as flat search, and EXPLAIN and the profile count them as flat, but Lance used to search them with the first index on the column in manifest order, which need not be the index the FE selected (for example while an index is rebuilt under a new name). Indexes the FE cannot plan with (no fragment coverage metadata, no recorded metric, or a legacy index whose details Lance cannot infer from its files, which the FE skips; `lanceVectorIndexStatus=UNKNOWN_COVERAGE` / `METRIC_MISMATCH` / `NO_MATCH`) are therefore no longer used implicitly, and such searches become exact flat searches. Results can change only where the unplanned index was approximate; latency can grow on such datasets. Legacy indexes that lack details but whose details Lance infers, such as those written by Lance 0.10 to 0.24, are planned and searched as before (`test_lance_vector_search_legacy_index`). - A vector split that carries index segments while the request says `use_index=false` is rejected. The FE plans segments only when `use_index` allows them, so it never sends such a split; the BE now takes `use_index` from the segments and would otherwise have to override either the plan or the request. Known limitations (documented): - A branch of a branch, or a branch of a shallow clone, can fail to find the index files it inherited, because Lance writers before v13 record them under the intermediate branch (lance-format/lance#9176). The search fails with an error. - After a branch is deleted and recreated under the same name, or a dataset is replaced at the same URI, version numbers repeat. Since #68615 the BE's Lance keys the index list and the row-id index by the manifest e_tag (lance-format/lance#8904, #8930), but the BE's process-wide Lance session still keys two entries by version alone, until eviction or restart. On datasets with stable row ids, the live-row mask that vector and full-text search prefilter with can come from the old data, so rows deleted in the new data can reappear or live rows can be missing (lance-format/lance#9079, open). For a manifest without an e_tag (V1 naming, or a store that returns no ETag), the index list can come from the old data, on the BE and in the FE's Lance session until the catalog is refreshed. Neither is introduced by this PR. - `SHOW INDEX` and `lance_index_entries()` keep showing the latest version of `main`. ### Release note `vector_search()` and `full_text_search()` accept `version`, `timestamp`, `tag` and `branch` to search a historical version, the version at a time, a tag or a branch of a Lance table, including REST Namespace managed tables. Vector search no longer uses an index the planner did not select, and `full_text_search()` rejects prepared-statement placeholders. ### Check List (For Author) - Test: Regression test / Unit Test - Regression: `external_table_p0/lance/test_lance_search_snapshot` (filesystem catalog) and `test_lance_rest_search_snapshot` (REST, storage-versioned and managed) on a new fixture `search_snapshot.lance`. It has seven versions (appends, a vector index, an FTS index, an append no index covers, a delete, a vector index rebuild), a tag, and a branch `dev` whose versions and fragment ids overlap main's with different rows, plus a tag at a non-latest branch version. `search_snapshot_pruned.lance` lost a version to cleanup and the index files of a tagged version. `search_snapshot_evolved.lance` adds a column and renames the vector column after its index is built, then appends data the index does not cover. The suites check rows, distances, two-phase reads (on a branch, a tag, the columns of a version before and after a rename, and a managed table through REST), EXPLAIN, full-text search across a delete, `use_index=false` with a selector, every selector error, and a managed branch named with a Unicode 16 letter (`unicode_branch.lance`, built by pylance 12.0.0) through a table scan, a search and a two-phase read. `test_lance_rest_catalog.out` and `test_lance_rest_time_travel.out` list the new REST tables. `test_lance_vector_search_legacy_index` searches a vector index written by pylance 0.18.2 without index details and checks that it is planned (`USED`, one segment) and returns the flat search's rows. All 30 suites in `external_table_p0/lance` pass locally. - FE unit tests: `LanceSearchSelectorTest` (property rules, time zones, selector forwarding), `LancePreparedSearchTest` (both TVFs pass their selector to the table; each analysis resolves it exactly once, and a moved tag is resolved again on the next `EXECUTE`; search statements bypass the SQL cache; placeholders rejected), `LanceScanNodeTest` (EXPLAIN, errors naming the branch), `LanceCatalogClientTest` (branch names, including letters newer than JDK 17's Unicode tables), and, with real datasets and locally only like the other JNI tests, `LanceSearchSnapshotTest` (historical index coverage, branches, tags, schema evolution, cleanup, a snapshot planned before later commits) and `LanceManagedVersioningTest` (managed versions, tags, branches, unfinalized heads through the search entry points). - BE unit tests in `lance_reader_test.cpp`: missing fragment ids on normal, vector and FTS splits, unpinned versions, error and profile contents, a split without index segments searching flat although an index covers it, and a split with index segments under `use_index=false` being rejected. The indexed variants of the existing multi-vector tests now pass the index segments the FE would plan. - Behavior changed: Yes. New TVF properties; vector splits without index segments always search flat; `full_text_search` rejects `?`; managed branch names are validated; the error for a time in removed history now reads "Lance cannot select the version at or before '...'" instead of "Lance cannot resolve FOR TIME AS OF '...'", because the TVF reaches it too; BE errors name the dataset version and URI. - Does this need documentation: Yes (apache/doris-website#4185) |
||
|
Gabriel
|
b8c03bfab2 |
[feature](flight) Support native Variant V2 results for ADBC on branch-4.1 (#68667)
### What problem does this PR solve? Flight SQL exposes VARIANT as UTF8, preventing ADBC clients from receiving native binary Variant V2 values. On branch-4.1, return Variant V2 through the `arrow.parquet.variant` extension by default, with `struct<metadata: binary not null, value: binary not null>` storage. SQL NULL is a null struct; Variant null is an encoded non-null value. Clients without a Variant extension implementation can access the physical struct and field metadata. Support **Variant V2 only**. Legacy Variant returns an unsupported error, including nested legacy fields, constants, all-null columns and empty results. Reject legacy types during BE Flight schema conversion and defensively in the legacy SerDe. Remove the output-format session variable and result-sink flag entirely. Users who need text can explicitly `CAST(... AS STRING)`. Keep execution, GetTables, GetSchema and prepared-statement schemas consistent, including nested Variant fields. Native Variant output does not depend on a heartbeat capability or a cluster-wide BE support check. Retain the Variant V2 type requirement and reject non-native execution schemas. Reuse the V2 binary representation and compact dictionaries per selected row, preserving physical scalar types and decimal scales without repeatedly copying unrelated dictionary keys. Keep the value helper private to `data_type_variant_v2_serde.cpp`. Native encoding accepts up to 128 nested levels. Iceberg output keeps its existing behavior. Document the V2 requirement and client decoding limitations in the Python Flight sample README; parsing JSON does not bypass the configured Variant representation. ### Release note Arrow Flight SQL / ADBC returns Variant V2 as native Arrow Variant binary by default, without a session switch. Legacy Variant is unsupported. Use an explicit SQL cast to STRING for text output. ### Validation - The native-output update passed all 46 selected Arrow/Flight tests using a focused ASAN binary; the binary conversion code is unchanged by the capability removal. - For the capability removal, regenerated the heartbeat Thrift bindings, compiled the affected FE sources and BE heartbeat translation unit, and passed all 6 focused FE tests covering native schemas, legacy/UTF8 rejection and the existing Paimon heartbeat capability. - FE Checkstyle, clang-format 16 and diff checks passed. - The updated Groovy regression script and Python sample syntax were checked in the preceding update. - A full production build and live cluster regression/ADBC execution were not rerun for this update; CI validation is requested. ### Check List (For Author) - Test - [x] Regression test updated - [x] Unit Test - Behavior changed: - [x] Yes. Variant V2 uses native binary by default; legacy Variant requires an explicit text cast. - Does this need documentation? - [x] Yes, usage, V2-only scope and client decoding limitations are documented in the Python Flight sample README. |
||
|
linrrarity
|
3d78b6b31f |
branch-4.1: [fix](be) Prevent shared quantile state mutation with TDigest COW (#68498) (#68759)
pick: https://github.com/apache/doris/pull/68498 |
||
|
github-actions[bot]andMryange
|
ea18ba1f84 |
branch-4.1: [fix](be) Defer Bloom allocation for IN runtime filters #68763 (#68789)
Cherry-picked from #68763 Co-authored-by: Mryange <yanxuecheng@selectdb.com> |
||
|
daidai
|
c8c0ddca4d |
branch-4.1: [fix](iceberg) keep identity partition values of old specs after evolving to unpartitioned(#68124) (#68259)
Cherry-picked from #68124. |
||
 
|
1512485c45 |
branch-4.1:[fix](rowid-fetch) fix TopN lazy materialization crash with duplicateprojected columns (#64372) (#67563)
### What problem does this PR solve? Problem Summary: bp #64372 ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> Co-authored-by: York Cao <52438394+Baymine@users.noreply.github.com> Co-authored-by: caokaihua1 <caokaihua1@jd.com> |
||
|
daidai
|
2f84f2fb53 |
branch-4.1: [fix](iceberg) Resolve dropped equality-delete fields in manifest-cache planning (#68433)
### What problem does this PR solve? Issue Number: None Related PR: #67479, #67687 Problem Summary: #67479 lets `DeleteFileIndex` resolve equality-delete fields that were later dropped, by looking them up in historical schemas. It covers the default planner but misses the manifest-cache planner in `IcebergScanNode`, which still passes only `specsById`. When the manifest cache is enabled, planning fails with `Cannot find field for ID N` and falls back to the SDK scan: results are correct, but each query logs a WARN and skips the cache. The master port (#67687) already covers this path. This PR applies the same one-line change to branch-4.1: pass `icebergTable.schemas()` to the index. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Manual test: with the manifest cache enabled, `EXPLAIN VERBOSE` shows `failures=0` instead of `1`, and the fallback WARN is gone. - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label |
||
|
Gabriel
|
c6db37c5be |
[fix](flight-sql) Report actual server and Arrow versions on branch-4.1 (#68770)
### What problem does this PR solve? Flight SQL GetSqlInfo always reports server version `1.0` and Arrow version `18.2.0`, so ADBC clients receive stale metadata after upgrades. Report the running FE build version and short commit hash, using the same format as FE heartbeat metadata, and read the Arrow version from the loaded Flight SQL JAR. If the Arrow manifest has no implementation version, report `unknown`. Keep the info code registered because Arrow's SqlInfoBuilder rejects explicit requests for unregistered codes. ### Release note Fix stale server and Arrow library versions returned by Flight SQL GetSqlInfo. ### Validation - Added three Flight RPC tests covering all SQL info, the server version alone, and the Arrow version alone. - All three tests reproduce the stale version bug before the fix and pass after it. - All three also pass with the Flight SQL dependency loaded from an exploded directory without its manifest; Arrow allocator debug checks are enabled. - Compiled the changed producer and tests against the existing FE dependency classpath. No full FE/BE build or deployed-cluster test was run. - FE reactor Checkstyle validation passed with zero violations. ### Check List (For Author) - [x] Unit Test - Behavior changed: - [x] Yes. Version metadata follows the running server and Arrow library. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label |
||
|
Gabriel
|
3982e0bef6 |
[fix](catalog) Validate external table partition models on branch-4.1 (#68769)
### What problem does this PR solve?
On branch-4.1, external CREATE TABLE can silently discard unsupported
partition expressions or explicit partition definitions. For example, a
Paimon table declared with `date_trunc(ts, 'day')` may be created
without partition keys.
Validate partition models through a shared external-table entry point
before descriptor conversion and remote creation. Hive, Paimon, and
MaxCompute accept column partitions; Iceberg retains its supported
transforms and schema-specific validation. Reject explicit Doris
partition definitions for these formats, preserve Elasticsearch's
existing partition checks, and reject partition clauses for engines that
do not support them.
Also remove identical duplicate Nimbus and Hadoop COS dependency
declarations inherited from the base branch. Maven 3.10 rejects these
declarations before Checkstyle can run; dependency versions and
exclusions are preserved.
### Release note
External CREATE TABLE reports errors for unsupported partition models
instead of silently ignoring them.
### Check List (For Author)
- Test
- [x] Regression test: a standalone P0 suite covers invalid models for
all external engines without external services. Catalog suites cover
Paimon, Hive, Iceberg, and MaxCompute, including inferred engines and
supported partition behavior.
- [x] Unit Test: shared analysis checks, Paimon rejection before remote
creation, and supported partition syntax.
- Behavior changed:
- [x] Yes. Unsupported external partition models fail before table
creation.
- Does this need documentation?
- [x] No.
### Validation
- New rejection tests failed before the corresponding validation
changes.
- All 38 selected FE tests pass: 24 Paimon metadata tests, nine Iceberg
table-creation tests, and five external-table partition analysis tests.
- The full `mvn clean checkstyle:check` passes with Maven 3.10.0 (the CI
version), with zero violations across all 28 modules.
- All five new or updated Groovy suites compile with Groovy 4.0.19. SQL
regression suites were not executed locally; catalog integration tests
require their configured external services.
|
||
|
Dongyang Li
|
ee1a064d07 |
[fix](ci) Backport MinIO image migration to branch-4.1 (#68777)
### What problem does this PR solve? Related PR: #67908, #67897 Problem Summary: Backport the final MinIO image migration from master to branch-4.1. The upstream `minio/minio` and `minio/mc` repositories reject pulls, preventing third-party services from starting and causing External regression to be skipped. Update all 14 applicable references in 8 files to `doristhirdpartydocker/minio` and `doristhirdpartydocker/mc`. Preserve existing release tags, startup arguments, environment variables, and branch-specific test logic. Align the two previously untagged Iceberg/Paimon sample images with the release tags selected in #67908. The master-only `docker/runtime/all-in-one/4.1/compose/cloud.yml` does not exist on branch-4.1 and is not introduced by this backport. |
||
|
Gabriel
|
508f636eff |
[improvement](lance) Upgrade prewarm dependency and remove install guard (branch-4.1) (#68698)
### What problem does this PR solve? Related issue: #68692 (M1) Related PRs: lance-format/lance-c#94, #68687 Upgrade lance-c from `cd63420bfbe27f6f0a1edcc873b9191af7d52852` to `98468344bc9d56aa7ed4a4192e844afa62b34ba1` to include the upstream synchronous index-prewarm C/C++ APIs. Refresh the retained Foyer patch without changing its added/removed implementation payload, and verify that patch application preserves the prewarm APIs and shared-session regression. Remove the Lance-specific installation fingerprint mechanism introduced by #68687: delete `lance-install.sh` and its test harness, remove the CI harness step, and restore the existing dependency build flow. An existing installation is no longer rejected or rebuilt because of a Lance revision fingerprint; missing APIs are reported by the normal compilation or linking process. This delivers the dependency portion of M1. SQL parsing, BE RPCs, and shared-session integration remain outside this PR; the parent issue stays open. ### Validation After rebasing onto the current `branch-4.1`: - Lance archive checksum verified; the Foyer patch's added/removed payload matches the target branch. - `lance-prefilter-patch-test.sh`: passed fresh extraction, cached-source reuse, re-extraction, stale-marker refresh, invalid-patch rejection, and prewarm API/regression preservation checks. - `download-thirdparty-fallback-test.sh` and `adbc-jni-config-test.sh`: passed. - Affected shell syntax, workflow YAML parsing, and `git diff --check`: passed. - Confirmed that the removed installation scripts and fingerprint symbols have no remaining references, and the build/CI scripts match their versions before the fingerprint mechanism was introduced. No FE/BE source files change. Full Doris compilation and runtime suites were not rerun locally and remain for PR CI. ### Release note Prepare the Lance dependency for index prewarm and remove Lance-specific installation fingerprint checks. No new SQL interface is exposed. ### Check List (For Author) - Test: - [x] Existing dependency/script regression tests - [x] Manual validation described above - Behavior changed: - [x] Yes. Upgrade lance-c and restore the existing dependency build flow without Lance revision fingerprint checks. - Does this need documentation? - [x] No new SQL interface; upstream API documentation is included in the dependency. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label |
||
|
Gabriel
|
7fda449b7a |
[fix](arrow) Correct year-zero DATE conversion on branch-4.1 (#68649)
### What problem does this PR solve? Related issue: #67366 Related PR: #67449 On branch-4.1, Arrow Flight SQL returns `0000-01-02` for a stored DATE of `0000-01-01`. Doris follows the MySQL calendar, which excludes year zero's leap day, while Arrow DATE uses the proleptic Gregorian calendar. Subtracting the Unix epoch day number directly shifts dates in `0000-01-01` through `0000-02-28` by one day, including elements of `ARRAY<DATE>`. Adapt the DATE portion of #67449 to branch-4.1, with credit to its author. Convert between the calendars at Arrow and ORC write boundaries and Arrow/default Parquet/ORC read boundaries. Use the same conversion for ORC statistics so pushed-down MIN/MAX agrees with row decoding. Reject unrepresentable external dates and ignore undefined payloads in null Arrow slots. The two DATE regression suites and their expected results are copied unchanged from #67449. Add its DATE unit coverage plus an explicit null-payload test and an exhaustive comparison against the civil calendar for all 3,652,424 representable dates. Also align Iceberg DATE day/bucket partition transforms with file ordinals. For ORC/Parquet, metadata is usable only when its raw DATE range proves that every ordinal is representable, including the interior of the range. Unsafe ranges fall back to decoding: strict reads retain their errors, non-strict Parquet reads retain conversion-generated NULLs, and COUNT uses materialized values when physical null counts are insufficient. ORC uses a metadata-only SARG to check projected DATE leaves per stripe: safe excluded stripes retain SARG pruning, while unsafe stripes remain readable with SDK row-group pruning disabled. ORC COUNT validates every DATE leaf of its argument in each selected stripe, and strict Parquet COUNT validates projected DATE descendants before reading only levels. Scope: DATE, including its Iceberg partition transforms. This does not include DATETIME changes or the independent legacy file scanner conversions used when `enable_file_scanner_v2=false`. V1 reader changes and mixed-version/rolling-upgrade compatibility gates are explicitly outside this PR's scope. ### Validation - Baseline: 9 of the original 10 DATE unit tests fail; the unchanged Flight regression reproduces the one-day shift. - Fixed: 250 unit tests pass under ASAN, including the complete ORC reader suite, DATE SerDe exhaustive calendar check, Iceberg partition transforms, Parquet statistics, DATE scans and COUNT tests. These compile the modified branch-4.1 sources against cached test dependencies. - The unchanged `test_date_year_zero` Flight regression passes locally, including nullable and array values. - `test_outfile_date_year_zero` passes for Parquet and ORC using the original rows and golden assertions, with S3 transport replaced by local outfile/TVF only for the local run. The committed suite retains S3 transport for CI. - Local integration runs use the same fix rebuilt into an existing ASAN/UBSAN BE build at `5bcb996a5ff`, with an existing FE. A full branch-4.1 build and the original S3 run remain for CI. - clang-format 16 passes for all affected C++ files. Source whitespace checks pass; generated golden files retain their conventional trailing blank line from #67449. - Review coverage: new tests reproduce partition mismatches, invalid-interior MIN/MAX bypass, ORC SARG/Parquet footer/page pruning inconsistencies, and Parquet IS NULL/COUNT discrepancies. All five reproducing tests fail before the review fixes. - Added a DATE-only Spark/Iceberg partitioned write/read regression, including selective day/bucket predicates and partition metadata comparison. Its Groovy compilation passes; execution requires the external CI environment. - Follow-up review: scalar/nested ORC COUNT and strict nested Parquet COUNT reproductions fail before the guards and pass afterward. ARRAY, STRUCT, MAP, all-null values, safe ranges, non-strict Parquet, and unprojected DATE siblings are covered. - External ORC regression: retain the original LZ4 file as a BE fixture and verify that ordinal `-719530` raises a data-quality error. Replace the obsolete `1900-01-01` golden result with an explicit error assertion, restoring the scanner setting afterward. The fixture test and Groovy compilation pass locally. The complete group6 suite is also validated locally with HDFS transport replaced by LOCAL TVF; the committed suite retains HDFS for CI. - Pruning review: both new tests fail before the fixes and pass afterward. Partial ORC STRUCT projection retains SARG when the invalid DATE sibling is not read; selecting that DATE still preserves its error. Multi-page Parquet tests retain the unsafe DATE page while pruning safe pages for IS NULL and range predicates, through both single-slot and multi-slot OR evaluators. - Mixed-stripe ORC coverage: the original file-wide guard prunes zero safe stripes; the fix prunes both safe excluded stripes and still reports the invalid DATE in the retained stripe. The active SDK options outlive their reader, and the cached SARG evaluator is discarded before opening an unsafe range. - External group6 coverage: the original `non_vec_orc_scanner.orc` fixture encodes `20191111` as an invalid epoch-day ordinal. Replace its obsolete `1900-01-01` golden result with an explicit data-quality error assertion and keep the original bytes as a small BE fixture; projecting other columns still succeeds. - Final local integration after the stripe fix: all three suites pass with no failures or skips (complete group6: 41 fixtures, 39 golden-result assertions and two expected errors; Flight year zero; Parquet/ORC outfile year zero). These use the incremental ASAN/UBSAN build and local transport substitutions described above. - Metadata safety review: the malformed pruned-stripe index and truncated nested DATE statistics reproduce SDK bounds assertions before the fix. Both regressions pass afterward; the DATE preflight now adds no file read calls or bytes compared with the same excluded stripe without DATE projection. - Historical-file migration: an explicit offline tool converts only confirmed legacy Doris DATE ordinals, including nested values. It publishes a separate output atomically and refuses overwrites. Tests cover year-zero boundaries, NULLs, hidden null-container payloads, nested ORC/Parquet round trips, schema metadata, timestamp precision, and failed-write cleanup. - Final follow-up validation: all 250 ASAN tests and 10 migration-tool tests pass. The final incremental production BE passes the three existing local suites plus six legacy migration/read/export/read checks, covering both ORC and Parquet. Original HDFS/S3 transport and the full branch build remain for CI as described above. ### Release note Fix year-zero DATE values being returned one day late through Arrow Flight SQL, align DATE encoding and decoding in the default Parquet/ORC paths, and keep Iceberg DATE partition values consistent with file contents. Existing files written with the former year-zero encoding cannot be distinguished automatically from correctly encoded files. Before switching their consumers to the corrected reader, use the explicit [legacy DATE migration tool and procedure](https://github.com/apache/doris/blob/082429b00f2392f8e8eabcee6e6a5a94a403e7aa/tools/legacy_date_migration/README.md) on files confirmed to use the old Doris encoding, or re-export from authoritative original data. The tool writes new ORC/Parquet files without modifying the source. Existing Iceberg files must be migrated through a new table or a supported table-format transaction that recomputes partition metadata; do not replace files in existing snapshots. The proleptic-only `0000-02-29` remains unrepresentable as a Doris DATE. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - Behavior changed: - [x] Yes, external DATE ordinals use the correct calendar; internal DATE arithmetic is unchanged. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label4.1.5-rc01 |
||
|
Gabriel
|
e233c4ad51 |
[fix](arrow-flight) Preserve nested logical type metadata (#68614)
### What problem does this PR solve?
Flight SQL attaches `doris_type` metadata to top-level LARGEINT/IP
fields, but ARRAY items, MAP keys/values and STRUCT children lose it. A
nested LARGEINT therefore arrives as an Arrow string that clients cannot
distinguish from ordinary text. JSON and VARIANT also lack the marker at
the top level.
For example, in `SELECT named_struct('number', CAST(17 AS LARGEINT),
'text', '17')`, both children currently appear as unannotated strings.
This change annotates only the LARGEINT child, allowing clients to
restore its logical type safely.
Related PR: #67530 (closed while awaiting type-mapping refactoring).
This change targets the current `branch-4.1` schema and converter
implementation.
Move schema construction behind the abstract `ArrowSchemaConvertor`
interface. Doris, Flight, legacy Flight, Parquet, Iceberg and Paimon own
their conversion policies; recursive Doris type conversion no longer
accepts protocol or metadata flags. Flight selects its compatibility
implementation once during result-sink preparation. Iceberg retains
table field IDs and schema metadata, and Paimon consumes the
writer-provided IPC schema so nested nullability and physical layouts
remain authoritative.
Build nested fields through the existing metadata helper and add
JSON/VARIANT to that helper. Preserve Arrow storage types, child names,
nullability and encoded values; in particular, MAP keys remain
non-nullable and LARGEINT retains its string encoding for the full
signed 128-bit range. Metadata is carried at every nesting depth and
survives Arrow IPC serialization. No session setting is required.
Rolling upgrades use an optional result-sink capability flag. Old FEs
omit it, so upgraded BEs return the exact legacy metadata layout for
both schema fetches and result streams. Upgraded FEs enable the flag and
fill missing markers from the planned Doris output types before
comparing and advertising schemas. Physical types, names, nullability,
dictionaries, conflicting markers and unrelated metadata still
participate in strict schema equality. Mixed old/new BE endpoints
produce the same advertised metadata in either order. An old BE's stream
can still lack the markers advertised by the upgraded FE.
PyArrow does not automatically interpret custom field metadata.
Applications can use `doris_type=LARGEINT` to convert the corresponding
values to integers without guessing from their text.
### Release note
Preserve Doris logical type metadata for LARGEINT, IPV4, IPV6, JSON and
VARIANT inside Arrow arrays, maps and structs. JSON and VARIANT now also
include the marker at the top level.
### Check List (For Author)
- Test:
- BE ASAN: all 84 tests passed with `bash run-be-ut.sh --run
--filter='*MetadataTest*:ArrowSchemaConvertorTest.*:ArrowBlockConvertorTest.*:ArrowSchemaUtilTest.*:DataTypeSerDeArrowTest.*:VParquetWriterTest.*:ArrowFlightTimestampTest.*'
-j 48`. Coverage includes metadata compatibility, independent converter
policies, table schemas and field IDs, Arrow SerDe, timestamp bindings,
and Parquet writer round trips.
- The new polymorphic schema-converter test verifies that nested
timestamp and LARGEINT metadata policies stay independent across Doris,
Flight and legacy Flight instances.
- Before the fix, 8 of the 9 new tests failed on missing metadata; the
native-type negative control passed.
- Coverage includes all five logical types at the root and in
ARRAY/STRUCT/MAP, LARGEINT map keys, deep nesting, names/nullability,
signed 128-bit extremes, NULL values, and Arrow IPC round trips.
- FE: 13 tests passed (5 new compatibility tests and 8 existing Arrow
schema helper tests). Tests cover mixed-version schema fetches in both
orders, all-old/all-new endpoints, strict rejection of physical and
unrelated metadata differences, and result-sink capability negotiation.
The pre-fix FE fails the mixed-version and all-old metadata cases.
- Legacy BE schema tests cover all five logical types, LARGEINT map
keys, and IPC schema round trips. Populated LARGEINT batches also run
through the legacy schema path to verify values, NULLs, and IPC remain
compatible.
- FE Checkstyle, clang-format 16 and `git diff --check` passed.
- The optional ADBC integration script has five independent cases for
LARGEINT, IPV4, IPV6, JSON and VARIANT. JSON/VARIANT use ARRAY_REPEAT to
support nested values in the default mode.
- A temporary FE analyzer test checks all five exact sample queries and
asserts ARRAY<JSON>/ARRAY<VARIANT> result types. The original combined
query fails analysis; the updated queries pass.
- Python syntax/import checks passed; all five live ADBC tests were
skipped because no cluster URI was provided. No live-cluster result is
claimed.
- Behavior changed: additional Arrow field metadata only; physical types
and values remain compatible.
- Documentation: describe nested metadata and client conversion in the
Python Flight SQL sample README.
|
||
|
Gavin Chou
|
6217a2c618 |
branch-4.1: [fix](be) Stop background workers before startup exit #68652 (#68653)
### What problem does this PR solve? Issue Number: None Related PR: #68652 Problem Summary: This PR backports the startup-failure lifecycle fix from #68652 to `branch-4.1`. When a BE service fails to start with `enable_graceful_exit_check=true`, the startup failure path calls `exit(1)` while StorageEngine background workers are still running. Process-wide static destruction can then race with a path GC worker that is using the function-local static RE2 matcher in `TabletManager::get_tablet_id_and_schema_hash_from_path()`, causing a null-address crash in `re2::Prog::SearchOnePass`. The fix stops and joins created server starters, releases the backend service, and explicitly destroys `ExecEnv` before calling `exit(1)`. The immediate `_exit(1)` path remains unchanged when graceful exit checking is disabled. The backport has the same stable patch-id as #68652. ### Release note None ### Check List (For Author) - Test: - [x] Manual test - Targeted clang-format 16 check passed for `be/src/service/doris_main.cpp` on `branch-4.1`. - A local BE build could not start because the macOS environment does not have Homebrew; `run buildall` is requested below for full build validation. - Behavior changed: - [ ] Yes - [x] No. Only abnormal BE startup cleanup ordering changes. - Does this need documentation? - [x] No - [ ] Yes |
||
|
Gabriel
|
b9dc93d45e |
[improvement](lance) Push down array membership and boolean scalar index predicates (#68687)
### What problem does this PR solve? Ordinary Lance scans currently retain array label membership predicates in Doris, and scalar index planning misses usable boolean drivers. This adds Substrait pushdown for built-in `array_contains` on a direct `List<Utf8>` column with a non-NULL constant string, and for nonempty constant-string `arrays_overlap` (including the normal Nereids rewrite of three or more membership disjuncts). Positive indexable boolean conditions can select a LabelList/BTree/Bitmap segment. OR requires usable drivers on both branches of the same selected index. Literal starts_with prefixes and LIKE filters with a usable leading prefix remain eligible for BTree planning. OR branches requiring a refine filter retain fragment scans. Missing FE field/index metadata and competing logical indexes on one column preserve native automatic index selection and fragment ownership. For example, two label conditions joined by AND now reach Lance together and can reduce the index candidate set before row materialization. NULL needles, other array types, dynamic operands, UDFs, and `array_contains_all` remain residual. Doris `array_contains_all` tests a contiguous ordered subsequence, so translating it to Lance set containment would change results. Each task still selects one logical index; this change does not implement multi-column index intersection. Fragment coverage, residual evaluation, and limit handling are preserved. Complement-only predicates and expressions exceeding the pinned native node/depth budget use parallel fragment scans with scalar-index use disabled. An independent positive condition retains one search per index segment even when another conjunct is a complement. This avoids repeated segment-wide searches and prevents large overlap or unindexable OR filters from serializing fallback scans. Dependency: [lance-format/lance-c#93](https://github.com/lance-format/lance-c/pull/93), merged in upstream main at `cd63420bfbe27f6f0a1edcc873b9191af7d52852`, which is now the pinned dependency. The existing Foyer payload is reapplied without implementation changes because [lance-c#73](https://github.com/lance-format/lance-c/pull/73) is still open and supplies cache APIs used by Doris. Remove that remaining patch after #73 merges. The build now records and validates the installed Lance source/archive/patch fingerprint, detects incomplete publication, and refuses stale external build output. Before removing an existing installation, it verifies required rebuild inputs and rejects external definitions that differ from the checkout. The third-party CI script job runs the installation harness under the existing workflow path filters. These third-party/build changes are synchronized to #68689 for master. This PR targets `branch-4.1`. Documentation: https://github.com/apache/doris-website/pull/4184 (English and Chinese; draft pending implementation). ### Validation - 70 focused FE JUnit tests passed after compiling the changed converter/planner/scan sources and tests against an existing FE dependency build. Coverage includes metadata fallbacks, competing same-column indexes in both metadata orders, balanced wide conjunctions, short IN expansion, literal/refined prefixes, nested OR guards, and existing complement/overlap limits. The new regression assertions failed before the fixes. - FE Checkstyle passed with zero violations; the updated Groovy regression suite passed syntax validation. - Executed 12 plans produced by the actual FE converter and scan planner through 30 pinned lance-c scans. Verified result rows and index counters for unavailable field IDs, omitted index metadata, unequal same-column index coverage, a 34-predicate conjunction, 62/63-label overlap combined with two-value IN, literal prefixes, refined LIKE, and direct/nested OR fallback. No segment fallback occurred; metadata/ambiguity cases retained native index loads. Previous integration also covered fully indexed, partially indexed, and unindexed datasets. - The installation harness passed on master and branch-4.1 using independent external source definitions: legacy/matching installs, missing helper/vars/patch/downloader/builder, mismatched and synchronized pin/patch updates, incomplete artifacts, stale builder output, interrupted publication, and retry. Rejected preflight cases verify that the entire installation and builder invocation count remain unchanged. The missing-helper regression failed before the fix. Shell syntax, workflow YAML, and CI path-filter checks passed. - The unchanged pinned upstream main plus Foyer dependency previously passed 75 Rust unit tests, 376 C API tests, and all 3 native C/C++/static OSS consumer tests. Third-party extraction/cache-refresh/patch-failure checks also passed on both dependency definitions. - SQL regressions assert pushed/residual EXPLAIN output, fragment versus segment grouping, actual index searches/candidates/fallbacks, partial coverage, 64/65-label boundaries, mixed complements, and selective LIMIT. Full Doris compilation and distributed SQL execution on this revision are pending CI. ### Release note Support ordinary Lance array-label membership pushdown and safe boolean scalar index planning. ### Check List (For Author) - Test - [x] Regression test added (SQL execution pending CI) - [x] Unit Test - [x] Manual integration test (FE-produced Substrait through lance-c, described above) - Behavior changed: - [x] Yes. Compatible label predicates are evaluated in Lance; eligible positive boolean drivers use scalar segments, while broad complements and unsupported or oversized expressions retain fragment parallelism. - Does this need documentation? - [x] Yes: https://github.com/apache/doris-website/pull/4184 ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Confirm the upstream dependency is ready |
||
|
github-actions[bot]andJerry Hu
|
8d664b3c93 |
branch-4.1: [fix](jsonb) Stop truncating JSON text at a U+0000 inside a string #68244 (#68735)
Cherry-picked from #68244 Co-authored-by: Jerry Hu <hushenggang@selectdb.com> |
||
|
linrrarity
|
cdf997970c |
branch-4.1: [fix](array-func) Handle outer-null containers with hidden nested data (#68022) (#68696)
pick: https://github.com/apache/doris/pull/68022 |
||
 github-actions[bot]andXin Liao
|
6c9f6c7845 |
branch-4.1: [fix](storage) Do not cache the failure of loading segment rows in BetaRowset #68675 (#68683)
Cherry-picked from #68675 Co-authored-by: Xin Liao <liaoxin@selectdb.com> |
||
|
Gabriel
|
99ae82ac36 |
[improvement](lance) Expose query parallelism and complete row fetch profiles (branch-4.1) (#68690)
### What problem does this PR solve?
Lance vector searches cannot configure the SDK's query parallelism from
Doris, and second-phase row-ID fetch profiles omit available read
counters. Materialization also nests two timers for the same `ExecTime`
counter, counting the synchronous fetch wait twice.
This PR targets `branch-4.1` and:
- Adds optional `vector_search("query_parallelism" = "4", ...)` support
through FE validation, Thrift, and the existing Lance-C API. Accepts
`-1`, `0`, and positive integers; omitting it preserves the SDK default.
EXPLAIN shows an explicit setting.
- Carries row-ID fetch call counts, returned row counts, and per-dataset
Foyer cache/origin logical read bytes through the fetch RPC into each
`RowIDFetcher` profile, preserving their units. Statistics are captured
after the reader closes.
- Removes the nested materialization execution timer; the operator
framework continues to count the fetch wait once.
The read-byte counters exclude metadata/index IO and block-alignment
amplification. The current take API does not expose physical request
counts or a separate decode timer; zero Foyer counters on bypassed paths
do not mean zero physical IO. These boundaries are documented in
`docs/lance-ann-profile.md`.
No third-party dependency or patch changes are required.
### Validation
- 43 focused FE tests passed, including option bounds, omitted defaults,
Thrift round trips, and EXPLAIN propagation.
- FE Checkstyle passed with zero violations.
- Clang-format 16 checks passed for every changed C++ source/header.
- C++ syntax checks passed for all changed translation units and both
affected BE test files using regenerated Thrift definitions.
- Groovy compilation passed for the changed regression suite.
- Added BE tests for parameter validation, result preservation,
row-fetch counters, RPC profile propagation, and avoiding duplicate
execution timing.
- Added external regression coverage comparing indexed results across
parallelism settings and rejecting invalid values. Full BE UT and
external regression execution remain for CI; local C++ checks were
syntax-only.
### Release note
Add configurable Lance vector query parallelism and row-ID fetch read
counters, and correct materialization execution-time accounting.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- Behavior changed:
- [x] Yes. Explicit query parallelism is now accepted; existing defaults
are preserved. Materialization profiles no longer double-count fetch
execution time.
- Does this need documentation?
- [x] Yes. Updated `docs/lance-ann-profile.md` in this PR.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
|
||
|
Gabriel
|
78aa9c996a |
[improvement](lance) Add ANN prefilter optimization and profiles (branch-4.1) (#68615)
Unfiltered ANN searches scoped to complete index segments can materialize redundant row-ID allowlists, and the existing scan profile leaves the remaining search work unexplained. Update the upstream dependency and expose ANN stage timings on `branch-4.1`. This includes the dependency integration from #68613: adopt the merged lance-format/lance#9599 and lance-format/lance#9602 through lance-format/lance-c#89, retain the merged lance-format/lance#9537 PQ scoring fix, remove superseded local lance-c patches, and retain only the Foyer cache integration. Add BE profile counters for index opening, partition loading/preparation, prefilter readiness, CPU queue wait, partition search, query lookup-table preparation, fused distance/TopK work, and result materialization. Export operator baselines for ANN, sort/merge, take, and vector-distance work. Extend the existing indexed multivector regression to check that stage timers reach the Doris profile, alongside existing result and prefilter assertions. Timings accumulate across concurrent work and overlap parent stages. Distance/TopK includes candidate filtering in fused paths; these counters must not be summed to reconstruct wall time. `docs/lance-ann-profile.md` documents the boundaries, supported paths, and relationship to the existing scanner and prefilter timers. The BE changes add observability; the prefilter optimization itself remains in the upstream dependency. Pin lance-c `9bd730add2ac70316c1d642b8459011e2dd92022`, using the merged Lance #9602 revision `68c12dfd7efe02f90ad2d7f3a239a7eb64884e57`. This includes parallel-path partition-preparation timing. The upstream callback regression checks timer presence and nanosecond units while accepting valid zero durations. The Foyer wrapper caches immutable ObjectMeta and Attributes independently of HTTP response extensions, avoiding repeated HEAD requests on warm range reads. It does not replay transport extensions from cached metadata. The HTTP regression checks HEAD/GET counts, returned bytes/ranges/metadata, fresh dataset scopes, and explicit HEAD bypass. Generate the retained Foyer patch from source commit `24c7ca4bcb9422c113b0d3e07e4efe1173b0bc9f` relative to the pinned lance-c baseline. The header records both revisions and the regeneration command. The source is submitted as https://github.com/zhangstar333/lance-c/pull/3 against the branch behind lance-format/lance-c#73, on top of merged source-alignment PR #2. Doris consumes the exact reviewed source revision while that supplementary PR awaits merge. Scope metadata, size, and block keys to a random namespace for each live underlying object-store instance. The wrapping API omits a complete stable backend identity; identical bucket/path names alone cannot distinguish S3-compatible endpoints. Weak identity records preserve sharing for the same live store without retaining it, while new stores and process restarts start cold. A fresh Dataset open that creates a new store consequently needs to warm its own cache. This deliberately reduces reuse across instances to prevent returning another origin's data; it does not claim unchanged cache-hit rates or production latency. Avoid inserting an unchanged size entry into the WriteOnInsertion hybrid cache: look up both tiers first and populate only absent or invalid size records. Regression tests cover real HTTP endpoints sharing bucket/path/ETag, batched data and NotFound isolation, replaced origins after disk recovery, weak-reference lifetime, and actual disk-write bytes. Five warm range reads wrote 40 KB before the fix and zero bytes after it in the regression. Fingerprint the patch in the downloader so existing source trees refresh after updates, including legacy empty markers. Identical patches reuse cached sources. Check platform definitions under nounset with simulated Darwin x86_64/arm64, and make the optional ADBC source guard safe when unset on master. Downloader lifecycle and platform handling remain downstream in Doris. Validation: 461 regular Rust tests passed on the final source; Rust formatting and diff checks passed. GNU patch and git apply checks passed, and all 87 tracked source files match the recorded source commit. Downloader tests passed on master, branch-4.1, and the downstream hotfix branch for fresh extraction, idempotence, re-extraction, generic markers, legacy/mismatched Foyer markers, and patch failure. Shell syntax and simulated macOS initialization passed. All three opt-in native consumer tests passed: C calls, C++ calls, and static OSS HTTP transport. Full Doris builds and BE integration execution remain in PR CI. |
||
|
github-actions[bot]andCalvin Kirs
|
08fc786e28 |
branch-4.1: [fix](auth) Reject privilege object names with more than three parts in GRANT/REVOKE #68660 (#68682)
Cherry-picked from #68660 Co-authored-by: Calvin Kirs <guoqiang@selectdb.com> |