mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
master
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 linrrarity
|
7acfd9bfc2 |
[improvement](be) Reduce transitive dependencies in shared headers (#69717)
Related PR: https://github.com/apache/doris/issues/66901 ### What problem does this PR solve? Several shared BE headers transitively include protobuf, Thrift, thread-pool, and storage-reader implementations. Translation units that only need declarations must still parse these definitions, increasing compilation work and the rebuild scope when those headers change. Recent CI builds failed during CMake configuration because the reverse-reach counts for threadpool.h and rle_encoding.h exceeded the build-hygiene limits. This PR audits all 32 header-budget entries and reduces unnecessary dependencies without changing the budgets or exception lists. The changes: - Extract `ColumnIteratorOptions` into a dedicated header so `vexpr_context.h` does not require the full `column_reader.h` dependency chain. - Use forward declarations for `thread-pool`, `protobuf`, `cache`, and `WorkloadGroup `types where complete definitions are unnecessary. - Move `UniqueId` conversions, `protobuf` methods in numeric `SerDe`, and `selected construction/destruction` operations into `.cpp` files. - Extract `SyncSizeCallback` into a dedicated header to reduce RPC dependencies propagated through the runtime-filter interface. - Add direct includes at actual use sites and verify independent compilation. Object layouts, move semantics, explicit template instantiations, and per-value decoding implementations are preserved. Moving some definitions into .cpp files changes their visibility for inlining in ordinary separate compilation. Query runtime performance has not been measured in this PR. ### Dependency reductions The following counts come from Ninja dependencies after completed builds, covering 1,531 production translation units and 52 generated translation units: | Header | TUs Before | TUs Now | New Limit | |---|---:|---:|---:| | util/rle_encoding.h | 846 | 231 | 250 | | util/threadpool.h | 1,056 | 585 | 600 | | gen_cpp/types.pb.h | 1,862 | 1,225 | 1,250 | | gen_cpp/segment_v2.pb.h | 1,231 | 1,156 | 1,200 | | gen_cpp/internal_service.pb.h | 777 | 557 | 600 | | gen_cpp/FrontendService_types.h | 593 | 293 | 300 | | gen_cpp/cloud.pb.h | 325 | 30 | 50 | | gen_cpp/BackendService_types.h | 195 | 115 | 150 | | gen_cpp/data.pb.h | 228 | 90 | 100 | | storage/options.h | 502 | 293 | 350 | | io/fs/s3_file_system.h | 65 | 31 | 50 | | gen_cpp/Types_types.h | 2,078 | 2,073 | 2,100 | | gen_cpp/PaloInternalService_types.h | 1,093 | 1,093 | 1,100 | | util/brpc_closure.h | 62 | 35 | 50 | | gen_cpp/Status_types.h | 37 | 37 | 41 | | runtime/workload_group/workload_group.h | 43 | 38 | 47 | Across the 16 monitored heavy headers, measured header-to-TU dependency pairs decrease from 8,108 to 6,617, an 18.39% reduction. These removals affect 742 distinct translation units. The number of production compilation units remains unchanged. The transitive project-header closure of dependency.h decreases from 325 to 246 headers, a 24.31% reduction. This uses the hygiene script’s static include graph and is reported separately from the measured Ninja dependency counts. ### Query runtime performance ### TPC-H SF1000 | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 59.615 | -9.93% | | master 2026-10-13 | 60.776 | -11.65% | | master 2026-10-15 | 57.021 | -5.83% | | master 2026-10-20 | 55.877 | -3.90% | | master 2026-10-27 | 65.987 | -18.62% | | PR #69717 | 53.697 | — | ### SSB SF1000 | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 11.418 | +3.71% | | master 2026-10-13 | 11.749 | +0.79% | | master 2026-10-15 | 10.653 | +11.16% | | master 2026-10-20 | 10.773 | +9.92% | | master 2026-10-27 | 11.471 | +3.23% | | PR #69717 | 11.842 | — | ### ClickBench Configuration: cluster_type=single, ecs.c9i.4xlarge_16c32g, one 500 GB disk, full analyze, PR #69717 release package. | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 24.717 | -1.83% | | master 2026-10-13 | 26.406 | -8.11% | | master 2026-10-15 | 25.079 | -3.25% | | master 2026-10-20 | 24.629 | -1.48% | | master 2026-10-27 | 23.643 | +2.63% | | PR #69717 single rerun | 24.265 | — | ### TPC-DS SF1000 (1 TB) Comparisons use the PR result from 2026-10-29. The 2026-10-28 rerun is listed separately. | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 168.926 | -7.40% | | master 2026-10-13 | 170.676 | -8.35% | | master 2026-10-15 | 163.170 | -4.13% | | master 2026-10-20 | 163.926 | -4.58% | | master 2026-10-27 | 195.853 | -20.13% | | PR #69717 2026-10-29 main result | 156.423 | — | ### Release note None ### Check List (For Author) - Test: Three complete Release builds; standard ASAN BE/FE build and final incremental build; all BE UT sources compiled and linked; 29 selected unit tests passed with -j50 --coverage; 1,024-row service read, aggregate, shuffle join, sort and cast checks; header hygiene and changed-line format checks passed. Clang-tidy covered all 71 files and retained two existing warnings in relocated protobuf methods (function size and signedness). - Behavior changed: No - Does this need documentation: No |
||
|
linrrarity
|
e8e44449f1 |
[improvement](be) Add optional structured output for batched AI functions (#69302)
### What problem does this PR solve? doc: https://github.com/apache/doris-website/pull/4442 Problem Summary: Improve batched AI function output reliability by adding optional structured output constraints and validating the returned results. Reuse static format definitions and preserve embedded NUL characters when parsing responses. Add the resource property `ai.enable_structured_output` (default: `true`). When disabled, requests remain batched and rely on the system prompt without provider format fields. For example, OpenAI Chat Completions requests include: ```json { "response_format": { "type": "json_schema", "json_schema": { "name": "ai_batch_result", "strict": true, "schema": { "type": "object", "properties": {"results": {"type": "array", "items": {"type": "string"}}}, "required": ["results"], "additionalProperties": false } } } } ``` ### Release note Add ai.enable_structured_output (default true). Disabling it omits provider format fields while retaining batching, prompts and result validation. |
||
|
linrrarity
|
6e5a8929fb |
[fix](nereids) Fold UTC_DATE and UTC_TIME in FE (#69393)
### What problem does this PR solve? Issue Number: close #69368 Related PR: #57347 Problem Summary: `UTC_DATE()` and `UTC_TIME()` lack FE constant folding support, preventing partition pruning for predicates such as `ts >= UTC_DATE()`. ### Release note Add FE constant folding for `UTC_DATE()` and both overloads of `UTC_TIME()`, using the statement start time in UTC. |
||
|
linrrarity
|
cc8a24a13e |
[improvement](profile) Add query-wide AI function request and token statistics (#69046)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
AI function queries lack visibility into request counts, retries,
failures, latency, and token consumption.
This PR collects shared per-fragment statistics for scalar AI functions,
text/multimodal EMBED, and AI_AGG. Query profiles expose an AISummary
section aggregating the latest reports across backends, fragments, and
query execution attempts, while retaining fragment-level details.
Repeated reports do not double-count statistics.
Token usage includes provider-reported values from all HTTP attempts,
including retries and error responses. Coverage counters distinguish
missing usage from explicitly reported zero usage
```text
AIFunctionSummary:
- AIFunctionCancelledCount: 0
- AIFunctionCompletionTokens: 3.615K (3615)
- AIFunctionCompletionUsageCount: 8
- AIFunctionDnsTime: 20.216ms
- AIFunctionErrorCount: 0
- AIFunctionFirstAttemptCount: 8
- AIFunctionFirstByteTime: 1sec487ms
- AIFunctionHttpTime: 22sec896ms
- AIFunctionInputRows: 10
- AIFunctionLogicalRequestCount: 8
- AIFunctionNewConnectionCount: 8
- AIFunctionParseErrorCount: 0
- AIFunctionParseTime: 1.698ms
- AIFunctionPromptTokens: 1.704K (1704)
- AIFunctionPromptUsageCount: 8
- AIFunctionRequestCount: 8
- AIFunctionRetryCount: 0
- AIFunctionRetryWaitTime: 0ns
- AIFunctionSkippedNullRows: 0
- AIFunctionTcpConnectTime: 13.898ms
- AIFunctionTimeoutCount: 0
- AIFunctionTlsHandshakeTime: 166.740ms
- AIFunctionTotalTokens: 5.319K (5319)
- AIFunctionTotalUsageCount: 8
- AIFunctionUsageParseTime: 903.169us
```
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
|
||
|
linrrarity
|
f6359ae806 |
[Enhance](thirdparty) Enable phmap intrinsic int128 support on Clang … (#69144)
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: The bundled parallel-hashmap 1.3.8 disables intrinsic 128-bit integer support for Clang on AArch64. This selects the fallback 64-bit hash mixing path, contributing to slow hash-set lookups on ARM. Backport the ARM detection fix from upstream parallel-hashmap PR [#290:](https://github.com/greg7mdp/parallel-hashmap/pull/290) ### Release note |
||
|
linrrarity
|
25a15f932a |
[Fix](test) Isolate profile manager in cloud insert retry tests (#69015)
### What problem does this PR solve? Related PR: #68658 Problem Summary: CloudInsertProfileRetryTest uses the ProfileManager singleton, which starts a periodic storage thread. After an INSERT retry succeeds, this thread can persist the finished profile and clear its execution profiles before the test assertions run, causing intermittent failures. Use a dedicated ProfileManager without starting its periodic thread and redirect singleton access through a scoped static mock. Preserve the explicit background storage probe to verify that profiles remain ineligible for storage while the outer retry decision is pending. |
||
|
linrrarity
|
70f7dcbe11 |
[improvement](be) Reuse HTTP connections across AI function requests (#68816)
### What problem does this PR solve? Issue Number: None Problem Summary: AI scalar functions, EMBED and AI\_AGG recreate HTTP clients for each request sequence, losing connection reuse between batches and Blocks. This PR introduces `AIRequestExecutor` with exclusive client borrowing and a bounded idle cache. Scalar functions retain the cache in their execution contexts, while AI\_AGG shares it across local groups, flushes and resets. It also clears responses between attempts, detaches cancellation callbacks before returning clients, sets JSON headers for LOCAL providers without an API key, and fixes `max_retries` to count retries after the initial request. Unity compilation issues are fixed by adding missing protobuf and RuntimeState includes and locally suppressing CLucene conversion warnings, without changing CMakeLists. ### Release note AI functions reuse HTTP connections across batches and Blocks. `max_retries=0` performs one request; `max_retries=3` allows up to four attempts |
||
|
linrrarity
|
04da0ca662 |
[fix](profile) Fix execution profile leak when dispatch fails before profile registration (#68658)
### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: An ExecutionProfile is registered before query dispatch. If dispatch fails before the first `updateProfile(false)` call, the corresponding Profile has not been added to `profileIdToProfileMap`. When cleanup is invoked, the previous `removeProfile` implementation removed execution profiles only after finding a historical Profile entry. If no such entry existed, the ExecutionProfile remained in `queryIdToExecutionProfiles` and continued to retain query state in FE memory. This was reproduced with a UDF JAR download failure: three failed queries left three ExecutionProfiles and no historical Profile entries. Pass the current Profile to `removeProfile` and remove its ExecutionProfiles directly, regardless of whether a historical entry exists. Add tests covering a missing historical entry, an existing entry, and multiple ExecutionProfiles. ### Release note Fix FE memory growth caused by execution profiles retained when query dispatch fails before profile registration. |
||
|
linrrarity
|
da4ec5dca3 |
[fix](be) Prevent shared quantile state mutation with TDigest COW (#68498)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Copies of `QuantileState` share a `TDigest`. When CTEs, joins, or
`explode` reuse a state, updating one copy can contaminate another
group's quantile result. Concurrent percentile queries can also race
while compressing the shared digest.
Add copy-on-write in `QuantileState`, using the current reference count
to detach before sample writes. Synchronize compression and shared
reads, and preserve reserved vector capacity when copying `TDigest`.
Serialization keeps its existing binary format; sizing finishes pending
compression so subsequent queries cannot change the serialized length.
This moves compression earlier and can affect approximate results.
```sql
SET enable_cte_materialize = true;
SET inline_cte_referenced_threshold = 0;
WITH t AS (
SELECT number % 2 AS g,
quantile_union(to_quantile_state(100 * (number % 2), 2048)) AS q
FROM numbers("number" = "8192")
GROUP BY g
)
SELECT g, quantile_percent(q, 0.5) FROM t
UNION ALL
SELECT -1, quantile_percent(quantile_union(q), 0.5) FROM t;
```
before:
```text
std::vector<doris::Centroid>::operator[](size_type) const: Assertion '__n < this->size()' failed.
*** Query id: e5437298fa6e4f14-9b6a546907f92425 ***
*** tablet id: 0 ***
*** Aborted at 1790250779 (unix time) try "date -d @1790250779" if you are using GNU date ***
*** Current BE git commitID: 765e2dfefe ***
*** SIGABRT unknown detail explain (@0x3fe0007a9c2) received by PID 502210 (TID 547405 OR 0x113767afd640) from PID 502210; stack trace: ***
F20260924 19:52:59.574612 547409 tdigest.h:458] Check failed: index >= _processed_weight - _weight(n - 1) / 2.0 (4095 vs. 8189)
*** Check failure stack trace: ***
@ 0x560d751ce0b6 google::LogMessageFatal::~LogMessageFatal()
@ 0x560d4c8a1600 doris::TDigest::quantile_processed()
@ 0x560d4c884644 doris::TDigest::quantile()
@ 0x560d4c876beb doris::QuantileState::get_value_by_percentile()
@ 0x560d555b14b5 doris::FunctionQuantileStatePercent::execute_impl()
@ 0x560d555b1902 doris::FunctionQuantileStatePercent::execute_impl()
@ 0x560d563a3d19 doris::PreparedFunctionImpl::_execute_skipped_constant_deal()
@ 0x560d55df44cb doris::PreparedFunctionImpl::default_execute()
@ 0x560d55d4aa7c doris::PreparedFunctionImpl::execute()
@ 0x560d4adbbe7f doris::IFunctionBase::execute()
@ 0x560d53f79b64 doris::VectorizedFnCall::_do_execute()
@ 0x560d53f147a3 doris::VectorizedFnCall::execute_column_impl()
@ 0x560d53f3d44c doris::VExpr::execute_column()
@ 0x560d53f9d63b doris::VExprContext::execute()
@ 0x560d535ee6cf doris::OperatorXBase::do_projections()
@ 0x560d535f091a doris::OperatorXBase::get_block_after_projects()
@ 0x560d4f121015 doris::PipelineTask::execute()
@ 0x560d51b28ca0 doris::TaskScheduler::_do_work()
@ 0x560d51d357c5 doris::TaskScheduler::start()::$_0::operator()()
@ 0x560d51d3570d std::__invoke_impl<>()
@ 0x560d51d3562d _ZSt10__invoke_rIvRZN5doris13TaskScheduler5startEvE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EES5_E4typeEOS6_DpOS7_
@ 0x560d51d35305 std::_Function_handler<>::_M_invoke()
@ 0x560d4a838b3e std::function<>::operator()()
@ 0x560d710d3339 doris::FunctionRunnable::run()
@ 0x560d71013f64 doris::ThreadPool::dispatch_thread()
@ 0x560d710f38fd std::__invoke_impl<>()
@ 0x560d710f36b5 std::__invoke<>()
@ 0x560d710f35e1 _ZNSt5_BindIFMN5doris10ThreadPoolEFvvEPS1_EE6__callIvJEJLm0EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
@ 0x560d710f339c std::_Bind<>::operator()<>()
@ 0x560d710f328d std::__invoke_impl<>()
@ 0x560d710f318d _ZSt10__invoke_rIvRSt5_BindIFMN5doris10ThreadPoolEFvvEPS2_EEJEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESA_E4typeEOSB_DpOSC_
@ 0x560d710f2a65 std::_Function_handler<>::_M_invoke()
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at ../src/common/signal_handler.h:418
1# 0x00001543D423FC60 in /lib64/libc.so.6
2# __pthread_kill_implementation in /lib64/libc.so.6
3# gsignal in /lib64/libc.so.6
4# abort in /lib64/libc.so.6
5# 0x0000560D7AD406A7 in /mnt/disk9/linzhenqi/dv/doris/output/be/lib/doris_be
6# std::vector<doris::Centroid, std::allocator<doris::Centroid> >::operator[](unsigned long) const at /mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_vector.h:1282
7# doris::TDigest::_weight(long) const at ../src/util/tdigest.h:623
8# doris::TDigest::_update_cumulative() at ../src/util/tdigest.h:701
9# doris::TDigest::_process() at ../src/util/tdigest.h:749
10# doris::TDigest::quantile(float) in /mnt/disk9/linzhenqi/dv/doris/output/be/lib/doris_be
11# doris::QuantileState::get_value_by_percentile(float) const at ./be/src/core/value/quantile_state.cpp:140
12# doris::FunctionQuantileStatePercent::execute_impl(doris::FunctionContext*, doris::Block&, std::vector<unsigned int, std::allocator<unsigned int> > const&, unsigned int, unsigned long) const at ./be/src/exprs/function/function_quantile_state.cpp:202
13# non-virtual thunk to doris::FunctionQuantileStatePercent::execute_impl(doris::FunctionContext*, doris::Block&, std::vector<unsigned int, std::allocator<unsigned int> > const&, unsigned int, unsigned long) const in /mnt/disk9/linzhenqi/dv/doris/output/be/lib/doris_be
14# doris::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::Block&, std::vector<unsigned int, std::allocator<unsigned int> > const&, unsigned int, unsigned long) const at ./be/src/exprs/function/function.cpp:135
15# doris::PreparedFunctionImpl::default_execute(doris::FunctionContext*, doris::Block&, std::vector<unsigned int, std::allocator<unsigned int> > const&, unsigned int, unsigned long) const at ./be/src/exprs/function/function.cpp:268
16# doris::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::Block&, std::vector<unsigned int, std::allocator<unsigned int> > const&, unsigned int, unsigned long) const at ./be/src/exprs/function/function.cpp:274
17# doris::IFunctionBase::execute(doris::FunctionContext*, doris::Block&, std::vector<unsigned int, std::allocator<unsigned int> > const&, unsigned int, unsigned long) const at ../src/exprs/function/function.h:213
18# doris::VectorizedFnCall::_do_execute(doris::VExprContext*, doris::Block const*, doris::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, true>, 16ul, 15ul> const*, unsigned long, doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>&, doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>*) const at ./be/src/exprs/vectorized_fn_call.cpp:441
19# doris::VectorizedFnCall::execute_column_impl(doris::VExprContext*, doris::Block const*, doris::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, true>, 16ul, 15ul> const*, unsigned long, doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>&) const at ./be/src/exprs/vectorized_fn_call.cpp:477
20# doris::VExpr::execute_column(doris::VExprContext*, doris::Block const*, doris::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, true>, 16ul, 15ul> const*, unsigned long, doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>&) const at ./be/src/exprs/vexpr.cpp:1067
21# doris::VExprContext::execute(doris::Block const*, doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>&) at ./be/src/exprs/vexpr_context.cpp:91
22# doris::VExprContext::get_output_block_after_execute_exprs(std::vector<std::shared_ptr<doris::VExprContext>, std::allocator<std::shared_ptr<doris::VExprContext> > > const&, doris::Block const&, doris::Block*, bool) at ./be/src/exprs/vexpr_context.cpp:464
23# doris::MultiCastDataStreamerSourceOperatorX::get_block_impl(doris::RuntimeState*, doris::Block*, bool*) at ./be/src/exec/operator/multi_cast_data_stream_source.cpp:112
24# doris::OperatorXBase::get_block(doris::RuntimeState*, doris::Block*, bool*) at ../src/exec/operator/operator.h:898
25# doris::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::Block*, bool*) at ./be/build_ASAN/../src/exec/operator/operator.cpp:435
26# doris::PipelineTask::execute(bool*) at ./be/src/exec/pipeline/pipeline_task.cpp:655
27# doris::TaskScheduler::_do_work(int) at ./be/src/exec/pipeline/task_scheduler.cpp:155
28# doris::TaskScheduler::start()::$_0::operator()() const at ./be/src/exec/pipeline/task_scheduler.cpp:64
29# void std::__invoke_impl<void, doris::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::TaskScheduler::start()::$_0&) at /mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63
30# std::enable_if<is_invocable_r_v<void, doris::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::TaskScheduler::start()::$_0&>(doris::TaskScheduler::start()::$_0&) at /mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:119
31# std::_Function_handler<void (), doris::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) at /mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292
32# std::function<void ()>::operator()() const at /mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
33# doris::FunctionRunnable::run() at ./be/src/util/threadpool.cpp:60
34# doris::ThreadPool::dispatch_thread() at ./be/src/util/threadpool.cpp:621
35# void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:76
36# std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:98
37# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) at /mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/functional:515
38# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() at /mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/functional:600
39# void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63
40# std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:119
41# std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) at /mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292
42# std::function<void ()>::operator()() const at /mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
43# doris::Thread::supervise_thread(void*) at ./be/src/util/thread.cpp:460
44# asan_thread_start(void*) in /mnt/disk9/linzhenqi/dv/doris/output/be/lib/doris_be
45# start_thread in /lib64/libc.so.6
46# clone3 in /lib64/libc.so.6
```
now:
```text
+------+--------------------------+
| g | quantile_percent(q, 0.5) |
+------+--------------------------+
| -1 | 0 |
| 0 | 0 |
| 1 | 100 |
+------+--------------------------+
```
### Release note
Fix backend crashes and incorrect quantile results when aggregate states
share inputs, including materialized CTE queries.
|
||
|
linrrarity
|
82f04c61c4 |
[fix](array-func) Handle outer-null containers with hidden nested data (#68022)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
With short-circuit evaluation disabled, array_map evaluated hidden
nested values belonging to outer-null array or map rows. Invalid hidden
values could therefore raise errors even though the visible result was
NULL.
Compact nested lambda inputs for non-null rows and rebuild offsets
before lambda execution, then preserve the original outer null map.
```sql
CREATE TABLE test_array_map_null_filt (
id INT,
a ARRAY<STRING>
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES('replication_num'='1');
INSERT INTO test_array_map_null_filt VALUES
(1, array('bad-number')),
(2, array('10'));
SET enable_strict_cast = true;
SET short_circuit_evaluation = false;
SELECT
id,
array_map(x -> CAST(x AS INT), if(id = 1, CAST(NULL AS ARRAY<STRING>), a)) AS result_array
FROM test_array_map_null_filt
ORDER BY id;
SELECT
id,
array_map(x -> CAST(x AS INT), if(id = 1, CAST(NULL AS ARRAY<STRING>), a)) AS result_array
FROM test_array_map_null_filt
ORDER BY id;
-- before:
ERROR 1105 (HY000): Exception, msg: (127.0.0.1)[INVALID_ARGUMENT]parse number fail, string: 'bad-number'
-- now:
+------+--------------+
| id | result_array |
+------+--------------+
| 1 | NULL |
| 2 | [10] |
+------+--------------+
```
### Release note
Fix lambda functions incorrectly evaluating nested values from NULL
array and map containers.
|
||
|
linrrarity
|
eb50752c02 |
[fix](function) Share DISTINCT keys for percentile_approx_array (#68489)
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: `percentile_approx_array` included the quantile array and optional compression value in its DISTINCT arguments. Calls over the same value column could therefore be classified as separate DISTINCT argument groups. The fix makes only the value expression a DISTINCT argument, allowing these calls to share one DISTINCT argument group. The unit test also verifies that different value expressions remain in separate groups. ### Release note Multiple DISTINCT `percentile_approx_array` calls over the same value expression can share one DISTINCT argument group. |
||
|
linrrarity
|
91d7975d56 |
[fix](function) Fix overflow in quarter arithmetic (#68410)
Problem Summary: `quarters_add` / `quarters_sub` uses the i32 middle value when converting quarters to months, and the oversized INT parameter may have an integer backturn, bypassing the subsequent date range check and returning the wrong date. This repair includes: 1. **BE quarter addition and subtraction**: Use Int64 to receive the quarterly offset and calculate the number of months to avoid overflow when multiplying by 3, and also avoid `quarters_sub` to narrowing back to i32 when calling addition after negative INT_MIN. Four date types share this repair, and the real bias is transferred to the existing date range for inspection and processing. 2. **FE quarter function calculation**: complete the `quarters_add` of TIMESTAMP_NS, using long multiplication; four date types `quarters_sub` directly calculate `-3L * quarter `, avoid losing in i32 first. The original long addition of DATE/DATETIME/TIMESTAMPTZ and the return mechanism after the failure of constant folding remain unchanged. 3. **Error message**: The offset parameter in the date cross-bound error uses Int64 to avoid truncating the number of months again when the error is reported. ### Release note Fix quarters_add and quarters_sub returning incorrect dates for excessively large quarter intervals; these inputs now report date-range errors. |
||
|
linrrarity
|
0482779cea |
[Enhance](system table) Push down predicates for table_stream_consumption metadata scan (#68227)
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Queries on `information_schema.table_stream_consumption` always scanned all streams and partitions, even when predicates such as `DB_NAME`, `STREAM_NAME`, `STREAM_ID`, or `UNIT` could exclude most of them. In cloud mode, this also caused unnecessary partition requests to MetaService. ### What is changed? - Forward schema-scan predicates from BE to FE for `table_stream_consumption`. - Push down `DB_NAME`, `STREAM_NAME`, and `STREAM_ID` predicates before scanning stream metadata. - Push down `UNIT` predicates after partition enumeration: - In cloud mode, only matching partitions are requested from MetaService. - In non-cloud mode, rows for unmatched partitions are not generated. |
||
|
linrrarity
|
eacc120738 |
[Enhance](ai_func) Support dedicate embed properties in AI RESOURCE (#67673)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
- Add dedicated embedding properties to AI resources:
- `ai.embed.endpoint`
- `ai.embed.provider_type`
- `ai.embed.model_name`
- `ai.embed.api_key`
- Require at least one complete property group when creating an AI
resource:
- the general `ai.*` group, or
- the `ai.embed.*` group
- Use `ai.embed.*` when executing the `EMBED` function if the embed
property group is configured.
- Fall back to the general `ai.*` properties otherwise.
- Mask `ai.embed.api_key` when displaying resource properties.
- Keep other AI functions using the general `ai.*` properties.
```text
Doris> CREATE RESOURCE 'default_combine_resource'
-> PROPERTIES (
-> 'type' = 'ai',
->
-> 'ai.provider_type' = 'QWEN',
-> 'ai.endpoint' = 'https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions',
-> 'ai.model_name' = 'qwen-plus',
-> 'ai.api_key' = '<general-api-key>',
-> 'ai.effort' = 'high',
->
-> 'ai.embed.provider_type' = 'QWEN',
-> 'ai.embed.endpoint' = 'https://dashscope.aliyuncs.com/compatible-mode/v1/embeddings',
-> 'ai.embed.model_name' = 'text-embedding-v4',
-> 'ai.embed.api_key' = '<embedding-api-key>',
->
-> 'ai.embed.mm.provider_type' = 'QWEN',
-> 'ai.embed.mm.endpoint' = 'https://dashscope.aliyuncs.com/api/v1/services/embeddings/multimodal-embedding/multimodal-embedding',
-> 'ai.embed.mm.model_name' = 'qwen3-vl-embedding',
-> 'ai.embed.mm.api_key' = '<multimodal-api-key>',
->
-> 'ai.dimensions' = '1024'
-> );
Query OK, 0 rows affected (0.026 sec)
Doris> set default_ai_resource = 'default_combine_resource';
Query OK, 0 rows affected (0.009 sec)
Doris> SELECT array_size(EMBED('this is a test'));
+-------------------------------------+
| array_size(EMBED('this is a test')) |
+-------------------------------------+
| 1024 |
+-------------------------------------+
1 row in set (1.729 sec)
Doris> EXPLAIN SELECT array_size(EMBED('this is a test'));
+---------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+---------------------------------------------------------------------------+
| PLAN FRAGMENT 0 |
| OUTPUT EXPRS: |
| array_size(EMBED('this is a test'))[#0] |
| PARTITION: UNPARTITIONED |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| VRESULT SINK |
| MYSQL_PROTOCOL |
| |
| 0:VUNION(11) |
| constant exprs: |
| cardinality(embed('default_combine_resource', 'this is a test')) |
| |
| |
| |
| ========== STATISTICS ========== |
+---------------------------------------------------------------------------+
17 rows in set (0.030 sec)
Doris> SELECT AI_TRANSLATE('this is a test', 'chinese');
+-------------------------------------------+
| AI_TRANSLATE('this is a test', 'chinese') |
+-------------------------------------------+
| 这是一个测试 |
+-------------------------------------------+
1 row in set (1.269 sec)
Doris> EXPLAIN SELECT AI_TRANSLATE('this is a test', 'chinese');
+--------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+--------------------------------------------------------------------------------+
| PLAN FRAGMENT 0 |
| OUTPUT EXPRS: |
| AI_TRANSLATE('this is a test', 'chinese')[#0] |
| PARTITION: UNPARTITIONED |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| VRESULT SINK |
| MYSQL_PROTOCOL |
| |
| 0:VUNION(11) |
| constant exprs: |
| ai_translate('default_combine_resource', 'this is a test', 'chinese') |
| |
| |
| |
| ========== STATISTICS ========== |
+--------------------------------------------------------------------------------+
17 rows in set (0.010 sec)
Doris> EXPLAIN SELECT EMBED(
-> CAST(
-> '{"content_type":"image/png","uri":"https://example.com/image.png"}'
-> AS JSON
-> )
-> );
+--------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+--------------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0 |
| OUTPUT EXPRS: |
| EMBED( |
| CAST( |
| '{"content_type":"image/png","uri":"https://example.com/image.png"}' |
| AS JSON |
| ) |
| )[#0] |
| PARTITION: UNPARTITIONED |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| VRESULT SINK |
| MYSQL_PROTOCOL |
| |
| 0:VUNION(11) |
| constant exprs: |
| embed('default_combine_resource', CAST('{"content_type":"image/png","uri":"https://example.com/image.png"}' AS json)) |
| |
| |
| |
| ========== STATISTICS ========== |
+--------------------------------------------------------------------------------------------------------------------------------+
```
### Release note
None
|
||
|
linrrarity
|
2624e053a3 |
[fix](be) Prevent BE crashes on malformed AI adapter responses (#68247)
### What problem does this PR solve?
Issue Number: N/A
Related PR: #68247
Problem Summary: AI adapters can receive valid JSON with unexpected nested value types, such as a numeric choice, a non-array embedding, or a string inside an embedding array. Calling RapidJSON object, array, or numeric accessors on those values can trigger assertions or invalid accesses in the BE. Validate each required type before accessing it, and share embedding-array validation and conversion across provider adapters. Malformed responses return a non-OK Status that callers propagate as a query error.
### Release note
Return query errors for malformed AI provider response types instead of risking a BE crash during response parsing.
### Check List (For Author)
- Test: Unit Test / Manual test
- Add 21 malformed-response unit tests; BE UT and regression CI passed for
|
||
 linrrarityandzhaochangle
|
216cd60300 |
[fix](be) Avoid rowset map race in backend tablets scanner (#68306)
pick from: https://github.com/apache/doris/pull/65288 Co-authored-by: zhaochangle <zhaochangle@selectdb.com> |
||
|
linrrarity
|
979e053f08 |
[Enhance](resource) Restrict AI resource creation by user identity (#68147)
Rstrict AI resource creation by user identity |
||
|
linrrarity
|
0988943259 |
[Fix](regexp) Fix empty constant pattern matching in regexp (#68146)
Problem Summary:
An empty regexp produces a zero-width match, so the expected result is
1.
The constant pattern was recognized as a substring pattern and routed to
the long-buffer substring optimization. However, this path treated an
empty needle as not found.
```sql
DROP TABLE IF EXISTS tmp_b8_c01;
CREATE TABLE tmp_b8_c01 (
id INT,
s STRING,
p STRING
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES("replication_num" = "1");
INSERT INTO tmp_b8_c01 VALUES (1, 'abc', '');
SELECT
regexp('abc', '') AS const_empty,
regexp(s, p) AS column_empty
FROM tmp_b8_c01
WHERE id = 1;
-- before:
+-------------+--------------+
| const_empty | column_empty |
+-------------+--------------+
| 0 | 1 |
+-------------+--------------+
-- now:
+-------------+--------------+
| const_empty | column_empty |
+-------------+--------------+
| 1 | 1 |
+-------------+--------------+
```
### What is changed?
- Handle an empty constant substring pattern as matching every non-null
input.
- Move the long-buffer substring implementation into
`constant_substring_fn`.
- Remove the separate execute_substring implementation and its special
dispatch logic.
- Preserve row-boundary checks to prevent matches across adjacent
ColumnString rows.
|
||
|
linrrarity
|
a4c062c6a8 |
[fix](meta path) Preserve metadata access path type for map_entries (#68040)
### What problem does this PR solve? When `transform_values` operates on physical nested Map columns with nested-column pruning enabled: - `MAP<..., ARRAY<...>>` may fail because `VALUES.OFFSET` is incorrectly marked as a DATA access path. - `MAP<..., STRUCT<...>>` may crash the BE because `VALUES.NULL` is incorrectly treated as a Struct data field, causing all physical Struct child iterators to be pruned. ### Root cause `AccessPathExpressionCollector.visitMapEntries` creates a new `CollectorContext` while translating access paths from `map_entries`. The new context did not inherit the original `ColumnAccessPathType`, so META paths such as `OFFSET` and `NULL` fell back to the default DATA type. ### What is changed? Propagate the original access-path type to the new context in `visitMapEntries`. Added regression coverage for: - `transform_values((k, v) -> size(v), map<int, array<int>>)` - `transform_values((k, v) -> v is null, map<int, struct<...>>)` The tests verify that `VALUES.OFFSET` and `VALUES.NULL` remain META access paths. |
||
|
linrrarity
|
e870abeee7 |
[Fix](test) make pyudf ut deterministic and meaningful (#67352)
Problem Summary:
Some Python UDF BE unit tests were flaky and provided limited
verification value. Several tests launched shell-based fake Python
processes that only created Unix socket files or stayed alive for a
fixed duration. These tests exercised process-management branches but
did not verify that Doris could start the production Python server,
establish an Arrow Flight connection, or execute an actual UDF.
This PR:
- Removes or rewrites low-value Python UDF tests that only exercised
implementation details or line coverage.
- Keeps meaningful coverage for process startup failures, shutdown,
recovery, pool initialization, and concurrent process selection.
- Fixes the race in the least-shared-process test by waiting until the
complete process pool is available before checking process distribution.
- Adds an end-to-end BE unit test that:
- Creates a Python runtime through the production venv scanner.
- Starts the real `python_server.py`.
- Connects through Arrow Flight.
- Executes a nullable inline scalar UDF.
- Verifies the returned values and null propagation.
- Adds `PYTHON_UDF_TEST_INTERPRETER` as the only Python interpreter
configuration for this test. It defaults to `/usr/bin/python`, matching
the community P0 pipeline, and can be overridden locally in
`custom_env.sh`.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
|
||
|
linrrarity
|
8dd6aafade |
[Fix](ai_func) Parse final text from Responses API output (#67671)
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: The Responses API returns heterogeneous items in the `output` array. Besides the final `message`, it may contain `reasoning` and tool-related items. The previous OpenAI adapter treated every output item as response text. This caused two problems: 1. An OpenAI-type reasoning item with an empty `content` array was reported as an invalid response. 2. An OpenAI-type `reasoning_text` item was incorrectly included in batch results, causing errors such as `expected 1 items but got 2`. This change updates the Responses API parser to: - Ignore non-`message` output items, including reasoning and tool items. - Only extract `output_text` parts from message content. - Preserve validation for malformed message and output-text structures. - Update the response format comments to match the current API structure. for example: ```sql CREATE RESOURCE 'deepseek-responses' PROPERTIES ( 'type'='ai', 'ai.provider_type'='deepseek', 'ai.endpoint'='https://api.deepseek.com/responses', 'ai.model_name' = 'deepseek-v4-flash', 'ai.api_key' = 'sk-xxx' ); SELECT id, ai_TRANSLATE('deepseek-responses', str_val, tar_language) AS Result FROM ai_test WHERE str_val IS NOT NULL; ``` before: ```text Doris> SELECT id, ai_TRANSLATE('deepseek-responses', str_val, tar_language) AS Result -> FROM ai_test WHERE str_val IS NOT NULL; ERROR 1105 (HY000): Exception, msg: (127.0.0.1)[RUNTIME_ERROR]Failed to parse ai_translate batch result, expected 1 items but got 2 ``` now ```text +------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | id | Result | +------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 4 | Satz 4, Apache Doris ist ein MPP-basiertes Echtzeit-Data-Warehouse, das für seine hohe Abfragegeschwindigkeit bekannt ist. | | 3 | phrase 3, Apache Doris est un entrepôt de données en temps réel basé sur MPP, connu pour sa vitesse de requête élevée. | | 6 | предложение 6, Apache Doris — это хранилище данных реального времени на основе MPP, известное высокой скоростью выполнения запросов. | | 2 | 句子2：Apache Doris 是一个基于 MPP 的实时数据仓库，以高查询速度著称。 | | 5 | 文5、Apache DorisはMPPベースのリアルタイムデータウェアハウスであり、高速なクエリ処理で知られています。 | | 8 | Frase 8, Apache Doris é um data warehouse em tempo real baseado em MPP, conhecido por sua alta velocidade de consulta. | | 9 | 문장 9, Apache Doris는 높은 쿼리 속도로 알려진 MPP 기반 실시간 데이터 웨어하우스입니다. | | 1 | sentence 1, Apache Doris is an MPP-based real-time data warehouse known for its high query speed. | | 7 | oración 7, Apache Doris es un almacén de datos en tiempo real basado en MPP conocido por su alta velocidad de consulta. | | 10 | الجملة 10، Apache Doris هو مستودع بيانات في الوقت الفعلي يعتمد على MPP ويُعرف بسرعته العالية في معالجة الاستعلامات. | +------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` ### Release note Fix parsing of OpenAI-compatible Responses API results containing reasoning output. |
||
|
linrrarity
|
ee4a91720f |
[Enhance](resource) Restrict modification of root-created AI resources (#67659)
- Persist whether an `AI resource` was created by root. - Only `root` can alter or drop root-created AI resources. - `ADMIN` users can still alter or drop AI resources created by `ADMIN`. |
||
|
linrrarity
|
de9ed8ee90 |
[fix](udf) Reject variadic user-defined functions (#67373)
User-defined functions exposed variadic DDL metadata without reliable
end-to-end support. Reject variadic declarations during `CREATE
FUNCTION` analysis for scalar, aggregate, table, and alias functions
while retaining variadic signature parsing for `DROP`, `SHOW`, and
historical metadata compatibility.
before:
```sql
CREATE FUNCTION py_add(INT, INT, ...)
RETURNS INT
PROPERTIES (
"type" = "PYTHON_UDF",
"symbol" = "evaluate",
"runtime_version" = "3.12.11",
"volatility" = "immutable"
)
AS $$
def evaluate(a, b, c):
return a + b + c
$$;
SELECT py_add(1, 2, 3);
-- ERROR 1105 (HY000): errCode = 2, detailMessage = Index 2 out of bounds for length 2
```
In `PythonUdfBuilder.java:82`:
```java
public Pair<PythonUdf, PythonUdf> build(String name, List<?> arguments) {
// exprs = (1, 2, 3), size = 3
// argTypes = [INT, INT], size = 2
List<Expression> exprs = arguments.stream().map(Expression.class::cast).collect(Collectors.toList());
List<DataType> argTypes = udf.getSignatures().get(0).argumentsTypes;
List<Expression> processedExprs = Lists.newArrayList();
for (int i = 0; i < exprs.size(); ++i) {
// when i = 2, argTypes.get(2), err occur!
processedExprs.add(TypeCoercionUtils.castIfNotSameType(exprs.get(i), argTypes.get(i)));
}
return Pair.ofSame(udf.withFreshVolatileIdentity().withChildren(processedExprs));
}
```
now:
```sql
CREATE FUNCTION py_add(INT, INT, ...)
RETURNS INT
PROPERTIES (
"type" = "PYTHON_UDF",
"symbol" = "evaluate",
"runtime_version" = "3.12.11",
"volatility" = "immutable"
)
AS $$
def evaluate(a, b, c):
return a + b + c
$$;
-- ERROR 1105 (HY000): errCode = 2, detailMessage = mismatched input ',' expecting ')'(line 1, pos 31)
```
### Release note
Reject variadic declarations for user-defined functions.
### Check List (For Author)
- Test: Unit Test
- ./run-fe-ut.sh --run org.apache.doris.catalog.CreateFunctionTest
- Behavior changed: Yes. Variadic user-defined function declarations are
rejected during analysis.
- Does this need documentation:
https://github.com/apache/doris-website/pull/4103
|
||
|
linrrarity
|
ddbaaab138 |
[fix](build) prevent math regression from global lance-c linkage (#67237)
Problem Summary: PR #65304 added the following libraries to `COMMON_THIRDPARTY`: ```cmake list(APPEND COMMON_THIRDPARTY m lance_c) ``` As a result, some math symbols are resolved by the system `libm` before the linker scans Doris' optimized compatibility implementations, causing a noticeable performance regression for functions such as `exp`, `log`, and `pow`: - exp: approximately 220% - ln: approximately 167% - log: approximately 197% - log2: approximately 165% Simply moving `-lm` and `lance_c` after `glibc-compatibility` exposes another issue. Once Lance is actually referenced, `liblance_c.a` pulls in Rust std objects containing late unresolved references to: - `posix_spawnp` - `posix_spawn_file_actions_init` - `posix_spawn_file_actions_destroy` - `posix_spawn_file_actions_adddup2` - `preadv` - `splice` The LDB toolchain resolves these references by extracting its monolithic `glibc-compatibility.c.o`, which also defines `getrandom`. This conflicts with Doris' explicitly linked `getrandom.c.o` and causes a duplicate-symbol error. #### Solution 1. Restore the required link order For `GLIBC_COMPATIBILITY=ON`, use the following order: ```text glibc-compatibility-explicit → glibc-compatibility → -lm → lance_c ``` This ordering ensures that: - Doris' optimized math implementations are scanned before `-lm`. - `-lm` is scanned before `lance_c`, preventing Rust compiler_builtins symbols such as cbrt from being selected. - `lance_c` remains available to its consumers. 2. Resolve the late Lance/Rust libc references Add a small set of hidden symbol-version adapters for the six strong libc references introduced by the Rust std object. Each adapter has two parts. For example, the `splice` adapter is conceptually: ```c extern __typeof__(splice) __doris_old_splice; __asm__( ".symver __doris_old_splice," "splice@GLIBC_2.5" ); __attribute__((visibility("hidden"))) ssize_t splice(...) { return __doris_old_splice(...); } ``` The hidden `splice` wrapper is included in the `glibc-compatibility-explicit` OBJECT target, so its symbol is visible to the linker before `liblance_c.a` is processed. #### performance ```text Doris> select count(exp(db)) from double_ranges; +----------------+ | count(exp(db)) | +----------------+ | 50000000 | +----------------+ 1 row in set (0.980 sec) Doris> select count(exp(db)) from double_ranges; +----------------+ | count(exp(db)) | +----------------+ | 50000000 | +----------------+ 1 row in set (0.347 sec) Doris> select count(ln(db)), count(ln(in_one)) from double_ranges; +---------------+-------------------+ | count(ln(db)) | count(ln(in_one)) | +---------------+-------------------+ | 50000000 | 50000000 | +---------------+-------------------+ 1 row in set (1.486 sec) Doris> select count(ln(db)), count(ln(in_one)) from double_ranges; +---------------+-------------------+ | count(ln(db)) | count(ln(in_one)) | +---------------+-------------------+ | 50000000 | 50000000 | +---------------+-------------------+ 1 row in set (0.607 sec) Doris> select count(log(db, db)), count(log(in_one, db)), count(log(db, in_one)), count(log(db, in_ten)) from double_ranges; +--------------------+------------------------+------------------------+------------------------+ | count(log(db, db)) | count(log(in_one, db)) | count(log(db, in_one)) | count(log(db, in_ten)) | +--------------------+------------------------+------------------------+------------------------+ | 50000000 | 49999990 | 50000000 | 50000000 | +--------------------+------------------------+------------------------+------------------------+ 1 row in set (5.944 sec) Doris> select count(log(db, db)), count(log(in_one, db)), count(log(db, in_one)), count(log(db, in_ten)) from double_ranges; +--------------------+------------------------+------------------------+------------------------+ | count(log(db, db)) | count(log(in_one, db)) | count(log(db, in_one)) | count(log(db, in_ten)) | +--------------------+------------------------+------------------------+------------------------+ | 50000000 | 49999990 | 50000000 | 50000000 | +--------------------+------------------------+------------------------+------------------------+ 1 row in set (2.356 sec) Doris> select count(log2(db)), count(log2(in_one)) from double_ranges; +-----------------+---------------------+ | count(log2(db)) | count(log2(in_one)) | +-----------------+---------------------+ | 50000000 | 50000000 | +-----------------+---------------------+ 1 row in set (1.616 sec) Doris> select count(log2(db)), count(log2(in_one)) from double_ranges; +-----------------+---------------------+ | count(log2(db)) | count(log2(in_one)) | +-----------------+---------------------+ | 50000000 | 50000000 | +-----------------+---------------------+ 1 row in set (0.655 sec) ``` |
||
|
linrrarity
|
a10c799e78 |
[Feature](lambda) Support some map lambda functions (#66968)
### What problem does this PR solve? Related PR: https://github.com/apache/doris/pull/67045, https://github.com/apache/doris/pull/67047 Doc: https://github.com/apache/doris-website/pull/4085 Problem Summary: This PR adds MAP lambda support under the Nereids planner. Supported higher-order functions: - `map_filter((k, v) -> predicate, map)` - `map_exists((k, v) -> predicate, map)` - `map_all((k, v) -> predicate, map)` - `map_apply((k, v) -> struct(new_key, new_value), map)` - `transform_keys((k, v) -> new_key, map)` - `transform_values((k, v) -> new_value, map)` Example: ```sql SELECT map_filter( (k, v) -> v > 10, map(1, 10, 2, 20) ); SELECT transform_values( (k, v) -> v + 1, map(1, 10, 2, 20) ); SELECT map_apply( (k, v) -> (k + 1, v * 2), map(1, 10, 2, 20) ); ``` ### Implementation Use `map_entries(m)` to expand the map parameter and reuse the original `array lambda` for execution. ```text map_apply((k, v) -> (k2, v2), m) ↓ map_from_entries( array_apply(entry: struct{k, v} -> (k2, v2), map_entries(m) ); map_filter((k, v) -> k + v > 0, m) ↓ %map_from_filtered_entries_unique%( array_map( entry: strct{k, v} -> if( e.k + e.v > 0, entry, null ), map_entries(m) ) transform_values((k, v) -> k + v + col, m) ↓ %map_from_entries_unique%( array_map( entry -> (e.k, e.k + e.v + col), map_entries(m) ) ) map_exists((k, v) -> k + v = 1, m) ↓ array_match_any( array_map( entry -> e.k + e.v = 1, map_entries(m) ) ) map_all((k, v) -> k + v = 1, m) ↓ array_match_all( array_map( entry -> e.k + e.v = 1, map_entries(m) ) ) ``` ### Internal helper function This PR adds two internal Map construction functions used by the rewritten expressions: - %map_from_entries_unique%: rebuilds a Map from an entry array without running key deduplication. Used by `transform_value` - %map_from_filtered_entries_unique%: **removes** null entries input instead of throw error like `map_from_entries`. Used by `map_filter` |
||
|
linrrarity
|
3ba29643fa |
[Fix](udf) Key UDF class cache by function ID and enable cleanup in cloud mode (#67046)
### What problem does this PR solve? Problem Summary: UDF cache cleanup has two problems: 1. Cloud mode does not register a worker for `CLEAN_UDF_CACHE`, so `DROP FUNCTION` cannot clean the cached UDF classloader. 2. UDF caches are cleaned by function signature. If a function is dropped and recreated with the same signature, a delayed cleanup task may delete the new cache, or the recreated function may reuse the stale cache. The FE removes the function metadata first and then submits `CleanUDFCacheTask` asynchronously. It does not wait for the BE cache cleanup result. In addition, the cleanup task does not report a completion result back to the FE. Therefore, even if the task cannot be submitted or the JNI cache cleanup fails, `DROP FUNCTION` still returns success to the client. ```sql CREATE FUNCTION test_udf(INT) RETURNS INT ...; -- implementation V1 SELECT test_udf(1); -- cache V1 DROP FUNCTION test_udf(INT); -- cache cleanup fails or is delayed CREATE FUNCTION test_udf(INT) RETURNS INT ...; -- implementation V2 SELECT test_udf(1); ```` Before this PR, the last query could reuse V1's cached classloader because V1 and V2 had the same signature. A delayed cleanup task for V1 could also remove V2's cache. ### How does this PR fix the problem? - Register the CLEAN_UDF_CACHE worker in cloud mode. - Use the function ID as the key for Java UDF cache lookup, insertion, and cleanup. = Fall back to signature-based cleanup when no valid function ID is provided for compatibility with older FEs. A recreated function receives a new function ID. Therefore, even if the previous function's cache is not successfully removed, the recreated function does not reuse it and can load and execute the correct implementation. A delayed cleanup task also removes only the old function's cache without affecting the recreated function. |
||
|
linrrarity
|
96ad9e088f |
[fix](nested column pruning) Resolve captured lambda access paths by ExprId (#67047)
### What problem does this PR solve? Related PR: #57204 Problem Summary: #### 1. Captured arguments from outer Lambdas could not be resolved Lambda arguments were associated with their bound arrays by argument name, and `visitArrayItemSlot` only searched the innermost Lambda scope. For example: ```sql SELECT array_map( m -> array_map( x -> element_at(map_values(m)[1], 'a') + x, [1]), maps) FROM t; ``` The inner Lambda captures m from the outer Lambda. The collector should produce:`s.data.*.VALUES.a` #### 2. `element_at` did not collect payload access for Lambda index/key arguments For example: ```sql SELECT id, array_map( (nested, i) -> element_at(nested, i), nested_arrays, indexes ) AS result FROM repro_lambda_access_path WHERE indexes IS NOT NULL ORDER BY id; -- ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INVALID_ARGUMENT]in array map function, the input column size are not equal completely, nested column data rows 1st size is 5, 2th size is 0. ``` In `EXPLAIN VERBOSE`: ```text | 0:VOlapScanNode(20) | | TABLE: repro_lambda_access_path.repro_lambda_access_path(repro_lambda_access_path), PREAGGREGATION: ON | | PREDICATES: indexes[#2] IS NOT NULL | | partitions=1/1 (repro_lambda_access_path) | | tablets=1/1, tabletList=1787714324292 | | cardinality=1, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | nested columns: | | nested_arrays: | | origin type: array<array<int>> | | all access paths: [nested_arrays.*.*] | | indexes: | | origin type: array<bigint> | | all access paths: [indexes.NULL] | | predicate access paths: [indexes.NULL] | | final projections: array_map((nested,i) -> element_at(nested, i), nested_arrays[#1], indexes[#2]) | | final project output tuple id: 1 | | tuple ids: 0 ``` The old visitElementAt handled secondary arguments through a direct generic visitor call:`visit(arguments.get(i), context);` When `i` is an `ArrayItemSlot` leaf, this call does not dispatch to `visitArrayItemSlot`. The collector therefore did not register payload access for the array bound to `i`. #### 3. Comparator-form array_sort did not preserve its returned payload ```sql SELECT id, array_sort( (x, y) -> IF( cardinality(x) < cardinality(y), -1, IF(cardinality(x) = cardinality(y), 0, 1) ), nested_arrays ) AS result FROM repro_lambda_access_path ORDER BY id; +------+--------------------------------------------------------------+ | id | result | +------+--------------------------------------------------------------+ | 1 | [[null, null], [null, null, null], [null, null, null, null]] | | 2 | [[null], [null, null]] | | 3 | [] | | 4 | NULL | +------+--------------------------------------------------------------+ ``` In `EXPLAIN VERBOSE`: ```text | 0:VOlapScanNode(15) | | TABLE: repro_lambda_access_path.repro_lambda_access_path(repro_lambda_access_path), PREAGGREGATION: ON | | partitions=1/1 (repro_lambda_access_path) | | tablets=1/1, tabletList=1787714324292 | | cardinality=4, avgRowSize=1592.5, numNodes=1 | | pushAggOp=NONE | | nested columns: | | nested_arrays: | | origin type: array<array<int>> | | all access paths: [nested_arrays.*.OFFSET] | | final projections: array_sort((x,y) -> if((cardinality(x) < cardinality(x)), -1, if((cardinality(x) = cardinality(x)), 0, 1)), nested_arrays[#1]) | | final project output tuple id: 1 | | tuple ids: 0 ``` For arr `ARRAY<ARRAY<INT>>`, the comparator only needs the inner-array offsets: `arr.*.OFFSET`. However, `array_sort` also returns the original inner arrays after reordering them. The old collector returned immediately after collecting the comparator paths and did not propagate the consumer's result path to `arr`. #### What is changed 1. Resolve Lambda bindings by `ExprId` - Associate each `ArrayItemReference` with its bound array by `ExprId`. - When visiting an `ArrayItemSlot`, search Lambda scopes from the innermost scope to the outermost scope. 2. Dynamically dispatch `element_at` index/key expressions 3. Preserve comparator paths and result paths for `array_sort` #### After fix: ```text Doris> SELECT id, -> array_map( -> (nested, i) -> element_at(nested, i), -> nested_arrays, -> indexes -> ) AS result -> FROM repro_lambda_access_path -> WHERE indexes IS NOT NULL -> ORDER BY id; +------+-----------+ | id | result | +------+-----------+ | 1 | [3, 4, 2] | | 2 | [9, 7] | | 3 | [] | +------+-----------+ Doris> SELECT id, -> array_sort( -> (x, y) -> -> IF( -> cardinality(x) < cardinality(y), -> -1, -> IF(cardinality(x) = cardinality(y), 0, 1) -> ), -> nested_arrays -> ) AS result -> FROM repro_lambda_access_path -> ORDER BY id; +------+-----------------------------------+ | id | result | +------+-----------------------------------+ | 1 | [[1, 2], [2, 3, 1], [4, 2, 1, 4]] | | 2 | [[9], [8, 7]] | | 3 | [] | | 4 | NULL | +------+-----------------------------------+ ``` |
||
|
linrrarity
|
725d6ce0e4 |
[Update](p0) Update PythonUdf conda mode config (#67068)
### What problem does this PR solve? ### Release note Update PythonUdf conda mode config from miniconda to miniforge |
||
|
linrrarity
|
04b57232a4 |
[Feat](map) Support function map_from_array/entries (#67045)
## Release note Support function `map_from_array` and `map_from_entries' doc: https://github.com/apache/doris-website/pull/4083 ```sql SELECT map_from_arrays([1, 2], [10, 20]) AS result; +--------------+ | result | +--------------+ | {1:10, 2:20} | +--------------+ SELECT map_from_entries(array(struct(1, 10), struct(2, 20))) AS result; +--------------+ | result | +--------------+ | {1:10, 2:20} | +--------------+ ``` |
||
|
linrrarity
|
ce6c03a95c |
[Fix](profile) Prevent concurrent historical profile loaders (#67012)
### What problem does this PR solve? Problem Summary: `ProfileManager` periodically calls `loadProfilesFromStorageIfFirstTime(false)` after FE becomes ready. The method previously tracked only whether historical profiles had finished loading. Because the storage scan runs asynchronously, repeated calls made before the first scan completed could each create a new `profile-loader` thread. This resulted in concurrent and duplicated profile storage scans. This PR adds an atomic loading flag to ensure that only one historical profile loader can run at a time: - Atomically acquire the loading state before starting the loader. - Ignore repeated calls while a loader is running. - Clear the loading state after success or failure so failed loads can be retried. - Recheck the loaded state after acquiring the flag to handle the race where the previous loader finishes between the initial check and the CAS operation. The existing loading logic and synchronous waiting behavior remain unchanged. |
||
|
linrrarity
|
342fd13104 |
[Fix](auth) Restrict processlist visibility by user identity (#66746)
Problem Summary:
Authenticated non-admin users could inspect sessions and active SQL
statements belonging to other users through
`information_schema.processlist`, `SHOW PROCESSLIST`
```sql
CREATE USER 'pl_victim' IDENTIFIED BY 'C123_567p';
CREATE USER 'pl_attacker' IDENTIFIED BY 'C123_567p';
GRANT SELECT_PRIV ON processlist_demo.* TO 'pl_victim';
GRANT SELECT_PRIV ON processlist_demo.* TO 'pl_attacker';
-- victim
SELECT SLEEP(120), 'victim_secret_token=demo-only-123';
-- attacker
SELECT User, Command, Info
FROM information_schema.processlist
WHERE User = 'pl_victim';
+-------------+---------+-------------------------------------------------------------------------------------------------------------------------+
| User | Command | Info |
+-------------+---------+-------------------------------------------------------------------------------------------------------------------------+
| pl_attacker | Query | SELECT User, Command, Info
FROM information_schema.processlist
WHERE User IN ('pl_victim', 'pl_attacker')
ORDER BY User |
| pl_victim | Query | SELECT SLEEP(120), 'victim_secret_token=demo-only-123' |
+-------------+---------+-------------------------------------------------------------------------------------------------------------------------+
```
The schema scanners did not consistently propagate the caller's user
identity to FE, and the FE metadata handlers did not always enforce
per-user visibility.
### Release note
Non-admin users can now see only their own sessions and active queries
through processlist and information_schema.active_queries. ADMIN users
retain cluster-wide visibility.
|
||
|
linrrarity
|
f45b322b2a |
[Fix](func) Keep multi-match regex cache owner alive (#66909)
### What problem does this PR solve? Problem Summary: `multi_match_any` compiles each pattern array and stores the result in a process-wide, direct-mapped cache. The previous implementation retained the cache entry's shared_ptr only while preparing the regex database and scratch space. The subsequent scan used a **raw** `Regexps*`. If another concurrent request inserted a different pattern array into the same cache bucket, the old cache entry could be replaced and destroyed while it was still being scanned, resulting in a use-after-free. This change keeps the cache entry's shared_ptr alive for the entire hs_scan operation. Cache eviction can still replace the global entry, but the compiled database is not destroyed until all active scans have completed. ### Release note Fix a potential use-after-free in multi_match_any when concurrent pattern arrays collide in the compiled-regex cache. |
||
|
linrrarity
|
cb0f7a434b |
[Opt](lambda) Aviod redundant copies during lambda execute (#66590)
`array_map` flattens array elements into an internal block before executing its lambda expression. The previous implementation introduced several redundant copies and materializations on this path: - Nested array elements were copied into new columns even when all elements could be processed in a single lambda batch. - Captured outer columns were expanded repeatedly for each outer row. - Constant captured columns were materialized as full columns. - Sparse, unused input positions could create unnecessary placeholder column work. - The first lambda result batch was copied into a newly created result column. - Internal lambda execution could receive the outer selector even though its input block had already been selected and flattened. These operations increased CPU usage and peak memory consumption, especially for large arrays, wide captured inputs, and complex lambda expressions. This change optimizes the `array_map` execution path by: - Reusing the nested array data columns directly when all nested rows fit within one lambda batch. - Preserving captured constant columns as `ColumnConst`. - Expanding row-dependent captured columns in one operation with selector-aware source row indices. - Representing unused sparse input positions with `ColumnNothing`. - Reusing lambda block column allocations across batches. - Taking ownership of the first lambda result column instead of cloning and copying it. - Passing no outer selector when executing the already-materialized internal lambda block. - Centralizing nullable array result construction to keep the execution path simpler. - Sizing internal lambda batches according to the runtime row and byte budgets. The optimization preserves array element order, nullable semantics, captured-column mapping, nested lambda behavior, and SQL result compatibility. Performance: ```text Before: Doris> select sum(array_sum(array_map(x -> x + 1, array(id % 5, id % 7, id % 11)))) from t_explode; +-----------------------------------------------------------------------+ | sum(array_sum(array_map(x -> x + 1, array(id % 5, id % 7, id % 11)))) | +-----------------------------------------------------------------------+ | 6499979 | +-----------------------------------------------------------------------+ 1 row in set (0.082 sec) After: Doris> select sum(array_sum(array_map(x -> x + 1, array(id % 5, id % 7, id % 11)))) from t_explode; +-----------------------------------------------------------------------+ | sum(array_sum(array_map(x -> x + 1, array(id % 5, id % 7, id % 11)))) | +-----------------------------------------------------------------------+ | 6499979 | +-----------------------------------------------------------------------+ 1 row in set (0.056 sec) Before: Doris> select sum(array_sum(array_map(x -> x + id, array(id % 5, id % 7, id % 11)))) from t_explode; +------------------------------------------------------------------------+ | sum(array_sum(array_map(x -> x + id, array(id % 5, id % 7, id % 11)))) | +------------------------------------------------------------------------+ | 375004249979 | +------------------------------------------------------------------------+ 1 row in set (0.091 sec) After: Doris> select sum(array_sum(array_map(x -> x + id, array(id % 5, id % 7, id % 11)))) from t_explode; +------------------------------------------------------------------------+ | sum(array_sum(array_map(x -> x + id, array(id % 5, id % 7, id % 11)))) | +------------------------------------------------------------------------+ | 375004249979 | +------------------------------------------------------------------------+ 1 row in set (0.070 sec) Before: Doris> select sum(array_sum(array_map(x -> x + 1, arr))) from t_explode; +--------------------------------------------+ | sum(array_sum(array_map(x -> x + 1, arr))) | +--------------------------------------------+ | 380997500000 | +--------------------------------------------+ 1 row in set (6.759 sec) After: Doris> select sum(array_sum(array_map(x -> x + 1, arr))) from t_explode; +--------------------------------------------+ | sum(array_sum(array_map(x -> x + 1, arr))) | +--------------------------------------------+ | 380997500000 | +--------------------------------------------+ 1 row in set (5.807 sec) ``` |
||
|
linrrarity
|
bc86a22d6f |
[Fix](build) Manage datasketches-cpp in the BE CMake build tree (#66511)
Related PR: #63143 Problem Summary: The BE build previously configured and installed datasketches-cpp from `contrib/datasketches-cpp` into `thirdparty/installed` before configuring the BE itself. Its CMake cache was stored under `contrib/datasketches-cpp/build/Release`, which was **NOT** managed by `clean_be()`. As a result, `build.sh --be --clean` could still fail before reaching `clean_be()` when the cached compiler path belonged to another workspace or no longer existed. ```text install datasketches-cpp to thirdparty path before build be Update datasketches-cpp submodule ... /mnt/disk9/linzhenqi/d1/doris Current commit ID of datasketches-cpp submodule: de8553ba372e618382c2e7b44b0ffc9422b9458c, expected is de8553ba372e618382c2e7b44b0ffc9422b9458c -- The CXX compiler identification is unknown CMake Error at CMakeLists.txt:25 (project): The CMAKE_CXX_COMPILER: /mnt/disk7/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/clang++ is not a full path to an existing compiler tool. Tell CMake where to find the compiler by setting either the environment variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path to the compiler, or to the compiler name if it is in the PATH. -- Configuring incomplete, errors occurred! ``` The standalone installation also mixed a contrib dependency into the thirdparty installation directory without using the thirdparty build lifecycle. ### Release note - Keep datasketches-cpp as a contrib submodule. - Add datasketches-cpp to the BE CMake build tree with `add_subdirectory`. - Link `Exprs` against the upstream `DataSketches::HLL` interface target. - Mark the datasketches include directories as system headers so upstream warnings are not promoted to errors by Doris's `-Werror` settings. - Use the upstream build-tree include path for `hll.hpp`. - Remove the standalone CMake configure/install commands from `build.sh`. - Remove the now-unused `TP_INSTALLED_DIR` variable. The datasketches CMake state is now stored under the BE build directory and is removed together with the rest of the BE build artifacts by `clean_be()`. |
||
|
linrrarity
|
712c078c61 |
[fix](be) Adapt nullable column unnesting to new interface (#66474)
Related PR: conflict between https://github.com/apache/doris/pull/66242 and https://github.com/apache/doris/pull/66031 Problem Summary: fix be compile: ```text In file included from ../src/exprs/function/ai/ai_functions.cpp:19: In file included from ../src/exprs/function/ai/ai_classify.h:20: ../src/exprs/function/ai/ai_functions.h:312:66: error: too few arguments to function call, expected 2, have 0 312 | prompt_columns.emplace_back(argument.unnest_nullable().column); | ~~~~~~~~~~~~~~~~~~~~~~~~ ^ ../src/core/block/column_with_type_and_name.h:86:27: note: 'unnest_nullable' declared here 86 | ColumnWithTypeAndName unnest_nullable(const NullableColumnInfo& info, | ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 87 | bool replace_null_data_to_default) const; | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In file included from ../src/exprs/function/ai/ai_functions.cpp:29: ../src/exprs/function/ai/embed.h:74:56: error: too few arguments to function call, expected 2, have 0 74 | ColumnPtr input_column = input.unnest_nullable().column; | ~~~~~~~~~~~~~~~~~~~~~ ^ ../src/core/block/column_with_type_and_name.h:86:27: note: 'unnest_nullable' declared here 86 | ColumnWithTypeAndName unnest_nullable(const NullableColumnInfo& info, | ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 87 | bool replace_null_data_to_default) const; | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2 errors generated. ``` |
||
|
linrrarity
|
a7953ebc3a |
[Enhancement](ai_func) Skip Null inputs in AI functions (#66242)
Problem Summary: The framework's default NULL implementation unwraps Nullable arguments and executes AI functions for every input row. For partially NULL inputs, the nested placeholder values of NULL rows are still included in prompts and sent to external AI providers. This causes unnecessary remote requests and token consumption. It also requires special handling for embedding results to preserve the original row order without duplicating large embedding vectors. This PR: - Disables the framework's default NULL implementation for all `AIFunction` subclasses. - Determines the Nullable return type in the `AIFunction` base class. - Extracts nested prompt columns and merges argument null maps in the common AI execution path. - Skips NULL rows before building prompts or sending requests. - Restores NULL rows in the final result while preserving the original row order. - Handles both text and multimodal Nullable inputs for `EMBED`. - Expands embedding array offsets in place, avoiding a copy of the nested Float32 embedding data. - Returns a constant NULL column when all input rows are NULL. ### Release note Fix AI scalar functions to skip NULL input rows instead of sending their placeholder values to external AI providers. |
||
|
linrrarity
|
dc4fc151b9 |
[Feature](func) Support AggFn percentile_approx_array (#65847)
### Release note doc: https://github.com/apache/doris-website/pull/4016 Support function `PERCENTILE_APPROX_ARRAY` performance: ```text Doris> select percentile_approx(FUniqID, 0.01), percentile_approx(FUniqID, 0.99) from hits_100m; +----------------------------------+----------------------------------+ | percentile_approx(FUniqID, 0.01) | percentile_approx(FUniqID, 0.99) | +----------------------------------+----------------------------------+ | 0 | 9.170476181709914e+18 | +----------------------------------+----------------------------------+ 1 row in set (13.962 sec) Doris> select percentile_approx_array(FUniqID, [0.01, 0.99]) from hits_100m; +------------------------------------------------+ | percentile_approx_array(FUniqID, [0.01, 0.99]) | +------------------------------------------------+ | [0, 9.170476181709914e+18] | +------------------------------------------------+ 1 row in set (7.869 sec) ``` |
||
|
linrrarity
|
0dde27390a |
[Fix](user-property) Preserve legacy user properties during desrialization (#65781)
Related PR: https://github.com/apache/doris/pull/37243 Problem Summary: (Before doris-3.0)Older persisted `CommonUserProperties` JSON uses long field names such as `resourceTags` and `sqlBlockRules`. The current serialized names are `rt` and `sbr`, while the compatibility aliases only covered the singular forms `resourceTag` and `sqlBlockRule`. As a result, legacy user properties may be silently ignored during metadata deserialization, causing resource tags or SQL block rules to be reset to their default values. This change adds compatibility aliases for the legacy plural field names: - `resourceTags` - `sqlBlockRules` |
||
|
linrrarity
|
f70306ce04 |
[Opt](bitmap) Optimize bitmap op count rewrites (#65483)
Problem Summary: ### Background `bitmap_count(bitmap_op(...))` currently materializes an intermediate bitmap before counting. For patterns such as `bitmap_count(bitmap_and(...))`, `bitmap_count(bitmap_or(...))`, `bitmap_count(bitmap_xor(...))`, and `bitmap_count(bitmap_and_not(...))`, Doris already has corresponding `bitmap_*_count` functions that can compute the cardinality directly. Also, `bitmap_and_count`, `bitmap_or_count`, `bitmap_xor_count`, and `bitmap_and_not_count` are semantically always not null, but their previous metadata could be marked nullable when bitmap arguments were nullable. ### Changes - Add Nereids expression rewrite: - `bitmap_count(bitmap_and(...))` -> `bitmap_and_count(...)` - `bitmap_count(bitmap_or(...))` -> `bitmap_or_count(...)` - `bitmap_count(bitmap_xor(...))` -> `bitmap_xor_count(...)` - `bitmap_count(bitmap_not(...))` / `bitmap_and_not(...)` -> `bitmap_and_not_count(...)` - Mark bitmap op count scalar functions as always not nullable in Nereids. - Add BE v2 implementations for bitmap op count return type inference: - `bitmap_and_count_v2` - `bitmap_or_count_v2` - `bitmap_xor_count_v2` - `bitmap_and_not_count_v2` - Add `new_version_bitmap_op_count` query option to select the v2 BE implementations. - Optimize binary bitmap count execution paths to avoid intermediate bitmap materialization. - Fix `BitmapValue::or_cardinality` for SET/SET input. - Merge `bitmap_and_not` / `bitmap_andnot` aliases into `bitmap_not`. |
||
|
linrrarity
|
3adf9c8c88 |
[Chore](Collaborator) Add Breakwater agent to collaborators (#65980)
### Release note Add Breakwater agent to collaborators |
||
|
linrrarity
|
307bf9e333 |
[Refactor](bitmap) Refactor BitmapValue small_set and roaring_bitmap (#65421)
Problem Summary: This PR optimizes bitmap execution performance by updating the forked Roaring64Map implementation and refactoring the small-cardinality SET representation in BitmapValue. 1. Updates Doris's forked Roaring64Map implementation based on CRoaring v4.7.2: https://github.com/RoaringBitmap/CRoaring/blob/v4.7.2/cpp/roaring/roaring64map.hh Doris-specific serialization/deserialization behavior is preserved. 2. Refactors BitmapValue's small-cardinality SET representation from `phmap::flat_hash_set` to an inline fixed-capacity container backed by `std::array<uint64_t, 32>`. Main SET optimizations include: - Store small bitmap values inline to avoid heap allocation in common SET operations. - Add batch insert paths so add_many and SET deserialization can copy/process values in bulk. - Optimize subset operations for SET: - sub_range filters values directly without sorting. - sub_limit uses filtering plus nth_element to select the smallest limited subset without fully sorting all values. - offset_limit uses nth_element-based rank selection to avoid fully sorting the SET. - Add direct intersects and contains_all helpers to avoid materializing temporary bitmaps in `bitmap_has_any` and `bitmap_has_all`. 3. Compatibility: - No new bitmap type code is introduced. - Existing EMPTY, SINGLE32, SINGLE64, SET, SET_V2, BITMAP32/64, and BITMAP32/64_V2 formats remain readable. - SET still serializes using the historical SET format. 4. Performance BITMAP_FROM_ARRAY before: ```text Doris> SELECT COUNT(*) FROM bitmap_test WHERE array_size(bitmap_to_array(large_bitmap)) > 0; +----------+ | COUNT(*) | +----------+ | 5000000 | +----------+ 1 row in set (28.962 sec) Doris> SELECT COUNT(*) FROM bitmap_test WHERE array_size(bitmap_to_array(small_bitmap)) > 0; +----------+ | COUNT(*) | +----------+ | 5000000 | +----------+ 1 row in set (1.267 sec) ``` now: ```text Doris> SELECT COUNT(*) FROM bitmap_test WHERE array_size(bitmap_to_array(large_bitmap)) > 0; +----------+ | COUNT(*) | +----------+ | 5000000 | +----------+ 1 row in set (21.722 sec) Doris> SELECT COUNT(*) FROM bitmap_test WHERE array_size(bitmap_to_array(small_bitmap)) > 0; +----------+ | COUNT(*) | +----------+ | 5000000 | +----------+ 1 row in set (0.506 sec) ``` --- BITMAP_CONTAINS before: ```text Doris> SELECT COUNT(*) FROM bitmap_test WHERE bitmap_contains(small_bitmap, id % 100); +----------+ | COUNT(*) | +----------+ | 5000000 | +----------+ 1 row in set (0.995 sec) ``` now: ```text Doris> SELECT COUNT(*) FROM bitmap_test WHERE bitmap_contains(small_bitmap, id % 100); +----------+ | COUNT(*) | +----------+ | 5000000 | +----------+ 1 row in set (0.290 sec) ``` --- SUB_BITMAP before: ```text Doris> SELECT COUNT(*) FROM bitmap_test WHERE bitmap_count(sub_bitmap(large_bitmap, 100, 500)) > 0; +----------+ | COUNT(*) | +----------+ | 5000000 | +----------+ 1 row in set (1 min 3.175 sec) ``` now: ```text Doris> SELECT COUNT(*) FROM bitmap_test WHERE bitmap_count(sub_bitmap(large_bitmap, 100, 500)) > 0; +----------+ | COUNT(*) | +----------+ | 5000000 | +----------+ 1 row in set (51.151 sec) ``` --- BITMAP_SUBSET_LIMIT before: ```text Doris> SELECT COUNT(*) -> FROM bitmap_test -> WHERE bitmap_count(bitmap_subset_limit(large_bitmap, id % 10000, 500)) > 0; +----------+ | COUNT(*) | +----------+ | 5000000 | +----------+ 1 row in set (1 min 11.807 sec) ``` now: ```text Doris> SELECT COUNT(*) -> FROM bitmap_test -> WHERE bitmap_count(bitmap_subset_limit(large_bitmap, id % 10000, 500)) > 0; +----------+ | COUNT(*) | +----------+ | 5000000 | +----------+ 1 row in set (48.117 sec) ``` |
||
|
linrrarity
|
bd10160454 |
[Test](bitmap) Harden BitmapValue serialization test (#65491)
### Release note - Strengthen BitmapValue serialization tests for deterministic and non-deterministic formats. - Keep exact byte assertions for non-SET bitmap encodings. - Add fixed historical SET binary inputs and verify deserialized bitmap contents semantically. - Avoid pinning bitmap_to_base64 output for SET because hash-set iteration order makes serialized bytes unstable. |
||
|
linrrarity
|
5aa3c6c0d9 |
[Feature](func) Support REGEXP_EXTRACT_ALL_ARRAY (#61156)
1. Support function `REGEXP_EXTRACT_ALL_ARRAY` doc: https://github.com/apache/doris-website/pull/3460 function `regexp_extract_all` returns `string` type, `regexp_extract_all_array` behaves the same but returns an `array<string>` type. ```sql -- returns a String type Doris> SELECT regexp_extract_all('AbCdE', '([[:lower:]]+)C([[:lower:]]+)'); +--------------------------------------------------------------+ | regexp_extract_all('AbCdE', '([[:lower:]]+)C([[:lower:]]+)') | +--------------------------------------------------------------+ | ['b'] | +--------------------------------------------------------------+ 1 row in set (0.13 sec) Doris> SELECT array_size(regexp_extract_all('AbCdE', '([[:lower:]]+)C([[:lower:]]+)')); ERROR 1105 (HY000): errCode = 2, detailMessage = Can not find the compatibility function signature: cardinality(VARCHAR(65533)) -- returns an Array<String> type Doris> SELECT regexp_extract_all_array('AbCdE', '([[:lower:]]+)C([[:lower:]]+)'); +--------------------------------------------------------------------+ | regexp_extract_all_array('AbCdE', '([[:lower:]]+)C([[:lower:]]+)') | +--------------------------------------------------------------------+ | ["b"] | +--------------------------------------------------------------------+ 1 row in set (1.08 sec) Doris> SELECT array_size(regexp_extract_all_array('AbCdE', '([[:lower:]]+)C([[:lower:]]+)')); +--------------------------------------------------------------------------------+ | array_size(regexp_extract_all_array('AbCdE', '([[:lower:]]+)C([[:lower:]]+)')) | +--------------------------------------------------------------------------------+ | 1 | +--------------------------------------------------------------------------------+ ``` 2. Refactor regexp function implementation - Generalize `regexp_extract` / `regexp_extract_all` execution through a shared function wrapper. - Split `regexp_extract_all` output handling into string output and array output handlers. - Merge `regexp_replace` and `regexp_replace_one` implementation with a template parameter. |
||
|
linrrarity
|
4dfcb4f143 |
[Feature](function) Support function DAMERAU_LEVENSHTEIN_DISTANCE (#65278)
### Release note Support function `DAMERAU_LEVENSHTEIN_DISTANCE` doc: https://github.com/apache/doris-website/pull/3971 |
||
|
linrrarity
|
9ec9d5ffde |
[Fix](fold_const) MAKE_SET constant folding should clear (#64907)
related PR: #56367 Problem Summary: `MAKE_SET` uses `bit &= ~(1 << pos)` for clearing bits at high positions in the FE constant folding path, which leads to incorrect clearing of high bits when `pos >= 32` due to integer shift modulo. For Java `int` shifts, the shift distance is masked with 0x1F, which means only the low 5 bits are used: - `pos = 0..31` -> normal - `pos = 32` -> treated as `0` - `pos = 33` -> treated as `1` - `pos = 64` -> treated as `0` again For inputs like: `MAKE_SET(4294967296, ...)`(4294967296 == 1L << 32) expect: `bit &= ~(1 << 32)` got: `bit &= ~(1 << 0)` That does **not** clear bit 32 at all. So: - `bit` stays unchanged and `pos` stays 32 - the loop never makes progress - the same string is appended again and again - Java throws OutOfMemoryError: Java heap space before(FE constant folding failed or FE OOM): ```text Doris> EXPLAIN SELECT MAKE_SET(4294967296, -> 'a00', 'a01', 'a02', 'a03', 'a04', 'a05', 'a06', 'a07', -> 'a08', 'a09', 'a10', 'a11', 'a12', 'a13', 'a14', 'a15', -> 'a16', 'a17', 'a18', 'a19', 'a20', 'a21', 'a22', 'a23', -> 'a24', 'a25', 'a26', 'a27', 'a28', 'a29', 'a30', 'a31', -> 'a32', 'a33') AS ms; +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | ms[#0] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCOL | | | | 0:VUNION(11) | | constant exprs: | | make_set(4294967296, 'a00', 'a01', 'a02', 'a03', 'a04', 'a05', 'a06', 'a07', 'a08', 'a09', 'a10', 'a11', 'a12', 'a13', 'a14', 'a15', 'a16', 'a17', 'a18', 'a19', 'a20', 'a21', 'a22', 'a23', 'a24', 'a25', 'a26', 'a27', 'a28', 'a29', 'a30', 'a31', 'a32', 'a33') | | | | | | | | ========== STATISTICS ========== | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 17 rows in set (28.822 sec) ``` ```text 2026-06-26 23:27:50,285 INFO (mysql-nio-pool-0|303) [StmtExecutor.executeByNereids():821] Command(EXPLAIN SELECT MAKE_SET(4294967296, 'a00', 'a01', 'a02', 'a03', 'a04', 'a05', 'a06', 'a07', 'a08', 'a09', 'a10', 'a11', 'a12', 'a13', 'a14', 'a15', 'a16', 'a17', 'a 18', 'a19', 'a20', 'a21', 'a22', 'a23', 'a24', 'a25', 'a26', 'a27', 'a28', 'a29', 'a30', 'a31', 'a32', 'a33') AS ms) process fail ed. org.apache.doris.nereids.exceptions.AnalysisException: Nereids cost too much time (32s > 30s). You should increment timeout by set 'nerei ds_timeout_second' or disable check timeout by set 'enable_nereids_timeout' to false. Time consuming details, parse time: 6ms, plan time: {"plan":-1,"garbage_collect":-1,"lock_tables":0,"analyze":2,"rewrite":-1,"fold_const_by_be":0,"collect_partitions":-1,"optimize":-1,"tra nslate":-1,"init_scan_node":-1,"finalize_scan_node":-1,"create_scan_range":-1,"distribute":-1} ``` now: ```text Doris> SET debug_skip_fold_constant = 0; Doris> EXPLAIN SELECT MAKE_SET(4294967296, -> 'a00', 'a01', 'a02', 'a03', 'a04', 'a05', 'a06', 'a07', -> 'a08', 'a09', 'a10', 'a11', 'a12', 'a13', 'a14', 'a15', -> 'a16', 'a17', 'a18', 'a19', 'a20', 'a21', 'a22', 'a23', -> 'a24', 'a25', 'a26', 'a27', 'a28', 'a29', 'a30', 'a31', -> 'a32', 'a33') AS ms; +----------------------------------+ | Explain String(Nereids Planner) | +----------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | ms[#0] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCOL | | | | 0:VUNION(12) | | constant exprs: | | 'a32' | | | | | | | | ========== STATISTICS ========== | +----------------------------------+ ``` |
||
|
linrrarity
|
1097d51a86 |
[Enhance](func) Delete useless code of split_by_char (#64801)
### Release note
There is a function registration part for `SPLIT_BY_CHAR` in FE, but
there is no corresponding implementation in BE.
```sql
Doris> SELECT SPLIT_BY_CHAR('apple:banana:cherry', ':');
-- ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Could not find function split_by_char, arg String: String, String: String return Array(Nullable(String))
```
Since the functionality of this function has already been implemented by
`SPLIT_BY_STRING`, the implementation of this function is not meaningful
and can be directly deleted.
|
||
|
linrrarity
|
c7f355922c |
[Fix](thirdparty) Unify arrow cpp standard with be (#64390)
Related PR: https://github.com/apache/doris/pull/63191 Problem Summary: Arrow 17 defaults to `C++17` when CMAKE_CXX_STANDARD is not specified, while Doris BE is built with `C++20`. This can make header-defined inline/template code from Arrow Flight and its dependencies be compiled under different C++ standard modes in the same final binary. In particular, Arrow Status-related inline paths may generate different implementations across C++17 and C++20, such as different initialization strategies for function-local static std::string objects: code: ```cpp const std::string& get_empty_string() { static const std::string s = ""; return s; } ``` cpp17 lazy initialization: ```asm get_empty_string[abi:cxx11](): push rbp mov rbp, rsp sub rsp, 64 cmp byte ptr [rip + guard variable for get_empty_string[abi:cxx11]()::s[abi:cxx11]], 0 jne .LBB0_4 lea rdi, [rip + guard variable for get_empty_string[abi:cxx11]()::s[abi:cxx11]] call __cxa_guard_acquire@PLT cmp eax, 0 je .LBB0_4 lea rdx, [rbp - 33] mov qword ptr [rbp - 32], rdx mov rax, qword ptr [rbp - 32] mov qword ptr [rbp - 8], rax lea rdi, [rip + get_empty_string[abi:cxx11]()::s[abi:cxx11]] lea rsi, [rip + .L.str] call std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::basic_string<std::allocator<char>>(char const*, std::allocator<char> const&) jmp .LBB0_3 .LBB0_3: lea rax, [rbp - 33] mov qword ptr [rbp - 24], rax mov rdi, qword ptr [rbp - 24] call std::__new_allocator<char>::~__new_allocator() [base object destructor] lea rdi, [rip + std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::~basic_string() [base object destructor]] lea rsi, [rip + get_empty_string[abi:cxx11]()::s[abi:cxx11]] lea rdx, [rip + __dso_handle] call __cxa_atexit@PLT lea rdi, [rip + guard variable for get_empty_string[abi:cxx11]()::s[abi:cxx11]] call __cxa_guard_release@PLT ``` cpp20 constant initialization: ```asm get_empty_string[abi:cxx11](): push rbp mov rbp, rsp lea rax, [rip + get_empty_string[abi:cxx11]()::s[abi:cxx11]] pop rbp ret get_empty_string[abi:cxx11]()::s[abi:cxx11]: .quad get_empty_string[abi:cxx11]()::s[abi:cxx11]+16 .quad 0 .zero 16 ``` Mixing those definitions through `weak/COMDAT` symbols is not a supported build model and can surface as runtime crashes in Flight error/status handling paths. |
||
|
linrrarity
|
3370aa003f |
[Opt](reservoir-sampler) Optimize percentile_reservoir batch aggregation (#64593)
Optimize `percentile_reservoir` aggregation performance by reducing per-row aggregate function overhead and using faster internal sorting for reservoir samples. before: ```sql Doris> select percentile_reservoir(FUniqID, 0.9999) from hits_100m; +---------------------------------------+ | percentile_reservoir(FUniqID, 0.9999) | +---------------------------------------+ | 9.222511254540202e+18 | +---------------------------------------+ 1 row in set (1.292 sec) ``` now: ```sql Doris> select percentile_reservoir(FUniqID, 0.9999) from hits_100m; +---------------------------------------+ | percentile_reservoir(FUniqID, 0.9999) | +---------------------------------------+ | 9.222511254540202e+18 | +---------------------------------------+ 1 row in set (0.537 sec) ``` |
||
|
linrrarity
|
800b6a100c |
[Fix](test) Allow bitmap base64 serialization variants (#64505)
### Release note Allow doc bitmap function tests to accept both SET and SINGLE bitmap serialization outputs. In fuzzy BE config mode, bitmap values may be serialized with different valid internal representations, causing bitmap_to_base64 golden output to vary for single-value and small bitmap cases. Replace fixed qt checks with explicit allowed-result assertions and remove the unstable golden entries. |
||
|
linrrarity
|
f8c8dd7b61 |
[Fix](nereids) Fix month nullable datetimev2 literal binding (#64459)
Related PR: https://github.com/apache/doris/pull/64127 Problem Summary: `nullable('...')` string literals are unwrapped during signature search and coerced like bare literals. For: ```sql month(nullable('2021-12-31 12:23:34')) ``` the argument is coerced to DATETIMEV2(6). But Month declared its datetime overload as `DateTimeV2Type.SYSTEM_DEFAULT`, which is `DATETIMEV2(0)`. `DATETIMEV2(6)` does not match that narrow signature, so signature matching falls through to the DATEV2 overload and reports: ```text ERROR 1105 (HY000): errCode = 2, detailMessage = argument 1 requires datev2 type, however 'cast(nullable('2025-02-20 12:12:12') as DATETIMEV2(6))' is of datetimev2 type ``` |
||
|
linrrarity
|
c6ba5357ad |
[Fix](ut) Include UDF and OLAP BE tests (#64448)
### Release note Add the missing olap and udf test directories to the BE UT CMake whitelist. |
||
|
linrrarity
|
04bf5f6217 |
[Imporve](p0) Add doc example p0 (#64282)
### Release note Add some p0 from https://github.com/apache/doris-website/pull/3913 |
||
|
linrrarity
|
43800f3bf5 |
[Fix](nereids) Fix nullable date literal binding and date *_diff folding (#64127)
## Problem Summary
Two issues in the **FE constant folding** of date/time `*_diff`
functions
(`days_diff`, `weeks_diff`, `hours_diff`, `minutes_diff`,
`seconds_diff`,
`milliseconds_diff`, `microseconds_diff`). Both only affect FE folding —
the BE
results were already correct — so `debug_skip_fold_constant` on/off
could
disagree for the same query. (`datediff` / `to_days` already fold via
`calc_daynr` and are **not** changed here; this PR only brings the
remaining
`*_diff` folding in line with them.)
### 1. Nullable string/date literals were resolved to TIMESTAMPTZ
unexpectedly
During signature search an argument is only type-coerced when
`argument.isLiteral()` is true. A literal wrapped in `nullable()` /
`non_nullable()` is a function node, not a `Literal`, so the check
fails: the
datetime coercion **and** the "no time zone → don't prefer TIMESTAMPTZ"
scoring
are both skipped. With the scoring gone every candidate signature ties,
and the
first one — the TIMESTAMPTZ overload — wins. So a plain, no-time-zone
literal
that happens to be wrapped in `nullable()` is cast to `timestamptz`
instead of
`datetimev2`.
The change unwraps `nullable()` / `non_nullable()` so the inner literal
is
coerced exactly like a bare literal. Genuinely timezone-bearing literals
(e.g. `'2021-12-31 00:30:00+08:00'`) still bind to TIMESTAMPTZ.
before:
```
explain select days_diff(nullable('2021-12-31 12:23:34'), nullable('0000-01-01 00:00:00'));
0:VUNION(11)
constant exprs:
days_diff(CAST(nullable('2021-12-31 12:23:34') AS timestamptz(6)), CAST(nullable('0000-01-01 00:00:00') AS timestamptz(6)))
```
now:
```
0:VUNION(11)
constant exprs:
days_diff(CAST(nullable('2021-12-31 12:23:34') AS datetimev2(6)), CAST(nullable('0000-01-01 00:00:00') AS datetimev2(6)))
```
### 2. Wrong day count when the range crosses year `0000`
The folding used Java `ChronoUnit.*.between(...)`. Java's `java.time`
follows the
ISO proleptic Gregorian calendar in which year 0 **is** a leap year
(`0000-02-29` exists). Doris/BE use the MySQL-style `calc_daynr`, in
which year 0
is **not** a leap year. So whenever the interval crosses `0000-02-29`,
FE folding
was exactly one day larger than the BE result.
The change folds these functions from `calc_daynr`
(`DateLiteral.getTotalDays()`)
for the day count, plus the sub-day remainder for the time-aware units —
matching `datediff` / `to_days` and the BE. `TimestampTzLiteral`
overloads are
added so timezone-bearing literals fold on the FE too.
before:
```
set debug_skip_fold_constant=false; -- FE folds
select days_diff('0001-01-01 00:00:00', '0000-01-01 00:00:00');
+---------------------------------------------------------+
| days_diff('0001-01-01 00:00:00', '0000-01-01 00:00:00') |
+---------------------------------------------------------+
| 366 | -- wrong
+---------------------------------------------------------+
set debug_skip_fold_constant=true; -- BE
select days_diff('0001-01-01 00:00:00', '0000-01-01 00:00:00');
+---------------------------------------------------------+
| 365 | -- correct
+---------------------------------------------------------+
```
now: FE folding matches BE, both return `365`.
|
||
|
linrrarity
|
3ece7b5ddc |
[Enhancement](pyudf) Enhance Python process poll init and repair (#64039)
Problem Summary: Python UDF process pool initialization previously required the whole pool to finish initialization before BE could continue serving the query. In abnormal environments, Python process startup may hang or take a very long time in paths such as: - `fork` / child process creation - waiting for the Python Flight socket to appear - terminating and waiting for a failed child process to exit When one process slot gets stuck, the whole process pool initialization can be blocked. As a result, FE may hit the send fragments RPC timeout before BE returns a meaningful Python UDF error: `RpcException, msg: timeout when waiting for send fragments rpc, query timeout:900, left timeout for this operation:30`. be.log: ```text Initializing Python process pool for version 3.8.19 with 8 processes Python process pool initialization progress for version 3.8.19: waiting_slot=4/8, success=3, failed=0, elapsed_ms=20508 Python process pool initialization progress for version 3.8.19: waiting_slot=4/8, success=3, failed=0, elapsed_ms=40508 Python process pool initialization progress for version 3.8.19: waiting_slot=4/8, success=3, failed=0, elapsed_ms=60508 Python process pool initialization progress for version 3.8.19: waiting_slot=4/8, success=3, failed=0, elapsed_ms=80508 Python process pool initialization progress for version 3.8.19: waiting_slot=4/8, success=3, failed=0, elapsed_ms=100508 Python process pool initialization progress for version 3.8.19: waiting_slot=4/8, success=3, failed=0, elapsed_ms=120508 ``` ### Solution Change Python process pool initialization from "wait until all processes are created" to "return once at least one usable process is available". The pool no longer treats full-size initialization as a prerequisite for serving queries. Once one Python process is alive, the current query can proceed. Missing or failed process slots are repaired asynchronously by the existing health check / repair path. - Bound process pool initialization time, so BE can return `SERVICE_UNAVAILABLE` before FE send fragments RPC timeout. - Allow partial pool availability: initialization succeeds as long as one usable Python process exists. - Mark the first initialization round as completed after success or timeout, then rely on health check / repair to fill missing slots. - Add bounded wait/reap logic for Python child shutdown to avoid blocking indefinitely in `wait`. - Protect late init / repair workers from writing back after shutdown, and discard late duplicate processes safely. - Share repair guarding between foreground repair and health check to avoid duplicate repair pressure. |
||
|
linrrarity
|
adbff78c0a |
Revert "[Fix](variance) Fix sample variance/stddev NaN res for single value" (#64099)
Reverts apache/doris#63605 |
||
|
linrrarity
|
daab2c0e82 |
[Feature](function) Support function ARRAY_CROSS_PRODUCT (#64031)
Issue Number: https://github.com/apache/doris/issues/48203 Related PR: https://github.com/apache/doris/pull/59223 doc: https://github.com/apache/doris-website/pull/3891 Problem Summary: Support function `ARRAY_CROSS_PRODUCT` ```sql Doris> SELECT CROSS_PRODUCT([1, 2, 3], [2, 3, 4]); +-------------------------------------+ | CROSS_PRODUCT([1, 2, 3], [2, 3, 4]) | +-------------------------------------+ | [-1, 2, -1] | +-------------------------------------+ 1 row in set (0.021 sec) Doris> SELECT CROSS_PRODUCT([1, 2, 3], NULL); +--------------------------------+ | CROSS_PRODUCT([1, 2, 3], NULL) | +--------------------------------+ | NULL | +--------------------------------+ 1 row in set (0.009 sec) Doris> SELECT CROSS_PRODUCT([1, NULL, 3], [1, 2, 3]); ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INVALID_ARGUMENT]function array_cross_product cannot have null Doris> SELECT CROSS_PRODUCT([1, 2, 3, 4], [1, 2, 3, 4]); ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INVALID_ARGUMENT]function array_cross_product requires both input arrays to have exactly 3 elements, got 4 and 4 ``` |
||
|
linrrarity
|
d073c953e8 |
[Fix](pyudf) Convert nested map value correctly (#63907)
Problem Summary:
Fix Python UDF nested complex type conversion when `MAP` appears inside
`ARRAY`, `STRUCT`, or vectorized inputs.
Previously, Python UDF argument conversion mostly relied on PyArrow's
default conversions(`Scalar.as_py()`, `Array.to_pylist()`,
`Array.to_pandas()`). Those APIs convert a top-level Arrow `MAP` into
Python-friendly values in some paths, but nested `MAP` values are
exposed as list-of-tuples. For example, `ARRAY<MAP<STRING, INT>>` could
arrive in Python as `[[('a', 1)]]` instead of `[{'a': 1}]`. This made
user UDF code see nested maps as `list` instead of `dict`.
This PR introduces a recursive Arrow-value conversion helper and applies
it consistently across Python UDF argument conversion paths. The helper
manually reconstructs Python values according to the Arrow type:
- `MAP` -> `dict`
- `LIST` / `LARGE_LIST` -> `list`
- `STRUCT` -> `dict`
before
```sql
CREATE FUNCTION py_deep_nested_debug(ARRAY<MAP<STRING, ARRAY<INT>>> )
RETURNS STRING
PROPERTIES (
"type" = "PYTHON_UDF",
"symbol" = "evaluate",
"runtime_version" = "3.12.11",
"always_nullable" = "true"
)
AS $$
def evaluate(arr):
if arr is None:
return 'None'
return 'outer_type={}, outer_repr={}'.format(type(arr).__name__, repr(arr))
$$;
SELECT py_deep_nested_debug([{'a': [1, 2], 'b': [3]}, {'c': [4, 5, 6]}]);
+-------------------------------------------------------------------------------+
| py_deep_nested_debug([{'a': [1, 2], 'b': [3]}, {'c': [4, 5, 6]}]) |
+-------------------------------------------------------------------------------+
| outer_type=list, outer_repr=[[('a', [1, 2]), ('b', [3])], [('c', [4, 5, 6])]] |
+-------------------------------------------------------------------------------+
```
now:
```text
SELECT py_deep_nested_debug([{'a': [1, 2], 'b': [3]}, {'c': [4, 5, 6]}]);
+-------------------------------------------------------------------------+
| py_deep_nested_debug([{'a': [1, 2], 'b': [3]}, {'c': [4, 5, 6]}]) |
+-------------------------------------------------------------------------+
| outer_type=list, outer_repr=[{'a': [1, 2], 'b': [3]}, {'c': [4, 5, 6]}] |
+-------------------------------------------------------------------------+
```
|
||
|
linrrarity
|
a2e76e080e |
[Fix](nereids) Preserve negative zero sign in SIGNBIT constant folding (#63954)
Problem Summary:
`signbit` in Nereids FE constant folding used `value < 0` to determine
the sign, which treats `-0.0` as non-negative and folds it to `false`.
This is inconsistent with:
- the runtime BE implementation, which uses `std::signbit`
- the documented `signbit` behavior, which distinguishes `+0.0` and
`-0.0`
`+0.0` and `-0.0` compare equal numerically, so `value < 0` cannot
distinguish them. Their difference is only recorded in the
floating-point sign bit. Using raw bits makes FE constant folding
consistent with BE runtime semantics.
before:
```text
Doris> set debug_skip_fold_constant=true;
Query OK, 0 rows affected (0.024 sec)
Doris> select signbit(cast('+0.0' as double)) , signbit(cast('-0.0' as double));
+---------------------------------+---------------------------------+
| signbit(cast('+0.0' as double)) | signbit(cast('-0.0' as double)) |
+---------------------------------+---------------------------------+
| 0 | 1 |
+---------------------------------+---------------------------------+
1 row in set (0.108 sec)
Doris> set debug_skip_fold_constant=false;
Query OK, 0 rows affected (0.002 sec)
Doris> select signbit(cast('+0.0' as double)) , signbit(cast('-0.0' as double));
+---------------------------------+---------------------------------+
| signbit(cast('+0.0' as double)) | signbit(cast('-0.0' as double)) |
+---------------------------------+---------------------------------+
| 0 | 0 |
+---------------------------------+---------------------------------+
```
now:
```text
Doris> set debug_skip_fold_constant=true;
Query OK, 0 rows affected (0.012 sec)
Doris> select signbit(cast('+0.0' as double)) , signbit(cast('-0.0' as double));
+---------------------------------+---------------------------------+
| signbit(cast('+0.0' as double)) | signbit(cast('-0.0' as double)) |
+---------------------------------+---------------------------------+
| 0 | 1 |
+---------------------------------+---------------------------------+
1 row in set (0.070 sec)
Doris> set debug_skip_fold_constant=false;
Query OK, 0 rows affected (0.002 sec)
Doris> select signbit(cast('+0.0' as double)) , signbit(cast('-0.0' as double));
+---------------------------------+---------------------------------+
| signbit(cast('+0.0' as double)) | signbit(cast('-0.0' as double)) |
+---------------------------------+---------------------------------+
| 0 | 1 |
+---------------------------------+---------------------------------+
1 row in set (0.010 sec)
```
|
||
|
linrrarity
|
c243238746 |
[Enhancement](udf) support volatility for udaf && udtf (#63611)
Related PR: https://github.com/apache/doris/pull/62698 Problem Summary: Add volatility metadata support for UDAF and UDTF definitions, so user-defined aggregate and table functions can preserve and expose their volatility semantics consistently with UDFs. |
||
|
linrrarity
|
487f783334 |
[Enhancement](udf) Do not check file when inline code exists (#63906)
Before this change, even when inline code was present, the FE would still attempt to parse and validate the `FILE` in the `CREATE FUNCTION` statement. However, during subsequent execution, even if `FILE` is valid, it would not be used. Therefore, when inline code is present, we can omit checking the `FILE` field when creating the table. before ```sql DROP FUNCTION IF EXISTS py_inline_file_udf(INT); CREATE FUNCTION py_inline_file_udf(INT) RETURNS INT PROPERTIES ( "type"="PYTHON_UDF", "file"="http://127.0.0.1:12345/non_existent.zip", "symbol"="evaluate", "runtime_version"="3.12.11", "always_nullable"="true" ) AS $$ def evaluate(x): if x is None: return None return x + 100 $$; SELECT py_inline_file_udf(val) FROM t_repro ORDER BY id; -- errCode = 2, detailMessage = cannot to compute object's checksum. ``` now ```sql Doris> SELECT py_inline_file_udf(val) FROM t_repro ORDER BY id; +-------------------------+ | py_inline_file_udf(val) | +-------------------------+ | 110 | | 120 | | 130 | +-------------------------+ ``` |
||
|
linrrarity
|
c59abe09e6 |
[Fix](FoldConst) Preserve NaN in numeric constant folding (#63870)
Problem Summary:
Some numeric functions returned different results between normal
execution and FE constant folding when the input was NaN. The shared
boundary helper `inputOutOfBound()` treated `Double.NaN` as out of bound
because range comparisons with NaN are false, causing functions such as
`ln`, `log`, `log2`, `log10`, `dlog10`, and `power` to fold NaN cases
incorrectly to NULL or skip folding.
before
```sql
Doris> set debug_skip_fold_constant=true;
Doris> select log(2 ,cast('nan' as double));
+-------------------------------+
| log(2 ,cast('nan' as double)) |
+-------------------------------+
| NaN |
+-------------------------------+
1 row in set (0.033 sec)
Doris> set debug_skip_fold_constant=false;
Query OK, 0 rows affected (0.009 sec)
Doris> select log(2 ,cast('nan' as double));
+-------------------------------+
| log(2 ,cast('nan' as double)) |
+-------------------------------+
| NULL |
+-------------------------------+
1 row in set (0.008 sec)
```
now
```sql
Doris> set debug_skip_fold_constant=true;
Query OK, 0 rows affected (0.001 sec)
Doris> select log(2 ,cast('nan' as double));
+-------------------------------+
| log(2 ,cast('nan' as double)) |
+-------------------------------+
| NaN |
+-------------------------------+
1 row in set (0.018 sec)
Doris> set debug_skip_fold_constant=false;
Query OK, 0 rows affected (0.001 sec)
Doris> select log(2 ,cast('nan' as double));
+-------------------------------+
| log(2 ,cast('nan' as double)) |
+-------------------------------+
| NaN |
+-------------------------------+
1 row in set (0.003 sec)
```
|
||
|
linrrarity
|
d7c033f7f3 |
[Fix](variance) Fix sample variance/stddev NaN res for single value (#63605)
Problem Summary:
Fix `VAR_SAMP`, `VARIANCE_SAMP`, and `STDDEV_SAMP` to return `NaN` when
the number of valid input values is less than or equal to 1. Sample
variance/stddev are undefined for `n <= 1`, so returning `0.0` is
misleading.
before:
```sql
CREATE TABLE t (id INT, v DOUBLE) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES('replication_num'='1');
INSERT INTO t VALUES (1, 5.0); -- 单行
SELECT VAR_SAMP(v), STDDEV_SAMP(v) FROM t;
+-------------+----------------+
| VAR_SAMP(v) | STDDEV_SAMP(v) |
+-------------+----------------+
| 0 | 0 |
+-------------+----------------+
```
now:
```sql
SELECT VAR_SAMP(v), STDDEV_SAMP(v) FROM t;
+-------------+----------------+
| VAR_SAMP(v) | STDDEV_SAMP(v) |
+-------------+----------------+
| NaN | NaN |
+-------------+----------------+
```
doc: https://github.com/apache/doris-website/pull/3765
|
||
|
linrrarity
|
aa68e4bd9e |
[Enhancement](udf) Reject bitmap, hll, and quantile_state in udf create (#63849)
Problem Summary: UDF creation currently allows `BITMAP`, `HLL`, and `QUANTILE_STATE` in function signatures, but these object types are not exposed to Java/Python UDF runtimes as first-class values. They are effectively bridged as opaque bytes, and marked unsupported in [doc](https://doris.apache.org/docs/dev/query-data/udf/python-user-defined-function#data-type-mapping) |
||
|
linrrarity
|
f318357c82 |
[Fix](topn) Reject non-positive topn count argument (#63350)
Problem Summary:
According to Doris doc: `topn count must be a positive integer`, but
before this, negative numbers and 0 were not handled specially,
returning `{}`, leading to unexpected results.
before
```sql
Doris> SELECT TOPN(page_id, 0) as top_zero_pages FROM page_visits;
+----------------+
| top_zero_pages |
+----------------+
| {} |
+----------------+
Doris> SELECT TOPN(page_id, -1) as top_neg_one_pages FROM page_visits;
+-------------------+
| top_neg_one_pages |
+-------------------+
| {} |
+-------------------+
```
now:
```sql
Doris> SELECT TOPN(page_id, 0) as top_pages FROM page_visits;
ERROR 1105 (HY000): errCode = 2, detailMessage = topn requires second parameter must be a constant positive integer: topn(cast(page_id as VARCHAR(65533)), 0)
Doris> SELECT TOPN(page_id, -1) as top_pages FROM page_visits;
ERROR 1105 (HY000): errCode = 2, detailMessage = topn requires second parameter must be a constant positive integer: topn(cast(page_id as VARCHAR(65533)), -1)
```
|
||
|
linrrarity
|
cfe94e17d0 |
[Enhancement](udf) Support volatility property for scalar UDF (#62698)
Problem Summary:
Previously, UDFs could be treated as deterministic in optimizer-related
paths, which is unsafe for UDFs whose results are not stable across
evaluations. That may cause invalid rewrite/planning decisions and lead
to incorrect query semantics in some cases.
Introduce `immutable`, `stable`, and `volatile` semantics through
`"volatility" = "immutable|stable|volatile"`, persist the property in
function metadata, and use it to drive deterministic and
volatile-expression behavior in Nereids.
Immutable UDFs are treated as deterministic, stable UDFs avoid volatile
identity handling while remaining non-deterministic, and volatile UDFs
receive per-call volatile identities to protect unsafe rewrites.
```sql
CREATE TABLE cte_uuid_seed (id INT) ENGINE=OLAP DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO cte_uuid_seed VALUES (1),(2),(3);
DROP FUNCTION IF EXISTS py_uuid_token(INT);
CREATE FUNCTION py_uuid_token(INT)
RETURNS STRING
PROPERTIES (
"type" = "PYTHON_UDF",
"symbol" = "py_uuid_token_impl",
"always_nullable" = "false",
"runtime_version" = "3.12.11"
)
AS $$
import uuid
def py_uuid_token_impl(x):
return f"{x}-{uuid.uuid4()}"
$$;
```
before:
```sql
SET enable_cte_materialize = true;
SET inline_cte_referenced_threshold = 10;
-- treated as volatile func(UniqueFunction), which caused wrong planning
WITH cte AS (SELECT id, py_uuid_token(id) AS token FROM cte_uuid_seed)
SELECT id, COUNT(DISTINCT token) AS distinct_tokens
FROM (SELECT id, token FROM cte UNION ALL SELECT id, token FROM cte) u
GROUP BY id ORDER BY id;
+------+-----------------+
| id | distinct_tokens |
+------+-----------------+
| 1 | 2 |
| 2 | 2 |
| 3 | 2 |
+------+-----------------+
```
now
```sql
+------+-----------------+
| id | distinct_tokens |
+------+-----------------+
| 1 | 1 |
| 2 | 1 |
| 3 | 1 |
+------+-----------------+
```
doc: https://github.com/apache/doris-website/pull/3570
|
||
|
linrrarity
|
bb3890ac6c |
[Fix](ttz) Fix TIMESTAMPTZ elapsed-time semantics to use UTC (#63161)
Problem Summary:
Fix TIMESTAMPTZ handling for elapsed-time semantics.
TIMESTAMPTZ represents an absolute instant. For calculations that depend
on the elapsed interval between two timestamps, such as time diff
functions and time-window matching logic, Doris should use the stored
UTC time values directly. This follows PostgreSQL-style semantics and
avoids treating TIMESTAMPTZ as local DATETIME before calculation.
### Before
The previous behavior could fall back to TIMESTAMPTZ-to-DATETIME
conversion before evaluating elapsed-time logic. That conversion depends
on the session time zone and produces local wall-clock time.
This is unstable around daylight saving time transitions. During DST
fall-back or spring-forward, local wall-clock time can repeat or skip,
so two TIMESTAMPTZ values with a fixed UTC interval may produce
different or unexpected elapsed-time results after conversion to
DATETIME.
```sql
Doris> SET time_zone = '+00:00';
Doris> SELECT milliseconds_diff(
-> CAST('2024-11-03 01:05:00 -05:00' AS TIMESTAMPTZ(6)),
-> CAST('2024-11-03 01:55:00 -04:00' AS TIMESTAMPTZ(6))) AS ms_utc;
+--------+
| ms_utc |
+--------+
| 600000 |
+--------+
Doris> SET time_zone = 'America/New_York';
-- 内部计算 cast 成 datetime(local_time), 在夏令时/冬令时转换节点结果不稳定)
Doris> SELECT milliseconds_diff(
-> CAST('2024-11-03 01:05:00 -05:00' AS TIMESTAMPTZ(6)),
-> CAST('2024-11-03 01:55:00 -04:00' AS TIMESTAMPTZ(6))) AS ms_ny;
+----------+
| ms_ny |
+----------+
| -3000000 |
+----------+
```
### Now
Handle TIMESTAMPTZ directly in affected elapsed-time paths, including
scalar time interval calculations and time-window matching logic, so
they operate on UTC values instead of local DATETIME values.
Add regression coverage for DST transition cases to verify that results
are based on absolute UTC elapsed time.
```sql
Doris> SET time_zone = 'America/New_York';
Doris> SELECT milliseconds_diff(
-> CAST('2024-11-03 01:05:00 -05:00' AS TIMESTAMPTZ(6)),
-> CAST('2024-11-03 01:55:00 -04:00' AS TIMESTAMPTZ(6))) AS ms_ny;
+--------+
| ms_ny |
+--------+
| 600000 |
+--------+
```
|
||
|
linrrarity
|
c5f2693d14 |
[Fix](p0) rename duplicate pyudf across p0 suites (#63214)
Problem Summary:
prevent failures caused by conflicting function names across test
suites.
```text
Exception in pythonudf_p0/test_pythonudf_mixed_params.groovy(line 242):
sql """ DROP FUNCTION IF EXISTS py_vec_add_int(INT, INT); """
// If another suite creates a function with the same name at this time, it will cause the test to fail.
sql """
CREATE FUNCTION py_vec_add_int(INT, INT)
Exception:
java.sql.SQLException: errCode = 2, detailMessage = function already exists
```
|
||
|
linrrarity
|
8b3912260d |
[Fix](arrow flight) Fix arrow::Status inline static empty msg core (#63191)
Related PR: #62489 Problem Summary: ```cpp static const std::string no_message = ""; return ok() ? no_message : state_->msg; ``` In the clang-built Doris BE binary, this inline static `std::string` is emitted as a weak/COMDAT object and placed in `.data.rel.ro`. ```bash $ readelf -sW "$BIN" | c++filt | grep 'Status::message.*no_message' 105229: 000000007d6958e0 32 OBJECT WEAK DEFAULT 28 arrow::Status::message[abi:cxx11]() const::no_message[abi:cxx11] $ readelf -SW doris/output/be/lib/doris_be | grep '\[ *28\]' [28] .data.rel.ro PROGBITS 000000007d0f7720 7d0f6720 7e4208 00 WA 0 0 32 ``` After relocation, RELRO makes this section read-only. However, C++ function-local statics are lazily initialized on first execution, so the first call to `Status::message()` tries to construct `no_message` at runtime. The `std::string` constructor writes to the object storage in `.data.rel.ro`, which triggers `SIGSEGV invalid permissions for mapped object` like: *FromStatus(arrow::Status)* ```text *** Query id: 0-0 *** *** is nereids: 0 *** *** tablet id: 0 *** *** Aborted at 1778559900 (unix time) try "date -d @1778559900" if you are using GNU date *** *** Current BE git commitID: f02e9e680c8 *** *** SIGSEGV invalid permissions for mapped object (@0xaaaaf1ecbf28) received by PID 3634450 (TID 3636517 OR 0xfffa35fe97c0) from PID 18446744073473408808; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/selectdb-core/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo_t*, void*) [clone .part.0] in /opt/jdk/lib/server/libjvm.so 2# JVM_handle_linux_signal in /opt/jdk/lib/server/libjvm.so 3# 0x0000FFFFAE091830 in linux-vdso.so.1 4# std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) at /root/toolchain/ldb-toolchain-v0.21/bin/../lib/gcc/aarch64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:632 5# arrow::flight::internal::TransportStatus::FromStatus(arrow::Status const&) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 6# arrow::flight::transport::grpc::ToGrpcStatus(arrow::Status const&, grpc::ServerContext*) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 7# arrow::flight::transport::grpc::(anonymous namespace)::GrpcServiceHandler::DoGet(grpc::ServerContext*, arrow::flight::protocol::Ticket const*, grpc::ServerWriter<arrow::flight::protocol::FlightData>*) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 8# grpc::Status grpc::internal::CatchingFunctionHandler<grpc::internal::ServerStreamingHandler<arrow::flight::protocol::FlightService::Service, arrow::flight::protocol::Ticket, arrow::flight::protocol::FlightData>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}>(grpc::internal::ServerStreamingHandler<arrow::flight::protocol::FlightService::Service, arrow::flight::protocol::Ticket, arrow::flight::protocol::FlightData>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}&&) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 9# grpc::internal::ServerStreamingHandler<arrow::flight::protocol::FlightService::Service, arrow::flight::protocol::Ticket, arrow::flight::protocol::FlightData>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 10# grpc::Server::SyncRequest::ContinueRunAfterInterception() in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 11# grpc::Server::SyncRequest::Run(std::shared_ptr<grpc::Server::GlobalCallbacks> const&, bool) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 12# grpc::ThreadManager::MainWorkLoop() in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 13# grpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::__invoke(void*) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 14# grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::__invoke(void*) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 15# start_thread in /lib64/libc.so.6 16# thread_start in /lib64/libc.so.6 ``` *ToStatus()* ```text *** SIGSEGV invalid permissions for mapped object (@0x56448ff1ef38) received by PID 66637 (TID 67372 OR 0x7f2687017640) from PID 18446744071829581624; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at ../src/common/signal_handler.h:417 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F2836CE7520 in /lib/x86_64-linux-gnu/libc.so.6 4# std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.h:707 5# arrow::Status::WithDetail(std::shared_ptr<arrow::StatusDetail>) const in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 6# arrow::flight::internal::TransportStatus::ToStatus() const in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 7# arrow::flight::transport::grpc::FromGrpcStatus(grpc::Status const&, grpc::ClientContext*) in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 8# arrow::flight::transport::grpc::(anonymous namespace)::GrpcClientInterceptorAdapter::Intercept(grpc::experimental::InterceptorBatchMethods*) in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 9# grpc::internal::InterceptorBatchMethodsImpl::RunInterceptors() in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 10# grpc::internal::CallOpSet<grpc::internal::CallOpRecvInitialMetadata, grpc::internal::CallOpClientRecvStatus, grpc::internal::CallNoOp<3>, grpc::internal::CallNoOp<4>, grpc::internal::CallNoOp<5>, grpc::internal::CallNoOp<6> >::FinalizeResult(void**, bool*) in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 11# grpc::ClientReaderWriter<arrow::flight::protocol::FlightData, arrow::flight::protocol::FlightData>::Finish() in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 12# arrow::flight::transport::grpc::(anonymous namespace)::FinishableDataStream<grpc::ClientReaderWriter<arrow::flight::protocol::FlightData, arrow::flight::protocol::FlightData>, arrow::flight::internal::FlightData>::DoFinish() in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 13# arrow::flight::transport::grpc::(anonymous namespace)::WritableDataStream<grpc::ClientReaderWriter<arrow::flight::protocol::FlightData, arrow::flight::protocol::FlightData>, arrow::flight::internal::FlightData>::DoFinish() in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 14# arrow::flight::internal::ClientDataStream::Finish(arrow::Status) in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 15# arrow::flight::ClientStreamReader::Next() in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 16# doris::PythonClient::read_batch(std::shared_ptr<arrow::RecordBatch>*) at ./be/build_RELEASE/../src/udf/python/python_client.cpp:136 17# doris::PythonUDFClient::evaluate(arrow::RecordBatch const&, std::shared_ptr<arrow::RecordBatch>*) at ./be/build_RELEASE/../src/udf/python/python_udf_client.cpp:36 ``` ### fix Move the empty message/detail sentinels out of the header inline path and make non-OK statuses return `state_->msg` directly. This avoids touching the empty OK-message sentinel on error paths and prevents the inline weak/COMDAT `std::string` object from being lazily constructed from a read-only mapping. |
||
|
linrrarity
|
2cfa2426e4 |
[Fix](pyudf) Fix python udf error propagation (#62613)
Problem Summary:
before
```sql
CREATE FUNCTION py_err_stats_test(INT)
RETURNS INT
PROPERTIES (
"type"="PYTHON_UDF",
"symbol"="evaluate",
"runtime_version"="3.12.11",
"always_nullable"="true"
) AS $$
def evaluate(x):
raise TypeError("consistent_error_42")
$$;
SELECT py_err_stats_test(1);
+----------------------+
| py_err_stats_test(1) |
+----------------------+
| NULL |
+----------------------+
```
now
```sql
SELECT py_err_stats_test(1);
-- ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[RUNTIME_ERROR]Flight stream finish failed with message: Error in scalar UDF execution at row 0: consistent_error_42. Detail: Python exception: Traceback (most recent call last):
File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 614, in _scalar_call
res = self._eval_func(*converted_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 3, in evaluate
TypeError: consistent_error_42
```
|
||
|
linrrarity
|
54d7885340 |
[Fix](pyudf) clear stale UDAF state cache on drop (#63062)
Fix Python UDAF stale cache reuse after dropping and recreating an
inline UDAF with the same name/signature.
The Python server previously keyed `UDAF state` managers by `function
name and argument types`, so a recreated inline UDAF could reuse the old
loaded Python class. This fix includes the FE function id in the Python
UDAF metadata/cache key and clears `UDAF state` manager cache during
`DROP FUNCTION` cleanup.
```sql
set enable_sql_cache = 0;
DROP FUNCTION IF EXISTS py_udaf_bug_repro(INT);
drop database if exists db001;
create database db001;
use db001;
-- 0. Prepare test data
DROP TABLE IF EXISTS t_udaf_cache_bug_test;
CREATE TABLE t_udaf_cache_bug_test (
id INT,
val INT
) DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES("replication_num"="1");
INSERT INTO t_udaf_cache_bug_test VALUES (1, 10), (2, 20), (3, 30);
-- At this moment, the total of the entire table val is 60.
-- 1. Create V1 version of UDAF (Logic: Accumulate and multiply by 10)
DROP FUNCTION IF EXISTS py_udaf_bug_repro(INT);
select sleep(10);
CREATE AGGREGATE FUNCTION py_udaf_bug_repro(INT)
RETURNS BIGINT
PROPERTIES (
"type"="PYTHON_UDF",
"symbol"="RecreateUDAF",
"runtime_version"="3.12.11",
"always_nullable"="true"
)
AS $$
class RecreateUDAF:
def __init__(self):
self.total = 0
@property
def aggregate_state(self):
return self.total
def accumulate(self, val):
if val is not None:
self.total += val
def merge(self, other):
self.total += other
def finish(self):
return self.total * 10 # V1: 乘以 10
$$;
-- 2. Verify V1 Logic
SELECT py_udaf_bug_repro(val) FROM t_udaf_cache_bug_test;
-- Expected Return: 600 (60 * 10)
-- Actual Return: 600 (Correct)
-- 3. Drop the old function and create a V2 version of the UDAF with the same name (logic: accumulate and multiply by 100)
DROP FUNCTION IF EXISTS py_udaf_bug_repro(INT);
select sleep(10);
select sleep(10);
CREATE AGGREGATE FUNCTION py_udaf_bug_repro(INT)
RETURNS BIGINT
PROPERTIES (
"type"="PYTHON_UDF",
"symbol"="RecreateUDAF",
"runtime_version"="3.12.11",
"always_nullable"="true"
)
AS $$
class RecreateUDAF:
def __init__(self):
self.total = 0
@property
def aggregate_state(self):
return self.total
def accumulate(self, val):
if val is not None:
self.total += val
def merge(self, other):
self.total += other
def finish(self):
return self.total * 100 # V2: Logic modified to multiply by 100
$$;
-- 4. Verify V2 Logic
SELECT py_udaf_bug_repro(val) FROM t_udaf_cache_bug_test;
-- Expected Return: 6000 (60 * 100)
-- Actual Return: 600 ([Bug occurs] Still outputs the old cached 600)
```
|
||
|
linrrarity
|
9896994e9e |
[Fix](pyudf) clear Nereids UDF registry on drop database (#62950)
Nereids resolves UDF calls from FunctionRegistry, while SHOW FUNCTIONS
reads catalog
metadata. After DROP DATABASE and recreating the same database name,
catalog metadata
could be empty but FunctionRegistry still contained stale Python UDF
builders, causing
SELECT to bind and execute the old function body.
```sql
DROP DATABASE IF EXISTS registry_test_db
CREATE DATABASE registry_test_db;
USE registry_test_db;
DROP FUNCTION IF EXISTS py_exc_cache_test(INT);
CREATE FUNCTION py_exc_cache_test(INT)
RETURNS INT
PROPERTIES (
"type" = "PYTHON_UDF",
"symbol" = "evaluate",
"runtime_version" = "3.12.11",
"always_nullable" = "true"
)
AS $$
def evaluate(x):
if x is None:
return None
return x + 1
$$;
-- Normal operation
SELECT py_exc_cache_test(10); -- 11
-- Directly delete, but the code didn't clean up the FunctionRegistry under db.
DROP DATABASE registry_test_db FORCE;
CREATE DATABASE registry_test_db;
USE registry_test_db;
-- show functions 走catalog的 db.getFunctions()
SHOW FUNCTIONS LIKE 'py_exc_cache_test';
-- empty
-- Function execution, go through FunctionRegistry, use the remaining FunctionRegistry, expected to be 11 (bug)
SELECT py_exc_cache_test(10);
-- Create a new function with the same name, but the execution logic is to append to the end of the Registry.
-- Normal call still goes through the previous version of x + 1 after drop dp (bug)
DROP FUNCTION IF EXISTS py_exc_cache_test(INT);
CREATE FUNCTION py_exc_cache_test(INT)
RETURNS INT
PROPERTIES (
"type" = "PYTHON_UDF",
"symbol" = "evaluate",
"runtime_version" = "3.12.11",
"always_nullable" = "true"
)
AS $$
def evaluate(x):
if x is None:
return None
return x + 999
$$;
SELECT py_exc_cache_test(10); -- still 11 (bug)
```
|
||
|
linrrarity
|
892eae27bb |
[Fix](ai_agg) isolate AI_AGG query_ctx per aggregate state (#63080)
Problem Summary: `AI_AGG` previously stored `QueryContext` in a process-level static pointer. Concurrent AI_AGG queries could overwrite that pointer, causing one query to read another query's AI resource metadata, query options, or timeout state. The aggregate function instance now receives the query context from `AggFnEvaluator` and binds it to each AggregateFunctionAIAggData state when the state is created, reset, or deserialized. |
||
|
linrrarity
|
497cbf5c27 |
[Log](pyudf) Add progress logs for python process pool init (#62974)
Improve observability for rare Python UDF process pool initialization stalls. Previously, when initialization was blocked or unusually slow, logs only showed the start of pool creation, making it difficult to tell whether BE was still waiting on process startup or where time was being spent. This adds lightweight progress and elapsed-time logs to help diagnose initialization hangs without increasing normal-case log volume. And reduce the `max_python_process_num` to 16 in regression test, we don't need so many processes for testing. |
||
|
linrrarity
|
75b381a893 |
[Enhancement](pyudf) Support parameterless calls for pythonUDF (#62624)
Problem Summary:
```sql
CREATE FUNCTION py_pkg_versions()
RETURNS STRING
PROPERTIES (
"type" = "PYTHON_UDF",
"symbol" = "evaluate",
"runtime_version" = "3.12.11",
"always_nullable" = "true"
)
AS $$
import json
import sys
def evaluate():
versions = {"python": sys.version}
try:
import numpy
versions["numpy"] = numpy.__version__
except:
versions["numpy"] = "not_found"
try:
import pandas
versions["pandas"] = pandas.__version__
except:
versions["pandas"] = "not_found"
try:
import jieba
versions["jieba"] = jieba.__version__
except:
versions["jieba"] = "not_found"
return json.dumps(versions)
$$;
```
before:
```sql
SELECT py_pkg_versions();
-- errCode = 2, detailMessage = (172.20.49.73)[INVALID_ARGUMENT]Python UDF input types is empty
```
now:
```sql
SELECT py_pkg_versions();
+------------------------------------------------------------------------------------------------------------------------------------------------------+
| py_pkg_versions() |
+------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"python": "3.12.11 | packaged by conda-forge | (main, Jun 4 2025, 14:45:31) [GCC 13.3.0]", "numpy": "2.4.3", "pandas": "3.0.1", "jieba": "0.42.1"} |
+------------------------------------------------------------------------------------------------------------------------------------------------------+
```
|
||
|
linrrarity
|
b3af03dafc |
[Fix](fragment) avoid query-ctx map clear self-deadlock when stop FragmentMgr (#62954)
Problem Summary:
Previously `ConcurrentContextMap::clear()` destroyed map values while
holding the shard_mutex. For `_query_ctx_map_delay_delete`, releasing
the last QueryContext reference can run `QueryContext::~QueryContext()`,
which calls `FragmentMgr::remove_query_context()` and re-enters the same
delay-delete map
through `erase()`. This can throw std::system_error with "Resource
deadlock avoided" during BE shutdown.
```bash
terminate called after throwing an instance of 'std::system_error'
what(): Resource deadlock avoided
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1777288253 (unix time) try "date -d @1777288253" if you are using GNU date ***
*** Current BE git commitID:
|
||
|
linrrarity
|
3f3c79c01c |
[Fix](pyudf) make Python server pool selection alive-aware and version-isolated (#62620)
`PythonServerManager` does not check if the python process corresponding to the version is alive when retrieving the process, which may cause errors like: ```text java.lang.IllegalStateException: PYTHON_UDF_BLOCKED suite=python_udf_cross_feature_import_storage scenario=python_udf_cross_feature_import_storage.inline_runtime reason=inline probe failed. reason=errCode = 2, detailMessage = (172.20.49.73)[INTERNAL_ERROR]IOError: Flight stream finish failed with gRPC code 14, message: failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/doris_python_udf_55799.sock: Connection refused ``` After modification, before obtaining the Python process, it will check if the process is alive to ensure the availability of this feature. |
||
|
linrrarity
|
92f993c68e | [chore](collaborator) fix collaborator github username (#62963) | ||
|
linrrarity
|
882d02581c |
[Fix](decimal) Fix unaligned memory access in read_column_from_arrow (#62759)
Related PR: #58002 Problem Summary: `arrow::Decimal256Array::Value()` returns raw bytes (`const uint8_t*`), we previously reinterpret_cast it to `Decimal256*` which is a misaligned access . ```text ../src/core/data_type_serde/data_type_decimal_serde.cpp:376:21: runtime error: reference binding to misaligned address 0x1338f3345b12 for type 'const FieldType' (aka 'const Decimal<integer<256, int>>'), which requires 8 byte alignment 0x1338f3345b12: note: pointer points here 00 c2 3e 20 00 00 10 63 2d 5e c7 6b 05 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ #0 0x56203d1eb751 in doris::DataTypeDecimalSerDe<(doris::PrimitiveType)35>::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const be/build_ASAN/../src/core/data_type_serde/data_type_decimal_serde.cpp:375:25 #1 0x56203d9c6d6b in doris::DataTypeNullableSerDe::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const be/build_ASAN/../src/core/data_type_serde/data_type_nullable_serde.cpp:350:26 #2 0x562055dd83b5 in doris::FromRecordBatchToBlockConverter::convert(doris::Block*) be/build_ASAN/../src/format/arrow/arrow_block_convertor.cpp:131:9 #3 0x562055dd9229 in doris::convert_from_arrow_batch(std::shared_ptr<arrow::RecordBatch> const&, std::vector<std::shared_ptr<doris::IDataType const>, ``` |
||
|
linrrarity
|
ac6d282671 |
[Opt](func) Improve AggFunc Percentile performance (#62520)
### Key Optimizations - Reworked exact percentile state into a simpler `values + PercentileLevels + inited_flag` layout. - Removed the old Counts-based exact percentile path and replaced it with a lightweight `PercentileLevels` metadata structure. - Reduced add-path overhead by directly appending values into a contiguous buffer and initializing `percentile levels` only once. - Optimized multi-quantile finalize(`percentile_array`) by sorting quantiles through permutation once, then reusing incremental `nth_element` selection. ### Performance before: ```text Doris> select percentile(FUniqID, 0.91) from hits_100m; +---------------------------+ | percentile(FUniqID, 0.91) | +---------------------------+ | 8.749975206154981e+18 | +---------------------------+ 1 row in set (6.149 sec) Doris> select percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56]) from hits_100m; +------------------------------------------------------------------------------------------------+ | percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56]) | +------------------------------------------------------------------------------------------------+ | [0, 5.097819197233503e+18, 5.68906185719365e+18, 6.281280528073601e+18, 6.869583632113269e+18] | +------------------------------------------------------------------------------------------------+ 1 row in set (28.984 sec) Doris> select percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56, 0.67, 0.78, 0.89, 0.91]) from hits_100m; ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[E-3113]string column length is too large: total_length=7200000149, element_number=0, rows=0 ``` now ```text Doris> select percentile(FUniqID, 0.91) from hits_100m; +---------------------------+ | percentile(FUniqID, 0.91) | +---------------------------+ | 8.749975206154981e+18 | +---------------------------+ 1 row in set (2.107 sec) Doris> select percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56]) from hits_100m; +------------------------------------------------------------------------------------------------+ | percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56]) | +------------------------------------------------------------------------------------------------+ | [0, 5.097819197233503e+18, 5.68906185719365e+18, 6.281280528073601e+18, 6.869583632113269e+18] | +------------------------------------------------------------------------------------------------+ 1 row in set (5.903 sec) Doris> select percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56, 0.67, 0.78, 0.89, 0.91]) from hits_100m; +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56, 0.67, 0.78, 0.89, 0.91]) | +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | [0, 5.097819197233503e+18, 5.68906185719365e+18, 6.281280528073601e+18, 6.869583632113269e+18, 7.459044216605808e+18, 8.044902013692362e+18, 8.640523572825436e+18, 8.749975206154981e+18] | +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (8.866 sec) ``` |
||
|
linrrarity
|
af71398d29 | [Fix](thirdparty) Fix pyudf core cased by Arrow Flight unstable status reconstruction (#62489) | ||
|
linrrarity
|
2a8afb9aa5 |
[Opt](ai-func) Improving AI function performance (#62494)
### Release note
Improving the performance of AI functions through batch sending, embed
controls the number of (text/file) items sent in a single batch through
the variable `embed_max_batch_size`, and the remaining functions
internally maintain a conservative context window.
The current sending format is similar to:
```json
"input": [
{"role": "system", "content": "system_prompt here"},
{"role": "user",
"content": [
{"idx": 1, "text": "xxx"},
{"idx": 2, "text": "xxx"},
]
}
]
```
performance:
```sql
-- AI_CLASSIFY
SELECT
COUNT(*) AS total_rows,
SUM(IF(res = 'science', 1, 0)) AS excepte_eq_res
FROM (
SELECT AI_CLASSIFY('deepseek-chat', str, ['science', 'sport']) AS res
FROM test_str
) t;
-- before
+------------+----------------+
| total_rows | excepte_eq_res |
+------------+----------------+
| 100 | 100 |
+------------+----------------+
1 row in set (2 min 11.579 sec)
-- now
+------------+----------------+
| total_rows | excepte_eq_res |
+------------+----------------+
| 100 | 100 |
+------------+----------------+
1 row in set (10.487 sec)
-- AI_FILTER
SELECT
COUNT(*) AS total_rows,
SUM(IF(res = 1, 1, 0)) AS zero_res_rows
FROM (
SELECT AI_FILTER('deepseek-chat', str) AS res
FROM test_str
) t;
-- before
+------------+---------------+
| total_rows | zero_res_rows |
+------------+---------------+
| 100 | 0 |
+------------+---------------+
1 row in set (2 min 2.979 sec)
-- now
+------------+---------------+
| total_rows | zero_res_rows |
+------------+---------------+
| 100 | 0 |
+------------+---------------+
1 row in set (5.007 sec)
-- EMBED
select count(embed('qwen-embed', str)) FROM test_str;
-- before
+---------------------------------+
| count(embed('qwen-embed', str)) |
+---------------------------------+
| 100 |
+---------------------------------+
1 row in set (4 min 4.888 sec)
-- now
set embed_max_batch_size = 10;
+---------------------------------+
| count(embed('qwen-embed', str)) |
+---------------------------------+
| 100 |
+---------------------------------+
1 row in set (23.424 sec)
-- Multimodal_Embed
SELECT COUNT(EMBED('qwen_mul_embed', to_json(file))) FROM test_jpg2;
-- before: can't get results for a long time(over 20 mins).
-- now
set embed_max_batch_size = 20;
+----------------------------------------------------+
| .... |
| 1152 |
+----------------------------------------------------+
1142 rows in set (1 min 13.577 sec)
```
|
||
|
linrrarity
|
443e1acf43 |
[Feature](multimodal_embed) Support multimodal(file) embed (#62147)
Support image/video/audio embedding from json.
Supported multimodal_embed AI provider : Gemini(image, video, audio),
Qwen(image, video), Voyage(image, video), Jina(image)
Supported authentication methods include IAM + EXTERNAL_ID / AK + SK
```text
-- ima role, video
mysql> SELECT ARRAY_SIZE(
-> EMBED(
-> 'qwen_mul_embed',
-> CAST('{
'> "uri": "s3://selectdb-qa-test-3/lzq-multimodal-test/video/45944deac7d96c872f559f2ef94ea0a9.mp4",
'> "content_type": "video/mp4",
'> "provider": "AWS",
'> "endpoint": "s3.us-east-1.amazonaws.com",
'> "region": "us-east-1",
'> "role_arn": "arn:aws:iam::447051187841:role/lzq-role-test",
'> "external_id": "1001"
'> }' AS JSON)
-> )
-> ) AS video_embed_size;
+------------------+
| video_embed_size |
+------------------+
| 2560 |
+------------------+
-- ima role, video
mysql> SELECT ARRAY_SIZE(
-> EMBED(
-> 'qwen_mul_embed',
-> CAST('{
'> "uri": "s3://selectdb-qa-test-3/lzq-multimodal-test/image/ai-agg.png",
'> "content_type": "image/png",
'> "provider": "AWS",
'> "endpoint": "s3.us-east-1.amazonaws.com",
'> "region": "us-east-1",
'> "role_arn": "arn:aws:iam::447051187841:role/lzq-role-test",
'> "external_id": "1001"
'> }' AS JSON)
-> )
-> ) AS img_embed_size;
+----------------+
| img_embed_size |
+----------------+
| 2560 |
+----------------+
-- gemini multimodal embed
mysql> SELECT ARRAY_SIZE(
-> EMBED(
-> 'gemini_mul_embed',
-> CAST('{
'> "uri": "s3://selectdb-qa-test-3/lzq-multimodal-test/image/ai-agg.png",
'> "content_type": "image/png",
'> "provider": "AWS",
'> "endpoint": "s3.us-east-1.amazonaws.com",
'> "region": "us-east-1",
'> "role_arn": "arn:aws:iam::447051187841:role/lzq-role-test",
'> "external_id": "1001"
'> }' AS JSON)
-> )
-> ) AS img_embed_size;
+----------------+
| img_embed_size |
+----------------+
| 3072 |
+----------------+
-- ak, sk
Doris> SELECT ARRAY_SIZE(
-> EMBED(
-> 'qwen_mul_embed',
-> CAST('{
'> "uri": "s3://<bucket>/<key>/test_img.png",
'> "content_type": "image/png",
'> "endpoint": "cos.ap-hongkong.myqcloud.com",
'> "region": "ap-hongkong",
'> "ak": "<ak>",
'> "sk": "<sk>"
'> }' AS JSON)
-> )
-> ) AS img_embed_size_ak_sk;
+----------------------+
| img_embed_size_ak_sk |
+----------------------+
| 2560 |
+----------------------+
```
|
||
|
linrrarity
|
0013e26953 |
[Update](p0) control python version via conf in p0-test (#62338)
Newer versions of Conda can no longer keep Python pinned to `3.8.10` while remaining compatible with several required test dependencies (e.g. pyarrow, pandas) from conda-forge.(will automatically upgrade to `3.8.19`) To ensure the test environment can be successfully created and remains stable, This PR allows controlling the Python UDF version in p0 by modifying the `pythonUdfRuntimeVersion` in `regression_conf.groovy`. |
||
|
linrrarity
|
06d114b4ce |
[Fix](pyudf) Fix import conflicts for modules with the same top-level name (#62167)
Related PR: https://github.com/apache/doris/pull/61280 Problem Summary: When two imported modules share the same top-level package name, python_server imports them package by package, and clean up `sys.modules` after import, which causes incorrect lock key assignment and leads to thread contention or interference between imported states. For example, the symbols `pkg.mod.func1` and `pkg.mod.func2` may conflict when importing `pkg` or `pkg.mod` ```text 2026-04-07 16:35:30.108 ERROR [suite-thread-1] (ScriptContext.groovy:122) - Run test_pythonudtf_data_types_module in /mnt/disk7/linzhenqi/dv/doris/regression-test/suites/pythonudtf_p0/test_pythonudtf_data_types_module.groovy failed java.sql.SQLException: errCode = 2, detailMessage = (127.0.0.1)[RUNTIME_ERROR]Flight stream finish failed with message: 'pyudtf_module'. Detail: Python exception: Traceback (most recent call last): File "pyarrow/_flight.pyx", line 2315, in pyarrow._flight._do_exchange File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 2499, in do_exchange self._handle_exchange_udtf(python_udf_meta, reader, writer) File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 2316, in _handle_exchange_udtf adaptive_udtf = loader.load() ^^^^^^^^^^^^^ File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 877, in load func = self.load_udf_from_module(location, package_name, module_name, func_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 1135, in load_udf_from_module return self._load_package_udf( ^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 1088, in _load_package_udf udf_module = self._get_or_import_module(location, full_module_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 978, in _get_or_import_module module = importlib.import_module(full_module_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/disk7/linzhenqi/miniconda3/envs/py312/lib/python3.12/importlib/__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1387, in _gcd_import File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 1387, in _gcd_import File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 946, in _load_unlocked KeyError: 'pyudtf_module' ... 2026-04-07 16:35:30.169 ERROR [suite-thread-6] (ScriptContext.groovy:122) - Run test_pythonudtf_exceptions_module in /mnt/disk7/linzhenqi/dv/doris/regression-test/suites/pythonudtf_p0/test_pythonudtf_exceptions_module.groovy failed java.sql.SQLException: errCode = 2, detailMessage = (127.0.0.1)[RUNTIME_ERROR]Flight stream finish failed with message: 'pyudtf_module'. Detail: Python exception: Traceback (most recent call last): File "pyarrow/_flight.pyx", line 2315, in pyarrow._flight._do_exchange File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 2499, in do_exchange self._handle_exchange_udtf(python_udf_meta, reader, writer) File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 2316, in _handle_exchange_udtf adaptive_udtf = loader.load() ^^^^^^^^^^^^^ File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 877, in load func = self.load_udf_from_module(location, package_name, module_name, func_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 1135, in load_udf_from_module return self._load_package_udf( ^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 1088, in _load_package_udf udf_module = self._get_or_import_module(location, full_module_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 978, in _get_or_import_module module = importlib.import_module(full_module_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/disk7/linzhenqi/miniconda3/envs/py312/lib/python3.12/importlib/__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1387, in _gcd_import File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1337, in _find_and_load_unlocked KeyError: 'pyudtf_module' ``` |
||
|
linrrarity
|
7827e1bd03 | [Enhancement](mmhash) Support mmhash3_u64_v2 (#61846) | ||
|
linrrarity
|
5fc3698fa5 |
[Fix](pyudf) Fix error type conversion (#61729)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
1. `LargeInt` is mapped to arrow's `str` type, but no conversion is
performed in python_server which leads to an error due to the mismatch
between the final result and the expect return type(pa.str) after the
internal calculation is converted to int.
```text
Doris> CREATE FUNCTION py_largeint_inc(LARGEINT)
-> RETURNS LARGEINT
-> PROPERTIES (
-> "type"="PYTHON_UDF",
-> "symbol"="evaluate",
-> "runtime_version"="3.12.11",
-> "always_nullable"="true"
-> )
-> AS $$
-> def evaluate(x):
-> if x is None:
-> return None
-> return int(x) + 1
-> $$;
Query OK, 0 rows affected (0.024 sec)
Doris> SELECT py_largeint_inc(1);
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[RUNTIME_ERROR]Expected bytes, got a 'int' object. Detail: Python exception: Traceback (most recent call last):
File "pyarrow/_flight.pyx", line 2315, in pyarrow._flight._do_exchange
File "/mnt/disk7/linzhenqi/d1/doris/output/be/plugins/python_udf/python_server.py", line 2481, in do_exchange
self._handle_exchange_udf(python_udf_meta, reader, writer)
File "/mnt/disk7/linzhenqi/d1/doris/output/be/plugins/python_udf/python_server.py", line 2013, in _handle_exc
```
now:
```text
Doris> select py_largeint_inc(1);
+--------------------+
| py_largeint_inc(1) |
+--------------------+
| 2 |
+--------------------+
```
2. Udaf does not maintain metadata for special types
(IPv4/IPv6/LargeInt), causing the input_type to be passed and not
processed normally.
```text
Doris> CREATE AGGREGATE FUNCTION udaf_count_loopback_ipv6_inline(IPV6)
-> RETURNS BIGINT
-> PROPERTIES (
-> "type" = "PYTHON_UDF",
-> "symbol" = "CountLoopbackIPv6UDAF",
-> "runtime_version" = "3.9.18"
-> )
-> AS $$
-> class CountLoopbackIPv6UDAF:
-> def __init__(self):
-> self.count = 0
->
-> def accumulate(self, value):
-> if value is not None and value.is_loopback:
-> self.count += 1
->
-> def merge(self, other_state):
-> if other_state is not None:
-> self.count += other_state
->
-> def finish(self):
-> return self.count
->
-> @property
-> def aggregate_state(self):
-> return self.count
-> $$;
Query OK, 0 rows affected (0.004 sec)
Doris> SELECT udaf_count_loopback_ipv6_inline(ip_v6) as loopback_ipv6_count
-> FROM test_pythonudaf_ipv6_table;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR][INTERNAL_ERROR]ACCUMULATE operation failed for place_id=140343167410176
0# doris::PythonUDAFClient::accumulate(long, bool, arrow::RecordBatch const&, long, long) at ../src/common/status.h:500
1# doris::AggregatePythonUDAFData::add(long, doris::IColumn const**, long, long, std::vector<std::shared_ptr<doris::IDataType const>, std::allocator<std::shared_ptr<doris::IDataType const> > > const&) at ./be/build_RELEASE/../src/exprs/aggregate/aggregate_f
```
now
```text
Doris> SELECT udaf_count_loopback_ipv6_inline(ip_v6) as loopback_ipv6_count
-> FROM test_pythonudaf_ipv6_table;
+---------------------+
| loopback_ipv6_count |
+---------------------+
| 1 |
+---------------------+
```
|
||
|
linrrarity
|
b34106dcdf |
[Feature](func) Support table function json_each, json_each_text (#60910)
doc: https://github.com/apache/doris-website/pull/3422 ```text Doris> SELECT k, v -> FROM (SELECT 1) dummy -> LATERAL VIEW json_each('{"a":"foo","b":"bar"}') t AS k, v; +------+-------+ | k | v | +------+-------+ | a | "foo" | | b | "bar" | +------+-------+ 2 rows in set (0.01 sec) Doris> SELECT k, v -> FROM (SELECT 1) dummy -> LATERAL VIEW json_each_text('{"a":"foo","b":"bar"}') t AS k, v; +------+------+ | k | v | +------+------+ | a | foo | | b | bar | +------+------+ 2 rows in set (0.01 sec) ``` |
||
|
linrrarity
|
9326ce7923 |
[Fix](pyudf) Fix concurrent race condition when import module (#61280)
### What problem does this PR solve? Issue Number: close #xxx Related PR: https://github.com/apache/doris/pull/60630 Problem Summary: ### Release note 1. sys.modules uses `modules_name` as a key to distinguish imported modules, but the current behavior uses `location + modules_name` as a key to differentiate and acquire a lock for lib loading, which can cause a concurrent race condition when two threads import the same named module from different locations at the same time. 2. Currently, importing the same package with different file paths causes issues. ```text Doris> CREATE FUNCTION func1(INT) -> RETURNS INT -> PROPERTIES ( -> "type" = "PYTHON_UDF", -> "file" = "file:///mnt/disk7/linzhenqi/dv/path_a/pkg.zip", -> "symbol" = "pkg.mdu_a.func", -> "runtime_version" = "3.9.18", -> "always_nullable" = "false" -> ); Query OK, 0 rows affected (0.02 sec) Doris> select func1(1); +----------+ | func1(1) | +----------+ | 2 | +----------+ 1 row in set (2.70 sec) Doris> CREATE FUNCTION func10(INT) -> RETURNS INT -> PROPERTIES ( -> "type" = "PYTHON_UDF", -> "file" = "file:///mnt/disk7/linzhenqi/dv/path_b/pkg.zip", -> "symbol" = "pkg.mdu_b.func", -> "runtime_version" = "3.9.18", -> "always_nullable" = "false" -> ); Query OK, 0 rows affected (0.01 sec) Doris> select func10(1); ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[RUNTIME_ERROR]Failed to load packaged UDF 'pkg.mdu_b.func': No module named 'pkg.mdu_b'. Detail: Python exception: Traceback (most recent call last): File "/mnt/disk7/linzhenqi/d4/doris/output/be/plugins/python_udf/python_server.py", line 1058, in _load_package_udf udf_module = self._get_or_import_module(location, full_module_name) File "/mnt/disk7/linzhenqi/d4/doris/output/be/plugins/python_udf/python_server.py", line 949, in _get_or_import_module modul -- pkg does not exist in path_b mdu_a Doris> CREATE FUNCTION func_no_module(INT) -> RETURNS INT -> PROPERTIES ( -> "type" = "PYTHON_UDF", -> "file" = "file:///mnt/disk7/linzhenqi/dv/path_b/pkg.zip", -> "symbol" = "pkg.mdu_a.func", -> "runtime_version" = "3.9.18", -> "always_nullable" = "false" -> ); Query OK, 0 rows affected (0.00 sec) -- The pkg cache was used by mistake, and the pkg of the func1 cache was found. Doris> select func_no_module(1); +-------------------+ | func_no_module(1) | +-------------------+ | 2 | +-------------------+ 1 row in set (0.04 sec) ``` In the `python_server`, the `location` is the cache directory, composed of the `function_id`. Thus, the internal `module_cache` only needs `location` as its key. After importing a cache, immediately remove the mapping maintained in `sys`. now ```text ``` |
||
|
linrrarity
|
7064d4b58b |
[Fix](p0) Fixed the issue that testFoldConstant was interfered by enable_sql_cache and could not be tested correctly. (#61328)
`testFoldConst `used in the p0 regression, due to `enable_sql_cache`, causes the second query to directly reuse the result of the first query, thus failing to correctly perform the correctness test of constant folding. limit `enable_sql_cache = false` in `testFoldConst, and fix the constant folding implementation error in maketime. ```sql Doris> set debug_skip_fold_constant=false; Query OK, 0 rows affected (0.00 sec) Doris> SELECT MAKETIME(111, 0, 23.1234567); +------------------------------+ | MAKETIME(111, 0, 23.1234567) | +------------------------------+ | 00:00:00.399623 | +------------------------------+ ``` now ```sql Doris> set debug_skip_fold_constant=false; Query OK, 0 rows affected (0.00 sec) Doris> SELECT MAKETIME(111, 0, 23.1234567); +------------------------------+ | MAKETIME(111, 0, 23.1234567) | +------------------------------+ | 111:00:23.123457 | +------------------------------+ ``` |
||
|
linrrarity
|
406c4833b1 | [Fix](tz) Fix some issues about timezone (#61151) | ||
|
linrrarity
|
2a680f5fca |
[Fix](pythonUdf) Fix pythonUdf helper cmd coredump when enable_python_udf_support is not set (#61196)
When `enable_python_udf_support` is not set to true, a null pointer
dereference occurs when invoking helper commands (`SHOW PYTHON VERSION/
PACKAGES IN`).
```text
/home/zcp/repo_center/doris_master/doris/be/src/udf/python/python_env.cpp:305:38: runtime error: member call on null pointer of type 'doris::PythonEnvScanner'
#0 0x559edede162c in doris::PythonVersionManager::env_infos_to_thrift() const /home/zcp/repo_center/doris_master/doris/be/src/udf/python/python_env.cpp:305:38
#1 0x559ede63acd4 in doris::BaseBackendService::get_python_envs(std::vector>&) /home/zcp/repo_center/doris_master/doris/be/src/service/backend_service.cpp:1316:47
#2 0x559edf99796a in doris::BackendServiceProcessor::process_get_python_envs(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/BackendService.cpp:7789:13
#3 0x559edf94a69c in doris::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string, std::allocator> const&, int, void*) /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/BackendService.cpp:6466:3
#4 0x559eac6bad06 in apache::thrift::TDispatchProcessor::process(std::shared_ptr, std::shared_ptr, void*) /home/zcp/repo_center/doris_master/doris/thirdparty/installed/include/thrift/TDispatchProcessor.h:121:12
#5 0x559ee47c071a in apache::thrift::server::TConnectedClient::run() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1c871a)
#6 0x559ee47c1a45 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1c9a45)
#7 0x559ee47c557f in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd57f)
#8 0x559ee47c52a5 in void std::__invoke_impl), std::shared_ptr>(std::__invoke_other, void (*&&)(std::shared_ptr), std::shared_ptr&&) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd2a5)
#9 0x559ee47c521c in std::__invoke_result), std::shared_ptr>::type std::__invoke), std::shared_ptr>(void (*&&)(std::shared_ptr), std::shared_ptr&&) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd21c)
#10 0x559ee47c51f1 in void std::thread::_Invoker), std::shared_ptr>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd1f1)
#11 0x559ee47c51b4 in std::thread::_Invoker), std::shared_ptr>>::operator()() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd1b4)
#12 0x559ee47c4fd8 in std::thread::_State_impl), std::shared_ptr>>>::_M_run() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1ccfd8)
#13 0x559ef61b5f3f in execute_native_thread_routine archive64.c
#14 0x559eac535d26 in asan_thread_start(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x51f3dd26)
#15 0x7fcba62bfac2 in start_thread nptl/pthread_create.c:442:8
#16 0x7fcba635184f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/zcp/repo_center/doris_master/doris/be/src/udf/python/python_env.cpp:305:38
```
now:
```text
Doris> show python packages in '3.19.18';
ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to get python packages from any backend: [E42] Set 'python_venv_interpreter_paths' in be.conf to enable PythonUDF feature
Doris> show python versions;
ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to get python envs from any backend: [E-236] Set 'python_venv_interpreter_paths' in be.conf to enable PythonUDF feature
```
|
||
|
linrrarity
|
e95a17ef80 |
[Enhancement](udf) clear cache when droping function (#60630)
1. drop python udf file cache when drop function
2. Fixed the issue where the same-named module from two different paths
would incorrectly use the cache.
```sql
-- same1(x) -> x + 1
-- same2(x) -> x + 2
Doris> CREATE FUNCTION same1(INT)
-> RETURNS INT
-> PROPERTIES (
-> "type" = "PYTHON_UDF",
-> "file" = "file:///mnt/disk7/linzhenqi/d2/same_name_py_test/1/1.zip",
-> "symbol" = "same_name.same_name",
-> "runtime_version" = "3.9.18",
-> "always_nullable" = "false"
-> );
Query OK, 0 rows affected (0.04 sec)Doris> select same1(1);
+----------+
| same1(1) |
+----------+
| 2 |
+----------+
1 row in set (6.25 sec)
Doris> CREATE FUNCTION same2(INT)
-> RETURNS INT
-> PROPERTIES (
-> "type" = "PYTHON_UDF",
-> "file" = "file:///mnt/disk7/linzhenqi/d2/same_name_py_test/2/2.zip",
-> "symbol" = "same_name.same_name",
-> "runtime_version" = "3.9.18",
-> "always_nullable" = "false"
-> );
Query OK, 0 rows affected (0.00 sec)
-- due to the same module name, it incorrectly hit the cache of same1.
Doris> select same2(1);
+----------+
| same2(1) |
+----------+
| 2 |
+----------+
1 row in set (0.09 sec)
```
|
||
|
linrrarity
|
3bf910c795 | [Enhancement](pyudf) Support MemTracker in PythonUdf (#60655) | ||
|
linrrarity
|
e8bc244aa5 |
[Enhancement](pyudf) add helper command to show more py info (#60751)
doc: https://github.com/apache/doris-website/pull/3402 Add the auxiliary commands `SHOW PYTHON VERSIONS` and `SHOW PYTHON PACKAGES IN '<VERSION>'` to display more PythonUDF-related information. ```sql Doris> show python versions; +---------+---------+---------+---------------------------------------------+--------------------------------------------------------+ | Version | EnvName | EnvType | BasePath | ExecutablePath | +---------+---------+---------+---------------------------------------------+--------------------------------------------------------+ | 3.9.18 | py39 | conda | /mnt/disk7/linzhenqi/miniconda3/envs/py39 | /mnt/disk7/linzhenqi/miniconda3/envs/py39/bin/python | | 3.8.10 | py3810 | conda | /mnt/disk7/linzhenqi/miniconda3/envs/py3810 | /mnt/disk7/linzhenqi/miniconda3/envs/py3810/bin/python | | 3.12.11 | py312 | conda | /mnt/disk7/linzhenqi/miniconda3/envs/py312 | /mnt/disk7/linzhenqi/miniconda3/envs/py312/bin/python | +---------+---------+---------+---------------------------------------------+--------------------------------------------------------+ 3 rows in set (0.02 sec) Doris> show python packages in '3.9.18'; +-----------------+-------------+ | Package | Version | +-----------------+-------------+ | pyarrow | 21.0.0 | | Bottleneck | 1.4.2 | | jieba | 0.42.1 | | six | 1.17.0 | | wheel | 0.45.1 | | python-dateutil | 2.9.0.post0 | | tzdata | 2025.3 | | setuptools | 80.9.0 | | numpy | 2.0.1 | | psutil | 7.0.0 | | pandas | 2.3.3 | | mkl_random | 1.2.8 | | pip | 25.3 | | snownlp | 0.12.3 | | pytz | 2025.2 | | mkl_fft | 1.3.11 | | mkl-service | 2.4.0 | | numexpr | 2.10.1 | +-----------------+-------------+ ``` |
||
|
linrrarity
|
888a046009 |
[Fix](regexp) make dot match newline in regexp_fn by default (#60831)
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note doc: https://github.com/apache/doris-website/pull/3410 Keep the behavior of `regexp_fn` consistent with `regexp` before: ```text Doris> SELECT REGEXP_EXTRACT('foo\nbar', '^(.+)$', 1); +-----------------------------------------+ | REGEXP_EXTRACT('foo\nbar', '^(.+)$', 1) | +-----------------------------------------+ | | +-----------------------------------------+ ``` After ```text Doris> SELECT REGEXP_EXTRACT('foo\nbar', '^(.+)$', 1); +-----------------------------------------+ | REGEXP_EXTRACT('foo\nbar', '^(.+)$', 1) | +-----------------------------------------+ | foo bar | +-----------------------------------------+ ``` None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> |
||
|
linrrarity
|
6c07a1c910 |
[Fix](func) Fixing microsecond_from_unixtime decimal arg type to DECIMAL(18, 6) (#60829)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
```sql
mysql> SELECT MICROSECOND_FROM_UNIXTIME(1145.14);
+------------------------------------+
| MICROSECOND_FROM_UNIXTIME(1145.14) |
+------------------------------------+
| 140000 |
+------------------------------------+
1 row in set (0.02 sec)
mysql> select @@debug_skip_fold_constant;
+----------------------------+
| @@debug_skip_fold_constant |
+----------------------------+
| 0 |
+----------------------------+
1 row in set (0.01 sec)
mysql> set debug_skip_fold_constant=true;
Query OK, 0 rows affected (0.02 sec)
mysql> SELECT MICROSECOND_FROM_UNIXTIME(1145.14);
ERROR 1105 (HY000): RpcException, msg: send fragments failed. io.grpc.StatusRuntimeException: UNAVAILABLE: Keepalive failed. The connection is likely gone, host: 10.16.10.3
```
```text
F20260225 00:04:40.698660 618581 status.h:467] Bad cast from type:doris::vectorized::ColumnDecimal<(doris::PrimitiveType)28>* to doris::vectorized::ColumnDecimal<(doris::PrimitiveType)29> const*
*** Check failure stack trace: ***
@ 0x55dbc5c4dc4f google::LogMessage::SendToLog()
@ 0x55dbc5c44260 google::LogMessage::Flush()
@ 0x55dbc5c47959 google::LogMessageFatal::~LogMessageFatal()
@ 0x55db8f09709b doris::Status::FatalError<>()
@ 0x55dbb7936b66 _ZZ11assert_castIPKN5doris10vectorized13ColumnDecimalILNS0_13PrimitiveTypeE29EEEL18TypeCheckOnRelease1EPKNS1_7IColumnEET_OT1_ENKUlOSB_E_clISA_EES6_SE_
@ 0x55dbb7935f74 assert_cast<>()
@ 0x55dbbcbf2f1f doris::vectorized::FunctionTimeFieldFromUnixtime<>::execute_impl()
@ 0x55dbbcbf3762 doris::vectorized::FunctionTimeFieldFromUnixtime<>::execute_impl()
@ 0x55dbba5ca2d8 doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal()
@ 0x55dbba5ad094 doris::vectorized::PreparedFunctionImpl::default_implementation_for_constant_arguments()
@ 0x55dbba5af1fd doris::vectorized::PreparedFunctionImpl::default_execute()
@ 0x55dbba5af624 doris::vectorized::PreparedFunctionImpl::execute()
```
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
|
||
|
linrrarity
|
c20d38dd9f |
[Enhancement](udf) SHOW FULL FUNCTION display more information (#60690)
Before
```text
+-----------+-------------+---------------+-------------------+------------+
| Signature | Return Type | Function Type | Intermediate Type | Properties |
+-----------+-------------+---------------+-------------------+------------+
| same2 | | | | |
+-----------+-------------+---------------+-------------------+------------+
```
After
```text
+------------------------------------+-------------+----------------------+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lzq(int) | text | SCALAR/PYTHON_UDF | | ID=1770634239619, CHECKSUM=8c0e13ec9d1f3447ec2dd0efa0251b76, OBJECT_FILE=file:///mnt/disk7/linzhenqi/doooooris/lzq.zip, NULLABLE_MODE=ALWAYS_NOT_NULLABLE, RUNTIME_VERSION=3.9.18, SYMBOL=lzq_py_test.lzq_print |
| py_add(int, int) | int | SCALAR/PYTHON_UDF | | ID=1770634239620, NULLABLE_MODE=ALWAYS_NULLABLE, RUNTIME_VERSION=3.10.12, SYMBOL=evaluate, INLINE_CODE=
def evaluate(a, b):
return a + b
|
| py_split_string_module(text) | text | UDTF/PYTHON_UDF | | ID=1770871349701, CHECKSUM=c6230f1580bfdb4639ebcf7042119dcf, OBJECT_FILE=file:///mnt/disk7/linzhenqi/d4/doris/regression-test/suites/pythonudtf_p0/udtf_scripts/pyudtf.zip, NULLABLE_MODE=ALWAYS_NULLABLE, RUNTIME_VERSION=3.8.10, SYMBOL=pyudtf_module.basic_udtf.split_string_udtf |
| py_split_string_module_outer(text) | text | UDTF/PYTHON_UDF | | ID=1770871349702, CHECKSUM=c6230f1580bfdb4639ebcf7042119dcf, OBJECT_FILE=file:///mnt/disk7/linzhenqi/d4/doris/regression-test/suites/pythonudtf_p0/udtf_scripts/pyudtf.zip, NULLABLE_MODE=ALWAYS_NULLABLE, RUNTIME_VERSION=3.8.10, SYMBOL=pyudtf_module.basic_udtf.split_string_udtf |
| python_udaf_sum_int(int) | bigint | AGGREGATE/PYTHON_UDF | | ID=1770871349699, CHECKSUM=8968fc20fc629d1e193359c235927bed, OBJECT_FILE=file:///mnt/disk7/linzhenqi/ddoooooris/doris/regression-test/suites/pythonudaf_p0/udaf_scripts/pyudaf.zip, NULLABLE_MODE=ALWAYS_NULLABLE, RUNTIME_VERSION=3.8.10, INIT_FN=, UPDATE_FN=, MERGE_FN=, SYMBOL=sum_int.SumInt |
+------------------------------------+-------------+----------------------+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
|
||
|
linrrarity
|
48fc0ae2bf |
[Fix](keywords) add composite time units to nonReserved (#60775)
### What problem does this PR solve? Issue Number: close #xxx Related PR: #60347 Problem Summary: Before: ```text Doris> select now() day_hour; ERROR 1105 (HY000): errCode = 2, detailMessage = extraneous input 'day_hour' expecting {<EOF>, ';'}(line 1, pos 13) ``` After: ```text Doris> select now() day_hour; +---------------------+ | day_hour | +---------------------+ | 2026-02-16 02:31:36 | +---------------------+ ``` ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> |
||
|
linrrarity
|
b5b09c2469 |
[Enhancement](func) Add fold constant for PREVIOUS_DAY (#60755)
Related PR: https://github.com/apache/doris/pull/60680 |
||
|
linrrarity
|
5699e269ca |
[Feature](func) Support function PREVIOUS_DAY (#60680)
### What problem does this PR solve? Issue Number: close #xxx Related PR: ### Release note doc: https://github.com/apache/doris-website/pull/3390 Given an input date value, return the date of the day of the week before it. ```text Doris> select previous_day('2023-04-06', 'Monday'); +--------------------------------------+ | previous_day('2023-04-06', 'Monday') | +--------------------------------------+ | 2023-04-03 | +--------------------------------------+ 1 row in set (0.13 sec) Doris> select previous_day('2023-04-06', 'Tue'); +-----------------------------------+ | previous_day('2023-04-06', 'Tue') | +-----------------------------------+ | 2023-04-04 | +-----------------------------------+ 1 row in set (0.04 sec) Doris> select previous_day('2023-04-06 20:13:14', 'Fr'); +-------------------------------------------+ | previous_day('2023-04-06 20:13:14', 'Fr') | +-------------------------------------------+ | 2023-03-31 | +-------------------------------------------+ 1 row in set (0.04 sec) ``` None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [x] Yes. https://github.com/apache/doris-website/pull/3390 ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> |
