mirror of
https://github.com/apache/doris.git
synced 2026-11-04 17:16:21 +00:00
yangshijieandyangshijie
4a37aec7d2
This PR references ByteDance's implementation. Co-Author: @WencongLiu ## Description This PR introduces native support for **Python User-Defined Functions (UDF), User-Defined Aggregate Functions (UDAF), and User-Defined Table Functions (UDTF)** in Doris, enabling users to extend SQL capabilities with custom Python logic for complex data processing scenarios. ## Key Features ### 🚀 **Three Function Types** - **UDF**: Scalar functions with row-by-row or vectorized execution (3-10x performance gain with Pandas/Arrow mode) - **UDAF**: Snowflake-style stateful aggregation with distributed merge support - **UDTF**: Table-valued functions that generate multiple output rows from a single input row ### 🔧 **Production-Grade Architecture** - **High-Performance Communication**: Arrow Flight RPC over Unix sockets with zero-copy columnar data transfer - **Multi-Version Support**: Flexible environment management via Conda or venv, allowing different UDFs to use different Python versions (e.g., 3.9, 3.10, 3.12) ### 🎯 **Deep Integration** - Seamless integration with Doris vectorized execution engine - Native support for Doris data types (including complex types like ARRAY, MAP, STRUCT) - Automatic conversion between Doris types and Python/Arrow types ## Architecture Highlights ``` ┌──────────────────────────────────────────────────────┐ │ Doris BE (C++) │ │ ┌────────────────────────────────────────────────┐ │ │ │ PythonServerManager (Process Pool) │ │ │ │ ├─ Health Check Thread (60s interval) │ │ │ │ ├─ Load Balancing (min ref count) │ │ │ │ └─ Auto Recovery (dead process detection) │ │ │ └────────────────────────────────────────────────┘ │ │ ┌────────────────────────────────────────────────┐ │ │ │ PythonClient (Arrow Flight RPC) │ │ │ │ ├─ UDF: Scalar/Vectorized evaluation │ │ │ │ ├─ UDAF: Stateful aggregation with merge │ │ │ │ └─ UDTF: ListArray batch processing │ │ │ └────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────┘ ↕ Unix Socket ┌──────────────────────────────────────────────────────┐ │ Python Process (python_server.py) │ │ ┌────────────────────────────────────────────────┐ │ │ │ FlightServer (Arrow Flight bidirectional) │ │ │ │ ├─ AdaptivePythonUDF (auto mode selection) │ │ │ │ ├─ UDAFStateManager (Snowflake interface) │ │ │ │ └─ UDFLoader (inline/module code execution) │ │ │ └────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────┘ ``` ## Configuration Add to `be.conf`: ```properties # Enable Python UDF support enable_python_udf_support = true # Choose environment management mode (conda or venv) python_env_mode = conda # For Conda mode python_conda_root_path = /path/to/miniconda3 # For venv mode python_env_mode = venv python_venv_root_path = /doris/python_envs python_venv_interpreter_paths = /opt/python3.9/bin/python3.9:/opt/python3.12/bin/python3.12 # Process pool size (0 = use CPU core count) max_python_process_num = 0 ``` ## Technical Highlights ### 1. Environment Management - **Multi-version support**: Each UDF can specify its own Python version - **Two modes**: Conda (full environment isolation) or venv (lightweight) - **Automatic discovery**: Scans available Python environments at BE startup ### 2. Process Pool Management - **Shared pool**: One pool per Python version, shared across all threads - **Load balancing**: Distributes requests to processes with minimum load - **Health monitoring**: Background thread checks process health every 60 seconds - **Auto recovery**: Automatically recreates dead processes ### 3. Communication Protocol - **Arrow Flight RPC**: High-performance, language-agnostic RPC framework - **Unix Socket**: Local IPC for minimal latency and enhanced security - **Bidirectional streaming**: Efficient batch data transfer ### 4. Execution Modes - **Scalar mode**: Process one value at a time (simple functions) - **Vectorized mode**: Process entire columns with NumPy/Pandas (10-100x faster) - **Adaptive selection**: Automatically chooses mode based on function signature ### 5. UDAF State Management (Snowflake Style) - **5 lifecycle methods**: `__init__`, `accumulate`, `merge`, `finish`, `aggregate_state` - **Distributed aggregation**: Serialization/deserialization for shuffle operations - **Efficient state handling**: Place-based mapping avoids redundant transfers ## Limitations 1. **Performance**: Python UDFs are slower than native C++ built-in functions. Best suited for complex logic that's difficult to implement in SQL. 2. **Type support**: Special Doris types like HLL and Bitmap are not yet supported. 3. **Concurrency**: Parallelism is limited by `max_python_process_num` setting. ## Related Documentation - Python UDF User Guide - Python UDAF User Guide - Python UDTF User Guide - Python Environment Configuration Guide --- **This PR enables users to leverage the rich Python ecosystem (NumPy, Pandas, scikit-learn, etc.) directly within Doris SQL queries, significantly expanding the platform's data processing capabilities.** ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: yangshijie <jasonsjyang@tencent.com>
113 lines
1.5 KiB
Plaintext
113 lines
1.5 KiB
Plaintext
-- This file is automatically generated. You should know what you did if you want to edit this
|
|
-- !select_1 --
|
|
1 10 1000 1010
|
|
2 20 2000 2020
|
|
3 30 3000 3030
|
|
4 40 4000 4040
|
|
5 50 5000 5050
|
|
|
|
-- !select_2 --
|
|
1 100 1000 1100
|
|
2 200 2000 2200
|
|
3 300 3000 3300
|
|
4 400 4000 4400
|
|
5 500 5000 5500
|
|
|
|
-- !select_3 --
|
|
1 1000 10000 11000
|
|
2 2000 20000 22000
|
|
3 3000 30000 33000
|
|
4 4000 40000 44000
|
|
5 5000 50000 55000
|
|
|
|
-- !select_4 --
|
|
1 1.5 10.5 12
|
|
2 2.5 20.5 23
|
|
3 3.5 30.5 34
|
|
4 4.5 40.5 45
|
|
5 5.5 50.5 56
|
|
|
|
-- !select_5 --
|
|
1 10 100 1000 1110
|
|
2 20 200 2000 2220
|
|
3 30 300 3000 3330
|
|
4 40 400 4000 4440
|
|
5 50 500 5000 5550
|
|
|
|
-- !select_6 --
|
|
1 10 100 1000
|
|
2 20 200 4000
|
|
3 30 300 9000
|
|
4 40 400 16000
|
|
5 50 500 25000
|
|
|
|
-- !select_7 --
|
|
1 1.5 10.5 7
|
|
2 2.5 20.5 8.199999999999999
|
|
3 3.5 30.5 8.714285714285714
|
|
4 4.5 40.5 9
|
|
5 5.5 50.5 9.181818181818182
|
|
|
|
-- !select_8 --
|
|
1 1000 1.5 2001.5
|
|
2 2000 2.5 4002.5
|
|
3 3000 3.5 6003.5
|
|
4 4000 4.5 8004.5
|
|
5 5000 5.5 10005.5
|
|
|
|
-- !select_9 --
|
|
1 test1 TEST1
|
|
2 test2 TEST2
|
|
3 test3 TEST3
|
|
4 test4 TEST4
|
|
5 test5 TEST5
|
|
|
|
-- !select_10 --
|
|
1 true false
|
|
2 false true
|
|
3 true false
|
|
4 false true
|
|
5 true false
|
|
|
|
-- !select_11 --
|
|
1 10 10000 10010
|
|
2 20 20000 20020
|
|
3 30 30000 30030
|
|
4 40 40000 40040
|
|
5 50 50000 50050
|
|
|
|
-- !select_12 --
|
|
1 1000 1500
|
|
2 2000 3000
|
|
3 3000 4500
|
|
4 4000 6000
|
|
5 5000 7500
|
|
|
|
-- !select_13 --
|
|
1 test1 \N
|
|
|
|
-- !select_14 --
|
|
1 10000 11000
|
|
|
|
-- !select_15 --
|
|
1 10.5 12.0
|
|
|
|
-- !select_16 --
|
|
1 true 1001
|
|
|
|
-- !select_17 --
|
|
1 2024-01-01 2024-01-01
|
|
|
|
-- !select_18 --
|
|
1 1000 false
|
|
|
|
-- !select_19 --
|
|
1 test1 \N
|
|
|
|
-- !select_20 --
|
|
1 test1 true \N
|
|
|
|
-- !select_22 --
|
|
1 1.5 1001
|
|
|