mirror of
https://github.com/apache/doris.git
synced 2026-11-04 18:02:19 +00:00
yangshijieandyangshijie
4a37aec7d2
This PR references ByteDance's implementation. Co-Author: @WencongLiu ## Description This PR introduces native support for **Python User-Defined Functions (UDF), User-Defined Aggregate Functions (UDAF), and User-Defined Table Functions (UDTF)** in Doris, enabling users to extend SQL capabilities with custom Python logic for complex data processing scenarios. ## Key Features ### 🚀 **Three Function Types** - **UDF**: Scalar functions with row-by-row or vectorized execution (3-10x performance gain with Pandas/Arrow mode) - **UDAF**: Snowflake-style stateful aggregation with distributed merge support - **UDTF**: Table-valued functions that generate multiple output rows from a single input row ### 🔧 **Production-Grade Architecture** - **High-Performance Communication**: Arrow Flight RPC over Unix sockets with zero-copy columnar data transfer - **Multi-Version Support**: Flexible environment management via Conda or venv, allowing different UDFs to use different Python versions (e.g., 3.9, 3.10, 3.12) ### 🎯 **Deep Integration** - Seamless integration with Doris vectorized execution engine - Native support for Doris data types (including complex types like ARRAY, MAP, STRUCT) - Automatic conversion between Doris types and Python/Arrow types ## Architecture Highlights ``` ┌──────────────────────────────────────────────────────┐ │ Doris BE (C++) │ │ ┌────────────────────────────────────────────────┐ │ │ │ PythonServerManager (Process Pool) │ │ │ │ ├─ Health Check Thread (60s interval) │ │ │ │ ├─ Load Balancing (min ref count) │ │ │ │ └─ Auto Recovery (dead process detection) │ │ │ └────────────────────────────────────────────────┘ │ │ ┌────────────────────────────────────────────────┐ │ │ │ PythonClient (Arrow Flight RPC) │ │ │ │ ├─ UDF: Scalar/Vectorized evaluation │ │ │ │ ├─ UDAF: Stateful aggregation with merge │ │ │ │ └─ UDTF: ListArray batch processing │ │ │ └────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────┘ ↕ Unix Socket ┌──────────────────────────────────────────────────────┐ │ Python Process (python_server.py) │ │ ┌────────────────────────────────────────────────┐ │ │ │ FlightServer (Arrow Flight bidirectional) │ │ │ │ ├─ AdaptivePythonUDF (auto mode selection) │ │ │ │ ├─ UDAFStateManager (Snowflake interface) │ │ │ │ └─ UDFLoader (inline/module code execution) │ │ │ └────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────┘ ``` ## Configuration Add to `be.conf`: ```properties # Enable Python UDF support enable_python_udf_support = true # Choose environment management mode (conda or venv) python_env_mode = conda # For Conda mode python_conda_root_path = /path/to/miniconda3 # For venv mode python_env_mode = venv python_venv_root_path = /doris/python_envs python_venv_interpreter_paths = /opt/python3.9/bin/python3.9:/opt/python3.12/bin/python3.12 # Process pool size (0 = use CPU core count) max_python_process_num = 0 ``` ## Technical Highlights ### 1. Environment Management - **Multi-version support**: Each UDF can specify its own Python version - **Two modes**: Conda (full environment isolation) or venv (lightweight) - **Automatic discovery**: Scans available Python environments at BE startup ### 2. Process Pool Management - **Shared pool**: One pool per Python version, shared across all threads - **Load balancing**: Distributes requests to processes with minimum load - **Health monitoring**: Background thread checks process health every 60 seconds - **Auto recovery**: Automatically recreates dead processes ### 3. Communication Protocol - **Arrow Flight RPC**: High-performance, language-agnostic RPC framework - **Unix Socket**: Local IPC for minimal latency and enhanced security - **Bidirectional streaming**: Efficient batch data transfer ### 4. Execution Modes - **Scalar mode**: Process one value at a time (simple functions) - **Vectorized mode**: Process entire columns with NumPy/Pandas (10-100x faster) - **Adaptive selection**: Automatically chooses mode based on function signature ### 5. UDAF State Management (Snowflake Style) - **5 lifecycle methods**: `__init__`, `accumulate`, `merge`, `finish`, `aggregate_state` - **Distributed aggregation**: Serialization/deserialization for shuffle operations - **Efficient state handling**: Place-based mapping avoids redundant transfers ## Limitations 1. **Performance**: Python UDFs are slower than native C++ built-in functions. Best suited for complex logic that's difficult to implement in SQL. 2. **Type support**: Special Doris types like HLL and Bitmap are not yet supported. 3. **Concurrency**: Parallelism is limited by `max_python_process_num` setting. ## Related Documentation - Python UDF User Guide - Python UDAF User Guide - Python UDTF User Guide - Python Environment Configuration Guide --- **This PR enables users to leverage the rich Python ecosystem (NumPy, Pandas, scikit-learn, etc.) directly within Doris SQL queries, significantly expanding the platform's data processing capabilities.** ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: yangshijie <jasonsjyang@tencent.com>
56 lines
1.3 KiB
Plaintext
56 lines
1.3 KiB
Plaintext
-- This file is automatically generated. You should know what you did if you want to edit this
|
|
-- !concurrent_scalar_1 --
|
|
1 30 200 0.6 hello_world 5
|
|
2 70 1200 0.7777777777777778 foo_bar 3
|
|
3 110 3000 0.8461538461538461 test_case 4
|
|
4 150 5600 0.8823529411764706 python_udf 6
|
|
5 190 9000 0.9047619047619048 doris_db 5
|
|
|
|
-- !concurrent_scalar_2 --
|
|
1 40 11
|
|
2 100 7
|
|
3 160 9
|
|
4 220 10
|
|
5 280 8
|
|
|
|
-- !concurrent_scalar_3 --
|
|
3 50 60
|
|
4 70 80
|
|
5 90 100
|
|
|
|
-- !concurrent_vector_1 --
|
|
1 30 10 20
|
|
2 70 10 40
|
|
3 110 10 60
|
|
4 150 10 80
|
|
5 190 10 100
|
|
|
|
-- !concurrent_mixed_1 --
|
|
1 30 30 hello_world 100
|
|
2 70 70 foo_bar 300
|
|
3 110 110 test_case 500
|
|
4 150 150 python_udf 700
|
|
5 190 190 doris_db 900
|
|
|
|
-- !concurrent_with_agg --
|
|
5 550 100 6
|
|
|
|
-- !concurrent_group_by --
|
|
group1 3 210 0.7413105413105413
|
|
group2 2 340 0.8935574229691876
|
|
|
|
-- !concurrent_type_mix --
|
|
1 30 0.6 10_hello
|
|
2 70 0.7777777777777778 30_foo
|
|
3 110 0.8461538461538461 50_test
|
|
4 150 0.8823529411764706 70_python
|
|
5 190 0.9047619047619048 90_doris
|
|
|
|
-- !concurrent_stress --
|
|
1 30 200 110 100 0.6 1.666666666666667 hello_world 5 5 10
|
|
2 70 1200 130 200 0.7777777777777778 1.285714285714286 foo_bar 3 3 30
|
|
3 110 3000 150 300 0.8461538461538461 1.181818181818182 test_case 4 4 50
|
|
4 150 5600 170 400 0.8823529411764706 1.133333333333333 python_udf 6 3 70
|
|
5 190 9000 190 500 0.9047619047619048 1.105263157894737 doris_db 5 2 90
|
|
|