mirror of
https://github.com/apache/doris.git
synced 2026-11-04 18:29:59 +00:00
yangshijieandyangshijie
4a37aec7d2
This PR references ByteDance's implementation. Co-Author: @WencongLiu ## Description This PR introduces native support for **Python User-Defined Functions (UDF), User-Defined Aggregate Functions (UDAF), and User-Defined Table Functions (UDTF)** in Doris, enabling users to extend SQL capabilities with custom Python logic for complex data processing scenarios. ## Key Features ### 🚀 **Three Function Types** - **UDF**: Scalar functions with row-by-row or vectorized execution (3-10x performance gain with Pandas/Arrow mode) - **UDAF**: Snowflake-style stateful aggregation with distributed merge support - **UDTF**: Table-valued functions that generate multiple output rows from a single input row ### 🔧 **Production-Grade Architecture** - **High-Performance Communication**: Arrow Flight RPC over Unix sockets with zero-copy columnar data transfer - **Multi-Version Support**: Flexible environment management via Conda or venv, allowing different UDFs to use different Python versions (e.g., 3.9, 3.10, 3.12) ### 🎯 **Deep Integration** - Seamless integration with Doris vectorized execution engine - Native support for Doris data types (including complex types like ARRAY, MAP, STRUCT) - Automatic conversion between Doris types and Python/Arrow types ## Architecture Highlights ``` ┌──────────────────────────────────────────────────────┐ │ Doris BE (C++) │ │ ┌────────────────────────────────────────────────┐ │ │ │ PythonServerManager (Process Pool) │ │ │ │ ├─ Health Check Thread (60s interval) │ │ │ │ ├─ Load Balancing (min ref count) │ │ │ │ └─ Auto Recovery (dead process detection) │ │ │ └────────────────────────────────────────────────┘ │ │ ┌────────────────────────────────────────────────┐ │ │ │ PythonClient (Arrow Flight RPC) │ │ │ │ ├─ UDF: Scalar/Vectorized evaluation │ │ │ │ ├─ UDAF: Stateful aggregation with merge │ │ │ │ └─ UDTF: ListArray batch processing │ │ │ └────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────┘ ↕ Unix Socket ┌──────────────────────────────────────────────────────┐ │ Python Process (python_server.py) │ │ ┌────────────────────────────────────────────────┐ │ │ │ FlightServer (Arrow Flight bidirectional) │ │ │ │ ├─ AdaptivePythonUDF (auto mode selection) │ │ │ │ ├─ UDAFStateManager (Snowflake interface) │ │ │ │ └─ UDFLoader (inline/module code execution) │ │ │ └────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────┘ ``` ## Configuration Add to `be.conf`: ```properties # Enable Python UDF support enable_python_udf_support = true # Choose environment management mode (conda or venv) python_env_mode = conda # For Conda mode python_conda_root_path = /path/to/miniconda3 # For venv mode python_env_mode = venv python_venv_root_path = /doris/python_envs python_venv_interpreter_paths = /opt/python3.9/bin/python3.9:/opt/python3.12/bin/python3.12 # Process pool size (0 = use CPU core count) max_python_process_num = 0 ``` ## Technical Highlights ### 1. Environment Management - **Multi-version support**: Each UDF can specify its own Python version - **Two modes**: Conda (full environment isolation) or venv (lightweight) - **Automatic discovery**: Scans available Python environments at BE startup ### 2. Process Pool Management - **Shared pool**: One pool per Python version, shared across all threads - **Load balancing**: Distributes requests to processes with minimum load - **Health monitoring**: Background thread checks process health every 60 seconds - **Auto recovery**: Automatically recreates dead processes ### 3. Communication Protocol - **Arrow Flight RPC**: High-performance, language-agnostic RPC framework - **Unix Socket**: Local IPC for minimal latency and enhanced security - **Bidirectional streaming**: Efficient batch data transfer ### 4. Execution Modes - **Scalar mode**: Process one value at a time (simple functions) - **Vectorized mode**: Process entire columns with NumPy/Pandas (10-100x faster) - **Adaptive selection**: Automatically chooses mode based on function signature ### 5. UDAF State Management (Snowflake Style) - **5 lifecycle methods**: `__init__`, `accumulate`, `merge`, `finish`, `aggregate_state` - **Distributed aggregation**: Serialization/deserialization for shuffle operations - **Efficient state handling**: Place-based mapping avoids redundant transfers ## Limitations 1. **Performance**: Python UDFs are slower than native C++ built-in functions. Best suited for complex logic that's difficult to implement in SQL. 2. **Type support**: Special Doris types like HLL and Bitmap are not yet supported. 3. **Concurrency**: Parallelism is limited by `max_python_process_num` setting. ## Related Documentation - Python UDF User Guide - Python UDAF User Guide - Python UDTF User Guide - Python Environment Configuration Guide --- **This PR enables users to leverage the rich Python ecosystem (NumPy, Pandas, scikit-learn, etc.) directly within Doris SQL queries, significantly expanding the platform's data processing capabilities.** ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: yangshijie <jasonsjyang@tencent.com>
53 lines
875 B
Plaintext
53 lines
875 B
Plaintext
-- This file is automatically generated. You should know what you did if you want to edit this
|
|
-- !mixed_1 --
|
|
1060 8 200
|
|
|
|
-- !mixed_2 --
|
|
BOO 190 3
|
|
CLO 270 2
|
|
ELE 600 3
|
|
|
|
-- !mixed_3 --
|
|
Books 380 3
|
|
Clothing 540 2
|
|
Electronics 1200 3
|
|
|
|
-- !mixed_4 --
|
|
BOO 190 3 39.9
|
|
CLO 270 2 69.90000000000001
|
|
ELE 600 3 199.9
|
|
|
|
-- !mixed_5 --
|
|
Books 190 36.67433333333333 3
|
|
Clothing 270 57.4175 2
|
|
Electronics 600 161.5833333333333 3
|
|
|
|
-- !mixed_6 --
|
|
Books BOO 380 3 39.9
|
|
Clothing CLO 540 2 69.90000000000001
|
|
Electronics ELE 1200 3 199.9
|
|
|
|
-- !mixed_7 --
|
|
Books 190 3
|
|
Clothing 270 2
|
|
Electronics 600 3
|
|
|
|
-- !mixed_8 --
|
|
2120 8
|
|
|
|
-- !mixed_9 --
|
|
Books 140 2
|
|
Clothing 270 2
|
|
Electronics 600 3
|
|
|
|
-- !mixed_10 --
|
|
BOO 190 380 3 39.9 36.67433333333333
|
|
CLO 270 540 2 69.90000000000001 57.4175
|
|
ELE 600 1200 3 199.9 161.5833333333333
|
|
|
|
-- !mixed_verify --
|
|
Books 190 190 3 3
|
|
Clothing 270 270 2 2
|
|
Electronics 600 600 3 3
|
|
|