mirror of
https://github.com/apache/doris.git
synced 2026-11-04 15:39:47 +00:00
linrrarity
5fc3698fa5
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
1. `LargeInt` is mapped to arrow's `str` type, but no conversion is
performed in python_server which leads to an error due to the mismatch
between the final result and the expect return type(pa.str) after the
internal calculation is converted to int.
```text
Doris> CREATE FUNCTION py_largeint_inc(LARGEINT)
-> RETURNS LARGEINT
-> PROPERTIES (
-> "type"="PYTHON_UDF",
-> "symbol"="evaluate",
-> "runtime_version"="3.12.11",
-> "always_nullable"="true"
-> )
-> AS $$
-> def evaluate(x):
-> if x is None:
-> return None
-> return int(x) + 1
-> $$;
Query OK, 0 rows affected (0.024 sec)
Doris> SELECT py_largeint_inc(1);
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[RUNTIME_ERROR]Expected bytes, got a 'int' object. Detail: Python exception: Traceback (most recent call last):
File "pyarrow/_flight.pyx", line 2315, in pyarrow._flight._do_exchange
File "/mnt/disk7/linzhenqi/d1/doris/output/be/plugins/python_udf/python_server.py", line 2481, in do_exchange
self._handle_exchange_udf(python_udf_meta, reader, writer)
File "/mnt/disk7/linzhenqi/d1/doris/output/be/plugins/python_udf/python_server.py", line 2013, in _handle_exc
```
now:
```text
Doris> select py_largeint_inc(1);
+--------------------+
| py_largeint_inc(1) |
+--------------------+
| 2 |
+--------------------+
```
2. Udaf does not maintain metadata for special types
(IPv4/IPv6/LargeInt), causing the input_type to be passed and not
processed normally.
```text
Doris> CREATE AGGREGATE FUNCTION udaf_count_loopback_ipv6_inline(IPV6)
-> RETURNS BIGINT
-> PROPERTIES (
-> "type" = "PYTHON_UDF",
-> "symbol" = "CountLoopbackIPv6UDAF",
-> "runtime_version" = "3.9.18"
-> )
-> AS $$
-> class CountLoopbackIPv6UDAF:
-> def __init__(self):
-> self.count = 0
->
-> def accumulate(self, value):
-> if value is not None and value.is_loopback:
-> self.count += 1
->
-> def merge(self, other_state):
-> if other_state is not None:
-> self.count += other_state
->
-> def finish(self):
-> return self.count
->
-> @property
-> def aggregate_state(self):
-> return self.count
-> $$;
Query OK, 0 rows affected (0.004 sec)
Doris> SELECT udaf_count_loopback_ipv6_inline(ip_v6) as loopback_ipv6_count
-> FROM test_pythonudaf_ipv6_table;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR][INTERNAL_ERROR]ACCUMULATE operation failed for place_id=140343167410176
0# doris::PythonUDAFClient::accumulate(long, bool, arrow::RecordBatch const&, long, long) at ../src/common/status.h:500
1# doris::AggregatePythonUDAFData::add(long, doris::IColumn const**, long, long, std::vector<std::shared_ptr<doris::IDataType const>, std::allocator<std::shared_ptr<doris::IDataType const> > > const&) at ./be/build_RELEASE/../src/exprs/aggregate/aggregate_f
```
now
```text
Doris> SELECT udaf_count_loopback_ipv6_inline(ip_v6) as loopback_ipv6_count
-> FROM test_pythonudaf_ipv6_table;
+---------------------+
| loopback_ipv6_count |
+---------------------+
| 1 |
+---------------------+
```
105 lines
1.1 KiB
Plaintext
105 lines
1.1 KiB
Plaintext
-- This file is automatically generated. You should know what you did if you want to edit this
|
|
-- !select_data --
|
|
1 10 10.5 A
|
|
2 20 20.5 A
|
|
3 30 30.5 B
|
|
4 40 40.5 B
|
|
5 50 50.5 C
|
|
|
|
-- !test1 --
|
|
150 150
|
|
|
|
-- !test2 --
|
|
A 30 30
|
|
B 70 70
|
|
C 50 50
|
|
|
|
-- !test3 --
|
|
30.5 30.5
|
|
|
|
-- !test4 --
|
|
A 15.5 15.5
|
|
B 35.5 35.5
|
|
C 50.5 50.5
|
|
|
|
-- !test5 --
|
|
5 5
|
|
|
|
-- !test6 --
|
|
A 2 2
|
|
B 2 2
|
|
C 1 1
|
|
|
|
-- !test7 --
|
|
50 50
|
|
|
|
-- !test8 --
|
|
A 20 20
|
|
B 40 40
|
|
C 50 50
|
|
|
|
-- !test_null1 --
|
|
150 150
|
|
|
|
-- !test_null2 --
|
|
5 5
|
|
|
|
-- !test_null3 --
|
|
A 30 30
|
|
B 70 70
|
|
C 50 50
|
|
|
|
-- !test_window1 --
|
|
1 A 10 30 30
|
|
2 A 20 30 30
|
|
3 B 30 70 70
|
|
4 B 40 70 70
|
|
5 C 50 50 50
|
|
|
|
-- !test_window2 --
|
|
1 A 10 10 10
|
|
2 A 20 30 30
|
|
3 B 30 30 30
|
|
4 B 40 70 70
|
|
5 C 50 50 50
|
|
|
|
-- !test_multiple --
|
|
A 30 30 2 2 20 20 30.5 30.5
|
|
B 70 70 2 2 40 40 47.16666666666666 47.16666666666666
|
|
C 50 50 1 1 50 50 50.5 50.5
|
|
|
|
-- !test_global1 --
|
|
150 150
|
|
|
|
-- !test_global2 --
|
|
A 30 30
|
|
B 70 70
|
|
C 50 50
|
|
|
|
-- !test_empty_parallel --
|
|
0
|
|
|
|
-- !test_global_empty_parallel --
|
|
0
|
|
|
|
-- !test_largeint1 --
|
|
1000
|
|
|
|
-- !test_largeint2 --
|
|
A 300
|
|
B 700
|
|
|
|
-- !test_ipv4_udaf1 --
|
|
3
|
|
|
|
-- !test_ipv4_udaf2 --
|
|
A 2
|
|
B 1
|
|
|
|
-- !test_ipv6_udaf1 --
|
|
1
|
|
|
|
-- !test_ipv6_udaf2 --
|
|
A 1
|
|
B 0
|