### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Document:
https://github.com/apache/doris-website/pull/4131https://github.com/apache/doris-website/pull/4166
Doris does not have a native UUID SQL and storage type, so UUID values
must be stored as strings and cannot use a fixed-width representation or
UUID-aware generation and version functions. Add a native 128-bit UUID
type across FE, BE, IDL, storage, serialization, expressions, runtime
filters, external and JNI bridges, and schema metadata. Accept canonical
and compact hexadecimal input, normalize output to lowercase canonical
form, preserve unsigned canonical ordering, and add native UUID v4 and
v7 generation and version inspection while keeping the existing
string-returning uuid() behavior compatible.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---------
Co-authored-by: Dongyang Li <lidongyang@selectdb.com>
## User-visible changes at a glance
Everything an operator or a user can observe, in one place. Rationale
for each row is in the
sections further down. Rows marked **Action** need something done at
upgrade time; everything else
is laid down by the package.
### 1. Deployment layout
| What | Before | After | Upgrade |
| --- | --- | --- | --- |
| Java extension jars | `lib/java_extensions/<module>/*.jar`, all fat
jars flattened onto one BE classpath | `plugins/jni/<plugin>/` — one
self-contained directory per plugin: `paimon` `hudi` `iceberg` `jdbc`
`max-compute` `trino-connector` `java-udf` `java-writer` | none |
| Classes shared between BE and Java | the whole `preload-extensions`
fat jar (80MB) plus `java-udf`, at the head of the classpath |
`lib/jni/spi/doris-jni-spi.jar` (zero dependencies) +
`doris-jni-bootstrap.jar`. `preload-extensions` and `java-common` are
gone | none |
| Hadoop config read by the Java plugins | `conf/core-site.xml`,
`conf/hdfs-site.xml` (they were on BE's classpath) |
`plugins/hadoop_conf/`, BE config `jni_plugin_hadoop_conf_dir`; falls
back to `$HADOOP_CONF_DIR` when that directory does not exist |
**Action** — copy the files there, or set `HADOOP_CONF_DIR`, or keep the
properties on the catalog. `conf/*.xml` still serves BE's native
(libhdfs) reader |
| JindoFS / JuiceFS jars (BE) | `lib/jindofs`, `lib/juicefs` (older
builds: `lib/java_extensions/<name>`) |
`plugins/jni_fs/{jindofs,juicefs}`, BE config `jni_plugin_fs_dir` — read
by the native reader **and** appended to every plugin's classpath |
**Action** — move the jars. The old paths are still read for the native
`oss-hdfs://` / `jfs://` reader, with a WARN, but they do **not** reach
the Java plugins |
| User Java function jars | `custom_lib/` and `plugins/java_extensions/`
were on BE's classpath | same two directories, now read by the java-udf
plugin itself | none for the directories; see the UDF row in §2 |
| FE-side jars | `fe/lib/<name>` | unchanged | none |
### 2. Behaviour changes
| Area | Before | After | Impact |
| --- | --- | --- | --- |
| Java UDF classpath | a UDF resolved against whatever BE shipped —
log4j, commons-lang3, hadoop `Writable` types, … | a UDF sees only the
jar named by `CREATE FUNCTION` plus the contract (`org.joda.time.` and
the Hive UDF base classes) | **Action** — package every other library
into that jar |
| `static_load = true` on a **UDTF** | read but never applied, so always
false for table functions | actually statically loaded: classes compiled
once, classloader lives for the life of the BE, shared across queries |
**Action if unwanted** — set `static_load = false` |
| Java function caching | keyed by signature: two same-named functions
in different databases shared one compiled class, and a variadic
function could never be invalidated (DROP + CREATE kept running the old
code until BE restart) | keyed by function id | bug fix; nothing about
how a function is declared changes |
| TVF sink `writer_class` | a Java class name | a plugin factory,
`<plugin>:<factory>` — e.g. `java-writer:local-file`. A class name is
rejected with a message saying so | master only — introduced after 4.0,
no released version accepts the old spelling |
| JDBC catalog `driver_url` | the jar behind the URL was read as-is |
verified against the MD5 the catalog carries, once per driver jar, on
the scanner, the writer and the connection test alike | a catalog whose
jar was replaced in place now fails with `Checksum mismatch for JDBC
driver`; a catalog defined without a checksum is unaffected |
| Hudi time travel (`FOR TIME AS OF` / `FOR VERSION AS OF`) with
`use_hive_sync_partition = true` | pruned against the HMS partition
listing | reads every partition; EXPLAIN shows `partition=0/0` |
correctness — the HMS listing holds what is synced *now* and cannot be
trusted at a past instant. `use_hive_sync_partition = false` prunes as
before |
| Query profile node names | `PaimonJniScanner`, `HadoopHudiJniScanner`,
`JdbcJniScanner`, … | the plugin name: `paimon`, `hudi`, `jdbc`, … on
both the v1 and v2 scan paths | **Action** — anything that parses
profiles (dashboards, support tooling, archived comparisons) has to
follow |
| `file://` under hudi | hadoop's `LocalFileSystem` | hive-apache's
`ProxyLocalFileSystem` (a subclass overriding only `rename`) | each
plugin now fills its own filesystem registry; no functional difference
expected |
| UDF `expiration_time` / `EXPIRATION_TIME` | accepted, ignored | still
accepted, still echoed by `SHOW CREATE FUNCTION`, still ignored | none —
time-based eviction was removed before this PR and is not reintroduced |
### 3. Process, signals, logs, metrics
| Area | Before | After | Impact |
| --- | --- | --- | --- |
| JVM lifecycle | created at BE startup whether or not any Java code
runs; a broken jar in the shared layer could stop BE from starting |
created lazily on first Java use; a plugin that fails to load becomes a
plugin-level `FAILED` state instead of `exit(1)` | a BE that touches no
Java feature runs without a JVM |
| Signals | the JVM installed its own SIGINT / SIGTERM / SIGQUIT
handlers | JVM runs with `-Xrs` (installs none); BE installs its own
SIGQUIT handler that does nothing | no Java shutdown hook runs on
SIGTERM / SIGINT; `kill -3 <be_pid>` no longer prints a JVM thread dump
(use `jcmd` / `jstack`) and no longer risks killing the BE |
| `log/jni.log` | `log/jni.log`, log4j2, 10MB × 5 per plugin |
`log/jni.log.0` (JUL rotation naming), 100MB × 10, one file for all
plugins | **Action** — log collectors matching the exact name have to
follow |
| `log/trinoconnector0.log` | written by the Trino connector | removed;
Trino logs into `jni.log` | its FileHandler sat on the JUL **root**
logger, which after isolation would collect every plugin's output |
| Hadoop's own logging on the native reader path | configured by a
`log4j2.properties` inside `java-common` |
`conf/hadoop_log4j2.properties` → `log/hadoop.log`, named by
`bin/start_be.sh` when present | without it Log4j2 falls back to
ERROR-only-to-console, i.e. into `be.out` |
| `jvm_*` metrics on `/metrics` | always exported | exported once the
process has a JVM, whichever Java feature brought it up — an HDFS read
through libhdfs included | a BE using no Java feature at all reports
none |
### 4. New knobs
| Name | Kind | Default | What it is |
| --- | --- | --- | --- |
| `jni_plugin_dir` | BE config | `${DORIS_HOME}/plugins/jni` | the
plugin root |
| `jni_plugin_hadoop_conf_dir` | BE config |
`${DORIS_HOME}/plugins/hadoop_conf` | hadoop config for the Java
plugins; falls back to `$HADOOP_CONF_DIR` |
| `jni_plugin_fs_dir` | BE config | `${DORIS_HOME}/plugins/jni_fs` |
third-party filesystem jars, read by the native reader and by every
plugin. Moving it means moving both halves — `bin/start_be.sh` reads the
same config |
| `GET /api/jni_plugin_status` | HTTP API (admin) | — | per-plugin
state: loaded, or failed with the reason, plus the declared API version.
Deliberately does not start a JVM to answer |
### 5. Nothing to do if
- The deployment has no Java UDF/UDTF and no JNI catalog (paimon, hudi,
iceberg, jdbc, max-compute,
trino-connector) — the layout change is invisible and the BE simply
never creates a JVM.
- Catalogs carry their hadoop properties explicitly,
`hadoop.config.resources` included.
- `HADOOP_CONF_DIR` already points at the hadoop configuration — the
plugins read it when
`plugins/hadoop_conf/` does not exist.
- No JindoFS / JuiceFS, no profile-parsing tooling, no `kill -3` habit.
> **One standing caveat.** jindo-core carries a native library, and a
JVM binds one native library
> to exactly one classloader. A single BE process must not reach jindo
from two plugins at once,
> nor from a plugin and through libhdfs at the same time. That is
inherent to plugin isolation, not
> to the new directory: a single shared loader for these jars would need
a shared hadoop, which is
> what this PR removes. The FE records the same caveat for its paimon
connector plugin.
---
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
**Draft.** Opened for direction review. See "State of this PR" at the
bottom for what is
still missing.
Today every `be-java-extensions` module is flattened onto one classpath.
`bin/start_be.sh`
puts an 80MB `preload-extensions` fat jar plus `java-udf` in front of
everything, each
scanner is a second fat jar loaded by a child-first loader, and BE
resolves a scanner by
Java class name. Three consequences, all of which have caused production
incidents:
- **Silent version arbitration.** Between the shared layer and the
scanners there are
45,530 same-named classes, thousands of which differ in bytes (parquet
2.12 vs 2.10,
iceberg, hive UDF contracts, two log4j 1.x). Which one runs is decided
by classpath
order, with no warning. A scanner cannot upgrade a library without
changing what every
other scanner sees.
- **Implicit provider contracts.** libthrift comes from `java-udf`;
`org.apache.doris.thrift.*`
from `preload-extensions`; hadoop from `lib/hadoop_hdfs`. None of this
is declared
anywhere - the only record is pom comments left behind by past
incidents, written in the
providing module rather than the consuming one.
- **A JVM nobody asked for.** BE creates a JVM at startup whether or not
any Java code is
ever used, and a broken jar in the shared layer can stop BE from
starting.
This PR replaces that with Trino-style hard isolation: a shared SPI, a
loader, and one
directory per plugin.
**Architecture**
```
be/lib/jni/spi/ doris-jni-spi.jar zero dependencies, the only classes BE and plugins share
doris-jni-bootstrap.jar the loader and the registry BE calls
be/plugins/jni/ paimon/ hudi/ iceberg/ jdbc/ max-compute/ trino-connector/ java-udf/
java-writer/
```
The plugins live under `plugins/` rather than `lib/` because `lib/` is
the engine tree a package
upgrade replaces wholesale. The directory is the BE config
`jni_plugin_dir`.
A plugin's classloader has the **platform** loader as its parent and
delegates only the
`org.apache.doris.jni.spi.` prefix. Nothing else is shared - each plugin
carries its own
hadoop, its own filesystem implementations, its own logging bridges. BE
addresses a plugin
by `(plugin name, factory name)` instead of by class name, so the wiring
is one table in
`be/src/util/jni_plugin_registry.h` rather than string literals spread
over the readers.
The system classpath is now `conf/` + `lib/jni/spi` + `lib/hadoop_hdfs`
+ `plugins/jni_fs` - all of
it for BE itself and for libhdfs; no plugin can reach it.
`preload-extensions` and `java-common`
are gone.
`plugins/jni_fs` is the one directory read from both sides: the
third-party filesystem jars
(JindoFS for `oss://` and `oss-hdfs://`, JuiceFS for `jfs://`) go on the
system classpath for the
native libhdfs reader **and** are appended to every plugin's own
classpath by `PluginRegistry`. No
plugin declares them - hadoop reaches a `FileSystem` by class name out
of a `Configuration`, so
nothing links against them - and a plugin classloader cannot see the
system classpath, so without
this a Java scanner would lose those schemes entirely. Each plugin still
loads its own copy in its
own classloader; only the files are shared. BE config
`jni_plugin_fs_dir`.
**Also in this PR**
- **The JVM bootstrap moves into Doris**
(`be/src/util/jvm_launcher.{h,cpp}`). It used to
belong to libhdfs, which meant BE's JNI ability depended on libhdfs
being linked. The
three-step bootstrap libhdfs performed is reproduced exactly, including
`FileSystem.loadFileSystems()` - hadoop records its providers once,
under the first
caller's TCCL, so skipping it defers that scan to whichever plugin
thread gets there
first. The JVM is now created lazily: no Java query and no HDFS access
means no JVM.
- **A plugin failing to load is a plugin-level FAILED state**, not
`exit(1)`.
- **User Java UDFs see only the contract** - `org.joda.time.` and the
Hive UDF base classes -
and nothing else Doris happens to have on its classpath.
**Sizes** (fat jar -> plugin directory): jdbc 5.6MB -> 3.8MB, java-udf
80MB -> 20MB,
trino-connector 110MB -> 73MB, max-compute 67MB -> 60MB.
**Behaviour changes worth calling out**
- `log/jni.log` becomes `log/jni.log.0` (JUL rotation naming, 100MB x
10). Larger than the
10MB x 5 the replaced log4j2 configuration gave each plugin, because
this one file now holds what
several of them held separately - Trino in particular had 500MB x 10 to
itself and is chatty at
INFO, and rolling away the plugin-load diagnostics is exactly what this
file must not do.
- `log/trinoconnector0.log` is removed. Its FileHandler was attached to
the JUL **root**
logger, which is process-wide, so after isolation it would collect every
plugin's log.
Trino logs through `io.airlift.log.Logger` (JUL) and lands in `jni.log`
anyway.
- The `expiration_time` UDF property is not carried into the new module.
It has been
ignored since time-based eviction was removed; it stays accepted and
stays ineffective.
- Profile node names use the plugin name (`paimon`, `hudi`, `jdbc`, ...)
instead of the class name
(`PaimonJniScanner`, `HadoopHudiJniScanner`, `JdbcJniScanner`), on both
the v1 and v2 scan paths.
Anything that parses a query profile - dashboards, support tooling,
archived comparisons - sees
the new names.
- Under hudi, `file://` now resolves to hive-apache's
`ProxyLocalFileSystem` rather than
hadoop's `LocalFileSystem` (a subclass overriding only `rename`).
Previously the static
registry was filled once by BE's TCCL; now each plugin fills its own.
- **A user Java UDF sees only its own jar plus the contract**
(`org.joda.time.` and the Hive UDF
base classes). Everything else it uses has to be inside the jar named by
`CREATE FUNCTION` -
log4j, commons-lang3, hadoop's `Writable` types and so on used to
resolve against whatever BE
shipped, and no longer do. `custom_lib` and `plugins/java_extensions`
also leave the BE
classpath; the java-udf plugin reads both directories itself.
- **`writer_class` on a TVF sink names a plugin factory**,
`<plugin>:<factory>` - for example
`java-writer:local-file`. A Java class name is rejected with a message
saying so. Introduced
after 4.0, so no released version accepts the old spelling.
- **Hadoop configuration files for the Java plugins move to
`plugins/hadoop_conf/`** (BE config
`jni_plugin_hadoop_conf_dir`). A plugin's classloader cannot reach BE's
classpath, so a
`core-site.xml` dropped into `conf/` is no longer read by the paimon /
hudi / iceberg / jdbc
plugins - it still is by the native (libhdfs) reader. Catalogs that
carry their hadoop
properties explicitly, including `hadoop.config.resources`, are
unaffected.
- **JindoFS and JuiceFS jars move to
`plugins/jni_fs/{jindofs,juicefs}`** on the BE, from
`lib/{jindofs,juicefs}` (the FE keeps `fe/lib/<name>`). One copy on
disk, two readers:
`bin/start_be.sh` puts the directory on the system classpath for the
native libhdfs reader, and
`PluginRegistry` appends the
same jars to every plugin's classpath, after that plugin's own, so a
plugin's hadoop still wins.
`start_be.sh` still reads the two old locations as a fallback, with a
WARN, so an in-place upgrade
keeps resolving these schemes for the native reader before the new tree
is laid down - but only
the new location reaches the plugins.
Caveat, the same one the FE records for its paimon connector plugin:
jindo-core carries a native
library and a JVM binds one of those to exactly one classloader, so a
single BE process must not
reach jindo from two plugins at once, nor from a plugin and through
libhdfs at the same time. That
is inherent to plugin isolation rather than to this directory: a single
shared loader for these
jars is impossible, since they need the hadoop that lives inside each
plugin.
- **The JVM is created with `-Xrs`**, so it installs no SIGINT / SIGTERM
/ SIGQUIT handler of its
own. That is what keeps a shutdown signal from becoming a Java
`Shutdown.exit()` on a serving
BE now that the JVM comes up on demand. Consequences: no Java shutdown
hook runs on those
signals, and `kill -3` no longer produces a JVM thread dump (`jcmd` /
`jstack` still do). BE
installs a handler for SIGQUIT itself so that the signal does nothing -
left at its default it
would terminate the process and dump core, turning the habitual `kill -3
<be_pid>` from "print
the thread dump" into "kill the BE".
- **A statically loaded Java function is cached by its function id**,
not by its signature. Two
same-named functions in different databases used to share one compiled
class, and a variadic
function could never be invalidated at all - FE spells a variadic
signature with a trailing
`"..."` when it describes a function to execute and without one when it
drops it, so DROP plus
CREATE went on running the old code until the BE restarted. Both are
fixed; nothing about how a
function is declared changes.
- **`jvm_*` metrics appear on `/metrics` once the process has a JVM**,
whichever Java feature
brought it up - including an HDFS read through libhdfs on a BE that uses
no JNI table format and
no Java UDF. A BE that touches no Java feature at all has no JVM, and
reports none.
`GET /api/jni_plugin_status` (admin) reports the plugin state - loaded,
failed with the reason,
declared API version - and deliberately does not start a JVM to answer.
- **A UDTF declared with `static_load = true` is now actually statically
loaded.** The property was
read but never applied to table functions, so it was always false for
them; from here on such a
UDTF is cached process-wide, its classloader lives for the life of the
BE, and its classes are
shared between queries - which is what the property has always meant for
scalar and aggregate
functions.
- **A JDBC catalog's `driver_url` is verified against the MD5 the
catalog carries**, once per driver
jar rather than per query, by the scanner, the writer and the connection
test alike. A catalog
whose jar no longer matches its recorded checksum now fails with
"Checksum mismatch for JDBC
driver" instead of reading through whatever is behind the URL. A catalog
defined without a
checksum is unaffected.
- **Hadoop configuration for the Java plugins falls back to
`HADOOP_CONF_DIR`** when
`plugins/hadoop_conf/` does not exist, so a deployment that resolves its
HDFS HA nameservice
through that variable keeps working without copying files.
- **A `FOR TIME AS OF` / `FOR VERSION AS OF` query on a hudi catalog
with
`use_hive_sync_partition = true` now reports `partition=0/0` in EXPLAIN
and reads every
partition.** That listing comes from HMS, which holds what is synced
*now* and can only be
subtracted from, so it is a subset of the pinned universe - pruning
against it would silently
drop rows a time-travel query must read. The connector therefore
declines to claim the listing is
snapshot-exact, and an empty universe means scan-all. Coarse, never
short; the
`use_hive_sync_partition = false` listing comes from hudi's metadata
table, is snapshot-exact, and
prunes as before.
**New build-time gate**
`tools/be-java-plugins/check_plugin_layout.py` runs from `build.sh`,
right after the plugins are
deployed, over `output/be/lib/jni/spi` and `output/be/plugins/jni`. It
enforces five rules: the
SPI jar carries only SPI packages; no plugin ships an SPI class; no
class inside a plugin
directory resolves to different bytes in different jars; every class
referenced by Doris's own
code in a plugin resolves inside that directory; and the jar declaring
the `DorisPlugin` service
carries the `Doris-Jni-Plugin-Api-Version` this build serves. The
known-benign exceptions are
allowlisted with the reason written down.
It runs there rather than in a GitHub workflow because it needs a built
output tree, which only a
full BE build produces - no workflow in `.github/` builds BE with the
Java extensions. A violation
fails the build; a missing `python3` or `jdeps` is a warning, since that
is the check not running
rather than a verdict on the tree. The file also states what it does
**not** prove: it is static
and starts from Doris's classes, so anything reached by ServiceLoader or
reflection is invisible
to it.
Docs: https://github.com/apache/doris-website/pull/4091
Problem Summary:
Doris DATETIME only support at most six fractional digits, it's not
sufficient for event times, traces, etc which require higher precision
of sub-seconds.
Introduce `TIMESTAMP_NS` as a separate fixed-scale SQL type backed by
signed Int64 Unix-epoch nanoseconds. It always has scale 9 and supports
values from `1677-09-21 00:12:43.145224192` through `2262-04-11
23:47:16.854775807`. Wire the type through Thrift and protobuf metadata,
the FE catalog and Nereids, BE columns and SerDe, storage and indexes,
result protocols, JNI and Java UDF transport, complex and VARIANT types,
and schema metadata.
Add exact casts, comparisons, type inference, and constant folding
across date-like types while preserving nanoseconds and rejecting
partial or precision-losing implicit conversions. Add timestamp-aware
scalar and calendar functions, aggregation, joins and ASOF joins, scan
predicates, runtime filters, partition pruning, query-cache
normalization, automatic partitions, and MTMV roll-up. Keep FE folding
and BE runtime behavior consistent at signed endpoints, leap years, DST
gaps, calendar-clamped boundaries, and fractional formatting and
rounding boundaries.
Keep existing DATETIME and DATETIMEV2 scale semantics unchanged.
FROM_UNIXTIME retains a DECIMAL64 microsecond path for scale 0 through 6
and uses the DECIMAL128 nanosecond path only for scale 7 through 9. The
%f formatter rounds half-up to microseconds with second carry, while %n
preserves the nanosecond fraction.
- Test: Unit Test and Regression Test
- Added targeted BE unit coverage for TIMESTAMP_NS values, casts,
functions, SerDe, predicates, aggregates, joins, and compatibility paths
- Added targeted FE unit coverage for literals, casts, constant folding,
type inference, partitions, query-cache and MTMV boundaries, and
statement-time semantics
- Added regression coverage under datatype_p0/timestamp_ns and focused
MTMV and Java UDF suites
- Built FE and BE with the standard build scripts
- Behavior changed: Yes. Adds TIMESTAMP_NS and nanosecond function
semantics without changing the scale limits of DATETIME or DATETIMEV2.
- Does this need documentation: Yes. Document TIMESTAMP_NS syntax,
range, casts and coercion rules, function behavior, and integration
support.
### Release note
Add `TIMESTAMP_NS`, a fixed-scale nanosecond timestamp type stored as
signed Unix-epoch nanoseconds. It supports SQL literals, casts,
date/time functions, storage and indexes, partitions, joins, aggregates,
complex types, Java UDF/JNI transport, and FE constant folding and type
inference. Existing DATETIME and DATETIMEV2 types remain limited to
scale 0 through 6.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
Problem Summary:
UDF cache cleanup has two problems:
1. Cloud mode does not register a worker for `CLEAN_UDF_CACHE`, so `DROP
FUNCTION` cannot clean the cached UDF classloader.
2. UDF caches are cleaned by function signature. If a function is
dropped and recreated with the same signature, a delayed cleanup task
may delete the new cache, or the recreated function may reuse the stale
cache.
The FE removes the function metadata first and then submits
`CleanUDFCacheTask` asynchronously. It does not wait for the BE cache
cleanup result.
In addition, the cleanup task does not report a completion result back
to the FE. Therefore, even if the task cannot be submitted or the JNI
cache cleanup fails, `DROP FUNCTION` still returns success to the
client.
```sql
CREATE FUNCTION test_udf(INT) RETURNS INT ...; -- implementation V1
SELECT test_udf(1); -- cache V1
DROP FUNCTION test_udf(INT); -- cache cleanup fails or is delayed
CREATE FUNCTION test_udf(INT) RETURNS INT ...; -- implementation V2
SELECT test_udf(1);
````
Before this PR, the last query could reuse V1's cached classloader
because V1 and V2 had the same signature. A delayed cleanup task for V1
could also remove V2's cache.
### How does this PR fix the problem?
- Register the CLEAN_UDF_CACHE worker in cloud mode.
- Use the function ID as the key for Java UDF cache lookup, insertion,
and cleanup.
= Fall back to signature-based cleanup when no valid function ID is
provided for compatibility with older FEs.
A recreated function receives a new function ID. Therefore, even if the
previous function's cache is not successfully removed, the recreated
function does not reuse it and can load and execute the correct
implementation. A delayed cleanup task also removes only the old
function's cache without affecting the recreated function.
### What problem does this PR solve?
Issue Number: close#65185
Related PR: every phase PR is listed in #65185
Problem Summary:
This is the merge of the `branch-catalog-spi` feature branch, i.e. the
Catalog SPI
migration tracked in #65185. Per-phase design, review history and the
per-connector
status table live in that issue. The framework itself is documented in
`fe/fe-connector/README.md` (architecture, module map, how to add a
connector) and
`fe/fe-connector/AGENTS.md` (build/test recipes, architecture gates,
invariants).
Before: fe-core hard-coded every external data source under
`fe/fe-core/.../datasource/{hive,iceberg,paimon,hudi,trinoconnector,maxcompute,...}`,
reached from the catalog and planner layers through `switch-case` and
`instanceof *ExternalTable`. Adding or upgrading a source meant changing
FE core, and
every source's client stack lived on the single FE classpath.
After: each source ships as a self-contained plugin under
`fe/fe-connector/` (hive, hms,
iceberg, paimon, hudi, trino, maxcompute, jdbc, es), loaded child-first
from
`Config.connector_plugin_root`. fe-core keeps only generic
infrastructure — the
`PluginDriven*` catalog / database / table / scan-node bridge plus one
per-statement
`ConnectorMetadata` funnel — and no compile-time knowledge of any
specific source:
`CatalogFactory` routes on `ConnectorProvider.getType()` alone. Two
`validate`-phase
gates keep it that way: a connector may not import fe-core internals
(`build-support/check-fe-connector-imports.sh`), and only
`PluginDrivenMetadata` may
call `Connector#getMetadata`
(`build-support/check-fe-core-metadata-funnel.sh`).
Catalogs persisted by an older FE (`IcebergExternalCatalog`,
`PaimonExternalDatabase`,
...) still deserialize and migrate transparently to the plugin-driven
types.
### Release note
External catalogs (hive, hms, iceberg, paimon, hudi, trino-connector,
maxcompute, jdbc,
es) are now served by connector plugins, shipped as zips under
`fe/plugins/connector/`
(`connector_plugin_root`). Existing catalogs keep working across
upgrade; no DDL change
is required.
The default of `trino_connector_plugin_dir` moves from
`plugins/connectors` to
`plugins/trino_plugins`; the old directory is still read as a fallback.
The deprecated `lakesoul` catalog type is removed — creating or
replaying one now fails
with "Lakesoul catalog is no longer supported".
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
Covered by the connector modules' own unit tests, fe-core unit tests
(including a
4.1.3 metadata upgrade-compatibility suite), the two gate self-test
scripts under
`build-support/tests/`, and the `external_table_p0` /
`external_table_p2` regression
suites.
- Behavior changed:
- [ ] No.
- [x] Yes. See the release note: plugin-based packaging and loading of
external
catalogs, the `trino_connector_plugin_dir` default, and the removal of
the
`lakesoul` catalog type. Everything else is held to behavior parity.
- Does this need documentation?
- [ ] No.
- [x] Yes. Doc PR to follow: connector plugin layout,
`connector_plugin_root`,
the `trino_connector_plugin_dir` default, and the `lakesoul` removal.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove the two consumers of the ~122MB
org.apache.doris:hive-catalog-shade fat jar under fe/be-java-extensions.
1. Delete the dead avro-scanner module. Its AvroJNIScanner is never
routed to (no FE code references the class name it would be loaded by),
it is absent from the shared preload-extensions jar, and its only
regression test (test_tvf_avro.groovy) is entirely commented out.
Removes the module, its be-java-extensions/pom.xml and build.sh
registrations, and the dead test plus its .out data.
2. Slim java-udf: replace the hive-catalog-shade dependency with a new
minimal hive-udf-shade module that shades hive-exec:3.1.3:core down to
only the Hive UDF contract classes a user UDF extends -- ql.exec.UDF and
its load-time closure (DefaultUDFMethodResolver/UDFMethodResolver),
ql.exec.Description, and ql.metadata.HiveException. The shaded jar is 20
KB / 10 classes; java-udf's fat jar goes from 12807 hive classes to 10
(all metastore/serde2/objectinspector/relocated-thrift dropped).
The fe/pom.xml version pin and the fe-core / fe-connector-hms /
fe-common consumers still depend on hive-catalog-shade and are
intentionally left untouched (out of scope).
### What problem does this PR solve?
Problem Summary:
doc https://github.com/apache/doris-website/pull/3845
```
CREATE FUNCTION print_12() RETURNS int
PROPERTIES (
"file" = "file:///path/to/java-udf-demo-jar-with-dependencies.jar",
"symbol" = "org.apache.doris.udf.Print",
"always_nullable"="true",
"type" = "JAVA_UDF",
"static_load" = "true", // default value is false
"expiration_time" = "60" // default value is 360 minutes
);
```
```
before in the java-udf could use static_load and expiration_time to control the cache jar times in BE.
which use a backgroud thread to scan the jars every ten minutes, check it's init times, and then drop it if time expire.
those will cause some long running query failed when the backgroud thread remove it.
Now, remove the expiration_time, and the jar will be clean when drop fucntion immediately
```
### What problem does this PR solve?
Problem Summary:
before the UDF use static load will cache some values in UdfClassCache,
but not cache the classload, those will cause some NoClassDefFoundError
when users UDF have invoke some thirdparty class.
### What problem does this PR solve?
- We should keep fe-common focused and avoid putting unrelated logic
into it.
- JUnit should be declared with test scope, as it is only used for
testing.
https://github.com/apache/doris/pull/58280
It also fixes the Paimon S3AFileSystem ClassNotFoundException issue
reported in #58280.
### What problem does this PR solve?
In the past, Paimon needed to access HMS, so we included this module.
However, since we've now resolved metadata access through other means,
this module is no longer needed and can be safely removed.
### Release note
#57979
This PR reverts recent modifications to the be-extension package
management. The changes introduced class loading problems that caused
the hadoop-jin catalog system tables to become unusable.
### changes
#### Remove Ambrai & K8s Kernel Deployment Support
These features were never documented, had no user feedback, and
generally should be implemented externally rather than in the core
engine.
#### Standardize to commons-collections4
Completely migrated to commons-collections4.
Ensures API consistency and avoids accidental introduction of outdated
libraries.
#### Remove Unused Dependencies
Full dependency audit performed.
Removed multiple unused or obsolete dependencies to reduce footprint and
avoid security risks.
#### Refactor be-java-extension Dependencies
Reorganized dependency structure:
Common dependencies extracted to a standalone minimal module.
Business-specific logic isolated into separate extension(s).
Improves maintainability, reduces coupling, and prevents unnecessary
dependency leakage.
### Impact
Project size reduced by ~800 MB.
Cleaner module boundaries.
### What problem does this PR solve?
Problem Summary:
```
mysql> select java_udf_int_test_nullable2(1) from test_query_qa.baseall;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.3)[JNI_ERROR]UdfRuntimeException: UDF failed to evaluate | CAUSED BY: IllegalArgumentException: Input value cannot be
```
In some error reports, when an InternalError is returned, users may not
immediately understand the issue and must check the logs in detail. To
improve this, we've introduced a JNI_ERROR error code. When users see
this error code, they can quickly recognize its meaning, especially in
the context of UDF calls.
Additionally, this error will be followed by a UdfRuntimeException
message, making it clear to users that the error occurred within a UDF.
### What problem does this PR solve?
Problem Summary:
in some user case,
- the udf-jars package have some resource file, it's maybe about
hundreds MBs,
so if every instances load the jar, it's easy to cause the BE JVM OOM.
- or in some udf-jars have some init operator cause many times, so wants
all instance could only init it's once.
follow up https://github.com/apache/doris/pull/40404
support class cache for java-udaf
Refactor the java udf code to reduce the unless code
1. `JavaUdfDataTypeSet` to `JavaUdfDataTypeMap` speed up is support
search
2. Rewrite the JavaUdfDataType `toString()`
3. remove the unless `getKeyType` and `setKeyType`
4. remove the unless child
method：`getInputConverters/getOutputConverter`
### What problem does this PR solve?
Problem Summary:
Use explicit build directory in pom.xml, or the IDEA will report error
sometimes like:
```
Error scala: Output path .../eval/target/test-classes is shared between: Module 'xxx' tests, Module 'xxx' tests
Output path .../xxx/target/classes is shared between: Module 'xxx' production, Module 'xxx' production
Please configure separate output paths to proceed with the compilation.
```
### What problem does this PR solve?
Problem Summary:
support handle const column in java-udf
the before and after execute time
```
mysql [ssb]>select count(test_query_qa.java_udf_test3(c_custkey,"asd")) from test_udf;
+-----------------------------------------+
| count(java_udf_test3(c_custkey, 'asd')) |
+-----------------------------------------+
| 36000000 |
+-----------------------------------------+
1 row in set (0.68 sec)
------------
mysql [ssb]>select count(test_query_qa.java_udf_test3(c_custkey,"asd")) from test_udf;
+-----------------------------------------+
| count(java_udf_test3(c_custkey, 'asd')) |
+-----------------------------------------+
| 36000000 |
+-----------------------------------------+
1 row in set (0.52 sec)
```
### Release note
support handle const column in java-udf
In this PR #34980, we cache the class load of java udf.
But when calling init() method, it is still very time consuming
when calling `Class.forName()`.
This PR add a new class `UdfClassCache` to cache more things
so that the `init()` method will not cost too much time.
## Proposed changes
```
CREATE FUNCTION print_12() RETURNS int
PROPERTIES (
"symbol" = "org.apache.doris.udf.AddOne",
"type" = "JAVA_UDF",
"file" = "file:///mnt/ava-udf-demo-jar-with-dependencies.jar",
"static_load" = "true", // default value is false
"expiration_time" = "60" // default value is 360 minutes
);
```
if set static_load=true, the udf ClassLoader will be cache, and will be
clear when expiration timeout
or use drop function stmt.
Follow https://github.com/apache/doris/pull/25302, and use the unified jni framework to refactor java udaf.
This PR has removed the old interfaces to run java udf/udaf. Thanks to the ease of use of the new framework, the core code for modifying UDAF does not exceed 100 lines, and the logic is similar to that of UDF.
Use the unified jni framework to refactor java udf.
The unified jni framework takes VectorTable as the container to transform data between c++ and java, and hide the details of data format conversion.
In addition, the unified framework supports complex and nested types.
The performance of basic types remains consistent, with a 30% improvement in string types and an order of magnitude improvement in complex types.
The java-udf module has become increasingly large and difficult to manage, making it inconvenient to package and use as needed. It needs to be split into multiple sub-modules, such as : java-commom、java-udf、jdbc-scanner、hudi-scanner、 paimon-scanner.
Co-authored-by: lexluo <lexluo@tencent.com>
TengJianPingandDongyang Li
Mingyu Chen (Rayner)
linrrarity
Mingyu Chen (Rayner)andClaude Opus 4.7
zhangstar333
Calvin Kirs
HappenLee
Mingyu Chen (Rayner)
zhangstar333
Calvin Kirs
Gabriel
Ashin Gau
Mryange
lexluo09andlexluo