mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Mingyu Chen (Rayner)
1407093484
## User-visible changes at a glance
Everything an operator or a user can observe, in one place. Rationale
for each row is in the
sections further down. Rows marked **Action** need something done at
upgrade time; everything else
is laid down by the package.
### 1. Deployment layout
| What | Before | After | Upgrade |
| --- | --- | --- | --- |
| Java extension jars | `lib/java_extensions/<module>/*.jar`, all fat
jars flattened onto one BE classpath | `plugins/jni/<plugin>/` — one
self-contained directory per plugin: `paimon` `hudi` `iceberg` `jdbc`
`max-compute` `trino-connector` `java-udf` `java-writer` | none |
| Classes shared between BE and Java | the whole `preload-extensions`
fat jar (80MB) plus `java-udf`, at the head of the classpath |
`lib/jni/spi/doris-jni-spi.jar` (zero dependencies) +
`doris-jni-bootstrap.jar`. `preload-extensions` and `java-common` are
gone | none |
| Hadoop config read by the Java plugins | `conf/core-site.xml`,
`conf/hdfs-site.xml` (they were on BE's classpath) |
`plugins/hadoop_conf/`, BE config `jni_plugin_hadoop_conf_dir`; falls
back to `$HADOOP_CONF_DIR` when that directory does not exist |
**Action** — copy the files there, or set `HADOOP_CONF_DIR`, or keep the
properties on the catalog. `conf/*.xml` still serves BE's native
(libhdfs) reader |
| JindoFS / JuiceFS jars (BE) | `lib/jindofs`, `lib/juicefs` (older
builds: `lib/java_extensions/<name>`) |
`plugins/jni_fs/{jindofs,juicefs}`, BE config `jni_plugin_fs_dir` — read
by the native reader **and** appended to every plugin's classpath |
**Action** — move the jars. The old paths are still read for the native
`oss-hdfs://` / `jfs://` reader, with a WARN, but they do **not** reach
the Java plugins |
| User Java function jars | `custom_lib/` and `plugins/java_extensions/`
were on BE's classpath | same two directories, now read by the java-udf
plugin itself | none for the directories; see the UDF row in §2 |
| FE-side jars | `fe/lib/<name>` | unchanged | none |
### 2. Behaviour changes
| Area | Before | After | Impact |
| --- | --- | --- | --- |
| Java UDF classpath | a UDF resolved against whatever BE shipped —
log4j, commons-lang3, hadoop `Writable` types, … | a UDF sees only the
jar named by `CREATE FUNCTION` plus the contract (`org.joda.time.` and
the Hive UDF base classes) | **Action** — package every other library
into that jar |
| `static_load = true` on a **UDTF** | read but never applied, so always
false for table functions | actually statically loaded: classes compiled
once, classloader lives for the life of the BE, shared across queries |
**Action if unwanted** — set `static_load = false` |
| Java function caching | keyed by signature: two same-named functions
in different databases shared one compiled class, and a variadic
function could never be invalidated (DROP + CREATE kept running the old
code until BE restart) | keyed by function id | bug fix; nothing about
how a function is declared changes |
| TVF sink `writer_class` | a Java class name | a plugin factory,
`<plugin>:<factory>` — e.g. `java-writer:local-file`. A class name is
rejected with a message saying so | master only — introduced after 4.0,
no released version accepts the old spelling |
| JDBC catalog `driver_url` | the jar behind the URL was read as-is |
verified against the MD5 the catalog carries, once per driver jar, on
the scanner, the writer and the connection test alike | a catalog whose
jar was replaced in place now fails with `Checksum mismatch for JDBC
driver`; a catalog defined without a checksum is unaffected |
| Hudi time travel (`FOR TIME AS OF` / `FOR VERSION AS OF`) with
`use_hive_sync_partition = true` | pruned against the HMS partition
listing | reads every partition; EXPLAIN shows `partition=0/0` |
correctness — the HMS listing holds what is synced *now* and cannot be
trusted at a past instant. `use_hive_sync_partition = false` prunes as
before |
| Query profile node names | `PaimonJniScanner`, `HadoopHudiJniScanner`,
`JdbcJniScanner`, … | the plugin name: `paimon`, `hudi`, `jdbc`, … on
both the v1 and v2 scan paths | **Action** — anything that parses
profiles (dashboards, support tooling, archived comparisons) has to
follow |
| `file://` under hudi | hadoop's `LocalFileSystem` | hive-apache's
`ProxyLocalFileSystem` (a subclass overriding only `rename`) | each
plugin now fills its own filesystem registry; no functional difference
expected |
| UDF `expiration_time` / `EXPIRATION_TIME` | accepted, ignored | still
accepted, still echoed by `SHOW CREATE FUNCTION`, still ignored | none —
time-based eviction was removed before this PR and is not reintroduced |
### 3. Process, signals, logs, metrics
| Area | Before | After | Impact |
| --- | --- | --- | --- |
| JVM lifecycle | created at BE startup whether or not any Java code
runs; a broken jar in the shared layer could stop BE from starting |
created lazily on first Java use; a plugin that fails to load becomes a
plugin-level `FAILED` state instead of `exit(1)` | a BE that touches no
Java feature runs without a JVM |
| Signals | the JVM installed its own SIGINT / SIGTERM / SIGQUIT
handlers | JVM runs with `-Xrs` (installs none); BE installs its own
SIGQUIT handler that does nothing | no Java shutdown hook runs on
SIGTERM / SIGINT; `kill -3 <be_pid>` no longer prints a JVM thread dump
(use `jcmd` / `jstack`) and no longer risks killing the BE |
| `log/jni.log` | `log/jni.log`, log4j2, 10MB × 5 per plugin |
`log/jni.log.0` (JUL rotation naming), 100MB × 10, one file for all
plugins | **Action** — log collectors matching the exact name have to
follow |
| `log/trinoconnector0.log` | written by the Trino connector | removed;
Trino logs into `jni.log` | its FileHandler sat on the JUL **root**
logger, which after isolation would collect every plugin's output |
| Hadoop's own logging on the native reader path | configured by a
`log4j2.properties` inside `java-common` |
`conf/hadoop_log4j2.properties` → `log/hadoop.log`, named by
`bin/start_be.sh` when present | without it Log4j2 falls back to
ERROR-only-to-console, i.e. into `be.out` |
| `jvm_*` metrics on `/metrics` | always exported | exported once the
process has a JVM, whichever Java feature brought it up — an HDFS read
through libhdfs included | a BE using no Java feature at all reports
none |
### 4. New knobs
| Name | Kind | Default | What it is |
| --- | --- | --- | --- |
| `jni_plugin_dir` | BE config | `${DORIS_HOME}/plugins/jni` | the
plugin root |
| `jni_plugin_hadoop_conf_dir` | BE config |
`${DORIS_HOME}/plugins/hadoop_conf` | hadoop config for the Java
plugins; falls back to `$HADOOP_CONF_DIR` |
| `jni_plugin_fs_dir` | BE config | `${DORIS_HOME}/plugins/jni_fs` |
third-party filesystem jars, read by the native reader and by every
plugin. Moving it means moving both halves — `bin/start_be.sh` reads the
same config |
| `GET /api/jni_plugin_status` | HTTP API (admin) | — | per-plugin
state: loaded, or failed with the reason, plus the declared API version.
Deliberately does not start a JVM to answer |
### 5. Nothing to do if
- The deployment has no Java UDF/UDTF and no JNI catalog (paimon, hudi,
iceberg, jdbc, max-compute,
trino-connector) — the layout change is invisible and the BE simply
never creates a JVM.
- Catalogs carry their hadoop properties explicitly,
`hadoop.config.resources` included.
- `HADOOP_CONF_DIR` already points at the hadoop configuration — the
plugins read it when
`plugins/hadoop_conf/` does not exist.
- No JindoFS / JuiceFS, no profile-parsing tooling, no `kill -3` habit.
> **One standing caveat.** jindo-core carries a native library, and a
JVM binds one native library
> to exactly one classloader. A single BE process must not reach jindo
from two plugins at once,
> nor from a plugin and through libhdfs at the same time. That is
inherent to plugin isolation, not
> to the new directory: a single shared loader for these jars would need
a shared hadoop, which is
> what this PR removes. The FE records the same caveat for its paimon
connector plugin.
---
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
**Draft.** Opened for direction review. See "State of this PR" at the
bottom for what is
still missing.
Today every `be-java-extensions` module is flattened onto one classpath.
`bin/start_be.sh`
puts an 80MB `preload-extensions` fat jar plus `java-udf` in front of
everything, each
scanner is a second fat jar loaded by a child-first loader, and BE
resolves a scanner by
Java class name. Three consequences, all of which have caused production
incidents:
- **Silent version arbitration.** Between the shared layer and the
scanners there are
45,530 same-named classes, thousands of which differ in bytes (parquet
2.12 vs 2.10,
iceberg, hive UDF contracts, two log4j 1.x). Which one runs is decided
by classpath
order, with no warning. A scanner cannot upgrade a library without
changing what every
other scanner sees.
- **Implicit provider contracts.** libthrift comes from `java-udf`;
`org.apache.doris.thrift.*`
from `preload-extensions`; hadoop from `lib/hadoop_hdfs`. None of this
is declared
anywhere - the only record is pom comments left behind by past
incidents, written in the
providing module rather than the consuming one.
- **A JVM nobody asked for.** BE creates a JVM at startup whether or not
any Java code is
ever used, and a broken jar in the shared layer can stop BE from
starting.
This PR replaces that with Trino-style hard isolation: a shared SPI, a
loader, and one
directory per plugin.
**Architecture**
```
be/lib/jni/spi/ doris-jni-spi.jar zero dependencies, the only classes BE and plugins share
doris-jni-bootstrap.jar the loader and the registry BE calls
be/plugins/jni/ paimon/ hudi/ iceberg/ jdbc/ max-compute/ trino-connector/ java-udf/
java-writer/
```
The plugins live under `plugins/` rather than `lib/` because `lib/` is
the engine tree a package
upgrade replaces wholesale. The directory is the BE config
`jni_plugin_dir`.
A plugin's classloader has the **platform** loader as its parent and
delegates only the
`org.apache.doris.jni.spi.` prefix. Nothing else is shared - each plugin
carries its own
hadoop, its own filesystem implementations, its own logging bridges. BE
addresses a plugin
by `(plugin name, factory name)` instead of by class name, so the wiring
is one table in
`be/src/util/jni_plugin_registry.h` rather than string literals spread
over the readers.
The system classpath is now `conf/` + `lib/jni/spi` + `lib/hadoop_hdfs`
+ `plugins/jni_fs` - all of
it for BE itself and for libhdfs; no plugin can reach it.
`preload-extensions` and `java-common`
are gone.
`plugins/jni_fs` is the one directory read from both sides: the
third-party filesystem jars
(JindoFS for `oss://` and `oss-hdfs://`, JuiceFS for `jfs://`) go on the
system classpath for the
native libhdfs reader **and** are appended to every plugin's own
classpath by `PluginRegistry`. No
plugin declares them - hadoop reaches a `FileSystem` by class name out
of a `Configuration`, so
nothing links against them - and a plugin classloader cannot see the
system classpath, so without
this a Java scanner would lose those schemes entirely. Each plugin still
loads its own copy in its
own classloader; only the files are shared. BE config
`jni_plugin_fs_dir`.
**Also in this PR**
- **The JVM bootstrap moves into Doris**
(`be/src/util/jvm_launcher.{h,cpp}`). It used to
belong to libhdfs, which meant BE's JNI ability depended on libhdfs
being linked. The
three-step bootstrap libhdfs performed is reproduced exactly, including
`FileSystem.loadFileSystems()` - hadoop records its providers once,
under the first
caller's TCCL, so skipping it defers that scan to whichever plugin
thread gets there
first. The JVM is now created lazily: no Java query and no HDFS access
means no JVM.
- **A plugin failing to load is a plugin-level FAILED state**, not
`exit(1)`.
- **User Java UDFs see only the contract** - `org.joda.time.` and the
Hive UDF base classes -
and nothing else Doris happens to have on its classpath.
**Sizes** (fat jar -> plugin directory): jdbc 5.6MB -> 3.8MB, java-udf
80MB -> 20MB,
trino-connector 110MB -> 73MB, max-compute 67MB -> 60MB.
**Behaviour changes worth calling out**
- `log/jni.log` becomes `log/jni.log.0` (JUL rotation naming, 100MB x
10). Larger than the
10MB x 5 the replaced log4j2 configuration gave each plugin, because
this one file now holds what
several of them held separately - Trino in particular had 500MB x 10 to
itself and is chatty at
INFO, and rolling away the plugin-load diagnostics is exactly what this
file must not do.
- `log/trinoconnector0.log` is removed. Its FileHandler was attached to
the JUL **root**
logger, which is process-wide, so after isolation it would collect every
plugin's log.
Trino logs through `io.airlift.log.Logger` (JUL) and lands in `jni.log`
anyway.
- The `expiration_time` UDF property is not carried into the new module.
It has been
ignored since time-based eviction was removed; it stays accepted and
stays ineffective.
- Profile node names use the plugin name (`paimon`, `hudi`, `jdbc`, ...)
instead of the class name
(`PaimonJniScanner`, `HadoopHudiJniScanner`, `JdbcJniScanner`), on both
the v1 and v2 scan paths.
Anything that parses a query profile - dashboards, support tooling,
archived comparisons - sees
the new names.
- Under hudi, `file://` now resolves to hive-apache's
`ProxyLocalFileSystem` rather than
hadoop's `LocalFileSystem` (a subclass overriding only `rename`).
Previously the static
registry was filled once by BE's TCCL; now each plugin fills its own.
- **A user Java UDF sees only its own jar plus the contract**
(`org.joda.time.` and the Hive UDF
base classes). Everything else it uses has to be inside the jar named by
`CREATE FUNCTION` -
log4j, commons-lang3, hadoop's `Writable` types and so on used to
resolve against whatever BE
shipped, and no longer do. `custom_lib` and `plugins/java_extensions`
also leave the BE
classpath; the java-udf plugin reads both directories itself.
- **`writer_class` on a TVF sink names a plugin factory**,
`<plugin>:<factory>` - for example
`java-writer:local-file`. A Java class name is rejected with a message
saying so. Introduced
after 4.0, so no released version accepts the old spelling.
- **Hadoop configuration files for the Java plugins move to
`plugins/hadoop_conf/`** (BE config
`jni_plugin_hadoop_conf_dir`). A plugin's classloader cannot reach BE's
classpath, so a
`core-site.xml` dropped into `conf/` is no longer read by the paimon /
hudi / iceberg / jdbc
plugins - it still is by the native (libhdfs) reader. Catalogs that
carry their hadoop
properties explicitly, including `hadoop.config.resources`, are
unaffected.
- **JindoFS and JuiceFS jars move to
`plugins/jni_fs/{jindofs,juicefs}`** on the BE, from
`lib/{jindofs,juicefs}` (the FE keeps `fe/lib/<name>`). One copy on
disk, two readers:
`bin/start_be.sh` puts the directory on the system classpath for the
native libhdfs reader, and
`PluginRegistry` appends the
same jars to every plugin's classpath, after that plugin's own, so a
plugin's hadoop still wins.
`start_be.sh` still reads the two old locations as a fallback, with a
WARN, so an in-place upgrade
keeps resolving these schemes for the native reader before the new tree
is laid down - but only
the new location reaches the plugins.
Caveat, the same one the FE records for its paimon connector plugin:
jindo-core carries a native
library and a JVM binds one of those to exactly one classloader, so a
single BE process must not
reach jindo from two plugins at once, nor from a plugin and through
libhdfs at the same time. That
is inherent to plugin isolation rather than to this directory: a single
shared loader for these
jars is impossible, since they need the hadoop that lives inside each
plugin.
- **The JVM is created with `-Xrs`**, so it installs no SIGINT / SIGTERM
/ SIGQUIT handler of its
own. That is what keeps a shutdown signal from becoming a Java
`Shutdown.exit()` on a serving
BE now that the JVM comes up on demand. Consequences: no Java shutdown
hook runs on those
signals, and `kill -3` no longer produces a JVM thread dump (`jcmd` /
`jstack` still do). BE
installs a handler for SIGQUIT itself so that the signal does nothing -
left at its default it
would terminate the process and dump core, turning the habitual `kill -3
<be_pid>` from "print
the thread dump" into "kill the BE".
- **A statically loaded Java function is cached by its function id**,
not by its signature. Two
same-named functions in different databases used to share one compiled
class, and a variadic
function could never be invalidated at all - FE spells a variadic
signature with a trailing
`"..."` when it describes a function to execute and without one when it
drops it, so DROP plus
CREATE went on running the old code until the BE restarted. Both are
fixed; nothing about how a
function is declared changes.
- **`jvm_*` metrics appear on `/metrics` once the process has a JVM**,
whichever Java feature
brought it up - including an HDFS read through libhdfs on a BE that uses
no JNI table format and
no Java UDF. A BE that touches no Java feature at all has no JVM, and
reports none.
`GET /api/jni_plugin_status` (admin) reports the plugin state - loaded,
failed with the reason,
declared API version - and deliberately does not start a JVM to answer.
- **A UDTF declared with `static_load = true` is now actually statically
loaded.** The property was
read but never applied to table functions, so it was always false for
them; from here on such a
UDTF is cached process-wide, its classloader lives for the life of the
BE, and its classes are
shared between queries - which is what the property has always meant for
scalar and aggregate
functions.
- **A JDBC catalog's `driver_url` is verified against the MD5 the
catalog carries**, once per driver
jar rather than per query, by the scanner, the writer and the connection
test alike. A catalog
whose jar no longer matches its recorded checksum now fails with
"Checksum mismatch for JDBC
driver" instead of reading through whatever is behind the URL. A catalog
defined without a
checksum is unaffected.
- **Hadoop configuration for the Java plugins falls back to
`HADOOP_CONF_DIR`** when
`plugins/hadoop_conf/` does not exist, so a deployment that resolves its
HDFS HA nameservice
through that variable keeps working without copying files.
- **A `FOR TIME AS OF` / `FOR VERSION AS OF` query on a hudi catalog
with
`use_hive_sync_partition = true` now reports `partition=0/0` in EXPLAIN
and reads every
partition.** That listing comes from HMS, which holds what is synced
*now* and can only be
subtracted from, so it is a subset of the pinned universe - pruning
against it would silently
drop rows a time-travel query must read. The connector therefore
declines to claim the listing is
snapshot-exact, and an empty universe means scan-all. Coarse, never
short; the
`use_hive_sync_partition = false` listing comes from hudi's metadata
table, is snapshot-exact, and
prunes as before.
**New build-time gate**
`tools/be-java-plugins/check_plugin_layout.py` runs from `build.sh`,
right after the plugins are
deployed, over `output/be/lib/jni/spi` and `output/be/plugins/jni`. It
enforces five rules: the
SPI jar carries only SPI packages; no plugin ships an SPI class; no
class inside a plugin
directory resolves to different bytes in different jars; every class
referenced by Doris's own
code in a plugin resolves inside that directory; and the jar declaring
the `DorisPlugin` service
carries the `Doris-Jni-Plugin-Api-Version` this build serves. The
known-benign exceptions are
allowlisted with the reason written down.
It runs there rather than in a GitHub workflow because it needs a built
output tree, which only a
full BE build produces - no workflow in `.github/` builds BE with the
Java extensions. A violation
fails the build; a missing `python3` or `jdeps` is a warning, since that
is the check not running
rather than a verdict on the tree. The file also states what it does
**not** prove: it is static
and starts from Doris's classes, so anything reached by ServiceLoader or
reflection is invisible
to it.