nprasad2077/doris
1
0 Fork 0
mirror of https://github.com/apache/doris.git synced 2026-11-04 07:31:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
8ebf20dc3acb2a57ac7e52f9e73790cb8ffea459
doris/fe/be-java-extensions/trino-connector-scanner/pom.xml
T
Gabriel 8ebf20dc3a [feature](fe) Support MongoDB ObjectId and Trino JSON types (#69969) 
### What problem does this PR solve?

Issue Number: N/A

MongoDB exposes ObjectId as a plugin-defined Trino type. Doris did not
register these types in either plugin loader, rejected ObjectId during
schema mapping, and discarded the connector's default hidden `_id`
column. Register plugin-defined types in FE and BE, map ObjectId
recursively to `VARBINARY(12)`, and expose the default MongoDB `_id`
while retaining the hiding of other columns. Existing byte readers
preserve all twelve bytes without UTF-8 conversion.

Trino JSON columns also lacked a Doris type mapping. Map the built-in
Trino JSON type recursively to native Doris JSON, and parse its UTF-8
JNI carrier into JSONB. Preserve SQL NULL independently of the JSON
value `null`, including empty and repeated batches. Reuse the existing
JSON parser and return malformed-input errors. Correct nested JSON text
serialization so arrays, maps and structs contain JSON values with their
original quoting; previously an object could become invalid text such as
`["{"a":1}"]`. Coordinate typed ARRAY/MAP/STRUCT text parsing with that
formatter, retaining JSON string quotes and using uppercase `NULL` for
SQL-null JSON children versus lowercase JSON `null`. Both strict and
non-strict casts retain the original JSON type and null semantics.
Delegate complete child tokens to `from_string_as_nested`: scalar SerDes
remove outer quotes, JSONB retains JSON quotes and accepts legacy SQL
apostrophe wrappers, and Nullable handles SQL nulls and conversion
errors using the child's paired null-token policy. Generic complex
parsing no longer queries the concrete JSONB type.

Require the fixture's final network-facing MongoDB process before
marking it ready, and use bounded tmpfs mounts for both image-declared
data directories to avoid orphaned anonymous volumes.

Add an opt-in MongoDB 8.0.20 Docker fixture and five regression suites
with **138 result checks and ten expected-error checks**. Coverage
includes primitive and nested types, BSON documents and Extended JSON,
all JSON value shapes, null and missing fields, numeric/date boundaries,
Unicode and escapes, schema discovery/permissions, scalar and JSON
predicates, joins, aggregates, subqueries, windows, views,
internal-table joins, scalar JSON materialization, multi-page reads and
repeated early termination. Golden files are generated and compared with
the standard regression runner. Source-side MongoDB queries
independently confirm critical counts and null semantics.

MongoDB remains an operator-installed Trino 435 plugin; no MongoDB
runtime dependency is added to Doris. Use the complete matching plugin
ZIP on FE and BE.

ObjectId retains existing VARBINARY operator/storage restrictions; use
`from_binary(_id)` for hexadecimal filtering, joining, grouping or
materialization. Nested JSON is readable through catalogs, while
internal-table creation retains existing complex-type restrictions,
including rejecting `ARRAY<JSON>`. BSON documents inferred automatically
remain ROW/STRUCT; use an explicit `json` schema declaration to read a
field as a whole JSON value.

### Validation

- FE package and native ASAN BE built through `build.sh --fe` /
`build.sh --be`. UI, Hive UDF, OBS/COS dependencies and unrelated Java
plugins were excluded locally; the native build covers the initial
connector and JSON changes. The final SerDe interface refactor was
compiled and linked through the ASAN BE unit-test build; the standalone
BE package and external integration suites were not rerun for that
refactor. Final executable passed the GLIBC_2.17 compatibility gate.
- `run-be-ut.sh --run`: **33 related BE unit tests passed**, including
existing JSON serde, general complex-type parsing and formatting tests.
Typed JSON text covers 66 combinations of container, value shape and
strictness, plus 48 legacy apostrophe-wrapper combinations, deeper
nesting and malformed children in both modes. Reverting JNI support
reproduces three failures; the typed complex JSON test fails before the
reader fix and passes after it.
- `run-regression-test.sh --run -d
external_table_p0/trino_connector/mongodb`: **five suites passed**, with
**138 executed result checks and ten expected errors**, real MongoDB and
the complete Trino 435 plugin ZIP.
- Checkstyle/validate, clang-format 16 for all affected C++ files,
header hygiene and clang-tidy passed.
- **17 related regression suites passed** across the final comparison
and an Elasticsearch rerun after adjusting only local fixture
CPU/direct-memory limits. This includes the array-function
documentation, JSON/array/aggregate coverage, all four Elasticsearch
fixture versions, spill-enabled bucket-shuffle plans, HTTP TVF and FE/BE
version metrics.
- The generated-version test verifies refreshed numbers without
recompiling a cached consumer. Warmed FE builds verify changed and
restored release metadata after cache hits; bytecode inspection confirms
numeric fields are read at runtime.
- Gated MongoDB startup reproduces and fixes the initialization-server
readiness race; repeated start/stop cycles leave no anonymous data
volumes.
- Independent MongoDB queries confirmed source-side null semantics and
multi-page counts/sums.

Refresh the affected array-function documentation and Elasticsearch
golden results through the regression runner. Keep the bucket-shuffle
regression focused on physical INTERSECT/EXCEPT nodes when spill is
enabled by disabling their two spill rewrite rules. Pin the HTTP TVF
INT96 fixture timezone to UTC to preserve its stored wall-clock values
and existing golden results.

Refresh the FE build-version class after Maven cache restoration so
packaged FE/BE version metrics report the current revision. Keep numeric
version fields runtime-initialized so cached consumers observe refreshed
metadata. A standalone generator test covers replacing Version.class
without recompiling its consumer and invokes the actual packaging
function with a PATH-only JDK to verify both tool selection and
deployed-jar metadata.

The fixture README documents setup, JSON schema behavior, typed SQL text
null spelling, coverage and current limitations. MongoDB integration
suites are disabled by default and require the matching plugin on FE and
BE.

### Release note

Support MongoDB ObjectId columns, including the default `_id` and nested
fields, as `VARBINARY(12)` through Trino connector catalogs. Support
Trino JSON columns as native Doris JSON, including nested JSON fields,
and preserve JSON values and SQL-null distinctions through typed
complex-text round trips.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
- [x] Manual test: real MongoDB fixture, complete connector ZIP, exact
bytes and independent source-side JSON checks
- Behavior changed:
- [x] Yes. ObjectId and JSON fields become queryable with their native
Doris types, the default hidden MongoDB `_id` is exposed, and nested
JSON values retain their own quotes and null semantics.
- Does this need documentation?
- [x] Yes. Setup, JSON schema selection and limitations are documented
in `docker/thirdparties/docker-compose/mongodb/README.md`.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
2026-11-03 10:27:59 +08:00

354 lines
16 KiB
XML
Raw Blame History

<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>be-java-extensions</artifactId>
<groupId>org.apache.doris</groupId>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>trino-connector-scanner</artifactId>
<name>Doris BE trino-connector Plugin</name>
<description>
Reads any table a Trino connector can read. FE plans the scan with the same Trino connector
and JSON-serializes the split, the table handle and the column handles to this plugin, which
deserializes them against the connector and materializes rows - so the plugin carries the
Trino engine, and the connectors themselves are loaded from plugins/trino_plugins at runtime
by a second, nested classloader.
</description>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.doris</groupId>
<artifactId>jni-spi</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<!--
The Trino engine. Not just the SPI: deserializing a handle needs the connector's own
jackson modules and the block encodings, and building the session Trino hands a connector
needs the session property machinery, so what runs here is Trino's own bootstrap path with
a single catalog on it.
-->
<dependency>
<groupId>io.trino</groupId>
<artifactId>trino-main</artifactId>
<!--
ATTN: this replaces the exclusion list fe/pom.xml manages for trino-main rather than
adding to it - maven does not merge the two - so the first block below is that list,
copied. Dropping one of those silently puts a second logging backend or the Servlet
5.0 repackage back into the closure.
The second block is what BE does not need. trino-main is the whole Trino engine, and
BE runs one narrow path through it: load the installed connectors, deserialize one
split against one of them, pull pages. It never starts a Trino server, so nothing
constructs an endpoint, authenticates a client, announces itself to a discovery
service or exports a trace. Each group is excluded at its subtree's head, and what
survives is asserted by loading the deployed directory and reading rows through it -
see TrinoConnectorPluginTest and the acceptance recipe in the migration notes.
-->
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>log4j-over-slf4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-to-slf4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
</exclusion>
<exclusion>
<groupId>io.airlift</groupId>
<artifactId>bootstrap</artifactId>
</exclusion>
<exclusion>
<groupId>io.trino</groupId>
<artifactId>re2j</artifactId>
</exclusion>
<exclusion>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
</exclusion>
<exclusion>
<groupId>com.squareup.okio</groupId>
<artifactId>okio</artifactId>
</exclusion>
<exclusion>
<groupId>org.eclipse.jetty.toolchain</groupId>
<artifactId>jetty-jakarta-servlet-api</artifactId>
</exclusion>
<!-- Trino's HTTP server and the REST stack it serves: jetty's server side, jersey,
hk2, jaxb, the JMX and openmetrics endpoints, and the discovery service it
announces itself to (with the embedded leveldb that stores announcements).
BE serves nothing. The http *client* stays: it is reachable from the parts of
the engine BE does construct. -->
<exclusion>
<groupId>io.airlift</groupId>
<artifactId>http-server</artifactId>
</exclusion>
<exclusion>
<groupId>io.airlift</groupId>
<artifactId>jaxrs</artifactId>
</exclusion>
<exclusion>
<groupId>io.airlift</groupId>
<artifactId>jmx-http</artifactId>
</exclusion>
<exclusion>
<groupId>io.airlift</groupId>
<artifactId>openmetrics</artifactId>
</exclusion>
<exclusion>
<groupId>io.airlift</groupId>
<artifactId>event</artifactId>
</exclusion>
<exclusion>
<groupId>io.airlift.discovery</groupId>
<artifactId>discovery-server</artifactId>
</exclusion>
<exclusion>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-alpn-java-client</artifactId>
</exclusion>
<!-- Client authentication: JWT, OAuth2/OIDC. A Trino server authenticates the
clients that connect to it; nothing connects to BE's copy of the engine. -->
<exclusion>
<groupId>com.nimbusds</groupId>
<artifactId>nimbus-jose-jwt</artifactId>
</exclusion>
<exclusion>
<groupId>com.nimbusds</groupId>
<artifactId>oauth2-oidc-sdk</artifactId>
</exclusion>
<exclusion>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt-api</artifactId>
</exclusion>
<exclusion>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt-impl</artifactId>
</exclusion>
<exclusion>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt-jackson</artifactId>
</exclusion>
<!-- Trace export: the connector cache hands Trino OpenTelemetry.noop(), so the
OTLP exporter this drags in has nothing to export to. Two things stay on
purpose. The API and context packages, because Trino delegates those to a
plugin's classloader and an installed connector has to resolve them there;
and the two SDK modules trino-main declares itself, which this exclusion does
not reach - they are 0.4MB and trino-main naming them directly is a better
signal about its own paths than an analysis of BE's. -->
<exclusion>
<groupId>io.airlift</groupId>
<artifactId>tracing</artifactId>
</exclusion>
<!-- Reconfigures java.util.logging for a Trino server. BE's plugin runtime owns
that configuration - this being present is an invitation to a second one. -->
<exclusion>
<groupId>io.airlift</groupId>
<artifactId>log-manager</artifactId>
</exclusion>
<!-- Retry policy and latency histograms for the server's remote tasks. -->
<exclusion>
<groupId>dev.failsafe</groupId>
<artifactId>failsafe</artifactId>
</exclusion>
<exclusion>
<groupId>org.hdrhistogram</groupId>
<artifactId>HdrHistogram</artifactId>
</exclusion>
<!-- 8.6MB, and unreferenced: io.airlift:configuration declares it, but no class in
this closure names net.bytebuddy - not as a type and not as a string, which is
how hibernate-validator would have reached it had it still used it. -->
<exclusion>
<groupId>net.bytebuddy</groupId>
<artifactId>byte-buddy</artifactId>
</exclusion>
</exclusions>
</dependency>
<!--
The plugin/connector bootstrap classes FE and BE share, kept out of fe-common precisely so
that depending on them does not put trino-main on FE's own classpath.
-->
<dependency>
<groupId>org.apache.doris</groupId>
<artifactId>fe-trino-connector-common</artifactId>
<version>${project.version}</version>
</dependency>
<!--
Reached transitively through the shared preload jar until this module became a plugin:
guava for the connector cache, commons-lang3 for the root-cause message the cache reports,
commons-io for reading /proc/<pid>/cmdline. Compiling was never the problem - all three
were on the compile classpath as transitive `provided` dependencies - which is why a
missing declaration here only ever shows up as a NoClassDefFoundError in a deployed
plugin directory.
-->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
</dependency>
<!--
Logging: this plugin and the libraries inside it must reach the one file BE's plugin
runtime configures - log/jni.log, written through java.util.logging. So the plugin speaks
slf4j and deploys slf4j-jdk14. Trino's own io.airlift.log.Logger already writes to
java.util.logging, so it needs no bridge at all.
log4j-to-slf4j is the bridge for the log4j2 callers in this closure:
fe-trino-connector-common, which is shared with FE and logs the plugin scan there.
Without it log4j-api finds no provider and falls back to writing on stderr.
The three log4j artifacts below are narrowed to test because the fe parent pom declares
them compile on EVERY module: log4j-core is a second logging implementation writing
wherever its own configuration says (nowhere, since BE's log4j2 config is not on a
plugin's classpath), log4j-slf4j2-impl would route slf4j into it, and log4j-1.2-api is not
the pure facade its name suggests - its org.apache.log4j.Logger resolves log4j-core
classes. Maven cannot drop an inherited dependency, only narrow its scope.
-->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<version>${slf4j.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-to-slf4j</artifactId>
<version>${log4j2.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-1.2-api</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.trino</groupId>
<artifactId>trino-mongodb</artifactId>
<version>${trino.version}</version>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<build>
<finalName>trino-connector-scanner</finalName>
<directory>${project.basedir}/target/</directory>
<plugins>
<!--
Give the tests the same logging the deployed plugin has: slf4j-jdk14 as slf4j's only
provider, log4j-to-slf4j as log4j2's. Both artifacts excluded here are declared test
only because maven cannot drop what the fe parent pom declares compile on every
module - narrowing the scope keeps them out of the plugin directory, and this keeps
them out of the test JVM, where each would be a second provider competing for the
same slot.
-->
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<classpathDependencyExcludes>
<classpathDependencyExclude>
org.apache.logging.log4j:log4j-core
</classpathDependencyExclude>
<classpathDependencyExclude>
org.apache.logging.log4j:log4j-slf4j2-impl
</classpathDependencyExclude>
</classpathDependencyExcludes>
</configuration>
</plugin>
<!--
A plugin is a directory of jars, not one jar with everything shaded into it. Its own
jar is built by the parent's jar plugin; its runtime closure lands beside it here.
`provided` dependencies are excluded by scope, which is what keeps the SPI out.
-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy-plugin-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<includeScope>runtime</includeScope>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Reference in new issue View Git Blame Copy Permalink