mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Gabriel
8ebf20dc3a
### What problem does this PR solve?
Issue Number: N/A
MongoDB exposes ObjectId as a plugin-defined Trino type. Doris did not
register these types in either plugin loader, rejected ObjectId during
schema mapping, and discarded the connector's default hidden `_id`
column. Register plugin-defined types in FE and BE, map ObjectId
recursively to `VARBINARY(12)`, and expose the default MongoDB `_id`
while retaining the hiding of other columns. Existing byte readers
preserve all twelve bytes without UTF-8 conversion.
Trino JSON columns also lacked a Doris type mapping. Map the built-in
Trino JSON type recursively to native Doris JSON, and parse its UTF-8
JNI carrier into JSONB. Preserve SQL NULL independently of the JSON
value `null`, including empty and repeated batches. Reuse the existing
JSON parser and return malformed-input errors. Correct nested JSON text
serialization so arrays, maps and structs contain JSON values with their
original quoting; previously an object could become invalid text such as
`["{"a":1}"]`. Coordinate typed ARRAY/MAP/STRUCT text parsing with that
formatter, retaining JSON string quotes and using uppercase `NULL` for
SQL-null JSON children versus lowercase JSON `null`. Both strict and
non-strict casts retain the original JSON type and null semantics.
Delegate complete child tokens to `from_string_as_nested`: scalar SerDes
remove outer quotes, JSONB retains JSON quotes and accepts legacy SQL
apostrophe wrappers, and Nullable handles SQL nulls and conversion
errors using the child's paired null-token policy. Generic complex
parsing no longer queries the concrete JSONB type.
Require the fixture's final network-facing MongoDB process before
marking it ready, and use bounded tmpfs mounts for both image-declared
data directories to avoid orphaned anonymous volumes.
Add an opt-in MongoDB 8.0.20 Docker fixture and five regression suites
with **138 result checks and ten expected-error checks**. Coverage
includes primitive and nested types, BSON documents and Extended JSON,
all JSON value shapes, null and missing fields, numeric/date boundaries,
Unicode and escapes, schema discovery/permissions, scalar and JSON
predicates, joins, aggregates, subqueries, windows, views,
internal-table joins, scalar JSON materialization, multi-page reads and
repeated early termination. Golden files are generated and compared with
the standard regression runner. Source-side MongoDB queries
independently confirm critical counts and null semantics.
MongoDB remains an operator-installed Trino 435 plugin; no MongoDB
runtime dependency is added to Doris. Use the complete matching plugin
ZIP on FE and BE.
ObjectId retains existing VARBINARY operator/storage restrictions; use
`from_binary(_id)` for hexadecimal filtering, joining, grouping or
materialization. Nested JSON is readable through catalogs, while
internal-table creation retains existing complex-type restrictions,
including rejecting `ARRAY<JSON>`. BSON documents inferred automatically
remain ROW/STRUCT; use an explicit `json` schema declaration to read a
field as a whole JSON value.
### Validation
- FE package and native ASAN BE built through `build.sh --fe` /
`build.sh --be`. UI, Hive UDF, OBS/COS dependencies and unrelated Java
plugins were excluded locally; the native build covers the initial
connector and JSON changes. The final SerDe interface refactor was
compiled and linked through the ASAN BE unit-test build; the standalone
BE package and external integration suites were not rerun for that
refactor. Final executable passed the GLIBC_2.17 compatibility gate.
- `run-be-ut.sh --run`: **33 related BE unit tests passed**, including
existing JSON serde, general complex-type parsing and formatting tests.
Typed JSON text covers 66 combinations of container, value shape and
strictness, plus 48 legacy apostrophe-wrapper combinations, deeper
nesting and malformed children in both modes. Reverting JNI support
reproduces three failures; the typed complex JSON test fails before the
reader fix and passes after it.
- `run-regression-test.sh --run -d
external_table_p0/trino_connector/mongodb`: **five suites passed**, with
**138 executed result checks and ten expected errors**, real MongoDB and
the complete Trino 435 plugin ZIP.
- Checkstyle/validate, clang-format 16 for all affected C++ files,
header hygiene and clang-tidy passed.
- **17 related regression suites passed** across the final comparison
and an Elasticsearch rerun after adjusting only local fixture
CPU/direct-memory limits. This includes the array-function
documentation, JSON/array/aggregate coverage, all four Elasticsearch
fixture versions, spill-enabled bucket-shuffle plans, HTTP TVF and FE/BE
version metrics.
- The generated-version test verifies refreshed numbers without
recompiling a cached consumer. Warmed FE builds verify changed and
restored release metadata after cache hits; bytecode inspection confirms
numeric fields are read at runtime.
- Gated MongoDB startup reproduces and fixes the initialization-server
readiness race; repeated start/stop cycles leave no anonymous data
volumes.
- Independent MongoDB queries confirmed source-side null semantics and
multi-page counts/sums.
Refresh the affected array-function documentation and Elasticsearch
golden results through the regression runner. Keep the bucket-shuffle
regression focused on physical INTERSECT/EXCEPT nodes when spill is
enabled by disabling their two spill rewrite rules. Pin the HTTP TVF
INT96 fixture timezone to UTC to preserve its stored wall-clock values
and existing golden results.
Refresh the FE build-version class after Maven cache restoration so
packaged FE/BE version metrics report the current revision. Keep numeric
version fields runtime-initialized so cached consumers observe refreshed
metadata. A standalone generator test covers replacing Version.class
without recompiling its consumer and invokes the actual packaging
function with a PATH-only JDK to verify both tool selection and
deployed-jar metadata.
The fixture README documents setup, JSON schema behavior, typed SQL text
null spelling, coverage and current limitations. MongoDB integration
suites are disabled by default and require the matching plugin on FE and
BE.
### Release note
Support MongoDB ObjectId columns, including the default `_id` and nested
fields, as `VARBINARY(12)` through Trino connector catalogs. Support
Trino JSON columns as native Doris JSON, including nested JSON fields,
and preserve JSON values and SQL-null distinctions through typed
complex-text round trips.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test: real MongoDB fixture, complete connector ZIP, exact
bytes and independent source-side JSON checks
- Behavior changed:
- [x] Yes. ObjectId and JSON fields become queryable with their native
Doris types, the default hidden MongoDB `_id` is exposed, and nested
JSON values retain their own quotes and null semantics.
- Does this need documentation?
- [x] Yes. Setup, JSON schema selection and limitations are documented
in `docker/thirdparties/docker-compose/mongodb/README.md`.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label