nprasad2077/doris
1
0 Fork 0
mirror of https://github.com/apache/doris.git synced 2026-11-04 07:31:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
8ebf20dc3acb2a57ac7e52f9e73790cb8ffea459
doris/regression-test/conf/regression-conf.groovy
T
Gabriel 8ebf20dc3a [feature](fe) Support MongoDB ObjectId and Trino JSON types (#69969) 
### What problem does this PR solve?

Issue Number: N/A

MongoDB exposes ObjectId as a plugin-defined Trino type. Doris did not
register these types in either plugin loader, rejected ObjectId during
schema mapping, and discarded the connector's default hidden `_id`
column. Register plugin-defined types in FE and BE, map ObjectId
recursively to `VARBINARY(12)`, and expose the default MongoDB `_id`
while retaining the hiding of other columns. Existing byte readers
preserve all twelve bytes without UTF-8 conversion.

Trino JSON columns also lacked a Doris type mapping. Map the built-in
Trino JSON type recursively to native Doris JSON, and parse its UTF-8
JNI carrier into JSONB. Preserve SQL NULL independently of the JSON
value `null`, including empty and repeated batches. Reuse the existing
JSON parser and return malformed-input errors. Correct nested JSON text
serialization so arrays, maps and structs contain JSON values with their
original quoting; previously an object could become invalid text such as
`["{"a":1}"]`. Coordinate typed ARRAY/MAP/STRUCT text parsing with that
formatter, retaining JSON string quotes and using uppercase `NULL` for
SQL-null JSON children versus lowercase JSON `null`. Both strict and
non-strict casts retain the original JSON type and null semantics.
Delegate complete child tokens to `from_string_as_nested`: scalar SerDes
remove outer quotes, JSONB retains JSON quotes and accepts legacy SQL
apostrophe wrappers, and Nullable handles SQL nulls and conversion
errors using the child's paired null-token policy. Generic complex
parsing no longer queries the concrete JSONB type.

Require the fixture's final network-facing MongoDB process before
marking it ready, and use bounded tmpfs mounts for both image-declared
data directories to avoid orphaned anonymous volumes.

Add an opt-in MongoDB 8.0.20 Docker fixture and five regression suites
with **138 result checks and ten expected-error checks**. Coverage
includes primitive and nested types, BSON documents and Extended JSON,
all JSON value shapes, null and missing fields, numeric/date boundaries,
Unicode and escapes, schema discovery/permissions, scalar and JSON
predicates, joins, aggregates, subqueries, windows, views,
internal-table joins, scalar JSON materialization, multi-page reads and
repeated early termination. Golden files are generated and compared with
the standard regression runner. Source-side MongoDB queries
independently confirm critical counts and null semantics.

MongoDB remains an operator-installed Trino 435 plugin; no MongoDB
runtime dependency is added to Doris. Use the complete matching plugin
ZIP on FE and BE.

ObjectId retains existing VARBINARY operator/storage restrictions; use
`from_binary(_id)` for hexadecimal filtering, joining, grouping or
materialization. Nested JSON is readable through catalogs, while
internal-table creation retains existing complex-type restrictions,
including rejecting `ARRAY<JSON>`. BSON documents inferred automatically
remain ROW/STRUCT; use an explicit `json` schema declaration to read a
field as a whole JSON value.

### Validation

- FE package and native ASAN BE built through `build.sh --fe` /
`build.sh --be`. UI, Hive UDF, OBS/COS dependencies and unrelated Java
plugins were excluded locally; the native build covers the initial
connector and JSON changes. The final SerDe interface refactor was
compiled and linked through the ASAN BE unit-test build; the standalone
BE package and external integration suites were not rerun for that
refactor. Final executable passed the GLIBC_2.17 compatibility gate.
- `run-be-ut.sh --run`: **33 related BE unit tests passed**, including
existing JSON serde, general complex-type parsing and formatting tests.
Typed JSON text covers 66 combinations of container, value shape and
strictness, plus 48 legacy apostrophe-wrapper combinations, deeper
nesting and malformed children in both modes. Reverting JNI support
reproduces three failures; the typed complex JSON test fails before the
reader fix and passes after it.
- `run-regression-test.sh --run -d
external_table_p0/trino_connector/mongodb`: **five suites passed**, with
**138 executed result checks and ten expected errors**, real MongoDB and
the complete Trino 435 plugin ZIP.
- Checkstyle/validate, clang-format 16 for all affected C++ files,
header hygiene and clang-tidy passed.
- **17 related regression suites passed** across the final comparison
and an Elasticsearch rerun after adjusting only local fixture
CPU/direct-memory limits. This includes the array-function
documentation, JSON/array/aggregate coverage, all four Elasticsearch
fixture versions, spill-enabled bucket-shuffle plans, HTTP TVF and FE/BE
version metrics.
- The generated-version test verifies refreshed numbers without
recompiling a cached consumer. Warmed FE builds verify changed and
restored release metadata after cache hits; bytecode inspection confirms
numeric fields are read at runtime.
- Gated MongoDB startup reproduces and fixes the initialization-server
readiness race; repeated start/stop cycles leave no anonymous data
volumes.
- Independent MongoDB queries confirmed source-side null semantics and
multi-page counts/sums.

Refresh the affected array-function documentation and Elasticsearch
golden results through the regression runner. Keep the bucket-shuffle
regression focused on physical INTERSECT/EXCEPT nodes when spill is
enabled by disabling their two spill rewrite rules. Pin the HTTP TVF
INT96 fixture timezone to UTC to preserve its stored wall-clock values
and existing golden results.

Refresh the FE build-version class after Maven cache restoration so
packaged FE/BE version metrics report the current revision. Keep numeric
version fields runtime-initialized so cached consumers observe refreshed
metadata. A standalone generator test covers replacing Version.class
without recompiling its consumer and invokes the actual packaging
function with a PATH-only JDK to verify both tool selection and
deployed-jar metadata.

The fixture README documents setup, JSON schema behavior, typed SQL text
null spelling, coverage and current limitations. MongoDB integration
suites are disabled by default and require the matching plugin on FE and
BE.

### Release note

Support MongoDB ObjectId columns, including the default `_id` and nested
fields, as `VARBINARY(12)` through Trino connector catalogs. Support
Trino JSON columns as native Doris JSON, including nested JSON fields,
and preserve JSON values and SQL-null distinctions through typed
complex-text round trips.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
- [x] Manual test: real MongoDB fixture, complete connector ZIP, exact
bytes and independent source-side JSON checks
- Behavior changed:
- [x] Yes. ObjectId and JSON fields become queryable with their native
Doris types, the default hidden MongoDB `_id` is exposed, and nested
JSON values retain their own quotes and null semantics.
- Does this need documentation?
- [x] Yes. Setup, JSON schema selection and limitations are documented
in `docker/thirdparties/docker-compose/mongodb/README.md`.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
2026-11-03 10:27:59 +08:00

376 lines
12 KiB
Groovy
Raw Blame History

// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
/* ******* Do not commit this file unless you know what you are doing ******* */
// **Note**: default db will be create if not exist
defaultDb = "regression_test"
// add useLocalSessionState so that the jdbc will not send
// init cmd like: select @@session.tx_read_only
// at each time we connect.
// add allowLoadLocalInfile so that the jdbc can execute mysql load data from client.
jdbcUrl = "jdbc:mysql://127.0.0.1:9030/?useLocalSessionState=true&allowLoadLocalInfile=true&zeroDateTimeBehavior=round"
targetJdbcUrl = "jdbc:mysql://127.0.0.1:9030/?useLocalSessionState=true&allowLoadLocalInfile=true&zeroDateTimeBehavior=round"
jdbcUser = "root"
jdbcPassword = ""
feSourceThriftAddress = "127.0.0.1:9020"
feTargetThriftAddress = "127.0.0.1:9020"
syncerAddress = "127.0.0.1:9190"
feSyncerUser = "root"
feSyncerPassword = ""
feHttpAddress = "127.0.0.1:8030"
feHttpUser = "root"
feHttpPassword = ""
// ccr, need BE enable_feature_binlog=true and FE enable_feature_binlog = true
ccrDownstreamUrl = "jdbc:mysql://127.0.0.1:9030/?useLocalSessionState=true&allowLoadLocalInfile=true"
ccrDownstreamUser = "root"
ccrDownstreamPassword = ""
ccrDownstreamFeThriftAddress = "127.0.0.1:9030"
// set DORIS_HOME by system properties
// e.g. java -DDORIS_HOME=./
suitePath = "${DORIS_HOME}/regression-test/suites"
dataPath = "${DORIS_HOME}/regression-test/data"
pluginPath = "${DORIS_HOME}/regression-test/plugins"
realDataPath = "${DORIS_HOME}/regression-test/realdata"
sslCertificatePath = "${DORIS_HOME}/regression-test/ssl_default_certificate"
// suite configs
suites = {
//// equals to:
//// suites.test_suite_1.key1 = "val1"
//// suites.test_suite_1.key2 = "val2"
////
//test_suite_1 {
// key1 = "val1"
// key2 = "val2"
//}
//test_suite_2 {
// key3 = "val1"
// key4 = "val2"
//}
}
// docker image
image = ""
dockerCoverageOutputDir = "" // if not empty, will save docker coverage output files
dockerEndDeleteFiles = false
dorisComposePath = "${DORIS_HOME}/docker/runtime/doris-compose/doris-compose.py"
// do run docker test because pipeline not support build image now
excludeDockerTest = true
// will test <group>/<suite>.groovy
// empty group will test all group
testGroups = ""
// empty suite will test all suite
testSuites = ""
// empty directories will test all directories
testDirectories = ""
// this groups will not be executed
excludeGroups = ""
// this suites will not be executed
excludeSuites = "test_broker_load"
// this directories will not be executed
excludeDirectories = "000_the_start_sentinel_do_not_touch," + // keep this line as the first line
"segcompaction_p2," +
"workload_manager_p1," +
"cloud_p0/cache," +
"load_p0/msk_load," +
"zzz_the_end_sentinel_do_not_touch," +
"dialect_compatible"// keep this line as the last line
customConf1 = "test_custom_conf_value"
// for test csv with header
enableHdfs=false // set to true if hdfs is ready
hdfsFs = "hdfs://127.0.0.1:9000"
hdfsUser = "doris-test"
hdfsPasswd = ""
brokerName = "broker_name"
// broker load test config
enableBrokerLoad=true
// for s3 releated cases, "aliyun" or "aliyun-internal" or "tencent" or "huawei" or "azure" or "gcp"
// if s3Source is set, s3Endpoint s3BucketName s3Region s3Provider will be filled with default value if not set
s3Source="aliyun"
// s3Endpoint = ""
// s3BucketName = ""
// s3Region = ""
// s3Provider = ""
ak="***********"
sk="***********"
// jdbc connector test config
// To enable jdbc test, you need first start mysql/pg container.
// See `docker/thirdparties/run-thirdparties-docker.sh`
enableJdbcTest=false
mysql_57_port=3316
pg_14_port=5442
oracle_11_port=1521
sqlserver_2022_port=1433
clickhouse_22_port=8123
doris_port=9030
mariadb_10_port=3326
db2_11_port=50000
oceanbase_port=2881
oceanbase_cdc_port=2883
// hive catalog test config
// To enable hive/paimon test, you need first start hive container.
// See `docker/thirdparties/run-thirdparties-docker.sh`
enableHiveTest=false
enablePaimonTest=false
// port of hive2 docker
hive2HmsPort=9083
hive2HdfsPort=8020
hive2ServerPort=10000
hive2PgPort=5432
// port of hive3 docker
hive3HmsPort=9383
hive3HdfsPort=8320
hive3ServerPort=13000
hive3PgPort=5732
// kafka test config
// to enable kafka test, you need firstly to start kafka container
// See `docker/thirdparties/run-thirdparties-docker.sh`
enableKafkaTest=false
kafka_port=19193
// fluss catalog test config
// to enable fluss test, you need firstly to start fluss containers
// See `docker/thirdparties/run-thirdparties-docker.sh -c fluss`
enableFlussTest=false
fluss_coordinator_port=19123
// The object store the fluss lake lives in. Fluss removes every lake option whose
// name holds key, secret or password before it hands a table's properties to a
// client, so a suite that reads the lake has to tell its own catalog how to reach
// that store; the credentials are the minio defaults and are spelled out there.
fluss_minio_port=19125
// elasticsearch catalog test config
// See `docker/thirdparties/run-thirdparties-docker.sh`
enableEsTest=false
es_5_port=59200
es_6_port=19200
es_7_port=29200
es_8_port=39200
//hive catalog test config for bigdata
enableExternalHiveTest = false
extHiveHmsHost = "***.**.**.**"
extHiveHmsPort = 7004
extHdfsPort = 4007
extHiveServerPort= 7001
extHiveHmsUser = "****"
extHiveHmsPassword= "***********"
dfsNameservices=""
dfsNameservicesPort=8020
dfsHaNamenodesHdfsCluster=""
dfsNamenodeRpcAddress1=""
dfsNamenodeRpcAddress2=""
dfsNamenodeRpcAddress3=""
hadoopSecurityAuthentication = ""
hadoopKerberosKeytabPath = ""
hadoopKerberosPrincipal = ""
hadoopSecurityAutoToLocal = ""
//paimon catalog test config for bigdata
enableExternalPaimonTest = false
//mysql jdbc connector test config for bigdata
enableExternalMysqlTest = false
extMysqlHost = "***.**.**.**"
extMysqlPort = 3306
extMysqlUser = "****"
extMysqlPassword = "***********"
//postgresql jdbc connector test config for bigdata
enableExternalPgTest = false
extPgHost = "***.**.**.*"
extPgPort = 5432
extPgUser = "****"
extPgPassword = "***********"
// minio external test config
enableExternalMinioTest = false
extMinioHost = "***.**.**.**"
extMinioPort = 9000
extMinioDomain = "myminio.com"
extMinioAk = "minioadmin"
extMinioSk = "minioadmin"
extMinioRegion = "us-east-1"
extMinioBucket = "test-bucket"
enableExternalHudiTest = false
hudiEmrCatalog = "***********"
enableObjStorageTest=false
enableMaxComputeTest=false
aliYunAk="***********"
dlfUid="***********"
aliYunSk="***********"
hwYunAk="***********"
hwYunSk="***********"
txYunAk="***********"
txYunSk="***********"
//arrow flight sql test config
extArrowFlightSqlHost = "127.0.0.1"
extArrowFlightSqlPort = 8081
extArrowFlightSqlUser = "root"
extArrowFlightSqlPassword= ""
extArrowFlightHttpPort= 8030
// iceberg rest catalog config
iceberg_rest_uri_port=18181
iceberg_scan_planning_rest_uri_port=18182
iceberg_minio_port=19001
iceberg_rest_uri_port_s3=19181
iceberg_rest_uri_port_oss=19182
iceberg_rest_uri_port_cos=19183
iceberg_rest_uri_port_obs=19184
iceberg_rest_uri_port_gcs=19185
iceberg_rest_uri_port_hdfs=19186
iceberg_rest_hdfs_port=20020
// polaris rest catalog config
polaris_rest_uri_port=20181
polaris_minio_port=20001
// If the failure suite num exceeds this config
// all following suite will be skipped to fast quit the run.
// <=0 means no limit.
max_failure_num=0
// used for exporting test
s3ExportBucketName = ""
externalEnvIp="127.0.0.1"
// Optional Trino JDBC regressions use the Trino service in the Iceberg Docker stack.
// Match trino_jdbc_port to its published TRINO_JDBC_PORT and externalEnvIp to the Docker host.
enableTrinoJdbcTest = false
trino_jdbc_port = 8080
// trino-connector catalog test config
enableTrinoConnectorTest = false
// Optional MongoDB fixture: run-thirdparties-docker.sh -c mongodb.
enableMongoDBTest = false
mongodb_port = 27017
mongodb_user = "mongodb_test"
mongodb_password = "mongodb_test"
enableKerberosTest=false
kerberosHmsPort=9883
kerberosHdfsPort=8820
enableRefactorParamsHdfsTest=true
// cloud
metaServiceToken = "greedisgood9999"
instanceId = "default_instance_id"
multiClusterInstance = "default_instance_id"
storageProvider = "oss"
cbsS3Ak = "*******"
cbsS3Sk = "*******"
cbsS3Endpoint = "oss-cn-beijing.aliyuncs.com"
cbsS3Bucket = "test-bucket"
cbsS3Prefix = "test-cluster-prefix"
// External Env
enableExternalIcebergTest = false
// The properties string of iceberg catalog
icebergS3TablesCatalog = ""
enableExternalHudiTest = false
// The properties string of hudi catalog
hudiEmrCatalog = ""
icebergS3TablesCatalog=""
icebergS3TablesCatalogGlueRest=""
// The path of the cert configuration file for the testing framework
// is consistent with the path of the cert file for the cluster
enableTLS=false
tlsVerifyMode="strict"
keyStorePath="/your/keystore.p12"
keyStorePassword="yourPwd"
keyStoreType="PKCS12"
trustStorePath="/your/truststore.p12"
trustStorePassword="yourPwd"
trustStoreType="PKCS12"
trustCert="/your/certificate.crt"
trustCACert="/your/ca.crt"
trustCAKey="/your/certificate.key"
enableTestTvfAnonymous="true"
anymousS3Uri="https://datasets-documentation.s3.eu-west-3.amazonaws.com/aapl_stock.csv"
anymousS3Region="eu-west-3"
anymousS3ExpectDataCount="8365"
awsInstanceProfileRegion="us-east-1"
// Object storage IAM regression tests run one provider per invocation: AWS or GCP.
objectStorageIamProvider=""
objectStorageIamEndpoint=""
objectStorageIamRegion=""
objectStorageIamBucket=""
objectStorageIamPrefix="regression/object_storage_iam_p0"
objectStorageIamDataPath="regression/tpch/sf0.01/customer.csv.gz"
// AWS authentication.
objectStorageIamAwsRoleArn=""
objectStorageIamAwsExternalId=""
// GCP authentication. Credential provider types are comma-separated DEFAULT and/or
// COMPUTE_ENGINE. DEFAULT also tests omitted authentication properties; leaving both GCP
// settings empty tests only omitted properties. A non-empty account adds COMPUTE_ENGINE
// impersonation. The IAM data fixture must be private so anonymous reads can be rejected.
objectStorageIamGcpCredentialProviderTypes=""
objectStorageIamGcpImpersonationServiceAccount=""
// Optional public GCS fixture for anonymous TVF reads: gzip CSV with a '|' separator.
// Configure a gs:// URI and its positive row count to enable test_gcp_tvf_anonymous.
// This public fixture must differ from the private objectStorageIamDataPath fixture.
objectStorageIamGcpAnonymousUri=""
objectStorageIamGcpAnonymousExpectedRows=""
// hudi p0 external regression test config
// To enable hudi test, you need first start hudi container.
// See `docker/thirdparties/run-thirdparties-docker.sh -c hudi`
enableHudiTest=true
// hudi catalog config
hudiHmsPort=19083
hudiMinioPort=19100
hudiMinioAccessKey="minio"
hudiMinioSecretKey="minio123"
icebergDlfRestCatalog="'type' = 'iceberg', 'warehouse' = 'new_dlf_iceberg_catalog', 'iceberg.catalog.type' = 'rest', 'iceberg.rest.uri' = 'http://cn-beijing-vpc.dlf.aliyuncs.com/iceberg', 'iceberg.rest.sigv4-enabled' = 'true', 'iceberg.rest.signing-name' = 'DlfNext', 'iceberg.rest.access-key-id' = 'ak', 'iceberg.rest.secret-access-key' = 'sk', 'iceberg.rest.signing-region' = 'cn-beijing', 'iceberg.rest.vended-credentials-enabled' = 'true', 'io-impl' = 'org.apache.iceberg.rest.DlfFileIO', 'fs.oss.support' = 'true'"
// For python UDF test, set the runtime version of python, default: 3.8.10
// pythonUdfRuntimeVersion = ""
Reference in new issue View Git Blame Copy Permalink