nprasad2077/doris
1
0 Fork 0
mirror of https://github.com/apache/doris.git synced 2026-11-04 07:31:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
master
doris/.licenserc.yaml
T
Mingyu Chen (Rayner)andkid d420f9c41b [fix](fe) Load FE images that carry Lance catalogs written by 4.1.4+ (#69575) 
### What problem does this PR solve?

Issue Number: None

Related PR: #65730, #67235, #68781

Problem Summary:

**In short.** This PR makes master read the Lance catalogs that
branch-4.1 persists (#65730, released in 4.1.4): an FE image written by
4.1.4 or later with a Lance catalog in it now loads. Before this PR the
image failed to parse, so a cluster with a Lance catalog could not be
upgraded from any released 4.1.4 / 4.1.4.1 / 4.1.5 build (or from 4.2)
to a master-based build: the FE did not start. The Lance catalog's GSON
labels are mapped to the plugin-driven classes, as the Iceberg, Paimon
and LakeSoul labels already are, and the catalog stays registered,
unusable until a Lance connector plugin is installed.

**Background.** branch-4.1 added a Lance catalog as fe-core classes in
#65730 (released in 4.1.4). Like every external catalog, it is persisted
in the image's `datasource` module as GSON JSON whose `clazz` field
names the class, `LanceExternalCatalog`, with `logType` `LANCE`. master
has no Lance classes: external catalogs are served by connector plugins,
and the labels of the catalogs that used to be fe-core classes are
redirected to `PluginDrivenExternalCatalog` /
`PluginDrivenExternalDatabase` / `PluginDrivenMvccExternalTable` by
`registerCompatibleSubtype` in `GsonUtils`. A Lance connector plugin
does not exist on master yet; it will be ported later.

branch-4.1 also had, for a while, a durable Lance index job framework
(#67235) that wrote edit-log op 500 (`OP_LANCE_INDEX_JOB_UPSERT`) and an
image module named `lanceIndexJobManager`. #68781 reverted it on
branch-4.1 and branch-4.2 before any release: among the release tags
only 4.1.5-rc01 has the writer, while 4.1.5 and 4.2.0-rc01 keep just a
tombstone that reads and discards both. master never had the framework,
and this PR does not make it read that metadata either.

The upgrade fixtures on master (`upgrade/413`) were generated from
4.1.3, before Lance existed, so no test caught this.

**The problem, and what it cost.** Compared on master against fixtures
emitted by real 4.1.5 bytecode:

| Metadata from a 4.1.4+ FE | master before this PR | after this PR |
|---|---|---|
| image whose `datasource` module has a Lance catalog (every 4.1.4+
cluster with one) | image fails to parse: `cannot deserialize interface
org.apache.doris.datasource.CatalogIf subtype named
LanceExternalCatalog; did you forget to register a subtype?`; the FE
cannot start | loads; the catalog is a `PluginDrivenExternalCatalog` of
type `lance` with all its properties, comment and auto-analyze policies;
using it fails with `No ConnectorProvider found for plugin-driven
catalog: <name>, type: lance. Ensure the connector plugin is installed.`
|
| `OP_CREATE_CATALOG` of a Lance catalog | replays as a degraded catalog
(the entry carries only properties) | unchanged |

The only labels in the GSON registries of 4.1.4.1, 4.1.5 and 4.2.0-rc01
that master does not know are `LanceExternalCatalog`,
`LanceExternalDatabase` and `LanceExternalTable`.

**How this PR fixes it.**

- `GsonUtils`: `LanceExternalCatalog` → `PluginDrivenExternalCatalog`,
`LanceExternalDatabase` → `PluginDrivenExternalDatabase`,
`LanceExternalTable` → `PluginDrivenMvccExternalTable`
(`LanceExternalTable` implemented `MvccTable`). 4.1.x never persists an
external database or table, so only the catalog label can actually
appear; the other two complete the registry, as for the other migrated
connectors.
- `InitCatalogLog.Type` keeps `LANCE`, so the `logType` 4.1.4+ wrote
still deserializes to itself. Nothing depends on it: a Lance catalog
always persists its `type` property (no resource type creates one), and
`PluginDrivenExternalCatalog#gsonPostProcess` rewrites the logType to
`PLUGIN`.
- `BdbjeJournalDataProcNode`, which renders `SHOW PROC
'/bdbje/<db>/<journalId>'` in BDBJE debug mode, prints a record read
without a payload as `null`, as `BDBTool` already does, instead of
failing with a `NullPointerException`. This is an existing master issue
that came up in review, not a Lance one: the readers of the retired
operation types already on master (the PL/SQL ones of #58700 and the
external-catalog ones) keep no payload.

Until a Lance connector plugin is installed, an upgraded cluster lists
its Lance catalogs and fails only when one is used; installing the
plugin makes them work without any metadata change, since the catalog
properties are handed to the connector unchanged.

**Results.**

- New golden fixtures `fe/fe-core/src/test/resources/upgrade/415/`,
emitted by 4.1.5-release bytecode like `upgrade/413` (generator and
`PROVENANCE.txt` included, byte-deterministic): the `datasource` module
with three Lance catalogs (filesystem on S3, filesystem on `file://`,
REST with bearer auth), their `OP_CREATE_CATALOG` entries, and 4.1.5's
label registries.
- `Legacy415LanceUpgradeTest` (7 tests): the image loads and every Lance
catalog becomes a `PluginDrivenExternalCatalog` of type `lance` with the
`PLUGIN` logType; every persisted property, the comment,
`lastUpdateTime` and the auto-analyze policy survive; the persisted
logType names still deserialize; without a plugin the catalog fails only
when used, with the message above; a provider claiming `lance` receives
exactly the persisted properties, from the image and from the journal;
the journal entries replay with and without a plugin.
- `LanceGsonCompatReplayTest` (4 tests): the three labels replay as the
expected classes (the table as the MVCC variant), and they are the only
labels 4.1.5 registered beyond 4.1.3.
- Each of these changes, undone on its own, turns tests red: dropping
the catalog label (5 tests, with the parse error above), the database
label, mapping the table label to the non-MVCC class, or removing
`LANCE`.
- `BdbjeJournalDataProcNodeTest` (2 tests): a retired PL/SQL record, in
its written form, shows its journal ID, operation name, size and `null`
data in the debug view; a live `OP_SAVE_NEXTID` record still shows its
payload. With `getData().toString()` back, the first test fails with a
`NullPointerException`.
- The three classes above with 13 related upgrade, GSON and journal
suites (`Legacy413ImageMigrationTest`, `*GsonCompatReplayTest`,
`LegacyExternalTableTypeReplayTest`, `CatalogMgrTest`,
`ExternalCatalogTest`, `ConstraintPersistTest`, `BDBJEJournalTest`,
`BDBToolTest`, `BDBDebuggerTest`, ...): 103 tests pass; fe-core
checkstyle passes.

Co-authored-by: kid <lucian1412@outlook.com>
2026-10-29 13:04:08 +08:00

178 lines
8.5 KiB
YAML
Raw Permalink Blame History

header:
license:
spdx-id: Apache-2.0
copyright-owner: Apache Software Foundation
paths-ignore:
- "dist"
- "licenses"
- "**/*.md"
- "LICENSE"
- "LICENSE.txt"
- "NOTICE"
- "NOTICE.txt"
- "doap_Doris.rdf"
- ".clang-format"
- ".clang-format-ignore"
- ".clang-tidy"
- ".gitattributes"
- ".gitignore"
- ".gitmodules"
- ".gitleaks.toml"
- ".licenserc.yaml"
- ".rat-excludes"
- ".github/**"
- ".dockerignore"
- "be/src/common/status.cpp"
- "be/src/common/status.h"
- "be/src/env/env.h"
- "be/src/env/env_posix.cpp"
- "**/clucene/**"
- "**/apache-orc/**"
- "**/glibc-compatibility/**"
- "**/gutil/**"
- "**/go.sum"
- "**/test_data/**"
- "**/jmockit/**"
- "**/*.json"
- "**/*.csv"
- "**/*.dat"
- "**/*.svg"
- "**/*.md5"
- "**/*.patch"
- "**/*.log"
- "**/*.sql"
- "**/*.lock"
- "**/*.out"
- "**/*.ldif"
- "**/*.parquet"
- "docs/.markdownlintignore"
- "fe/fe-core/src/test/resources/data/net_snmp_normal"
- "fe/fe-core/src/test/resources/stageUtilTest.txt"
# Recorded method-surface baseline; ConnectorMetadataSurfaceTest reads every
# non-blank line as a method signature, so a header would break the test.
- "fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt"
# Recorded plugin API surface baselines, one per plugin family. Same reason as
# above: each *PluginSurfaceTest.readBaseline() adds every non-blank line to the
# expected signature set, so a header would be read back as phantom signatures.
- "fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt"
- "fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt"
- "fe/fe-authentication/fe-authentication-spi/src/test/resources/authentication-plugin-surface.txt"
- "fe/fe-authorization/fe-authorization-spi/src/test/resources/authorization-plugin-surface.txt"
- "fe/fe-core/src/test/resources/lineage-plugin-surface.txt"
- "fe/be-java-extensions/jni-spi/src/test/resources/jni-plugin-surface.txt"
# Golden access-control decision matrix. AccessControlBehaviorBaselineTest regenerates
# the whole file and compares it to this one line for line, so a header would read back
# as extra rows; it prints the regenerated path on mismatch instead.
- "fe/fe-core/src/test/resources/access-control-behavior-baseline.txt"
# Hive-generated CSV oracle. The FE test compares every line with regenerated
# data, and both BE readers consume every line as a record; a header would
# become unexpected test data. Exclude only this fixture, not all TSV files.
- "fe/fe-connector/fe-connector-hive/src/test/resources/hive-csv-oracle.tsv"
# Golden protocol traffic. MysqlPacketGoldenTest and FlightResultGoldenTest regenerate
# each file in full and compare it to this one byte for byte, so a header would read back
# as unexpected leading content; both tests print the regeneration command on mismatch.
- "fe/fe-core/src/test/resources/protocol-golden/mysql-packets.txt"
- "fe/fe-core/src/test/resources/protocol-golden/flight-results.txt"
# Connector plugin settings templates. build.sh seeds each connector's live
# <name>.conf from its template verbatim (cp -n), so the template's content IS
# the file an administrator edits in plugins/connector/<dir>/. Matched by name
# rather than by a **/*.conf.template glob, so a new one is a deliberate entry
# here rather than something a wildcard silently absorbs.
- "fe/fe-connector/fe-connector-adbc/src/main/resources/adbc.conf.template"
- "fe/fe-connector/fe-connector-hive/src/main/resources/hms.conf.template"
- "fe/fe-connector/fe-connector-iceberg/src/main/resources/iceberg.conf.template"
- "fe/fe-connector/fe-connector-jdbc/src/main/resources/jdbc.conf.template"
- "fe/fe-connector/fe-connector-paimon/src/main/resources/paimon.conf.template"
- "fe/fe-connector/fe-connector-trino/src/main/resources/trino-connector.conf.template"
# Golden 4.1.3 upgrade fixtures, emitted by Gen413Fixtures running real 4.1.3
# bytecode (see the sibling PROVENANCE.txt) and stamped "do not edit by hand".
# These are the generator's only non-binary outputs -- its .bin files are skipped
# as binary and its .json files by the **/*.json rule above -- so listing them
# here covers everything it writes. Deliberately NOT ignoring the directory:
# upgrade/413/generator/Gen413Fixtures.java is real source and must stay checked.
- "fe/fe-core/src/test/resources/upgrade/413/PROVENANCE.txt"
- "fe/fe-core/src/test/resources/upgrade/413/editlog.index"
- "fe/fe-core/src/test/resources/upgrade/413/labels.db.txt"
- "fe/fe-core/src/test/resources/upgrade/413/labels.ds.txt"
- "fe/fe-core/src/test/resources/upgrade/413/labels.tbl.txt"
# Same for the golden 4.1.5 (Lance catalog) fixtures emitted by Gen415Fixtures;
# upgrade/415/generator/Gen415Fixtures.java stays checked.
- "fe/fe-core/src/test/resources/upgrade/415/PROVENANCE.txt"
- "fe/fe-core/src/test/resources/upgrade/415/editlog.index"
- "fe/fe-core/src/test/resources/upgrade/415/labels.db.txt"
- "fe/fe-core/src/test/resources/upgrade/415/labels.ds.txt"
- "fe/fe-core/src/test/resources/upgrade/415/labels.tbl.txt"
- "fe/fe-core/src/main/java/software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.java"
- "fe/fe-core/src/main/antlr4/org/apache/doris/nereids/JavaLexer.g4"
- "fe/fe-core/src/main/antlr4/org/apache/doris/nereids/JavaParser.g4"
- "fe/.idea/vcs.xml"
- "be/dict/ik/*"
- "be/dict/pinyin/*"
- "be/dict/icu/**"
- "be/src/common/signal_handler.h"
- "be/src/olap/lru_cache.cpp"
- "be/src/olap/lru_cache.h"
- "be/src/olap/skiplist.h"
- "be/src/runtime/string_search.hpp"
- "be/src/util/coding.cpp"
- "be/src/util/coding.h"
- "be/src/util/condition_variable.cpp"
- "be/src/util/condition_variable.h"
- "be/src/util/jsonb_document.h"
- "be/src/util/jsonb_error.h"
- "be/src/util/jsonb_parser.h"
- "be/src/util/jsonb_parser_simd.h"
- "be/src/util/jsonb_stream.h"
- "be/src/util/jsonb_updater.h"
- "be/src/util/jsonb_utils.h"
- "be/src/util/jsonb_writer.h"
- "be/src/util/murmur_hash3.cpp"
- "be/src/util/murmur_hash3.h"
- "be/src/util/mustache/mustache.cc"
- "be/src/util/mustache/mustache.h"
- "be/src/util/sse2neo.h"
- "be/src/util/sse2neon.h"
- "be/src/storage/index/inverted/analysis/analyzer/kuromoji/dict/darts.h"
- "be/src/pch/*"
- "be/test/data"
- "be/test/expected_result"
- "build-support/run_clang_format.py"
- "regression-test/data"
- "regression-test/suites/query_p0/sql_functions/conditional_functions/data.txt"
- "docs/.vuepress/public/css/animate.min.css"
- "docs/.vuepress/public/js/jquery.min.js"
- "docs/.vuepress/public/js/wow.min.js"
- "docs/package-lock.json"
- "regression-test/script/README"
- "regression-test/suites/load_p0/stream_load/data"
- "docker/thirdparties/docker-compose/hive/scripts/**"
- "docker/thirdparties/docker-compose/kerberos/paimon_data/**"
- "docker/thirdparties/docker-compose/kerberos/sql/**"
- "docker/thirdparties/docker-compose/iceberg/spark-defaults.conf.tpl"
# Generated Paimon schema and snapshot metadata. These files are copied verbatim into
# MinIO by the Iceberg third-party initializer and cannot carry source comments.
- "docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/paimon/**/schema/**"
- "docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/paimon/**/snapshot/**"
- "docker/thirdparties/docker-compose/postgresql/certs/**"
- "docker/thirdparties/docker-compose/mysql/certs/**"
- "conf/mysql_ssl_default_certificate/*"
- "conf/mysql_ssl_default_certificate/client_certificate/ca.pem"
- "conf/mysql_ssl_default_certificate/client_certificate/client-cert.pem"
- "conf/mysql_ssl_default_certificate/client_certificate/client-key.pem"
- "regression-test/ssl_default_certificate/*"
- "regression-test/pipeline/**"
- "extension/beats/go.mod"
- "extension/beats/go.sum"
- "extension/dbt-doris/dev-requirements.txt"
- "pytest/hdfs"
- "pytest/qe"
- "pytest/sys/data"
- "pytest/deploy/*.conf"
- "tools/jeprof"
- "tools/FlameGraph/*"
- "thirdparty/LICENSE.txt"
- "fs_brokers/cdc_client/src/main/java/io/debezium/**"
- "fs_brokers/cdc_client/src/main/java/com/github/shyiko/**"
comment: on-failure
Reference in new issue View Git Blame Copy Permalink