mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
[fix](fe) Load FE images that carry Lance catalogs written by 4.1.4+ (#69575)
### What problem does this PR solve? Issue Number: None Related PR: #65730, #67235, #68781 Problem Summary: **In short.** This PR makes master read the Lance catalogs that branch-4.1 persists (#65730, released in 4.1.4): an FE image written by 4.1.4 or later with a Lance catalog in it now loads. Before this PR the image failed to parse, so a cluster with a Lance catalog could not be upgraded from any released 4.1.4 / 4.1.4.1 / 4.1.5 build (or from 4.2) to a master-based build: the FE did not start. The Lance catalog's GSON labels are mapped to the plugin-driven classes, as the Iceberg, Paimon and LakeSoul labels already are, and the catalog stays registered, unusable until a Lance connector plugin is installed. **Background.** branch-4.1 added a Lance catalog as fe-core classes in #65730 (released in 4.1.4). Like every external catalog, it is persisted in the image's `datasource` module as GSON JSON whose `clazz` field names the class, `LanceExternalCatalog`, with `logType` `LANCE`. master has no Lance classes: external catalogs are served by connector plugins, and the labels of the catalogs that used to be fe-core classes are redirected to `PluginDrivenExternalCatalog` / `PluginDrivenExternalDatabase` / `PluginDrivenMvccExternalTable` by `registerCompatibleSubtype` in `GsonUtils`. A Lance connector plugin does not exist on master yet; it will be ported later. branch-4.1 also had, for a while, a durable Lance index job framework (#67235) that wrote edit-log op 500 (`OP_LANCE_INDEX_JOB_UPSERT`) and an image module named `lanceIndexJobManager`. #68781 reverted it on branch-4.1 and branch-4.2 before any release: among the release tags only 4.1.5-rc01 has the writer, while 4.1.5 and 4.2.0-rc01 keep just a tombstone that reads and discards both. master never had the framework, and this PR does not make it read that metadata either. The upgrade fixtures on master (`upgrade/413`) were generated from 4.1.3, before Lance existed, so no test caught this. **The problem, and what it cost.** Compared on master against fixtures emitted by real 4.1.5 bytecode: | Metadata from a 4.1.4+ FE | master before this PR | after this PR | |---|---|---| | image whose `datasource` module has a Lance catalog (every 4.1.4+ cluster with one) | image fails to parse: `cannot deserialize interface org.apache.doris.datasource.CatalogIf subtype named LanceExternalCatalog; did you forget to register a subtype?`; the FE cannot start | loads; the catalog is a `PluginDrivenExternalCatalog` of type `lance` with all its properties, comment and auto-analyze policies; using it fails with `No ConnectorProvider found for plugin-driven catalog: <name>, type: lance. Ensure the connector plugin is installed.` | | `OP_CREATE_CATALOG` of a Lance catalog | replays as a degraded catalog (the entry carries only properties) | unchanged | The only labels in the GSON registries of 4.1.4.1, 4.1.5 and 4.2.0-rc01 that master does not know are `LanceExternalCatalog`, `LanceExternalDatabase` and `LanceExternalTable`. **How this PR fixes it.** - `GsonUtils`: `LanceExternalCatalog` → `PluginDrivenExternalCatalog`, `LanceExternalDatabase` → `PluginDrivenExternalDatabase`, `LanceExternalTable` → `PluginDrivenMvccExternalTable` (`LanceExternalTable` implemented `MvccTable`). 4.1.x never persists an external database or table, so only the catalog label can actually appear; the other two complete the registry, as for the other migrated connectors. - `InitCatalogLog.Type` keeps `LANCE`, so the `logType` 4.1.4+ wrote still deserializes to itself. Nothing depends on it: a Lance catalog always persists its `type` property (no resource type creates one), and `PluginDrivenExternalCatalog#gsonPostProcess` rewrites the logType to `PLUGIN`. - `BdbjeJournalDataProcNode`, which renders `SHOW PROC '/bdbje/<db>/<journalId>'` in BDBJE debug mode, prints a record read without a payload as `null`, as `BDBTool` already does, instead of failing with a `NullPointerException`. This is an existing master issue that came up in review, not a Lance one: the readers of the retired operation types already on master (the PL/SQL ones of #58700 and the external-catalog ones) keep no payload. Until a Lance connector plugin is installed, an upgraded cluster lists its Lance catalogs and fails only when one is used; installing the plugin makes them work without any metadata change, since the catalog properties are handed to the connector unchanged. **Results.** - New golden fixtures `fe/fe-core/src/test/resources/upgrade/415/`, emitted by 4.1.5-release bytecode like `upgrade/413` (generator and `PROVENANCE.txt` included, byte-deterministic): the `datasource` module with three Lance catalogs (filesystem on S3, filesystem on `file://`, REST with bearer auth), their `OP_CREATE_CATALOG` entries, and 4.1.5's label registries. - `Legacy415LanceUpgradeTest` (7 tests): the image loads and every Lance catalog becomes a `PluginDrivenExternalCatalog` of type `lance` with the `PLUGIN` logType; every persisted property, the comment, `lastUpdateTime` and the auto-analyze policy survive; the persisted logType names still deserialize; without a plugin the catalog fails only when used, with the message above; a provider claiming `lance` receives exactly the persisted properties, from the image and from the journal; the journal entries replay with and without a plugin. - `LanceGsonCompatReplayTest` (4 tests): the three labels replay as the expected classes (the table as the MVCC variant), and they are the only labels 4.1.5 registered beyond 4.1.3. - Each of these changes, undone on its own, turns tests red: dropping the catalog label (5 tests, with the parse error above), the database label, mapping the table label to the non-MVCC class, or removing `LANCE`. - `BdbjeJournalDataProcNodeTest` (2 tests): a retired PL/SQL record, in its written form, shows its journal ID, operation name, size and `null` data in the debug view; a live `OP_SAVE_NEXTID` record still shows its payload. With `getData().toString()` back, the first test fails with a `NullPointerException`. - The three classes above with 13 related upgrade, GSON and journal suites (`Legacy413ImageMigrationTest`, `*GsonCompatReplayTest`, `LegacyExternalTableTypeReplayTest`, `CatalogMgrTest`, `ExternalCatalogTest`, `ConstraintPersistTest`, `BDBJEJournalTest`, `BDBToolTest`, `BDBDebuggerTest`, ...): 103 tests pass; fe-core checkstyle passes. Co-authored-by: kid <lucian1412@outlook.com>
This commit is contained in:
Mingyu Chen (Rayner)andkid authored and 
GitHub committed
1 parent
2760a7663b
commit
d420f9c41b
20 files changed
+984
-16
No files matched your search
@@ -96,6 +96,13 @@ header:
|
||||
- "fe/fe-core/src/test/resources/upgrade/413/labels.db.txt"
|
||||
- "fe/fe-core/src/test/resources/upgrade/413/labels.ds.txt"
|
||||
- "fe/fe-core/src/test/resources/upgrade/413/labels.tbl.txt"
|
||||
# Same for the golden 4.1.5 (Lance catalog) fixtures emitted by Gen415Fixtures;
|
||||
# upgrade/415/generator/Gen415Fixtures.java stays checked.
|
||||
- "fe/fe-core/src/test/resources/upgrade/415/PROVENANCE.txt"
|
||||
- "fe/fe-core/src/test/resources/upgrade/415/editlog.index"
|
||||
- "fe/fe-core/src/test/resources/upgrade/415/labels.db.txt"
|
||||
- "fe/fe-core/src/test/resources/upgrade/415/labels.ds.txt"
|
||||
- "fe/fe-core/src/test/resources/upgrade/415/labels.tbl.txt"
|
||||
- "fe/fe-core/src/main/java/software/amazon/awssdk/core/client/builder/SdkDefaultClientBuilder.java"
|
||||
- "fe/fe-core/src/main/antlr4/org/apache/doris/nereids/JavaLexer.g4"
|
||||
- "fe/fe-core/src/main/antlr4/org/apache/doris/nereids/JavaParser.g4"
|
||||
|
||||
@@ -52,7 +52,9 @@ public class BdbjeJournalDataProcNode implements ProcNodeInterface {
|
||||
BDBDebugger.JournalEntityWrapper entity = env.getJournalEntity(dbName, journalId);
|
||||
|
||||
short opCode = entity.entity == null ? -1 : entity.entity.getOpCode();
|
||||
String data = entity.entity == null ? entity.errMsg : entity.entity.getData().toString();
|
||||
// The readers of retired operation types (e.g. the PL/SQL ones) consume the payload without keeping it;
|
||||
// print such a record the way BDBTool does.
|
||||
String data = entity.entity == null ? entity.errMsg : String.valueOf(entity.entity.getData());
|
||||
result.addRow(Lists.newArrayList(entity.journalId.toString(),
|
||||
OperationType.getOpName(opCode), entity.size.toString(), data));
|
||||
|
||||
|
||||
@@ -42,6 +42,10 @@ public class InitCatalogLog implements Writable {
|
||||
*
|
||||
* <p>HUDI is absent because it was never produced: hudi tables have always lived in an hms catalog and
|
||||
* no {@code HudiExternalCatalog} ever existed, so no image can carry it.</p>
|
||||
*
|
||||
* <p>LANCE is written by the 4.1.4+ FEs that shipped a Lance catalog. A Lance catalog always persists its
|
||||
* {@code type} property (no resource type creates one), so nothing is backfilled from it; it stays so
|
||||
* that every name a released image carries still deserializes to itself.</p>
|
||||
*/
|
||||
public enum Type {
|
||||
HMS,
|
||||
@@ -54,6 +58,7 @@ public class InitCatalogLog implements Writable {
|
||||
TEST,
|
||||
TRINO_CONNECTOR,
|
||||
REMOTE_DORIS,
|
||||
LANCE,
|
||||
PLUGIN,
|
||||
UNKNOWN;
|
||||
}
|
||||
|
||||
@@ -413,7 +413,12 @@ public class GsonUtils {
|
||||
// was never migrated to a connector, so the remapped catalog has no backing connector and
|
||||
// errors on access (it must be dropped); this only keeps old images/edit-logs loadable.
|
||||
.registerCompatibleSubtype(
|
||||
PluginDrivenExternalCatalog.class, "LakeSoulExternalCatalog");
|
||||
PluginDrivenExternalCatalog.class, "LakeSoulExternalCatalog")
|
||||
// Migrate Lance catalogs written by 4.1.4+ (4.1.x / 4.2) to PluginDriven on deserialization.
|
||||
// Until a lance connector plugin is installed the catalog has no backing connector and errors
|
||||
// on access; without this label the whole image fails to parse and the FE cannot start.
|
||||
.registerCompatibleSubtype(
|
||||
PluginDrivenExternalCatalog.class, "LanceExternalCatalog");
|
||||
if (Config.isNotCloudMode()) {
|
||||
dsTypeAdapterFactory
|
||||
.registerSubtype(InternalCatalog.class, InternalCatalog.class.getSimpleName());
|
||||
@@ -472,7 +477,10 @@ public class GsonUtils {
|
||||
PluginDrivenExternalDatabase.class, "HMSExternalDatabase")
|
||||
// Migrate old LakeSoul databases to PluginDriven on deserialization (deprecated, no connector)
|
||||
.registerCompatibleSubtype(
|
||||
PluginDrivenExternalDatabase.class, "LakeSoulExternalDatabase");
|
||||
PluginDrivenExternalDatabase.class, "LakeSoulExternalDatabase")
|
||||
// Migrate Lance databases (4.1.4+) to PluginDriven on deserialization
|
||||
.registerCompatibleSubtype(
|
||||
PluginDrivenExternalDatabase.class, "LanceExternalDatabase");
|
||||
|
||||
private static RuntimeTypeAdapterFactory<TableIf> tblTypeAdapterFactory = RuntimeTypeAdapterFactory.of(
|
||||
TableIf.class, "clazz").registerSubtype(ExternalTable.class, ExternalTable.class.getSimpleName())
|
||||
@@ -508,6 +516,10 @@ public class GsonUtils {
|
||||
// PluginDrivenMvccExternalTable — matching what buildTableInternal rebuilds it as on replay.
|
||||
.registerCompatibleSubtype(
|
||||
PluginDrivenMvccExternalTable.class, "HMSExternalTable")
|
||||
// Lance tables (4.1.4+) migrate to the MVCC variant: LanceExternalTable implemented MvccTable
|
||||
// (dataset versions back snapshots/time-travel)
|
||||
.registerCompatibleSubtype(
|
||||
PluginDrivenMvccExternalTable.class, "LanceExternalTable")
|
||||
.registerSubtype(BrokerTable.class, BrokerTable.class.getSimpleName())
|
||||
.registerSubtype(EsTable.class, EsTable.class.getSimpleName())
|
||||
.registerSubtype(FunctionGenTable.class, FunctionGenTable.class.getSimpleName())
|
||||
|
||||
+121
@@ -0,0 +1,121 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
package org.apache.doris.common.proc;
|
||||
|
||||
import org.apache.doris.common.AnalysisException;
|
||||
import org.apache.doris.common.Config;
|
||||
import org.apache.doris.common.io.Text;
|
||||
import org.apache.doris.common.jmockit.Deencapsulation;
|
||||
import org.apache.doris.journal.JournalEntity;
|
||||
import org.apache.doris.journal.bdbje.BDBDebugger;
|
||||
import org.apache.doris.persist.OperationType;
|
||||
|
||||
import com.google.common.collect.Lists;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.Assertions;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockito.Mockito;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.DataInputStream;
|
||||
import java.io.DataOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* SHOW PROC '/bdbje/dbName/journalId', the BDBJE debug-mode view of a single journal record.
|
||||
*/
|
||||
public class BdbjeJournalDataProcNodeTest {
|
||||
private static final String DB_NAME = "1";
|
||||
|
||||
private boolean debugMode;
|
||||
private BDBDebugger.BDBDebugEnv debugEnv;
|
||||
|
||||
@BeforeEach
|
||||
public void setUp() {
|
||||
debugMode = Config.enable_bdbje_debug_mode;
|
||||
debugEnv = BDBDebugger.get().getEnv();
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
public void tearDown() {
|
||||
Config.enable_bdbje_debug_mode = debugMode;
|
||||
Deencapsulation.setField(BDBDebugger.get(), "debugEnv", debugEnv);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void recordOfARetiredOperationTypeIsPrintable() throws Exception {
|
||||
// The readers of retired operation types consume the payload without keeping it, e.g. the PL/SQL ones
|
||||
// that #58700 retired, whose records are Text-framed JSON.
|
||||
BDBDebugger.JournalEntityWrapper plsqlPackage = record(8L, OperationType.OP_DROP_PLSQL_PACKAGE,
|
||||
"{\"name\":\"pkg1\",\"catalogId\":0,\"dbName\":\"db1\"}");
|
||||
Assertions.assertNull(plsqlPackage.entity.getData());
|
||||
installDebugEnv(plsqlPackage);
|
||||
|
||||
// MUTATION: with getData().toString() the row fails with a NullPointerException.
|
||||
Assertions.assertEquals(Lists.newArrayList("8", "OP_DROP_PLSQL_PACKAGE",
|
||||
plsqlPackage.size.toString(), "null"), fetchRow(8L));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void liveRecordStillShowsItsPayload() throws Exception {
|
||||
BDBDebugger.JournalEntityWrapper nextId = record(9L, OperationType.OP_SAVE_NEXTID, "12345");
|
||||
installDebugEnv(nextId);
|
||||
|
||||
Assertions.assertEquals(Lists.newArrayList("9", "OP_SAVE_NEXTID", nextId.size.toString(), "12345"),
|
||||
fetchRow(9L));
|
||||
}
|
||||
|
||||
/** A journal record as BDBDebugEnv#getJournalEntity reads it back from BDBJE. */
|
||||
private static BDBDebugger.JournalEntityWrapper record(long journalId, short opCode, String payload)
|
||||
throws IOException {
|
||||
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
|
||||
try (DataOutputStream out = new DataOutputStream(bytes)) {
|
||||
out.writeShort(opCode);
|
||||
// The int-length framing that both Text#write and Text.writeString produce.
|
||||
Text.writeString(out, payload);
|
||||
}
|
||||
BDBDebugger.JournalEntityWrapper wrapper = new BDBDebugger.JournalEntityWrapper(journalId);
|
||||
wrapper.size = bytes.size();
|
||||
JournalEntity entity = new JournalEntity();
|
||||
try (DataInputStream in = new DataInputStream(new ByteArrayInputStream(bytes.toByteArray()))) {
|
||||
entity.readFields(in);
|
||||
Assertions.assertEquals(-1, in.read(), "the record must be consumed exactly");
|
||||
}
|
||||
wrapper.entity = entity;
|
||||
return wrapper;
|
||||
}
|
||||
|
||||
private static void installDebugEnv(BDBDebugger.JournalEntityWrapper... records) {
|
||||
BDBDebugger.BDBDebugEnv env = Mockito.mock(BDBDebugger.BDBDebugEnv.class);
|
||||
for (BDBDebugger.JournalEntityWrapper record : records) {
|
||||
Mockito.when(env.getJournalEntity(DB_NAME, record.journalId)).thenReturn(record);
|
||||
}
|
||||
Deencapsulation.setField(BDBDebugger.get(), "debugEnv", env);
|
||||
Config.enable_bdbje_debug_mode = true;
|
||||
}
|
||||
|
||||
private static List<String> fetchRow(long journalId) throws AnalysisException {
|
||||
ProcResult result = new BdbjeJournalDataProcNode(DB_NAME, journalId).fetchResult();
|
||||
Assertions.assertEquals(BdbjeJournalDataProcNode.TITLE_NAMES, result.getColumnNames());
|
||||
Assertions.assertEquals(1, result.getRows().size());
|
||||
return result.getRows().get(0);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
package org.apache.doris.datasource;
|
||||
|
||||
import org.apache.doris.catalog.DatabaseIf;
|
||||
import org.apache.doris.catalog.TableIf;
|
||||
import org.apache.doris.datasource.mvcc.PluginDrivenMvccExternalTable;
|
||||
import org.apache.doris.datasource.plugin.PluginDrivenExternalCatalog;
|
||||
import org.apache.doris.datasource.plugin.PluginDrivenExternalDatabase;
|
||||
import org.apache.doris.persist.gson.GsonUtils;
|
||||
|
||||
import com.google.common.collect.ImmutableSet;
|
||||
import com.google.common.collect.Maps;
|
||||
import com.google.common.collect.Sets;
|
||||
import org.junit.jupiter.api.Assertions;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
/**
|
||||
* Guards loading the Lance catalogs/databases/tables that 4.1.4+ FEs persisted (4.1.x and 4.2 shipped Lance as
|
||||
* fe-core classes; this branch has no such classes). Their GSON "clazz" labels must be redirected by the
|
||||
* {@code registerCompatibleSubtype} mappings in {@link GsonUtils} to the generic PluginDriven classes —
|
||||
* otherwise an image from a cluster with a Lance catalog fails with a {@code JsonParseException} (label not
|
||||
* registered) and the FE cannot start.
|
||||
*
|
||||
* <p>The table label lands on the MVCC variant: {@code LanceExternalTable} implemented {@code MvccTable}
|
||||
* (dataset versions back snapshots and time travel), so replaying it as the base
|
||||
* {@code PluginDrivenExternalTable} would silently downgrade it.</p>
|
||||
*
|
||||
* <p>Each case round-trips a valid PluginDriven object through GSON and rewrites only the "clazz" discriminator
|
||||
* to the legacy label, like {@link PaimonGsonCompatReplayTest}. The real 4.1.5 bytes of a Lance catalog are
|
||||
* covered by {@link Legacy415LanceUpgradeTest}; a database or table is never persisted by 4.1.5.</p>
|
||||
*/
|
||||
public class LanceGsonCompatReplayTest {
|
||||
|
||||
private static String swapClazz(String json, String currentTag, String legacyTag) {
|
||||
String needle = "\"clazz\":\"" + currentTag + "\"";
|
||||
// Sanity: the polymorphic serialization must emit the discriminator we are about to rewrite.
|
||||
Assertions.assertTrue(json.contains(needle),
|
||||
"expected discriminator " + needle + " in serialized json: " + json);
|
||||
return json.replace(needle, "\"clazz\":\"" + legacyTag + "\"");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testLegacyLanceCatalogTagReplaysAsPluginDriven() {
|
||||
Map<String, String> props = Maps.newHashMap();
|
||||
props.put("type", "lance");
|
||||
PluginDrivenExternalCatalog catalog =
|
||||
new PluginDrivenExternalCatalog(1L, "lance_ctl", "", props, "c", null);
|
||||
String baseJson = GsonUtils.GSON.toJson(catalog, CatalogIf.class);
|
||||
|
||||
String json = swapClazz(baseJson, "PluginDrivenExternalCatalog", "LanceExternalCatalog");
|
||||
// MUTATION: removing the registerCompatibleSubtype throws "cannot deserialize ... subtype named
|
||||
// LanceExternalCatalog" here; a wrong target class fails the assertSame.
|
||||
CatalogIf<?> restored = GsonUtils.GSON.fromJson(json, CatalogIf.class);
|
||||
Assertions.assertSame(PluginDrivenExternalCatalog.class, restored.getClass(),
|
||||
"legacy 'LanceExternalCatalog' tag must replay as PluginDrivenExternalCatalog");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testLegacyLanceDatabaseTagReplaysAsPluginDriven() {
|
||||
PluginDrivenExternalDatabase db = new PluginDrivenExternalDatabase();
|
||||
db.id = 2L;
|
||||
db.name = "lance_db";
|
||||
String baseJson = GsonUtils.GSON.toJson(db, DatabaseIf.class);
|
||||
|
||||
String json = swapClazz(baseJson, "PluginDrivenExternalDatabase", "LanceExternalDatabase");
|
||||
// MUTATION: dropping the db registerCompatibleSubtype makes this throw on deserialize.
|
||||
DatabaseIf<?> restored = GsonUtils.GSON.fromJson(json, DatabaseIf.class);
|
||||
Assertions.assertSame(PluginDrivenExternalDatabase.class, restored.getClass(),
|
||||
"legacy 'LanceExternalDatabase' tag must replay as PluginDrivenExternalDatabase");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testLegacyLanceTableTagReplaysAsMvccPluginDriven() {
|
||||
PluginDrivenMvccExternalTable table = new PluginDrivenMvccExternalTable();
|
||||
table.id = 3L;
|
||||
table.name = "lance_tbl";
|
||||
table.dbName = "lance_db";
|
||||
String baseJson = GsonUtils.GSON.toJson(table, TableIf.class);
|
||||
|
||||
String json = swapClazz(baseJson, "PluginDrivenMvccExternalTable", "LanceExternalTable");
|
||||
TableIf restored = GsonUtils.GSON.fromJson(json, TableIf.class);
|
||||
// assertSame, not instanceof: a mapping to the base PluginDrivenExternalTable would also pass instanceof.
|
||||
Assertions.assertSame(PluginDrivenMvccExternalTable.class, restored.getClass(),
|
||||
"legacy 'LanceExternalTable' tag must replay as PluginDrivenMvccExternalTable,"
|
||||
+ " not the base PluginDrivenExternalTable");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void lanceLabelsAreTheOnlyOnesAReleaseAddedAfter413() throws Exception {
|
||||
// The three cases above are complete only if Lance is all that 4.1.5's registries added on top of
|
||||
// 4.1.3's (which Legacy413LabelRegistryTest covers label by label). Both inventories are dumped
|
||||
// reflectively by the fixture generators, so a label added in a later release cannot hide here.
|
||||
Assertions.assertEquals(ImmutableSet.of("LanceExternalCatalog"), addedSince413("ds"));
|
||||
Assertions.assertEquals(ImmutableSet.of("LanceExternalDatabase"), addedSince413("db"));
|
||||
Assertions.assertEquals(ImmutableSet.of("LanceExternalTable"), addedSince413("tbl"));
|
||||
}
|
||||
|
||||
private static Set<String> addedSince413(String which) throws Exception {
|
||||
return Sets.difference(Legacy415Fixtures.legacyLabels(which).keySet(),
|
||||
Legacy413Fixtures.legacyLabels(which).keySet());
|
||||
}
|
||||
}
|
||||
@@ -34,6 +34,9 @@ import java.util.Map;
|
||||
* <p>Those files were produced by real 4.1.3 bytecode, not written by hand; see the PROVENANCE.txt
|
||||
* next to them for how to regenerate and for the five wire-format details that make hand-authored
|
||||
* fixtures wrong.
|
||||
*
|
||||
* <p>The package-private overloads taking a resource root are shared with the fixture families of later
|
||||
* releases that persisted new labels ({@link Legacy415Fixtures}), which are generated the same way.
|
||||
*/
|
||||
public final class Legacy413Fixtures {
|
||||
|
||||
@@ -56,7 +59,11 @@ public final class Legacy413Fixtures {
|
||||
}
|
||||
|
||||
private static CatalogMgr loadCatalogMgr(String fileName) throws IOException {
|
||||
try (InputStream in = open(fileName);
|
||||
return loadCatalogMgr(ROOT, fileName);
|
||||
}
|
||||
|
||||
static CatalogMgr loadCatalogMgr(String root, String fileName) throws IOException {
|
||||
try (InputStream in = open(root, fileName);
|
||||
DataInputStream dis = new DataInputStream(in)) {
|
||||
return CatalogMgr.read(dis);
|
||||
}
|
||||
@@ -64,32 +71,34 @@ public final class Legacy413Fixtures {
|
||||
|
||||
/** Raw bytes of one edit-log entry, in JournalEntity wire form (opCode short + Text-framed JSON). */
|
||||
public static byte[] journalEntry(String fileName) throws IOException {
|
||||
try (InputStream in = open("editlog/" + fileName)) {
|
||||
return readAll(in);
|
||||
}
|
||||
return resource(ROOT, "editlog/" + fileName);
|
||||
}
|
||||
|
||||
/** Names of every edit-log fixture with the given op code, e.g. {@code 320}. */
|
||||
public static List<String> journalEntryNames(int opCode) {
|
||||
return journalEntryNames(ROOT, opCode);
|
||||
}
|
||||
|
||||
static List<String> journalEntryNames(String root, int opCode) {
|
||||
List<String> names = new ArrayList<>();
|
||||
for (String name : listEditLogFixtures()) {
|
||||
for (String name : listEditLogFixtures(root)) {
|
||||
if (name.startsWith("op" + opCode + "-")) {
|
||||
names.add(name);
|
||||
}
|
||||
}
|
||||
if (names.isEmpty()) {
|
||||
throw new IllegalStateException("no edit-log fixture for op " + opCode
|
||||
+ "; the resource index at " + ROOT + "editlog.index is stale");
|
||||
+ "; the resource index at " + root + "editlog.index is stale");
|
||||
}
|
||||
return names;
|
||||
}
|
||||
|
||||
private static List<String> listEditLogFixtures() {
|
||||
private static List<String> listEditLogFixtures(String root) {
|
||||
// Resource directories are not enumerable from a jar, and surefire may well run these classes off one.
|
||||
// The generator writes a flat index next to the fixtures precisely so this does not need directory
|
||||
// listing; keeping it in one place means a fixture added without regenerating the index fails loudly
|
||||
// in journalEntryNames() rather than silently reducing coverage.
|
||||
try (InputStream in = open("editlog.index")) {
|
||||
try (InputStream in = open(root, "editlog.index")) {
|
||||
String text = new String(readAll(in), StandardCharsets.UTF_8);
|
||||
List<String> names = new ArrayList<>();
|
||||
for (String line : text.split("\n")) {
|
||||
@@ -111,8 +120,12 @@ public final class Legacy413Fixtures {
|
||||
* @return label to the fully-qualified name of the class 4.1.3 mapped it to, in file order
|
||||
*/
|
||||
public static Map<String, LegacyLabel> legacyLabels(String which) throws IOException {
|
||||
return legacyLabels(ROOT, which);
|
||||
}
|
||||
|
||||
static Map<String, LegacyLabel> legacyLabels(String root, String which) throws IOException {
|
||||
Map<String, LegacyLabel> labels = new LinkedHashMap<>();
|
||||
try (InputStream in = open("labels." + which + ".txt")) {
|
||||
try (InputStream in = open(root, "labels." + which + ".txt")) {
|
||||
String text = new String(readAll(in), StandardCharsets.UTF_8);
|
||||
for (String line : text.split("\n")) {
|
||||
if (line.trim().isEmpty() || line.startsWith("#")) {
|
||||
@@ -148,11 +161,18 @@ public final class Legacy413Fixtures {
|
||||
}
|
||||
}
|
||||
|
||||
private static InputStream open(String relative) throws IOException {
|
||||
InputStream in = Legacy413Fixtures.class.getResourceAsStream(ROOT + relative);
|
||||
/** All bytes of one fixture file under the release directory {@code root}. */
|
||||
static byte[] resource(String root, String relative) throws IOException {
|
||||
try (InputStream in = open(root, relative)) {
|
||||
return readAll(in);
|
||||
}
|
||||
}
|
||||
|
||||
private static InputStream open(String root, String relative) throws IOException {
|
||||
InputStream in = Legacy413Fixtures.class.getResourceAsStream(root + relative);
|
||||
if (in == null) {
|
||||
throw new IOException("missing 4.1.3 fixture: " + ROOT + relative
|
||||
+ " -- see src/test/resources/upgrade/413/PROVENANCE.txt");
|
||||
throw new IOException("missing upgrade fixture: " + root + relative
|
||||
+ " -- see src/test/resources" + root + "PROVENANCE.txt");
|
||||
}
|
||||
return in;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
package org.apache.doris.datasource;
|
||||
|
||||
import org.apache.doris.common.io.Text;
|
||||
import org.apache.doris.datasource.Legacy413Fixtures.LegacyLabel;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.DataInputStream;
|
||||
import java.io.IOException;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* Access to the golden Doris 4.1.5 metadata fixtures under {@code src/test/resources/upgrade/415/}: the Lance
|
||||
* catalogs 4.1.4 introduced, in the bytes a real 4.1.5 FE writes (see the PROVENANCE.txt next to them).
|
||||
*/
|
||||
public final class Legacy415Fixtures {
|
||||
|
||||
/** The lastUpdateTime the generator stamps on every catalog, so the bytes are deterministic. */
|
||||
public static final long FIXED_UPDATE_TIME = 1753000000000L;
|
||||
|
||||
private static final String ROOT = "/upgrade/415/";
|
||||
|
||||
private Legacy415Fixtures() {
|
||||
}
|
||||
|
||||
/** The image's "datasource" module as a 4.1.5 FE would have written it. */
|
||||
public static CatalogMgr loadCatalogMgr() throws IOException {
|
||||
return Legacy413Fixtures.loadCatalogMgr(ROOT, "datasource.module.bin");
|
||||
}
|
||||
|
||||
/** The same module's JSON payload, unframed, for assertions on what is literally on the wire. */
|
||||
public static String catalogMgrJson() throws IOException {
|
||||
byte[] framed = Legacy413Fixtures.resource(ROOT, "datasource.module.bin");
|
||||
try (DataInputStream in = new DataInputStream(new ByteArrayInputStream(framed))) {
|
||||
return Text.readString(in);
|
||||
}
|
||||
}
|
||||
|
||||
/** Raw bytes of one edit-log entry, in JournalEntity wire form (opCode short + Text-framed JSON). */
|
||||
public static byte[] journalEntry(String fileName) throws IOException {
|
||||
return Legacy413Fixtures.resource(ROOT, "editlog/" + fileName);
|
||||
}
|
||||
|
||||
/** Names of every edit-log fixture with the given op code, e.g. {@code 320}. */
|
||||
public static List<String> journalEntryNames(int opCode) {
|
||||
return Legacy413Fixtures.journalEntryNames(ROOT, opCode);
|
||||
}
|
||||
|
||||
/**
|
||||
* One of 4.1.5's own GSON label registries, read out of the generated {@code labels.*.txt}.
|
||||
*
|
||||
* @param which one of {@code ds}, {@code db}, {@code tbl}
|
||||
*/
|
||||
public static Map<String, LegacyLabel> legacyLabels(String which) throws IOException {
|
||||
return Legacy413Fixtures.legacyLabels(ROOT, which);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
package org.apache.doris.datasource;
|
||||
|
||||
import org.apache.doris.common.Pair;
|
||||
import org.apache.doris.common.io.Text;
|
||||
import org.apache.doris.connector.ConnectorFactory;
|
||||
import org.apache.doris.connector.ConnectorPluginManager;
|
||||
import org.apache.doris.datasource.log.CatalogLog;
|
||||
import org.apache.doris.datasource.log.InitCatalogLog;
|
||||
import org.apache.doris.datasource.plugin.PluginDrivenExternalCatalog;
|
||||
import org.apache.doris.datasource.upgrade.RecordingConnectorProvider;
|
||||
import org.apache.doris.persist.OperationType;
|
||||
import org.apache.doris.persist.gson.GsonUtils;
|
||||
|
||||
import com.google.common.base.Throwables;
|
||||
import com.google.common.collect.ImmutableList;
|
||||
import com.google.gson.JsonElement;
|
||||
import com.google.gson.JsonObject;
|
||||
import com.google.gson.JsonParser;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.Assertions;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.DataInputStream;
|
||||
import java.io.IOException;
|
||||
import java.util.HashSet;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.TreeMap;
|
||||
|
||||
/**
|
||||
* Upgrading a 4.1.4+ cluster that has Lance catalogs to this branch.
|
||||
*
|
||||
* <p>4.1.4 shipped Lance as fe-core classes ({@code LanceExternalCatalog} and friends); this branch has no such
|
||||
* classes and, until a Lance connector plugin lands, no provider for the "lance" type either. The fixtures were
|
||||
* emitted by real 4.1.5 bytecode (src/test/resources/upgrade/415/PROVENANCE.txt). What must hold: the image
|
||||
* loads and the FE starts, the catalog keeps everything the user configured, it fails only when used while the
|
||||
* plugin is missing, and a plugin that claims "lance" later is handed exactly the persisted properties.
|
||||
*
|
||||
* <p>This class lives in {@code org.apache.doris.datasource} because {@link ExternalCatalog#logType} and
|
||||
* {@link ExternalCatalog#tableAutoAnalyzePolicy} are protected.
|
||||
*/
|
||||
public class Legacy415LanceUpgradeTest {
|
||||
|
||||
private static final long[] LANCE_IDS = {10101L, 10102L, 10103L};
|
||||
|
||||
@AfterEach
|
||||
public void tearDown() {
|
||||
// The plugin manager is a process-wide static shared with every other test class in this fork.
|
||||
ConnectorFactory.initPluginManager(new ConnectorPluginManager());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void everyLanceCatalogInThe415ImageLoadsAsPluginDriven() throws Exception {
|
||||
// MUTATION: without the LanceExternalCatalog compatible subtype the whole module fails to parse
|
||||
// ("cannot deserialize ... subtype named LanceExternalCatalog"), i.e. the upgraded FE cannot start.
|
||||
CatalogMgr mgr = Legacy415Fixtures.loadCatalogMgr();
|
||||
|
||||
Assertions.assertEquals(4, mgr.getCatalogNum(),
|
||||
"the 4.1.5 fixture holds the internal catalog plus 3 Lance catalogs; a different count means the "
|
||||
+ "fixture was regenerated without updating this test");
|
||||
Assertions.assertSame(InternalCatalog.class, mgr.getCatalog(0L).getClass());
|
||||
for (long id : LANCE_IDS) {
|
||||
ExternalCatalog catalog = (ExternalCatalog) mgr.getCatalog(id);
|
||||
Assertions.assertSame(PluginDrivenExternalCatalog.class, catalog.getClass(),
|
||||
"catalog " + id + " must be served by the connector framework, but loaded as "
|
||||
+ catalog.getClass().getSimpleName());
|
||||
Assertions.assertEquals("lance", catalog.getType());
|
||||
Assertions.assertEquals(InitCatalogLog.Type.PLUGIN, catalog.logType,
|
||||
"a migrated catalog must report the PLUGIN logType, otherwise buildDbForInit takes the "
|
||||
+ "legacy path");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void everyPropertyAndUserVisibleStateSurvivesTheMigration() throws Exception {
|
||||
CatalogMgr mgr = Legacy415Fixtures.loadCatalogMgr();
|
||||
|
||||
for (long id : LANCE_IDS) {
|
||||
ExternalCatalog catalog = (ExternalCatalog) mgr.getCatalog(id);
|
||||
Map<String, String> loaded = catalog.getCatalogProperty().getProperties();
|
||||
persistedProperties(id).forEach((key, value) -> Assertions.assertEquals(value, loaded.get(key),
|
||||
"property '" + key + "' of catalog " + id + " did not survive loading the 4.1.5 image"));
|
||||
}
|
||||
|
||||
ExternalCatalog s3 = (ExternalCatalog) mgr.getCatalog(10101L);
|
||||
Assertions.assertEquals("g1_lance_s3", s3.getName());
|
||||
Assertions.assertEquals("lance on s3", s3.getComment(), "the user's COMMENT must survive the upgrade");
|
||||
// Before anything initialises the catalog: makeSureInitialized() overwrites lastUpdateTime.
|
||||
Assertions.assertEquals(Legacy415Fixtures.FIXED_UPDATE_TIME, s3.getLastUpdateTime());
|
||||
Map<Pair<String, String>, String> policy = s3.tableAutoAnalyzePolicy;
|
||||
Assertions.assertEquals(1, policy.size(), "auto-analyze policies must survive the upgrade");
|
||||
Assertions.assertEquals("enable", policy.get(Pair.of("db1", "tbl1")));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void everyLogTypeNameThe415ImageCarriesStillDeserializesToItself() throws Exception {
|
||||
// Gson reads an enum name it does not know as null instead of failing, so losing LANCE would not break
|
||||
// loading: a Lance catalog always persists its "type" property and nothing is backfilled from the
|
||||
// logType. This pins the rule InitCatalogLog.Type documents -- every name a released image carries
|
||||
// stays -- for the name 4.1.4 added.
|
||||
Set<String> names = new HashSet<>();
|
||||
for (Map.Entry<String, JsonElement> entry : idToCatalogJson().entrySet()) {
|
||||
JsonElement logType = entry.getValue().getAsJsonObject().get("logType");
|
||||
if (logType != null) {
|
||||
names.add(logType.getAsString());
|
||||
}
|
||||
}
|
||||
Assertions.assertEquals(java.util.Collections.singleton("LANCE"), names,
|
||||
"the fixture's premise changed; regenerate the expectations with the fixture");
|
||||
for (String name : names) {
|
||||
InitCatalogLog.Type type = GsonUtils.GSON.fromJson("\"" + name + "\"", InitCatalogLog.Type.class);
|
||||
Assertions.assertNotNull(type, "logType '" + name + "' written by 4.1.5 is no longer a known name");
|
||||
Assertions.assertEquals(name, type.name());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void lanceCatalogWithoutItsPluginFailsOnlyWhenUsed() throws Exception {
|
||||
// The state right after upgrading to a build that ships no Lance connector: the image loaded (above),
|
||||
// and touching the catalog reports what is missing instead of something obscure.
|
||||
ConnectorFactory.initPluginManager(new ConnectorPluginManager());
|
||||
ExternalCatalog catalog = (ExternalCatalog) Legacy415Fixtures.loadCatalogMgr().getCatalog(10102L);
|
||||
|
||||
RuntimeException e = Assertions.assertThrows(RuntimeException.class, catalog::makeSureInitialized);
|
||||
Assertions.assertTrue(Throwables.getStackTraceAsString(e).contains(
|
||||
"No ConnectorProvider found for plugin-driven catalog: g1_lance_local, type: lance"),
|
||||
Throwables.getStackTraceAsString(e));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void lanceConnectorIsHandedExactlyThePersistedProperties() throws Exception {
|
||||
// Once a plugin claims "lance", the migrated catalog must reach it with nothing dropped or renamed --
|
||||
// credentials and namespace options included -- or it loads fine and then cannot connect to anything.
|
||||
RecordingConnectorProvider.Registry registry =
|
||||
RecordingConnectorProvider.installFor(ImmutableList.of("lance"));
|
||||
CatalogMgr mgr = Legacy415Fixtures.loadCatalogMgr();
|
||||
|
||||
for (long id : LANCE_IDS) {
|
||||
ExternalCatalog catalog = (ExternalCatalog) mgr.getCatalog(id);
|
||||
catalog.makeSureInitialized();
|
||||
|
||||
RecordingConnectorProvider lance = registry.get("lance");
|
||||
Assertions.assertEquals(catalog.getName(), lance.calls.get(lance.calls.size() - 1).catalogName);
|
||||
Map<String, String> delivered = lance.lastProperties();
|
||||
persistedProperties(id).forEach((key, value) -> Assertions.assertEquals(value, delivered.get(key),
|
||||
"property '" + key + "' of catalog " + id + " did not reach the connector unchanged"));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void everyCreateLanceCatalogEntryFrom415ReplaysWithoutThePlugin() throws Exception {
|
||||
// OP_CREATE_CATALOG journals only the property map, so this path never depended on the class label; it
|
||||
// is pinned because replay builds the connector synchronously on the FE's startup thread, where
|
||||
// EditLog turns any exception into System.exit(-1). Without the plugin the catalog registers degraded.
|
||||
ConnectorFactory.initPluginManager(new ConnectorPluginManager());
|
||||
|
||||
for (String fixture : Legacy415Fixtures.journalEntryNames(OperationType.OP_CREATE_CATALOG)) {
|
||||
CatalogLog log = readCatalogLog(fixture);
|
||||
CatalogIf<?> catalog = CatalogFactory.createFromLog(log);
|
||||
|
||||
Assertions.assertSame(PluginDrivenExternalCatalog.class, catalog.getClass(),
|
||||
"replay of " + fixture + " produced " + catalog.getClass().getSimpleName());
|
||||
Assertions.assertEquals(log.getCatalogName(), catalog.getName());
|
||||
Assertions.assertEquals("lance", ((ExternalCatalog) catalog).getType());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void everyCreateLanceCatalogEntryFrom415ReplaysThroughALancePlugin() throws Exception {
|
||||
RecordingConnectorProvider.Registry registry =
|
||||
RecordingConnectorProvider.installFor(ImmutableList.of("lance"));
|
||||
|
||||
for (String fixture : Legacy415Fixtures.journalEntryNames(OperationType.OP_CREATE_CATALOG)) {
|
||||
CatalogLog log = readCatalogLog(fixture);
|
||||
CatalogIf<?> catalog = CatalogFactory.createFromLog(log);
|
||||
|
||||
Assertions.assertSame(PluginDrivenExternalCatalog.class, catalog.getClass());
|
||||
RecordingConnectorProvider lance = registry.get("lance");
|
||||
Assertions.assertEquals(log.getCatalogName(), lance.calls.get(lance.calls.size() - 1).catalogName,
|
||||
"the provider claiming 'lance' must be asked to build the connector for " + fixture);
|
||||
Map<String, String> delivered = lance.lastProperties();
|
||||
log.getProps().forEach((key, value) -> Assertions.assertEquals(value, delivered.get(key),
|
||||
"property '" + key + "' did not survive the journey from the 4.1.5 journal to the connector"));
|
||||
}
|
||||
}
|
||||
|
||||
/** The property map catalog {@code id} has on the wire, read straight out of the 4.1.5 module's JSON. */
|
||||
private static Map<String, String> persistedProperties(long id) throws IOException {
|
||||
JsonObject properties = idToCatalogJson().getAsJsonObject(String.valueOf(id))
|
||||
.getAsJsonObject("catalogProperty").getAsJsonObject("properties");
|
||||
Map<String, String> result = new TreeMap<>();
|
||||
properties.entrySet().forEach(e -> result.put(e.getKey(), e.getValue().getAsString()));
|
||||
Assertions.assertTrue(result.size() > 2, "catalog " + id + " has an implausibly small property map");
|
||||
return result;
|
||||
}
|
||||
|
||||
private static JsonObject idToCatalogJson() throws IOException {
|
||||
return JsonParser.parseString(Legacy415Fixtures.catalogMgrJson()).getAsJsonObject()
|
||||
.getAsJsonObject("idToCatalog");
|
||||
}
|
||||
|
||||
private static CatalogLog readCatalogLog(String fixture) throws IOException {
|
||||
byte[] bytes = Legacy415Fixtures.journalEntry(fixture);
|
||||
try (DataInputStream in = new DataInputStream(new ByteArrayInputStream(bytes))) {
|
||||
in.readShort(); // op code; the payload class is not on the wire
|
||||
return GsonUtils.GSON.fromJson(Text.readString(in), CatalogLog.class);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
Golden FE metadata fixtures for upgrading a 4.1.4+ cluster that has Lance catalogs.
|
||||
|
||||
WHAT THIS IS
|
||||
============
|
||||
Every .bin / .json / .txt / .index file in this directory except this one was produced by REAL Apache Doris
|
||||
4.1.5 bytecode, the same way the 4.1.3 fixtures in ../413 were (read ../413/PROVENANCE.txt for why
|
||||
hand-written fixtures are not good enough). 4.1.4 added the Lance catalog; 4.1.4, 4.1.4.1, 4.1.5 and
|
||||
4.2.0-rc01 register the same GSON labels, and the only ones 4.1.3 did not have are LanceExternalCatalog,
|
||||
LanceExternalDatabase and LanceExternalTable. 4.1.5 is the newest release at the time of writing.
|
||||
|
||||
PROVENANCE
|
||||
==========
|
||||
source tag : 4.1.5-release (annotated tag; commit 917b20be83878e98f94e3ebf7efccc80346e3e50)
|
||||
gson : 2.10.1 (identical on 4.1.5 and on the branch under test)
|
||||
generator : generator/Gen415Fixtures.java (committed here verbatim)
|
||||
generated on : 2026-10-24
|
||||
sha256 datasource.module.bin : 7a435e135f73d1f2f9b9fa099dc6756e6b01529bf660ced736d923724ebb425d
|
||||
|
||||
HOW TO REGENERATE
|
||||
=================
|
||||
mkdir -p /path/to/doris-415 && git archive 4.1.5-release | tar -x -C /path/to/doris-415
|
||||
cd /path/to/doris-415
|
||||
DORIS_HOME=$PWD DORIS_THIRDPARTY=/path/to/doris/thirdparty bash generated-source.sh
|
||||
# compile only, and resolve the classpath inside the same reactor so the 4.1.5 modules come from their
|
||||
# own target/classes rather than from ~/.m2 (where 1.2-SNAPSHOT is the CURRENT branch's):
|
||||
mvn -o -f fe/pom.xml -pl fe-core -am -Dmaven.test.skip=true -Dcheckstyle.skip=true -Dlicense.skip=true \
|
||||
-Drat.skip=true -Dmaven.build.cache.enabled=false -Dmdep.includeScope=runtime \
|
||||
compile dependency:build-classpath > build.log
|
||||
# the last "Dependencies classpath:" block in build.log is fe-core's; prepend fe-core's own classes:
|
||||
CP=$PWD/fe/fe-core/target/classes:$(grep -A1 'Dependencies classpath:' build.log | tail -1)
|
||||
javac -d gen-classes -cp "$CP" Gen415Fixtures.java
|
||||
java -cp "gen-classes:$CP" org.apache.doris.tools.Gen415Fixtures <outDir>
|
||||
|
||||
Output is byte-deterministic across JVM runs (verified: two runs produce identical files).
|
||||
|
||||
!! NEVER run `mvn install` in the 4.1.5 tree: both trees resolve ${revision}=1.2-SNAPSHOT.
|
||||
|
||||
CONTENTS
|
||||
========
|
||||
datasource.module.bin the image's "datasource" module, framed exactly as MetaWriter writes it: the
|
||||
internal catalog plus three Lance catalogs (ids 10101-10103), each built through
|
||||
4.1.5's own CatalogFactory.createFromLog, i.e. what a catalog created on 4.1.4+
|
||||
looks like. Their props always carry "type": 4.1.5 has no Lance resource type, so
|
||||
there is no resource-backed (4.1.3 "G2") Lance shape.
|
||||
10101 g1_lance_s3 filesystem namespace on S3-compatible storage, comment, taap
|
||||
10102 g1_lance_local filesystem namespace on file://
|
||||
10103 g1_lance_rest REST namespace with bearer auth and namespace options
|
||||
catalogMgr.json the same JSON unframed, for human diffing. Not read by the tests.
|
||||
labels.{ds,db,tbl}.txt 4.1.5's own GSON label registries, read reflectively out of GsonUtils.
|
||||
editlog/op320-*.bin OP_CREATE_CATALOG for each Lance catalog, in JournalEntity wire form.
|
||||
|
||||
NOT HERE, ON PURPOSE
|
||||
====================
|
||||
Lance databases / tables 4.1.5 never persists an external database or table (only CatalogMgr.idToCatalog
|
||||
is serialized), so no image or journal can carry those two labels; they are
|
||||
covered by labels.{db,tbl}.txt.
|
||||
@@ -0,0 +1 @@
|
||||
{"idToCatalog":{"0":{"clazz":"InternalCatalog"},"10101":{"clazz":"LanceExternalCatalog","id":10101,"name":"g1_lance_s3","logType":"LANCE","catalogProperty":{"properties":{"type":"lance","lance.catalog.type":"filesystem","warehouse":"s3://warehouse/lance","s3.endpoint":"http://minio:9000","s3.access_key":"ak","s3.secret_key":"sk","s3.region":"us-east-1","use_path_style":"true","use_meta_cache":"true"}},"initialized":false,"lastUpdateTime":1753000000000,"taap":[[{"first":"db1","second":"tbl1"},"enable"]],"comment":"lance on s3"},"10102":{"clazz":"LanceExternalCatalog","id":10102,"name":"g1_lance_local","logType":"LANCE","catalogProperty":{"properties":{"type":"lance","lance.catalog.type":"filesystem","warehouse":"file:///tmp/lance_wh","use_meta_cache":"true"}},"initialized":false,"lastUpdateTime":1753000000000,"taap":{},"comment":""},"10103":{"clazz":"LanceExternalCatalog","id":10103,"name":"g1_lance_rest","logType":"LANCE","catalogProperty":{"properties":{"type":"lance","lance.catalog.type":"rest","lance.rest.uri":"http://lance-rest:2333","lance.rest.security.type":"bearer","lance.rest.bearer-token":"tok","lance.namespace.delimiter":".","lance.namespace.root_database":"default","use_meta_cache":"true"}},"initialized":false,"lastUpdateTime":1753000000000,"taap":{},"comment":""}}}
|
||||
Binary file not shown.
@@ -0,0 +1,4 @@
|
||||
# generated by Gen415Fixtures -- do not edit by hand
|
||||
op320-create-g1_lance_local.bin
|
||||
op320-create-g1_lance_rest.bin
|
||||
op320-create-g1_lance_s3.bin
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,248 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
package org.apache.doris.tools;
|
||||
|
||||
import org.apache.doris.common.FeConstants;
|
||||
import org.apache.doris.common.io.Text;
|
||||
import org.apache.doris.datasource.CatalogFactory;
|
||||
import org.apache.doris.datasource.CatalogIf;
|
||||
import org.apache.doris.datasource.CatalogLog;
|
||||
import org.apache.doris.datasource.CatalogMgr;
|
||||
import org.apache.doris.datasource.ExternalCatalog;
|
||||
import org.apache.doris.persist.gson.GsonUtils;
|
||||
import org.apache.doris.persist.gson.RuntimeTypeAdapterFactory;
|
||||
|
||||
import java.io.DataOutputStream;
|
||||
import java.io.File;
|
||||
import java.io.FileOutputStream;
|
||||
import java.lang.reflect.Field;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Paths;
|
||||
import java.security.MessageDigest;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.TreeMap;
|
||||
import java.util.concurrent.ConcurrentMap;
|
||||
|
||||
/**
|
||||
* Generates golden metadata fixtures EXACTLY as Doris 4.1.5 writes them, for the Lance catalogs that
|
||||
* 4.1.4 introduced.
|
||||
*
|
||||
* <p>This class is NOT part of Doris. It runs only against a throwaway tree checked out at the 4.1.5-release
|
||||
* tag, so that the fixtures it emits are produced by real 4.1.5 bytecode rather than by someone's idea of
|
||||
* what 4.1.5 writes. The 4.1.3 fixtures next to these (upgrade/413) were made the same way.</p>
|
||||
*
|
||||
* <p>Usage: {@code java -cp <fe-core classes + deps> org.apache.doris.tools.Gen415Fixtures <outDir>}</p>
|
||||
*/
|
||||
public class Gen415Fixtures {
|
||||
|
||||
/** Fixed so the output is byte-deterministic across runs; the same value the 4.1.3 generator stamps. */
|
||||
private static final long FIXED_UPDATE_TIME = 1753000000000L;
|
||||
|
||||
private final List<CatalogIf<?>> catalogs = new ArrayList<>();
|
||||
private final List<String> failures = new ArrayList<>();
|
||||
|
||||
public static void main(String[] args) throws Exception {
|
||||
if (args.length < 1) {
|
||||
throw new IllegalArgumentException("usage: Gen415Fixtures <outDir>");
|
||||
}
|
||||
FeConstants.runningUnitTest = true;
|
||||
new Gen415Fixtures().run(new File(args[0]));
|
||||
}
|
||||
|
||||
private void run(File outDir) throws Exception {
|
||||
if (!outDir.exists() && !outDir.mkdirs()) {
|
||||
throw new IllegalStateException("cannot create " + outDir);
|
||||
}
|
||||
buildLance();
|
||||
|
||||
CatalogMgr mgr = newCatalogMgrWith(catalogs);
|
||||
String json = GsonUtils.GSON.toJson(mgr);
|
||||
|
||||
writeFramed(new File(outDir, "datasource.module.bin"), json);
|
||||
Files.write(Paths.get(new File(outDir, "catalogMgr.json").toURI()), json.getBytes(StandardCharsets.UTF_8));
|
||||
dumpLabels(outDir);
|
||||
buildEditLogFixtures(new File(outDir, "editlog"));
|
||||
|
||||
System.out.println("=== catalogs written: " + catalogs.size());
|
||||
for (CatalogIf<?> c : catalogs) {
|
||||
System.out.println(" " + c.getId() + " " + c.getClass().getSimpleName() + " " + c.getName());
|
||||
}
|
||||
System.out.println("=== module sha256: " + sha256(json.getBytes(StandardCharsets.UTF_8)));
|
||||
if (!failures.isEmpty()) {
|
||||
System.out.println("=== FAILURES (" + failures.size() + "):");
|
||||
failures.forEach(f -> System.out.println(" " + f));
|
||||
throw new IllegalStateException(failures.size() + " fixture(s) could not be built");
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------------------
|
||||
// Lance catalogs, built through 4.1.5's own CatalogFactory replay path, i.e. what a catalog CREATED ON
|
||||
// 4.1.4+ looks like. Their props always carry "type": Resource.ResourceType has no LANCE value, so a
|
||||
// resource-backed Lance catalog cannot exist and there is no 4.1.3-style G2 family to add here.
|
||||
// ---------------------------------------------------------------------------------------------------
|
||||
private void buildLance() {
|
||||
// The shape of the external_table_p0/lance suites: a filesystem namespace on S3-compatible storage.
|
||||
lance(10101, "g1_lance_s3", "lance on s3", props("type", "lance",
|
||||
"lance.catalog.type", "filesystem", "warehouse", "s3://warehouse/lance",
|
||||
"s3.endpoint", "http://minio:9000", "s3.access_key", "ak", "s3.secret_key", "sk",
|
||||
"s3.region", "us-east-1", "use_path_style", "true"), true);
|
||||
lance(10102, "g1_lance_local", "", props("type", "lance",
|
||||
"lance.catalog.type", "filesystem", "warehouse", "file:///tmp/lance_wh"), false);
|
||||
lance(10103, "g1_lance_rest", "", props("type", "lance",
|
||||
"lance.catalog.type", "rest", "lance.rest.uri", "http://lance-rest:2333",
|
||||
"lance.rest.security.type", "bearer", "lance.rest.bearer-token", "tok",
|
||||
"lance.namespace.delimiter", ".", "lance.namespace.root_database", "default"), false);
|
||||
}
|
||||
|
||||
private void lance(long id, String name, String comment, Map<String, String> props, boolean withTaap) {
|
||||
try {
|
||||
CatalogLog log = new CatalogLog();
|
||||
log.setCatalogId(id);
|
||||
log.setCatalogName(name);
|
||||
log.setComment(comment);
|
||||
log.setProps(props);
|
||||
CatalogIf<?> c = CatalogFactory.createFromLog(log);
|
||||
decorate(c, withTaap);
|
||||
catalogs.add(c);
|
||||
} catch (Throwable t) {
|
||||
failures.add(name + ": " + t);
|
||||
}
|
||||
}
|
||||
|
||||
private void decorate(CatalogIf<?> c, boolean withTaap) {
|
||||
ExternalCatalog ec = (ExternalCatalog) c;
|
||||
ec.setLastUpdateTime(FIXED_UPDATE_TIME);
|
||||
if (withTaap) {
|
||||
// Empty map serializes as {}, non-empty as an array of [key, value] 2-tuples
|
||||
// (enableComplexMapKeySerialization); keep one catalog on each encoding.
|
||||
ec.setAutoAnalyzePolicy("db1", "tbl1", "enable");
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------------------
|
||||
// Edit-log fixtures: JournalEntity wire form is writeShort(opCode) + payload.write(out), and
|
||||
// CatalogLog.write is Text.writeString(out, GSON.toJson(this)). OP_CREATE_CATALOG only; the other
|
||||
// catalog ops carry no catalog type and are covered by the 4.1.3 fixtures.
|
||||
// ---------------------------------------------------------------------------------------------------
|
||||
private void buildEditLogFixtures(File dir) throws Exception {
|
||||
if (!dir.exists() && !dir.mkdirs()) {
|
||||
throw new IllegalStateException("cannot create " + dir);
|
||||
}
|
||||
for (CatalogIf<?> c : catalogs) {
|
||||
writeOp(dir, 320, "create-" + c.getName(), c.constructEditLog());
|
||||
}
|
||||
|
||||
String[] names = dir.list();
|
||||
if (names == null) {
|
||||
throw new IllegalStateException("cannot list " + dir);
|
||||
}
|
||||
java.util.Arrays.sort(names);
|
||||
StringBuilder sb = new StringBuilder("# generated by Gen415Fixtures -- do not edit by hand\n");
|
||||
for (String n : names) {
|
||||
sb.append(n).append('\n');
|
||||
}
|
||||
Files.write(Paths.get(new File(dir.getParentFile(), "editlog.index").toURI()),
|
||||
sb.toString().getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
|
||||
private void writeOp(File dir, int opCode, String name, CatalogLog log) throws Exception {
|
||||
File f = new File(dir, String.format("op%d-%s.bin", opCode, name));
|
||||
try (DataOutputStream out = new DataOutputStream(new FileOutputStream(f))) {
|
||||
out.writeShort(opCode);
|
||||
log.write(out);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------------------
|
||||
// Label inventory, read reflectively out of 4.1.5's own registries (a regex scrape drops labels whose
|
||||
// class name contains a digit).
|
||||
// ---------------------------------------------------------------------------------------------------
|
||||
private void dumpLabels(File outDir) throws Exception {
|
||||
dumpOneRegistry(outDir, "labels.ds.txt", "dsTypeAdapterFactory");
|
||||
dumpOneRegistry(outDir, "labels.db.txt", "dbTypeAdapterFactory");
|
||||
dumpOneRegistry(outDir, "labels.tbl.txt", "tblTypeAdapterFactory");
|
||||
}
|
||||
|
||||
private void dumpOneRegistry(File outDir, String fileName, String fieldName) throws Exception {
|
||||
Field f = GsonUtils.class.getDeclaredField(fieldName);
|
||||
f.setAccessible(true);
|
||||
RuntimeTypeAdapterFactory<?> factory = (RuntimeTypeAdapterFactory<?>) f.get(null);
|
||||
Field l = RuntimeTypeAdapterFactory.class.getDeclaredField("labelToSubtype");
|
||||
l.setAccessible(true);
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Class<?>> labelToSubtype = (Map<String, Class<?>>) l.get(factory);
|
||||
StringBuilder sb = new StringBuilder();
|
||||
sb.append("# generated from 4.1.5 by Gen415Fixtures -- do not edit by hand\n");
|
||||
sb.append("# <label>\\t<target class FQN>\\t<ABSTRACT if the label can never be emitted>\n");
|
||||
new TreeMap<>(labelToSubtype).forEach((label, clazz) -> sb.append(label).append('\t')
|
||||
.append(clazz.getName())
|
||||
.append(java.lang.reflect.Modifier.isAbstract(clazz.getModifiers()) ? "\tABSTRACT" : "")
|
||||
.append('\n'));
|
||||
Files.write(Paths.get(new File(outDir, fileName).toURI()), sb.toString().getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------------------
|
||||
// helpers (same as the 4.1.3 generator)
|
||||
// ---------------------------------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* A CatalogMgr holding the internal catalog (added by its own ctor) plus the given externals, injected
|
||||
* straight into idToCatalog: addCatalog() needs a live Env. Only idToCatalog is @SerializedName.
|
||||
*/
|
||||
private CatalogMgr newCatalogMgrWith(List<CatalogIf<?>> externals) throws Exception {
|
||||
CatalogMgr mgr = new CatalogMgr();
|
||||
Field f = CatalogMgr.class.getDeclaredField("idToCatalog");
|
||||
f.setAccessible(true);
|
||||
@SuppressWarnings("unchecked")
|
||||
ConcurrentMap<Long, CatalogIf<?>> idToCatalog = (ConcurrentMap<Long, CatalogIf<?>>) f.get(mgr);
|
||||
for (CatalogIf<?> c : externals) {
|
||||
idToCatalog.put(c.getId(), c);
|
||||
}
|
||||
return mgr;
|
||||
}
|
||||
|
||||
private static void writeFramed(File file, String json) throws Exception {
|
||||
try (DataOutputStream out = new DataOutputStream(new FileOutputStream(file))) {
|
||||
Text.writeString(out, json);
|
||||
}
|
||||
}
|
||||
|
||||
private static Map<String, String> props(String... kv) {
|
||||
if (kv.length % 2 != 0) {
|
||||
throw new IllegalArgumentException("odd number of props");
|
||||
}
|
||||
Map<String, String> m = new LinkedHashMap<>();
|
||||
for (int i = 0; i < kv.length; i += 2) {
|
||||
m.put(kv[i], kv[i + 1]);
|
||||
}
|
||||
return m;
|
||||
}
|
||||
|
||||
private static String sha256(byte[] bytes) throws Exception {
|
||||
MessageDigest md = MessageDigest.getInstance("SHA-256");
|
||||
StringBuilder sb = new StringBuilder();
|
||||
for (byte b : md.digest(bytes)) {
|
||||
sb.append(String.format("%02x", b));
|
||||
}
|
||||
return sb.toString();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
# generated from 4.1.5 by Gen415Fixtures -- do not edit by hand
|
||||
# <label>\t<target class FQN>\t<ABSTRACT if the label can never be emitted>
|
||||
EsExternalDatabase org.apache.doris.datasource.es.EsExternalDatabase
|
||||
ExternalDatabase org.apache.doris.datasource.ExternalDatabase ABSTRACT
|
||||
ExternalInfoSchemaDatabase org.apache.doris.datasource.infoschema.ExternalInfoSchemaDatabase
|
||||
ExternalMysqlDatabase org.apache.doris.datasource.infoschema.ExternalMysqlDatabase
|
||||
HMSExternalDatabase org.apache.doris.datasource.hive.HMSExternalDatabase
|
||||
IcebergExternalDatabase org.apache.doris.datasource.iceberg.IcebergExternalDatabase
|
||||
JdbcExternalDatabase org.apache.doris.datasource.jdbc.JdbcExternalDatabase
|
||||
LakeSoulExternalDatabase org.apache.doris.datasource.lakesoul.LakeSoulExternalDatabase
|
||||
LanceExternalDatabase org.apache.doris.datasource.lance.LanceExternalDatabase
|
||||
MaxComputeExternalDatabase org.apache.doris.datasource.maxcompute.MaxComputeExternalDatabase
|
||||
PaimonExternalDatabase org.apache.doris.datasource.paimon.PaimonExternalDatabase
|
||||
TestExternalDatabase org.apache.doris.datasource.test.TestExternalDatabase
|
||||
TrinoConnectorExternalDatabase org.apache.doris.datasource.trinoconnector.TrinoConnectorExternalDatabase
|
||||
@@ -0,0 +1,26 @@
|
||||
# generated from 4.1.5 by Gen415Fixtures -- do not edit by hand
|
||||
# <label>\t<target class FQN>\t<ABSTRACT if the label can never be emitted>
|
||||
CloudInternalCatalog org.apache.doris.cloud.datasource.CloudInternalCatalog
|
||||
EsExternalCatalog org.apache.doris.datasource.es.EsExternalCatalog
|
||||
HMSExternalCatalog org.apache.doris.datasource.hive.HMSExternalCatalog
|
||||
IcebergDLFExternalCatalog org.apache.doris.datasource.iceberg.IcebergDLFExternalCatalog
|
||||
IcebergExternalCatalog org.apache.doris.datasource.iceberg.IcebergExternalCatalog ABSTRACT
|
||||
IcebergGlueExternalCatalog org.apache.doris.datasource.iceberg.IcebergGlueExternalCatalog
|
||||
IcebergHMSExternalCatalog org.apache.doris.datasource.iceberg.IcebergHMSExternalCatalog
|
||||
IcebergHadoopExternalCatalog org.apache.doris.datasource.iceberg.IcebergHadoopExternalCatalog
|
||||
IcebergJdbcExternalCatalog org.apache.doris.datasource.iceberg.IcebergJdbcExternalCatalog
|
||||
IcebergRestExternalCatalog org.apache.doris.datasource.iceberg.IcebergRestExternalCatalog
|
||||
IcebergS3TablesExternalCatalog org.apache.doris.datasource.iceberg.IcebergS3TablesExternalCatalog
|
||||
InternalCatalog org.apache.doris.datasource.InternalCatalog
|
||||
JdbcExternalCatalog org.apache.doris.datasource.jdbc.JdbcExternalCatalog
|
||||
LakeSoulExternalCatalog org.apache.doris.datasource.lakesoul.LakeSoulExternalCatalog
|
||||
LanceExternalCatalog org.apache.doris.datasource.lance.LanceExternalCatalog
|
||||
MaxComputeExternalCatalog org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog
|
||||
PaimonDLFExternalCatalog org.apache.doris.datasource.paimon.PaimonDLFExternalCatalog
|
||||
PaimonExternalCatalog org.apache.doris.datasource.paimon.PaimonExternalCatalog
|
||||
PaimonFileExternalCatalog org.apache.doris.datasource.paimon.PaimonFileExternalCatalog
|
||||
PaimonHMSExternalCatalog org.apache.doris.datasource.paimon.PaimonHMSExternalCatalog
|
||||
PaimonRestExternalCatalog org.apache.doris.datasource.paimon.PaimonRestExternalCatalog
|
||||
RemoteDorisExternalCatalog org.apache.doris.datasource.doris.RemoteDorisExternalCatalog
|
||||
TestExternalCatalog org.apache.doris.datasource.test.TestExternalCatalog
|
||||
TrinoConnectorExternalCatalog org.apache.doris.datasource.trinoconnector.TrinoConnectorExternalCatalog
|
||||
@@ -0,0 +1,29 @@
|
||||
# generated from 4.1.5 by Gen415Fixtures -- do not edit by hand
|
||||
# <label>\t<target class FQN>\t<ABSTRACT if the label can never be emitted>
|
||||
BrokerTable org.apache.doris.catalog.BrokerTable
|
||||
Dictionary org.apache.doris.dictionary.Dictionary
|
||||
EsExternalTable org.apache.doris.datasource.es.EsExternalTable
|
||||
EsTable org.apache.doris.catalog.EsTable
|
||||
ExternalInfoSchemaTable org.apache.doris.datasource.infoschema.ExternalInfoSchemaTable
|
||||
ExternalMysqlTable org.apache.doris.datasource.infoschema.ExternalMysqlTable
|
||||
ExternalTable org.apache.doris.datasource.ExternalTable
|
||||
FunctionGenTable org.apache.doris.catalog.FunctionGenTable
|
||||
HMSExternalTable org.apache.doris.datasource.hive.HMSExternalTable
|
||||
HiveTable org.apache.doris.catalog.HiveTable
|
||||
IcebergExternalTable org.apache.doris.datasource.iceberg.IcebergExternalTable
|
||||
InlineView org.apache.doris.catalog.InlineView
|
||||
JdbcExternalTable org.apache.doris.datasource.jdbc.JdbcExternalTable
|
||||
JdbcTable org.apache.doris.catalog.JdbcTable
|
||||
LakeSoulExternalTable org.apache.doris.datasource.lakesoul.LakeSoulExternalTable
|
||||
LanceExternalTable org.apache.doris.datasource.lance.LanceExternalTable
|
||||
MTMV org.apache.doris.catalog.MTMV
|
||||
MaxComputeExternalTable org.apache.doris.datasource.maxcompute.MaxComputeExternalTable
|
||||
MysqlDBTable org.apache.doris.catalog.MysqlDBTable
|
||||
MysqlTable org.apache.doris.catalog.MysqlTable
|
||||
OdbcTable org.apache.doris.catalog.OdbcTable
|
||||
OlapTable org.apache.doris.catalog.OlapTable
|
||||
PaimonExternalTable org.apache.doris.datasource.paimon.PaimonExternalTable
|
||||
SchemaTable org.apache.doris.catalog.SchemaTable
|
||||
TestExternalTable org.apache.doris.datasource.test.TestExternalTable
|
||||
TrinoConnectorExternalTable org.apache.doris.datasource.trinoconnector.TrinoConnectorExternalTable
|
||||
View org.apache.doris.catalog.View
|
||||
Reference in new issue
Block a user