mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Mingyu Chen (Rayner)andClaude Opus 4.7
a73cdc0173
### What problem does this PR solve? Issue Number: close #65185 Related PR: every phase PR is listed in #65185 Problem Summary: This is the merge of the `branch-catalog-spi` feature branch, i.e. the Catalog SPI migration tracked in #65185. Per-phase design, review history and the per-connector status table live in that issue. The framework itself is documented in `fe/fe-connector/README.md` (architecture, module map, how to add a connector) and `fe/fe-connector/AGENTS.md` (build/test recipes, architecture gates, invariants). Before: fe-core hard-coded every external data source under `fe/fe-core/.../datasource/{hive,iceberg,paimon,hudi,trinoconnector,maxcompute,...}`, reached from the catalog and planner layers through `switch-case` and `instanceof *ExternalTable`. Adding or upgrading a source meant changing FE core, and every source's client stack lived on the single FE classpath. After: each source ships as a self-contained plugin under `fe/fe-connector/` (hive, hms, iceberg, paimon, hudi, trino, maxcompute, jdbc, es), loaded child-first from `Config.connector_plugin_root`. fe-core keeps only generic infrastructure — the `PluginDriven*` catalog / database / table / scan-node bridge plus one per-statement `ConnectorMetadata` funnel — and no compile-time knowledge of any specific source: `CatalogFactory` routes on `ConnectorProvider.getType()` alone. Two `validate`-phase gates keep it that way: a connector may not import fe-core internals (`build-support/check-fe-connector-imports.sh`), and only `PluginDrivenMetadata` may call `Connector#getMetadata` (`build-support/check-fe-core-metadata-funnel.sh`). Catalogs persisted by an older FE (`IcebergExternalCatalog`, `PaimonExternalDatabase`, ...) still deserialize and migrate transparently to the plugin-driven types. ### Release note External catalogs (hive, hms, iceberg, paimon, hudi, trino-connector, maxcompute, jdbc, es) are now served by connector plugins, shipped as zips under `fe/plugins/connector/` (`connector_plugin_root`). Existing catalogs keep working across upgrade; no DDL change is required. The default of `trino_connector_plugin_dir` moves from `plugins/connectors` to `plugins/trino_plugins`; the old directory is still read as a fallback. The deprecated `lakesoul` catalog type is removed — creating or replaying one now fails with "Lakesoul catalog is no longer supported". ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: Covered by the connector modules' own unit tests, fe-core unit tests (including a 4.1.3 metadata upgrade-compatibility suite), the two gate self-test scripts under `build-support/tests/`, and the `external_table_p0` / `external_table_p2` regression suites. - Behavior changed: - [ ] No. - [x] Yes. See the release note: plugin-based packaging and loading of external catalogs, the `trino_connector_plugin_dir` default, and the removal of the `lakesoul` catalog type. Everything else is held to behavior parity. - Does this need documentation? - [ ] No. - [x] Yes. Doc PR to follow: connector plugin layout, `connector_plugin_root`, the `trino_connector_plugin_dir` default, and the `lakesoul` removal. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
19 lines
812 B
Bash
19 lines
812 B
Bash
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
git submodule deinit -f .
|