mirror of
https://github.com/apache/doris.git
synced 2026-11-04 18:29:59 +00:00
Mingyu Chen (Rayner)
ef58609449
### What problem does this PR solve? Issue Number: close #xxx Related PR: #66403 (merged) Problem Summary: Doris can now read [Apache Fluss](https://fluss.apache.org/) tables. This adds a **fluss catalog**, reading all three shapes a fluss table can take: | Table | How it is read | |---|---| | log table | the fluss change log, one scan range per bucket | | primary-key table | a kv snapshot plus the change log after it, merged by key | | tiered table (`table.datalake.enabled`) | its **paimon lake plus the log written after it**, in one scan | For a tiered table the lake half is not re-implemented: it is planned by the **paimon connector**, on the snapshot the fluss coordinator pinned, so that half gets native ORC/Parquet, deletion vectors and the file cache for free — and the fluss plugin ships **no paimon dependency at all** (it borrows the paimon plugin through `createSiblingConnector`, the way hive borrows iceberg and hudi). For a tiered **primary-key** table the two halves are merged BY KEY, natively in BE: FE wraps each lake split with the key set of that bucket's log tail, the BE reader drops the lake rows that tail replaced before materialization, and the tail is replayed once as its own range. ```sql CREATE CATALOG fluss PROPERTIES ( "type" = "fluss", "fluss.bootstrap.servers" = "host:9123" ); SELECT * FROM fluss.db.tbl; -- lake + log, merged SELECT * FROM fluss.db.`tbl$lake`; -- the lake alone SELECT * FROM fluss.db.`tbl$log`; -- the part that is not in the lake yet ``` #### Naming a segment, and choosing a path `tbl$lake` and `tbl$log` name **which segment of the table** is read, and together they partition it: `tbl$lake ⊎ tbl$log == tbl`. They are named after where the data is, not after how fresh it is, because a stopped tiering service makes "the recent part" a lie while "the part still in the log" stays true. How the whole table is read is a different question, and it is a **session variable** rather than a suffix, because the useful third value cannot be spelled as a name: ```sql SET fluss_union_read_mode = 'required'; -- fail rather than fall back to a fluss-only read SET fluss_union_read_mode = 'disabled'; -- read fluss alone SET fluss_union_read_mode = ''; -- follow the catalog (the default) ``` `required` is the one a regression baseline and a bug report want: it turns "quietly read it the other way" into an error. The variable overrides `fluss.union_read.mode` for one statement and changes no row that comes back — only which reader produced it. #### Lake settings a fluss cluster cannot send Fluss removes every lake option whose name contains `key`, `secret` or `password` before it hands a table's properties to a client, so a lake in an object store cannot have its credentials learned from the cluster at all. A catalog states them itself, under paimon's own spelling so that the tail of a `datalake.paimon.*` line can be copied straight out of a cluster's configuration: ```sql CREATE CATALOG fluss PROPERTIES ( "type" = "fluss", "fluss.bootstrap.servers" = "host:9123", "fluss.lake.paimon.s3.endpoint" = "http://minio:9000", "fluss.lake.paimon.s3.access-key" = "...", "fluss.lake.paimon.s3.secret-key" = "..." ); ``` Overrides are applied key by key, not as a replacement set, so stating the metastore leaves the warehouse alone. The keys that describe **storage** are taken out of what the lake catalog gets and handed to the engine's storage layer instead, under the names it binds them by: storage is configured once per catalog and read by the FE (which reads manifests while planning) and the BE (which reads the data files) alike, and a storage setting carried to the lake catalog would reach at most the FE — sending the two halves to different endpoints with nothing reporting the disagreement. The lake's paimon catalog may be `filesystem`, `hive` or `rest`. #### Where a scan's time went Because a single fluss scan can be four different reads at once — fluss's own log, a paimon lake, the log tail that supersedes part of it, and the suppression itself — the scan reports where its time and rows went, per side and per kind of range: ``` TableReader ├── FlussLogReadTime 8s25ms the JNI side, everything forwarded to it │ ├── FlussPkTailRangeNum 2 registered only for the kinds this scan read │ ├── FlussPkTailRangeReadTime 6s2ms │ └── FlussLogRowsReturned 2 ├── FlussLakeReadTime 8s25ms the paimon side, tail read and suppression included │ ├── FlussLakeRangeNum 1 │ ├── FlussLakeSuppressRangeNum 2 │ └── FlussLakeRowsReturned 7 after suppression; + SuppressedRows = before ├── FlussUnionTailReadTime 5s20ms one round trip to fluss per bucket └── FlussUnionSuppressTime 16.1us per block, grows with the lake's rows ``` A side's total minus its range kinds' is what standing that child up cost — JNI class loading, or the paimon stack's setup — so "slow scan" resolves to a specific half and a specific cause without subtracting counters by hand. ### Reviewing this 85 commits, rebased on master (`f88735f8f62`). Two earlier PRs carried the parts of this work that were **not** in the fluss module and have both landed, so neither is part of this branch any more: #66403 (five independent fixes a sibling-connector read depends on) and #66589 (removing FileScannerV2's per-range table reader rebuild, which this branch's own refactor left without a producer). What remains is the connector itself plus some changes to shared code. These three commits reshape the shared BE scan path and are called out here so a reviewer knows to look at them: | Commit | Touches | Why it is here | |---|---|---| | `[feat](be) Dispatch a wrapped lake split to the reader that unwraps it` | `be/src/exec/scan/file_scanner_v2.cpp` | Teaches the V2 scanner that a fluss range may carry a paimon payload, so it can answer "can V2 read this" by the same key the fluss reader dispatches on. | | `[refactor](be) file_scanner_v2: one table format for every fluss range` | `be/src/exec/scan/file_scanner_v2.cpp`, `file_scan_operator.cpp` | A review request on this PR: the scanner should not see two `table_format_type`s for one table. Every fluss range is now the format `fluss`, and the split-level dispatch moved inside the connector's own reader — the same arrangement paimon and hudi already have. | | `[opt](fluss) Retain each log-tail key once, not once per change-log record` | `be/src/format_v2/expr/equality_delete_predicate.{h,cpp}` | Adds a static `distinct_rows` to the class that owns the equality, rather than deduplicating beside the caller: a caller with its own idea of when two keys match could drop a row this predicate would not have suppressed. Additive; no existing behaviour changes. | Beyond these three, the branch also touches shared code outside the fluss module: the connector SPI (`ConnectorCapability`, `ConnectorScanRequest`); in fe-core the plugin-driven table and scan classes, `SlotTypeReplacer`, `LogicalFileScan`, `DescribeCommand`, `SessionVariable`, `CatalogMgr`, `DatasourcePrintableMap`, `MetadataGenerator`, `LineageUtils` and `KafkaRoutineLoadJob`; the paimon, hive and iceberg connectors; the paimon BE scanner and the JNI plugin toolkit; and in BE `parquet_scan.cpp` / `column_data.h`, `jni_table_reader.cpp` and `jni_plugin_registry.h`. Build, thrift and pipeline files (`build.sh`, `run-fe-ut.sh`, `PlanNodes.thrift`, the external pipeline's regression conf) change as well. The connector commits are grouped so that each group compiles, runs and has its own acceptance: 1. **metadata** — module, type mapping, catalog/table/partition metadata, the e2e docker environment 2. **log tables** — thrift payload, FE planning, the BE java scanner, the BE C++ glue 3. **primary-key tables** — kv snapshot + change log 4. **the lake** — `$lake` through the paimon sibling, and lake-plus-log for log tables 5. **merging a primary-key table's halves by key** — FE planning, the tail replayer, the BE suppression reader 6. **coverage** — a partition-column-type gate, and five more e2e suites 7. **shape and observability** — the property layout every other connector uses, one table format for every range, and the per-side/per-range-kind profile above 8. **saying what to read and where the lake is** — `$log`, the session variable, catalog-level lake settings, the storage keys, the non-filesystem lake catalogs, and two more e2e suites ### Things worth saying out loud - **This depends on `fluss 1.0.0`**, released on 2026-09-21 and taken from Maven Central; the APIs this uses are not in 0.9.1, and 1.0.0 is the first release that has them. The docker environment runs the official `apache/fluss:1.0.0` and `apache/fluss-quickstart-flink:1.20-1.0.0` images, with the one jar the server image lacks (`paimon-s3`) mounted into its paimon plugin directory. Moving from the development snapshot to 1.0.0 surfaced two behaviour changes, both handled here: every fluss partition now carries its own bucket count (1.0.0 supports rescaling), so a partition is planned by the count it was created with rather than the table's; and the paimon table the tiering service writes no longer carries `__bucket`/`__offset`/`__timestamp` (paimon 2.0's `lakestream.enabled` takes their place), which the suites now expect. - **Connector plugin API 12.0.** The two `ConnectorCapability` constants this series adds (`SUPPORTS_FIELD_ID_ACCESS_PATH`, `SUPPORTS_SYS_TABLE_NESTED_COLUMN_PRUNE`) are a plugin-surface change, so `connector.plugin.api.version` moves from master's 11.0 (#68320) to 12.0, in the same commit that records the two constants in the plugin-surface baseline. - **The e2e suites run in the External Regression pipeline.** It starts the fluss docker component and its conf sets `enableFlussTest=true`, so the suites under `external_table_p0/fluss` exercise a real cluster instead of returning early. The local default stays `false`, for developers who have not started the fluss environment. - **The e2e lake lives in an object store**, in a minio the fluss compose brings up itself. That is not "one more storage backend to cover": with the lake in a directory, the whole credential path is unexecuted, because a warehouse that needs no credentials never asks for any. With it in a bucket, every lake-reading case in the suite exercises the catalog-stated credentials, and the FE's manifest reads and the BE's data-file reads are shown to be configured from the same answer. `FLUSS_PAIMON_WAREHOUSE` still takes a directory path, so a storage problem can be told from a lake problem by switching back. - **Two things have no end-to-end coverage and are held up by unit tests alone**, which is worth knowing before trusting them: - a `hive` or `rest` lake catalog. The e2e environment's lake is a filesystem catalog, and standing up a metastore beside it is a separate piece of work. - `s3.path.style.access`. It is carried to the engine's storage layer like any other storage key, but the e2e object store is addressed by IP, where the addressing rules fall back to path style on their own — so no query in this environment can distinguish the setting taking effect from it being ignored. An object store named by a hostname would. - Two pre-existing bugs in neighbouring code were found by this work and are **not** fixed here, so as not to widen the diff. Both reproduce without fluss: an equality predicate on a microsecond `TIMESTAMP` pushed into paimon matches no row (a plain paimon catalog over the same warehouse behaves the same), and `java-common`'s VARBINARY read-back decodes what it wrote as a StringView, so it always reads zeros (the write side and BE agree; only the Java read-back is wrong, and no production path takes it). - `TIME` is mapped to `UNSUPPORTED` rather than to a string or to elapsed millis: Doris has no type meaning what fluss's TIME means, and handing back a plausible value of a different meaning is not something a later error would catch. The paimon and iceberg connectors mark their own TIME the same way, which also keeps `tbl` and `tbl$lake` agreeing on one schema. - Predicate pass-through into the lake half and the write path are follow-ups. - The profile counters are registered lazily, so a pure log-table query's profile carries no lake or primary-key lines at all rather than a column of zeros. `FlussUnionSuppressTime` is nested inside the existing `FinalizeBlockTime` rather than taken out of it, so no established counter changed meaning.
137 lines
6.2 KiB
XML
137 lines
6.2 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!--
|
|
Licensed to the Apache Software Foundation (ASF) under one
|
|
or more contributor license agreements. See the NOTICE file
|
|
distributed with this work for additional information
|
|
regarding copyright ownership. The ASF licenses this file
|
|
to you under the Apache License, Version 2.0 (the
|
|
"License"); you may not use this file except in compliance
|
|
with the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing,
|
|
software distributed under the License is distributed on an
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations
|
|
under the License.
|
|
-->
|
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
|
<parent>
|
|
<artifactId>be-java-extensions</artifactId>
|
|
<groupId>org.apache.doris</groupId>
|
|
<version>${revision}</version>
|
|
</parent>
|
|
<modelVersion>4.0.0</modelVersion>
|
|
<artifactId>plugin-toolkit</artifactId>
|
|
<name>Doris BE Plugin Toolkit</name>
|
|
<description>
|
|
Utilities shared by the plugins' source, never by their runtime.
|
|
|
|
Unlike jni-spi this jar is NOT on BE's classpath: each plugin that wants it declares a
|
|
normal compile dependency and gets its own copy inside its own directory. So two plugins
|
|
using it share the source but not the classes, which is the whole point - a bug fix reaches
|
|
both, while neither can be affected by the other's version of anything. Trino's
|
|
trino-plugin-toolkit works the same way.
|
|
|
|
The corollary is that anything added here is duplicated into every consumer, so it must be
|
|
code that genuinely has more than one, and it must stay dependency-free: a dependency here
|
|
would be forced on every plugin that wants any part of the toolkit.
|
|
</description>
|
|
|
|
<properties>
|
|
<maven.compiler.source>8</maven.compiler.source>
|
|
<maven.compiler.target>8</maven.compiler.target>
|
|
</properties>
|
|
|
|
<dependencies>
|
|
<!--
|
|
The SPI a helper here may build on - NestedProjection walks the ColumnType tree BE hands a
|
|
scanner. Provided, never compile: the enforcer below only pre-authorises it in that scope,
|
|
because a compile-scope SPI would be copied into every consuming plugin's directory and
|
|
the loader rejects a plugin that ships its own SPI copy.
|
|
-->
|
|
<dependency>
|
|
<groupId>org.apache.doris</groupId>
|
|
<artifactId>jni-spi</artifactId>
|
|
<version>${project.version}</version>
|
|
<scope>provided</scope>
|
|
</dependency>
|
|
<!--
|
|
Narrowed to test scope for the same reason as in jni-spi: the fe parent pom declares
|
|
these for every module, and a child cannot drop an inherited dependency. See that pom
|
|
for the full explanation. Do not "fix" these to compile scope.
|
|
-->
|
|
<dependency>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-api</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
<dependency>
|
|
<groupId>org.apache.logging.log4j</groupId>
|
|
<artifactId>log4j-slf4j2-impl</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
<dependency>
|
|
<groupId>org.apache.logging.log4j</groupId>
|
|
<artifactId>log4j-core</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
<dependency>
|
|
<groupId>org.apache.logging.log4j</groupId>
|
|
<artifactId>log4j-1.2-api</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
</dependencies>
|
|
|
|
<build>
|
|
<finalName>doris-plugin-toolkit</finalName>
|
|
<plugins>
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-enforcer-plugin</artifactId>
|
|
<executions>
|
|
<execution>
|
|
<id>enforce-plugin-toolkit-stays-dependency-free</id>
|
|
<phase>validate</phase>
|
|
<goals>
|
|
<goal>enforce</goal>
|
|
</goals>
|
|
<configuration>
|
|
<rules>
|
|
<bannedDependencies>
|
|
<searchTransitive>true</searchTransitive>
|
|
<excludes>
|
|
<exclude>*</exclude>
|
|
</excludes>
|
|
<includes>
|
|
<!--
|
|
jni-spi is pre-authorised, but only in provided scope: a
|
|
compile-scope SPI would be copied into every consuming
|
|
plugin's directory, and the loader rejects a plugin that
|
|
ships its own SPI copy.
|
|
-->
|
|
<include>org.apache.doris:jni-spi:*:*:provided</include>
|
|
<include>*:*:*:*:test</include>
|
|
</includes>
|
|
<message>
|
|
plugin-toolkit must stay dependency-free. Its jar is copied
|
|
into every plugin that uses any part of it, so a dependency
|
|
added here is imposed on all of them - including plugins
|
|
that only wanted one unrelated helper. Put code that needs
|
|
a library in the plugin that needs it.
|
|
</message>
|
|
</bannedDependencies>
|
|
</rules>
|
|
<fail>true</fail>
|
|
</configuration>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
</plugins>
|
|
</build>
|
|
</project>
|