mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Calvin Kirs
5e2d9a75a6
### What problem does this PR solve? Issue Number: close #xxx Related PR: #65586 (the branch-4.1 change this ports) Problem Summary: Hadoop's `FileSystem.CACHE` keys entries only by `(scheme, authority, UGI)`. Different catalogs/TVFs accessing the same bucket/namenode with different credentials could receive each other's cached `FileSystem` instance (cross-credential contamination), so Doris forced `fs.<schema>.impl.disable.cache=true` everywhere — which makes every access (e.g. every JNI scanner split) create a brand-new `FileSystem` instance (closing/releasing depends on GC). This PR makes the cache credential-aware instead of disabling it. It is the master counterpart of #65586; the 4.1 code no longer exists here (the connector-plugin refactor moved storage properties into the `fe-filesystem/*` SPI plugins and replaced the typed `StorageProperties` hierarchy with `StorageAdapter`), so this is the same design re-layered rather than a cherry-pick. **BE/FE shared patch (hadoop-deps):** - Ship a patched copy of `org.apache.hadoop.fs.FileSystem` (verbatim from hadoop-common 3.4.2, hunks marked `DORIS-PATCH`): `Cache.Key` carries one extra dimension read from the reserved conf property `doris.fs.cache.key.<scheme>`, falling back to the scheme-less `doris.fs.cache.key`. Absent/empty value = byte-for-byte vanilla behavior. - A second `DORIS-PATCH` in `loadFileSystems` binds the `ServiceLoader` scan to `FileSystem.class.getClassLoader()` instead of the thread context loader. `SERVICE_FILE_SYSTEMS` is a process-wide, first-caller-wins registry; now that plugins share the kernel's `FileSystem` (below), whichever plugin touched it first would otherwise freeze the scheme registry to that plugin's classpath. - maven-enforcer pins `hadoop.version=3.4.2`: upgrading hadoop fails the build until the patched file is re-synced. The vendored file is recorded in `LICENSE.txt` and excluded from checkstyle (scoped to `org/apache/hadoop/fs/**` so `fe-connector-hms`' `org/apache/hadoop/hive/**` stays checked). **Why the key is per scheme.** One `Configuration` is routinely merged from several storage definitions — a catalog can hold an object store *and* HDFS — and every consumer merges them with `putAll`. Under a single shared property name that merge is last-writer-wins: every scheme ends up tagged with one arbitrary storage's fingerprint, which is both unstable (map iteration order) and unsafe — two catalogs differing only in their object-store credentials but sharing an HDFS definition would collapse onto the same key. Distinct per-scheme names make the merge lossless, so **no merge site has to know this mechanism exists** and no consumer code changes: the fingerprint is set entirely inside the storage layer, on the way out of the property getters. **Load-order guarantees (four classpaths):** - BE loads `be/lib/hadoop_hdfs/hadoop-deps.jar` before `lib/*.jar` (existing). - FE declares `hadoop-deps` as fe-core's first dependency (all transitives excluded) and `start_fe.sh` prepends the jar explicitly — `collectJars`/glob order does not deliver it (`"hadoop-common-x.y.z.jar"` sorts *before* `"hadoop-deps.jar"`). - Cloud meta-service: `build.sh` now also copies the module's own jar (not just `copy-dependencies`' transitives) into `cloud/output/lib/hadoop_hdfs/`, and `cloud/script/start.sh` prepends it the same way. Without this the meta-service ran on the vanilla class and silently ignored the key. - **Plugins.** Filesystem plugins already resolve `org.apache.hadoop.*` parent-first (`FileSystemPluginManager.FS_PARENT_FIRST_PREFIXES`), so they see the FE copy. Connector plugins (iceberg/paimon/hudi/hive) bundle their own `hadoop-common` and loaded it child-first, so they would have run on a vanilla `FileSystem`. This PR adds `org.apache.hadoop.` to `ConnectorPluginManager.CONNECTOR_PARENT_FIRST_PREFIXES`, i.e. **no connector pom or plugin zip changes**. Parent-first here is a delegation *order*, not an exclusion: anything the kernel lacks still falls back to the plugin's own jars, so `org.apache.hadoop.hbase.*` (hudi) and `org.apache.hadoop.hive.*` still come from the plugin. Every hadoop-namespace artifact the four plugins bundle resolves to the identical GAV as the kernel's (`hadoop-common/auth/annotations/hdfs-client/aws` 3.4.2, `hadoop-shaded-guava` 1.2.0, `hadoop-shaded-protobuf_3_25` 1.5.0, and paimon's `hadoop-huaweicloud` 3.1.1-hw-46, which fe-core also carries via `${huaweiobs.version}`). **FE fingerprint injection (storage layer only):** - `FsCacheKeys` (fe-filesystem-api) computes a stable 32-hex SHA-256 fingerprint from the concrete class name (as salt) plus a sorted identity map, exposed as `FileSystemProperties.fsCacheFingerprint()` and published under `doris.fs.cache.key.<scheme>` for every scheme the storage serves. - **The identity is the effective credential set, not just the typed one.** It is `matchedProperties()` (the user-supplied storage definition, credentials included) plus every raw property that reaches the effective `Configuration` without being a declared `@ConnectorProperty` alias: - `fs.` / `dfs.` / `hadoop.` / `juicefs.` — overlaid verbatim *after* the storage's own map by `IcebergCatalogFactory.buildHadoopConfiguration`, `HudiScanPlanProvider.buildHadoopConf` and `HdfsProperties.extractUserOverriddenHdfsConfig`; - `paimon.s3.` / `paimon.s3a.` / `paimon.fs.s3.` / `paimon.fs.oss.` — *re-keyed* onto `fs.s3a.` by `MetaStoreParseUtils.applyStorageConfig` (every HMS-backed catalog) and `PaimonCatalogFactory`, so they never appear verbatim yet decide the effective S3A credentials. - Every key, value and the salt are **length-prefix framed** before hashing (`len ':' value`, `null` → `-1:`), so the encoding is injective: a value containing newlines or `=` cannot be smuggled to forge another definition's fingerprint. - The scheme set is `getSupportedSchemes()` ∪ `legacyCacheSchemes()`. Neither alone is enough: the legacy set is wrong for the S3-compatible dialects (COS declares `{cos, cosn}`, but Doris normalizes `cos://` to `s3://` and the FileSystem is opened as `s3a`, so a fingerprint published only under `cos` would never be read), while `getSupportedSchemes()` is empty for providers with no scheme identity. - `StorageAdapter` publishes it into `getBackendConfigProperties()` (now a defensive copy) and into `hadoopStorageConfig`; the SPI providers publish it into their own hadoop/backend maps in place of the removed disable-cache keys. **Connector code is untouched.** - REST vended credentials need no special handling either: a vended overlay replaces the `...key.<scheme>` of the scheme it actually re-credentials and leaves the other storages' entries intact. - The blanket `fs.<schema>.impl.disable.cache=true` defaults are removed; explicit user-provided values are still honored as an escape hatch. The HDFS plugin's `HdfsConfigBuilder` keeps its per-instance disable: it exists for manual lifecycle management (`DFSFileSystem.close`), not credential isolation. `OutFileClause` no longer mutates the returned backend map (it is a copy now); it injects `fs.defaultFS` into the source properties and rebuilds the descriptor. **Notes:** - Rolling upgrade: upgrade BE before FE (standard order). Old FE + new BE stays safe because old FE still sends the disable flags; new FE + old BE would re-expose the vanilla cache. - `FileSystem.CACHE` has no eviction, so rotating vended credentials accumulate entries over time — still strictly better than today's new-instance-per-access behavior. ### Release note Hadoop FileSystem cache is now enabled and keyed by a per-storage credential fingerprint (`doris.fs.cache.key`) instead of being disabled globally; explicit `fs.<schema>.impl.disable.cache` settings are still honored. ### Check List (For Author) - Test - [x] Unit test: - `DorisFileSystemCacheKeyTest` (hadoop-deps, 8) — patched class loaded, vanilla default behavior, distinct keys isolate, equal keys share, per-scheme key beats the generic one, generic key still honored without a per-scheme entry, an unrelated scheme's key is ignored, service scan ignores the thread context classloader - `FsCacheKeysTest` (fe-filesystem-api, 7) — fingerprint stable and order-independent, embedded separators cannot forge another property set, key/value boundary unambiguous, `null` distinct from empty, salt framed too, documented shape, per-scheme lower-cased property name - `StorageAdapterFsCacheFingerprintTest` (fe-core, 13) — stability / credential sensitivity, raw `fs.*` overlays changing the HDFS *and* S3 fingerprints, connector re-keyed `paimon.*` credentials changing the S3 fingerprint, smuggled separators, unrelated properties ignored, both injection channels, per-scheme publication (incl. S3 dialects published under `s3a`), a two-storage `putAll` merge keeping both fingerprints independent, defensive copy, no blanket disable + explicit override honored, patched `Cache.Key` on the FE classpath - `ConnectorPluginHadoopPatchTest` (fe-core, 3) — a connector plugin resolves the patched `FileSystem` from the kernel; without the `org.apache.hadoop.` prefix it gets its own unpatched copy; parent-first still falls back to the plugin when the kernel has nothing to offer - `OutFileTest`, `HdfsPropertiesTest`, `AzureFileSystemPropertiesTest` extended; updated HDFS / OSS-HDFS / JFS SPI parity goldens - [ ] Regression test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. - Behavior changed: - [ ] No. - [x] Yes. Hadoop FileSystem caching is enabled again with credential-aware keys; blanket cache disabling removed. Connector plugins now resolve `org.apache.hadoop.*` from the FE kernel rather than child-first. - Does this need documentation? - [ ] No. - [x] Yes. `doris.fs.cache.key` is a reserved property; upgrade order note (BE before FE). ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label