mirror of
https://github.com/apache/doris.git
synced 2026-11-04 18:02:19 +00:00
hui lai
b8dc326111
### What problem does this PR solve? GCS access currently relies on HMAC keys or unsigned requests. This PR adds native GCP IAM authentication across FE, BE, and Cloud, allowing Doris to access GCS using Application Default Credentials (ADC), a Compute Engine service account, or an impersonated service account. ### Authentication and configuration ```properties provider=GCP gs.credential_provider_type=DEFAULT # Optional: access GCS as this service account gs.impersonation_service_account=target@project.iam.gserviceaccount.com ``` - `DEFAULT` uses Google's Application Default Credentials chain. - `COMPUTE_ENGINE` uses the service account attached to the VM. - `gs.impersonation_service_account` optionally selects a target service account using the source identity's impersonation permissions. - GCP configurations without HMAC credentials or an explicit authentication mode default to ADC. Public-bucket access requires explicit `ANONYMOUS`. Existing GCS HMAC authentication remains supported. Native IAM authentication rejects conflicting HMAC or AWS credential properties and requires a supported Google Storage HTTPS service endpoint. ### How it works - **FE:** The GCS filesystem plugin resolves authentication properties and creates refreshable Google OAuth credentials. Object operations use bearer-authenticated GCS XML API requests; namespace operations use the Hadoop GCS connector. The Iceberg connector applies the same GCP identity to GCS storage access. - **BE and Cloud:** Typed GCP identity settings are carried through Thrift, Protobuf, and Cloud object metadata. The C++ storage client uses google-cloud-cpp OAuth credentials to authenticate GCS requests and refresh access tokens. - **Resources and storage vaults:** Persist and propagate the authentication mode and optional impersonation target. Partial credential updates preserve omitted settings and allow impersonation to be cleared explicitly. Anonymous GCS storage vaults are rejected. - **Data and error-log access:** Integrate IAM authentication with GCS-backed catalogs, TVFs, load, export, outfile, and backup/restore. EXPORT and OUTFILE accept `gs://` paths. Backend error-log downloads use GCS V4 URLs signed through IAM `signBlob`. FE/Iceberg package the Hadoop GCS connector; BE/Cloud package the google-cloud-cpp OAuth libraries. The participating FE, BE, and MetaService processes must support the new credential fields before native IAM is enabled. Native IAM support for Paimon/Hudi and FE presigned uploads is outside this PR's scope. ### Release note Support native GCP IAM authentication for GCS using ADC, Compute Engine service accounts, and optional service-account impersonation. GCP configurations without HMAC credentials default to ADC; anonymous access must be selected explicitly.
649 lines
36 KiB
Python
Executable File
649 lines
36 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
"""Check a deployed BE plugin tree against the rules the plugin isolation depends on.
|
|
|
|
Run it on a built output tree:
|
|
|
|
tools/be-java-plugins/check_plugin_layout.py [output/be/lib/jni/spi] [output/be/plugins/jni]
|
|
|
|
Six checks, four of which have caught a real regression during the plugin migration:
|
|
|
|
spi-jar-purity doris-jni-spi.jar carries only the SPI packages. Everything in that jar
|
|
is shared by BE and by every plugin at once, so a class that slips in
|
|
can never be upgraded by one plugin alone.
|
|
plugin-ships-no-spi No jar inside a plugin directory carries an SPI class. A plugin that
|
|
brings its own copy gets a second, incompatible identity for the same
|
|
type; the loader rejects it, but only when someone runs a query.
|
|
duplicate-classes Within one plugin directory, a class name that resolves to different
|
|
bytes in different jars. One directory is one flat classloader, so the
|
|
winner is jar-name order - not a decision anyone made. The one decided
|
|
case is a SHADOWING jar: one whose manifest names, in
|
|
Doris-Shadows-Classes, the classes it exists to replace (hadoop-deps,
|
|
carrying the Doris-patched org.apache.hadoop.fs.FileSystem). PluginRuntime
|
|
searches such a jar first whatever its name, so those classes - nested
|
|
ones included - may duplicate a sibling's.
|
|
shadowing-jar A jar that declares Doris-Shadows-Classes carries every class it names
|
|
and nothing else. The loader trusts the attribute for search order; this
|
|
is what keeps it honest, so a jar cannot ride to the front of the
|
|
classpath with more than the classes it declares.
|
|
closure-self-contained
|
|
Every class referenced from a plugin's ROOT jars resolves inside that
|
|
plugin directory (plus the SPI jar and the JDK). The roots are Doris's own
|
|
jars AND every jar holding a class Doris hands to hadoop as a string -
|
|
the fs.*.impl and credentials-provider values in fe/fe-filesystem/**, see
|
|
_NAMED_BY_DORIS. This is the check that fires when a dependency is
|
|
switched to `provided` and its jar stops being deployed while the code
|
|
still calls into it, and - because of that second root set - when a
|
|
bundled filesystem is missing a library of its OWN.
|
|
api-version-stamp The jar declaring the DorisPlugin service carries the
|
|
Doris-Jni-Plugin-Api-Version this build serves. That attribute is the only
|
|
thing PluginRuntime's version gate reads, and until this check existed
|
|
nothing verified it reached the artifact.
|
|
|
|
WHAT THIS DOES NOT PROVE. The last check is static, so it says nothing about anything reached by
|
|
a ServiceLoader or by a reflective lookup Doris does not make itself - JDBC drivers, logging
|
|
backends, jackson modules. A jar can disappear from a plugin directory, pass every check here, and
|
|
still fail at runtime. The only thing that proves a closure complete is loading the plugin out of
|
|
the deployed directory for real. Treat a green run as "no STATICALLY visible hole", nothing more.
|
|
|
|
BUNDLED THIRD-PARTY FILESYSTEMS used to be the sharpest edge of that and have drawn blood twice. A
|
|
plugin packages hadoop-aws or hadoop-huaweicloud for one URI scheme; hadoop reaches it by class
|
|
NAME out of a Configuration, so no Doris class references it and a walk rooted at Doris's own jars
|
|
never entered it. Its own dependencies were then invisible too - hadoop-huaweicloud calls
|
|
commons-lang 2.x from seven classes and hadoop-aws's AssumedRoleCredentialProvider calls
|
|
software.amazon.awssdk.services.sts from twenty-three, and neither jar carries a copy. Both used to
|
|
come from the shared preload classpath and now have to be declared by each plugin that bundles
|
|
them; both shipped broken before this was checked.
|
|
|
|
_NAMED_BY_DORIS closes that particular hole by making those jars roots. Note what it is NOT: it
|
|
does not follow a Class.forName and it is not reflection analysis. It is a hand-maintained list of
|
|
the class names Doris itself writes into a Configuration, and every reference it then finds is an
|
|
ordinary constant-pool entry. A filesystem Doris reaches some other way is still invisible, and so
|
|
is anything those roots reach reflectively - the per-plugin
|
|
resolvesEveryFilesystemSchemeAScanCanArriveOn tests do not close that either, because
|
|
getFileSystemClass returns a Class without linking it.
|
|
|
|
The shared filesystem directory (plugins/jni_fs, appended to every plugin classloader for
|
|
jindofs/juicefs) is outside this check entirely: it is not a plugin directory, so nothing here
|
|
looks at it - not the duplicate scan, and not the closure walk, which will happily report a class
|
|
as missing that a BE resolves out of there at runtime.
|
|
|
|
Exit status: 0 if every check passes, 1 if any fails, 2 if the tree or the tools are unusable.
|
|
"""
|
|
import collections
|
|
import functools
|
|
import os
|
|
import subprocess
|
|
import sys
|
|
import zipfile
|
|
|
|
SPI_PREFIX = "org/apache/doris/jni/spi/"
|
|
DORIS_PREFIX = "org/apache/doris/"
|
|
|
|
_OBS = ("Two builds of the Huawei OBS SDK: the standalone esdk-obs-java-optimised and the copy "
|
|
"shaded into the hadoop-huaweicloud fat jar. Both are present on BE's system classpath "
|
|
"today with the same overlap, and PluginRuntime sorts jar URLs, so the copy that wins "
|
|
"here is the copy that wins today.")
|
|
|
|
# Classes allowed to appear with DIFFERING bytes in two jars of the same plugin directory.
|
|
# Adding an entry means claiming the arbitrary winner is harmless - write down why.
|
|
# Match kinds: "basename" compares the last path segment, "prefix" compares the whole path.
|
|
DUPLICATE_ALLOWLIST = {
|
|
"*": [
|
|
("basename", "module-info.class",
|
|
"A plugin directory is a classpath, not a module path, so module descriptors are never "
|
|
"read. Multi-release jars additionally carry one per version directory."),
|
|
("basename", "package-info.class",
|
|
"Holds package annotations and no code. The copies that collide here are hadoop's, "
|
|
"which carry only its InterfaceAudience/InterfaceStability documentation annotations, "
|
|
"so which one wins changes nothing that executes."),
|
|
],
|
|
# No JindoFS entries any more. This build no longer copies the JindoFS jars into the iceberg
|
|
# and paimon directories - plugins/jni_fs is appended to every plugin classloader instead - so
|
|
# the ten class paths those copies used to collide on (org/apache/hadoop/fs/StreamCapabilities,
|
|
# PositionedReadable, four javax/annotation Checker types and their siblings) are back under
|
|
# this check. A real collision on any of them must fail the build rather than be waved through
|
|
# by an exemption whose reason no longer exists.
|
|
"iceberg": [("prefix", "com/obs/", _OBS), ("prefix", "com/oef/", _OBS)],
|
|
"paimon": [("prefix", "com/obs/", _OBS), ("prefix", "com/oef/", _OBS)],
|
|
"hudi": [("prefix", "com/obs/", _OBS), ("prefix", "com/oef/", _OBS)],
|
|
}
|
|
|
|
# Classes that Doris's own code in a plugin references but that the plugin deliberately does not
|
|
# ship. Every entry is a claim that the referencing code is never executed in BE - the class is
|
|
# in the jar because it came with a shared Doris artifact, not because a plugin path reaches it.
|
|
# Prefix match on the unresolved class name.
|
|
_TRIMMED = ("Reached only from {} inside a shared Doris jar that the plugin needs for other "
|
|
"reasons. {} was excluded from this closure on purpose; the referencing code is not "
|
|
"on any BE path, so the class is never loaded and the missing library never resolves.")
|
|
_FASTUTIL = _TRIMMED.format(
|
|
"fe-foundation's ConcurrentLong2LongHashMap / ConcurrentLong2ObjectHashMap", "fastutil")
|
|
# Holes INSIDE the bundled cloud filesystems, surfaced by the widened root set (_NAMED_BY_DORIS).
|
|
# These are not Doris's code and not Doris's choices; each entry says why the reference is never
|
|
# linked on a BE path, and every one of them was checked with javap against the deployed jar
|
|
# rather than assumed. Adding hadoop-aws or hadoop-huaweicloud to a plugin brings the whole list.
|
|
_S3A_CSE = ("hadoop-aws's EncryptionS3ClientFactory, the S3 client-side-encryption path "
|
|
"(fs.s3a.encryption.algorithm=CSE-KMS). It needs the Amazon S3 Encryption Client "
|
|
"(software.amazon.encryption.s3), which is not a transitive dependency of anything "
|
|
"here and was not on the shared preload classpath this plugin layout replaced either "
|
|
"- CSE-KMS therefore did not work before this change and does not now. Shipping the "
|
|
"kms module alone would not fix it. Doris never writes that property.")
|
|
_SDK_BUNDLE = ("the AWS SDK's own shaded HTTP client, which ships only in its `bundle` uber-jar; "
|
|
"the plugins deploy the modular `apache-client` instead. The two reference sites "
|
|
"are STSClientFactory.getSTSEndpoint (private, called only when "
|
|
"fs.s3a.assumed.role.sts.endpoint is set - Doris never writes it) and "
|
|
"ConfigureShadedAWSSocketFactory (reached only for fs.s3a.ssl.channel.mode values "
|
|
"naming the shaded OpenSSL path, which Doris never writes either).")
|
|
_SDK_V1 = ("hadoop-aws's V1ToV2AwsCredentialProviderAdapter, the shim for credential providers "
|
|
"written against AWS SDK v1. It is instantiated only for a provider CLASS NAME under "
|
|
"com.amazonaws.*, and every provider Doris names is either hadoop's own or an SDK v2 "
|
|
"one (see _NAMED_BY_DORIS). No plugin ships SDK v1 at all.")
|
|
_LOG4J1 = ("hadoop-aws's Log4JController, an optional log4j 1.x bridge that LogControllerFactory "
|
|
"loads reflectively and skips when it is absent. Doris runs log4j2.")
|
|
_OKHTTP_PLATFORM = ("the OBS SDK's shaded okhttp, whose Platform.findPlatform() probes for an "
|
|
"Android or Conscrypt runtime and catches ClassNotFoundException. Neither is "
|
|
"present in a BE, which is what the probe is for.")
|
|
_OBS_BASE64 = ("the OBS SDK's shaded XML builder, which prefers net.iharder.Base64 when it is on "
|
|
"the classpath and falls back to its own encoder when it is not.")
|
|
_GUAVA_CHECKED_FUTURE = ("Futures.immediateFailedCheckedFuture, removed in Guava 26; "
|
|
"hadoop-huaweicloud 3.1.1-hw-46 was built against an older one. Reached "
|
|
"only from SemaphoredDelegatingExecutor.submit()'s InterruptedException "
|
|
"branch. Pre-existing and byte-identical to what BE's system classpath "
|
|
"carried before this layout - no plugin can fix it without shipping a "
|
|
"Guava that the rest of the tree has moved past.")
|
|
_S3A_COMMITTER = ("hadoop-aws's S3A output committers, which extend hadoop-mapreduce-client-core "
|
|
"types. BE reads table formats; it never runs a MapReduce job, and nothing "
|
|
"here resolves a committer. NOT in _BUNDLED_S3A on purpose: the prefix is far "
|
|
"wider than the committers and covers "
|
|
"org.apache.hadoop.mapreduce.lib.input.FileInputFormat, which paimon and hudi "
|
|
"DO reach - see the per-plugin note in CLOSURE_ALLOWLIST.")
|
|
_BUNDLED_S3A = [
|
|
("software.amazon.encryption.s3.", _S3A_CSE),
|
|
("software.amazon.awssdk.services.kms.", _S3A_CSE),
|
|
("software.amazon.awssdk.thirdparty.", _SDK_BUNDLE),
|
|
("com.amazonaws.", _SDK_V1),
|
|
("org.apache.log4j.", _LOG4J1),
|
|
]
|
|
_BUNDLED_OBS = [
|
|
("android.", _OKHTTP_PLATFORM),
|
|
("org.conscrypt.", _OKHTTP_PLATFORM),
|
|
("net.iharder.", _OBS_BASE64),
|
|
("com.google.common.util.concurrent.CheckedFuture", _GUAVA_CHECKED_FUTURE),
|
|
]
|
|
CLOSURE_ALLOWLIST = {
|
|
# The mapreduce exemption is iceberg's ALONE. iceberg-metadata-scanner is the only plugin that
|
|
# bundles hadoop-aws WITHOUT hadoop-mapreduce-client-core, so it is the only one where the S3A
|
|
# committers' mapreduce references are genuinely unresolvable. paimon-scanner and
|
|
# hadoop-hudi-scanner declare that artifact on purpose (commit d50394e645d: every parquet read
|
|
# from a paimon catalog failed with NoClassDefFoundError on
|
|
# org.apache.hadoop.mapreduce.lib.input.FileInputFormat, which this prefix would wave through),
|
|
# so on those two the prefix stays guarded and dropping the jar again fails the build.
|
|
"iceberg": [("it.unimi.dsi.fastutil.", _FASTUTIL),
|
|
("org.apache.hadoop.mapreduce.", _S3A_COMMITTER)] + _BUNDLED_S3A + _BUNDLED_OBS,
|
|
"paimon": [("it.unimi.dsi.fastutil.", _FASTUTIL)] + _BUNDLED_S3A + _BUNDLED_OBS,
|
|
"hudi": [("it.unimi.dsi.fastutil.", _FASTUTIL)] + _BUNDLED_S3A + _BUNDLED_OBS,
|
|
"java-udf": [
|
|
("it.unimi.dsi.fastutil.", _FASTUTIL),
|
|
("org.roaringbitmap.", _TRIMMED.format("fe-common's org.apache.doris.common.io codecs",
|
|
"RoaringBitmap")),
|
|
("com.fasterxml.jackson.", _TRIMMED.format("fe-common's org.apache.doris.common.util",
|
|
"jackson")),
|
|
("com.google.protobuf.", _TRIMMED.format("fe-common's org.apache.doris.persist.gson",
|
|
"protobuf")),
|
|
("io.netty.", _TRIMMED.format("fe-common's org.apache.doris.common", "netty")),
|
|
("org.apache.commons.io.", _TRIMMED.format("fe-common's org.apache.doris.common",
|
|
"commons-io")),
|
|
("org.apache.commons.codec.", _TRIMMED.format("fe-common's org.apache.doris.common.io",
|
|
"commons-codec")),
|
|
("org.apache.logging.log4j.core.", _TRIMMED.format(
|
|
"fe-common's log4j plugin classes", "log4j-core")),
|
|
],
|
|
}
|
|
|
|
|
|
# gcs-connector 3.1.18 is a shaded uber-jar. Making its Hadoop filesystem a root
|
|
# also scans optional transports, test helpers and native-image substitutions.
|
|
# Match BOTH ends of these edges: a missing library referenced by Doris or the
|
|
# GCS filesystem/authentication implementation must still fail the check.
|
|
_GCS = "com.google.cloud.hadoop.repackaged.gcs."
|
|
_GCS_NETTY = _GCS + "io.grpc.netty.shaded.io.netty."
|
|
_GCS_CODECS = ("Optional Netty codecs bundled in the GCS jar. GCS uses HTTP/gRPC, not "
|
|
"SPDY, JBoss marshalling or protobuf-nano. Its gzip/deflate path uses "
|
|
"JDK zlib; Brotli and Zstd HTTP handlers probe isAvailable() before use.")
|
|
_GCS_ANNOTATIONS = ("Compile-time annotations in the shaded GCS jar; the generated "
|
|
"AutoValue implementations are already bundled.")
|
|
# Each tuple is (referencing class prefix, missing class prefix, reason).
|
|
GCS_CLOSURE_ALLOWLIST = [
|
|
(_GCS + "com.google.api.client.extensions.appengine.http.", "com.google.appengine.api.urlfetch.",
|
|
"App Engine's UrlFetch transport; BE uses the normal JVM HTTP transport."),
|
|
("com.google.cloud.hadoop.", "com.google.auto.value.AutoValue", _GCS_ANNOTATIONS),
|
|
(_GCS + "com.google.common.flogger.", "org.checkerframework.checker.nullness.compatqual.",
|
|
_GCS_ANNOTATIONS),
|
|
(_GCS_NETTY + "util.concurrent.", "org.jetbrains.annotations.Async", _GCS_ANNOTATIONS),
|
|
(_GCS_NETTY + "util.", "com.oracle.svm.core.annotate.",
|
|
"Graal native-image substitutions are not used by BE's JVM."),
|
|
(_GCS_NETTY + "util.internal.Hidden$NettyBlockHoundIntegration", "reactor.blockhound.",
|
|
"Optional BlockHound integration; BE does not install a BlockHound agent."),
|
|
(_GCS_NETTY + "handler.ssl.", "org.bouncycastle.",
|
|
"Optional PEM/self-signed certificate helpers. GCS uses trust-store TLS; "
|
|
"BouncyCastlePemReader checks availability and JDK PEM parsing is the fallback."),
|
|
(_GCS_NETTY + "handler.ssl.JettyAlpnSslEngine", "org.eclipse.jetty.alpn.",
|
|
"Legacy Jetty ALPN for old JDKs; BE requires JDK 17 and uses JDK ALPN."),
|
|
(_GCS_NETTY + "handler.ssl.JettyNpnSslEngine", "org.eclipse.jetty.npn.",
|
|
"Legacy NPN transport; GCS uses HTTP/2 ALPN on JDK 17."),
|
|
(_GCS + "io.grpc.testing.", "org.junit.",
|
|
"Upstream GrpcCleanupRule/GrpcServerRule test helpers bundled in the shaded runtime jar."),
|
|
(_GCS + "io.opentelemetry.sdk.", _GCS + "io.opentelemetry.api.incubator.",
|
|
"SdkTracer, SdkLogger and instrument builders probe incubator availability "
|
|
"with Class.forName and use the stable implementation when it is absent."),
|
|
("com.google.cloud.hadoop.fs.gcs.auth.GcsDtFetcher",
|
|
"com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem",
|
|
"Unused pre-shading constant-pool entry in 3.1.18: javap shows no instruction "
|
|
"or signature using it. The actual filesystem implementation is relocated."),
|
|
] + [(_GCS_NETTY + "handler.codec.", target, _GCS_CODECS) for target in (
|
|
"com.aayushatharva.brotli4j.", "com.github.luben.zstd.", "com.jcraft.jzlib.",
|
|
"com.ning.compress.", "lzma.sdk.", "net.jpountz.lz4.", "net.jpountz.xxhash.",
|
|
"org.jboss.marshalling.", _GCS + "com.google.protobuf.nano.",
|
|
)]
|
|
|
|
|
|
def _gcs_optional_reference(plugin, source, target):
|
|
return plugin in ("hudi", "iceberg", "paimon") and any(
|
|
source.startswith(src) and target.startswith(dst)
|
|
for src, dst, _reason in GCS_CLOSURE_ALLOWLIST)
|
|
|
|
|
|
def _allowed(rules, path):
|
|
for kind, pattern, _reason in rules:
|
|
if kind == "basename" and path.rsplit("/", 1)[-1] == pattern:
|
|
return True
|
|
if kind == "prefix" and path.startswith(pattern):
|
|
return True
|
|
return False
|
|
|
|
|
|
def _class_entries(jar):
|
|
with zipfile.ZipFile(jar) as zf:
|
|
return [(i.filename, i.CRC) for i in zf.infolist() if i.filename.endswith(".class")]
|
|
|
|
|
|
# Memoized because the same jar is read by four of the five checks - the duplicate scan, the
|
|
# sole-provider scan, the api-version scan and the root selection below - and a lake-format
|
|
# plugin directory holds a hundred of them, one of which is a 180 MB fat jar. A tuple rather than
|
|
# the list zipfile hands back, so a caller cannot mutate what the next one reads.
|
|
@functools.lru_cache(maxsize=None)
|
|
def _names(jar):
|
|
with zipfile.ZipFile(jar) as zf:
|
|
return tuple(zf.namelist())
|
|
|
|
|
|
def check_spi_jar_purity(spi_dir, fail):
|
|
jar = os.path.join(spi_dir, "doris-jni-spi.jar")
|
|
# Both jars, because the shared layer is both: without doris-jni-bootstrap.jar there is no
|
|
# loader at all and every Java feature fails at runtime with a FindClass error naming none of
|
|
# this. Only the SPI jar's contents are checked below - the loader may carry whatever it needs.
|
|
loader = os.path.join(spi_dir, "doris-jni-bootstrap.jar")
|
|
if not os.path.isfile(loader):
|
|
fail("spi-jar-purity",
|
|
"%s is missing; without the loader no plugin can be loaded at all" % loader)
|
|
if not os.path.isfile(jar):
|
|
fail("spi-jar-purity", "%s is missing; the shared layer was not deployed" % jar)
|
|
return
|
|
stray = [n for n in _names(jar)
|
|
if not n.endswith("/")
|
|
and not n.startswith(SPI_PREFIX)
|
|
and not n.startswith("META-INF/")]
|
|
for n in sorted(stray):
|
|
fail("spi-jar-purity",
|
|
"doris-jni-spi.jar contains %s. Only %s and META-INF/ may be in this jar - it is "
|
|
"loaded once and shared by BE and every plugin." % (n, SPI_PREFIX))
|
|
|
|
|
|
def check_plugin_ships_no_spi(plugin, jars, fail):
|
|
for jar in jars:
|
|
carried = [n for n in _names(jar) if n.startswith(SPI_PREFIX) and n.endswith(".class")]
|
|
if carried:
|
|
fail("plugin-ships-no-spi",
|
|
"plugin '%s': %s carries %d SPI class(es), e.g. %s. Declare jni-spi as "
|
|
"<scope>provided</scope> so the SPI comes from lib/jni/spi instead."
|
|
% (plugin, os.path.basename(jar), len(carried), carried[0]))
|
|
|
|
|
|
SHADOWS_CLASSES_ATTRIBUTE = "Doris-Shadows-Classes"
|
|
|
|
|
|
def _shadowed(cls_paths, name):
|
|
"""Whether class-file entry `name` is one of `cls_paths` or nested inside one of them."""
|
|
for cls in cls_paths:
|
|
if name == cls + ".class" or name.startswith(cls + "$"):
|
|
return True
|
|
return False
|
|
|
|
|
|
def shadowed_classes(plugin, jars, fail):
|
|
"""
|
|
The class paths (org/apache/hadoop/fs/FileSystem, without .class) that the shadowing jars of
|
|
this directory declare, after holding each jar to its declaration. PluginRuntime moves a jar
|
|
carrying the attribute to the front of the classpath on the attribute's presence alone, so
|
|
this is the only place the VALUE is checked: every named class must be in the jar (a
|
|
declaration that shadows nothing is a build mistake, most likely a rename), and the jar must
|
|
hold nothing beyond the named classes and their nested classes (anything else would shadow a
|
|
sibling silently, which is the accident duplicate-classes exists to catch).
|
|
"""
|
|
shadowed = []
|
|
for jar in jars:
|
|
declared = _manifest_attribute(jar, SHADOWS_CLASSES_ATTRIBUTE)
|
|
if declared is None:
|
|
continue
|
|
cls_paths = [c.strip().replace(".", "/") for c in declared.split(",") if c.strip()]
|
|
entries = [n for n in _names(jar) if n.endswith(".class")]
|
|
for cls in cls_paths:
|
|
if cls + ".class" not in entries:
|
|
fail("shadowing-jar",
|
|
"plugin '%s': %s declares %s: %s but carries no %s.class; the attribute "
|
|
"names classes the jar replaces, and this one is not in it."
|
|
% (plugin, os.path.basename(jar), SHADOWS_CLASSES_ATTRIBUTE, declared, cls))
|
|
stray = sorted(n for n in entries if not _shadowed(cls_paths, n))
|
|
if stray:
|
|
fail("shadowing-jar",
|
|
"plugin '%s': %s declares %s: %s but also carries %d undeclared class(es), e.g. %s. "
|
|
"A shadowing jar is searched before every other jar in the directory, so it may hold "
|
|
"only the classes it declares (and their nested classes)."
|
|
% (plugin, os.path.basename(jar), SHADOWS_CLASSES_ATTRIBUTE, declared, len(stray),
|
|
stray[0]))
|
|
shadowed.extend(cls_paths)
|
|
return shadowed
|
|
|
|
|
|
def check_duplicate_classes(plugin, jars, fail):
|
|
rules = DUPLICATE_ALLOWLIST.get("*", []) + DUPLICATE_ALLOWLIST.get(plugin, [])
|
|
shadowed = shadowed_classes(plugin, jars, fail)
|
|
by_name = collections.defaultdict(lambda: collections.defaultdict(list))
|
|
for jar in jars:
|
|
for name, crc in _class_entries(jar):
|
|
by_name[name][crc].append(os.path.basename(jar))
|
|
for name in sorted(by_name):
|
|
by_crc = by_name[name]
|
|
if len(by_crc) < 2 or _allowed(rules, name) or _shadowed(shadowed, name):
|
|
continue
|
|
where = "; ".join("%08x in %s" % (crc, ", ".join(sorted(set(js))))
|
|
for crc, js in sorted(by_crc.items()))
|
|
fail("duplicate-classes",
|
|
"plugin '%s': %s resolves to different bytes depending on jar order (%s). Keep one "
|
|
"copy, or add it to DUPLICATE_ALLOWLIST with the reason the winner does not matter."
|
|
% (plugin, name, where))
|
|
|
|
|
|
SERVICE_ENTRY = "META-INF/services/org.apache.doris.jni.spi.DorisPlugin"
|
|
API_VERSION_ATTRIBUTE = "Doris-Jni-Plugin-Api-Version"
|
|
API_VERSION_RESOURCE = "META-INF/doris/jni-plugin-api-version.properties"
|
|
|
|
|
|
def _manifest_attribute(jar, attribute):
|
|
with zipfile.ZipFile(jar) as zf:
|
|
try:
|
|
manifest = zf.read("META-INF/MANIFEST.MF").decode("utf-8", "replace")
|
|
except KeyError:
|
|
return None
|
|
# Manifest continuation lines start with a single space; unfold before matching.
|
|
manifest = manifest.replace("\r\n", "\n").replace("\r", "\n").replace("\n ", "")
|
|
for line in manifest.split("\n"):
|
|
name, sep, value = line.partition(":")
|
|
if sep and name.strip() == attribute:
|
|
return value.strip()
|
|
return None
|
|
|
|
|
|
def served_api_version(spi_jar):
|
|
"""The major.minor this build serves, out of the resource maven filtering writes."""
|
|
with zipfile.ZipFile(spi_jar) as zf:
|
|
try:
|
|
text = zf.read(API_VERSION_RESOURCE).decode("utf-8", "replace")
|
|
except KeyError:
|
|
return None
|
|
for line in text.splitlines():
|
|
key, sep, value = line.partition("=")
|
|
if sep and key.strip() == "api.version":
|
|
return value.strip()
|
|
return None
|
|
|
|
|
|
def check_api_version_stamp(plugin, jars, served, fail):
|
|
"""
|
|
The manifest attribute PluginRuntime.checkApiVersion compares against, checked on the artifact
|
|
rather than in the pom that is supposed to produce it. A plugin jar built without the parent's
|
|
<archive> configuration - a module that overrides maven-jar-plugin, say - carries no stamp or a
|
|
stale one, and nothing else notices: the loader is the first to look, at which point the failure
|
|
is a deployment-time "built against plugin API 2.0 but this BE serves 3.0" on a jar that was
|
|
never built against 2.0 at all.
|
|
|
|
Only the jar declaring the service is FAILED on, because that is the only one the loader reads.
|
|
|
|
Every other stamped jar in the directory is reported as a warning instead. Nothing at runtime
|
|
reads those stamps, so a mismatch there is not a defect in the deployed tree - but it is a
|
|
reliable sign of a jar that was not rebuilt, which nothing else in this script can see (this
|
|
check reads every jar's entry list, which _names memoizes; the closure check does not care
|
|
when a class was compiled).
|
|
That is not hypothetical: a tree where every jar was stamped 3.0 except one left over at 2.0
|
|
passed all five checks, and the leftover was a stale build artifact, not a deliberate one.
|
|
"""
|
|
if served is None:
|
|
fail("api-version-stamp",
|
|
"the SPI jar carries no %s; the version gate has nothing to compare against"
|
|
% API_VERSION_RESOURCE)
|
|
return
|
|
_warn_stale_stamps(plugin, jars, served)
|
|
for jar in jars:
|
|
if SERVICE_ENTRY not in _names(jar):
|
|
continue
|
|
declared = _manifest_attribute(jar, API_VERSION_ATTRIBUTE)
|
|
if declared is None:
|
|
fail("api-version-stamp",
|
|
"plugin '%s': %s declares the DorisPlugin service but its manifest carries no %s. "
|
|
"It comes from <jni.plugin.api.version> in fe/be-java-extensions/pom.xml; a module "
|
|
"that redefines maven-jar-plugin's <archive> loses it."
|
|
% (plugin, os.path.basename(jar), API_VERSION_ATTRIBUTE))
|
|
elif declared != served:
|
|
fail("api-version-stamp",
|
|
"plugin '%s': %s is stamped %s=%s but this build serves %s. Rebuild the module; a "
|
|
"mismatch here is what PluginRuntime.checkApiVersion rejects at deployment."
|
|
% (plugin, os.path.basename(jar), API_VERSION_ATTRIBUTE, declared, served))
|
|
return
|
|
# No provider jar at all is the sole-provider check's verdict to give, not this one's.
|
|
|
|
|
|
def _warn_stale_stamps(plugin, jars, served):
|
|
"""Stamped jars that do NOT declare the service and disagree with the served version.
|
|
|
|
A warning, never a failure: the loader reads the service jar's stamp and nothing else, so
|
|
asserting on these would be asserting on something no runtime consults - and a plugin is
|
|
free to bundle a third-party jar that carries its own unrelated build stamps.
|
|
"""
|
|
for jar in jars:
|
|
if SERVICE_ENTRY in _names(jar):
|
|
continue
|
|
declared = _manifest_attribute(jar, API_VERSION_ATTRIBUTE)
|
|
if declared is not None and declared != served:
|
|
print(" WARN plugin '%s': %s is stamped %s=%s while this build serves %s. Nothing "
|
|
"reads that stamp, so this is not a failure - but it usually means the jar was "
|
|
"not rebuilt." % (plugin, os.path.basename(jar), API_VERSION_ATTRIBUTE,
|
|
declared, served))
|
|
|
|
|
|
def _doris_owned(jars):
|
|
owned = []
|
|
for jar in jars:
|
|
if any(n.startswith(DORIS_PREFIX) and n.endswith(".class") for n in _names(jar)):
|
|
owned.append(jar)
|
|
return owned
|
|
|
|
|
|
# Implementation classes Doris hands to hadoop AS A STRING, so that no Doris class references
|
|
# them and the walk from Doris's own jars never enters the jar they live in. Every one of these
|
|
# is written into a Configuration by fe/fe-filesystem/**; the list is short because the write
|
|
# points are - fs.*.impl and the two credentials-provider keys, nothing else.
|
|
#
|
|
# Why this matters more than it looks: hadoop-aws's AssumedRoleCredentialProvider resolves
|
|
# software.amazon.awssdk.services.sts.* from its constant pool, and hadoop-huaweicloud's
|
|
# OBSFileSystem resolves org.apache.commons.lang.*. Both are ordinary static references that jdeps
|
|
# reports the moment the jar holding them is a root - and both were real, shipped holes that every
|
|
# other check here passed over. Treating the jar as a root is not the same as trusting reflection
|
|
# analysis: nothing here follows a Class.forName, it just starts the walk somewhere Doris pointed.
|
|
#
|
|
# Entries are class file paths. A name absent from a plugin costs nothing: that plugin simply
|
|
# bundles no such filesystem, and the root set is what it was.
|
|
_NAMED_BY_DORIS = (
|
|
# Native GCP Hadoop access in the isolated Hudi, Paimon and Iceberg scanners.
|
|
"com/google/cloud/hadoop/fs/gcs/GoogleHadoopFileSystem.class",
|
|
# fs.s3.impl / fs.s3a.impl / fs.cos.impl / fs.cosn.impl / fs.gs.impl, and the fallback
|
|
# fs.obs.impl - one jar, hadoop-aws, and the S3A credential providers live in it too.
|
|
"org/apache/hadoop/fs/s3a/S3AFileSystem.class",
|
|
"org/apache/hadoop/fs/s3a/SimpleAWSCredentialsProvider.class",
|
|
"org/apache/hadoop/fs/s3a/auth/AssumedRoleCredentialProvider.class",
|
|
# fs.obs.impl and fs.AbstractFileSystem.obs.impl - hadoop-huaweicloud.
|
|
"org/apache/hadoop/fs/obs/OBSFileSystem.class",
|
|
"org/apache/hadoop/fs/obs/OBS.class",
|
|
# fs.s3a.aws.credentials.provider and fs.s3a.assumed.role.credentials.provider, when the
|
|
# catalog names a provider type rather than a key pair - the AWS SDK's own auth module.
|
|
"software/amazon/awssdk/auth/credentials/WebIdentityTokenFileCredentialsProvider.class",
|
|
"software/amazon/awssdk/auth/credentials/InstanceProfileCredentialsProvider.class",
|
|
"software/amazon/awssdk/auth/credentials/ContainerCredentialsProvider.class",
|
|
# fs.oss.impl / fs.AbstractFileSystem.oss.impl. Deployed to plugins/jni_fs rather than into a
|
|
# plugin directory, so normally absent here; listed so that a plugin which does bundle
|
|
# JindoFS gets the same treatment as the rest.
|
|
"com/aliyun/jindodata/oss/JindoOssFileSystem.class",
|
|
)
|
|
|
|
|
|
def _named_by_doris(jars):
|
|
"""Jars holding a class Doris names by string. See _NAMED_BY_DORIS."""
|
|
wanted = set(_NAMED_BY_DORIS)
|
|
return [jar for jar in jars if wanted.intersection(_names(jar))]
|
|
|
|
|
|
def check_closure_self_contained(plugin, jars, spi_jar, fail):
|
|
owned = _doris_owned(jars)
|
|
if not owned:
|
|
fail("closure-self-contained",
|
|
"plugin '%s' has no jar containing %s classes; it cannot implement the SPI."
|
|
% (plugin, DORIS_PREFIX))
|
|
return
|
|
roots = owned + [jar for jar in _named_by_doris(jars) if jar not in owned]
|
|
classpath = os.pathsep.join(jars + [spi_jar])
|
|
proc = subprocess.run(["jdeps", "--multi-release", "17", "-verbose:class", "-cp", classpath]
|
|
+ roots,
|
|
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
|
universal_newlines=True)
|
|
if proc.returncode != 0:
|
|
# Never treat this as a pass. An earlier version of this script analysed every jar in the
|
|
# directory; jdeps then built a module graph, died on a module-info requiring something
|
|
# absent, printed nothing, and the run looked clean.
|
|
fail("closure-self-contained",
|
|
"plugin '%s': jdeps exited %d, so nothing was checked.\n%s"
|
|
% (plugin, proc.returncode, proc.stderr.strip()[:600]))
|
|
return
|
|
rules = CLOSURE_ALLOWLIST.get(plugin, [])
|
|
missing = collections.defaultdict(set)
|
|
for line in proc.stdout.splitlines():
|
|
if "not found" not in line:
|
|
continue
|
|
parts = line.split("->")
|
|
if len(parts) != 2:
|
|
continue
|
|
src, target = parts[0].strip(), parts[1].strip().split()[0]
|
|
if "." not in src or "." not in target:
|
|
continue
|
|
if any(target.startswith(prefix) for prefix, _reason in rules):
|
|
continue
|
|
if _gcs_optional_reference(plugin, src, target):
|
|
continue
|
|
missing[target].add(src)
|
|
for target in sorted(missing):
|
|
fail("closure-self-contained",
|
|
"plugin '%s': %s is referenced by %s but is in no jar of this plugin directory. "
|
|
"Either ship the jar it lives in, or add the package to CLOSURE_ALLOWLIST with the "
|
|
"reason that code never runs."
|
|
% (plugin, target, sorted(missing[target])[0]))
|
|
|
|
|
|
def main(argv):
|
|
if len(argv) > 1 and argv[1] in ("-h", "--help"):
|
|
print(__doc__)
|
|
return 0
|
|
home = os.environ.get("DORIS_HOME", ".")
|
|
spi_dir = argv[1] if len(argv) > 1 else os.path.join(
|
|
home, "output", "be", "lib", "jni", "spi")
|
|
plugins_dir = argv[2] if len(argv) > 2 else os.path.join(
|
|
home, "output", "be", "plugins", "jni")
|
|
if not os.path.isdir(spi_dir):
|
|
sys.stderr.write("%s is not a deployed spi directory\n" % spi_dir)
|
|
return 2
|
|
if not os.path.isdir(plugins_dir):
|
|
sys.stderr.write("%s is not a deployed plugin family root\n" % plugins_dir)
|
|
return 2
|
|
try:
|
|
subprocess.run(["jdeps", "--version"], stdout=subprocess.DEVNULL,
|
|
stderr=subprocess.DEVNULL)
|
|
except OSError:
|
|
sys.stderr.write("jdeps not on PATH; it ships with the JDK this build already needs\n")
|
|
return 2
|
|
|
|
failures = []
|
|
|
|
def fail(check, message):
|
|
failures.append((check, message))
|
|
|
|
check_spi_jar_purity(spi_dir, fail)
|
|
spi_jar = os.path.join(spi_dir, "doris-jni-spi.jar")
|
|
served = served_api_version(spi_jar) if os.path.isfile(spi_jar) else None
|
|
plugins = sorted(d for d in os.listdir(plugins_dir)
|
|
if os.path.isdir(os.path.join(plugins_dir, d)))
|
|
if not plugins:
|
|
sys.stderr.write("no plugin directories under %s\n" % plugins_dir)
|
|
# Not "could not run" when the shared layer has already been found broken: 2 is mapped to a
|
|
# WARN by build.sh, which would report a real SPI failure as a check that did not happen.
|
|
# When there ARE failures this falls through to the reporting tail instead of returning
|
|
# here, because this is the only place they get printed and build.sh says "see above".
|
|
if not failures:
|
|
return 2
|
|
plugins = []
|
|
for plugin in plugins:
|
|
pdir = os.path.join(plugins_dir, plugin)
|
|
jars = sorted(os.path.join(pdir, f) for f in os.listdir(pdir) if f.endswith(".jar"))
|
|
if not jars:
|
|
fail("plugin-directory-nonempty", "plugin directory '%s' holds no jar" % plugin)
|
|
continue
|
|
print("checking %-18s %3d jars" % (plugin, len(jars)))
|
|
check_plugin_ships_no_spi(plugin, jars, fail)
|
|
check_duplicate_classes(plugin, jars, fail)
|
|
check_closure_self_contained(plugin, jars, spi_jar, fail)
|
|
check_api_version_stamp(plugin, jars, served, fail)
|
|
|
|
if not failures:
|
|
print("\nOK: %d plugins pass all six checks. This does NOT prove the closures are "
|
|
"complete - see the note at the top of this file." % len(plugins))
|
|
return 0
|
|
print("\n%d problem(s):" % len(failures))
|
|
for check, message in failures:
|
|
print("\n[%s] %s" % (check, message))
|
|
return 1
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main(sys.argv))
|