mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
hui lai
b8dc326111
### What problem does this PR solve? GCS access currently relies on HMAC keys or unsigned requests. This PR adds native GCP IAM authentication across FE, BE, and Cloud, allowing Doris to access GCS using Application Default Credentials (ADC), a Compute Engine service account, or an impersonated service account. ### Authentication and configuration ```properties provider=GCP gs.credential_provider_type=DEFAULT # Optional: access GCS as this service account gs.impersonation_service_account=target@project.iam.gserviceaccount.com ``` - `DEFAULT` uses Google's Application Default Credentials chain. - `COMPUTE_ENGINE` uses the service account attached to the VM. - `gs.impersonation_service_account` optionally selects a target service account using the source identity's impersonation permissions. - GCP configurations without HMAC credentials or an explicit authentication mode default to ADC. Public-bucket access requires explicit `ANONYMOUS`. Existing GCS HMAC authentication remains supported. Native IAM authentication rejects conflicting HMAC or AWS credential properties and requires a supported Google Storage HTTPS service endpoint. ### How it works - **FE:** The GCS filesystem plugin resolves authentication properties and creates refreshable Google OAuth credentials. Object operations use bearer-authenticated GCS XML API requests; namespace operations use the Hadoop GCS connector. The Iceberg connector applies the same GCP identity to GCS storage access. - **BE and Cloud:** Typed GCP identity settings are carried through Thrift, Protobuf, and Cloud object metadata. The C++ storage client uses google-cloud-cpp OAuth credentials to authenticate GCS requests and refresh access tokens. - **Resources and storage vaults:** Persist and propagate the authentication mode and optional impersonation target. Partial credential updates preserve omitted settings and allow impersonation to be cleared explicitly. Anonymous GCS storage vaults are rejected. - **Data and error-log access:** Integrate IAM authentication with GCS-backed catalogs, TVFs, load, export, outfile, and backup/restore. EXPORT and OUTFILE accept `gs://` paths. Backend error-log downloads use GCS V4 URLs signed through IAM `signBlob`. FE/Iceberg package the Hadoop GCS connector; BE/Cloud package the google-cloud-cpp OAuth libraries. The participating FE, BE, and MetaService processes must support the new credential fields before native IAM is enabled. Native IAM support for Paimon/Hudi and FE presigned uploads is outside this PR's scope. ### Release note Support native GCP IAM authentication for GCS using ADC, Compute Engine service accounts, and optional service-account impersonation. GCP configurations without HMAC credentials default to ADC; anonymous access must be selected explicitly.
56 lines
2.2 KiB
XML
56 lines
2.2 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!--
|
|
Licensed to the Apache Software Foundation (ASF) under one
|
|
or more contributor license agreements. See the NOTICE file
|
|
distributed with this work for additional information
|
|
regarding copyright ownership. The ASF licenses this file
|
|
to you under the Apache License, Version 2.0 (the
|
|
"License"); you may not use this file except in compliance
|
|
with the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing,
|
|
software distributed under the License is distributed on an
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations
|
|
under the License.
|
|
-->
|
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
|
<modelVersion>4.0.0</modelVersion>
|
|
<parent>
|
|
<groupId>org.apache.doris</groupId>
|
|
<artifactId>fe</artifactId>
|
|
<version>${revision}</version>
|
|
<relativePath>../pom.xml</relativePath>
|
|
</parent>
|
|
<artifactId>fe-storage-protocol</artifactId>
|
|
<name>Doris object storage protocol adapters</name>
|
|
<description>Host-side translation of typed filesystem identities into Doris RPC messages.</description>
|
|
<dependencies>
|
|
<dependency>
|
|
<groupId>org.apache.doris</groupId>
|
|
<artifactId>fe-filesystem-api</artifactId>
|
|
<version>${revision}</version>
|
|
</dependency>
|
|
<dependency>
|
|
<groupId>org.apache.doris</groupId>
|
|
<artifactId>fe-grpc</artifactId>
|
|
<version>${revision}</version>
|
|
</dependency>
|
|
<dependency>
|
|
<groupId>org.apache.doris</groupId>
|
|
<artifactId>fe-thrift</artifactId>
|
|
<version>${revision}</version>
|
|
</dependency>
|
|
<dependency>
|
|
<groupId>org.junit.jupiter</groupId>
|
|
<artifactId>junit-jupiter</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
</dependencies>
|
|
</project>
|