mirror of
https://github.com/apache/doris.git
synced 2026-11-04 16:03:23 +00:00
hui lai
b8dc326111
### What problem does this PR solve? GCS access currently relies on HMAC keys or unsigned requests. This PR adds native GCP IAM authentication across FE, BE, and Cloud, allowing Doris to access GCS using Application Default Credentials (ADC), a Compute Engine service account, or an impersonated service account. ### Authentication and configuration ```properties provider=GCP gs.credential_provider_type=DEFAULT # Optional: access GCS as this service account gs.impersonation_service_account=target@project.iam.gserviceaccount.com ``` - `DEFAULT` uses Google's Application Default Credentials chain. - `COMPUTE_ENGINE` uses the service account attached to the VM. - `gs.impersonation_service_account` optionally selects a target service account using the source identity's impersonation permissions. - GCP configurations without HMAC credentials or an explicit authentication mode default to ADC. Public-bucket access requires explicit `ANONYMOUS`. Existing GCS HMAC authentication remains supported. Native IAM authentication rejects conflicting HMAC or AWS credential properties and requires a supported Google Storage HTTPS service endpoint. ### How it works - **FE:** The GCS filesystem plugin resolves authentication properties and creates refreshable Google OAuth credentials. Object operations use bearer-authenticated GCS XML API requests; namespace operations use the Hadoop GCS connector. The Iceberg connector applies the same GCP identity to GCS storage access. - **BE and Cloud:** Typed GCP identity settings are carried through Thrift, Protobuf, and Cloud object metadata. The C++ storage client uses google-cloud-cpp OAuth credentials to authenticate GCS requests and refresh access tokens. - **Resources and storage vaults:** Persist and propagate the authentication mode and optional impersonation target. Partial credential updates preserve omitted settings and allow impersonation to be cleared explicitly. Anonymous GCS storage vaults are rejected. - **Data and error-log access:** Integrate IAM authentication with GCS-backed catalogs, TVFs, load, export, outfile, and backup/restore. EXPORT and OUTFILE accept `gs://` paths. Backend error-log downloads use GCS V4 URLs signed through IAM `signBlob`. FE/Iceberg package the Hadoop GCS connector; BE/Cloud package the google-cloud-cpp OAuth libraries. The participating FE, BE, and MetaService processes must support the new credential fields before native IAM is enabled. Native IAM support for Paimon/Hudi and FE presigned uploads is outside this PR's scope. ### Release note Support native GCP IAM authentication for GCS using ADC, Compute Engine service accounts, and optional service-account impersonation. GCP configurations without HMAC credentials default to ADC; anonymous access must be selected explicitly.