Problem Summary:
S3 endpoint scheme handling was inconsistent across FE, BE, and Cloud,
and the previous implementation distinguished Doris internal buckets
from user-configured external buckets. That classification leaked into
S3Conf construction and client-cache identity even though the desired
transport rule is the same for every AWS/S3-compatible endpoint.
This PR applies one rule across S3 client construction and
Resource/Storage Vault persistence boundaries:
- Keep the existing FE, BE, and Cloud s3_client_http_scheme
compatibility setting.
- Default every endpoint without an explicit scheme to HTTPS.
- If s3_client_http_scheme is explicitly set to http, use HTTP for
endpoints without a scheme.
- Preserve endpoints that already contain an explicit http:// or
https:// scheme.
- Remove the internal/external bucket classification from BE and Cloud
S3 configuration.
- Apply the same behavior to FE native S3 clients, the S3 filesystem
plugin, connectivity checks, stages, resources, and Iceberg S3 FileIO
handoffs.
- Normalize scheme-less S3 Resource and Storage Vault endpoint metadata
before persistence using the same configured default.
Azure is intentionally unchanged in this PR. Its existing
provider-specific behavior already defaults a bare endpoint to HTTPS and
preserves an explicit scheme; whether Azure should share the S3
compatibility setting is a separate policy decision.
### Release note
S3 endpoints without an explicit scheme now use HTTPS by default across
FE, BE, and Cloud. Set s3_client_http_scheme = http for compatibility
with scheme-less HTTP endpoints. Explicit HTTP and HTTPS endpoint
schemes are unchanged.
Previously the procedure of recycler instance is single-threaded, which
is not full sufficiently parallel. And there exists many network IO
operation. So this pr tries to spilt recycle tasks into different stage
which can be parallel. And make the delete operations parallel.
Gavin Chou
Refrain
abmdocrt
Pxl
Lei Zhang
AlexYue