mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
hui lai
c71d1aec0f
### 1. What does this PR do?
BE and Cloud Recycler previously maintained separate object-storage
abstractions and separate S3/Azure implementations. Although both sides
called the same cloud-provider SDKs, credential construction, error
conversion, metrics, pagination, batch deletion, and compatibility
behavior were duplicated and could evolve differently.
This PR consolidates those implementations under `common/cpp/client`
while keeping the original decorator-oriented client abstraction:
- `ObjStorageClient` is the common interface used by upper layers and
implemented by all concrete clients and decorators.
- `S3ObjStorageClient` and `AzureObjStorageClient` implement provider
operations through the AWS and Azure SDKs.
- `RateLimitedObjStorageClient` implements the same interface and
decorates another `ObjStorageClient`.
- BE's `S3ClientFactory` and Recycler's `S3Accessor` construct the
provider client and wrap it with their own rate-limit policy.
- Shared request/response types, page-based listing, lazy iteration,
recursive deletion orchestration, credentials, metrics, and error
conversion live in the common layer.
Before the refactor, BE and Recycler reached the SDKs through separate
stacks:
```text
BE call sites Recycler call sites
| |
v v
BE object client Recycler S3Accessor
+ BE limiter + Recycler limiter
| |
v v
BE S3 / Azure clients Recycler S3 / Azure clients
| |
v v
AWS / Azure SDK AWS / Azure SDK
```
After the refactor, BE and Recycler keep their own upper-layer
composition points and policies, while both use the same common
interface, decorator, and provider implementations:
```text
AFTER
+-------------------------- BE --------------------------+ +---------------------- Recycler ----------------------+
| BE call sites S3RateLimiterManager | | Recycler call sites AccessorRateLimiter |
| | | | | | | |
| v v | | v v |
| S3ClientFactory BE policy | | S3Accessor Recycler policy |
+-----------------------------+--------------------------+ +-----------------------------+-------------------------+
| |
+------------------------------+-------------------------------+
|
v
ObjStorageClient
^
|
RateLimitedObjStorageClient
(policy_: ObjStorageRateLimitPolicy)
|
+----------+----------+
| |
v v
S3ObjStorageClient AzureObjStorageClient
| |
v v
AWS SDK Azure SDK
```
This keeps cross-cutting behavior composable. Future decorators, such as
tracing, retries, or additional admission controls, can be added without
changing provider implementations or introducing a facade coupled to
every policy.
### 2. How are the different behaviors unified?
| Behavior | Unified behavior |
| --- | --- |
| Client API | BE and Recycler use the same `ObjStorageClient` interface
and request/response types. `doris::io` aliases keep BE call sites
source-compatible, while Recycler adapters preserve its integer-facing
API. |
| Provider clients | `S3ObjStorageClient` and `AzureObjStorageClient`
are shared by both callers and directly implement `ObjStorageClient`. |
| Rate limiting | `RateLimitedObjStorageClient` wraps any
`ObjStorageClient`. BE and Recycler inject their own
`ObjStorageRateLimitPolicy` when constructing the client. |
| Error model | `ObjectStorageResponse` carries a Doris status and
preserves the HTTP status and request ID when available. Recycler
adapters retain caller-facing contracts such as the 0/1/negative
`exists` result. |
| Listing | `list_objects` returns one page. `ObjectListIterator` owns
the continuation token and requests the next page only after consuming
the current page. |
| Missing S3 prefix | The shared S3 implementation preserves the
S3-compatible `NoSuchKey`-as-empty behavior for both callers. |
| Batch deletion | Provider implementations enforce their SDK limits: up
to 1000 objects for S3 and 256 subrequests for Azure. |
| Recursive deletion | `ObjStorageClient` provides the shared
provider-independent listing, expiration filtering, grouping, execution,
and error-propagation flow. Recycler can supply its `SyncExecutor`
adapter; BE uses the synchronous fallback. |
| AWS credentials | `AwsCredentialFactory` implements static
credentials, the default provider chain, role ARN, and external ID once.
|
| Azure credentials | `AzureAuthFactory` creates the shared-key
credential and container client for both callers. Shared error
conversion retains the BE TLS diagnostic context. |
| Metrics | Provider clients share request-failure metrics and latency
timers. The previous BE slow-S3-upload logging behavior is retained. |
### 3. Rate-limit boundary and follow-up
The decorator currently preserves the pre-refactor BE semantics:
- A direct `list_objects` or `delete_objects` call is admitted once
before delegation.
- `delete_objects_recursively` is admitted once as one logical PUT
operation and then delegates to the wrapped client.
- Therefore, list pages and provider delete batches performed inside
recursive deletion do not yet pass back through the decorator
individually.
Making recursive deletion invoke the decorated page-list and
delete-batch operations requires a separate change to avoid either
bypassing the decorator or double-counting requests. That request-level
admission behavior will be handled in a follow-up PR rather than mixed
into this structural refactor. This temporarily changes Recycler
recursive-delete admission to match the existing BE behavior.
28 lines
1.2 KiB
CMake
28 lines
1.2 KiB
CMake
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
# where to put generated libraries
|
|
set(LIBRARY_OUTPUT_PATH "${BUILD_DIR}/src/common_cpp")
|
|
|
|
file(GLOB_RECURSE SRC_FILES CONFIGURE_DEPENDS *.cpp)
|
|
if(NOT BUILD_AZURE STREQUAL "ON")
|
|
list(REMOVE_ITEM SRC_FILES
|
|
"${CMAKE_CURRENT_SOURCE_DIR}/obj-client/azure_obj_storage_client.cpp"
|
|
"${CMAKE_CURRENT_SOURCE_DIR}/obj-client/auth/azure_auth_factory.cpp")
|
|
endif()
|
|
add_library(CommonCPP STATIC ${SRC_FILES})
|