Cherry-picked from #67076
**Depends on #67554** (the branch-4.1 backport of #66851) (the
aws-sdk-cpp 1.11.219 -> 1.11.221 pin). BE cannot compile
this without `GeneralHTTPCredentialsProvider.h`, which first ships in
1.11.221, so that PR has to
land and the pre-built thirdparty artifacts be re-published first.
### Backport notes
branch-4.1 does not have the `common/cpp/obj-client/` refactor, so this
could not be applied
verbatim:
- The `CredProviderType::Container` switch case lives in **two** places
on this branch,
`be/src/util/s3_util.cpp` and `cloud/src/recycler/s3_accessor.cpp`,
instead of the single
`common/cpp/obj-client/auth/aws_credential_factory.cpp` upstream. Both
now call
`create_container_credentials_provider()`, and the then-unused
`<aws/core/platform/Environment.h>` include was dropped from both.
- `common/cpp/aws_common.{h,cpp}` and
`common/cpp/custom_aws_credentials_provider_chain.cpp` carry
the same change as upstream (the new `AWS_CONTAINER_*` constants,
`container_credentials_available()`
and `create_container_credentials_provider()`); the declarations follow
this branch's local
indentation in `aws_common.h`.
- The tests reach the same providers through this branch's entry points:
`S3ClientFactory::get_aws_credentials_provider()` and
`S3Accessor::_get_aws_credentials_provider_v2()`.
- `ContainerProviderTypeIsUsableAsStsBaseProvider` is **not** carried
over. Upstream it exists to
reach `AwsCredentialFactory::create()` below `S3ClientFactory`; this
branch has no such layer, so
the case would have been a second copy of
`ContainerProviderTypeReadsRotatedTokenFileForPodIdentity`
-- both go through the very same
`_create_credentials_provider(Container)`. The other five new
cases, the two new test headers and the MSK IAM cases are all carried
over unchanged.
The other files (`be/src/runtime/aws_msk_iam_auth.cpp`,
`be/test/runtime/aws_msk_iam_auth_test.cpp`,
`gensrc/proto/cloud.proto`, `gensrc/thrift/AgentService.thrift`) applied
cleanly.
`clang-format` 16 was run over every changed file under `be/src`,
`be/test`, `cloud/src` and
`cloud/test` (the paths the Code Formatter workflow checks) and reports
no diff. BE was **not**
compiled locally: this branch's `thirdparty/installed` is still on
aws-sdk-cpp 1.11.219, so the new
header is not available until the backport of #66851 lands.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01DzUYFcGHQH3bnLGCjpncVj
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
<!-- pr-watcher:conflict-pick
task_id=conflict-pick:ccb064f018a074242ce0193b0487f993c16feedbe9a1cb47bff45f24bdabda53
-->
### What problem does this PR solve?
Related PR: #65420
Backport
[apache/doris#65420](https://github.com/apache/doris/pull/65420) to
`branch-4.1`.
- Source merge commit: `c8b4a1c6d2105fef78a9c4db87e6a7ddde00c3c8`
- Target base commit: `984c535c98710a115b795be278710a1d987c67a5`
### Conflict resolution
- `be/src/common/config.cpp`: Added the source CPU-aware limiter
definitions while retaining branch-4.1’s
${DORIS_HOME}/plugins/connectors Trino path. (adapted)
- `be/src/util/s3_util.h`: Removed the obsolete factory rate_limiter API
required by the source change while retaining branch-4.1’s private
get_aws_credentials_provider declaration. (adapted)
- `be/src/util/s3_util.cpp`: Removed the branch-4.1 cache hash local
that became unused after applying the source factory cache-key refactor;
-Werror compilation exposed it. (adapted)
- `regression-test/data/cloud_p0/s3/test_s3_rate_limiter.out`: Removed
the extra blank line at EOF required by repository diff-check rules
without changing expected rows. (adapted)
### Checks actually run
- `git diff HEAD^ HEAD --check`: passed (exit 0)
- `clang-format --dry-run --Werror <changed C/C++ files>`: passed (exit
0)
No submodules were initialized. This automation only runs
formatting/whitespace, provenance,
conflict-marker, and clean-worktree checks. It does not run builds, unit
tests, or regression tests.
Co-authored-by: liaoxin <liaoxin@selectdb.com>
### What problem does this PR solve?
Backport #64038 to `branch-4.1`.
Add observability for S3 local rate limiting and S3/Azure 429 responses
so throttling-related performance issues can be diagnosed and alerted
more directly.
### What is changed?
- Add explicit S3 local rate limiter bvars for sleep duration, sleep
count, and rejected count.
- Rename the old ambiguous S3 local rate limiter bvars so their meaning
is clear.
- Log the first and every N local S3 rate limiter throttled/rejected
requests without reading bvar values for decisions.
- Add S3 429 retry/failure bvars for S3/Azure object clients, including
Azure batch delete failures.
- Capture the limiter configuration and result under the limiter lock to
avoid concurrent reset races.
- Adapt the Cloud configuration change to `branch-4.1` without bringing
in unrelated master-only fault-injection settings.
### New or renamed bvar metrics
Renamed local S3 GET limiter metrics:
- `get_rate_limit_ns` -> `s3_get_rate_limit_sleep_ns`
- `get_rate_limit_exceed_req_num` -> `s3_get_rate_limit_sleep_count`
Renamed local S3 PUT limiter metrics:
- `put_rate_limit_ns` -> `s3_put_rate_limit_sleep_ns`
- `put_rate_limit_exceed_req_num` -> `s3_put_rate_limit_sleep_count`
New local S3 limiter rejection metrics:
- `s3_get_rate_limit_rejected_count`
- `s3_put_rate_limit_rejected_count`
New S3/Azure 429 metrics:
- `s3_request_retry_too_many_requests_count`
- `s3_request_failed_too_many_requests_count`
Metric examples:
```text
s3_get_rate_limit_sleep_ns : 123456789
s3_get_rate_limit_sleep_count : 42
s3_get_rate_limit_rejected_count : 3
s3_put_rate_limit_sleep_ns : 987654321
s3_put_rate_limit_sleep_count : 27
s3_put_rate_limit_rejected_count : 1
s3_request_retry_too_many_requests_count : 8
s3_request_failed_too_many_requests_count : 2
```
Example local limiter logs:
```text
S3 GET request is throttled by local rate limiter, sleep_ms=12, sleep_count=1000, token_per_second=1000, bucket_tokens=2000, token_limit=5000
S3 PUT request is rejected by local rate limiter, rejected_count=1, token_per_second=1000, bucket_tokens=2000, token_limit=5000
```
### Tests
- `sh run-cloud-ut.sh --run --filter='s3_rate_limiter_test:*'` (7/7
passed)
- `sh run-be-ut.sh --run --filter='S3UTILTest.*:S3FileWriterTest.*'`
(24/24 passed)
- `sh run-be-ut.sh --run
--filter='AzureObjStorageClientTlsHelperTest.*'` (2/2 passed)
Co-authored-by: gavinchou <gavinchou@apache.org>
## Proposed changes
Backport #60344 / eaa4ef92b9 to
branch-4.1.
This adds BE-to-MS RPC throttling support, including:
- host-level MS RPC rate limiters
- table-level MS backpressure throttling
- schema table exposure for backend MS RPC table throttlers
- token bucket rate limiter naming updates used by common S3/rate-limit
code
Conflict resolution notes:
- Kept branch-4.1 schema table numbering and added
SCH_BACKEND_MS_RPC_TABLE_THROTTLERS after the existing branch-local
entries.
- Kept branch-4.1 cloud config layout and added the new MS RPC
throttling configs before the compile-check boundary.
## Validation
- `git diff HEAD^ HEAD --check`
- `./run-be-ut.sh --run
--filter=HostLevelMSRpcRateLimitersTest.*:HostLevelMSRpcRateLimitersConfigTest.*:StrictQpsLimiterTest.*:TableRpcQpsRegistryTest.*:TableRpcThrottlerTest.*:MSBackpressureHandlerTest.*:LoadRelatedRpcTest.*:RpcThrottleStateMachineTest.*:RpcThrottleCoordinatorTest.*:RpcThrottleIntegrationTest.*
-j32`
- 71 tests from 10 test suites passed.
## Summary
Pick #60608 to branch-4.1
Add new operation show_segment_data to meta_tool for reading and
displaying segment file column metadata and statistics.
## Changes
- Add get_field_type_string() to convert FieldType to readable string
- Add get_encoding_string() to convert EncodingTypePB to readable string
- Add get_compression_string() to convert CompressionTypePB to readable
string
- Add print_column_meta() to print column metadata recursively
- Add show_segment_data() to display segment file info and statistics
- Add common/cpp/private_member_accessor.hpp helper for accessing
private members
## Test
- [x] Compilation passed (BE meta_tool target)
## Related Issue
close#60608
---------
Co-authored-by: gavinchou <gavinchou@apache.org>
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
* Print azure sdk log to `be.INFO`
* Set aws_log_level=3 (WARN) and azure_log_level=2 (Warning) by default,
because when request failed, we don't know what happened in previously
* Set azure MaxRetries equal to config::max_s3_client_retry(10)
* AWS SDK default retry policy check response xml `code` to determine
retry or not, but other object storage such as `OSS`, `COS` not
compatible with AWS S3, so adding a custome retry policy for solving the
problem
## Proposed changes
Issue Number: close #xxx
Fix clang-18 and libc++ compile problems
unique() of shared_ptr is deprecated in clang-18, it's not a strict
implementation, can use use_count() instead.
And fixed some other compie errors in both using clang-18 and libc++.
## Further comments
If this is a relatively large or complex change, kick off the discussion
at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
Previously the azure sdk would be compiled into BE by default. But
sometimes the BE would run in environment which only require AWS
s3-compatible functionality. So user can use `DISABLE_BUILD_AZURE` to
control whether build azure sdk into BE or not.
`export DISABLE_BUILD_AZURE=ON` to disable build with azure support
A compilation error in `s3_rate_limiter.cpp` due to a type mismatch
between `std::chrono::system_clock::time_point` and
`std::chrono::high_resolution_clock::time_point`，Updated the
`_update_remain_token` function to accept
`std::chrono::high_resolution_clock::time_point` instead of
`std::chrono::system_clock::time_point` on MacOS。
## Proposed changes
Extract SyncPoint to common cpp directory. Some code shared between BE
and Cloud will gradually be extracted to the `$repo_root/common/cpp`
directory in the future.
Mingyu Chen (Rayner)andOwen Leung
github-actions[bot]andbobhan1
Refrainandliaoxin
Gavin Chouandgavinchou
Lei Zhang
abmdocrt
deardeng
Lei Zhang
sparrow
AlexYue
神技圈子
plat1ko