mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Gavin Chouandgavinchou
68938c5ca3
### What problem does this PR solve? Add observability for S3 local rate limiting and S3 429 responses so throttling-related performance issues can be diagnosed and alerted more directly. ### What is changed? - Add explicit S3 local rate limiter bvars for sleep duration, sleep count, and rejected count. - Rename the old ambiguous S3 local rate limiter bvars so their meaning is clear. - Log the first and every N local S3 rate limiter throttled/rejected requests without reading bvar values for decisions. - Add S3 429 retry/failure bvars for S3/Azure object clients. - Update S3 rate limiter unit coverage for sleep and rejected metrics. ### New or renamed bvar metrics Renamed local S3 GET limiter metrics: - `get_rate_limit_ns` -> `s3_get_rate_limit_sleep_ns` - `get_rate_limit_exceed_req_num` -> `s3_get_rate_limit_sleep_count` Renamed local S3 PUT limiter metrics: - `put_rate_limit_ns` -> `s3_put_rate_limit_sleep_ns` - `put_rate_limit_exceed_req_num` -> `s3_put_rate_limit_sleep_count` New local S3 limiter rejection metrics: - `s3_get_rate_limit_rejected_count` - `s3_put_rate_limit_rejected_count` New S3 429 metrics: - `s3_request_retry_too_many_requests_count` - `s3_request_failed_too_many_requests_count` Metric examples: ```text s3_get_rate_limit_sleep_ns : 123456789 s3_get_rate_limit_sleep_count : 42 s3_get_rate_limit_rejected_count : 3 s3_put_rate_limit_sleep_ns : 987654321 s3_put_rate_limit_sleep_count : 27 s3_put_rate_limit_rejected_count : 1 s3_request_retry_too_many_requests_count : 8 s3_request_failed_too_many_requests_count : 2 ``` Example local limiter logs: ```text S3 GET request is throttled by local rate limiter, sleep_ms=12, sleep_count=1000, token_per_second=1000, bucket_tokens=2000, token_limit=5000 S3 PUT request is rejected by local rate limiter, rejected_count=1, token_per_second=1000, bucket_tokens=2000, token_limit=5000 ``` ### Tests - `sh run-cloud-ut.sh --run --filter=s3_rate_limiter_test:*` - `sh run-be-ut.sh --run --filter=S3FileWriterTest.*` Note: an earlier BE run used an incorrect lowercase filter (`s3_file_writer_test:*`), which matched no suite and started running the full BE suite; it was stopped and replaced by the correct `S3FileWriterTest.*` run above. --------- Co-authored-by: gavinchou <gavinchou@apache.org>
56 lines
1.9 KiB
C++
56 lines
1.9 KiB
C++
// Licensed to the Apache Software Foundation (ASF) under one
|
|
// or more contributor license agreements. See the NOTICE file
|
|
// distributed with this work for additional information
|
|
// regarding copyright ownership. The ASF licenses this file
|
|
// to you under the Apache License, Version 2.0 (the
|
|
// "License"); you may not use this file except in compliance
|
|
// with the License. You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing,
|
|
// software distributed under the License is distributed on an
|
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
// KIND, either express or implied. See the License for the
|
|
// specific language governing permissions and limitations
|
|
// under the License.
|
|
|
|
#pragma once
|
|
|
|
#include <aws/core/client/AWSError.h>
|
|
#include <aws/core/client/DefaultRetryStrategy.h>
|
|
|
|
#ifdef USE_AZURE
|
|
#include <azure/core/http/policies/policy.hpp>
|
|
#endif
|
|
|
|
namespace doris {
|
|
void record_object_request_failed(int http_code);
|
|
|
|
class S3CustomRetryStrategy final : public Aws::Client::DefaultRetryStrategy {
|
|
public:
|
|
S3CustomRetryStrategy(int maxRetries, bool retry_slow_down = true);
|
|
~S3CustomRetryStrategy() override;
|
|
|
|
bool ShouldRetry(const Aws::Client::AWSError<Aws::Client::CoreErrors>& error,
|
|
long attemptedRetries) const override;
|
|
|
|
private:
|
|
bool _retry_slow_down;
|
|
};
|
|
|
|
#ifdef USE_AZURE
|
|
class AzureRetryRecordPolicy final : public Azure::Core::Http::Policies::HttpPolicy {
|
|
public:
|
|
AzureRetryRecordPolicy() = default;
|
|
~AzureRetryRecordPolicy() override = default;
|
|
|
|
std::unique_ptr<HttpPolicy> Clone() const override;
|
|
std::unique_ptr<Azure::Core::Http::RawResponse> Send(
|
|
Azure::Core::Http::Request& request,
|
|
Azure::Core::Http::Policies::NextHttpPolicy nextPolicy,
|
|
Azure::Core::Context const& context) const override;
|
|
};
|
|
#endif
|
|
} // namespace doris
|