mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
master
37
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 hui lai
|
b8dc326111 |
[feature](storage) Support native GCP IAM authentication (#68540)
### What problem does this PR solve? GCS access currently relies on HMAC keys or unsigned requests. This PR adds native GCP IAM authentication across FE, BE, and Cloud, allowing Doris to access GCS using Application Default Credentials (ADC), a Compute Engine service account, or an impersonated service account. ### Authentication and configuration ```properties provider=GCP gs.credential_provider_type=DEFAULT # Optional: access GCS as this service account gs.impersonation_service_account=target@project.iam.gserviceaccount.com ``` - `DEFAULT` uses Google's Application Default Credentials chain. - `COMPUTE_ENGINE` uses the service account attached to the VM. - `gs.impersonation_service_account` optionally selects a target service account using the source identity's impersonation permissions. - GCP configurations without HMAC credentials or an explicit authentication mode default to ADC. Public-bucket access requires explicit `ANONYMOUS`. Existing GCS HMAC authentication remains supported. Native IAM authentication rejects conflicting HMAC or AWS credential properties and requires a supported Google Storage HTTPS service endpoint. ### How it works - **FE:** The GCS filesystem plugin resolves authentication properties and creates refreshable Google OAuth credentials. Object operations use bearer-authenticated GCS XML API requests; namespace operations use the Hadoop GCS connector. The Iceberg connector applies the same GCP identity to GCS storage access. - **BE and Cloud:** Typed GCP identity settings are carried through Thrift, Protobuf, and Cloud object metadata. The C++ storage client uses google-cloud-cpp OAuth credentials to authenticate GCS requests and refresh access tokens. - **Resources and storage vaults:** Persist and propagate the authentication mode and optional impersonation target. Partial credential updates preserve omitted settings and allow impersonation to be cleared explicitly. Anonymous GCS storage vaults are rejected. - **Data and error-log access:** Integrate IAM authentication with GCS-backed catalogs, TVFs, load, export, outfile, and backup/restore. EXPORT and OUTFILE accept `gs://` paths. Backend error-log downloads use GCS V4 URLs signed through IAM `signBlob`. FE/Iceberg package the Hadoop GCS connector; BE/Cloud package the google-cloud-cpp OAuth libraries. The participating FE, BE, and MetaService processes must support the new credential fields before native IAM is enabled. Native IAM support for Paimon/Hudi and FE presigned uploads is outside this PR's scope. ### Release note Support native GCP IAM authentication for GCS using ADC, Compute Engine service accounts, and optional service-account impersonation. GCP configurations without HMAC credentials default to ADC; anonymous access must be selected explicitly. |
||
 Gavin Chou
|
493259c368 |
[improvement](metrics) Encode version metrics with fixed-width components (#66666)
Related PR: #68655 Problem Summary: The existing version gauges concatenate variable-width decimal components with `0`. Their numeric ordering can disagree with version ordering; for example, `4.99.0` is encoded as `409900`, which is greater than `5.0.0` encoded as `50000`. Appending hotfix only when it is positive also changes the value's magnitude. This change treats `major`, `minor`, `patch`, and `hotfix` as four base-100 digits. Every component therefore occupies exactly two decimal positions: `5.1.2.3` is logically encoded as `05|01|02|03` and exported as the numeric value `5010203`. Source generation rejects components outside `0..99`, so the encoding remains unambiguous and order-preserving. BE and Cloud share one C++ encoder, FE applies the equivalent calculation, and the existing metric names and build labels remain unchanged. Unit tests cover representative values, ordering, and bounds; the cross-process regression reconstructs the expected value from labels for FE, BE, and Cloud. ### Prometheus exposition examples The following examples assume build version `doris-5.1.2.3` and short Git hash `abc1234`. Each version component occupies two decimal positions, so `5.1.2.3` is logically encoded as `05|01|02|03` and exported as the numeric value `5010203`. FE (`/metrics`): ```text # HELP doris_fe_version # TYPE doris_fe_version gauge doris_fe_version{version="doris-5.1.2.3", major="5", minor="1", patch="2", hotfix="3", short_hash="abc1234"} 5010203 ``` BE (`/brpc_metrics`): ```text # HELP doris_be_version # TYPE doris_be_version gauge doris_be_version{version="doris-5.1.2.3",major="5",minor="1",patch="2",hotfix="3",short_hash="abc1234"} 5010203 ``` Cloud meta-service/recycler (`/brpc_metrics`): ```text # HELP doris_cloud_version # TYPE doris_cloud_version gauge doris_cloud_version{version="doris-5.1.2.3",major="5",minor="1",patch="2",hotfix="3",short_hash="abc1234"} 5010203 ``` ### Release note Encode FE, BE, and Cloud version metric values with fixed-width base-100 components. |
||
  Mingyu Chen (Rayner)andClaude Code
|
b41856ab7e |
[fix](build) Move the macOS toolchain to LLVM 22 so ASAN works on macOS 26.4+ (#68595)
### What problem does this PR solve?
Issue Number: N/A
Related PR: apache/doris-website#4180 (the macOS build guide change that
goes with this PR)
Problem Summary:
**Context.**
On macOS the only entry point for the BE unit tests is `run-be-ut.sh`,
and it defaults to `BUILD_TYPE_UT=ASAN`. Since macOS 26.4 that default
cannot start a single process: every binary linked against llvm.org's
`libclang_rt.asan_osx_dynamic` deadlocks during runtime initialisation,
before `main()`. `env.sh` selects that same toolchain on macOS
(`llvm@20`), so a developer who follows the documented setup gets a
build that hangs rather than one that fails.
How the macOS toolchain gets chosen matters for the rest of this
description:
- **Selection.** `env.sh` writes `custom_env_mac.sh`, which puts `$(brew
--prefix)/opt/<formula>/bin` on `PATH` for every formula in its
`CELLARS` list. Unless `DORIS_CLANG_HOME` is already set, `env.sh` then
takes the first `clang` on `PATH` and derives `CC`/`CXX` from it.
- **Installation.** `CELLARS` only selects; it installs nothing.
Developers install Homebrew formulae by following the macOS build guide,
and each CI workflow installs from its own `brew install` list.
- **Third-party libraries.** `thirdparty/installed` comes from one of
two places:
- the published `doris-thirdparty-prebuilt-darwin-*.tar.xz`, which
`apache/doris-thirdparty`'s `build-target.yml` builds by running
`thirdparty/build-thirdparty.sh` from apache/doris master;
- a local run of that same script. `build.sh` also starts one by itself
when `thirdparty/installed` lacks a library it checks for.
**1. The problem, and what it cost**
*The deadlock.* `sample` on the stalled process shows the whole chain:
```
__malloc_init (libsystem_malloc) <- early libSystem init calls malloc_default_zone()
`- wrap_malloc_default_zone (asan runtime)
`- AsanInitFromRtl -> AsanInitInternal -> InitializeShadowMemory
`- MemoryRangeIsAvailable -> MemoryMappingLayout::Next -> get_dyld_hdr()
`- dyld_shared_cache_iterate_text_swift <- macOS 26.4 reimplemented this in Swift
`- _Block_copy -> malloc
`- __sanitizer_mz_malloc (asan's own malloc)
`- AsanInitFromRtl() <- re-enters init
`- StaticSpinMutex::LockSlow <- spins on the lock it already holds
```
The cause is an OS-side change, not a Doris one. It also does not mean
ASAN is broken on macOS 26: Apple's own clang sanitizer runtime runs
fine on the same host, and only llvm.org's compiler-rt is affected.
Upstream fixed it by weak-importing `_dyld_get_dyld_header` and using
it, when present, instead of walking the shared cache
(llvm/llvm-project#182943, main `2e7d07a`; backport #188913,
release/22.x `7b6514c`). **The fix shipped only in 22.1.8.** 20.1.8 is
the last 20.x release and 21.1.8 the last 21.x, so no version of
`llvm@20` can ever be made to work.
*Why it looks like a hang and not an error.*
- `be/CMakeLists.txt` reaches `storage/index/ann` unconditionally, and
that directory's `cmake-protect` target calls `add_subdirectory()` on
`contrib/openblas`.
- OpenBLAS runs an instrumented `getarch` probe from its **configure**
step (`contrib/openblas/cmake/prebuild.cmake:1513`,
`execute_process(COMMAND .../getarch 0 ...)`), and `execute_process` has
no timeout.
- So every ASAN build stops at `-- Running getarch` and never moves
again. It sits at about 90% CPU and prints no diagnostic.
`run-be-ut.sh` on macOS was therefore unusable at its default setting,
and the failure gave the developer nothing to act on.
*The second obstacle, once the toolchain is bumped.*
- clang 22 added `-Wc2y-extensions` and folds it into `-Wpedantic`.
- `__COUNTER__` only reached the C standard in C2y (WG14 N3457).
`be/src/runtime/runtime_profile.h:73-85` uses it to give two
`SCOPED_TIMER` / `SCOPED_RAW_TIMER` expansions on the same line distinct
names.
- As a result, a clang 22 build of any TU that includes that header
fails under `-Werror`. Three representative unity TUs were enough to hit
it, so the failure is not confined to one module.
*The third obstacle: clang 22 cannot build the third-party libraries.*
The first revision of this PR moved every macOS toolchain reference to
`llvm@22`, including the third-party build. CI failed both macOS
third-party jobs (7m49s, 6m23s) on the first library that exercises a
new clang 22 error:
```
unixODBC 2.3.7 src/SQLBrowseConnectW.c:424:82:
error: incompatible pointer types passing 'SQLSMALLINT *' (aka 'short *')
to parameter of type 'int *' [-Wincompatible-pointer-types]
```
clang 16, clang 20 and Apple clang 21 report this as a warning; clang 22
makes it an error. unixODBC is only the *first* failure: the build
aborts there, so every library after it is untested against clang 22.
The macOS third-party libraries therefore have to stay on `llvm@20`
while the BE moves to `llvm@22`.
Leaving the third-party build alone does not achieve that, because it
sources `env.sh`:
- With `CELLARS` naming `llvm@22`, a machine without `llvm@22` gets a
non-existent directory on `PATH`.
- `command -v clang` then falls through to `/usr/bin/clang` (Apple
clang), which is worse than either explicit choice.
- The job that publishes the prebuilt archive in
`apache/doris-thirdparty` is exactly such a machine: it installs only
`llvm@20` and sets no `DORIS_CLANG_HOME`.
*Two latent defects clang 22 then surfaced.* The bump is not a pure
version change. clang 22's stricter diagnostics stop the build on two
pre-existing bugs, and both are worth fixing on their own merits.
| # | Site | Diagnostic | What is actually wrong |
|---|---|---|---|
| 1 | `be/src/load/group_commit/wal/wal_dirs_info.cpp:98` |
`-Wunused-result` | `LOG(INFO) << "… err: {}", e.what();` — the `,` is
the comma operator, not an argument separator, so the statement is
`(LOG(INFO) << "…{}") , (e.what())`. The `{}` is never substituted, and
`e.what()` is evaluated and then discarded: **the error message has
never been logged**, only the literal `{}`. A repo-wide scan for the
same shape finds exactly this one site (`cloud/src/common/logging.h`
matches only inside macro definitions and is not a bug). |
| 2 | `common/cpp/sync_point.cpp:208,210` | `-Wthread-safety-analysis` |
The function holds `std::unique_lock lock(mutex_)`, then releases and
re-takes the mutex with a raw `mutex_.unlock()` / `mutex_.lock()` pair
around the callback, which bypasses the lock's ownership tracking. If
the callback throws, the re-lock is skipped while `~unique_lock` still
believes it owns the mutex, so the destructor unlocks a mutex that is
not held (UB) and `num_callbacks_running_` is never decremented.
`lock.unlock()` / `lock.lock()` still runs the callback unlocked, as
intended, and keeps the ownership state correct on every path. |
**2. What this PR does, and why it helps**
- **`env.sh`**: the macOS `CELLARS` list moves from `llvm@20` to
`llvm@22`, with a comment recording that 22.1.8 is the minimum and why.
This is the single line that decides which clang a macOS developer
builds the BE with.
- **`thirdparty/build-thirdparty.sh`**: on Darwin, right after sourcing
`env.sh` and next to the existing environment sanitization, the script
unconditionally points `DORIS_CLANG_HOME`, `CC` and `CXX` at `$(brew
--prefix llvm@20)` and puts its `bin/` first on `PATH`. If `llvm@20` is
not installed, it stops with a `brew install llvm@20` hint rather than
falling back to another compiler.
- The override has to come after `env.sh` and has to be unconditional.
`env.sh` sources `custom_env.sh`, which may export `DORIS_CLANG_HOME`
for the BE, and `build.sh` has already exported the BE's `llvm@22`
values before it starts this script.
- Every macOS third-party build runs this script: `build.sh`'s automatic
rebuild, a manual run, the pull request check in `build-thirdparty.yml`,
and the `apache/doris-thirdparty` job that publishes
`doris-thirdparty-prebuilt-darwin-*.tar.xz`. This one place therefore
keeps all of them, and the published archive, on `llvm@20`.
- `.github/workflows/build-thirdparty.yml` is unchanged: its macOS jobs
already install `llvm@20`.
- **`be/CMakeLists.txt`**: add `-Wno-c2y-extensions` for clang 19 and
newer, in its own `add_compile_options` call **after** `-Wpedantic`.
- The position matters: a later `-Wpedantic` turns the group back on.
That is also why passing the flag through `EXTRA_CXX_FLAGS` does not
work: that variable lands near the front of the command line.
- The version gate matters because the `c2y-extensions` group only
exists from clang 19 on. clang rejects an unknown `-Wno-` option like
any other unknown warning option (`-Wunknown-warning-option`, fatal
under `-Werror`), and this file accepts clang 16 and newer.
- For clang 19 and newer the command line is exactly what it was.
- **`.github/workflows/be-ut-mac.yml`**: install `llvm@22` for the BE
and keep `llvm@20` next to it.
- The job builds the BE on top of the downloaded darwin-arm64 prebuilt.
When that archive lacks a library `build.sh` checks for, `build.sh`
rebuilds the third-party libraries from source, and that rebuild now
needs `llvm@20`. The archive falls behind like this whenever master
changes the set of checked libraries before the next archive is
published.
- Both formulae are keg-only, so installing both does not change which
one the BE uses.
- **`be/src/load/group_commit/wal/wal_dirs_info.cpp` and
`common/cpp/sync_point.cpp`**: the two fixes from the table above. Each
is one line, the smallest change that removes the defect rather than
suppressing the warning. No `-Wno-unused-result` /
`-Wno-thread-safety-analysis` is added, because those diagnostics point
at real bugs and should keep firing.
- **apache/doris-website#4180**: the macOS build guide installs
`llvm@22` and `llvm@20` instead of only `llvm@20`. Without it, a
developer who follows the guide after this PR has no `llvm@22`, and the
BE silently builds with Apple clang.
What this PR deliberately does not do is add `llvm@20` to `env.sh`'s
`CELLARS`:
- `CELLARS` installs nothing.
- `build-thirdparty.sh` does not use `PATH` to find `llvm@20`.
- The `CELLARS` loop prepends each entry to `PATH`. An `llvm@20` entry
after `llvm@22` would put the BE back on `llvm@20` and bring the
deadlock back.
**3. The classes, and how they call each other**
```
env.sh CELLARS := llvm@22 (selects a compiler; installs nothing)
|- generates custom_env_mac.sh: $(brew --prefix)/opt/<cellar>/bin prepended to PATH
|- DORIS_CLANG_HOME := dirname($(command -v clang))/.. -> CC / CXX / ASAN_SYMBOLIZER_PATH
'- is sourced by:
|- build.sh, run-be-ut.sh -> BE: llvm@22
'- thirdparty/build-thirdparty.sh:55
'- :100-110 on Darwin: DORIS_CLANG_HOME / CC / CXX := $(brew --prefix llvm@20)
llvm@20 missing -> exit 1, "brew install llvm@20"
-> third-party libraries: llvm@20
run by: build.sh (library missing from thirdparty/installed), manual runs,
build-thirdparty.yml (PR check), doris-thirdparty build-target.yml (publisher)
brew install lists, i.e. who installs which LLVM:
macOS build guide (apache/doris-website#4180) llvm@22 llvm@20
.github/workflows/be-ut-mac.yml llvm@22 llvm@20
.github/workflows/build-thirdparty.yml llvm@20
apache/doris-thirdparty build-target.yml llvm@20
be/CMakeLists.txt if (COMPILER_CLANG)
|- add_compile_options(-Wpedantic ...) <- enables the c2y group
'- if (CMAKE_CXX_COMPILER_VERSION >= 19)
add_compile_options(-Wno-c2y-extensions) <- must stay after the line above
|
be/src/runtime/runtime_profile.h:73-85
'- MACRO_CONCAT(SCOPED_TIMER, __COUNTER__) <- the only __COUNTER__ use in be/src, be/test
|
be/src/storage/index/ann/cmake-protect/CMakeLists.txt:48
'- add_subdirectory(contrib/openblas)
'- cmake/prebuild.cmake:1513 execute_process(getarch) <- where the hang surfaced
```
### Release note
None
### Check List (For Author)
- Test: Manual test on macOS 26.5.1 (arm64, dyld-1378), plus CI.
- `clang -fsanitize=address` hello world exits 0 with llvm@22 and still
deadlocks with llvm@20 on the same host.
- `sh run-be-ut.sh` with no environment overrides now selects
`Clang-22.1.8` and clears the `-- Running getarch` point that previously
hung forever: zero FAILED targets, and `doris_be_test` links.
- `sh run-be-ut.sh --run --filter=FormatRoundTest.*` starts the ASAN
binary and passes 8 tests.
- `-Wno-c2y-extensions` gate, tested on a `__COUNTER__` TU under
`-Wpedantic -Werror`: clang 16.0.6 rejects the flag (`unknown warning
option`), which is why it is gated; clang 20.1.8 accepts it; clang
22.1.8 fails without it and passes with it.
- `build-thirdparty.sh` compiler override, checked under `bash -x` with
both `custom_env.sh` and `build.sh` exporting the `llvm@22` values: the
effective `CC`/`CXX` and the first clang on `PATH` are llvm@20. With
`llvm@20` missing, the script exits 1 with the brew hint before building
anything.
- `be-ut-mac.yml`: the "Build BE" step script passes `bash -n` on bash
3.2 and 5.3, and its `cellars` array holds both `llvm@22` and `llvm@20`.
The workflow runs only on pushes to master and on schedule, so this PR
cannot run it.
- `build-support/check-build-hygiene.sh` passes.
- CI: `Build Third Party Libraries (macOS)`, `(macOS-arm64)` and
`(Linux)` pass at `a955357a30f` (run 36542107899), which already
contains the `build-thirdparty.sh` override. The first revision, which
moved these builds to llvm@22, failed both macOS jobs on unixODBC.
- Regression test / unit test: N/A (toolchain, flag and workflow
change).
- Behavior changed: Yes, for macOS builds only.
- The BE builds with llvm@22 instead of llvm@20.
- The third-party libraries, including the published prebuilt, stay on
llvm@20. Building them from source now requires llvm@20 installed next
to llvm@22; without it, the build stops with a `brew install llvm@20`
hint.
- Existing macOS developers need to run `brew install llvm@22`. Without
it, `env.sh` falls back to Apple clang in `/usr/bin` for newly
configured build directories.
- Does this need documentation: Yes, apache/doris-website#4180.
---------
Co-authored-by: Claude Code <noreply@anthropic.com>
|
||
 Owen Leung
|
d34f06b5da |
[fix](be): fix BE access to aws under eks pod identity (#67076)
### What problem does this PR solve? Issue Number: close #66554 **Problem Summary** Fixes the issue above by forwarding `AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE` alongside the inline token. Given a file path, the provider will re-read it before every fetch, so credentials survive token rotation. ECS, which sets only the inline variable `AWS_CONTAINER_AUTHORIZATION_TOKEN`, is unaffected. **Why the aws-sdp-cpp bump** The `GeneralHTTPCredentialsProvider` does not exist before 1.11.221. The only container provider available at the currently pinned version is `TaskRoleCredentialsProvider`, which takes the token as a string and has no notion of a token file. 1.11.221 is the first version whose `GeneralHTTPCredentialsProvider` re-reads the token file. |
||
 Xin Liao
|
6b5c53fdb4 |
[fix](s3) Keep the response stream usable when an error body overflows the read buffer (#66557)
Reading from object storage fails from time to time with ``` [INTERNAL_ERROR]failed to read from <key>: Failed to flush response stream (eof: 0, bad: 1) code=-1 type=1, request_id=failed to read ``` and succeeds when the same statement is run again. It has been hit by queries reading a rowset, by compaction, by an outfile export and by the download of an inverted index, always on an object storage that was answering `429` or `503` at that moment. `S3ObjStorageClient::get_object()` hands the buffer of the caller to the SDK as the response stream of the request, sized exactly like the requested range. The SDK writes the body of every response into that stream, the body of an error response included. The XML document of a `429 SlowDown` is a few hundred bytes, so a small ranged read cannot hold it - the read of the footer of a packed file asks for 12 bytes. `PreallocatedStreamBuf` does not implement `overflow()`, so the stream turns bad, the write callback of curl reports a short write and curl aborts the transfer with `CURLE_WRITE_ERROR`. The status code of the response is lost from there on: `CurlHttpClient::MakeRequest()` reads `CURLINFO_RESPONSE_CODE` only when curl succeeded, so the code stays at `REQUEST_NOT_MADE` (-1), and the flush check at the end of the same function replaces the retryable `NETWORK_CONNECTION` classification with `INTERNAL_FAILURE` (1). `S3CustomRetryStrategy::ShouldRetry()` declines to retry an error classified that way, and so does `S3FileReader::read_at_impl()`, which retries on `429` alone. A throttling error the server asked us to retry cancels the statement of the user instead, which is why running it again works. This also means the error carries no evidence of what really happened: the code of the response, the exception name and the request id of the object storage are all gone by the time the message is built. The fix lets the response stream grow: the body is written into the buffer of the caller as long as it fits, which is the case for every successful ranged read and keeps that path free of copies, and the remainder spills into a buffer of the stream itself, truncated at 1MB because only error documents are expected to overflow. The stream never turns bad, so curl completes the transfer, the SDK records the real status code and parses the error out of the body, and both the retry of the SDK and the retry of `S3FileReader` on `429` work again. A server or a proxy answering a ranged read with the whole object overflows the buffer as well. Such a read is still rejected, by the length check that follows the request, and now with a message that says so. Two misleading messages are fixed along the way: - `request_id=failed to read` is not a request id of the object storage. It is the string `S3FileReader` appended behind the empty request id of a failure raised by the client itself. The append is dropped and an empty request id is printed as `<empty>`. - The message of a failed read named neither the bucket nor the offset, leaving `failed to read from :` in the log whenever the key was empty. |
||
 bobhan1
|
c81ad5b840 |
[improvement](cloud) Add dry-run mode for BE-to-MS RPC rate limiting (#66977)
### What problem does this PR solve? Issue Number: N/A Related PR: #66969, #66940 Problem Summary: BE-to-MetaService host-level and table-level rate limiting needs production-safe observability before thresholds are enforced. This PR adds dry-run switches for both paths. Dry-run keeps limit evaluation, would-wait metrics, per-table QPS collection, and table-level MS_BUSY state transitions active, but does not delay requests. | Scope | New config | Default | Existing enforcement config | Purpose | | --- | --- | --- | --- | --- | | Host-level | `enable_ms_rpc_host_level_rate_limit_dry_run` | `true` | `enable_ms_rpc_host_level_rate_limit` (default `false`) | Evaluate each configured per-RPC token bucket, record would-wait observations, and log would-throttle decisions without sleeping. | | Table-level | `enable_ms_backpressure_handling_dry_run` | `true` | `enable_ms_backpressure_handling` (default `false`) | Keep per-table QPS collection and the MS_BUSY adaptive-throttling state machine active, but do not sleep on a throttle decision. | For each scope, the dry-run and enforcement configs interact as follows: - When both configs are `false`, rate-limit evaluation for that scope is disabled. - When only the enforcement config is `true`, the limiter is enforced and the request sleeps for the calculated wait time. - When the dry-run config is `true`, dry-run takes precedence regardless of the enforcement config: the limiter state, metrics, and logs are updated, but the request does not sleep. Both host-level and table-level dry-run paths reserve against the same limiter state used by enforcement, matching the FE dry-run model: dry-run suppresses sleeping but preserves reservation state. When a table-level downgrade restores a previous QPS limit, it resets queued reservations so stale wait debt does not carry into the relaxed limit; the final downgrade removes the limiter. The table-level coordinator uses 64-bit counters with overflow-free saturation at each timing threshold. Its MS_BUSY downgrade timer is reset to inactive when there is no pending upgrade history, while the upgrade cooldown counter is preserved and stops at the configured threshold so runtime cooldown changes retain the existing timing semantics. ### Signals after rate limiting is triggered Throttle-decision logs use `INFO` level and are suppressed independently per RPC type to at most one log per second. They are emitted only when the limiter calculates a positive wait. Host-level dry-run logs explicitly report the estimated wait: ```text [ms-throttle] host-level rate limiter dry run would throttle MS RPC request, rpc=get rowset, estimated_wait_ns=1250000, qps_limit=320 ``` With enforcement enabled and dry-run disabled, the corresponding log reports the actual sleep: ```text [ms-throttle] host-level rate limiter throttled MS RPC request, rpc=get rowset, sleep_ns=1250000, qps_limit=320 ``` Table-level dry-run and enforcement use the same trigger log so operators can search one stable pattern: ```text [ms-throttle] table-level rate limiter triggered for MS RPC request, rpc=commit_rowset, table_id=10001, wait_us=8200, current_qps=146.7, qps_limit=100 ``` For table-level dry-run, `wait_us` is the estimated wait and the request is not delayed. With enforcement enabled and dry-run disabled, the request sleeps for that wait. MS_BUSY-driven table-level state transitions also emit `INFO` logs, for example: ```text [ms-throttle] received MS_BUSY, triggering upgrade [ms-throttle] upgrade: rpc=commit_rowset, table_id=10001, current_qps=200, old_limit=0, new_limit=150 [ms-throttle] downgrade: rpc=commit_rowset, table_id=10001, removed limit ``` The same events are observable through bvars: host-level per-RPC latency recorders based on `host_level_ms_rpc_rate_limit_sleep`, table-level `ms_rpc_backpressure_throttle_wait_<rpc>` latency recorders, and `ms_rpc_backpressure_ms_busy_*`, `ms_rpc_backpressure_upgrade_*`, and `ms_rpc_backpressure_downgrade_*` counters/windows. ### Release note Add dry-run observation for BE-to-MetaService host-level and table-level RPC rate limiting without delaying requests. Both dry-run switches are enabled by default and take precedence over enforcement. Dry-run and enforcement share limiter reservation state, table-level downgrade resets queued reservations when restoring a previous QPS limit, and long-running table-level coordinator timers no longer overflow. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - `./run-be-ut.sh --run --filter='HostLevelMSRpcRateLimitersTest.*:StrictQpsLimiterTest.*:TableRpcThrottlerTest.*:MSBackpressureHandlerTest.*:RpcThrottleStateMachineTest.*:RpcThrottleIntegrationTest.*' -j100` (56 tests passed) - `./run-be-ut.sh --run --filter='TokenBucketRateLimiterTest.*:S3RateLimiterManagerTest.*:S3RateLimiterMetricsTest.*' -j100` (11 tests passed) - `./run-be-ut.sh --run --filter='MSBackpressureHandlerTest.*:RpcThrottleStateMachineTest.*:RpcThrottleCoordinatorTest.*:RpcThrottleIntegrationTest.*' -j100` (44 tests passed) - `./build.sh --be -j100` - `build-support/check-format.sh` - `build-support/check-build-hygiene.sh` - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [x] Yes. Dry-run observation for both limiter levels is enabled by default and does not delay MetaService RPCs; dry-run and enforcement share reservation state; table-level downgrade resets queued reservations; coordinator timing counters saturate without overflowing; actual and dry-run decisions emit per-RPC rate-limited `INFO` logs. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label |
||
|
hui lai
|
c71d1aec0f |
[refactor](storage) Unify BE and Recycler object clients (#66350)
### 1. What does this PR do?
BE and Cloud Recycler previously maintained separate object-storage
abstractions and separate S3/Azure implementations. Although both sides
called the same cloud-provider SDKs, credential construction, error
conversion, metrics, pagination, batch deletion, and compatibility
behavior were duplicated and could evolve differently.
This PR consolidates those implementations under `common/cpp/client`
while keeping the original decorator-oriented client abstraction:
- `ObjStorageClient` is the common interface used by upper layers and
implemented by all concrete clients and decorators.
- `S3ObjStorageClient` and `AzureObjStorageClient` implement provider
operations through the AWS and Azure SDKs.
- `RateLimitedObjStorageClient` implements the same interface and
decorates another `ObjStorageClient`.
- BE's `S3ClientFactory` and Recycler's `S3Accessor` construct the
provider client and wrap it with their own rate-limit policy.
- Shared request/response types, page-based listing, lazy iteration,
recursive deletion orchestration, credentials, metrics, and error
conversion live in the common layer.
Before the refactor, BE and Recycler reached the SDKs through separate
stacks:
```text
BE call sites Recycler call sites
| |
v v
BE object client Recycler S3Accessor
+ BE limiter + Recycler limiter
| |
v v
BE S3 / Azure clients Recycler S3 / Azure clients
| |
v v
AWS / Azure SDK AWS / Azure SDK
```
After the refactor, BE and Recycler keep their own upper-layer
composition points and policies, while both use the same common
interface, decorator, and provider implementations:
```text
AFTER
+-------------------------- BE --------------------------+ +---------------------- Recycler ----------------------+
| BE call sites S3RateLimiterManager | | Recycler call sites AccessorRateLimiter |
| | | | | | | |
| v v | | v v |
| S3ClientFactory BE policy | | S3Accessor Recycler policy |
+-----------------------------+--------------------------+ +-----------------------------+-------------------------+
| |
+------------------------------+-------------------------------+
|
v
ObjStorageClient
^
|
RateLimitedObjStorageClient
(policy_: ObjStorageRateLimitPolicy)
|
+----------+----------+
| |
v v
S3ObjStorageClient AzureObjStorageClient
| |
v v
AWS SDK Azure SDK
```
This keeps cross-cutting behavior composable. Future decorators, such as
tracing, retries, or additional admission controls, can be added without
changing provider implementations or introducing a facade coupled to
every policy.
### 2. How are the different behaviors unified?
| Behavior | Unified behavior |
| --- | --- |
| Client API | BE and Recycler use the same `ObjStorageClient` interface
and request/response types. `doris::io` aliases keep BE call sites
source-compatible, while Recycler adapters preserve its integer-facing
API. |
| Provider clients | `S3ObjStorageClient` and `AzureObjStorageClient`
are shared by both callers and directly implement `ObjStorageClient`. |
| Rate limiting | `RateLimitedObjStorageClient` wraps any
`ObjStorageClient`. BE and Recycler inject their own
`ObjStorageRateLimitPolicy` when constructing the client. |
| Error model | `ObjectStorageResponse` carries a Doris status and
preserves the HTTP status and request ID when available. Recycler
adapters retain caller-facing contracts such as the 0/1/negative
`exists` result. |
| Listing | `list_objects` returns one page. `ObjectListIterator` owns
the continuation token and requests the next page only after consuming
the current page. |
| Missing S3 prefix | The shared S3 implementation preserves the
S3-compatible `NoSuchKey`-as-empty behavior for both callers. |
| Batch deletion | Provider implementations enforce their SDK limits: up
to 1000 objects for S3 and 256 subrequests for Azure. |
| Recursive deletion | `ObjStorageClient` provides the shared
provider-independent listing, expiration filtering, grouping, execution,
and error-propagation flow. Recycler can supply its `SyncExecutor`
adapter; BE uses the synchronous fallback. |
| AWS credentials | `AwsCredentialFactory` implements static
credentials, the default provider chain, role ARN, and external ID once.
|
| Azure credentials | `AzureAuthFactory` creates the shared-key
credential and container client for both callers. Shared error
conversion retains the BE TLS diagnostic context. |
| Metrics | Provider clients share request-failure metrics and latency
timers. The previous BE slow-S3-upload logging behavior is retained. |
### 3. Rate-limit boundary and follow-up
The decorator currently preserves the pre-refactor BE semantics:
- A direct `list_objects` or `delete_objects` call is admitted once
before delegation.
- `delete_objects_recursively` is admitted once as one logical PUT
operation and then delegates to the wrapped client.
- Therefore, list pages and provider delete batches performed inside
recursive deletion do not yet pass back through the decorator
individually.
Making recursive deletion invoke the decorated page-list and
delete-batch operations requires a separate change to avoid either
bypassing the decorator or double-counting requests. That request-level
admission behavior will be handled in a follow-up PR rather than mixed
into this structural refactor. This temporarily changes Recycler
recursive-delete admission to match the existing BE behavior.
|
||
 Refrain
|
07ba0d1e44 |
[improvement](s3) Default scheme-less S3 endpoints to HTTPS (#65379)
Problem Summary: S3 endpoint scheme handling was inconsistent across FE, BE, and Cloud, and the previous implementation distinguished Doris internal buckets from user-configured external buckets. That classification leaked into S3Conf construction and client-cache identity even though the desired transport rule is the same for every AWS/S3-compatible endpoint. This PR applies one rule across S3 client construction and Resource/Storage Vault persistence boundaries: - Keep the existing FE, BE, and Cloud s3_client_http_scheme compatibility setting. - Default every endpoint without an explicit scheme to HTTPS. - If s3_client_http_scheme is explicitly set to http, use HTTP for endpoints without a scheme. - Preserve endpoints that already contain an explicit http:// or https:// scheme. - Remove the internal/external bucket classification from BE and Cloud S3 configuration. - Apply the same behavior to FE native S3 clients, the S3 filesystem plugin, connectivity checks, stages, resources, and Iceberg S3 FileIO handoffs. - Normalize scheme-less S3 Resource and Storage Vault endpoint metadata before persistence using the same configured default. Azure is intentionally unchanged in this PR. Its existing provider-specific behavior already defaults a bare endpoint to HTTPS and preserves an explicit scheme; whether Azure should share the S3 compatibility setting is a separate policy decision. ### Release note S3 endpoints without an explicit scheme now use HTTPS by default across FE, BE, and Cloud. Set s3_client_http_scheme = http for compatibility with scheme-less HTTP endpoints. Explicit HTTP and HTTPS endpoint schemes are unchanged. |
||
|
Refrainandliaoxin
|
c8b4a1c6d2 |
[improvement](be) Add CPU-aware S3 rate limiter (#65420)
Add CPU-aware GET/PUT QPS and bandwidth rate limiting for BE object-storage requests. --------- Co-authored-by: liaoxin <liaoxin@selectdb.com> |
||
 Gavin Chouandgavinchou
|
68938c5ca3 |
[fix](s3) Add S3 rate limit observability (#64038)
### What problem does this PR solve? Add observability for S3 local rate limiting and S3 429 responses so throttling-related performance issues can be diagnosed and alerted more directly. ### What is changed? - Add explicit S3 local rate limiter bvars for sleep duration, sleep count, and rejected count. - Rename the old ambiguous S3 local rate limiter bvars so their meaning is clear. - Log the first and every N local S3 rate limiter throttled/rejected requests without reading bvar values for decisions. - Add S3 429 retry/failure bvars for S3/Azure object clients. - Update S3 rate limiter unit coverage for sleep and rejected metrics. ### New or renamed bvar metrics Renamed local S3 GET limiter metrics: - `get_rate_limit_ns` -> `s3_get_rate_limit_sleep_ns` - `get_rate_limit_exceed_req_num` -> `s3_get_rate_limit_sleep_count` Renamed local S3 PUT limiter metrics: - `put_rate_limit_ns` -> `s3_put_rate_limit_sleep_ns` - `put_rate_limit_exceed_req_num` -> `s3_put_rate_limit_sleep_count` New local S3 limiter rejection metrics: - `s3_get_rate_limit_rejected_count` - `s3_put_rate_limit_rejected_count` New S3 429 metrics: - `s3_request_retry_too_many_requests_count` - `s3_request_failed_too_many_requests_count` Metric examples: ```text s3_get_rate_limit_sleep_ns : 123456789 s3_get_rate_limit_sleep_count : 42 s3_get_rate_limit_rejected_count : 3 s3_put_rate_limit_sleep_ns : 987654321 s3_put_rate_limit_sleep_count : 27 s3_put_rate_limit_rejected_count : 1 s3_request_retry_too_many_requests_count : 8 s3_request_failed_too_many_requests_count : 2 ``` Example local limiter logs: ```text S3 GET request is throttled by local rate limiter, sleep_ms=12, sleep_count=1000, token_per_second=1000, bucket_tokens=2000, token_limit=5000 S3 PUT request is rejected by local rate limiter, rejected_count=1, token_per_second=1000, bucket_tokens=2000, token_limit=5000 ``` ### Tests - `sh run-cloud-ut.sh --run --filter=s3_rate_limiter_test:*` - `sh run-be-ut.sh --run --filter=S3FileWriterTest.*` Note: an earlier BE run used an incorrect lowercase filter (`s3_file_writer_test:*`), which matched no suite and started running the full BE suite; it was stopped and replaced by the correct `S3FileWriterTest.*` run above. --------- Co-authored-by: gavinchou <gavinchou@apache.org> |
||
 Yixuan Wang
|
c8ed7602af |
[fix](s3) Avoid retrying object storage SlowDown errors (#63776)
### What problem does this PR solve? Object storage throttling errors can be retried by the SDK retry policy. When requests are already rate limited, these retries add **extra sleep time** and delay the caller from entering the next processing flow. This change disables retry for throttling responses in object storage clients: - S3 `SlowDown` errors are not retried. - Azure 429 `TooManyRequests` is not added to retryable status codes. Other retryable errors keep the existing retry behavior. |
||
|
Yixuan Wang
|
533f02b970 |
[improvement](cloud) Support configurable S3 credentials providers (#62788)
### What problem does this PR solve?
Support configuring the AWS S3 credentials provider type for cloud
storage vaults and S3 storage parameters. This allows IAM role based
access to use providers other than instance profile, such as
environment, system
properties, web identity, container, default chain, and anonymous
credentials. The credential provider type is propagated through FE
properties, thrift/proto definitions, meta service storage vault
handling, and cloud
recycler S3 accessor creation.
```
CREATE storage vault IF NOT EXISTS s3_vault_2
PROPERTIES (
"type"="S3",
"s3.endpoint"="xxx",
"s3.role_arn" = "xxx",
"s3.region" = "us-west-1",
"s3.root.path" = "xxx",
"s3.bucket" = "xxx",
"provider" = "S3",
"use_path_style" = "false",
"s3.credentials_provider_type" = "WEB_IDENTITY",
"s3_validity_check" = "false"
);
```
```
curl "127.0.0.1:5678/MetaService/http/get_obj_store_info
```
```
{
"id": "3",
"name": "s3_vault_2",
"obj_info": {
"ctime": "1777011933",
"mtime": "1777011933",
"id": "3",
"bucket": "xxx",
"prefix": "wyx_test",
"endpoint": "xxx",
"region": "us-west-1",
"provider": "S3",
"external_endpoint": "",
"sse_enabled": true,
"use_path_style": false,
"cred_provider_type": "WEB_IDENTITY",
"role_arn": "xxx",
"external_id": ""
}
}
```
|
||
|
bobhan1
|
eaa4ef92b9 |
[Opt](cloud) Add rate limit for BE to MS rpc (#60344)
This PR implements a two-layer MS (Meta Service) RPC rate limiting
system for Doris cloud mode BE:
1. **Host-level rate limiting** — Token-bucket based QPS limiter for all
21 MS RPC types, preventing a single BE from overwhelming the MS with
burst traffic.
2. **Table-level adaptive backpressure** — When MS returns `MS_BUSY`
error code, BE dynamically identifies and throttles the top-k
highest-QPS tables using a state machine, and automatically relaxes
limits after the pressure subsides.
---
## Part 1: BE Host-Level Rate Limiting
### Problem
In cloud mode, all BE nodes send RPCs (get_tablet, prepare_rowset,
commit_rowset, etc.) to a shared Meta Service. A single BE experiencing
load spikes (e.g., large batch imports, compaction storms) can send
excessive RPC traffic that overwhelms MS, degrading service for all BEs.
### Solution
Introduce `HostLevelMSRpcRateLimiters` — a per-BE, per-RPC-type rate
limiter using token bucket algorithm.
**Architecture:**
- All 21 MS RPC types are enumerated in `MetaServiceRPC` enum (defined
via X-macro for maintainability)
- Each RPC type has an independent `TokenBucketRateLimiterHolder` with
its own QPS limit
- QPS limits are configured per CPU core: `actual_qps = config_value ×
num_cores`
- Thread-safe design using `atomic_shared_ptr<RpcRateLimiter>` array for
lock-free concurrent access during `limit()` calls
- Each rate limiter includes a `bvar::LatencyRecorder` to monitor sleep
durations caused by rate limiting
**Configuration:**
| Config | Default | Description |
|--------|---------|-------------|
| `enable_ms_rpc_host_level_rate_limit` | `true` | Global enable/disable
switch |
| `ms_rpc_qps_default` | `100` | Default per-core QPS for all RPCs |
| `ms_rpc_qps_<rpc_name>` | `-1` | Per-RPC override (`-1` = use default,
`0` = disabled) |
All QPS configs are mutable (`DEFINE_mInt32`), allowing runtime
adjustment without restart. `reset_all()` re-reads configs and recreates
rate limiters.
**Integration:**
Rate limiting is applied inside the `retry_rpc()` template function in
`cloud_meta_mgr.cpp`, which wraps all MS RPC calls. The
`RpcRateLimitCtx` struct carries the rate limiter reference. Rate
limiting executes before each RPC attempt (including retries), with the
call to `apply_rate_limit()` performing a `bthread_usleep` if the token
bucket requires waiting.
**New files:**
- `be/src/cloud/cloud_ms_rpc_rate_limiters.h` / `.cpp`
- `be/test/cloud/cloud_ms_rpc_rate_limiters_test.cpp`
---
## Part 2: BE Table-Level Adaptive Backpressure
### Problem
Host-level rate limiting applies uniformly across all tables. When MS
reports overload (`MAX_QPS_LIMIT`), it's often caused by a small number
of high-traffic tables (e.g., tables with many concurrent stream load
jobs). A uniform rate limit would unnecessarily penalize all tables,
while the hot tables continue to dominate the RPC traffic.
### Solution
Implement table-level adaptive throttling for load-related RPCs. When MS
returns `MAX_QPS_LIMIT`, BE identifies the top-k highest-QPS tables and
progressively reduces their QPS limits, while leaving other tables
unaffected.
**Scope:** Only 5 load-related RPC types participate in table-level
throttling:
- `PREPARE_ROWSET`
- `COMMIT_ROWSET`
- `UPDATE_TMP_ROWSET`
- `UPDATE_PACKED_FILE_INFO`
- `UPDATE_DELETE_BITMAP`
**Architecture (4 components with clear separation of concerns):**
```
MS_BUSY signal (MAX_QPS_LIMIT)
│
▼
┌─────────────────────────┐ ┌──────────────────────────┐
│ RpcThrottleCoordinator │──────▶│ RpcThrottleStateMachine │
│ (timing control) │ │ (pure state logic) │
│ - upgrade cooldown │ │ - upgrade history stack │
│ - downgrade trigger │ │ - limit calculation │
└─────────────────────────┘ └──────────┬───────────────┘
│ Actions
▼
┌─────────────────────────┐ ┌──────────────────────────┐
│ TableRpcQpsRegistry │ │ TableRpcThrottler │
│ (QPS statistics) │ │ (limit enforcement) │
│ - per-table bvar │ │ - StrictQpsLimiter │
│ - top-k query │ │ - per (rpc, table) │
└─────────────────────────┘ └──────────────────────────┘
```
**Component details:**
1. **`TableRpcQpsRegistry`** — Tracks per-(rpc_type, table_id) QPS using
`bvar::PerSecond<bvar::Adder>`. Supports efficient top-k query via
min-heap. Configurable time window via `ms_rpc_table_qps_window_sec`
(immutable, default 10s).
2. **`RpcThrottleStateMachine`** — Pure state machine with no time
awareness or side effects. Maintains upgrade history as a stack for
clean rollback.
- `on_upgrade(snapshot)`: For each top-k table in the QPS snapshot,
calculates `new_limit = current_qps × ratio` (first time) or
`current_limit × ratio` (already limited), with a floor of
`ms_rpc_table_qps_limit_floor`. Returns `SET_LIMIT` actions.
- `on_downgrade()`: Pops the most recent upgrade from history. If the
table had a prior limit, restores it (`SET_LIMIT`). If no prior limit,
removes it (`REMOVE_LIMIT`).
3. **`RpcThrottleCoordinator`** — Timing control layer using tick counts
(1 tick = 1 ms).
- `report_ms_busy()`: Returns true if enough ticks have passed since
last upgrade (cooldown).
- `tick(n)`: Advances time by n ticks. Returns true if downgrade should
trigger (no MS_BUSY for `downgrade_after_ticks`).
4. **`TableRpcThrottler`** — Enforces QPS limits using
`StrictQpsLimiter` (strict fixed-interval, no burst allowed). Each
(rpc_type, table_id) pair has its own limiter. Returns the time point
when the request may execute; the caller sleeps until then.
5. **`MSBackpressureHandler`** — Orchestrator that wires all components
together:
- `on_ms_busy()`: Called when `retry_rpc` receives `MAX_QPS_LIMIT`.
Consults coordinator for cooldown, builds QPS snapshot from registry,
feeds to state machine, applies resulting actions to throttler.
- `before_rpc()` / `after_rpc()`: Called around each load-related RPC
for throttle enforcement and QPS recording.
- Background tick thread: Runs every 1 second, advances coordinator by
1000 ticks. Triggers downgrade when enough time has passed without
MS_BUSY.
**Upgrade/Downgrade lifecycle example:**
```
Time 0s: MS returns MAX_QPS_LIMIT
→ Upgrade level 1: top-2 tables (A: 100 qps, B: 80 qps)
→ A limited to 50 qps, B limited to 40 qps
Time 2s: MS returns MAX_QPS_LIMIT again (cooldown 5s not passed)
→ Skipped
Time 6s: MS returns MAX_QPS_LIMIT (cooldown passed)
→ Upgrade level 2: top-2 tables now (A: 50 qps, C: 60 qps)
→ A limited to 25 qps, C limited to 30 qps
Time 11s: No MS_BUSY for 5s
→ Downgrade: undo level 2
→ A restored to 50 qps, C limit removed
Time 16s: No MS_BUSY for 5s
→ Downgrade: undo level 1
→ A limit removed, B limit removed
```
**Configuration:**
| Config | Default | Mutable | Description |
|--------|---------|---------|-------------|
| `enable_ms_backpressure_handling` | `false` | Yes | Global
enable/disable switch |
| `ms_rpc_table_qps_window_sec` | `3` | No | bvar time window for QPS
calculation |
| `ms_backpressure_upgrade_interval_ms` | `3000` | Yes | Minimum
cooldown between upgrades |
| `ms_backpressure_upgrade_top_k` | `2` | Yes | Number of top tables to
throttle per upgrade |
| `ms_backpressure_throttle_ratio` | `0.75` | Yes | QPS decay ratio on
upgrade |
| `ms_rpc_table_qps_limit_floor` | `1.0` | Yes | Minimum QPS limit
(won't throttle below this) |
| `ms_backpressure_downgrade_interval_ms` | `3000` | Yes | Time without
MS_BUSY before downgrade |
**Observability (bvar metrics):**
- `ms_rpc_backpressure_upgrade_count` / `_60s` — Upgrade event counts
- `ms_rpc_backpressure_downgrade_count` / `_60s` — Downgrade event
counts
- `ms_rpc_backpressure_ms_busy_count` / `_60s` — MS_BUSY signal counts
- `ms_rpc_backpressure_throttle_wait_<rpc_name>` — Per-RPC-type throttle
wait latency
- `ms_rpc_backpressure_throttled_tables_<rpc_name>` — Number of
throttled tables per RPC type
**New files:**
- `be/src/cloud/cloud_throttle_state_machine.h` / `.cpp`
- `be/src/cloud/cloud_ms_backpressure_handler.h` / `.cpp`
- `be/test/cloud/cloud_throttle_state_machine_test.cpp`
- `be/test/cloud/cloud_ms_backpressure_handler_test.cpp`
**Also renamed (not part of the feature, cleanup):**
- `common/cpp/s3_rate_limiter.h/.cpp` →
`common/cpp/token_bucket_rate_limiter.h/.cpp` (more general naming since
it's now used beyond S3)
## Part 3: System Table for Table-Level Throttler Observability
### Problem
The table-level backpressure system operates transparently inside BE.
When issues arise, users and DBAs have no way to inspect which tables
are being throttled, what their QPS limits are, or what their current
QPS is — beyond checking raw bvar metrics.
### Solution
Add a new system table
`information_schema.backend_ms_rpc_table_throttlers` that exposes the
real-time state of the `TableRpcThrottler` on each BE. This table is a
**Backend-Partitioned Schema Table**, meaning each BE reports its own
throttling data, and queries are
distributed to all alive BEs and aggregated.
**Schema:**
| Column | Type | Description |
|--------|------|-------------|
| `BE_ID` | BIGINT | Backend ID |
| `TABLE_ID` | BIGINT | Table ID being throttled |
| `RPC_TYPE` | VARCHAR(64) | RPC type name (e.g., `PREPARE_ROWSET`,
`COMMIT_ROWSET`) |
| `QPS_LIMIT` | DOUBLE | Current QPS limit enforced on this (table, rpc)
pair |
| `CURRENT_QPS` | DOUBLE | Current observed QPS for this (table, rpc)
pair |
**Usage examples:**
```sql
-- View all currently throttled tables across all BEs
SELECT * FROM information_schema.backend_ms_rpc_table_throttlers;
-- View throttled tables on a specific BE
SELECT * FROM information_schema.backend_ms_rpc_table_throttlers WHERE BE_ID = 10001;
-- Find the most severely throttled tables
SELECT * FROM information_schema.backend_ms_rpc_table_throttlers ORDER BY QPS_LIMIT ASC;
```
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- Added host-level token-bucket rate limiting for all MS RPCs (enabled
by default via `enable_ms_rpc_host_level_rate_limit`)
- Added table-level adaptive backpressure handling triggered by MS
`MAX_QPS_LIMIT` response (disabled by default via
`enable_ms_backpressure_handling`)
|
||
|
Yixuan Wang
|
c96f3489d6 |
[fix](sync_point) Fix heap-use-after-free in SyncPoint during program exit (#61040)
### What problem does this PR solve? The SyncPoint singleton was being destroyed during program exit while background bthread timers were still accessing it, causing heap-use-after-free. Changed get_instance() to return a never-destructed heap-allocated pointer instead of a stack-allocated static object to prevent destruction order issues. This is a standard pattern for singletons accessed by background threads. ``` 11:47:40 #0 0x5582a963153d in operator new(unsigned long) (/root/doris/cloud/ut_build_ASAN/test/recycler_test+0x122b53d) (BuildId: 1191affa24589f52) 11:47:40 #1 0x5582ab0aa52f in doris::SyncPoint::SyncPoint() /root/doris/common/cpp/sync_point.cpp:77:9 11:47:40 #2 0x5582ab0aa46b in doris::SyncPoint::get_instance() /root/doris/common/cpp/sync_point.cpp:73:20 11:47:40 #3 0x5582aae33ee8 in doris::cloud::memkv::Transaction::get(std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::unique_ptr<doris::cloud::RangeGetIterator, std::default_delete<doris::cloud::RangeGetIterator>>*, bool, int) /root/doris/cloud/src/meta-store/mem_txn_kv.cpp:276:5 11:47:40 #4 0x5582a968f49f in doris::cloud::txn_get(doris::cloud::TxnKv*, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::unique_ptr<doris::cloud::RangeGetIterator, std::default_delete<doris::cloud::RangeGetIterator>>&) /root/doris/cloud/src/recycler/recycler.cpp:99:18 11:47:40 #5 0x5582a9657090 in doris::cloud::InstanceRecycler::scan_and_recycle(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::basic_string_view<char, std::char_traits<char>>, std::function<int (std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>)>, std::function<int ()>) /root/doris/cloud/src/recycler/recycler.cpp:2976:23 11:47:40 #6 0x5582a9686153 in doris::cloud::InstanceRecycler::recycle_rowsets() /root/doris/cloud/src/recycler/recycler.cpp:2587:15 11:47:40 #7 0x5582a96a8d47 in doris::cloud::RecyclerTest_recycle_empty_Test::TestBody() /root/doris/cloud/test/recycler_test.cpp:1136:5 ``` |
||
|
Gavin Chou
|
d489a79648 |
(tools) Add show_segment_data operation to meta_tool (#60608)
This is an improvement for debug tool meta_tool. Add new operation show_segment_data to meta_tool for reading and displaying segment file column metadata and statistics. - Add get_field_type_string() to convert FieldType to readable string - Add get_encoding_string() to convert EncodingTypePB to readable string - Add get_compression_string() to convert CompressionTypePB to readable string - Add print_column_meta() to print column metadata recursively - Add show_segment_data() to display segment file info and statistics This allows users to inspect segment file structure without requiring the full tablet schema. ``` meta_tool --operation=show_segment_data --file=020000000000004fd4412b500cb0833afcc5dd12e833c59b_0.dat === Segment File Info === File: 020000000000004fd4412b500cb0833afcc5dd12e833c59b_0.dat Num Rows: 2 Num Columns: 16 Compression: LZ4F === column_id_0: type=VARCHAR, nullable=false, encoding=DICT_ENCODING === Dictionary Page: offset=56, size=53 bytes Indexes: ORDINAL, ZONE_MAP Data Values (2 of 2 rows, showing first 2): [0] '10023--1-id' [1] '10023--1-score' === column_id_1: type=VARCHAR, nullable=false, encoding=DICT_ENCODING === Dictionary Page: offset=164, size=25 bytes Indexes: ORDINAL, ZONE_MAP Data Values (2 of 2 rows, showing first 2): [0] '0' [1] '0' ... ``` |
||
|
Yixuan Wang
|
54fff1b4fe |
[Enhancement](client) Supports dynamically changing the rate limiter config (#59465)
This PR adds support for dynamically updating S3 rate limiter configuration at runtime. It adds getter methods to S3RateLimiter and S3RateLimiterHolder, and attempts to trigger rate limiter updates when S3 clients are created. |
||
  zy-kkkandCalvin Kirs
|
86f0b3a089 |
[feat](catalog) Support passing credentials_provider_type to BE for S3 access (#59082)
#58740 This is the backend change for PR #58740, with additional test cases added on top of it. --------- Co-authored-by: Calvin Kirs <guoqiang@selectdb.com> |
||
 Lei Zhang
|
4db09207ae |
[feat](s3client) Use CustomAwsCredentialsProviderChain for aws sdk (#56065)
|
||
 abmdocrt
|
7d6fef7b34 | [Fix](Recycler) Fix recycler azure path fault (#54291) | ||
|
Lei Zhang
|
0648859a0a |
[opt](storage) Add log and metric when aws/azure sdk do retry operation (#51485)
* Print azure sdk log to `be.INFO` * Set aws_log_level=3 (WARN) and azure_log_level=2 (Warning) by default, because when request failed, we don't know what happened in previously * Set azure MaxRetries equal to config::max_s3_client_retry(10) |
||
|
Lei Zhang
|
a02ddbeb3a |
[fix](s3Client) Add ca_cert_file_paths conf for stsClient` and recycler (#50837)
* https://github.com/apache/doris/pull/32285 In previous, the pr add a `ca_cert_file_paths` config for be s3Client, but lack of recycler and stsClient |
||
|
Lei Zhang
|
b7e705b2aa | [feat](s3client) Add role-based authorization for s3client (#49541) | ||
|
Lei Zhang
|
00736e146b |
[opt](s3client) Add AWS SDK logger for recycler with error logging and enhance S3 client tests (#49422)
* Add aws sdk logger for recycler * Default set `aws_log_level=2` for logging aws sdk error * Add more ut for `S3ObjStorageClient` |
||
|
Lei Zhang
|
329d000e35 |
[fix](be/recycler) Fix S3RateLimiter bvar rate_limit_exceed_req_num incorrectly
* Fix S3RateLimiter bvar `rate_limit_exceed_req_num` incorrectly * Add more ut for S3RateLimter |
||
 deardeng
|
7b385c11fa | [fix](log) Fix s3_rate_limiter.cpp LOG incomplete (#48440) | ||
|
Gavin Chou
|
de856351f9 |
[fix](syncpoint) Fix incorrect syncpoint 'has_point()' intorduced by (#43461) (#43784)
Related PR: #43461 |
||
|
deardeng
|
92e4857309 |
[fix](sync point) Fix sync point not reg but construct default value (#43461)
Fix open ENABLE_INJECTION_POINT complie switch， be log
```
W20241107 11:43:11.205868 788 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:11.275723 788 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:11.415167 788 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:11.484220 788 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:11.553351 788 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:11.622395 788 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:11.691509 788 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:11.830601 788 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:11.899286 788 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:11.967985 788 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:13.841727 1032 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:13.945942 1032 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:14.156761 1032 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:38.900395 3064 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:39.397446 3064 status.h:423] meet error status: [IO_ERROR]inject io error
W20241107 11:43:39.896116 3064 status.h:423] meet error status: [IO_ERROR]inject io error
```
For example, in the BE code:
```
TEST_SYNC_POINT_RETURN_WITH_VALUE("LocalFileReader::read_at_impl",
Status::IOError("inject io error"))
```
However, this sync point is not used in the BE, although it is used in
the unit tests. But this macro, when expanded, will construct a default
value, and Status::IOError will print the stack log as soon as it is
constructed.
|
||
|
Lei Zhang
|
936ad96566 |
[fix](cloud) Fix aws sdk default retry policy not taking effect (#43575)
* AWS SDK default retry policy check response xml `code` to determine retry or not, but other object storage such as `OSS`, `COS` not compatible with AWS S3, so adding a custome retry policy for solving the problem |
||
 Lei Zhang
|
895754bf40 | [feat](cloud) Add injection point http api for ms (#41982) | ||
 sparrow
|
4f218b461a |
[Fix](compile) Fix clang-18 and libc++ compile problems (#34715)
## Proposed changes Issue Number: close #xxx Fix clang-18 and libc++ compile problems unique() of shared_ptr is deprecated in clang-18, it's not a strict implementation, can use use_count() instead. And fixed some other compie errors in both using clang-18 and libc++. ## Further comments If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... |
||
 AlexYue
|
ec2cbbc5d2 |
[enhance](RateLimit) Add bvar to monitor object storage rate limit sleep time and failure time (#38294)
Add one bvar to monitor the failed request due to exceeds rate limit. |
||
|
AlexYue
|
c14320cb3c |
[enhance](Azure) Make azure dependency compile one config (#38239)
Previously the azure sdk would be compiled into BE by default. But sometimes the BE would run in environment which only require AWS s3-compatible functionality. So user can use `DISABLE_BUILD_AZURE` to control whether build azure sdk into BE or not. `export DISABLE_BUILD_AZURE=ON` to disable build with azure support |
||
|
zy-kkk
|
ba0eb23db0 |
[fix][compile] Resolve time_point type mismatch on MacOS and Linux (#38049)
revert #38036 We cannot change the time_point type directly, but use macros to distinguish different compilation platforms, refer to #34054 |
||
|
AlexYue
|
529ba61503 |
[Improvement](ObjectStorage) Retry when encountering TooManyRequest response (#37199)
We can do retry for 429 using the embedded utility of corresponding sdk. |
||
 神技圈子
|
91b2030406 |
[fix] (compile) Resolve time_point type mismatch in s3_rate_limiter.cpp in mac (#38036)
A compilation error in `s3_rate_limiter.cpp` due to a type mismatch between `std::chrono::system_clock::time_point` and `std::chrono::high_resolution_clock::time_point`，Updated the `_update_remain_token` function to accept `std::chrono::high_resolution_clock::time_point` instead of `std::chrono::system_clock::time_point` on MacOS。 |
||
|
AlexYue
|
0ad6289e1e |
[feature](Recycler) Add rate limit for all obj storage client implementation (#37663)
We should enable rate limit for all kinds of network IO of recycler. This pr adds the rate limiter for obj storage client. |
||
 plat1ko
|
bca8b07d7b |
[refactor](cloud) Extract SyncPoint to common cpp (#37165)
## Proposed changes Extract SyncPoint to common cpp directory. Some code shared between BE and Cloud will gradually be extracted to the `$repo_root/common/cpp` directory in the future. |