mirror of
https://github.com/apache/doris.git
synced 2026-11-04 16:03:23 +00:00
Jerry Hu
af9dad5b0a
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Large non-streaming window partitions currently retain all input blocks
until the partition closes. A sufficiently large partition can therefore
exhaust BE memory even when query spill is enabled.
This change adds a capability-gated spill path for eligible
non-streaming window functions:
- keeps genuinely streaming windows on the existing in-memory path;
- incrementally detects partition boundaries and stores the original
input blocks in a batch store;
- reduces full-partition aggregates while collecting input, and spills
input blocks on memory revocation, on a configurable threshold, or in
forced-spill mode;
- records bounded peer-group metadata for `percent_rank` and
`cume_dist`, with independent metadata spill;
- publishes all partitions that end inside one input block as a single
`AnalyticSpillBatch` (per-partition row ends, reduce results and ntile
parameters; batch-relative peer group ends), so high-cardinality
partition keys do not create per-partition descriptors, aggregate-state
rebuilds or 1-row output blocks;
- after a batch is spilled, coalesces small blocks into a write buffer
and writes them once it reaches `spill_buffer_size_bytes`, flushing it
on revoke and seal;
- replays batches from the source operator in `batch_size` slices and
tracks peer-group metadata with the query MemTracker; the source
reserves replay memory only for the spilled files it is about to read
(data file and peer-group sidecar of the queued batch, and the next data
or sidecar record during replay), so in-memory batches are consumed
without a disk-read reservation;
- fills peer-function results once per peer/output intersection and
checks cancellation before each fill, without fixed 4096-row splits;
- keeps a sealed batch in memory unless it reaches
`spill_analytic_sink_mem_limit_bytes`, so batches are not written to
disk without memory pressure;
- preserves the existing path when any function or frame is unsupported;
- adds `spill_analytic_sink_mem_limit_bytes` and window-spill profile
counters.
The initial supported set is:
- full-partition `sum`/`sum0`/`count`/`avg` and fixed-length-state
`min`/`max`, including `OVER ()` / `OVER (PARTITION BY ...)` without
ORDER BY, which Nereids plans as `RANGE BETWEEN UNBOUNDED PRECEDING AND
CURRENT ROW` with no order-by expressions (every row is a peer, so the
frame is the whole partition);
- `ntile` with `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`;
- `percent_rank` and `cume_dist` with `RANGE BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW`.
The new source file shifts the BE unity batches so that
`async_result_writer.cpp` shares a translation unit with a file that
includes `local_exchanger.h`. Both declare `clear_blocks` with a default
argument, which fails with "redefinition of default argument"; the
redeclaration in `async_result_writer.cpp` no longer repeats the default
argument (its only call passes both arguments).
Performance (5M unique partition keys, `sum(v) over (partition by k)`,
SQL/query cache off): 0.14s with spill enabled vs 0.26s on the in-memory
path; the source emits ~4K rows per block.
Coalescing spill writes (50M narrow rows,
`spill_analytic_sink_mem_limit_bytes=1MB`, `sum(v) over ()`): spill
blocks 6152 -> 96, serialize time 958ms -> 388ms, spill file 38MB ->
11.8MB, query 1.81s -> 1.26s.
TPC-DS SF10000 (single BE, `exec_mem_limit=8GB`): full-partition
aggregates, ntile, percent_rank/cume_dist over `store_sales` (28.8B
rows) and a 930M-row single partition all completed with the window sink
peak at about 64MB per instance; 2.4B ticket-level partitions ran
entirely through in-memory batches.
### Release note
Eligible non-streaming window functions can spill partitions to local
storage when query spill is enabled.
### Check List (For Author)
- Test
- [x] Regression test
- New `spill_p0/window_function_spill`: 12 queries (full-partition
aggregates with an all-NULL partition, skew, no partition, high
cardinality, ntile, percent_rank/cume_dist, unsupported fallback,
streaming) each run with spill disabled, spill enabled and forced spill;
all three outputs are identical.
- [x] Unit Test
- Earlier full-suite run: `ENABLE_UNITY_BUILD=ON ./run-be-ut.sh --run
--filter='AnalyticSinkOperatorTest.*:BoundaryPoseTest.*'` — 38/38 passed
under ASAN, including write coalescing, replay slicing, revoke flush,
peer-group sidecar files, replay reservation for in-memory / data-only /
data-plus-sidecar batches and sidecar records read during replay, and
the proactive spill limit.
- Earlier full build: `./build.sh --be --fe` — passed.
- `build-support/check-format.sh`,
`build-support/check-build-hygiene.sh`, `git diff --check` — passed.
- [x] Manual test
- Profiles confirm `sum(v) over ()` spilling 5M rows and `partition by
k` using in-memory batches.
- A TPC-DS style validation (store/ticket/date partitions compared with
GROUP BY references) passed on a local scaled-down dataset with and
without spill.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Validation notes:
Latest follow-up (`a225ec4bd70`):
- Peer replay checks cancellation before each peer/output fill; output
is already bounded by `batch_size`, so there is no extra 4096-row split.
- Added focused unit cases for cancellation before filling a large peer,
filling a large peer in one iteration, and cancellation between uneven
peers.
- `BUILD_TYPE=ASAN ENABLE_UNITY_BUILD=ON ./build.sh --be -j16` — passed.
- `ENABLE_UNITY_BUILD=ON ./run-be-ut.sh --run
--filter='AnalyticSinkOperatorTest.Spill*' -j16` — the changed
production and test translation units compiled, but the test executable
could not be rebuilt/run because of existing
`be/test/exprs/function/function_string_test.cpp:2971-2987` compilation
errors (`Field` string construction and removed `FunctionContext` scope
values). The earlier 38/38 result above does not cover this follow-up.
- `build-support/clang-format.sh`, `build-support/check-format.sh`,
`build-support/check-build-hygiene.sh`, and `git diff --check` — passed.
- SQL regression was not rerun for this cancellation-check refactoring.
- Clang-tidy cannot fully run on the local toolchain (`stddef.h` not
found); reported diagnostics on the touched files are pre-existing
cognitive-complexity warnings or false positives from the failed parse.
- Behavior changed:
- [ ] No.
- [x] Yes. Eligible non-streaming analytic partitions can spill;
streaming and unsupported windows retain the existing behavior.
- Does this need documentation?
- [x] No. The implementation design is included in this PR; no
user-facing documentation change is required for this initial version.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
https://claude.ai/code/session_01J4WRvR5E8wQd6ew3LsJoAt