yujun
96f2558e16
### What problem does this PR solve? Issue Number: N/A (tracked in https://github.com/apache/doris/issues/65418) Related PR: #69286 Problem Summary: Adds the generic building blocks that incremental MIN/MAX maintenance builds on; the IVM integration itself is a separate PR. - `min_topk(x, k)` / `max_topk(x, k)`: an aggregate returning `ARRAY<STRUCT<value T, count BIGINT>>` — the k smallest (largest) distinct non-NULL values with their exact row counts, ordered by value. The array carries no truncation flag: a caller that needs one infers it from the length, treating a list of exactly k entries as possibly truncated. With `k = 0` the array holds at most the single `(extreme, count)` pair. `k` is a constant `BIGINT` with `0 <= k <= 65536`. - `array_sorted_merge(merge_fn, arr1, arr2, is_asc)`: a generic higher-order function that merges two arrays already sorted by the elements' natural order. `merge_fn` receives the two current candidates and returns the combined element, or NULL when the two elements must be kept separately. `is_asc` states the sort direction, so one function covers ascending and descending inputs, and calls can be chained to merge more than two arrays. The aggregates keep a sorted `(value, count)` state with checked int64 counting, so partial aggregation and merge stay exact (`topK(A ∪ B) ⊆ topK(A) ∪ topK(B)`). FLOAT/DOUBLE order NaN as greater than every number so the distinct list keeps a strict order. Returning the array alone keeps the aggregate in the same shape as the other array aggregates and leaves the truncation flag to the caller that actually persists a bound: incremental MIN/MAX maintenance still stores one, but as its own state column rather than inside the aggregate result. `array_sorted_merge` keeps its non-lambda `is_asc` argument through the higher-order binding: the existing pipeline passed only the lambda arguments to the bound function, so the binding contract gained a way to return extra inputs. ### Release note Add the `min_topk`/`max_topk` aggregate functions and the `array_sorted_merge` scalar function.
Doris BE development notes
Compile-time hygiene gates
BE configure runs build-support/check-build-hygiene.sh (option
ENABLE_BUILD_HYGIENE, default ON): a set of seconds-level text checks that
keep the backend's compile-time invariants from regressing. They exist because
include-graph regressions are silent -- the code still compiles, only every
build afterwards is slower. The gates make them loud, in the first seconds of
configure, with the mechanism and the fix in the message.
What each gate guards, and what to do when it fires:
- Header layering rules (
check-header-deps.py). A hub header (exec_env.h,thread_context.h, ...) must not reach a named subsystem, because everything a hub includes is reparsed by the ~1000 TUs behind it. Fix: forward-declare the type in the header and include the real header in the.cpp; if only declarations are needed by many files, route them through a*_fwd.h(see below). A genuinely leaf-like header can be added to the rule's exception set -- in the same PR, with the reasoning in the commit message. - Third-party bans. A few headers deliberately moved their
fmt/boost/<ranges>-using bodies out of line; the ban keeps the template machinery from coming back. Fix: put the code that needs the library into the matching.cpp. - Closure / reach budgets. The safety net for edges no rule names: light
hubs must stay light (
--closure <header>lists what grew), heavy payloads must not spread (--reach <header>ranks the spreading edges). Fix: cut the new edge, or -- when the growth is intended -- bump the number in the budget table in the same PR and say why in the commit message. The diff of the budget table is the review signal; the gate is never a dead end. - PCH whitelist.
pch/pch.h's quoted includes are pinned exactly: every header on the PCH rebuilds the whole backend when touched. Adding one is almost never right; if it is, change the whitelist in the same PR. - extern template pairing (
check-extern-template-pairing.py). Everyextern templatedeclaration in a header must have its explicit instantiation definition in a.cpp, and -- within a family that uses externs -- every definition must have its declaration. The missing-extern direction is the one silent case: it compiles and links, every TU just quietly re-instantiates the specialization. Fix: keep both sides in sync, spelling the template arguments the same way. - Unity-skip coverage (
check-unity-skip-coverage.py). A test that#includes a be/src.cppneeds that file opted out of unity batching (doris_skip_unity_inclusion), otherwise the BE UT link fails with duplicate symbols an hour later. The error names the exact entry to add.
The gates are python3 sources and resolve their own interpreter (python3,
then python, then /usr/libexec/platform-python, the RHEL8/AlmaLinux8 system
Python that the build-env image ships in place of /usr/bin/python3). They
deliberately ignore PYTHON, which env.sh exports as the interpreter the
build uses for code generation and which is python2 by default. Set
BUILD_HYGIENE_PYTHON to pin a specific interpreter.
Escape hatch for emergencies: configure with -DENABLE_BUILD_HYGIENE=OFF.
Self-tests live in build-support/tests/ (run.sh).
The *_fwd.h convention
*_fwd.h headers are the sanctioned way through a layering barrier: they
carry forward declarations (and lightweight aliases) only, so they cost
nothing to include and are exempt from the layering rules by suffix. When a
hub header needs a subsystem's type names but not its definitions, put the
declarations in <subsystem>_fwd.h (precedent:
exec/common/hash_table/phmap_fwd_decl.h) and include the real headers only
in .cpp files. Do not put function bodies, class bodies, or includes of
non-fwd project headers into a *_fwd.h.