[opt](build) Enforce BE compile-time hygiene at configure time (#66901)

### What problem does this PR solve?

Related PR: #66510 (the BE build-time series, #66615..#66826, that this
PR locks in)

Problem Summary:

The build-time series cut BE cold builds by ~49% and took
`olap_common.h` / `uid_util.h` off the everything-rebuilds line. Every
one of those wins can regress **silently**: a stray include, a lost
`extern template`, a header added to the PCH — the code still compiles,
only every build afterwards is slower, and nothing reports it. `#66052`
already showed the failure mode: the layering rules in
`check-header-deps.py` only worked because the author happened to
maintain them by hand; nothing in CI runs them.

This PR turns each known regression vector into a **fail-loud,
seconds-level gate that runs at configure time**, so a violation is a
build error on every path that builds the BE (CI, `build.sh`, the UT
build) with zero extra CI wiring — the same mounting pattern the FE side
already uses for its architecture gates (`exec-maven-plugin` running
`check-fe-core-metadata-funnel.sh` at `validate`).

**Checks** (pure text scans, ~1.3s combined, zero build dependency, zero
false positives on current master):

| Gate | Regression it stops |
|---|---|
| Header layering rules (30, +6 new) | a hub header (e.g. `exec_env.h`,
`function.h`, `pch.h`) re-growing an include edge into a subsystem that
was deliberately cut |
| Third-party ban table (new) |
`fmt`/`boost`/`concurrentqueue.h`/`<ranges>` returning to headers whose
bodies were moved out of line to contain template-instantiation cost
(`<ranges>` in a src header also breaks the `-fno-access-control` UT
build on libc++) |
| Two-axis budgets (new) | the edges nobody names: light hubs must stay
light (include-closure budget, zero slack), heavy payloads must not
spread (transitive-reach baseline +10%); `--closure` / `--reach`
subcommands rank the offending edges for the fix |
| pch whitelist (new) | any quoted include added to `pch/pch.h` — the
single most leveraged regression in the repo (touch one such header and
the whole backend plus the PCH rebuild) |
| `check-extern-template-pairing.py` (new) | the one **silent**
extern-template failure: a `.cpp` keeps the explicit instantiation but
the header loses the `extern` declaration, so every TU quietly
re-instantiates; also the loud direction (declaration with no
definition), reported at once instead of at link |
| `check-unity-skip-coverage.py` (new) | a test `#include`-ing a src
`.cpp` that is not opted out of unity batching — reported at configure
with the exact skip entry to add, instead of as duplicate symbols at the
end of the BE UT link |

`build-support/check-build-hygiene.sh` runs all three scripts;
`be/CMakeLists.txt` executes it at configure time. Escape hatches, by
design:

- `-DENABLE_BUILD_HYGIENE=OFF` (default ON; a missing `python3` is a
loud FATAL naming this switch, never a silent skip);
- every budget/whitelist/rule table lives in the scripts themselves, so
the legal way past a gate is a **one-line reviewed diff in the same PR**
— the gate is never a dead end. Every failure message carries the
mechanism (why this edge is expensive) and the concrete fix path.

**Found by the new gates on day one, fixed here:** `core/types.h` has
carried five `extern template struct fmt::formatter<Decimal*>;`
declarations with **no matching definition anywhere** since #53483 —
every member is defined in-class, so the linker never noticed; deleted.
Also made the FE funnel self-test's `sed -i` invocation portable to
BSD/macOS.

Blind spot, by design: instantiations expanded from macros (e.g.
`DECLARE_OPERATOR` in `operator.cpp`) are invisible to the pairing scan
on both sides — macro bodies are skipped; a comment at the site says so.

`be/README.md` (new) documents the gates, what to do when each fires,
and the `*_fwd.h` convention as the sanctioned way through a layering
barrier.

The same rules are encoded for AI-assisted development and review:
`be/AGENTS.md` (new — include discipline while writing BE code + review
checkpoints), `build-support/AGENTS.md` (new — maintenance discipline
for the gate scripts themselves), an include-src-cpp checkpoint in
`be/test/AGENTS.md`, and a Header Hygiene entry in the root `AGENTS.md`,
so coding agents and the review pipeline apply them automatically.

### Release note

None

### Check List (For Author)

- Test
    - [x] Manual test (add detailed scripts or steps below)
- `build-support/check-build-hygiene.sh` on master: all green, ~1.3s,
zero false positives.
- Self-tests `bash build-support/tests/run.sh` (7/7 PASS): red/green
injection per gate family — each seeded violation turns red with the
mechanism + fix in the message, restoring the file turns it green again.
- Real configure on macOS arm64: green path passes the gate section;
with an injected violation, configure dies at the gate in under a second
with the full message; `-DENABLE_BUILD_HYGIENE=OFF` bypasses; a missing
`python3` is a loud exit-2 FATAL naming the switch.
- This PR's own Linux CI exercises the configure mount on the second
platform.

- Behavior changed:
- [x] Yes. BE configure now fails fast (with mechanism + fix in the
message) on header-hygiene violations; emergency bypass via
`-DENABLE_BUILD_HYGIENE=OFF`. No runtime behavior change — the only
source-code changes are comments and the deletion of five never-defined
`extern template` declarations.

- Does this need documentation?
- [x] No. Developer-facing docs are included in-repo (`be/README.md`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01EyZL631YTeuNXa9WLjvk7s

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

AGENTS.md

@@ -30,6 +30,8 @@ All code must pass style checks before committing. Use the corresponding skill f
 
 **BE (C++) Static Analysis**: After building BE (which generates `compile_commands.json`), run `build-support/run-clang-tidy.sh` to check modified C++ files against the `.clang-tidy` config. The script parses `git diff` to filter warnings to changed lines where possible, reducing noise from pre-existing code (diagnostics from included headers may still appear). For Cloud C++ files, pass `--build-dir` pointing to the Cloud compilation database (e.g., `cloud/build_ASAN`). Try to fix all reported warnings; if a warning cannot be reasonably fixed, add a `// NOLINT` comment with justification and report it. See the `clang-tidy-check` skill for details.
 
+**BE (C++) Header Hygiene**: BE configure enforces compile-time hygiene gates (header layering rules, include-closure/reach budgets, pch whitelist, extern-template pairing, unity-skip coverage) via `build-support/check-build-hygiene.sh` (~1s, pure text). Run it directly after touching BE headers, includes, template instantiation lists, or unity skip lists — do not wait for CI. Violations are build errors whose messages carry the mechanism and fix; deliberate budget/whitelist changes are one-line table edits in the same commit with justification. Rules and review checkpoints: `be/AGENTS.md`; mechanics: `be/README.md`.
+
 **FE (Java) Style**: Checkstyle is integrated into the Maven build (`maven-checkstyle-plugin`). Running `build.sh --fe` automatically validates style via `mvn validate`. If checkstyle fails, fix the reported issues according to `fe/check/checkstyle/checkstyle.xml`. See the `fe-code-style` skill for details.
 
 ## Code Review

be/AGENTS.md

@@ -0,0 +1,70 @@
+# AGENTS.md — Doris BE
+
+Scope: `be/`. This file is operational — what to do while developing or
+reviewing BE C++ code so the compile-time invariants hold. The gate mechanics
+and the `*_fwd.h` convention are documented in `be/README.md` (read it when a
+gate fires). Repository-wide rules (commit format, build/test commands,
+clang-format/clang-tidy) live in the root `AGENTS.md`; nothing here overrides
+them.
+
+## Machine-Checked Obligations (configure-time gates)
+
+BE configure runs `build-support/check-build-hygiene.sh` (option
+`ENABLE_BUILD_HYGIENE`, default ON; scripts and self-tests live in
+`build-support/` and `build-support/tests/`). It is pure text and takes about
+a second — **run it directly after any change touching BE headers, includes,
+template instantiations, unity skip lists, or test files that include src
+sources**; do not wait for configure or CI to tell you.
+
+Every failure message carries the mechanism and the fix path. The legal way
+past a gate is never to bypass it, but one of:
+
+- fix the edge as the message says (forward-declare + include in the `.cpp`,
+  or route declarations through a `*_fwd.h`);
+- when the change is deliberate, edit the corresponding table
+  (rule exception set / `ANGLE_BANS` / `FORWARD_CLOSURE_BUDGETS` /
+  `REVERSE_REACH_BASELINES` / `PCH_QUOTED_WHITELIST` / `ALLOW`) **in the same
+  commit** and justify it in the commit message — the table diff is the
+  review signal.
+
+## Rules while writing BE code
+
+- **Adding an `#include` to a widely-included (hub) header is a design
+  decision, not a convenience.** Everything a hub includes is reparsed by
+  every TU behind it (~1000 TUs for `exec_env.h`, `thread_context.h`,
+  `runtime_state.h`, `function.h`, `dependency.h`, `column.h`, ...). Prefer
+  forward declarations; put the real include in the `.cpp`. If many files
+  need the declarations, use a `*_fwd.h` (declarations and lightweight
+  aliases only — never bodies or non-fwd project includes).
+- **Never add a quoted include to `pch/pch.h`.** Every header on the PCH
+  rebuilds the whole backend (plus the PCH itself) when touched; it is the
+  single most leveraged regression surface in the repo.
+- **Keep `extern template` families paired.** An explicit instantiation in a
+  `.cpp` needs the matching `extern template` in the header, spelled with the
+  same template arguments — a missing extern compiles and links fine and just
+  silently re-instantiates in every TU. Instantiations expanded from macros
+  (e.g. `DECLARE_OPERATOR` in `operator.cpp`) are invisible to the pairing
+  gate on both sides: keep those in sync by hand.
+- **A test that `#include`s a be/src `.cpp`** needs that file opted out of
+  unity batching via `doris_skip_unity_inclusion` in the owning
+  `be/src/.../CMakeLists.txt`; otherwise the BE UT link fails with duplicate
+  symbols an hour later. The gate error names the exact entry to add.
+- **Do not re-add banned third-party includes** (`fmt`/`boost`/
+  `concurrentqueue.h`/`<ranges>`) to the headers listed in `ANGLE_BANS`:
+  their bodies were deliberately moved out of line, and `<ranges>` in a src
+  header additionally breaks the `-fno-access-control` UT build on libc++.
+
+## Review checkpoints (AI review and self-review)
+
+- [ ] New includes in hub headers: could a forward declaration or `*_fwd.h`
+      carry this instead? Does the PR pay a closure/reach budget bump — and
+      if so, does the commit message justify it?
+- [ ] Any edit to a gate table (`RULES` exceptions, budgets, whitelist,
+      `ALLOW`) must be deliberate, minimal, and explained in the same
+      commit; an unexplained table edit is a red flag, not a fix.
+- [ ] Any change to `pch/pch.h` is near-always wrong; demand the reasoning.
+- [ ] New explicit instantiation lists or `extern template` blocks: both
+      sides present, same spelling? New test `#include` of a src `.cpp`:
+      skip entry present?
+- [ ] `git grep` for a deleted/renamed header in skip lists and gate tables:
+      stale entries fail configure loudly — fix them in the same PR.

be/CMakeLists.txt

@@ -159,6 +159,32 @@ endif()
 set(THIRDPARTY_DIR "$ENV{DORIS_THIRDPARTY}/installed")
 message(STATUS "THIRDPARTY_DIR is ${THIRDPARTY_DIR}")
 
+# Build-hygiene gates: seconds-level, zero-build-dependency text checks that
+# keep the backend's compile-time invariants from regressing (header layering
+# rules and budgets, extern-template pairing, unity-skip coverage for
+# test-included sources). Running them at configure time makes a violation a
+# build error on every path that builds the BE -- CI, build.sh, the UT build --
+# with no separate wiring, and it fails in the first seconds instead of
+# somewhere inside a long compile. Every failure message carries the mechanism
+# and the fix path; see build-support/check-build-hygiene.sh.
+option(ENABLE_BUILD_HYGIENE "Run the build-hygiene gates at configure time" ON)
+message(STATUS "ENABLE_BUILD_HYGIENE is ${ENABLE_BUILD_HYGIENE}")
+if (ENABLE_BUILD_HYGIENE)
+    execute_process(
+        COMMAND bash "${BASE_DIR}/../build-support/check-build-hygiene.sh"
+        ERROR_VARIABLE BUILD_HYGIENE_ERRORS
+        RESULT_VARIABLE BUILD_HYGIENE_RESULT
+    )
+    if (NOT BUILD_HYGIENE_RESULT EQUAL 0)
+        message(FATAL_ERROR
+            "build-hygiene gates failed (exit ${BUILD_HYGIENE_RESULT}):\n"
+            "${BUILD_HYGIENE_ERRORS}"
+            "Fix the violation (each message ends with its fix path), or "
+            "configure with -DENABLE_BUILD_HYGIENE=OFF to bypass in an "
+            "emergency.")
+    endif()
+endif()
+
 option(MAKE_TEST "ON for make unit test or OFF for not" OFF)
 message(STATUS "make test: ${MAKE_TEST}")
 option(BUILD_BENCHMARK "ON for make google benchmark or OFF for not" OFF)

be/README.md

@@ -0,0 +1,85 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Doris BE development notes
+
+## Compile-time hygiene gates
+
+BE configure runs `build-support/check-build-hygiene.sh` (option
+`ENABLE_BUILD_HYGIENE`, default `ON`): a set of seconds-level text checks that
+keep the backend's compile-time invariants from regressing. They exist because
+include-graph regressions are silent -- the code still compiles, only every
+build afterwards is slower. The gates make them loud, in the first seconds of
+configure, with the mechanism and the fix in the message.
+
+What each gate guards, and what to do when it fires:
+
+* **Header layering rules** (`check-header-deps.py`). A hub header
+  (`exec_env.h`, `thread_context.h`, ...) must not reach a named subsystem,
+  because everything a hub includes is reparsed by the ~1000 TUs behind it.
+  Fix: forward-declare the type in the header and include the real header in
+  the `.cpp`; if only declarations are needed by many files, route them
+  through a `*_fwd.h` (see below). A genuinely leaf-like header can be added
+  to the rule's exception set -- in the same PR, with the reasoning in the
+  commit message.
+* **Third-party bans**. A few headers deliberately moved their `fmt`/`boost`/
+  `<ranges>`-using bodies out of line; the ban keeps the template machinery
+  from coming back. Fix: put the code that needs the library into the matching
+  `.cpp`.
+* **Closure / reach budgets**. The safety net for edges no rule names: light
+  hubs must stay light (`--closure <header>` lists what grew), heavy payloads
+  must not spread (`--reach <header>` ranks the spreading edges). Fix: cut the
+  new edge, or -- when the growth is intended -- bump the number in the budget
+  table in the same PR and say why in the commit message. The diff of the
+  budget table is the review signal; the gate is never a dead end.
+* **PCH whitelist**. `pch/pch.h`'s quoted includes are pinned exactly: every
+  header on the PCH rebuilds the whole backend when touched. Adding one is
+  almost never right; if it is, change the whitelist in the same PR.
+* **extern template pairing** (`check-extern-template-pairing.py`). Every
+  `extern template` declaration in a header must have its explicit
+  instantiation definition in a `.cpp`, and -- within a family that uses
+  externs -- every definition must have its declaration. The missing-extern
+  direction is the one silent case: it compiles and links, every TU just
+  quietly re-instantiates the specialization. Fix: keep both sides in sync,
+  spelling the template arguments the same way.
+* **Unity-skip coverage** (`check-unity-skip-coverage.py`). A test that
+  `#include`s a be/src `.cpp` needs that file opted out of unity batching
+  (`doris_skip_unity_inclusion`), otherwise the BE UT link fails with
+  duplicate symbols an hour later. The error names the exact entry to add.
+
+The gates are python3 sources and resolve their own interpreter (`python3`,
+then `python`, then `/usr/libexec/platform-python`, the RHEL8/AlmaLinux8 system
+Python that the build-env image ships in place of `/usr/bin/python3`). They
+deliberately ignore `PYTHON`, which `env.sh` exports as the interpreter the
+*build* uses for code generation and which is python2 by default. Set
+`BUILD_HYGIENE_PYTHON` to pin a specific interpreter.
+
+Escape hatch for emergencies: configure with `-DENABLE_BUILD_HYGIENE=OFF`.
+Self-tests live in `build-support/tests/` (`run.sh`).
+
+## The `*_fwd.h` convention
+
+`*_fwd.h` headers are the sanctioned way through a layering barrier: they
+carry forward declarations (and lightweight aliases) only, so they cost
+nothing to include and are exempt from the layering rules by suffix. When a
+hub header needs a subsystem's type names but not its definitions, put the
+declarations in `<subsystem>_fwd.h` (precedent:
+`exec/common/hash_table/phmap_fwd_decl.h`) and include the real headers only
+in `.cpp` files. Do not put function bodies, class bodies, or includes of
+non-fwd project headers into a `*_fwd.h`.

be/src/core/column/column_decimal.cpp

@@ -591,6 +591,8 @@ typename ColumnDecimal<T>::CppNativeType ColumnDecimal<T>::get_fractional_part(s
     }
 }
 
+// A new instantiation here needs the matching 'extern template' declaration
+// in column_decimal.h (enforced by check-extern-template-pairing.py).
 template class ColumnDecimal<TYPE_DECIMAL32>;
 template class ColumnDecimal<TYPE_DECIMAL64>;
 template class ColumnDecimal<TYPE_DECIMALV2>;

be/src/core/column/column_string.cpp

@@ -774,6 +774,8 @@ bool ColumnStr<T>::is_valid_utf8() const {
     return true;
 }
 
+// A new instantiation here needs the matching 'extern template' declaration
+// in column_string.h (enforced by check-extern-template-pairing.py).
 template class ColumnStr<uint32_t>;
 template class ColumnStr<uint64_t>;
 } // namespace doris

be/src/core/column/column_vector.cpp

@@ -545,6 +545,10 @@ void ColumnVector<T>::replace_float_special_values() {
 }
 
 /// Explicit template instantiations - to avoid code bloat in headers.
+/// A new instantiation here needs the matching 'extern template' declaration
+/// in column_vector.h: without it, every TU that uses the specialization
+/// silently instantiates its own copy again (enforced by
+/// build-support/check-extern-template-pairing.py at configure time).
 template class ColumnVector<TYPE_BOOLEAN>;
 template class ColumnVector<TYPE_TINYINT>;
 template class ColumnVector<TYPE_SMALLINT>;

be/src/core/types.h

@@ -626,9 +626,3 @@ struct fmt::formatter<doris::Decimal<T>> {
         return fmt::format_to(ctx.out(), "{}", to_string(value.value));
     }
 };
-
-extern template struct fmt::formatter<doris::Decimal32>;
-extern template struct fmt::formatter<doris::Decimal64>;
-extern template struct fmt::formatter<doris::Decimal128V2>;
-extern template struct fmt::formatter<doris::Decimal128V3>;
-extern template struct fmt::formatter<doris::Decimal256>;

be/src/exec/operator/operator.cpp

@@ -819,6 +819,11 @@ Status AsyncWriterSink<Writer, Parent>::close(RuntimeState* state, Status exec_s
     return Base::close(state, exec_status);
 }
 
+// An instantiation added outside the macros below needs the matching
+// 'extern template' declaration in the operator's header (enforced by
+// build-support/check-extern-template-pairing.py). Instantiations expanded
+// through DECLARE_OPERATOR are invisible to that check -- macro bodies are
+// skipped -- so their extern pairing has to be kept in sync by hand.
 #define DECLARE_OPERATOR(LOCAL_STATE) template class DataSinkOperatorX<LOCAL_STATE>;
 DECLARE_OPERATOR(HashJoinBuildSinkLocalState)
 DECLARE_OPERATOR(ResultSinkLocalState)

be/test/AGENTS.md

@@ -3,3 +3,7 @@
 ## Access Control
 
 BE-UT has actually been configured to ignore access control, so it can access all private interfaces.
+
+## Including src .cpp files
+
+- [ ] A test that `#include`s a be/src `.cpp` (to reach file-static helpers) must have that file opted out of unity batching via `doris_skip_unity_inclusion` in the owning `be/src/.../CMakeLists.txt` — otherwise the batch object's copy of the definitions collides with the test's inlined copy and the BE UT link fails with duplicate symbols. Enforced at configure time by `build-support/check-unity-skip-coverage.py`; its error message names the exact entry to add. Prefer not including `.cpp` files at all when the code can be exercised through a header.

build-support/AGENTS.md

@@ -0,0 +1,49 @@
+# AGENTS.md — build-support
+
+Operational rules for the architecture/hygiene gates in this directory (the
+`check-*.sh` / `check-*.py` scripts wired into FE `mvn validate` and BE
+configure). These scripts fail other people's builds — treat them as
+production code with a stricter bar than the code they guard.
+
+## Gate maintenance discipline
+
+- **Zero false positives, non-negotiable.** A per-commit gate must be
+  deterministic, pure-text, and second-level. Anything heuristic
+  (nm audits, closure sweeps, wall-clock trends) belongs in offline reports,
+  never in the configure/validate path. One false positive burns more trust
+  than ten missed regressions.
+- **Fail loud, never silently skip.** A missing tool, a table entry pointing
+  at a renamed file, a sentinel nobody references any more — each is an
+  error with a message, not a silent pass (precedents:
+  `doris_skip_unity_inclusion` in `be/CMakeLists.txt`, the missing-python3
+  branch of `check-build-hygiene.sh`, the missing-header errors in
+  `check-header-deps.py`).
+- **Every failure message carries three parts**: the violation, the
+  *mechanism* (why this edge/entry is expensive — model:
+  `check-header-deps.py`'s reason/chain/fix form), and a concrete fix path
+  the reader can act on without opening the script.
+- **Escape hatches are tables in the script, not flags.** Budgets,
+  whitelists, exception sets and `ALLOW` lists live next to the rules so a
+  deliberate change is a one-line reviewed diff in the same commit. Do not
+  add bypass environment variables or config files.
+- **Rebaselining budgets**: forward closure budgets carry zero slack (bump =
+  explicit, justified edit); reverse reach baselines carry +10% and are
+  re-measured on the audit cadence with `check-header-deps.py --budget`.
+
+## Changing a gate script
+
+- Any behavior change to a `check-*` script requires updating its self-test
+  in `build-support/tests/` (red/green injection form: every seeded
+  violation turns red with the fix path in the message, restoring turns it
+  green) and running `bash build-support/tests/run.sh` — all of it, since
+  the entry scripts aggregate.
+- The BE hygiene self-tests briefly mutate working-tree files (backed up and
+  restored by EXIT traps): do not run them concurrently with a
+  build/configure of the same tree, and do not "fix" them by pointing at
+  fixtures — the gate tables name real headers on purpose.
+- Keep scripts portable: bash 3.2 (macOS), BSD *and* GNU userland — in
+  particular `sed -i` needs the `-i.bak` + `rm` form. Python: stdlib only,
+  no third-party imports.
+- Performance envelope: the combined configure gate is ~1s today; keep any
+  addition within a low single-digit second budget, with zero build
+  dependency (no compiler, no compile_commands.json).

build-support/check-build-hygiene.sh

@@ -0,0 +1,118 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Unified entry for the BE build-hygiene gates. Runs every seconds-level,
+# zero-build-dependency check that protects the backend's compile-time
+# invariants:
+#
+#   check-header-deps.py             header layering rules, third-party bans,
+#                                    the two-axis closure/reach budgets and the
+#                                    pch include lock
+#   check-extern-template-pairing.py extern template declarations and explicit
+#                                    instantiation definitions stay paired in
+#                                    both directions
+#   check-unity-skip-coverage.py     every src .cpp a test #includes is opted
+#                                    out of unity batching
+#
+# All checks are pure text scans and run in about a second combined. Every
+# failure message carries the mechanism (why the edge/entry is expensive) and
+# the fix path. All checks run even when an early one fails, so one pass shows
+# everything there is to fix.
+#
+# Mounted at configure time by be/CMakeLists.txt (ENABLE_BUILD_HYGIENE, default
+# ON), which is what makes a violation a build error rather than advice. To run
+# by hand:
+#
+#   build-support/check-build-hygiene.sh
+#
+# Exit code: 0 -- all checks passed; 1 -- at least one violation (details on
+# stderr); 2 -- environment problem (no Python 3 interpreter).
+#
+# Self-tests: build-support/tests/test-build-hygiene-*.sh (run.sh runs them).
+
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# The gates are python3 sources, so they need a Python 3 interpreter -- which is
+# not the interpreter the build itself uses. env.sh exports PYTHON as
+# DORIS_BUILD_PYTHON_VERSION, defaulting to `python`; on the official build-env
+# image (FROM almalinux:8) that is python2.7, under which every gate dies with a
+# SyntaxError on its first f-string. So resolve an interpreter here and never
+# read PYTHON.
+#
+# The probe order covers the two shapes that exist in practice: `python3` on
+# developer boxes and most distros, and /usr/libexec/platform-python -- the
+# RHEL8/AlmaLinux8 system interpreter (3.6), which the build-env image ships
+# while carrying no /usr/bin/python3 at all.
+#
+# BUILD_HYGIENE_PYTHON names an interpreter explicitly (a venv, a non-standard
+# prefix). It selects which python runs the gates; it cannot switch them off --
+# that is -DENABLE_BUILD_HYGIENE=OFF.
+is_python3() {
+    "$1" -c 'import sys; sys.exit(0 if sys.version_info >= (3, 6) else 1)' \
+        >/dev/null 2>&1
+}
+
+PYTHON=""
+if [ -n "${BUILD_HYGIENE_PYTHON:-}" ]; then
+    # An explicit choice is honoured strictly: if it is unusable, say so rather
+    # than quietly running the gates under some other interpreter.
+    if is_python3 "${BUILD_HYGIENE_PYTHON}"; then
+        PYTHON="${BUILD_HYGIENE_PYTHON}"
+    else
+        echo "check-build-hygiene: BUILD_HYGIENE_PYTHON=${BUILD_HYGIENE_PYTHON} is not found, or is not Python >= 3.6." >&2
+        echo "Point it at a Python 3 interpreter, or re-run with -DENABLE_BUILD_HYGIENE=OFF to skip the gates." >&2
+        exit 2
+    fi
+else
+    for candidate in python3 python /usr/libexec/platform-python; do
+        if is_python3 "${candidate}"; then
+            PYTHON="${candidate}"
+            break
+        fi
+    done
+    if [ -z "${PYTHON}" ]; then
+        echo "check-build-hygiene: no Python >= 3.6 found (tried python3, python, /usr/libexec/platform-python); the build-hygiene gates need one." >&2
+        echo "Install python3 or set BUILD_HYGIENE_PYTHON, or re-run with -DENABLE_BUILD_HYGIENE=OFF to skip the gates." >&2
+        exit 2
+    fi
+fi
+
+FAILURES=0
+for check in \
+    check-header-deps.py \
+    check-extern-template-pairing.py \
+    check-unity-skip-coverage.py; do
+    if ! "${PYTHON}" "${SCRIPT_DIR}/${check}"; then
+        FAILURES=$((FAILURES + 1))
+    fi
+done
+
+if [ "${FAILURES}" -ne 0 ]; then
+    echo "" >&2
+    echo "build hygiene: ${FAILURES} check(s) failed (details above)." >&2
+    echo "Each message ends with its fix path; the budget/whitelist tables live in the" >&2
+    echo "check scripts themselves, so a deliberate change is a one-line, reviewed diff." >&2
+    exit 1
+fi
+# Name the resolved interpreter: it is the one thing about this run that varies
+# per environment, and the configure log is where you look when a gate behaves
+# differently in CI than it did locally.
+echo "build hygiene: all checks passed (python: ${PYTHON} $("${PYTHON}" -c 'import sys; print("%d.%d.%d" % sys.version_info[:3])'))"

build-support/check-extern-template-pairing.py

@@ -0,0 +1,271 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Pairing guard for `extern template` families.
+
+The BE uses extern-template families (ColumnVector, ColumnDecimal, the operator
+and serde stacks, ...) to instantiate hot template specializations once instead
+of in every translation unit. Of all the ways such a pairing can rot, exactly
+one is silent: a .cpp carries the explicit instantiation definition but the
+header lost (or never gained) the matching `extern template` declaration. The
+code still compiles and links -- every TU just quietly goes back to implicitly
+instantiating that specialization, and the build slows down with no diagnostic
+anywhere. This script turns both pairing directions into an error:
+
+  forward   every `extern template` declaration in a be/src header must have a
+            matching explicit instantiation definition in some be/src .cpp
+            (missing ones eventually break the link, but only when a TU odr-uses
+            a non-inline member -- this reports them at once, with the file the
+            definition should go in).
+  reverse   within a COVERED family -- a template name that already has at least
+            one extern declaration -- every explicit instantiation definition
+            must have a matching extern declaration (the silent case). Families
+            with no externs at all are intentionally out of scope: instantiation
+            without extern is a legitimate TU-local idiom there.
+
+Matching is textual, on a normalized signature: whitespace folded, comments
+stripped, namespace qualifiers dropped, known type aliases canonicalized, and
+multi-line statements joined. Zero build dependency; runs in about half a
+second.
+
+Usage:
+    build-support/check-extern-template-pairing.py            # enforce
+    build-support/check-extern-template-pairing.py --list     # dump the pairing tables
+"""
+
+import argparse
+import collections
+import os
+import re
+import sys
+
+SRC_ROOT = "be/src"
+
+# Spellings that legitimately differ between a declaration site and its
+# definition site are rewritten to one canonical form ON BOTH SIDES before
+# matching. Because the rewrite is uniform, a pair that already spells its
+# arguments consistently can never be broken by an entry here; entries are only
+# ever needed when the header and the .cpp spell the same type differently.
+# Keep this table small -- the better fix for a new mismatch is to spell both
+# sides identically.
+TYPE_ALIASES = {
+    # core/types.h vectorized aliases vs the underlying fixed-width types
+    # (column_string.h declares ColumnStr<UInt32>, column_string.cpp
+    # instantiates ColumnStr<uint32_t>).
+    "UInt32": "uint32_t",
+    "UInt64": "uint64_t",
+    # wide:: aliases vs the underlying wide::integer specializations
+    # (wide_integer_to_string.h declares to_string(const Int128&), the .cpp
+    # instantiates to_string(const integer<128, signed>&)).
+    "Int128": "integer<128,signed>",
+    "UInt128": "integer<128,unsigned>",
+    "Int256": "integer<256,signed>",
+    "UInt256": "integer<256,unsigned>",
+}
+
+# Escape hatch, deliberately empty. An entry is
+#     ("be/src/path/to/file.ext", "<normalized signature>")
+# as printed by a failure (or by --list), and silences that one declaration or
+# definition. Adding one must be a reviewed decision with a comment saying why
+# the pairing is intentionally broken.
+ALLOW = set()
+
+EXTERN_START = re.compile(r"^\s*extern\s+template\b")
+# An explicit instantiation begins `template` followed by anything but `<`
+# (`template <...>` opens a template definition, not an instantiation).
+DEF_START = re.compile(r"^\s*template\s+[A-Za-z_:]")
+LINE_COMMENT = re.compile(r"//.*$")
+BLOCK_COMMENT = re.compile(r"/\*.*?\*/")
+
+
+def statements(path, start):
+    """Yields (lineno, statement) for namespace-scope statements matching
+    `start`, joining continuation lines up to the terminating semicolon and
+    skipping preprocessor directives and macro bodies (an instantiation-shaped
+    line inside a #define is not an instantiation)."""
+    with open(path, encoding="utf-8", errors="ignore") as handle:
+        lines = handle.read().splitlines()
+    in_macro = False
+    i = 0
+    while i < len(lines):
+        raw = lines[i]
+        if in_macro or raw.lstrip().startswith("#"):
+            in_macro = raw.rstrip().endswith("\\")
+            i += 1
+            continue
+        if not start.match(raw):
+            i += 1
+            continue
+        stmt, j = raw, i
+        while ";" not in stmt and j + 1 < len(lines) and j - i < 20:
+            j += 1
+            stmt += " " + lines[j]
+        if ";" in stmt and "\\" not in stmt:
+            yield i + 1, stmt.split(";")[0].strip()
+        i = j + 1
+
+
+def normalize(sig):
+    """Collapses a declaration/definition to the comparable skeleton."""
+    s = BLOCK_COMMENT.sub(" ", sig)
+    s = LINE_COMMENT.sub(" ", s)
+    s = re.sub(r"\s+", " ", s).strip()
+    s = re.sub(r"^extern\s+", "", s)
+    s = re.sub(r"^template\s+", "", s)
+    s = re.sub(r"^(class|struct)\s+", "", s)
+    s = re.sub(r"\b[A-Za-z_]\w*::", "", s)
+    for alias, canonical in TYPE_ALIASES.items():
+        s = re.sub(rf"\b{alias}\b", canonical, s)
+    return s.replace(" ", "")
+
+
+def family(norm):
+    """The template name a normalized signature instantiates: the identifier
+    before the argument list for a type, the function name for a function."""
+    paren = norm.find("(")
+    if paren != -1:
+        m = re.search(r"([A-Za-z_]\w*)(?:<[^()]*>)?\($", norm[: paren + 1])
+        if m:
+            return m.group(1)
+    m = re.match(r"([A-Za-z_]\w*)<", norm)
+    return m.group(1) if m else norm
+
+
+def walk(suffixes):
+    for directory, _, names in os.walk(SRC_ROOT):
+        for name in names:
+            if name.endswith(suffixes):
+                yield os.path.join(directory, name)
+
+
+def collect():
+    externs, defs = [], []
+    for path in walk((".h", ".hpp")):
+        with open(path, encoding="utf-8", errors="ignore") as handle:
+            if "extern template" not in handle.read():
+                continue
+        for lineno, stmt in statements(path, EXTERN_START):
+            externs.append((path, lineno, stmt, normalize(stmt)))
+    for path in walk((".cpp", ".cc")):
+        for lineno, stmt in statements(path, DEF_START):
+            if re.match(r"^\s*extern\b", stmt):
+                continue
+            defs.append((path, lineno, stmt, normalize(stmt)))
+    return externs, defs
+
+
+def enforce(externs, defs):
+    def_index = collections.defaultdict(list)
+    for path, lineno, stmt, norm in defs:
+        def_index[norm].append((path, lineno))
+    extern_index = collections.defaultdict(list)
+    covered = collections.defaultdict(set)  # family -> headers declaring it
+    for path, lineno, stmt, norm in externs:
+        extern_index[norm].append((path, lineno))
+        covered[family(norm)].add(path)
+
+    failures = 0
+    for path, lineno, stmt, norm in externs:
+        if norm in def_index or (path, norm) in ALLOW:
+            continue
+        failures += 1
+        print(
+            "error: extern template declaration pairs with no explicit "
+            "instantiation definition",
+            file=sys.stderr,
+        )
+        print(f"  decl:   {path}:{lineno}: {stmt};", file=sys.stderr)
+        print(f"  norm:   {norm}", file=sys.stderr)
+        print(
+            "  reason: 'extern template' promises the specialization is "
+            "instantiated in some .cpp; without one, any TU that odr-uses a "
+            "non-inline member fails to link -- and if every member is inline, "
+            "the declaration is dead weight",
+            file=sys.stderr,
+        )
+        print(
+            "  fix:    add the matching 'template class/struct/... ;' to the "
+            ".cpp that owns this family, or delete the declaration; an "
+            "intentionally unpaired declaration goes in ALLOW with a comment",
+            file=sys.stderr,
+        )
+    for path, lineno, stmt, norm in defs:
+        fam = family(norm)
+        if fam not in covered or norm in extern_index or (path, norm) in ALLOW:
+            continue
+        failures += 1
+        headers = ", ".join(sorted(covered[fam]))
+        print(
+            "error: explicit instantiation lacks the matching 'extern "
+            "template' declaration",
+            file=sys.stderr,
+        )
+        print(f"  def:    {path}:{lineno}: {stmt};", file=sys.stderr)
+        print(f"  norm:   {norm}", file=sys.stderr)
+        print(f"  family: {fam} is extern-covered in {headers}", file=sys.stderr)
+        print(
+            "  reason: every extern-declared specialization of this family is "
+            "instantiated once, but this one re-instantiates in every "
+            "including TU -- the one pairing mistake that is silent: it "
+            "compiles, links, and only slows the build down",
+            file=sys.stderr,
+        )
+        print(
+            "  fix:    add 'extern template ...;' next to the family's other "
+            "externs in the header above, spelling the template arguments the "
+            "same way as the definition, or list the definition in ALLOW with "
+            "a comment",
+            file=sys.stderr,
+        )
+    return failures
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--list",
+        action="store_true",
+        help="dump every declaration and definition with its normalized form",
+    )
+    args = parser.parse_args()
+
+    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    os.chdir(root)
+    externs, defs = collect()
+
+    if args.list:
+        for path, lineno, stmt, norm in externs:
+            print(f"decl {path}:{lineno}\n     {norm}")
+        for path, lineno, stmt, norm in defs:
+            print(f"def  {path}:{lineno}\n     {norm}")
+        return 0
+
+    failures = enforce(externs, defs)
+    if failures:
+        print(f"\n{failures} extern/instantiation pairing violation(s)", file=sys.stderr)
+        return 1
+    families = {family(n) for _, _, _, n in externs}
+    print(
+        f"extern/instantiation pairing: {len(externs)} declaration(s), "
+        f"{len(families)} covered family(s), all paired"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

build-support/check-header-deps.py

@@ -24,12 +24,30 @@ every translation unit, so one stray include can multiply rebuild cost by an ord
 of magnitude and does so silently: the code still compiles, only the build slows
 down. This script turns that into a build error instead.
 
-Each rule names a hub header and a directory prefix it must not reach. Keeping a
-hub out of a subsystem is what lets that subsystem's headers be edited cheaply.
+Four kinds of guard, all on the same text include graph (be/src + be/test, all
+ifdef branches -- not comparable to ninja-measured radii, each baseline compares
+only with itself):
+
+  RULES                    a hub header must not reach a subsystem. Keeping a
+                           hub out of a subsystem is what lets that subsystem's
+                           headers be edited cheaply.
+  ANGLE_BANS               a header must not include a named third-party header
+                           whose template machinery was deliberately moved out
+                           of line.
+  FORWARD/REVERSE budgets  the two-axis safety net for edges no rule names:
+                           light hubs must stay light (closure budget, zero
+                           slack), heavy payloads must not spread (reach
+                           baseline +10%).
+  PCH whitelist            pch/pch.h's quoted project includes are pinned
+                           exactly; the PCH is the single most leveraged
+                           regression surface in the repo.
 
 Usage:
-    build-support/check-header-deps.py            # enforce the rules
-    build-support/check-header-deps.py --report   # rank headers by rebuild cost
+    build-support/check-header-deps.py                  # enforce everything
+    build-support/check-header-deps.py --report         # rank headers by rebuild cost
+    build-support/check-header-deps.py --budget         # budgets only (rebaselining)
+    build-support/check-header-deps.py --closure HEADER # list a hub's closure, with chains
+    build-support/check-header-deps.py --reach HEADER   # rank an includer's edge weights
 """
 
 import argparse
@@ -288,19 +306,213 @@ RULES = [
         "rec_cte_shared_state.cpp; the brpc client stack must not ride a "
         "SharedState header",
     ),
+    (
+        "exprs/function/function.h",
+        "storage/",
+        {
+            # Plain result struct (<cstdint> only); the sanctioned carrier left
+            # behind when the zonemap machinery edge was cut.
+            "storage/index/zone_map/zonemap_filter_result.h",
+            # Plain id struct, methods defined in rowset_id.cpp; rides in via
+            # core/column/column.h.
+            "storage/rowset_id.h",
+        },
+        "function.h is the base header of every scalar function; cutting its "
+        "three storage edges (expr_zonemap_filter, inverted_index_iterator, "
+        "inverted_index_parser) removed 122,885 preprocessed lines from the "
+        "exprs TUs, and one edge flowing back re-couples the whole expression "
+        "layer to the storage stack",
+    ),
+    (
+        "core/field.h",
+        "util/json/path_in_data.h",
+        set(),
+        "a single using-alias here used to drag path_in_data.h and with it "
+        "gen_cpp/segment_v2.pb.h (11.6k lines) into every TU that sees a "
+        "Field; the alias users include path_in_data.h themselves now",
+    ),
+    (
+        "core/data_type/primitive_type.h",
+        "util/json/path_in_data.h",
+        set(),
+        "primitive_type.h is included by ~44 project headers transitively; "
+        "the path_in_data edge would put gen_cpp/segment_v2.pb.h behind the "
+        "most basic type-enum header in the backend",
+    ),
+    (
+        "core/value/variant/variant_field.h",
+        "util/json/path_in_data.h",
+        set(),
+        "the field.h cut moved the variant alias down here, so this header "
+        "is where the path_in_data edge would most naturally regrow; variant "
+        "consumers that need PathInData include it directly",
+    ),
+    (
+        "core/types.h",
+        "storage/olap_common.h",
+        set(),
+        "the int128/uint128 typedefs moved to core/extended_types.h precisely "
+        "so core/types.h stops paying for the storage domain; the binary_cast "
+        "include chain used to put all of olap_common behind two lines of "
+        "typedef",
+    ),
+    (
+        "pch/pch.h",
+        "storage/olap_common.h",
+        set(),
+        "every header on the PCH is on the everything-rebuilds line: touch it "
+        "and the whole backend plus the precompiled header itself rebuild; "
+        "olap_common.h alone used to carry 22 project headers onto that line",
+    ),
 ]
 
-# Not expressible as RULES entries (the scanner only follows quoted project
-# includes and <gen_cpp/...>): core/uint24.h and core/value/large_int_value.h
-# must not regain <fmt/compile.h> / <fmt/format.h>. Their to_string/to_buffer
-# bodies live in the matching .cpp files precisely so the FMT_COMPILE formatter
-# templates (53.5 CPU s over ~1150 TUs for the uint24 date format alone) are
-# instantiated once instead of in every includer.
+# Third-party bans, (header, banned include, why). The include graph above only
+# follows quoted project includes and <gen_cpp/...>, so forbidden *third-party*
+# edges get their own single-file check: the named header must not include the
+# banned spelling (a trailing '/' bans the whole directory, otherwise the match
+# is exact; angle and quoted forms are both caught). These are headers whose
+# formatting/queueing bodies were deliberately moved out of line -- the ban keeps
+# the heavy third-party template machinery from re-entering every includer.
+ANGLE_BANS = [
+    (
+        "core/uint24.h",
+        "fmt/",
+        "to_string/to_buffer bodies live in uint24.cpp precisely so the "
+        "FMT_COMPILE formatter templates (53.5 CPU s over ~1150 TUs for the "
+        "date format alone) are instantiated once instead of in every includer",
+    ),
+    (
+        "core/value/large_int_value.h",
+        "fmt/",
+        "the fmt formatting bodies moved to large_int_value.cpp under the same "
+        "contract as core/uint24.h: the formatter templates must be "
+        "instantiated once, not in every includer",
+    ),
+    (
+        "exec/pipeline/dependency.h",
+        "concurrentqueue.h",
+        "was a dead 152 KB third-party include here; the moodycamel users "
+        "(local_exchanger.h, scanner_context.h, async_result_writer.h) "
+        "include it themselves",
+    ),
+    (
+        "util/pretty_printer.h",
+        "boost/",
+        "one boost::algorithm::join dragged ~60k preprocessed lines into every "
+        "TU that sees runtime_profile.h; the join is a plain loop in "
+        "pretty_printer.cpp now",
+    ),
+    # <ranges> is both a heavy header (object_pool.h alone reaches most of the
+    # backend) and a build breaker: libc++'s <ranges> rejects the
+    # -fno-access-control flag that doris_be_test builds with, so a <ranges>
+    # include in a src header can take the whole UT build down (#66615).
+    (
+        "common/object_pool.h",
+        "ranges",
+        "widely-included header; <ranges> is heavy for every includer and "
+        "breaks the -fno-access-control UT build on libc++",
+    ),
+    (
+        "exprs/lambda_function/lambda_execution_context.h",
+        "ranges",
+        "rides into every lambda-capable expression TU; <ranges> is heavy and "
+        "breaks the -fno-access-control UT build on libc++",
+    ),
+    (
+        "format/table/iceberg_reader_mixin.h",
+        "ranges",
+        "rides the table-reader stack; <ranges> is heavy and breaks the "
+        "-fno-access-control UT build on libc++",
+    ),
+    (
+        "storage/index/inverted/query_v2/collect/top_k_collector.h",
+        "ranges",
+        "rides the inverted-index query stack; <ranges> is heavy and breaks "
+        "the -fno-access-control UT build on libc++",
+    ),
+    (
+        "storage/index/inverted/query_v2/composite_reader.h",
+        "ranges",
+        "rides the inverted-index query stack; <ranges> is heavy and breaks "
+        "the -fno-access-control UT build on libc++",
+    ),
+    (
+        "storage/index/inverted/query_v2/wand/block_wand.h",
+        "ranges",
+        "rides the inverted-index query stack; <ranges> is heavy and breaks "
+        "the -fno-access-control UT build on libc++",
+    ),
+]
+
+# Budgets: the edge rules and bans above pin the edges someone has already
+# thought about; the two budget axes below catch the ones nobody thought about.
+# Both read the same text include graph (be/src + be/test, all ifdef branches);
+# neither is comparable to ninja-measured rebuild radii -- each compares only
+# against its own baseline.
 #
-# Likewise <concurrentqueue.h> must not return to exec/pipeline/dependency.h:
-# it was a dead 152 KB third-party include there; the moodycamel users
-# (local_exchanger.h, scanner_context.h, async_result_writer.h) include it
-# themselves.
+# Forward axis, "this header must stay light": the number of project headers
+# transitively reachable from the hub (the hub itself excluded). Slack is ZERO
+# by design -- growing a hub's closure must be a visible, deliberate act, so the
+# legal way over the budget is to bump the number here in the same PR and say
+# why in the commit message.
+# Baselines: master 9a48f8120c0, 2026-08-18.
+FORWARD_CLOSURE_BUDGETS = {
+    "common/logging.h": 0,  # a leaf on purpose: logging must not drag project headers
+    "util/uid_util.h": 1,
+    "common/status.h": 6,
+    "runtime/exec_env.h": 10,
+    "core/pod_array.h": 18,
+    "util/pretty_printer.h": 31,
+    "storage/olap_common.h": 32,
+    "core/types.h": 35,
+    "runtime/runtime_state.h": 43,
+    "core/data_type/primitive_type.h": 44,
+    "runtime/thread_context.h": 55,
+    "core/field.h": 60,
+    "core/column/column.h": 66,
+    "exprs/function/function.h": 106,
+    "exec/pipeline/dependency.h": 357,
+    "pch/pch.h": 8,
+}
+
+# Reverse axis, "this heavy payload must not spread": how many TUs (src + test)
+# transitively include the header. New TUs legitimately reference these, so the
+# limit is baseline +10% (rounded up); rebaseline when the audit cadence
+# re-measures, or in the same PR with justification when a real growth burst is
+# intended.
+# Baselines: master 9a48f8120c0, 2026-08-18.
+REVERSE_REACH_BASELINES = {
+    "gen_cpp/segment_v2.pb.h": 1234,
+    "gen_cpp/PaloInternalService_types.h": 1133,
+    "util/threadpool.h": 988,
+    # parsed_page.h holds RleDecoder<bool> by value and constructs it in the
+    # header, so the whole storage read stack sees the RLE machinery; a live
+    # dependency, budgeted as-is (trimming it is a refactor, not a gate).
+    "util/rle_encoding.h": 823,
+    "gen_cpp/internal_service.pb.h": 741,
+    "gen_cpp/FrontendService_types.h": 576,
+    "storage/options.h": 464,
+    "gen_cpp/cloud.pb.h": 342,
+    "gen_cpp/BackendService_types.h": 222,
+    "gen_cpp/data.pb.h": 220,
+    "io/fs/s3_file_system.h": 109,  # the AWS SDK surface
+    "util/brpc_closure.h": 61,
+    "runtime/workload_group/workload_group.h": 42,  # thrift type universe carrier
+}
+REVERSE_SLACK = 0.10
+
+# The PCH is the single most leveraged file in the repo: every header on it is
+# on the everything-rebuilds line (touch one and the whole backend plus the
+# precompiled header itself rebuild). Its quoted project includes are therefore
+# pinned exactly; the transitive closure is capped by the pch/pch.h entry in
+# FORWARD_CLOSURE_BUDGETS. Generated <gen_cpp/...> includes are the PCH's whole
+# point and stay out of this lock.
+PCH_HEADER = "pch/pch.h"
+PCH_QUOTED_WHITELIST = {
+    "common/config.h",
+    "common/status.h",
+    "common/version_internal.h",
+}
 
 # Forward-declaration headers are the sanctioned way through a barrier: they carry
 # declarations only, so they cost nothing to include.
@@ -398,6 +610,213 @@ def enforce(includes):
     return failures
 
 
+def forward_closure(hub, includes):
+    """Project headers transitively reachable from `hub`, hub excluded."""
+    chains = reachable(hub, includes)
+    return sorted(
+        h for h in chains if h != hub and resolve(h, includes) is not None
+    )
+
+
+ANY_INCLUDE = re.compile(r'^\s*#\s*include\s+[<"]([^>"]+)[>"]')
+
+
+def enforce_angle_bans():
+    failures = 0
+    for header, banned, why in ANGLE_BANS:
+        path = os.path.join(INCLUDE_ROOT, header)
+        if not os.path.exists(path):
+            print(f"error: angle ban names a missing header: {header}", file=sys.stderr)
+            failures += 1
+            continue
+        with open(path, encoding="utf-8", errors="ignore") as handle:
+            spellings = [m.group(1) for m in map(ANY_INCLUDE.match, handle) if m]
+        hits = [
+            s
+            for s in spellings
+            if s == banned or (banned.endswith("/") and s.startswith(banned))
+        ]
+        for hit in hits:
+            failures += 1
+            print(f"error: {header} must not include <{hit}>", file=sys.stderr)
+            print(f"  reason: {why}", file=sys.stderr)
+            print(
+                "  fix:    move the code that needs it into the matching .cpp "
+                "(that is where the previous cut put it), or take this ban out "
+                "of ANGLE_BANS in the same PR and justify it in the commit "
+                "message",
+                file=sys.stderr,
+            )
+    return failures
+
+
+def enforce_budgets(includes):
+    failures = 0
+    for hub, budget in FORWARD_CLOSURE_BUDGETS.items():
+        if resolve(hub, includes) is None:
+            print(
+                f"error: forward budget names a missing header: {hub} "
+                "(renamed or moved? update FORWARD_CLOSURE_BUDGETS)",
+                file=sys.stderr,
+            )
+            failures += 1
+            continue
+        actual = len(forward_closure(hub, includes))
+        if actual > budget:
+            failures += 1
+            print(
+                f"error: {hub} include closure grew to {actual} project "
+                f"headers (budget {budget})",
+                file=sys.stderr,
+            )
+            print(
+                "  reason: everything this hub includes is reparsed by every "
+                "TU that includes the hub, so closure growth is a rebuild-cost "
+                "multiplier nobody sees in a diff; the budget makes it visible",
+                file=sys.stderr,
+            )
+            print(
+                f"  list:   build-support/check-header-deps.py --closure {hub}",
+                file=sys.stderr,
+            )
+            print(
+                "  fix:    cut the new edge (forward-declare the type, or "
+                "route it through a *_fwd.h), or bump the budget in "
+                "FORWARD_CLOSURE_BUDGETS in the same PR and justify it in the "
+                "commit message",
+                file=sys.stderr,
+            )
+    counts = translation_units_affected(includes)
+    for header, baseline in REVERSE_REACH_BASELINES.items():
+        limit = -(-baseline * (100 + int(REVERSE_SLACK * 100)) // 100)
+        actual = counts.get(header, 0)
+        is_project = resolve(header, includes) is not None
+        if not is_project and not header.startswith("gen_cpp/"):
+            print(
+                f"error: reverse budget names a missing header: {header} "
+                "(renamed or moved? update REVERSE_REACH_BASELINES)",
+                file=sys.stderr,
+            )
+            failures += 1
+            continue
+        if actual == 0:
+            # A watched header nobody includes any more is either renamed
+            # (the budget is watching nothing) or genuinely dead -- both mean
+            # the table must be updated, loudly.
+            print(
+                f"error: reverse budget sentinel is no longer referenced: "
+                f"{header} (renamed, or truly unused? update "
+                "REVERSE_REACH_BASELINES)",
+                file=sys.stderr,
+            )
+            failures += 1
+            continue
+        if actual > limit:
+            failures += 1
+            print(
+                f"error: {header} now reaches {actual} TUs "
+                f"(baseline {baseline} +{int(REVERSE_SLACK * 100)}% = {limit})",
+                file=sys.stderr,
+            )
+            print(
+                "  reason: this is a heavy payload (generated code / SDK "
+                "surface / template machinery); some header on a popular path "
+                "gained an include of it, taxing every TU downstream",
+                file=sys.stderr,
+            )
+            print(
+                f"  list:   build-support/check-header-deps.py --reach {header}",
+                file=sys.stderr,
+            )
+            print(
+                "  fix:    cut the spreading edge (forward-declare, or move "
+                "the include into the .cpp), or rebaseline in "
+                "REVERSE_REACH_BASELINES in the same PR and justify it in the "
+                "commit message",
+                file=sys.stderr,
+            )
+    return failures
+
+
+def enforce_pch(includes):
+    path = resolve(PCH_HEADER, includes)
+    if path is None:
+        print(f"error: {PCH_HEADER} not found", file=sys.stderr)
+        return 1
+    with open(path, encoding="utf-8", errors="ignore") as handle:
+        quoted = {m.group(1) for m in map(INCLUDE.match, handle) if m}
+    extra = sorted(quoted - PCH_QUOTED_WHITELIST)
+    missing = sorted(PCH_QUOTED_WHITELIST - quoted)
+    if not extra and not missing:
+        return 0
+    print("error: pch/pch.h quoted includes diverged from the whitelist", file=sys.stderr)
+    for header in extra:
+        print(f"  added:   \"{header}\"", file=sys.stderr)
+    for header in missing:
+        print(f"  removed: \"{header}\"", file=sys.stderr)
+    print(
+        "  reason: every header on the PCH is on the everything-rebuilds "
+        "line -- touch it and the whole backend plus the precompiled header "
+        "itself rebuild; adding one is the single most leveraged regression "
+        "in the repo",
+        file=sys.stderr,
+    )
+    print(
+        "  fix:    include the header in the TUs that need it instead, or "
+        "change PCH_QUOTED_WHITELIST in the same PR and justify it in the "
+        "commit message",
+        file=sys.stderr,
+    )
+    return 1
+
+
+def closure_listing(hub, includes):
+    if resolve(hub, includes) is None:
+        print(f"error: no such project header: {hub}", file=sys.stderr)
+        return 1
+    chains = reachable(hub, includes)
+    members = forward_closure(hub, includes)
+    print(f"{hub}: {len(members)} project header(s) in the include closure")
+    for member in members:
+        print("  " + " -> ".join(chains[member]))
+    return 0
+
+
+def reach_listing(header, includes):
+    """Direct includers of `header` ranked by how many TUs each edge carries."""
+    users = collections.defaultdict(set)
+    for path, headers in includes.items():
+        for h in headers:
+            users[h].add(path)
+    if not users.get(header):
+        print(f"error: nothing includes {header}", file=sys.stderr)
+        return 1
+
+    def tus(banned_edge=None):
+        seen, frontier = set(), [header]
+        while frontier:
+            current = frontier.pop()
+            for user in users.get(current, ()):
+                if banned_edge and (user, current) == banned_edge:
+                    continue
+                if user in seen:
+                    continue
+                seen.add(user)
+                if user.startswith(INCLUDE_ROOT + "/"):
+                    frontier.append(user[len(INCLUDE_ROOT) + 1:])
+        return sum(1 for f in seen if f.endswith((".cpp", ".cc")))
+
+    total = tus()
+    print(f"{header}: reaches {total} TU(s); direct includers by edge weight")
+    ranked = sorted(
+        ((total - tus((user, header)), user) for user in users[header]),
+        reverse=True,
+    )
+    for marginal, user in ranked:
+        print(f"  {marginal:>5} via this edge alone  {user}")
+    return 0
+
+
 def report(includes):
     counts = translation_units_affected(includes)
     ranked = sorted(counts.items(), key=lambda kv: kv[1], reverse=True)[:30]
@@ -413,6 +832,21 @@ def main():
         action="store_true",
         help="rank headers by how many translation units they force a rebuild of",
     )
+    parser.add_argument(
+        "--budget",
+        action="store_true",
+        help="run only the closure/reach budget checks (used when rebaselining)",
+    )
+    parser.add_argument(
+        "--closure",
+        metavar="HEADER",
+        help="list the project headers in HEADER's include closure, with chains",
+    )
+    parser.add_argument(
+        "--reach",
+        metavar="HEADER",
+        help="list HEADER's direct includers ranked by how many TUs each edge carries",
+    )
     args = parser.parse_args()
 
     root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -422,12 +856,35 @@ def main():
     if args.report:
         report(includes)
         return 0
+    if args.closure:
+        return closure_listing(args.closure, includes)
+    if args.reach:
+        return reach_listing(args.reach, includes)
 
-    failures = enforce(includes)
+    if args.budget:
+        failures = enforce_budgets(includes)
+    else:
+        failures = (
+            enforce(includes)
+            + enforce_angle_bans()
+            + enforce_budgets(includes)
+            + enforce_pch(includes)
+        )
     if failures:
-        print(f"\n{failures} header layering violation(s)", file=sys.stderr)
+        print(f"\n{failures} header hygiene violation(s)", file=sys.stderr)
         return 1
-    print(f"header layering: {len(RULES)} rule(s) satisfied")
+    if args.budget:
+        print(
+            f"header budgets: {len(FORWARD_CLOSURE_BUDGETS)} forward + "
+            f"{len(REVERSE_REACH_BASELINES)} reverse within budget"
+        )
+    else:
+        print(
+            f"header hygiene: {len(RULES)} layering rule(s), "
+            f"{len(ANGLE_BANS)} third-party ban(s), "
+            f"{len(FORWARD_CLOSURE_BUDGETS)}+{len(REVERSE_REACH_BASELINES)} "
+            "budget(s), pch whitelist: all satisfied"
+        )
     return 0
 
 

build-support/check-unity-skip-coverage.py

@@ -0,0 +1,226 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Unity-skip coverage for test-included source files.
+
+A few tests `#include` a be/src .cpp directly (to reach file-static helpers or
+to instantiate a template with test types). That only links because the source
+file is opted out of unity batching: if it stays in a batch, the batch object is
+pulled into the doris_be_test link for a *sibling's* symbol and its copy of the
+file's definitions collides with the test's inlined copy -- duplicate strong
+symbols, reported at the end of the BE UT build, an hour after the mistake.
+This script reports the mistake at configure time instead, with the exact skip
+entry to add.
+
+It is the mirror of the fail-loud check inside doris_skip_unity_inclusion()
+(be/CMakeLists.txt): that one catches skip entries pointing at files that no
+longer exist; this one catches files that need an entry and do not have one.
+
+A source file counts as covered when its owning CMakeLists (nearest ancestor
+with a CMakeLists.txt, else any CMakeLists naming it) does one of:
+
+  literal entry     the file's path appears in a list fed to
+                    doris_skip_unity_inclusion(), via set()/list(APPEND)/direct
+                    arguments;
+  whole-target      the skip list is initialized from the target's own source
+  opt-out           list (`set(<VAR> ${SRC_FILES})`), minus any
+                    `list(FILTER <VAR> EXCLUDE REGEX ...)` the file matches;
+  no unity          the owning CMakeLists never sets UNITY_BUILD, so there are
+                    no batches to collide with (archive link rules protect
+                    plain object files).
+
+Any other CMake idiom is unknown to this script on purpose: it fails loudly so
+the idiom is either replaced with a literal entry or taught here, instead of
+being silently guessed wrong.
+
+Usage:
+    build-support/check-unity-skip-coverage.py
+"""
+
+import os
+import re
+import sys
+
+TEST_ROOT = "be/test"
+SRC_ROOT = "be/src"
+
+TEST_INCLUDE = re.compile(r'^\s*#\s*include\s+"([^"]+\.(?:cpp|cc))"')
+SKIP_CALL = re.compile(r"doris_skip_unity_inclusion\s*\(([^)]*)\)", re.S)
+SET_BLOCK = re.compile(r"\bset\s*\(\s*(\w+)([^)]*)\)", re.S)
+APPEND_BLOCK = re.compile(r"\blist\s*\(\s*APPEND\s+(\w+)([^)]*)\)", re.S)
+FILTER_BLOCK = re.compile(
+    r'\blist\s*\(\s*FILTER\s+(\w+)\s+EXCLUDE\s+REGEX\s+"([^"]*)"\s*\)'
+)
+VAR_REF = re.compile(r"\$\{(\w+)\}")
+
+
+def strip_comments(text):
+    return "\n".join(line.split("#", 1)[0] for line in text.splitlines())
+
+
+def cpp_tokens(blob):
+    """Path-shaped tokens in a CMake argument blob, ${...} prefixes stripped."""
+    tokens = []
+    for token in blob.split():
+        if not token.endswith((".cpp", ".cc")):
+            continue
+        tokens.append(re.sub(r"\$\{\w+\}", "", token).lstrip("/"))
+    return tokens
+
+
+class CMakeFile:
+    def __init__(self, path):
+        self.path = path
+        self.directory = os.path.dirname(path)
+        with open(path, encoding="utf-8", errors="ignore") as handle:
+            text = strip_comments(handle.read())
+        self.sets_unity = "UNITY_BUILD" in text
+
+        sets = {}
+        for name, blob in SET_BLOCK.findall(text):
+            sets.setdefault(name, []).append(blob)
+        for name, blob in APPEND_BLOCK.findall(text):
+            sets.setdefault(name, []).append(blob)
+        filters = {}
+        for name, regex in FILTER_BLOCK.findall(text):
+            filters.setdefault(name, []).append(regex)
+
+        # Literal skip entries and whether the skip list is seeded from the
+        # target's whole source list.
+        self.entries = []
+        self.whole_target = False
+        self.whole_target_excludes = []
+        for blob in SKIP_CALL.findall(text):
+            self.entries.extend(cpp_tokens(blob))
+            for var in VAR_REF.findall(blob):
+                for var_blob in sets.get(var, ()):
+                    self.entries.extend(cpp_tokens(var_blob))
+                    if re.search(r"\$\{\w*SRC_FILES\}", var_blob):
+                        self.whole_target = True
+                        self.whole_target_excludes.extend(filters.get(var, ()))
+
+    def resolved_entries(self):
+        """Skip entries as repo-relative paths (handles ../ hops)."""
+        for entry in self.entries:
+            yield os.path.normpath(os.path.join(self.directory, entry))
+
+    def covers(self, repo_path):
+        """Does this CMakeLists skip-list `repo_path` (repo-relative .cpp)?"""
+        for resolved in self.resolved_entries():
+            if resolved == repo_path or resolved.endswith("/" + repo_path):
+                return True
+        if self.whole_target and repo_path.startswith(self.directory + "/"):
+            # cmake applies FILTER regexes to absolute paths; approximate with
+            # the repo-relative one, which shares every path component that the
+            # tree's regexes (".*/format_v2/.*") anchor on.
+            return not any(
+                re.search(regex, repo_path)
+                for regex in self.whole_target_excludes
+            )
+        return False
+
+
+def owning_cmakelists(repo_path, cmake_files):
+    directory = os.path.dirname(repo_path)
+    while directory.startswith(SRC_ROOT):
+        candidate = os.path.join(directory, "CMakeLists.txt")
+        if candidate in cmake_files:
+            return cmake_files[candidate]
+        directory = os.path.dirname(directory)
+    return None
+
+
+def test_included_sources():
+    """(test file, line, be/src-relative include) for every src .cpp a test
+    includes."""
+    found = []
+    for directory, _, names in os.walk(TEST_ROOT):
+        for name in names:
+            if not name.endswith((".cpp", ".cc", ".h", ".hpp")):
+                continue
+            path = os.path.join(directory, name)
+            with open(path, encoding="utf-8", errors="ignore") as handle:
+                for lineno, line in enumerate(handle, 1):
+                    match = TEST_INCLUDE.match(line)
+                    if not match:
+                        continue
+                    include = match.group(1)
+                    if os.path.exists(os.path.join(SRC_ROOT, include)):
+                        found.append((path, lineno, include))
+    return found
+
+
+def main():
+    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    os.chdir(root)
+
+    cmake_files = {}
+    for directory, _, names in os.walk(SRC_ROOT):
+        if "CMakeLists.txt" in names:
+            path = os.path.join(directory, "CMakeLists.txt")
+            cmake_files[path] = CMakeFile(path)
+
+    failures = 0
+    checked = 0
+    for test_path, lineno, include in test_included_sources():
+        checked += 1
+        repo_path = os.path.join(SRC_ROOT, include)
+        owner = owning_cmakelists(repo_path, cmake_files)
+        if owner is not None and not owner.sets_unity:
+            continue  # no unity batches in this target, nothing to collide with
+        if owner is not None and owner.covers(repo_path):
+            continue
+        if any(f.covers(repo_path) for f in cmake_files.values()):
+            continue  # cross-directory entry (e.g. appended into a sibling's list)
+        failures += 1
+        print(
+            f"error: {test_path}:{lineno} includes {repo_path}, which is not "
+            "opted out of unity batching",
+            file=sys.stderr,
+        )
+        print(
+            "  reason: the unity batch object containing this file gets pulled "
+            "into the doris_be_test link for a sibling's symbol, and its "
+            "definitions collide with the test's inlined copy -- duplicate "
+            "strong symbols at the end of the BE UT build",
+            file=sys.stderr,
+        )
+        if owner is not None:
+            entry = os.path.relpath(repo_path, owner.directory)
+            print(
+                f"  fix:    add ${{CMAKE_CURRENT_SOURCE_DIR}}/{entry} to the "
+                f"doris_skip_unity_inclusion list in {owner.path}",
+                file=sys.stderr,
+            )
+        else:
+            print(
+                "  fix:    add the file to the doris_skip_unity_inclusion "
+                "list of the CMakeLists whose source glob compiles it; for "
+                "the cross-directory entry form, see the adbc_reader.cpp "
+                "entry in be/src/format/CMakeLists.txt",
+                file=sys.stderr,
+            )
+    if failures:
+        print(f"\n{failures} unity-skip coverage violation(s)", file=sys.stderr)
+        return 1
+    print(f"unity-skip coverage: {checked} test-included source file(s) all covered")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

build-support/tests/test-build-hygiene-entry.sh

@@ -0,0 +1,101 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Self-test for the check-build-hygiene.sh unified entry:
+#   - clean tree exits 0 with the summary line
+#   - with violations in TWO different checks, BOTH are reported (the entry
+#     keeps running after a failing check) and the exit code is 1
+#   - a missing python3 is a loud error (exit 2) naming the OFF switch, never
+#     a silent skip
+#   - env.sh's PYTHON (python2 on the build-env image) is not consulted
+#
+# Injects into working-tree files, restores via EXIT trap. Do not run
+# concurrently with a build/configure of the same tree.
+#
+# Usage:  bash build-support/tests/test-build-hygiene-entry.sh
+
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+ENTRY="${ROOT}/build-support/check-build-hygiene.sh"
+
+UINT24="be/src/core/uint24.h"
+COLVEC="be/src/core/column/column_vector.h"
+BACKUP="$(mktemp -d)"
+cp "${ROOT}/${UINT24}" "${BACKUP}/uint24.h"
+cp "${ROOT}/${COLVEC}" "${BACKUP}/column_vector.h"
+restore() {
+    cp "${BACKUP}/uint24.h" "${ROOT}/${UINT24}"
+    cp "${BACKUP}/column_vector.h" "${ROOT}/${COLVEC}"
+    rm -rf "${BACKUP}"
+}
+trap restore EXIT
+
+FAILED=0
+fail() { echo "FAIL: $1"; FAILED=1; }
+
+# ---- clean tree: exit 0, summary line present ----
+OUT="$(bash "${ENTRY}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "clean tree expected exit 0, got ${EC}:"$'\n'"${OUT}"
+echo "${OUT}" | grep -q "build hygiene: all checks passed" \
+    || fail "clean run lacks the all-passed summary"
+
+# ---- two violations in two different checks: both reported, exit 1 ----
+printf '#include <fmt/format.h>\n' >> "${ROOT}/${UINT24}"
+printf 'extern template class ColumnVector<TYPE_FAKE_ENTRY_TEST>;\n' >> "${ROOT}/${COLVEC}"
+OUT="$(bash "${ENTRY}" 2>&1)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "violations expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "core/uint24.h must not include <fmt/format.h>" \
+    || fail "header-deps violation missing from aggregated output"
+echo "${OUT}" | grep -q "TYPE_FAKE_ENTRY_TEST" \
+    || fail "pairing violation missing from aggregated output (entry must run every check)"
+echo "${OUT}" | grep -q "2 check(s) failed" \
+    || fail "aggregated failure count missing or wrong"
+restore
+trap - EXIT
+BACKUP="$(mktemp -d)"  # restore() removed it; re-arm for the paths below
+cp "${ROOT}/${UINT24}" "${BACKUP}/uint24.h"
+cp "${ROOT}/${COLVEC}" "${BACKUP}/column_vector.h"
+trap restore EXIT
+
+# ---- unusable BUILD_HYGIENE_PYTHON: loud exit 2 with the OFF-switch hint ----
+OUT="$(BUILD_HYGIENE_PYTHON=/nonexistent/python3 bash "${ENTRY}" 2>&1)"; EC=$?
+[ "${EC}" -eq 2 ] || fail "missing python expected exit 2, got ${EC}"
+echo "${OUT}" | grep -q "not found" || fail "missing-python error not reported"
+echo "${OUT}" | grep -q "ENABLE_BUILD_HYGIENE=OFF" \
+    || fail "missing-python error lacks the OFF-switch hint"
+
+# ---- env.sh's PYTHON is not consulted ----
+# env.sh exports PYTHON=${DORIS_BUILD_PYTHON_VERSION:-python}, which is python2
+# on the build-env image; honouring it made every gate die with a SyntaxError on
+# its first f-string. The gates must resolve their own Python 3 regardless.
+OUT="$(PYTHON=/nonexistent/python2 bash "${ENTRY}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] \
+    || fail "the gates must resolve their own python3, not env.sh's PYTHON (exit ${EC}):"$'\n'"${OUT}"
+
+# ---- green again ----
+OUT="$(bash "${ENTRY}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "restored tree expected exit 0, got ${EC}:"$'\n'"${OUT}"
+
+if [ "${FAILED}" -eq 0 ]; then
+    echo "PASS: entry aggregates all checks, resolves its own python3, fails loud without one, and is green when clean."
+    exit 0
+fi
+exit 1

build-support/tests/test-build-hygiene-extern-pairing.sh

@@ -0,0 +1,91 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Self-test for check-extern-template-pairing.py: both pairing directions must
+# fire on injection and the clean tree must stay green (which also proves the
+# normalizer -- namespace dropping, alias table, comment stripping -- because
+# the real tree pairs ColumnStr<UInt32> with ColumnStr<uint32_t> and the wide
+# to_string declarations with their integer<...> definitions).
+#
+# Injects into working-tree files, restores via EXIT trap. Do not run
+# concurrently with a build/configure of the same tree.
+#
+# Usage:  bash build-support/tests/test-build-hygiene-extern-pairing.sh
+
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+GATE="${ROOT}/build-support/check-extern-template-pairing.py"
+# Not ${PYTHON}: env.sh exports that as the build's interpreter (python2 on
+# the build-env image). Same selector the gates themselves use.
+PYTHON="${BUILD_HYGIENE_PYTHON:-python3}"
+
+HEADER="be/src/core/column/column_vector.h"
+BACKUP="$(mktemp -d)"
+cp "${ROOT}/${HEADER}" "${BACKUP}/column_vector.h"
+restore() { cp "${BACKUP}/column_vector.h" "${ROOT}/${HEADER}"; rm -rf "${BACKUP}"; }
+trap restore EXIT
+
+FAILED=0
+fail() { echo "FAIL: $1"; FAILED=1; }
+
+# ---- baseline: clean tree green (normalizer handles the real alias pairs) ----
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "clean tree expected green, got ${EC}:"$'\n'"${OUT}"
+echo "${OUT}" | grep -q "all paired" || fail "clean run lacks the summary line"
+
+# ---- RED (the silent case): drop one extern, its definition loses its pair ----
+# Commenting the declaration out is exactly the "header forgot the extern"
+# regression the reverse direction exists for.
+sed -i.bak \
+    's|^extern template class ColumnVector<TYPE_BOOLEAN>;|// injected-out: extern template class ColumnVector<TYPE_BOOLEAN>;|' \
+    "${ROOT}/${HEADER}" && rm -f "${ROOT}/${HEADER}.bak"
+grep -q "injected-out" "${ROOT}/${HEADER}" || fail "reverse injection did not apply"
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "reverse injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "explicit instantiation lacks the matching 'extern template'" \
+    || fail "reverse violation not reported"
+echo "${OUT}" | grep -q "ColumnVector<TYPE_BOOLEAN>" \
+    || fail "reverse violation does not name the specialization"
+echo "${OUT}" | grep -q "extern-covered in be/src/core/column/column_vector.h" \
+    || fail "reverse violation does not point at the family's header"
+echo "${OUT}" | grep -q "fix:" || fail "reverse violation lacks a fix path"
+cp "${BACKUP}/column_vector.h" "${ROOT}/${HEADER}"
+
+# ---- RED: an extern declaration with no definition anywhere ----
+printf 'extern template class ColumnVector<TYPE_DATETIMEV2_FAKE_INJECTED>;\n' \
+    >> "${ROOT}/${HEADER}"
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "forward injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "pairs with no explicit instantiation definition" \
+    || fail "forward violation not reported"
+echo "${OUT}" | grep -q "TYPE_DATETIMEV2_FAKE_INJECTED" \
+    || fail "forward violation does not name the declaration"
+cp "${BACKUP}/column_vector.h" "${ROOT}/${HEADER}"
+
+# ---- GREEN again ----
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "restored tree expected green, got ${EC}:"$'\n'"${OUT}"
+
+if [ "${FAILED}" -eq 0 ]; then
+    echo "PASS: both pairing directions fire on injection and restore to green."
+    exit 0
+fi
+exit 1

build-support/tests/test-build-hygiene-header-deps.sh

@@ -0,0 +1,136 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Self-test for the check-header-deps.py gate families: layering rules,
+# third-party bans, forward/reverse budgets, pch whitelist.
+#
+# The gate runs against the real tree (its rule and budget tables name real
+# headers), so RED cases are proven by injecting one violation per family into
+# a working-tree file and GREEN by restoring it. Injected files are backed up
+# first and restored by an EXIT trap. Do not run concurrently with a
+# build/configure of the same tree.
+#
+# Usage:  bash build-support/tests/test-build-hygiene-header-deps.sh
+
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+GATE="${ROOT}/build-support/check-header-deps.py"
+# Not ${PYTHON}: env.sh exports that as the build's interpreter (python2 on
+# the build-env image). Same selector the gates themselves use.
+PYTHON="${BUILD_HYGIENE_PYTHON:-python3}"
+
+BACKUP="$(mktemp -d)"
+TARGETS=(
+    "be/src/core/types.h"
+    "be/src/core/uint24.h"
+    "be/src/common/logging.h"
+    "be/src/util/pretty_printer.h"
+    "be/src/pch/pch.h"
+)
+for t in "${TARGETS[@]}"; do
+    mkdir -p "${BACKUP}/$(dirname "${t}")"
+    cp "${ROOT}/${t}" "${BACKUP}/${t}"
+done
+restore() {
+    for t in "${TARGETS[@]}"; do
+        cp "${BACKUP}/${t}" "${ROOT}/${t}"
+    done
+    rm -rf "${BACKUP}"
+}
+trap restore EXIT
+
+FAILED=0
+fail() { echo "FAIL: $1"; FAILED=1; }
+
+run_gate() { "${PYTHON}" "${GATE}" 2>&1; }
+
+# ---- baseline: the clean tree is green ----
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 0 ] || { fail "clean tree expected green, got ${EC}:"$'\n'"${OUT}"; }
+
+# ---- RED: layering rule (types.h must not reach olap_common.h) ----
+printf '#include "storage/olap_common.h"\n' >> "${ROOT}/be/src/core/types.h"
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "rule injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "core/types.h must not reach storage/olap_common.h" \
+    || fail "rule violation not reported for types.h -> olap_common.h"
+echo "${OUT}" | grep -q "chain:" || fail "rule failure lacks the include chain"
+echo "${OUT}" | grep -q "fix:" || fail "rule failure lacks a fix path"
+cp "${BACKUP}/be/src/core/types.h" "${ROOT}/be/src/core/types.h"
+
+# ---- RED: third-party ban (uint24.h must not include fmt) ----
+printf '#include <fmt/format.h>\n' >> "${ROOT}/be/src/core/uint24.h"
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "angle-ban injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "core/uint24.h must not include <fmt/format.h>" \
+    || fail "angle-ban violation not reported"
+cp "${BACKUP}/be/src/core/uint24.h" "${ROOT}/be/src/core/uint24.h"
+
+# ---- RED: forward closure budget (logging.h is budgeted at 0) ----
+printf '#include "util/uid_util.h"\n' >> "${ROOT}/be/src/common/logging.h"
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "forward-budget injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "common/logging.h include closure grew" \
+    || fail "forward budget breach not reported"
+echo "${OUT}" | grep -q -- "--closure common/logging.h" \
+    || fail "forward budget failure lacks the --closure fix pointer"
+cp "${BACKUP}/be/src/common/logging.h" "${ROOT}/be/src/common/logging.h"
+
+# ---- RED: reverse reach budget (workload_group.h must not spread) ----
+printf '#include "runtime/workload_group/workload_group.h"\n' \
+    >> "${ROOT}/be/src/util/pretty_printer.h"
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "reverse-budget injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "runtime/workload_group/workload_group.h now reaches" \
+    || fail "reverse budget breach not reported"
+echo "${OUT}" | grep -q -- "--reach runtime/workload_group/workload_group.h" \
+    || fail "reverse budget failure lacks the --reach fix pointer"
+cp "${BACKUP}/be/src/util/pretty_printer.h" "${ROOT}/be/src/util/pretty_printer.h"
+
+# ---- RED: pch whitelist (quoted include added to pch.h) ----
+printf '#include "common/logging.h"\n' >> "${ROOT}/be/src/pch/pch.h"
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "pch injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "pch/pch.h quoted includes diverged from the whitelist" \
+    || fail "pch whitelist divergence not reported"
+echo "${OUT}" | grep -q 'added:   "common/logging.h"' \
+    || fail "pch failure does not name the added include"
+cp "${BACKUP}/be/src/pch/pch.h" "${ROOT}/be/src/pch/pch.h"
+
+# ---- diagnostics: --closure and --reach stay usable ----
+OUT="$("${PYTHON}" "${GATE}" --closure common/status.h 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "--closure expected exit 0, got ${EC}"
+echo "${OUT}" | grep -q "common/status.h: .* project header(s)" \
+    || fail "--closure output missing summary line"
+OUT="$("${PYTHON}" "${GATE}" --reach io/fs/s3_file_system.h 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "--reach expected exit 0, got ${EC}"
+echo "${OUT}" | grep -q "via this edge alone" \
+    || fail "--reach output missing edge ranking"
+
+# ---- GREEN again after all restores ----
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "restored tree expected green, got ${EC}:"$'\n'"${OUT}"
+
+if [ "${FAILED}" -eq 0 ]; then
+    echo "PASS: rules/bans/budgets/pch-lock all fire on injection, restore to green, diagnostics work."
+    exit 0
+fi
+exit 1

build-support/tests/test-build-hygiene-unity-skip.sh

@@ -0,0 +1,92 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Self-test for check-unity-skip-coverage.py. Seeds throwaway files (a src
+# .cpp that is in no skip list, plus probe tests under be/test) to prove:
+#   RED    a test including an unlisted, unity-batched src .cpp is flagged,
+#          with the exact skip entry to add
+#   SILENT the literal-entry idiom (cross-directory APPEND included)
+#   SILENT the whole-target opt-out idiom (set(<VAR> ${SRC_FILES}))
+#   SILENT a commented-out include
+# and that the clean tree is green before and after.
+#
+# Creates files only (never edits tracked ones); removes them via EXIT trap.
+#
+# Usage:  bash build-support/tests/test-build-hygiene-unity-skip.sh
+
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+GATE="${ROOT}/build-support/check-unity-skip-coverage.py"
+# Not ${PYTHON}: env.sh exports that as the build's interpreter (python2 on
+# the build-env image). Same selector the gates themselves use.
+PYTHON="${BUILD_HYGIENE_PYTHON:-python3}"
+
+PROBE_SRC="${ROOT}/be/src/util/tmp_hygiene_probe.cpp"
+PROBE_DIR="${ROOT}/be/test/tmp_hygiene_probe"
+cleanup() { rm -f "${PROBE_SRC}"; rm -rf "${PROBE_DIR}"; }
+trap cleanup EXIT
+
+FAILED=0
+fail() { echo "FAIL: $1"; FAILED=1; }
+
+# ---- baseline: clean tree green ----
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "clean tree expected green, got ${EC}:"$'\n'"${OUT}"
+
+# ---- fixtures ----
+echo "namespace doris { int tmp_hygiene_probe_symbol = 0; }" > "${PROBE_SRC}"
+mkdir -p "${PROBE_DIR}"
+# RED: util sets UNITY_BUILD and lists skips literally; the probe src is new,
+# so it is in no list.
+printf '#include "util/tmp_hygiene_probe.cpp"\n' > "${PROBE_DIR}/red_probe_test.cpp"
+# SILENT: cross-directory literal entry (appended into format's skip list).
+printf '#include "format_v2/table/adbc_reader.cpp"\n' > "${PROBE_DIR}/literal_probe_test.cpp"
+# SILENT: whole-target opt-out (format skip list is seeded from SRC_FILES).
+printf '#include "format/orc/orc_file_reader.cpp"\n' > "${PROBE_DIR}/wholetarget_probe_test.cpp"
+# SILENT: commented include is not an include.
+printf '//#include "format_v2/parquet/reader/native/decoder.cpp"\n' > "${PROBE_DIR}/comment_probe_test.cpp"
+
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "expected exit 1 with the red probe present, got ${EC}"
+N="$(echo "${OUT}" | grep -c '^error:')"
+[ "${N}" -eq 1 ] || fail "expected exactly 1 violation (silent probes must stay silent), got ${N}:"$'\n'"${OUT}"
+echo "${OUT}" | grep -q "red_probe_test.cpp:1 includes be/src/util/tmp_hygiene_probe.cpp" \
+    || fail "red probe not reported"
+echo "${OUT}" | grep -q 'add ${CMAKE_CURRENT_SOURCE_DIR}/tmp_hygiene_probe.cpp to the doris_skip_unity_inclusion list in be/src/util/CMakeLists.txt' \
+    || fail "fix path does not name the exact entry and CMakeLists"
+echo "${OUT}" | grep -q "duplicate strong symbols" \
+    || fail "failure lacks the mechanism explanation"
+
+# ---- GREEN again once the red probe is gone ----
+rm -f "${PROBE_DIR}/red_probe_test.cpp"
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "expected green with only silent probes, got ${EC}:"$'\n'"${OUT}"
+
+cleanup
+trap - EXIT
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "restored tree expected green, got ${EC}:"$'\n'"${OUT}"
+
+if [ "${FAILED}" -eq 0 ]; then
+    echo "PASS: unlisted include flagged with exact fix; literal/whole-target/comment cases stay silent."
+    exit 0
+fi
+exit 1

build-support/tests/test-fe-core-metadata-funnel.sh

@@ -142,8 +142,10 @@ bash "${GATE}" "${FX}" >/dev/null 2>&1 && CLEAN_EC=0 || CLEAN_EC=$?
 [ "${CLEAN_EC}" -eq 0 ] || fail "expected exit 0 on clean tree (only funnel/marked/no-arg/comment), got ${CLEAN_EC}"
 
 # ---- run 3: the marker is load-bearing -> strip it and both write calls are flagged ----
-sed -i 's/getMetadata-funnel-exempt/marker-removed-here/' \
+# (-i.bak + rm: the in-place flag's argument form differs between GNU and BSD sed)
+sed -i.bak 's/getMetadata-funnel-exempt/marker-removed-here/' \
     "${SRC}/write/WriterTrailing.java" "${SRC}/write/WriterAbove.java"
+rm -f "${SRC}/write/WriterTrailing.java.bak" "${SRC}/write/WriterAbove.java.bak"
 OUT3="$(bash "${GATE}" "${FX}" 2>&1)"; EC3=$?
 REP3="$(printf '%s\n' "${OUT3}" | grep -E "^${FX}.*:[0-9]+:" || true)"
 N3="$(printf '%s\n' "${REP3}" | grep -c 'getMetadata' || true)"