mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Mingyu Chen (Rayner)andClaude Fable 5
7933a82bdd
### What problem does this PR solve? Related PR: #66510 (the BE build-time series, #66615..#66826, that this PR locks in) Problem Summary: The build-time series cut BE cold builds by ~49% and took `olap_common.h` / `uid_util.h` off the everything-rebuilds line. Every one of those wins can regress **silently**: a stray include, a lost `extern template`, a header added to the PCH — the code still compiles, only every build afterwards is slower, and nothing reports it. `#66052` already showed the failure mode: the layering rules in `check-header-deps.py` only worked because the author happened to maintain them by hand; nothing in CI runs them. This PR turns each known regression vector into a **fail-loud, seconds-level gate that runs at configure time**, so a violation is a build error on every path that builds the BE (CI, `build.sh`, the UT build) with zero extra CI wiring — the same mounting pattern the FE side already uses for its architecture gates (`exec-maven-plugin` running `check-fe-core-metadata-funnel.sh` at `validate`). **Checks** (pure text scans, ~1.3s combined, zero build dependency, zero false positives on current master): | Gate | Regression it stops | |---|---| | Header layering rules (30, +6 new) | a hub header (e.g. `exec_env.h`, `function.h`, `pch.h`) re-growing an include edge into a subsystem that was deliberately cut | | Third-party ban table (new) | `fmt`/`boost`/`concurrentqueue.h`/`<ranges>` returning to headers whose bodies were moved out of line to contain template-instantiation cost (`<ranges>` in a src header also breaks the `-fno-access-control` UT build on libc++) | | Two-axis budgets (new) | the edges nobody names: light hubs must stay light (include-closure budget, zero slack), heavy payloads must not spread (transitive-reach baseline +10%); `--closure` / `--reach` subcommands rank the offending edges for the fix | | pch whitelist (new) | any quoted include added to `pch/pch.h` — the single most leveraged regression in the repo (touch one such header and the whole backend plus the PCH rebuild) | | `check-extern-template-pairing.py` (new) | the one **silent** extern-template failure: a `.cpp` keeps the explicit instantiation but the header loses the `extern` declaration, so every TU quietly re-instantiates; also the loud direction (declaration with no definition), reported at once instead of at link | | `check-unity-skip-coverage.py` (new) | a test `#include`-ing a src `.cpp` that is not opted out of unity batching — reported at configure with the exact skip entry to add, instead of as duplicate symbols at the end of the BE UT link | `build-support/check-build-hygiene.sh` runs all three scripts; `be/CMakeLists.txt` executes it at configure time. Escape hatches, by design: - `-DENABLE_BUILD_HYGIENE=OFF` (default ON; a missing `python3` is a loud FATAL naming this switch, never a silent skip); - every budget/whitelist/rule table lives in the scripts themselves, so the legal way past a gate is a **one-line reviewed diff in the same PR** — the gate is never a dead end. Every failure message carries the mechanism (why this edge is expensive) and the concrete fix path. **Found by the new gates on day one, fixed here:** `core/types.h` has carried five `extern template struct fmt::formatter<Decimal*>;` declarations with **no matching definition anywhere** since #53483 — every member is defined in-class, so the linker never noticed; deleted. Also made the FE funnel self-test's `sed -i` invocation portable to BSD/macOS. Blind spot, by design: instantiations expanded from macros (e.g. `DECLARE_OPERATOR` in `operator.cpp`) are invisible to the pairing scan on both sides — macro bodies are skipped; a comment at the site says so. `be/README.md` (new) documents the gates, what to do when each fires, and the `*_fwd.h` convention as the sanctioned way through a layering barrier. The same rules are encoded for AI-assisted development and review: `be/AGENTS.md` (new — include discipline while writing BE code + review checkpoints), `build-support/AGENTS.md` (new — maintenance discipline for the gate scripts themselves), an include-src-cpp checkpoint in `be/test/AGENTS.md`, and a Header Hygiene entry in the root `AGENTS.md`, so coding agents and the review pipeline apply them automatically. ### Release note None ### Check List (For Author) - Test - [x] Manual test (add detailed scripts or steps below) - `build-support/check-build-hygiene.sh` on master: all green, ~1.3s, zero false positives. - Self-tests `bash build-support/tests/run.sh` (7/7 PASS): red/green injection per gate family — each seeded violation turns red with the mechanism + fix in the message, restoring the file turns it green again. - Real configure on macOS arm64: green path passes the gate section; with an injected violation, configure dies at the gate in under a second with the full message; `-DENABLE_BUILD_HYGIENE=OFF` bypasses; a missing `python3` is a loud exit-2 FATAL naming the switch. - This PR's own Linux CI exercises the configure mount on the second platform. - Behavior changed: - [x] Yes. BE configure now fails fast (with mechanism + fix in the message) on header-hygiene violations; emergency bypass via `-DENABLE_BUILD_HYGIENE=OFF`. No runtime behavior change — the only source-code changes are comments and the deletion of five never-defined `extern template` declarations. - Does this need documentation? - [x] No. Developer-facing docs are included in-repo (`be/README.md`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01EyZL631YTeuNXa9WLjvk7s --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2.7 KiB
2.7 KiB
AGENTS.md — build-support
Operational rules for the architecture/hygiene gates in this directory (the
check-*.sh / check-*.py scripts wired into FE mvn validate and BE
configure). These scripts fail other people's builds — treat them as
production code with a stricter bar than the code they guard.
Gate maintenance discipline
- Zero false positives, non-negotiable. A per-commit gate must be deterministic, pure-text, and second-level. Anything heuristic (nm audits, closure sweeps, wall-clock trends) belongs in offline reports, never in the configure/validate path. One false positive burns more trust than ten missed regressions.
- Fail loud, never silently skip. A missing tool, a table entry pointing
at a renamed file, a sentinel nobody references any more — each is an
error with a message, not a silent pass (precedents:
doris_skip_unity_inclusioninbe/CMakeLists.txt, the missing-python3 branch ofcheck-build-hygiene.sh, the missing-header errors incheck-header-deps.py). - Every failure message carries three parts: the violation, the
mechanism (why this edge/entry is expensive — model:
check-header-deps.py's reason/chain/fix form), and a concrete fix path the reader can act on without opening the script. - Escape hatches are tables in the script, not flags. Budgets,
whitelists, exception sets and
ALLOWlists live next to the rules so a deliberate change is a one-line reviewed diff in the same commit. Do not add bypass environment variables or config files. - Rebaselining budgets: forward closure budgets carry zero slack (bump =
explicit, justified edit); reverse reach baselines carry +10% and are
re-measured on the audit cadence with
check-header-deps.py --budget.
Changing a gate script
- Any behavior change to a
check-*script requires updating its self-test inbuild-support/tests/(red/green injection form: every seeded violation turns red with the fix path in the message, restoring turns it green) and runningbash build-support/tests/run.sh— all of it, since the entry scripts aggregate. - The BE hygiene self-tests briefly mutate working-tree files (backed up and restored by EXIT traps): do not run them concurrently with a build/configure of the same tree, and do not "fix" them by pointing at fixtures — the gate tables name real headers on purpose.
- Keep scripts portable: bash 3.2 (macOS), BSD and GNU userland — in
particular
sed -ineeds the-i.bak+rmform. Python: stdlib only, no third-party imports. - Performance envelope: the combined configure gate is ~1s today; keep any addition within a low single-digit second budget, with zero build dependency (no compiler, no compile_commands.json).