linrrarity
7acfd9bfc2
Related PR: https://github.com/apache/doris/issues/66901 ### What problem does this PR solve? Several shared BE headers transitively include protobuf, Thrift, thread-pool, and storage-reader implementations. Translation units that only need declarations must still parse these definitions, increasing compilation work and the rebuild scope when those headers change. Recent CI builds failed during CMake configuration because the reverse-reach counts for threadpool.h and rle_encoding.h exceeded the build-hygiene limits. This PR audits all 32 header-budget entries and reduces unnecessary dependencies without changing the budgets or exception lists. The changes: - Extract `ColumnIteratorOptions` into a dedicated header so `vexpr_context.h` does not require the full `column_reader.h` dependency chain. - Use forward declarations for `thread-pool`, `protobuf`, `cache`, and `WorkloadGroup `types where complete definitions are unnecessary. - Move `UniqueId` conversions, `protobuf` methods in numeric `SerDe`, and `selected construction/destruction` operations into `.cpp` files. - Extract `SyncSizeCallback` into a dedicated header to reduce RPC dependencies propagated through the runtime-filter interface. - Add direct includes at actual use sites and verify independent compilation. Object layouts, move semantics, explicit template instantiations, and per-value decoding implementations are preserved. Moving some definitions into .cpp files changes their visibility for inlining in ordinary separate compilation. Query runtime performance has not been measured in this PR. ### Dependency reductions The following counts come from Ninja dependencies after completed builds, covering 1,531 production translation units and 52 generated translation units: | Header | TUs Before | TUs Now | New Limit | |---|---:|---:|---:| | util/rle_encoding.h | 846 | 231 | 250 | | util/threadpool.h | 1,056 | 585 | 600 | | gen_cpp/types.pb.h | 1,862 | 1,225 | 1,250 | | gen_cpp/segment_v2.pb.h | 1,231 | 1,156 | 1,200 | | gen_cpp/internal_service.pb.h | 777 | 557 | 600 | | gen_cpp/FrontendService_types.h | 593 | 293 | 300 | | gen_cpp/cloud.pb.h | 325 | 30 | 50 | | gen_cpp/BackendService_types.h | 195 | 115 | 150 | | gen_cpp/data.pb.h | 228 | 90 | 100 | | storage/options.h | 502 | 293 | 350 | | io/fs/s3_file_system.h | 65 | 31 | 50 | | gen_cpp/Types_types.h | 2,078 | 2,073 | 2,100 | | gen_cpp/PaloInternalService_types.h | 1,093 | 1,093 | 1,100 | | util/brpc_closure.h | 62 | 35 | 50 | | gen_cpp/Status_types.h | 37 | 37 | 41 | | runtime/workload_group/workload_group.h | 43 | 38 | 47 | Across the 16 monitored heavy headers, measured header-to-TU dependency pairs decrease from 8,108 to 6,617, an 18.39% reduction. These removals affect 742 distinct translation units. The number of production compilation units remains unchanged. The transitive project-header closure of dependency.h decreases from 325 to 246 headers, a 24.31% reduction. This uses the hygiene script’s static include graph and is reported separately from the measured Ninja dependency counts. ### Query runtime performance ### TPC-H SF1000 | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 59.615 | -9.93% | | master 2026-10-13 | 60.776 | -11.65% | | master 2026-10-15 | 57.021 | -5.83% | | master 2026-10-20 | 55.877 | -3.90% | | master 2026-10-27 | 65.987 | -18.62% | | PR #69717 | 53.697 | — | ### SSB SF1000 | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 11.418 | +3.71% | | master 2026-10-13 | 11.749 | +0.79% | | master 2026-10-15 | 10.653 | +11.16% | | master 2026-10-20 | 10.773 | +9.92% | | master 2026-10-27 | 11.471 | +3.23% | | PR #69717 | 11.842 | — | ### ClickBench Configuration: cluster_type=single, ecs.c9i.4xlarge_16c32g, one 500 GB disk, full analyze, PR #69717 release package. | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 24.717 | -1.83% | | master 2026-10-13 | 26.406 | -8.11% | | master 2026-10-15 | 25.079 | -3.25% | | master 2026-10-20 | 24.629 | -1.48% | | master 2026-10-27 | 23.643 | +2.63% | | PR #69717 single rerun | 24.265 | — | ### TPC-DS SF1000 (1 TB) Comparisons use the PR result from 2026-10-29. The 2026-10-28 rerun is listed separately. | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 168.926 | -7.40% | | master 2026-10-13 | 170.676 | -8.35% | | master 2026-10-15 | 163.170 | -4.13% | | master 2026-10-20 | 163.926 | -4.58% | | master 2026-10-27 | 195.853 | -20.13% | | PR #69717 2026-10-29 main result | 156.423 | — | ### Release note None ### Check List (For Author) - Test: Three complete Release builds; standard ASAN BE/FE build and final incremental build; all BE UT sources compiled and linked; 29 selected unit tests passed with -j50 --coverage; 1,024-row service read, aggregate, shuffle join, sort and cast checks; header hygiene and changed-line format checks passed. Clang-tidy covered all 71 files and retained two existing warnings in relocated protobuf methods (function size and signedness). - Behavior changed: No - Does this need documentation: No
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt