Jack
428c24187c
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: The auto .prx codec stores each position window in the smallest of PFOR, zstd and raw. Every read of a zstd window pays a decompression that PFOR does not, and the choice ignores that cost. On 99M AgentLogs rows 74% of the position windows read by an unfiltered phrase query were zstd, and decoding them took 58% of the index execution time of the query. PrxWindowLimits gets zstd_min_saving_percent, set from the new mutable config snii_prx_zstd_min_saving_percent. A readable PFOR frame is replaced by the zstd one only if zstd is smaller by at least that percentage of the PFOR frame. The default is 30; 0 picks the smaller frame as before and writes a byte-identical index (99M rows: 61,886,854,661 bytes with and without this change). Readers handle every codec, so the setting affects only segments written afterwards, and compaction rewrites with whatever is set then. The compression is still computed, so the write cost does not change. Effect of the setting, 10M rows (index size, DS7 = unfiltered phrase on the output column, position decode time from the query profile): 10 -> +0.02% size, no change for DS7; 20 -> +0.5%, no change; 30 -> +2.1%, decode 252 ms -> 85 ms; 50 -> +6.3%, no further gain; 100 (never zstd) -> +8.5%. 30 is the smallest value that removes the zstd windows of frequent terms, which zstd beats PFOR by 20-30% on and which phrase queries read most. On 99M rows with 30: index +1.9% (63.07 GB against 61.89 GB, still below the 65.70 GB of V3), load time within 2% of a control load, hot queries in the same process 0.63x (unfiltered phrase), 0.67x (phrase in input or output), 0.75x (phrase prefix), 0.77x, 0.87x, the others within 1.02x (one query measured 1.06x once and 1.00-1.05x in three repeats); cold runs 10-18% faster for the phrase queries. The best value depends on the data, since it is how much zstd beats PFOR on the window contents; it was calibrated on one corpus.
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt