Jack
eb16c4d81f
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: A SNII dictionary larger than the resident threshold (256 KB) is read block by block on demand. Every exact term lookup reads the term's DICT block (64 KB target), decompresses it with zstd, checks its CRC and parses its anchors, and the request-scoped DictBlockCache keeps nothing across queries. Every query therefore pays this again for every term in every segment. On a 99M-row AgentLogs table with 722 segments, counting one frequent term spent 68 ms of index time on these lookups against 16 ms for V3, and stack samples of postings-only and phrase queries showed 7-14% of SNII time in decompressing dictionary blocks alone. A SNII reader that the searcher cache keeps now keeps its recent exact lookups as well: a bounded LRU of up to 32 terms per reader, each at most 64 bytes long with at most 256 bytes of inline postings, which also remembers terms its dictionary turned out not to hold. `lookup()` and `lookup_batch()` check it before they read a block, so a batch reads only the blocks of the terms it does not hold. The cache's full footprint, about 25 KB, is reserved in the reader's searcher cache charge when the reader is inserted, as norms are, so the memory stays accounted. Only readers kept by the searcher cache enable it, and only when their dictionary is read on demand. The entries live inside the reader, so they go away with it and never outlive the file they describe. Caching the decoded blocks instead was not chosen: an exact lookup touches one 64 KB block per segment, so blocks would cost about 46 MB per hot term and column on that table, while a kept lookup costs a few hundred bytes. Keeping blocks in the searcher cache would also count against its fd-derived entry limit and push readers out. Prefix and pattern walks over the dictionary are not covered. Hot latency on the 99M-row table, cache off/on interleaved in one BE process: | Query | Off -> On | |---|---| | phrase 'retry budget depleted' (DS2) | 0.60 | | AND of two terms in two columns (A4) | 0.74-0.79 | | one term in two columns (A3) | 0.80 | | count of one frequent term (A2) | 0.84 | | regexp, its matched terms looked up again (DS9) | 0.89 | | phrase prefix (DS4) | 0.93 | | the other 19 queries | 0.92-1.05, within run-to-run noise | Against V3 on the same data (SNII/V3), DS2 went from 1.29 to 0.81, A4 from 1.24 to 0.95, A3 from 1.21 to 1.01 and A2 from 1.07 to 0.97.
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt