mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
lihangyu
64d2911f64
### What problem does this PR solve? Problem Summary: An inverted index on a VARIANT column is copied to every extracted subcolumn, so a segment holds one copy of it per path. Since #53980 every analyzed index calls `setOmitNorms(false)`, and CLucene writes norms densely: 12 header bytes plus **one byte per segment row for every indexed field**, including the rows that hold no value for that field. For a VARIANT column that cost is multiplied by the number of paths. On a real table (~1400 paths, 7 non-BKD indexes per path) a single base-compacted segment held 9891 `.nrm` files of 4,055,884 bytes each: **37.36 GiB of norms out of a 38.47 GiB `.idx`, 97.1%**. Segments written before the upgrade had no `.nrm` at all, and the non-norms bytes had not grown. Reproduced end to end on 1M rows / 1000 paths / 11 `field_pattern` indexes, by loading with 3.1 and then upgrading the same data in place to 4.1: | | index size after full compaction | `.nrm` | | --- | --- | --- | | 3.1 | 99.6 MiB | none | | 4.1, same data | 934.1 MiB | 875 files x 1,000,012 B = 834.5 MiB | **9.4x**, and the norms are the whole difference. A/B on the same data: norms on 457.3 MiB vs norms off 99.6 MiB (**4.6x**), with identical `MATCH` row counts and sums. #### What this PR changes This PR gives operators a way to reclaim that space without changing what anyone gets by default, and applies it the same way to the CLucene formats (V2/V3) and to SNII. 1. **A `norms` index property.** An analyzed index writes norms unless it sets `"norms" = "false"`, on a VARIANT path exactly as on an ordinary column. A subcolumn copy inherits the properties of the index it came from, so the property on a whole-column VARIANT index also settles every one of its subcolumn copies. 2. **A mutable BE config `inverted_index_skip_norms_for_variant`, off by default.** Turning it on drops norms for every index on a VARIANT path (an index declared with a `field_pattern`, or the copy an extracted subcolumn inherits, which carries the path as its index suffix), whatever that index's property says, so a cluster hit by the size regression can recover the space without rewriting its index definitions. 3. **One policy function.** `should_write_index_norms(const TabletIndex&)` holds the rule, and the three places that decide whether norms get written all call it: the CLucene writer, the SNII writer, and SNII direct compaction. SNII direct compaction rebuilds norms from the merged postings rather than copying them; the other SNII compaction path rebuilds the index through the SNII writer. 4. **BM25 scoring needs norms from every segment it reads, in both formats.** `MATCH` filtering is unaffected; a query that computes `score()` on an analyzed index fails with `INVERTED_INDEX_NOT_SUPPORTED` when any segment of the field has no norms. - SNII already worked this way (`resolve_snii_scoring_segment`); its message now names the property and the config instead of suggesting that compaction will bring the norms back. - CLucene did not. It keeps a field's token count in the `.nrm` header, and a row without norms reads the fake norm `encodeNorm(0)`, i.e. length 0. With no segment carrying norms, `avgdl` was 0 and `score()` came out NaN. With only some segments carrying norms, as while the config is being turned on, `avgdl` stayed positive and the rows without norms ranked as the shortest possible documents, above otherwise identical rows. `CollectionStatistics` now refuses such a segment before any score is computed. An index that is not analyzed never writes norms and is collected as before, so `SEARCH` scoring over keyword fields is unchanged. #### Compatibility - Defaults are unchanged: norms are still written everywhere unless someone opts out, and an index that has norms scores exactly as before. - No storage format change. - Turning the config on, or setting `"norms" = "false"`, is for indexes that are never ranked with `score()`. For such an index, `score()` fails as soon as one segment without norms exists. - A table upgraded from a version that did not write norms (before 4.0) now fails `score()` on its analyzed indexes until compaction has rewritten the old segments, where it used to return NaN or skewed scores. - Change the property or the config only after every BE runs this version. ### Release note Added the `norms` inverted index property and the BE config `inverted_index_skip_norms_for_variant` (off by default, mutable), which drops BM25 norms for inverted indexes on VARIANT paths and can cut the index size of a VARIANT column with many indexed paths by several times. Both apply to the V2, V3 and SNII index storage formats. A query that computes `score()` on an analyzed index now fails when a segment it reads has no norms, instead of returning NaN or inconsistent scores. ### Check List (For Author) - Test: Unit Test / Regression test - `be/test/storage/segment/inverted_index_writer_test.cpp`: `NormsFollowIndexNormsProperty` - `be/test/storage/index/snii_writer_test.cpp`: `SniiWriterNorms.WritesNormsFollowSharedNormsPolicy` - `be/test/storage/index/snii/compaction/snii_compaction_eligibility_test.cpp`: `DestinationWritesNormsFollowSharedNormsPolicy` - `be/test/storage/index/inverted/similarity/collection_statistics_test.cpp`: `LegacyV3RejectsSegmentWrittenWithoutNorms` (alone and next to a segment with norms), `LegacyV3KeywordIndexWithoutNormsIsStillCollected`, and the existing SNII rejection tests - `fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java` - Regression: `inverted_index_p0/test_variant_subcolumn_index_norms` (checks the `.nrm` files) and `inverted_index_p0/storage_format/test_storage_format_snii_norms` (reads norms off the scores): `MATCH` still filters, `score()` fails without norms, including on a mixed-generation table. Both are nonConcurrent, since they flip the BE config. Also rerun: `test_bm25_score_variant`, `test_omit_norms`, `test_search_score_topn_predicates`, `test_search_score_cache`, `test_search_score_topn_delete_predicate`, `test_storage_format_snii` - Behavior changed: No for defaults. `score()` on an analyzed index with a segment that has no norms now fails instead of returning NaN or skewed scores. - Does this need documentation: Yes (https://github.com/apache/doris-website/pull/4146)
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt