Jack
8d9e9c3f6d
### What problem does this PR solve? Issue Number: N/A Related PR: #67702 (closed) Problem Summary: A segment whose schema owns an inverted index always owns an index file, even when no logical index had anything to write: an all-NULL VARIANT column extracts no subcolumn, so no CLucene directory is ever opened and the container is closed with nothing in it. That empty container is committed by `close()`, not by `create_file()`, and `IndexFileWriter`'s empty-file path only closed three named implementations -- `StreamSinkFileWriter`, `S3FileWriter`, `PackedFileWriter`. `LocalFileWriter` and `HdfsFileWriter` were left open, and `~LocalFileWriter` aborts -- and DELETES -- a writer it was never asked to close. So on local storage the file silently disappeared and `show_nested_index_file` reported `E-6003 ... not found` where `E-6004 ... is empty` was the truth. Closing through `FileWriter::state()` instead of a type whitelist fixes that, but on its own it is not enough: it makes one latent `IndexBuilder` invariant violation real, and it leaves a second `IndexBuilder` bug -- already reachable today on the sink-node and remote paths, where the empty file does persist -- untouched. **`IndexBuilder` writes a container for every segment it rewrites**, including when the ALTER leaves the schema with no index at all: dropping the last index, or adding an index whose column does not exist. Until now the local writer's destructor happened to delete those files. Once they are closed they persist as orphans that nothing ever links, copies, uploads, removes or checksums -- `BetaRowset` decides whether to touch the container by asking `has_inverted_or_ann_index()` of the rowset's own schema, and both segment-writer paths (`SegmentFlusher::_create_segment_writer`, `VerticalBetaRowsetWriter`) already gate creation the same way. `IndexBuilder` was the one caller that did not. **`IndexBuilder` refuses to read an empty container.** `update_inverted_index_info()` tolerates `E-6003` ("this rowset predates the index") but not `E-6004` ("this rowset's index is empty"), even though both mean the same thing there -- there is nothing to carry over, build everything from the raw columns. `BUILD INDEX` / `ADD INDEX` / `DROP INDEX` on a table with an all-NULL VARIANT index therefore fails the whole ALTER with `[E-6004]inverted index file ... is empty`. So: three changes, one invariant. > a rowset's schema owns an inverted/ANN index **iff** every one of its segments > owns an index file, possibly zero length 1. `IndexFileWriter::begin_close()` / `finish_close()` close the empty container through `FileWriter` for any non-null, non-CLOSED writer, preserving the two-phase asynchronous protocol and propagating close errors. 2. `IndexBuilder::handle_single_rowset()` writes no container when the output rowset schema owns no index -- on the drop path and the add path alike. 3. `IndexBuilder::update_inverted_index_info()` reads an empty container exactly the way it reads a missing one. Consumers already accept `E-6004`: queries downgrade unconditionally (`SegmentIterator::_downgrade_without_index`), index compaction marks the segment ineligible and rebuilds, `CollectionStatistics` skips it. SNII is untouched -- its container always carries a header, so it is never zero length, and its `has_container` probe stays `E-6003`-only. #67702 made change 1 and then updated `regression-test/plugins/plugin_curl_requester.groovy` and `test_drop_index_with_format_v2.groovy` to expect `E-6004` after dropping every index. That expectation describes the orphan file, so this PR leaves both at `E-6003` and removes the orphan instead. ### Release note Persist empty V2/V3 inverted index files through every FileWriter implementation instead of only the remote ones; stop failing ALTER on a segment whose index file is legitimately empty; stop leaving an unreferenced index file behind when the last index of a table is dropped.
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt