mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
4bc6b148ca
### What problem does this PR solve? Introducing Ann index to doris. This pull request introduces foundational support for ANN (Approximate Nearest Neighbor) vector index functionality in the storage engine, including new runtime structures, configuration options, and initial integration with the build system. The changes lay the groundwork for ANN-based search and statistics collection, and begin integrating ANN index support into various storage and query execution paths. The implementation of ann index is based on [faiss](https://github.com/facebookresearch/faiss). Faiss could return distance directly, so this pr using [virtual slot ref](https://github.com/apache/doris/pull/52701) to return result from index. Each data segment of doris will have a faiss index if user creates a table with Ann index, and new segment generated by compaction will have a faiss index automatically. Currently, create index and build index is not supported, index defination be added to ddl if you want it. **ANN Index Feature Integration:** * Added new runtime structures and parameters for ANN index operations, including `AnnIndexStats`, `AnnIndexParam`, `RangeSearchParams`, `RangeSearchResult`, and others in `ann_search_params.h`, as well as `RangeSearchRuntimeInfo` for managing ANN range search context. [[1]](diffhunk://#diff-088dbea44296fb3669fe0cd22005df6aff33f8b60b20d5a49a68c6bbd22c29d1R1-R109) [[2]](diffhunk://#diff-ec34e664611a5877cab8f157919c35fe9b697428533c702536c97fd4f05769bdR1-R97) [[3]](diffhunk://#diff-d41283d91ba2756db2b45cadf964a78ad1a5c3360e1b854cb1a3d1f60817c804R1-R44) * Extended `StorageReadOptions` and `RowsetReaderContext` to include `ann_topn_runtime` for passing ANN runtime information through the storage read path. [[1]](diffhunk://#diff-8dec3cee74c5e0821835a7125bede7e3358bd3b5067b2748262193cb4cf80d48R126) [[2]](diffhunk://#diff-19fb296aa338021a0806017aa78ddadbea1791de3f4545724e34f8d9683a6551R95) [[3]](diffhunk://#diff-66ec81528c724b2f69e242d61f35d8a54d7bb5286a7e334c486166a3cc946642R106) * Added new ANN-related statistics fields (timing and row counts) to `OlapReaderStatistics` for monitoring ANN index operations. **Build System and Dependency Updates:** * Added `doris-faiss` and `doris-openblas` as submodules for ANN/vector index support, and integrated the new `Vector` library into the build process and as a dependency for relevant targets. [[1]](diffhunk://#diff-fe7afb5c9c916e521401d3fcfb4277d5071798c3baf83baf11d6071742823584R24-R31) [[2]](diffhunk://#diff-3507aac2aff9b5fe5f66d28967f3aa848491d4ced2466f6bf201ab3a97531837R532) [[3]](diffhunk://#diff-3507aac2aff9b5fe5f66d28967f3aa848491d4ced2466f6bf201ab3a97531837R787-R788) [[4]](diffhunk://#diff-d67261040b7ca84a64e8aeef5f7e1a8bab5efcc20fcdd3402f24160f56c29959R26) **Index Handling and Schema Integration:** * Updated index file writer accessors and naming from "inverted_index" to more generic "index" to accommodate ANN and other index types. [[1]](diffhunk://#diff-0c1c144f791918ef5b05ded169a7efb22a0ae67565e641cc03c31d4c2872729eL747-R748) [[2]](diffhunk://#diff-60cd05e044b4218e4a4d774abe89636fa0f6e1290dd0ff7892231d30770cd2b1L193-R193) * Changed index creation logic in `SegmentFlusher` to use `has_extra_index()` (supporting both inverted and ANN indexes) instead of `has_inverted_index()`. [[1]](diffhunk://#diff-7e9f53b4ef59bdb00d10393a2941be9201ddd46c3aab957d1dae8bc5d8898ebeL139-R139) [[2]](diffhunk://#diff-7e9f53b4ef59bdb00d10393a2941be9201ddd46c3aab957d1dae8bc5d8898ebeL157-R157) [[3]](diffhunk://#diff-7e9f53b4ef59bdb00d10393a2941be9201ddd46c3aab957d1dae8bc5d8898ebeL176-R176) [[4]](diffhunk://#diff-7e9f53b4ef59bdb00d10393a2941be9201ddd46c3aab957d1dae8bc5d8898ebeL193-R193) **Configuration:** * Introduced a new configuration option `opm_threads_limit` to control the maximum number of OpenMP threads used per Doris thread, which is relevant for vectorized/ANN computation. [[1]](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffR1578-R1580) [[2]](diffhunk://#diff-46e8c1ada0d43acf8c2965e46e90909089aada1f46531976c10605b837f8da3dR1634-R1635) These changes set up the infrastructure required for future development of ANN vector index features, including search, filtering, and statistics collection. Co-authored-by: chenlinzhong <490103404@qq.com> Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
32 lines
1.2 KiB
Plaintext
32 lines
1.2 KiB
Plaintext
[submodule ".github/actions/label-when-approved-action"]
|
|
path = .github/actions/label-when-approved-action
|
|
url = https://github.com/TobKed/label-when-approved-action
|
|
[submodule ".github/actions/get-workflow-origin"]
|
|
path = .github/actions/get-workflow-origin
|
|
url = https://github.com/potiuk/get-workflow-origin.git
|
|
[submodule ".github/actions/paths-filter"]
|
|
path = .github/actions/paths-filter
|
|
url = https://github.com/dorny/paths-filter
|
|
[submodule ".github/actions/action-pr-title"]
|
|
path = .github/actions/action-pr-title
|
|
url = https://github.com/deepakputhraya/action-pr-title.git
|
|
[submodule ".github/actions/ccache-action"]
|
|
path = .github/actions/ccache-action
|
|
url = https://github.com/hendrikmuhs/ccache-action
|
|
[submodule "contrib/clucene"]
|
|
path = contrib/clucene
|
|
url = https://github.com/apache/doris-thirdparty.git
|
|
branch = clucene
|
|
[submodule "contrib/apache-orc"]
|
|
path = contrib/apache-orc
|
|
url = https://github.com/apache/doris-thirdparty.git
|
|
branch = orc
|
|
[submodule "doris-faiss"]
|
|
path = contrib/faiss
|
|
url = https://github.com/apache/doris-thirdparty.git
|
|
branch = faiss
|
|
[submodule "doris-openblas"]
|
|
path = contrib/openblas
|
|
url = https://github.com/apache/doris-thirdparty.git
|
|
branch = openblas
|