HappenLee
c309a6aeb6
### What problem does this PR solve? Issue Number: N/A Related PR: #68369 Problem Summary: Doris maintains separate handwritten SSE, NEON and scalar UTF-8 validators. Replace them with the simdutf dependency added in #68369, keeping the existing boolean validation entry point and file-scan validation setting. SIMD selection and short-input handling use simdutf's runtime dispatch; no CPU-specific or length-specific dispatch is added in Doris. Remove the obsolete implementation and scalar-only API, update its remaining test caller, and drop the old source/distribution license and tooling exclusions. Bundle simdutf 9.2.0's complete BSD-3-Clause ISA-detection notice in the binary distribution and reference it from the license manifest. Fix the existing UTF-8 tests, whose `sizeof(vector)` loops only exercised the first test case, and extend coverage for Unicode boundaries, malformed and truncated sequences, embedded NUL, empty/null buffers, SIMD boundaries, unaligned exact-sized buffers, long inputs, independent row validation and the file-scan bypass. ### Release note None ### Check List (For Author) - Test: Unit tests and static checks - License follow-up: verified the bundled BSD notice byte-for-byte against simdutf v9.2.0, checked both license-tool exclusion removals, and executed the existing `build.sh` packaging function to confirm the manifest and complete notice are included intact. - License-Eye v0.8.0 passed with the repository configuration on the four follow-up files and the newly unexcluded `utf8_check.cpp` (1 valid source, 0 invalid, 4 ignored license/configuration files). The full local worktree scan stopped on the existing `doris-skills` submodule directory (`read doris-skills: is a directory`); the full repository license check is left to CI. - Passed all 9 UTF-8 utility tests in a standalone harness compiling the actual modified implementation, test source and generated Thrift types. Both the implementation and simdutf 9.2.0 were built with ASAN/UBSAN. Passed with automatic dispatch and the supported icelake, haswell, westmere and fallback implementations. - Passed repository formatting and build-hygiene checks; clang-tidy passed for the UTF-8 implementation and tests. - Attempted the standard BE ASAN test runner with UTF-8, ColumnString, SQL-function and Variant filters. The build stopped in unchanged `common/cpp/aws_common.cpp` because the local AWS dependency lacks `aws/core/auth/GeneralHTTPCredentialsProvider.h`; these integrated tests did not run. - clang-tidy for the one-line Variant test-helper update is blocked by an existing unmatched `NOLINTEND` in unchanged `be/src/core/types.h:576`. - Behavior changed: No; validation semantics and the file-scan bypass are preserved. - Does this need documentation: No
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt