mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
HappenLee
c309a6aeb6
### What problem does this PR solve? Issue Number: N/A Related PR: #68369 Problem Summary: Doris maintains separate handwritten SSE, NEON and scalar UTF-8 validators. Replace them with the simdutf dependency added in #68369, keeping the existing boolean validation entry point and file-scan validation setting. SIMD selection and short-input handling use simdutf's runtime dispatch; no CPU-specific or length-specific dispatch is added in Doris. Remove the obsolete implementation and scalar-only API, update its remaining test caller, and drop the old source/distribution license and tooling exclusions. Bundle simdutf 9.2.0's complete BSD-3-Clause ISA-detection notice in the binary distribution and reference it from the license manifest. Fix the existing UTF-8 tests, whose `sizeof(vector)` loops only exercised the first test case, and extend coverage for Unicode boundaries, malformed and truncated sequences, embedded NUL, empty/null buffers, SIMD boundaries, unaligned exact-sized buffers, long inputs, independent row validation and the file-scan bypass. ### Release note None ### Check List (For Author) - Test: Unit tests and static checks - License follow-up: verified the bundled BSD notice byte-for-byte against simdutf v9.2.0, checked both license-tool exclusion removals, and executed the existing `build.sh` packaging function to confirm the manifest and complete notice are included intact. - License-Eye v0.8.0 passed with the repository configuration on the four follow-up files and the newly unexcluded `utf8_check.cpp` (1 valid source, 0 invalid, 4 ignored license/configuration files). The full local worktree scan stopped on the existing `doris-skills` submodule directory (`read doris-skills: is a directory`); the full repository license check is left to CI. - Passed all 9 UTF-8 utility tests in a standalone harness compiling the actual modified implementation, test source and generated Thrift types. Both the implementation and simdutf 9.2.0 were built with ASAN/UBSAN. Passed with automatic dispatch and the supported icelake, haswell, westmere and fallback implementations. - Passed repository formatting and build-hygiene checks; clang-tidy passed for the UTF-8 implementation and tests. - Attempted the standard BE ASAN test runner with UTF-8, ColumnString, SQL-function and Variant filters. The build stopped in unchanged `common/cpp/aws_common.cpp` because the local AWS dependency lacks `aws/core/auth/GeneralHTTPCredentialsProvider.h`; these integrated tests did not run. - clang-tidy for the one-line Variant test-helper update is blocked by an existing unmatched `NOLINTEND` in unchanged `be/src/core/types.h:576`. - Behavior changed: No; validation semantics and the file-scan bypass are preserved. - Does this need documentation: No
40 lines
438 B
Plaintext
40 lines
438 B
Plaintext
.*json
|
|
.*rst
|
|
.*svg
|
|
.*xml
|
|
.*zip
|
|
.clang-format
|
|
.gitattributes
|
|
.gitignore
|
|
.gitleaks.toml
|
|
.rat-excludes
|
|
DISCLAIMER-WIP
|
|
LICENSE
|
|
NOTICE
|
|
gutil/*
|
|
glibc-compatibility/*
|
|
apache-orc/*
|
|
clucene/*
|
|
manifest
|
|
patches/*
|
|
data/*
|
|
test_data/*
|
|
test.zip
|
|
test.zip.md5
|
|
webroot/*
|
|
net_snmp_normal
|
|
.*log
|
|
lru_cache.*
|
|
jmockit/*
|
|
status.*
|
|
env*
|
|
lru*
|
|
signal_handler.h
|
|
skiplist.h
|
|
string_search.hpp
|
|
coding.*
|
|
condition_variable.*
|
|
murmur_hash3.*
|
|
.markdownlintignore
|
|
sse2neon.h
|