mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
HappenLee
c309a6aeb6
### What problem does this PR solve? Issue Number: N/A Related PR: #68369 Problem Summary: Doris maintains separate handwritten SSE, NEON and scalar UTF-8 validators. Replace them with the simdutf dependency added in #68369, keeping the existing boolean validation entry point and file-scan validation setting. SIMD selection and short-input handling use simdutf's runtime dispatch; no CPU-specific or length-specific dispatch is added in Doris. Remove the obsolete implementation and scalar-only API, update its remaining test caller, and drop the old source/distribution license and tooling exclusions. Bundle simdutf 9.2.0's complete BSD-3-Clause ISA-detection notice in the binary distribution and reference it from the license manifest. Fix the existing UTF-8 tests, whose `sizeof(vector)` loops only exercised the first test case, and extend coverage for Unicode boundaries, malformed and truncated sequences, embedded NUL, empty/null buffers, SIMD boundaries, unaligned exact-sized buffers, long inputs, independent row validation and the file-scan bypass. ### Release note None ### Check List (For Author) - Test: Unit tests and static checks - License follow-up: verified the bundled BSD notice byte-for-byte against simdutf v9.2.0, checked both license-tool exclusion removals, and executed the existing `build.sh` packaging function to confirm the manifest and complete notice are included intact. - License-Eye v0.8.0 passed with the repository configuration on the four follow-up files and the newly unexcluded `utf8_check.cpp` (1 valid source, 0 invalid, 4 ignored license/configuration files). The full local worktree scan stopped on the existing `doris-skills` submodule directory (`read doris-skills: is a directory`); the full repository license check is left to CI. - Passed all 9 UTF-8 utility tests in a standalone harness compiling the actual modified implementation, test source and generated Thrift types. Both the implementation and simdutf 9.2.0 were built with ASAN/UBSAN. Passed with automatic dispatch and the supported icelake, haswell, westmere and fallback implementations. - Passed repository formatting and build-hygiene checks; clang-tidy passed for the UTF-8 implementation and tests. - Attempted the standard BE ASAN test runner with UTF-8, ColumnString, SQL-function and Variant filters. The build stopped in unchanged `common/cpp/aws_common.cpp` because the local AWS dependency lacks `aws/core/auth/GeneralHTTPCredentialsProvider.h`; these integrated tests did not run. - clang-tidy for the one-line Variant test-helper update is blocked by an existing unmatched `NOLINTEND` in unchanged `be/src/core/types.h:576`. - Behavior changed: No; validation semantics and the file-scan bypass are preserved. - Does this need documentation: No
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
This directory is used to organize the list of third-party dependent licenses required by the binary release.
1. `LICENSE-dist.txt`
This file holds a list of all the licenses for third-party dependencies
2. `licenses/`
This directory holds the license files related to third-party dependencies.
2. `tools/`
This directory stores tools related to license organization.
3. `download_scripts/`
This directory stores scripts for downloading release binaries.