nprasad2077/doris
1
0 Fork 0
mirror of https://github.com/apache/doris.git synced 2026-11-04 07:31:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

[refactor](be) Replace handwritten UTF-8 validators with simdutf (#68406)

Browse Source 
### What problem does this PR solve?

Issue Number: N/A

Related PR: #68369

Problem Summary:

Doris maintains separate handwritten SSE, NEON and scalar UTF-8
validators. Replace them with the simdutf dependency added in #68369,
keeping the existing boolean validation entry point and file-scan
validation setting. SIMD selection and short-input handling use
simdutf's runtime dispatch; no CPU-specific or length-specific dispatch
is added in Doris.

Remove the obsolete implementation and scalar-only API, update its
remaining test caller, and drop the old source/distribution license and
tooling exclusions. Bundle simdutf 9.2.0's complete BSD-3-Clause
ISA-detection notice in the binary distribution and reference it from
the license manifest. Fix the existing UTF-8 tests, whose
`sizeof(vector)` loops only exercised the first test case, and extend
coverage for Unicode boundaries, malformed and truncated sequences,
embedded NUL, empty/null buffers, SIMD boundaries, unaligned exact-sized
buffers, long inputs, independent row validation and the file-scan
bypass.

### Release note

None

### Check List (For Author)

- Test: Unit tests and static checks
- License follow-up: verified the bundled BSD notice byte-for-byte
against simdutf v9.2.0, checked both license-tool exclusion removals,
and executed the existing `build.sh` packaging function to confirm the
manifest and complete notice are included intact.
- License-Eye v0.8.0 passed with the repository configuration on the
four follow-up files and the newly unexcluded `utf8_check.cpp` (1 valid
source, 0 invalid, 4 ignored license/configuration files). The full
local worktree scan stopped on the existing `doris-skills` submodule
directory (`read doris-skills: is a directory`); the full repository
license check is left to CI.
- Passed all 9 UTF-8 utility tests in a standalone harness compiling the
actual modified implementation, test source and generated Thrift types.
Both the implementation and simdutf 9.2.0 were built with ASAN/UBSAN.
Passed with automatic dispatch and the supported icelake, haswell,
westmere and fallback implementations.
- Passed repository formatting and build-hygiene checks; clang-tidy
passed for the UTF-8 implementation and tests.
- Attempted the standard BE ASAN test runner with UTF-8, ColumnString,
SQL-function and Variant filters. The build stopped in unchanged
`common/cpp/aws_common.cpp` because the local AWS dependency lacks
`aws/core/auth/GeneralHTTPCredentialsProvider.h`; these integrated tests
did not run.
- clang-tidy for the one-line Variant test-helper update is blocked by
an existing unmatched `NOLINTEND` in unchanged
`be/src/core/types.h:576`.
- Behavior changed: No; validation semantics and the file-scan bypass
are preserved.
- Does this need documentation: No
This commit is contained in:
HappenLee authored and GitHub committed 2026-09-23 21:52:12 +08:00
1 parent 9de831aeb8
commit c309a6aeb6
12 files changed
+200 -734
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.clang-format-ignore
-1
View File
@@ -8,6 +8,5 @@ be/src/util/sse2neo.h
be/src/util/sse2neon.h
be/src/util/mustache/mustache.h
be/src/util/mustache/mustache.cc
be/src/util/utf8_check.cpp
be/src/storage/index/inverted/analyzer/kuromoji/dict/darts.h
cloud/src/common/defer.h
.licenserc.yaml
-1
View File
@@ -126,7 +126,6 @@ header:
- "be/src/util/mustache/mustache.h"
- "be/src/util/sse2neo.h"
- "be/src/util/sse2neon.h"
- "be/src/util/utf8_check.cpp"
- "be/src/storage/index/inverted/analyzer/kuromoji/dict/darts.h"
- "be/src/pch/*"
- "be/test/data"
.rat-excludes
-1
View File
@@ -35,6 +35,5 @@ string_search.hpp
coding.*
condition_variable.*
murmur_hash3.*
utf8_check.cpp
.markdownlintignore
sse2neon.h
LICENSE.txt
-26
View File
@@ -412,32 +412,6 @@ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-------------------------------------------------------------------------------
be/src/util/utf8_check.cpp: MIT license
MIT License
Copyright (c) 2019 Yibo Cai
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
--------------------------------------------------------------------------------
be/src/util/condition_variable* : BSD-style license
Copyright (c) 2011 The Chromium Authors. All rights reserved.
be/src/util/simdutf8check.h
-313
View File
@@ -1,313 +0,0 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
#pragma once
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <string.h>
#include <x86intrin.h>
/*
* These functions are used for validating utf8 string.
* Details can be seen here: https://github.com/lemire/fastvalidate-utf-8
*/
/*
* legal utf-8 byte sequence
* http://www.unicode.org/versions/Unicode6.0.0/ch03.pdf - page 94
*
* Code Points 1st 2s 3s 4s
* U+0000..U+007F 00..7F
* U+0080..U+07FF C2..DF 80..BF
* U+0800..U+0FFF E0 A0..BF 80..BF
* U+1000..U+CFFF E1..EC 80..BF 80..BF
* U+D000..U+D7FF ED 80..9F 80..BF
* U+E000..U+FFFF EE..EF 80..BF 80..BF
* U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
* U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
* U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
*
*/
// all byte values must be no larger than 0xF4
static inline void checkSmallerThan0xF4(__m128i current_bytes, __m128i* has_error) {
// unsigned, saturates to 0 below max
*has_error = _mm_or_si128(*has_error, _mm_subs_epu8(current_bytes, _mm_set1_epi8(0xF4)));
}
static inline __m128i continuationLengths(__m128i high_nibbles) {
return _mm_shuffle_epi8(_mm_setr_epi8(1, 1, 1, 1, 1, 1, 1, 1, // 0xxx (ASCII)
0, 0, 0, 0, // 10xx (continuation)
2, 2, // 110x
3, // 1110
4), // 1111, next should be 0 (not checked here)
high_nibbles);
}
static inline __m128i carryContinuations(__m128i initial_lengths, __m128i previous_carries) {
__m128i right1 = _mm_subs_epu8(_mm_alignr_epi8(initial_lengths, previous_carries, 16 - 1),
_mm_set1_epi8(1));
__m128i sum = _mm_add_epi8(initial_lengths, right1);
__m128i right2 =
_mm_subs_epu8(_mm_alignr_epi8(sum, previous_carries, 16 - 2), _mm_set1_epi8(2));
return _mm_add_epi8(sum, right2);
}
static inline void checkContinuations(__m128i initial_lengths, __m128i carries,
__m128i* has_error) {
// overlap || underlap
// carry > length && length > 0 || !(carry > length) && !(length > 0)
// (carries > length) == (lengths > 0)
__m128i overunder = _mm_cmpeq_epi8(_mm_cmpgt_epi8(carries, initial_lengths),
_mm_cmpgt_epi8(initial_lengths, _mm_setzero_si128()));
*has_error = _mm_or_si128(*has_error, overunder);
}
// when 0xED is found, next byte must be no larger than 0x9F
// when 0xF4 is found, next byte must be no larger than 0x8F
// next byte must be continuation, ie sign bit is set, so signed < is ok
static inline void checkFirstContinuationMax(__m128i current_bytes, __m128i off1_current_bytes,
__m128i* has_error) {
__m128i maskED = _mm_cmpeq_epi8(off1_current_bytes, _mm_set1_epi8(0xED));
__m128i maskF4 = _mm_cmpeq_epi8(off1_current_bytes, _mm_set1_epi8(0xF4));
__m128i badfollowED = _mm_and_si128(_mm_cmpgt_epi8(current_bytes, _mm_set1_epi8(0x9F)), maskED);
__m128i badfollowF4 = _mm_and_si128(_mm_cmpgt_epi8(current_bytes, _mm_set1_epi8(0x8F)), maskF4);
*has_error = _mm_or_si128(*has_error, _mm_or_si128(badfollowED, badfollowF4));
}
// map off1_hibits => error condition
// hibits off1 cur
// C => < C2 && true
// E => < E1 && < A0
// F => < F1 && < 90
// else false && false
static inline void checkOverlong(__m128i current_bytes, __m128i off1_current_bytes, __m128i hibits,
__m128i previous_hibits, __m128i* has_error) {
__m128i off1_hibits = _mm_alignr_epi8(hibits, previous_hibits, 16 - 1);
__m128i initial_mins =
_mm_shuffle_epi8(_mm_setr_epi8(-128, -128, -128, -128, -128, -128, -128, -128, -128,
-128, -128, -128, // 10xx => false
0xC2, -128, // 110x
0xE1, // 1110
0xF1),
off1_hibits);
__m128i initial_under = _mm_cmpgt_epi8(initial_mins, off1_current_bytes);
__m128i second_mins =
_mm_shuffle_epi8(_mm_setr_epi8(-128, -128, -128, -128, -128, -128, -128, -128, -128,
-128, -128, -128, // 10xx => false
127, 127, // 110x => true
0xA0, // 1110
0x90),
off1_hibits);
__m128i second_under = _mm_cmpgt_epi8(second_mins, current_bytes);
*has_error = _mm_or_si128(*has_error, _mm_and_si128(initial_under, second_under));
}
struct processed_utf_bytes {
__m128i rawbytes;
__m128i high_nibbles;
__m128i carried_continuations;
};
static inline void count_nibbles(__m128i bytes, struct processed_utf_bytes* answer) {
answer->rawbytes = bytes;
answer->high_nibbles = _mm_and_si128(_mm_srli_epi16(bytes, 4), _mm_set1_epi8(0x0F));
}
// check whether the current bytes are valid UTF-8
// at the end of the function, previous gets updated
static struct processed_utf_bytes checkUTF8Bytes(__m128i current_bytes,
struct processed_utf_bytes* previous,
__m128i* has_error) {
struct processed_utf_bytes pb;
count_nibbles(current_bytes, &pb);
checkSmallerThan0xF4(current_bytes, has_error);
__m128i initial_lengths = continuationLengths(pb.high_nibbles);
pb.carried_continuations = carryContinuations(initial_lengths, previous->carried_continuations);
checkContinuations(initial_lengths, pb.carried_continuations, has_error);
__m128i off1_current_bytes = _mm_alignr_epi8(pb.rawbytes, previous->rawbytes, 16 - 1);
checkFirstContinuationMax(current_bytes, off1_current_bytes, has_error);
checkOverlong(current_bytes, off1_current_bytes, pb.high_nibbles, previous->high_nibbles,
has_error);
return pb;
}
static bool validate_utf8_fast(const char* src, size_t len) {
size_t i = 0;
__m128i has_error = _mm_setzero_si128();
struct processed_utf_bytes previous = {.rawbytes = _mm_setzero_si128(),
.high_nibbles = _mm_setzero_si128(),
.carried_continuations = _mm_setzero_si128()};
if (len >= 16) {
for (; i <= len - 16; i += 16) {
__m128i current_bytes = _mm_loadu_si128((const __m128i*)(src + i));
previous = checkUTF8Bytes(current_bytes, &previous, &has_error);
}
}
// last part
if (i < len) {
char buffer[16];
memset(buffer, 0, 16);
memcpy(buffer, src + i, len - i);
__m128i current_bytes = _mm_loadu_si128((const __m128i*)(buffer));
previous = checkUTF8Bytes(current_bytes, &previous, &has_error);
} else {
has_error = _mm_or_si128(
_mm_cmpgt_epi8(previous.carried_continuations,
_mm_setr_epi8(9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 1)),
has_error);
}
return _mm_testz_si128(has_error, has_error);
}
#ifdef __AVX2__
/*****************************/
static inline __m256i push_last_byte_of_a_to_b(__m256i a, __m256i b) {
return _mm256_alignr_epi8(b, _mm256_permute2x128_si256(a, b, 0x21), 15);
}
static inline __m256i push_last_2bytes_of_a_to_b(__m256i a, __m256i b) {
return _mm256_alignr_epi8(b, _mm256_permute2x128_si256(a, b, 0x21), 14);
}
// all byte values must be no larger than 0xF4
static inline void avxcheckSmallerThan0xF4(__m256i current_bytes, __m256i* has_error) {
// unsigned, saturates to 0 below max
*has_error =
_mm256_or_si256(*has_error, _mm256_subs_epu8(current_bytes, _mm256_set1_epi8(0xF4)));
}
static inline __m256i avxcontinuationLengths(__m256i high_nibbles) {
return _mm256_shuffle_epi8(_mm256_setr_epi8(1, 1, 1, 1, 1, 1, 1, 1, // 0xxx (ASCII)
0, 0, 0, 0, // 10xx (continuation)
2, 2, // 110x
3, // 1110
4, // 1111, next should be 0 (not checked here)
1, 1, 1, 1, 1, 1, 1, 1, // 0xxx (ASCII)
0, 0, 0, 0, // 10xx (continuation)
2, 2, // 110x
3, // 1110
4 // 1111, next should be 0 (not checked here)
),
high_nibbles);
}
static inline __m256i avxcarryContinuations(__m256i initial_lengths, __m256i previous_carries) {
__m256i right1 = _mm256_subs_epu8(push_last_byte_of_a_to_b(previous_carries, initial_lengths),
_mm256_set1_epi8(1));
__m256i sum = _mm256_add_epi8(initial_lengths, right1);
__m256i right2 = _mm256_subs_epu8(push_last_2bytes_of_a_to_b(previous_carries, sum),
_mm256_set1_epi8(2));
return _mm256_add_epi8(sum, right2);
}
static inline void avxcheckContinuations(__m256i initial_lengths, __m256i carries,
__m256i* has_error) {
// overlap || underlap
// carry > length && length > 0 || !(carry > length) && !(length > 0)
// (carries > length) == (lengths > 0)
__m256i overunder =
_mm256_cmpeq_epi8(_mm256_cmpgt_epi8(carries, initial_lengths),
_mm256_cmpgt_epi8(initial_lengths, _mm256_setzero_si256()));
*has_error = _mm256_or_si256(*has_error, overunder);
}
// when 0xED is found, next byte must be no larger than 0x9F
// when 0xF4 is found, next byte must be no larger than 0x8F
// next byte must be continuation, ie sign bit is set, so signed < is ok
static inline void avxcheckFirstContinuationMax(__m256i current_bytes, __m256i off1_current_bytes,
__m256i* has_error) {
__m256i maskED = _mm256_cmpeq_epi8(off1_current_bytes, _mm256_set1_epi8(0xED));
__m256i maskF4 = _mm256_cmpeq_epi8(off1_current_bytes, _mm256_set1_epi8(0xF4));
__m256i badfollowED =
_mm256_and_si256(_mm256_cmpgt_epi8(current_bytes, _mm256_set1_epi8(0x9F)), maskED);
__m256i badfollowF4 =
_mm256_and_si256(_mm256_cmpgt_epi8(current_bytes, _mm256_set1_epi8(0x8F)), maskF4);
*has_error = _mm256_or_si256(*has_error, _mm256_or_si256(badfollowED, badfollowF4));
}
// map off1_hibits => error condition
// hibits off1 cur
// C => < C2 && true
// E => < E1 && < A0
// F => < F1 && < 90
// else false && false
static inline void avxcheckOverlong(__m256i current_bytes, __m256i off1_current_bytes,
__m256i hibits, __m256i previous_hibits, __m256i* has_error) {
__m256i off1_hibits = push_last_byte_of_a_to_b(previous_hibits, hibits);
__m256i initial_mins =
_mm256_shuffle_epi8(_mm256_setr_epi8(-128, -128, -128, -128, -128, -128, -128, -128,
-128, -128, -128, -128, // 10xx => false
0xC2, -128, // 110x
0xE1, // 1110
0xF1, -128, -128, -128, -128, -128, -128, -128,
-128, -128, -128, -128, -128, // 10xx => false
0xC2, -128, // 110x
0xE1, // 1110
0xF1),
off1_hibits);
__m256i initial_under = _mm256_cmpgt_epi8(initial_mins, off1_current_bytes);
__m256i second_mins =
_mm256_shuffle_epi8(_mm256_setr_epi8(-128, -128, -128, -128, -128, -128, -128, -128,
-128, -128, -128, -128, // 10xx => false
127, 127, // 110x => true
0xA0, // 1110
0x90, -128, -128, -128, -128, -128, -128, -128,
-128, -128, -128, -128, -128, // 10xx => false
127, 127, // 110x => true
0xA0, // 1110
0x90),
off1_hibits);
__m256i second_under = _mm256_cmpgt_epi8(second_mins, current_bytes);
*has_error = _mm256_or_si256(*has_error, _mm256_and_si256(initial_under, second_under));
}
struct avx_processed_utf_bytes {
__m256i rawbytes;
__m256i high_nibbles;
__m256i carried_continuations;
};
static inline void avx_count_nibbles(__m256i bytes, struct avx_processed_utf_bytes* answer) {
answer->rawbytes = bytes;
answer->high_nibbles = _mm256_and_si256(_mm256_srli_epi16(bytes, 4), _mm256_set1_epi8(0x0F));
}
#endif // __AVX2__
be/src/util/utf8_check.cpp
+19 -321
View File
@@ -1,332 +1,29 @@
// Copyright (c) cyb70289(https://github.com/cyb70289). All rights reserved.
// Use of this source code is governed by a MIT license that can be
// found in the LICENSE file.
/*
* These functions are used for validating utf8 string.
* Details can be seen here: https://github.com/cyb70289/utf8/
*/
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
#include "util/utf8_check.h"
#if defined(__i386) || defined(__x86_64__)
#include "util/simdutf8check.h"
#elif defined(__aarch64__)
#include <arm_neon.h>
#endif
#include <simdutf.h>
/*
* http://www.unicode.org/versions/Unicode6.0.0/ch03.pdf - page 94
*
* Table 3-7. Well-Formed UTF-8 Byte Sequences
*
* +--------------------+------------+-------------+------------+-------------+
* | Code Points | First Byte | Second Byte | Third Byte | Fourth Byte |
* +--------------------+------------+-------------+------------+-------------+
* | U+0000..U+007F | 00..7F | | | |
* +--------------------+------------+-------------+------------+-------------+
* | U+0080..U+07FF | C2..DF | 80..BF | | |
* +--------------------+------------+-------------+------------+-------------+
* | U+0800..U+0FFF | E0 | A0..BF | 80..BF | |
* +--------------------+------------+-------------+------------+-------------+
* | U+1000..U+CFFF | E1..EC | 80..BF | 80..BF | |
* +--------------------+------------+-------------+------------+-------------+
* | U+D000..U+D7FF | ED | 80..9F | 80..BF | |
* +--------------------+------------+-------------+------------+-------------+
* | U+E000..U+FFFF | EE..EF | 80..BF | 80..BF | |
* +--------------------+------------+-------------+------------+-------------+
* | U+10000..U+3FFFF | F0 | 90..BF | 80..BF | 80..BF |
* +--------------------+------------+-------------+------------+-------------+
* | U+40000..U+FFFFF | F1..F3 | 80..BF | 80..BF | 80..BF |
* +--------------------+------------+-------------+------------+-------------+
* | U+100000..U+10FFFF | F4 | 80..8F | 80..BF | 80..BF |
* +--------------------+------------+-------------+------------+-------------+
*/
namespace doris {
bool validate_utf8_naive(const char* data, size_t len) {
while (len) {
int bytes;
const unsigned char byte1 = data[0];
/* 00..7F */
if (byte1 <= 0x7F) {
bytes = 1;
/* C2..DF, 80..BF */
} else if (len >= 2 && byte1 >= 0xC2 && byte1 <= 0xDF &&
(signed char)data[1] <= (signed char)0xBF) {
bytes = 2;
} else if (len >= 3) {
const unsigned char byte2 = data[1];
/* Is byte2, byte3 between 0x80 ~ 0xBF */
const int byte2_ok = (signed char)byte2 <= (signed char)0xBF;
const int byte3_ok = (signed char)data[2] <= (signed char)0xBF;
if (byte2_ok && byte3_ok &&
/* E0, A0..BF, 80..BF */
((byte1 == 0xE0 && byte2 >= 0xA0) ||
/* E1..EC, 80..BF, 80..BF */
(byte1 >= 0xE1 && byte1 <= 0xEC) ||
/* ED, 80..9F, 80..BF */
(byte1 == 0xED && byte2 <= 0x9F) ||
/* EE..EF, 80..BF, 80..BF */
(byte1 >= 0xEE && byte1 <= 0xEF))) {
bytes = 3;
} else if (len >= 4) {
/* Is byte4 between 0x80 ~ 0xBF */
const int byte4_ok = (signed char)data[3] <= (signed char)0xBF;
if (byte2_ok && byte3_ok && byte4_ok &&
/* F0, 90..BF, 80..BF, 80..BF */
((byte1 == 0xF0 && byte2 >= 0x90) ||
/* F1..F3, 80..BF, 80..BF, 80..BF */
(byte1 >= 0xF1 && byte1 <= 0xF3) ||
/* F4, 80..8F, 80..BF, 80..BF */
(byte1 == 0xF4 && byte2 <= 0x8F))) {
bytes = 4;
} else {
return false;
}
} else {
return false;
}
} else {
return false;
}
len -= bytes;
data += bytes;
}
return true;
}
#if defined(__i386) || defined(__x86_64__)
bool validate_utf8(const char* src, size_t len) {
return validate_utf8_fast(src, len);
}
#elif defined(__aarch64__)
/*
* Map high nibble of "First Byte" to legal character length minus 1
* 0x00 ~ 0xBF --> 0
* 0xC0 ~ 0xDF --> 1
* 0xE0 ~ 0xEF --> 2
* 0xF0 ~ 0xFF --> 3
*/
const uint8_t _first_len_tbl[] = {
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 2, 3,
};
/* Map "First Byte" to 8-th item of range table (0xC2 ~ 0xF4) */
static const uint8_t _first_range_tbl[] = {
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 8,
};
/*
* Range table, map range index to min and max values
* Index 0 : 00 ~ 7F (First Byte, ascii)
* Index 1,2,3: 80 ~ BF (Second, Third, Fourth Byte)
* Index 4 : A0 ~ BF (Second Byte after E0)
* Index 5 : 80 ~ 9F (Second Byte after ED)
* Index 6 : 90 ~ BF (Second Byte after F0)
* Index 7 : 80 ~ 8F (Second Byte after F4)
* Index 8 : C2 ~ F4 (First Byte, non ascii)
* Index 9~15 : illegal: u >= 255 && u <= 0
*/
static const uint8_t _range_min_tbl[] = {
0x00, 0x80, 0x80, 0x80, 0xA0, 0x80, 0x90, 0x80,
0xC2, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
};
static const uint8_t _range_max_tbl[] = {
0x7F, 0xBF, 0xBF, 0xBF, 0xBF, 0x9F, 0xBF, 0x8F,
0xF4, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
};
/*
* This table is for fast handling four special First Bytes(E0,ED,F0,F4), after
* which the Second Byte are not 80~BF. It contains "range index adjustment".
* - The idea is to minus byte with E0, use the result(0~31) as the index to
* lookup the "range index adjustment". Then add the adjustment to original
* range index to get the correct range.
* - Range index adjustment
* +------------+---------------+------------------+----------------+
* | First Byte | original range| range adjustment | adjusted range |
* +------------+---------------+------------------+----------------+
* | E0 | 2 | 2 | 4 |
* +------------+---------------+------------------+----------------+
* | ED | 2 | 3 | 5 |
* +------------+---------------+------------------+----------------+
* | F0 | 3 | 3 | 6 |
* +------------+---------------+------------------+----------------+
* | F4 | 4 | 4 | 8 |
* +------------+---------------+------------------+----------------+
* - Below is a uint8x16x2 table, data is interleaved in NEON register. So I'm
* putting it vertically. 1st column is for E0~EF, 2nd column for F0~FF.
*/
static const uint8_t _range_adjust_tbl[] = {
/* index -> 0~15 16~31 <- index */
/* E0 -> */ 2,
3, /* <- F0 */
0,
0,
0,
0,
0,
0,
0,
4, /* <- F4 */
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
/* ED -> */ 3,
0,
0,
0,
0,
0,
};
/* 2x ~ 4x faster than naive method */
/* Return true on success, false on error */
bool utf8_range(const char* data, size_t len) {
if (len >= 16) {
uint8x16_t prev_input = vdupq_n_u8(0);
uint8x16_t prev_first_len = vdupq_n_u8(0);
/* Cached tables */
const uint8x16_t first_len_tbl = vld1q_u8(_first_len_tbl);
const uint8x16_t first_range_tbl = vld1q_u8(_first_range_tbl);
const uint8x16_t range_min_tbl = vld1q_u8(_range_min_tbl);
const uint8x16_t range_max_tbl = vld1q_u8(_range_max_tbl);
const uint8x16x2_t range_adjust_tbl = vld2q_u8(_range_adjust_tbl);
/* Cached values */
const uint8x16_t const_1 = vdupq_n_u8(1);
const uint8x16_t const_2 = vdupq_n_u8(2);
const uint8x16_t const_e0 = vdupq_n_u8(0xE0);
uint8x16_t error = vdupq_n_u8(0);
while (len >= 16) {
const uint8x16_t input = vld1q_u8((const uint8_t*)data);
/* high_nibbles = input >> 4 */
const uint8x16_t high_nibbles = vshrq_n_u8(input, 4);
/* first_len = legal character length minus 1 */
/* 0 for 00~7F, 1 for C0~DF, 2 for E0~EF, 3 for F0~FF */
/* first_len = first_len_tbl[high_nibbles] */
const uint8x16_t first_len = vqtbl1q_u8(first_len_tbl, high_nibbles);
/* First Byte: set range index to 8 for bytes within 0xC0 ~ 0xFF */
/* range = first_range_tbl[high_nibbles] */
uint8x16_t range = vqtbl1q_u8(first_range_tbl, high_nibbles);
/* Second Byte: set range index to first_len */
/* 0 for 00~7F, 1 for C0~DF, 2 for E0~EF, 3 for F0~FF */
/* range |= (first_len, prev_first_len) << 1 byte */
range = vorrq_u8(range, vextq_u8(prev_first_len, first_len, 15));
/* Third Byte: set range index to saturate_sub(first_len, 1) */
/* 0 for 00~7F, 0 for C0~DF, 1 for E0~EF, 2 for F0~FF */
uint8x16_t tmp1, tmp2;
/* tmp1 = saturate_sub(first_len, 1) */
tmp1 = vqsubq_u8(first_len, const_1);
/* tmp2 = saturate_sub(prev_first_len, 1) */
tmp2 = vqsubq_u8(prev_first_len, const_1);
/* range |= (tmp1, tmp2) << 2 bytes */
range = vorrq_u8(range, vextq_u8(tmp2, tmp1, 14));
/* Fourth Byte: set range index to saturate_sub(first_len, 2) */
/* 0 for 00~7F, 0 for C0~DF, 0 for E0~EF, 1 for F0~FF */
/* tmp1 = saturate_sub(first_len, 2) */
tmp1 = vqsubq_u8(first_len, const_2);
/* tmp2 = saturate_sub(prev_first_len, 2) */
tmp2 = vqsubq_u8(prev_first_len, const_2);
/* range |= (tmp1, tmp2) << 3 bytes */
range = vorrq_u8(range, vextq_u8(tmp2, tmp1, 13));
/*
* Now we have below range indices caluclated
* Correct cases:
* - 8 for C0~FF
* - 3 for 1st byte after F0~FF
* - 2 for 1st byte after E0~EF or 2nd byte after F0~FF
* - 1 for 1st byte after C0~DF or 2nd byte after E0~EF or
* 3rd byte after F0~FF
* - 0 for others
* Error cases:
* 9,10,11 if non ascii First Byte overlaps
* E.g., F1 80 C2 90 --> 8 3 10 2, where 10 indicates error
*/
/* Adjust Second Byte range for special First Bytes(E0,ED,F0,F4) */
/* See _range_adjust_tbl[] definition for details */
/* Overlaps lead to index 9~15, which are illegal in range table */
uint8x16_t shift1 = vextq_u8(prev_input, input, 15);
uint8x16_t pos = vsubq_u8(shift1, const_e0);
range = vaddq_u8(range, vqtbl2q_u8(range_adjust_tbl, pos));
/* Load min and max values per calculated range index */
uint8x16_t minv = vqtbl1q_u8(range_min_tbl, range);
uint8x16_t maxv = vqtbl1q_u8(range_max_tbl, range);
/* Check value range */
error = vorrq_u8(error, vcltq_u8(input, minv));
error = vorrq_u8(error, vcgtq_u8(input, maxv));
prev_input = input;
prev_first_len = first_len;
data += 16;
len -= 16;
}
/* Delay error check till loop ends */
if (vmaxvq_u8(error)) return false;
/* Find previous token (not 80~BF) */
uint32_t token4;
vst1q_lane_u32(&token4, vreinterpretq_u32_u8(prev_input), 3);
const int8_t* token = (const int8_t*)&token4;
int lookahead = 0;
if (token[3] > (int8_t)0xBF)
lookahead = 1;
else if (token[2] > (int8_t)0xBF)
lookahead = 2;
else if (token[1] > (int8_t)0xBF)
lookahead = 3;
data -= lookahead;
len += lookahead;
}
/* Check remaining bytes with naive method */
return validate_utf8_naive(data, len);
}
bool validate_utf8(const char* src, size_t len) {
return utf8_range(src, len);
return simdutf::validate_utf8(src, len);
}
#else
bool validate_utf8(const char* src, size_t len) {
return validate_utf8_naive(src, len);
}
#endif
bool validate_utf8(const TFileScanRangeParams& params, const char* src, size_t len) {
if (params.__isset.file_attributes && !params.file_attributes.enable_text_validate_utf8) {
@@ -334,4 +31,5 @@ bool validate_utf8(const TFileScanRangeParams& params, const char* src, size_t l
}
return validate_utf8(src, len);
}
} // namespace doris
be/src/util/utf8_check.h
+1 -4
View File
@@ -22,11 +22,8 @@
#include <cstddef>
namespace doris {
// check utf8 code using simd instructions
// Return true - success, false fail
// Validate UTF-8 with simdutf runtime CPU dispatch. Empty input is valid.
bool validate_utf8(const char* src, size_t len);
// check utf8 use naive c++
bool validate_utf8_naive(const char* data, size_t len);
bool validate_utf8(const TFileScanRangeParams& params, const char* src, size_t len);
} // namespace doris
be/test/util/utf8_check_test.cpp
+131 -39
View File
@@ -17,45 +17,58 @@
#include "util/utf8_check.h"
#include <gtest/gtest-message.h>
#include <gtest/gtest-test-part.h>
#include <gtest/gtest.h>
#include <cstring>
#include <memory>
#include <string>
#include <string_view>
#include <vector>
#include "gtest/gtest_pred_impl.h"
namespace doris {
struct test {
const char* data;
int len;
};
class Utf8CheckTest : public testing::Test {
public:
Utf8CheckTest() {}
virtual ~Utf8CheckTest() {}
protected:
static void check(std::string_view input, bool expected) {
// Exact-sized buffers let an instrumented validator detect reads past the input.
auto data = std::make_unique<char[]>(input.size());
std::memcpy(data.get(), input.data(), input.size());
EXPECT_EQ(validate_utf8(data.get(), input.size()), expected);
}
private:
/* positive tests */
std::vector<test> pos = {{"", 0},
{"\x00", 1},
{"\x66", 1},
{"\x7F", 1},
{"\x00\x7F", 2},
{"\x7F\x00", 2},
{"\xC2\x80", 2},
{"\xDF\xBF", 2},
{"\xE0\xA0\x80", 3},
{"\xE0\xA0\xBF", 3},
{"\xED\x9F\x80", 3},
{"\xEF\x80\xBF", 3},
{"\xF0\x90\xBF\x80", 4},
{"\xF2\x81\xBE\x99", 4},
{"\xF4\x8F\x88\xAA", 4}};
std::vector<std::string_view> pos = {{"", 0},
{"\x00", 1},
{"f", 1},
{"\x7F", 1},
{"\x00\x7F", 2},
{"\x7F\x00", 2},
{"\xC2\x80", 2},
{"\xDF\xBF", 2},
{"\xE0\xA0\x80", 3},
{"\xE0\xA0\xBF", 3},
{"\xED\x9F\x80", 3},
{"\xEF\x80\xBF", 3},
{"\xF0\x90\xBF\x80", 4},
{"\xF2\x81\xBE\x99", 4},
{"\xF4\x8F\x88\xAA", 4},
{"\xED\x9F\xBF", 3},
{"\xEE\x80\x80", 3},
{"\xEF\xBF\xBF", 3},
{"\xF0\x90\x80\x80", 4},
{"\xF4\x8F\xBF\xBF", 4}};
/* negative tests */
std::vector<test> neg = {
std::vector<std::string_view> neg = {
{"\xC2", 1},
{"\xE0", 1},
{"\xE0\xA0", 2},
{"\xF0", 1},
{"\xF0\x90", 2},
{"\xF0\x90\x80", 3},
{"\xF5\x80\x80\x80", 4},
{"\xFE", 1},
{"\xFF", 1},
{"\x80", 1},
{"\xBF", 1},
{"\xC0\x80", 2},
@@ -96,28 +109,107 @@ private:
35}};
};
TEST_F(Utf8CheckTest, empty) {
EXPECT_TRUE(validate_utf8(pos[0].data, pos[0].len));
EXPECT_TRUE(validate_utf8(nullptr, 0));
check("", true);
}
TEST_F(Utf8CheckTest, normal) {
for (int i = 0; i < sizeof(pos) / sizeof(pos[0]); ++i) {
EXPECT_TRUE(validate_utf8(pos[i].data, pos[i].len));
for (const auto& value : pos) {
check(value, true);
}
}
TEST_F(Utf8CheckTest, abnormal) {
for (int i = 0; i < sizeof(neg) / sizeof(neg[0]); ++i) {
EXPECT_FALSE(validate_utf8(neg[i].data, neg[i].len));
for (const auto& value : neg) {
check(value, false);
}
}
TEST_F(Utf8CheckTest, naive) {
for (int i = 0; i < sizeof(pos) / sizeof(pos[0]); ++i) {
EXPECT_TRUE(validate_utf8_naive(pos[i].data, pos[i].len));
TEST_F(Utf8CheckTest, embedded_nul) {
check(std::string_view("a\0\xE4\xB8\xAD\0z", 7), true);
check(std::string_view("a\0\xFF", 3), false);
check(std::string_view("a\0\xE4\xB8", 4), false);
}
TEST_F(Utf8CheckTest, block_boundaries) {
// Exercise every sequence across 16-, 32- and 64-byte SIMD boundaries,
// both at the end of the input and followed by another complete block.
for (size_t prefix_size = 0; prefix_size < 130; ++prefix_size) {
SCOPED_TRACE(prefix_size);
for (size_t suffix_size : {0, 1, 16, 32, 64}) {
SCOPED_TRACE(suffix_size);
const std::string prefix(prefix_size, 'a');
const std::string suffix(suffix_size, 'b');
for (const auto& value : pos) {
check(prefix + std::string(value) + suffix, true);
}
for (const auto& value : neg) {
check(prefix + std::string(value) + suffix, false);
}
}
}
for (int i = 0; i < sizeof(neg) / sizeof(neg[0]); ++i) {
EXPECT_FALSE(validate_utf8_naive(neg[i].data, neg[i].len));
}
TEST_F(Utf8CheckTest, unaligned_inputs) {
for (size_t offset = 0; offset < 64; ++offset) {
SCOPED_TRACE(offset);
for (size_t size : {1, 15, 16, 17, 31, 32, 33, 63, 64, 65, 127, 128, 129, 1024}) {
SCOPED_TRACE(size);
auto data = std::make_unique<char[]>(offset + size);
std::memset(data.get(), 'a', offset + size);
EXPECT_TRUE(validate_utf8(data.get() + offset, size));
data[offset + size - 1] = '\xFF';
EXPECT_FALSE(validate_utf8(data.get() + offset, size));
}
}
}
TEST_F(Utf8CheckTest, long_inputs) {
for (size_t size : {1024, 4096, 65536}) {
SCOPED_TRACE(size);
std::string data(size, 'a');
check(data, true);
for (size_t offset : {size_t(0), size / 2, size - 1}) {
data[offset] = '\xFF';
check(data, false);
data[offset] = 'a';
}
data.clear();
for (size_t i = 0; i < size; ++i) {
data += "a\xE4\xB8\xAD\xF0\x9F\x98\x80";
}
check(data, true);
data.pop_back();
check(data, false);
}
}
TEST_F(Utf8CheckTest, independent_inputs) {
// A truncated sequence in one row must not consume the next row's bytes.
const std::string data = "\xE4\xB8\xAD";
EXPECT_FALSE(validate_utf8(data.data(), 2));
EXPECT_FALSE(validate_utf8(data.data() + 2, 1));
EXPECT_TRUE(validate_utf8(data.data(), data.size()));
}
TEST_F(Utf8CheckTest, file_scan_validation_setting) {
TFileScanRangeParams params;
const std::string invalid = "\xFF";
EXPECT_FALSE(validate_utf8(params, invalid.data(), invalid.size()));
TFileAttributes attributes;
params.__set_file_attributes(attributes);
EXPECT_FALSE(validate_utf8(params, invalid.data(), invalid.size()));
attributes.__set_enable_text_validate_utf8(false);
params.__set_file_attributes(attributes);
EXPECT_TRUE(validate_utf8(params, invalid.data(), invalid.size()));
attributes.__set_enable_text_validate_utf8(true);
params.__set_file_attributes(attributes);
EXPECT_FALSE(validate_utf8(params, invalid.data(), invalid.size()));
EXPECT_TRUE(validate_utf8(params, "valid", 5));
EXPECT_TRUE(validate_utf8(params, nullptr, 0));
}
} // namespace doris
be/test/util/variant/variant_test_utils.cpp
+1 -1
View File
@@ -108,7 +108,7 @@ uint8_t minimum_decimal_width(__int128 value) {
}
void require_valid_utf8(StringRef value, const char* description) {
if (value.size != 0 && !validate_utf8_naive(value.data, value.size)) {
if (value.size != 0 && !validate_utf8(value.data, value.size)) {
fail(description);
}
}
build-support/run-clang-tidy.sh
-1
View File
@@ -50,7 +50,6 @@ EXCLUDED_PATTERNS=(
"be/src/util/mustache/"
"be/src/util/sse2neo.h"
"be/src/util/sse2neon.h"
"be/src/util/utf8_check.cpp"
"cloud/src/common/defer.h"
"contrib/"
)
dist/LICENSE-dist.txt
Vendored
+2 -26
View File
@@ -397,32 +397,6 @@ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-------------------------------------------------------------------------------
be/src/util/utf8_check.cpp: MIT license
MIT License
Copyright (c) 2019 Yibo Cai
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
--------------------------------------------------------------------------------
be/src/util/condition_variable* : BSD-style license
Copyright (c) 2011 The Chromium Authors. All rights reserved.
@@ -1482,6 +1456,8 @@ The Apache Software License, Version 2.0
* benchmark: 1.8.0
* simdjson: 3.0.1
* simdutf: 9.2.0
- includes BSD-3-Clause code: simdutf/internal/isadetection.h
- additional license: licenses/LICENSE-simdutf-isadetection.txt
* libhdfs3: 2.3.8
* opentelemetry-proto: 0.18.0
* opentelemetry-cpp: 1.4.0
dist/licenses/LICENSE-simdutf-isadetection.txt
Vendored
+46
View File
@@ -0,0 +1,46 @@
simdutf 9.2.0: simdutf/internal/isadetection.h (BSD-3-Clause)
Source: https://github.com/simdutf/simdutf/blob/v9.2.0/include/simdutf/internal/isadetection.h
From
https://github.com/endorno/pytorch/blob/master/torch/lib/TH/generic/simd/simd.h
Highly modified.
Copyright (c) 2016- Facebook, Inc (Adam Paszke)
Copyright (c) 2014- Facebook, Inc (Soumith Chintala)
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou,
Iain Melvin, Jason Weston) Copyright (c) 2006 Idiap Research Institute
(Samy Bengio) Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert,
Samy Bengio, Johnny Mariethoz)
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. Neither the names of Facebook, Deepmind Technologies, NYU, NEC Laboratories
America and IDIAP Research Institute nor the names of its contributors may be
used to endorse or promote products derived from this software without
specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.