mirror of
https://github.com/apache/doris.git
synced 2026-11-04 18:29:59 +00:00
morrySnow
a6a7e52a73
### What problem does this PR solve? Problem Summary: Every SQL lexer currently copies its input through `CharStreams.fromString()` and calls `Character.toUpperCase()` for every case-insensitive lookahead. This is visible in lexer allocation and identifier-heavy parsing time. Add two strictly compatible fast paths: - Fold ASCII `a-z` with arithmetic and retain `Character.toUpperCase()` for every other code point. - Read strings without UTF-16 surrogates directly, avoiding ANTLR's copied code-point buffer. Inputs containing any surrogate retain the original ANTLR stream so code-point indices, `getText`, navigation, and errors remain unchanged. The public arbitrary-`CharStream` constructor is preserved. All production String entry points now use the factory. ANTLR's native `caseInsensitive` option was evaluated but rejected because it changed existing Unicode behavior such as the handling of `ſelect`. ### Benchmark Environment and method: - Baseline: `cf33a08bcd5`, artifact SHA-256 `51700bef953361218912abc6aec348cecdec3fe61031eba10cc1ebc534bc9183` - Candidate: `7aa7b5e6487`, artifact SHA-256 `6e8edf78befc0dc349be398aae454c172b5cb968ee98d88a0e056d4c21a9c5d0` - macOS 15.0.1 arm64, OpenJDK 17.0.20.1, 1 GiB heap, JMH 1.37 - 2 forks, 4 x 300 ms warmup, 7 x 400 ms measurement, `-prof gc` - Interleaved baseline/candidate runs; one candidate run affected by unrelated host load was discarded and repeated. The table averages two valid runs per artifact. Individual JMH scores use 99.9% confidence intervals. ```shell java -Xms1g -Xmx1g -jar <benchmark.jar> \ 'CaseInsensitiveStreamBenchmark.(createLexer|foldPrebuiltCharacters|parseStatement|tokenize)' \ -p workload=shortQuery,lowercaseIdentifiers,stringAndComment,unicode \ -f 2 -wi 4 -i 7 -w 300ms -r 400ms -prof gc -rf json ``` | Benchmark | Baseline us/op (B1 / B2) | Candidate us/op (C1 / C3) | Mean change | Baseline → candidate B/op | | -------------------------------------- | --------------------------: | --------------------------: | ----------: | ------------------------: | | Tokenize 64 lowercase identifiers | 10.826±0.078 / 11.408±0.125 | 8.708±0.071 / 8.704±0.096 | -21.7% | 14,799 → 10,464 (-29.3%) | | Parse 64 lowercase identifiers | 54.775±7.360 / 56.365±4.224 | 49.860±0.191 / 52.340±1.160 | -8.0% | 99,395 → 95,017 (-4.4%) | | Tokenize `select 1` | 0.451±0.004 / 0.485±0.011 | 0.348±0.069 / 0.324±0.010 | -28.2% | 1,008 → 816 (-19.0%) | | Parse strings/comments with Unicode | 27.047±5.371 / 28.435±2.932 | 24.744±0.401 / 24.743±0.425 | -10.8% | 24,167 → 23,784 (-1.6%) | | Tokenize supplementary Unicode control | 32.034±1.398 / 33.910±0.752 | 32.303±1.108 / 32.020±0.738 | -2.5% | 24,630 → 24,594 (-0.1%) | The isolated `createLexer` measurement for the surrogate-containing string/comment workload regresses by 5.9% because the compatibility guard scans for surrogates before falling back. The corresponding complete tokenize and parse paths improve by 3.6% and 10.8%; no end-to-end control workload regressed. Prebuilt lowercase character folding improves by 10.6%. The performance gains come from eliminating the copied input buffer for BMP-only SQL, avoiding its allocation, and replacing the common lowercase ASCII `Character.toUpperCase()` call with an arithmetic branch. Correctness corpus: - 4,610 tracked SQL files; baseline and candidate parse signatures match in legacy and ANSI modes. SHA-256: `69c811d0d80c52b40d8cd854e925ff4930aa6601c7d1e66541f2eb29f35db50a`. - 9,220 lexer cases (4,610 SQL files x both `noBackslashEscapes` modes); complete token tuple and lexer-error signatures match. SHA-256: `66780d4e1d9224ea27c8f715ab33edd247faed1571030ba9e90a8ef3f6212180`.