### What problem does this PR solve?
Problem Summary:
On glibc < 2.25, dynamic TLS can be allocated by libc and freed by
jemalloc, crashing during thread cleanup. Route all seven glibc
allocation entry points through the existing jemalloc wrappers, preserve
allocator API semantics, and add native interposition tests.
### Release note
Fix a BE crash during dynamic TLS cleanup on older glibc systems and
keep glibc allocation entry points paired with jemalloc.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
`./run-be-ut.sh --jemalloc-hook --run -j 2`: 12/12 passed on Linux
x86_64/glibc 2.32. Full BE, older glibc, ARM64 and customer JNI E2E were
not run.
- Behavior changed:
- [ ] No.
- [x] Yes. Glibc allocation entry points now consistently use jemalloc,
with allocator boundary semantics preserved.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
### What problem does this PR solve?
Issue Number: N/A
Problem Summary: Add `h3_lat_lng_to_cell`, `h3_cell_to_lat_lng`, and
`h3_cell_to_boundary` using official H3 4.5.0. Coordinates use degrees
in latitude/longitude order. Invalid inputs return NULL; centers contain
two coordinates and boundaries preserve H3 vertex order without closing
the ring. Input NULL propagation uses the default execution framework.
FE has no separate Java H3 evaluator; constant folding through BE
remains supported.
Build H3 as a static C library from its 18 upstream core sources.
Generate its versioned API header in the build directory, export include
paths through the target, enable PIC, and inherit Doris sanitizer flags.
Only the H3 target relaxes third-party warning errors. The wrapper
avoids upstream standalone tools and global build options. Include the
upstream license attribution and NOTICE.
### Release note
Add three H3 SQL functions for mapping coordinates to cells and
retrieving cell centers and boundaries.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. Add three SQL functions.
- Does this need documentation?
- [ ] No.
- [x] Yes. SQL function documentation is needed; no documentation PR
yet.
Validation on Linux x86_64:
- ASAN BE build and clean FE build passed; H3 objects carry ASAN/UBSAN
flags.
- `run-be-ut.sh --run --filter=FunctionH3Test.* -j48`: all three tests
passed.
- `run-regression-test.sh --run -d
query_p0/sql_functions/spatial_functions -s test_h3`: generated results
with `-forceGenOut`, inspected them, and reran successfully. Includes
constant-folding modes, nullable/non-nullable table columns, all 16
resolutions, stored nested arrays, array expansion, filtering/grouping,
and 35,000 mixed rows.
- Compared 10,000 inputs per function against official C/Python H3
4.5.0: exact agreement. ClickHouse 26.3.9.8 (H3 3.7.1) matched cell IDs
and coordinates within tolerance (maximum coordinate error 8.29e-8
degrees).
- Clang-format 16 and targeted clang-tidy checks passed; no sanitizer
diagnostics during these tests. ARM/macOS builds were not run.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
### What problem does this PR solve?
Issue Number: None
Related PR: #67180, #67538, #67918
Problem Summary:
MATCH and SEARCH each had their own Boolean, phrase, term-expansion and
scoring executors, written separately for CLucene (V1/V2) and SNII
indexes. Every query optimization had to be implemented once per
executor, the formats drifted apart in behavior, and cross-field queries
could mix document IDs from different physical partitions.
This PR runs both on one engine:
- **One logical form.** MATCH (`query/logical`) and SEARCH
(`exprs/function/search`) lower their queries to the same logical nodes.
- **One query engine.** Boolean logic (AND/OR/NOT, minimum_should_match,
WAND), term expansion (prefix, wildcard, regexp), exact and sloppy
phrases and BM25 scoring are implemented once, independent of the index
format.
- **Formats behind an SPI.** CLucene and SNII only provide dictionaries,
postings, positions, norms, statistics and batched I/O through
`query/spi`. The old per-format executors are removed.
- **Layered source tree.** `be/src/storage/index/` is reorganized by
layer, and `build-support/check-header-deps.py` enforces the allowed
include directions.
Index formats and writers are unchanged, and existing indexes remain
readable. FE is unchanged.
#### Source layout
```text
be/src/storage/index/
├── index_*.h/.cpp shared index files: file reader/writer, compound reader, storage formats
├── ann/ bloom_filter/ zone_map/
└── inverted/
├── analysis/ analyzers, tokenizers, char and token filters, normalizers, gram schemes
├── common/ descriptors, caches, statistics, profiles, single flight
├── reader/ text index reader, index selection, collection statistics
├── query/ format-independent query engine
│ ├── logical/ MATCH lowering and logical nodes (gram/ builds gram queries)
│ ├── engine/ Query/Weight/Scorer: boolean, term, phrase, expansion, WAND, union, collect
│ ├── exec/ doc-id sets and sinks, term waves, chained conjunctions
│ ├── phrase/ exact and sloppy phrase matchers
│ ├── scoring/ BM25
│ └── spi/ the source interface each format implements
├── clucene/ CLucene (V1/V2) reader, searcher, writer, compaction
│ └── spi/ CLucene index source and postings cursor
└── snii/ SNII: format/, encoding/, io/, bkd/, reader/, writer/, compaction/, query/
be/src/exprs/
├── vsearch.h/.cpp VSearchExpr, the SEARCH expression node
└── function/search/
├── function_search.* SEARCH entry: Boolean assembly, DSL result cache, NULL handling
├── search_lowering.* SEARCH DSL to logical nodes; binds each field to an index
├── function_search_leaf.* compiles each leaf into a query on its bound index
├── search_predicate_collector.* collects scored leaves for BM25 statistics
└── variant_inverted_index_search.* SEARCH on VARIANT subcolumns and NESTED groups
```
#### Performance
The 18 production SQL cases (V2 and SNII: point, range, absent-term and
phrase queries) were compared with master on fresh dual-BE pairs: 8
pairs, 16 samples per case and binary, alternating order. Every case is
within the 5% limit for both `InvertedIndexFilterTime` and client wall
time; the largest ratios are 1.032 and 1.038. SNII phrase cases take
25-60% less FilterTime. TPC-H, TPC-DS and ClickBench are within noise.
<details><summary>Per-case median ratios (PR / master)</summary>
Run on 2026-10-14 against master 7ac0eb2fdb on a busy shared host
(median 90% CPU), which widens the intervals; all 4,608 result checks
pass.
| Case | FilterTime | Client wall |
| --- | ---: | ---: |
| `v2/long/early/full` | 1.030 | 1.024 |
| `v2/long/early/range` | 1.018 | 0.999 |
| `v2/long/absent/full` | 1.032 | 1.014 |
| `v2/long/absent/range` | 1.018 | 0.995 |
| `v2/long/mid/full` | 1.006 | 1.038 |
| `v2/long/mid/range` | 1.013 | 1.012 |
| `v2/short/tiny` | 0.965 | 1.018 |
| `v2/search/scored_phrase` | 1.010 | 0.990 |
| `v2/search/phrase_2` | 0.954 | 0.951 |
| `snii/long/early/full` | 0.950 | 0.994 |
| `snii/long/early/range` | 0.961 | 1.022 |
| `snii/long/absent/full` | 0.728 | 0.847 |
| `snii/long/absent/range` | 0.891 | 0.983 |
| `snii/long/mid/full` | 0.821 | 0.924 |
| `snii/long/mid/range` | 0.975 | 1.007 |
| `snii/short/tiny` | 1.020 | 0.985 |
| `snii/search/scored_phrase` | 0.404 | 0.971 |
| `snii/search/phrase_2` | 0.746 | 0.986 |
</details>
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
**Context.** `run-be-ut.sh` is the only entry point for the BE unit
tests, and on macOS it is also the only way to run them: the `BE UT
(macOS)` workflow runs `build.sh --be`, which configures with
`-DMAKE_TEST=OFF`, so nothing under `be/test` is compiled there (the
workflow says so in its own comment at
`.github/workflows/be-ut-mac.yml:123`). On a clean worktree the script
therefore fails twice before a single test executes, and then stops
seven more times while compiling `be/test`.
**1. The problem, and what it cost**
*The script never initializes two of the submodules it needs.*
`run-be-ut.sh` initializes three submodules (`contrib/datasketches-cpp`,
`contrib/apache-orc`, `contrib/clucene`); `build.sh` initializes five,
and the two it adds — `contrib/faiss` and `contrib/openblas` — are
required before a single file is compiled. `be/CMakeLists.txt:1038` adds
`be/src/storage/index/ann` unconditionally, and that directory's
`cmake-protect/CMakeLists.txt:48,66` calls `add_subdirectory()` on both.
On a fresh clone or a new git worktree the configure step therefore ends
with
CMake Error at src/storage/index/ann/cmake-protect/CMakeLists.txt:48
(add_subdirectory):
The source directory .../contrib/openblas does not contain a
CMakeLists.txt file.
CMake Error at src/storage/index/ann/cmake-protect/CMakeLists.txt:66
(add_subdirectory):
The source directory .../contrib/faiss does not contain a CMakeLists.txt
file.
Running `build.sh` once first hides it, which is why it goes unnoticed.
*Seven `be/test` translation units do not compile on macOS.* The Linux
BE UT build is green for all seven. `run-be-ut.sh` invokes ninja without
`-k`, so the developer fixes one file, waits out another full compile,
and meets the next one — seven round trips, each of them tens of
minutes, for a task whose whole point is to run a test.
| # | TU | Error | Root cause |
|---|---|---|---|
| 1 | `exprs/function/function_map_inner_product_test.cpp` |
`'faiss/impl/platform_macros.h' file not found` | The `APPLE` branch of
`be/test/CMakeLists.txt:170` links `vector_search_test` as
`-Wl,-force_load,$<TARGET_FILE:vector_search_test>`: a file path, which
carries no usage requirements. Every other platform links
`-Wl,--whole-archive vector_search_test`, which is a target in the link
list and does propagate them. |
| 2 | `exprs/function/function_format_round_test.cpp` | `ambiguous
conversion for functional-style cast from 'long' to 'Decimal64'` (11
sites) | `Decimal64` is `Decimal<int64_t>`, so on Darwin the constructor
parameter is `long long` while these literals are `long`; every integral
and floating constructor is then an equally ranked conversion away. |
| 3 | `storage/index/snii/bench/bkd_native_vs_clucene_bench_test.cpp` |
six `-Werror,-Wformat` | `uint64_t` is `unsigned long long` on Darwin
and `unsigned long` on Linux, so `%lu` / `%ld` is right on one and wrong
on the other. |
| 4 | `storage/index/snii/bench/snii_vs_v3_benchmark_test.cpp` | `no
member named 'posix_fadvise'`, `mincore` argument type | `posix_fadvise`
/ `POSIX_FADV_DONTNEED` do not exist on Darwin; `mincore(2)` takes
`char*` there and `unsigned char*` on Linux. |
| 5 | `storage/index/snii/bkd/bkd_corruption_test.cpp` | `deduced
conflicting types ('size_type' vs 'uint64_t')` | An initializer list
mixing `payload.size()` (size_t) with `uint64_t` literals. |
| 6 | `storage/index/snii/writer/spimi_locality_bench_test.cpp` | libc++
`no matching function for call to object of type 'key_equal'` | The
transparent equality functor defines one argument order only. libc++
calls `key_equal` as `(key, lookup key)`, libstdc++ as `(lookup key,
key)`. |
| 7 | `storage/segment/hierarchical_data_iterator_test.cpp` | three `no
matching function for call to 'min'` | `kRows` is `size_t` and
`current_ordinal` is `ordinal_t` (`uint64_t`), which are distinct types
on Darwin, so `std::min` cannot deduce a common type. |
None of the seven is related to a local change; a clean master
reproduces all of them. #2 arrived on 2026-09-24 with #68397 (`[fix](be)
Preserve decimal precision in format_round`, by TengJianPing), which is
the commit that added `function_format_round_test.cpp` — after the last
full macOS sweep. So this class of breakage is still being produced:
"green on Linux, red on macOS" is invisible to every CI job we have, and
the two obstacles above make the one manual path that would find it
expensive enough that nobody walks it.
**2. What this PR does, and why it helps**
- `run-be-ut.sh`: the `update_submodule` list gains `contrib/openblas`
and `contrib/faiss`, with the same archive URLs `build.sh:941` already
uses, plus a comment that the list has to stay in sync with `build.sh`.
- `be/test/CMakeLists.txt`: the `APPLE` branch links `ann_index` in
addition to the `force_load` flag. `ann_index` is the target that
exports the faiss include directories `PUBLIC`, so the APPLE link line
now feeds the same compile and link inputs as every other platform,
instead of being a macOS special case.
- Six test files: the smallest change that removes the platform
difference, each spelled the way the file's neighbours already spell it
— `PRIu64` / `PRId64` (`core/decimal12.h`, `util/histogram.cpp`),
`int64_t(...)` around the literals (`storage/key_coder_test.cpp:392`,
`runtime/jsonb_document_test.cpp:433`), `#if
defined(POSIX_FADV_DONTNEED)` around the call.
Linux behaviour is unchanged: every edit is either inside an
`APPLE`-only path or a strictly more portable spelling of the same
value. On macOS `run-be-ut.sh` becomes a usable test line for the first
time.
One cost worth naming: because the APPLE link line now propagates
include directories, the compile command of every `be/test` TU changes
once, so the first macOS build after this lands rebuilds `be/test`
(about 1230 TUs). Later builds are incremental again.
**Branch pick.** #68397 was never picked to `branch-4.1`, so
`function_format_round_test.cpp` does not exist there. Of the seven
files, only `storage/segment/hierarchical_data_iterator_test.cpp` exists
on `branch-4.1`, with the same three `std::min(*rows, ROWS -
current_ordinal)` sites and the same types (`ROWS` is `size_t`,
`ordinal_t` is `uint64_t`), so it fails to compile on macOS there too;
that branch also has no `APPLE` `force_load` branch in
`be/test/CMakeLists.txt`, so the faiss part is master-only. If 4.1 is
meant to run BE UT on macOS, the pick is that one file's hunk.
**3. The classes, and how they call each other**
- `run-be-ut.sh` (changed): `update_submodule()` calls `git submodule
update --init --recursive <path>` and falls back to the branch tarball;
the two added lines are identical in shape to the three that were
already there.
- `be/test/CMakeLists.txt` (changed): the `APPLE` branch of the
`doris_be_test` link step.
- `be/src/storage/index/ann/cmake-protect/CMakeLists.txt` (unchanged,
the reason the submodules are mandatory): `add_subdirectory()` on
`contrib/openblas` and `contrib/faiss`, reached unconditionally from
`be/CMakeLists.txt:1038`.
- `ann_index` (unchanged target in `be/src/storage/index/ann`): links
`faiss` `PUBLIC`, so its `INTERFACE_INCLUDE_DIRECTORIES` carry the faiss
include path.
- `vector_search_test` (unchanged target in
`be/test/storage/index/ann`): links `ann_index` `PUBLIC`; on APPLE it
reaches `doris_be_test` as forced-in objects rather than as a target.
- The seven test TUs: unchanged in behaviour, three of them compiled in
a portable spelling.
```
run-be-ut.sh
|- update_submodule contrib/{datasketches-cpp, apache-orc, clucene} [already there]
'- update_submodule contrib/{openblas, faiss} [added here]
'- git submodule update --init --recursive
be/CMakeLists.txt:1038 add_subdirectory(be/src/storage/index/ann)
'- storage/index/ann/CMakeLists.txt:18 add_subdirectory(cmake-protect)
'- cmake-protect/CMakeLists.txt:48,66 add_subdirectory(contrib/{openblas, faiss})
'- target faiss --(PUBLIC)--> target ann_index --(PUBLIC)--> vector_search_test
|
be/test/CMakeLists.txt:170 if (APPLE) |
|- before: -Wl,-force_load,<path to libvector_search_test.a> -- propagates nothing
'- after: the same flag + ann_index -- propagates the faiss include dirs
|
function_map_inner_product_test.cpp <--'
(and every other be/test TU, which now
compiles with the same inputs as on Linux)
be/test/CMakeLists.txt add_executable(doris_be_test ${UT_FILES}) <- the other six TUs, no link involvement
```
### What problem does this PR solve?
Issue Number: close#66715
Related PR: #66510 (the BE build-time series this closes out)
Problem Summary:
BE compiles every TU with `-g -gdwarf-5`, and on this codebase that
debug info is not a rounding error — it is **95% of the bytes the
compiler writes**. Across the 7426 objects of a full Release build:
`__text` 150 MB, `.debug_*` **4.01 GB**. Type-level DWARF describes
every type a TU touched, so it scales with the header closure rather
than with the code.
Line tables alone keep stack traces, `perf` and `addr2line` resolving to
`file:line`; what they drop is variable-level DWARF (`print some_var` in
gdb/lldb).
This PR adds the switch and turns it on where nobody debugs.
**1. The switch** (`DORIS_DEV_DEBUG_INFO`, values `line-tables` /
`full`):
```bash
DORIS_DEV_DEBUG_INFO=line-tables ./build.sh --be
```
`-gline-tables-only` is appended **after** the existing `-g -gdwarf-5`
(clang lets the last debug-info flag win). Appending rather than
replacing is the point: with the switch unset the `if` contributes
nothing, so `build.sh` keeps producing byte-identical command lines. An
unrecognized value is a configure FATAL; `line-tables` requires clang
(gcc has no such flag).
Measured — macOS arm64, clang 20.1.8, `./build.sh --compile-bench -j 6`,
PCH on, ccache off, build dir recreated every run, same commit, three
back-to-back runs (default → line-tables → default, so drift is measured
rather than assumed):
| | default (run 1) | `line-tables` | default (run 3) | effect vs. mean
of defaults |
|---|---|---|---|---|
| cold build wall | 12m02s | **10m34s** | 11m52s | **-11.6%** |
| build phase wall | 11m28s | 10m03s | 11m22s | -11.9% |
| sum of TU wall | 3821s | 3341s | 3781s | -12.1% |
| sum of TU cpu | 3774s | 3281s | 3733s | -12.6% |
Machine drift between the two default runs: **-1.3%**, an order of
magnitude below the effect. Artifacts: objects 4.25 GB → 0.99 GB
(**-76.8%**), build dir 9.4 → 2.8 GB, type-level `.debug_*` 4.01 → 0.51
GB, heaviest TU peak RSS 3798 → 2618 MB (-31%), `doris_be` `__text`
**byte-identical**. Per file: 207 TUs improved, 16 regressed (all ≤
+2.6s, one target, scheduling noise).
**2. `run-be-ut.sh` defaults to `line-tables`.** The UT build has its
own cmake invocation, and it is the build a developer repeats most —
`MAKE_TEST=ON` adds all of `be/test` on top of `be/src`. A UT build is a
compile-and-run loop; when a test fails, the stack trace still resolves.
Sampled on 10 real `be/test` TUs (Debug, PCH on, same command with and
without the flag):
| | full | `line-tables` | delta |
|---|---|---|---|
| compile cpu | 55.0s | 51.5s | **-6.3%** |
| objects | 91.4 MB | 8.1 MB | **-91.1%** |
Smaller time win than the Release path (at `-O0`/`-Og` the DWARF is
bulky but cheap to emit), much larger size win.
`DORIS_DEV_DEBUG_INFO=full sh run-be-ut.sh` restores the full DWARF for
a debugger session. The default only applies under clang, so a
`DORIS_TOOLCHAIN=gcc` UT build is untouched rather than hitting the
clang-only FATAL.
**3. `be-ut-mac.yml` sets it explicitly.** That job builds with
`-DMAKE_TEST=OFF` — it only checks that the BE compiles on macOS, runs
no test and keeps no binary — so its full DWARF is pure cost. It is the
same `build.sh --be` path measured above (-11.6%), and its 5G ccache no
longer has to hold 4.25 GB of objects per build.
**Disclosure / blind spots:**
- All numbers are macOS/clang20; Linux is covered only by this PR's CI.
The gcc path was not exercised locally (no gcc toolchain on the bench
machine); it is guarded statically by `NOT COMPILER_CLANG` and by the
clang-only default in `run-be-ut.sh`.
- **No link-time win on macOS**: ld64 leaves DWARF in the objects and
builds a debug map, so `doris_be` is 336 MB either way (`__text`
byte-identical, 176 B of file difference) and the link is 0.9s → 0.8s.
On Linux ELF the linker *does* copy `.debug_*` into the binary, so the
binary should shrink there — not measured, so no number is claimed.
- The switch reaches BE plus the contrib targets that inherit the
directory options; `clucene` (347 TUs) and `apache-orc` (48 TUs) set
their own flags and keep full debug info.
- Changing the UT default changes the compile command line, so the first
UT build after this lands is a full rebuild and does not reuse existing
ccache entries. One-time cost.
- Interactive UT debugging loses variable inspection by default. That is
the trade this PR makes deliberately for the compile-and-run loop;
`DORIS_DEV_DEBUG_INFO=full` is the way back, and `build.sh` still
defaults to full DWARF.
- Orthogonal to `STRIP_DEBUG_INFO=ON`, which splits debug info off
**after** linking (saves binary/deploy size, saves no compile time — the
DWARF was already generated). This switch avoids generating it at all.
### Release note
None
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
- Three paired cold benchmark builds (`build.sh --compile-bench -j 6`),
same commit, same machine, back to back, the third quantifying drift.
- Sampled A/B over 10 real `be/test` TUs from the UT compile database,
same command with and without the flag, CPU time and object size.
- Verified the flag reaches the compiler on the `--be` path: 7545 of
7940 `compile_commands.json` entries carry `-gline-tables-only` (the
rest are the two flag-overriding contrib targets).
- Verified it on the UT path (`BUILD_TYPE_UT=Debug run-be-ut.sh`): 8741
of 9136 entries, same 395 contrib entries excepted; and that the new
default produces the same result with no environment variable set, while
`DORIS_DEV_DEBUG_INFO=full` produces zero flagged entries.
- Verified emitted code is unaffected: `doris_be` `__text`
byte-identical between the default and `line-tables` builds.
- Verified the unknown-value path: `DORIS_DEV_DEBUG_INFO=bogus` stops
configure with `Unknown DORIS_DEV_DEBUG_INFO value: bogus (supported:
line-tables, full)`.
- Behavior changed:
- [x] Yes. BE **unit-test** builds (`run-be-ut.sh`) and the macOS
BE-compile CI job now default to line-tables-only debug info: stack
traces, `perf` and `addr2line` are unaffected; variable-level DWARF is
not emitted unless `DORIS_DEV_DEBUG_INFO=full` is set. `build.sh` itself
still defaults to full debug info, and release builds are unchanged.
- Does this need documentation?
- [x] No. `build.sh -h` documents the switch and both values;
`run-be-ut.sh` prints the level it is building with.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #66842
Problem Summary:
**Second half of a two-PR split. Do not merge before #66842 has landed
*and* the
prebuilt third-party archives have been rebuilt from it** - see "Merge
order" at the
bottom. #66842 carries the third-party build changes; this PR carries
the consumers.
## 1. Move the remaining `hadoop_hdfs/` consumers onto
`hadoop_hdfs_3_4/`
The tree carried two libhdfs builds - 3.3.6.6 under
`installed/{include,lib}/hadoop_hdfs/`
and 3.4.2.4 under `.../hadoop_hdfs_3_4/` - and the two prefixes were
mixed up on the
consumer side:
| consumer | header | library |
|---|---|---|
| `be/src/io/fs/hdfs.h` | `hadoop_hdfs_3_4/hdfs.h` (3.4.2.4) | — |
| `be/src/io/hdfs_builder.cpp` | `hadoop_hdfs/hdfs.h` (**3.3.6.6**) | —
|
| `be/CMakeLists.txt` | — | `hadoop_hdfs_3_4/native/libhdfs.a` (3.4.2.4)
|
| `cloud/src/recycler/hdfs_accessor.{h,cpp}` | `hadoop_hdfs/hdfs.h`
(**3.3.6.6**) | — |
| `cloud/CMakeLists.txt` | — | `hadoop_hdfs/native/libhdfs.a`
(**3.3.6.6**) |
Both headers share the `LIBHDFS_HDFS_H` include guard, so
`hdfs_builder.cpp` compiled
against the 3.3.6.6 header (it came first) while BE linked the 3.4.2.4
archive.
So `cloud/`, `build.sh`'s `LAST_THIRDPARTY_LIB` sentinel,
`run-be-ut.sh`,
`run-cloud-ut.sh` and the thirdparty lifecycle test move to
`hadoop_hdfs_3_4/`, and
`be/src/io/hdfs_builder.cpp` drops its 3.3.6.6 include - `io/fs/hdfs.h`,
two lines
below it, already re-exports the 3.4.2.4 one. The unit-test runners
still stage the
jars at `lib/hadoop_hdfs/`, so the classpath loops below them are
unchanged.
Both prefixes exist in every third-party archive built before #66842, so
this half is
safe against the images in use today. It is #66842 - which stops
building 3.3.6.6 at
all - that makes it mandatory.
## 2. Link azure on aarch64 and macOS
`env.sh` forced `DISABLE_BUILD_AZURE=ON` on aarch64 and macOS, so BE and
the cloud
meta-service lost `+AZURE_BLOB` and `+AZURE_STORAGE_VAULT` there. #66842
made the
third-party recipe portable (`-ldl` does not exist on Apple; vcpkg ships
no prebuilt
tools for aarch64 Linux) and now builds azure on every platform. This
drops the
platform test so the consumers link it. `DISABLE_BUILD_AZURE=ON` still
opts out.
## Merge order
1. #66842 lands.
2. The apache/doris-thirdparty automation rebuilds the prebuilt archives
- it triggers
on `^thirdparty/` and takes ~3h - and the aarch64 build-env image is
rebuilt.
3. This PR lands.
Landing this one first fails at the BE link with
`ninja: error: '.../installed/lib/libazure-core.a' ... missing and no
known rule to
make it`, which is exactly what the BE UT (macOS) job of the un-split PR
hit.
### Release note
Azure Blob Storage support (`+AZURE_BLOB`, `+AZURE_STORAGE_VAULT`) is
now built on
aarch64 and macOS, not just x86_64 Linux.
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
`thirdparty/test/arrow-paimon-lifecycle-test.sh` covers the `build.sh`
sentinel
move; it is what the Arrow/Paimon Lifecycle Test job runs. The azure
half is
exercised by every BE/cloud build in this PR's own CI, against a
prebuilt that
by then carries the libraries.
- Behavior changed:
- [x] Yes.
- Azure is now built and linked on aarch64 and macOS. **The third-party
prebuilt
has to be rebuilt from #66842 before this lands**: BE/cloud on those
platforms
will pass `-DBUILD_AZURE=ON` and fail to link against an older archive
that has
no azure libraries. `DISABLE_BUILD_AZURE=ON` still opts out.
- Anything outside this repository reading the `hadoop_hdfs/` prefix
needs
updating; only `hadoop_hdfs_3_4/` remains after #66842.
- Does this need documentation?
- [x] No.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: daidai <changyuwei@selectdb.com>
> Part of the BE build-time optimization series tracked in #66715.
>
> Split out of **https://github.com/apache/doris/pull/66510**, which
carries the whole
> BE build-time batch. After the header-closure surgery in **#66400**
and **#66672**,
> this PR opens the second line of that batch: CMake unity builds. It
adds the
> infrastructure switch and converts the three lowest-risk glue segments
as a pilot;
> the heavier targets (Exec, Exprs, and the rest) follow in separate PRs
once this
> one has proven the mechanism cross-platform.
### What problem does this PR solve?
Related PR: #66510, #66672
Problem Summary:
Most of the BE's cold-build time is not spent compiling our code — it is
spent
**re-parsing the same shared header closure once per small `.cpp`**. For
the glue
directories this ratio is extreme: the 51 InformationSchema scanners
cost ~124 CPU s
(with PCH) of which almost everything is the closure re-parse; the ~58
http handlers
sum to ~5.6 min of slot time for a few thousand lines of handler logic.
CMake's built-in `UNITY_BUILD` (CMake ≥ 3.16; we require 3.19.2)
concatenates groups
of `.cpp` files into jumbo TUs, so a closure is parsed **once per
batch** instead of
once per file. This PR:
1. **Adds the switch** — `option(ENABLE_UNITY_BUILD ON)` in
`be/CMakeLists.txt`,
plumbed through `build.sh` and `run-be-ut.sh` exactly like `ENABLE_PCH`,
overridable from the environment / `custom_env.sh`. Every per-target
`UNITY_BUILD` property is set unconditionally from it -- including an
explicit
OFF, so the switch also wins over CMake's own `CMAKE_UNITY_BUILD` in a
reused
cache. Turn it OFF for per-file diagnostics, per-file tooling
(clang-tidy,
coverage), or the finest-grained incremental rebuilds.
2. **Pilots unity on three glue segments** (and only there — everything
else is
explicitly opted out per file, not by omission):
- **InformationSchema**: all 51 scanner TUs → 1 unity TU
(`UNITY_BUILD_BATCH_SIZE 0`).
- **Service, scoped to `http/`**: ~58 handler TUs → 1 unity TU. The
non-http
service sources (service entry points, arrow_flight) are heterogeneous
heavy TUs
that gain nothing from merging and stay individual.
- **Storage, scoped to `index/`**: ~111 TUs sharing one CLucene-heavy
closure →
4 unity TUs of ≤ 32 sources (batch size bounds jumbo-TU size and
memory).
3. **Makes the merged TUs legal C++** with two hygiene commits that
stand on their
own even without unity:
- 27 http action files each carried a private copy of the same
file-scope
constants (`HEADER_JSON` ×16, `TABLET_ID` ×7, `SCHEMA_HASH` ×4, …). They
move to
one shared header as C++17 inline variables, values unchanged.
- `prefix_query` declared `get_prefix_terms(IndexReader*)` relying on a
header-level `CL_NS_USE(index)` using-directive. Inside
`doris::segment_v2` that
unqualified name silently flips to `doris::segment_v2::IndexReader` as
soon as
any sibling source brings that type into scope — a landmine with or
without
unity. The declaration now spells out `lucene::index::IndexReader`.
4. **Fixes a latent bug unity exposed**: `schema_scanner_helper.h`
opened with
`#ifndef _SCHEMA_SCANNER_HELPER_H_` but never defined the macro (and one
include
sat outside the guard), so the guard never worked. Harmless while every
TU
included it exactly once; breaks immediately under unity. Now `#pragma
once`.
5. **Keeps the compile-bench tooling honest under unity** — with the
default ON,
`build.sh --compile-bench` produces a build directory whose sources are
jumbo TUs,
which two of the already-merged tools silently mis-read: `cut_impact.py`
dropped
every unity dependency block (and with it all 211 pilot sources) from
its TU set,
and `report.py` filed each batch's time under the build directory
instead of the
module it came from. `cut_impact.py` now refuses a unity build directory
with the
rebuild instruction — its analysis needs per-source closures, and
splitting a
batch's union back over its members would over-estimate instead of
under-report —
`report.py` attributes a unity TU to its target's directory, and
`rebuild_radius.py` says when its per-header counts are batch-granular.
`build-support/tests/test-compile-bench-unity.sh` pins all three
behaviours.
Files whose file-scope macros must not leak into unity siblings stay
individual via
`SKIP_UNITY_BUILD_INCLUSION`: `http_parser.cpp` (CR/LF),
`be_thread_stack_action.cpp`
(`UNW_LOCAL_ONLY`), and four index files (`CL_MAX_PATH` and friends,
`IS_CHINESE_CHAR`, `APPLY_FOR_PRIMITITYPE`). A future file whose
file-scope symbols
clash inside a unity TU can opt out the same way.
### Measured results
All numbers from the development branch this series is split from (which
also
carried the #66672 include cuts), macOS arm64 + clang 20, `-j14`,
`ENABLE_PCH=ON`, cold builds, back-to-back A/B:
| metric | before | after |
|---|---|---|
| build phase wall | 11m38s | **10m16s (-82.4 s / -11.8%)** |
| Σ per-TU CPU | 147.9 min | **134.8 min (-8.9%)** |
| TU count | 8384 | 8180 |
| failures | 0 | 0 |
Per-segment slot time (sum of compile time the segment occupies across
slots):
| segment | before | after | ratio |
|---|---|---|---|
| information_schema | 221.7 s | 17.0 s | **13×** |
| service/http | 273.6 s | 26.3 s | **10.4×** |
| storage/index | 264.2 s | 120.1 s | **2.2×** (remainder includes the
four macro SKIPs) |
Why unity rather than more PCH: a serial cold micro-benchmark of the
InformationSchema target alone measured **9.9× with PCH** (124 s → 12.5
s) and
**14.6× without PCH** (206 s → 14.1 s) — and unity *without* PCH still
beats
individual TUs *with* PCH by 8.8×. Unity removes the repeated parse
instead of
amortizing it, and the two compose.
Side effects on artifacts: `libInformationSchema.a` 334 MB → 29 MB,
`libStorage.a`
1492 MB → 1318 MB (linkonce_odr instantiations dedup inside each unity
TU). The
largest unity TU peaks at 2.1 GB RSS — *below* the largest existing
individual TU in
the tree (3.9 GB), so `-jN` memory envelopes are unchanged.
### Risk and verification
- **Unity changes TU grouping only; no code changes ride along** beyond
the two
hygiene commits described above (constant dedup with identical values,
one
qualified name, one include guard).
- **Archive symbol parity** was checked per target on the development
branch:
InformationSchema keeps all 2303 external defined symbols (plus 5 weak
`unique_ptr<SchemaXxxScanner>` instantiations that dedup), Service keeps
all 4054,
Storage keeps all external symbols with 8 weak linkonce_odr template
instantiations deduping away — which is the point of unity, not a loss.
- **This exact branch, rebased onto current master, full BE build from
scratch**
(macOS arm64, clang 20, `ENABLE_PCH=ON`, `ENABLE_UNITY_BUILD=ON`):
**8349/8349 ninja edges, zero failures, `doris_be` links.** The six
expected unity
TUs (InformationSchema ×1, Service http ×1, storage/index ×4) all
compile.
This includes `schema_tso_status_scanner.cpp`, added upstream after the
pilot was
measured — it lands inside the InformationSchema unity TU via the
existing
`GLOB_RECURSE` with zero CMakeLists edits, which is the intended
maintenance story.
- **The OFF path is verified on the same tree**: reconfiguring with
`ENABLE_UNITY_BUILD=OFF` removes every `unity_*.cxx` entry from
`compile_commands.json` and flips **exactly 219 ninja edges** — the
three targets'
per-file objects plus their archives and the final link, nothing else.
All of them
compile per-file with zero failures and `doris_be` links again. The
blast radius
of the switch is precisely the three pilot targets.
- **`OFF` also wins over a native-unity cache**: configuring with
`-DENABLE_UNITY_BUILD=OFF -DCMAKE_UNITY_BUILD=ON` produced 28 unity TUs
across the
three pilot targets before the gate fix and **0** after, while the
ordinary
`-DENABLE_UNITY_BUILD=ON` configure still produces exactly the 6
advertised
batches (InformationSchema ×1, Service ×1, Storage ×4).
- **These three segments have been building as unity TUs on the
development branch
since 2026-08-08**, through repeated full-tree builds and the BE UT
builds that
verified #66672 (the UT binaries link against these same target
libraries).
### Proactive disclosure
- **Cross-platform is the blind spot, and the default is deliberately ON
so this
PR's own CI closes it.** Every local build and measurement above is
macOS arm64 +
clang 20. Nothing here is platform-specific by construction, but with
the default
ON, the Linux compile lanes and every regression pipeline in this PR's
CI run
against unity builds — that is the validation. Please give the Linux gcc
lane in
particular a look. If some environment trips over unity after merge, the
escape hatches are, in order: per-user `ENABLE_UNITY_BUILD=OFF` (env or
`custom_env.sh`), per-file `SKIP_UNITY_BUILD_INCLUSION`, or a one-line
default
flip — the infrastructure stays either way.
- **The incremental-rebuild trade-off is real**: touching one `.cpp`
inside a unity
batch recompiles the whole batch. For the glue chosen here a batch
compiles in
~16–30 s, comparable to single mid-weight TUs elsewhere in the tree; and
`ENABLE_UNITY_BUILD=OFF` restores per-file granularity for workflows
that need it.
This is also why the pilot targets are glue directories and not the
hot-edit paths.
- **contrib (openblas/clucene) was evaluated and deliberately left
alone**: the
f2c-generated LAPACK sources and the snowball stemmers define clashing
file-scope
statics (`static c__1` and friends) — structurally un-unifiable without
rewriting
generated code — and contrib compiles once and rarely changes.
- **The rest of Storage and Service is opted out per file, on purpose.**
Merging
heavy heterogeneous TUs earns nothing (the closure parse is not the
dominant cost
there) and risks monster TUs. Follow-up PRs extend unity to Exec, Exprs
and the
remaining targets with the same SKIP discipline; on the development
branch the
full rollout takes the same tree from 10m16s to **6m18s**.
Related PR: #63143
Problem Summary:
The BE build previously configured and installed datasketches-cpp from
`contrib/datasketches-cpp` into `thirdparty/installed` before
configuring the BE itself.
Its CMake cache was stored under
`contrib/datasketches-cpp/build/Release`, which was **NOT** managed by
`clean_be()`. As a result, `build.sh --be --clean` could still fail
before reaching `clean_be()` when the cached compiler path belonged to
another workspace or no longer existed.
```text
install datasketches-cpp to thirdparty path before build be
Update datasketches-cpp submodule ...
/mnt/disk9/linzhenqi/d1/doris
Current commit ID of datasketches-cpp submodule: de8553ba372e618382c2e7b44b0ffc9422b9458c, expected is de8553ba372e618382c2e7b44b0ffc9422b9458c
-- The CXX compiler identification is unknown
CMake Error at CMakeLists.txt:25 (project):
The CMAKE_CXX_COMPILER:
/mnt/disk7/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/clang++
is not a full path to an existing compiler tool.
Tell CMake where to find the compiler by setting either the environment
variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
to the compiler, or to the compiler name if it is in the PATH.
-- Configuring incomplete, errors occurred!
```
The standalone installation also mixed a contrib dependency into the
thirdparty installation directory without using the thirdparty build
lifecycle.
### Release note
- Keep datasketches-cpp as a contrib submodule.
- Add datasketches-cpp to the BE CMake build tree with
`add_subdirectory`.
- Link `Exprs` against the upstream `DataSketches::HLL` interface
target.
- Mark the datasketches include directories as system headers so
upstream warnings are not promoted to errors by Doris's `-Werror`
settings.
- Use the upstream build-tree include path for `hll.hpp`.
- Remove the standalone CMake configure/install commands from
`build.sh`.
- Remove the now-unused `TP_INSTALLED_DIR` variable.
The datasketches CMake state is now stored under the BE build directory
and is removed together with the rest of the BE build artifacts by
`clean_be()`.
### What problem does this PR solve?
Issue Number: close#62497
follow up https://github.com/apache/doris/pull/64454
#### Problem Summary:
This PR adds a safer BE print stack API based on libunwind
signal-context unwinding.
Stack unwinding may call `dl_iterate_phdr()` to find DWARF/FDE metadata.
In signal-based stack capture, jemalloc profiling, or concurrent
`dlopen()` / `dlclose()`, re-entering glibc's loader-lock path may
deadlock or become unsafe.
#### Solution
Route GNU libunwind's PHDR lookup through Doris' lock-free PHDR cache.
The signal handler unwinds from the kernel-provided signal context,
while libunwind reads loaded-object metadata from Doris' PHDR snapshot
instead of calling glibc `dl_iterate_phdr()` directly.
Normal code paths still use the original live `dl_iterate_phdr()`
behavior.
The research project is at
https://github.com/JoverZhang/Doris-BE-Print-Stack-API-Research
Co-authored-by: zhaochangle <zhaochangle@selectdb.com>
Related PR: #63145
Problem Summary: This re-submits the OSS mTLS framework work from #63145
under my account and rebases it onto the latest apache/doris master. The
change ports the public mTLS scaffolding, configuration, protocol
startup split, certificate-auth contracts, and TLS validation tests
while excluding enterprise module directories.
After the rebase, the previous FE UT failures were fixed: the
ALTER/CREATE USER TLS unit tests now use Mockito static mocking instead
of external JMockit parameter injection, and the MetaServiceProxy
success-path test now stubs the mock client as using the latest channel
configuration so the proxy does not replace it before executing the
request.
---------
Co-authored-by: Siyang Tang <tangsiyang@selectdb.com>
### What problem does this PR solve?
> An aggregate function is required to process user data containing
Datasketches HLL sketches. In many data aggregation scenarios, users
pre‑aggregate detailed data in Hive using the sketching techniques
provided by Apache Datasketches, and then analyze the resulting sketches
across various OLAP engines. Compared with the HLL union aggregate
functions natively offered by these engines, there are two key diff to
using Datasketches HLL sketches: firstly, the use cases differ; and
secondly, HLL sketches can be used seamlessly across different
engines—for example, simultaneously in ES, Doris, and ClickHouse. Such
requirements are common in many production environments.
Issue Number:
- #63142(https://github.com/apache/doris/issues/63142)
- #26416
- #56246
Summary:
Implemented a built-in aggregate function that integrates the
Datasketches HLL sketch. This aggregate function cannot rely on the Java
UDF environment. Considering that in the Java UDF environment, Strings
are encoded in UTF-8, which corrupts the binary data of sketches, the
serialization/deserialization operations for sketches must be
implemented on the BE side. (additionally, since Apache Datasketches has
been added to the contrib directory via a git submodule, it will become
very easy to add other sketches such as theta sketch in the future.)
**see**: https://github.com/apache/doris/issues/63142
**use case**: see regression test &
https://github.com/apache/doris/issues/63142
---------
Co-authored-by: yuanyuhao <yuanyuhao@bytedance.com>
### What problem does this PR solve?
If you want to run unit tests using a real cloud provider, you can
configure it in `custom_env.sh`.
```
# Enable S3 client for unit tests
export ENABLE_S3_CLIENT=1
# S3 client config
export S3_AK="xxx"
export S3_SK="xxx"
export S3_ENDPOINT="xxx"
export S3_PROVIDER="xxx"
export S3_BUCKET="xxx"
export S3_REGION="xxx"
export S3_PREFIX=""
```
### What problem does this PR solve?
Use memchr to speed up BaseTextLineReaderContext.find_lf_crlf_line_sep
function.
---------
Co-authored-by: Zhiguo Wu <wuzhiguo@linaro.org>
### What problem does this PR solve?
add addr2line msg and remove variable use_dwarf
Clang generally defaults to using DWARFv5 for debugging information.
This has been the default for LLVM's Clang front-end since LLVM 14.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
1. Remove compile option `USE_MEM_TRACKER`. Linux compiles with
`USE_MEM_TRACKER=ON`, Mac and UT compiles with `USE_MEM_TRACKER=OFF`.
Now we don't use jemalloc hook to track memory. this option has not been
used for a long time and causes code confusion, so remove it.
2. `SCOPED_ATTACH_TASK` and other macros that operate `ThreadContext`
are effective in BE UT.
3. `thread_context()` removes the parameter `allow_null`.Introduce a new
macro `SCOPED_INIT_THREAD_CONTEXT` to initialize `ThreadContext` for
threads that do not execute Tasks.
4. Deprecated `SCOPED_SWITCH_RESOURCE_CONTEXT` macro, manually pass
`ResourceContext`, but `SCOPED_SWITCH_RESOURCE_CONTEXT` has not been
removed and may be used in the future.
### What problem does this PR solve?
remove libmysql libodbc link but do not remove those from thirdparty
(coz 2.1/3.0 may used)
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [x] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
DataTypeNumber, DataTypeDecimal, DataTypeString, DataTypeDateTimeV2
Implement a microbenchmark to quickly and efficiently identify issues
with the Doris file cache, including performance problems. Direct
end-to-end system tests that stress Doris to evaluate the efficiency of
the file cache component are relatively ineffective, as they do not
quickly yield the desired eviction rates and pressures, and may also be
affected by factors such as compaction. Therefore, we need to implement
direct testing of the file cache functionality and performance through
the IO layer (S3FileWriter + CachedRemoteReader) during upload and
download operations.
Previously the azure sdk would be compiled into BE by default. But
sometimes the BE would run in environment which only require AWS
s3-compatible functionality. So user can use `DISABLE_BUILD_AZURE` to
control whether build azure sdk into BE or not.
`export DISABLE_BUILD_AZURE=ON` to disable build with azure support
## Proposed changes
Revert #13660
`-l` and `dlopen` load Dynamic Link Library, will use the Allocator of
main program, Doris use Jemalloc by default.
but, `dlopen` load libjvm.so and use Jemalloc compiled with prefix,
overwriting malloc/free will incompatible with libjvm.so. see #34578 for
details.
In addition, jemalloc not recommend `dlopen` to load Dynamic Link
Library, `dlclose` will memory leak in that case.
https://github.com/jemalloc/jemalloc/issues/2404https://github.com/jemalloc/jemalloc/issues/1321https://github.com/jemalloc/jemalloc/issues/1890
`export LD_PRELOAD` to force `libjvm.so` use a separate `jemalloc.so`
will not solve the problem, but may cause a new crash.
```
received by PID 2368 (TID 2391 OR 0x7f445cafb700) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/selectdb-core/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /opt/jdk/lib/server/libjvm.so
2# JVM_handle_linux_signal in /opt/jdk/lib/server/libjvm.so
3# 0x00007F448DB40400 in /lib64/libc.so.6
4# je_arena_dalloc_promoted at /root/selectdb-core/thirdparty/src/jemalloc-5.3.0/doris_build/../src/arena.c:1277
5# je_free_default at /root/selectdb-core/thirdparty/src/jemalloc-5.3.0/doris_build/../src/jemalloc.c:3014
6# __pthread_create_2_1 in /lib64/libpthread.so.0
7# os::create_thread(Thread*, os::ThreadType, unsigned long) in /opt/jdk/lib/server/libjvm.so
8# CompilerThread::CompilerThread(CompileQueue*, CompilerCounters*) in /opt/jdk/lib/server/libjvm.so
9# CompileBroker::make_thread(CompileBroker::ThreadType, _jobject*, CompileQueue*, AbstractCompiler*, JavaThread*) [clone .constprop.0] in /opt/jdk/lib/server/libjvm.so
10# CompileBroker::possibly_add_compiler_threads(JavaThread*) in /opt/jdk/lib/server/libjvm.so
11# CompileBroker::compiler_thread_loop() in /opt/jdk/lib/server/libjvm.so
12# JavaThread::thread_main_inner() in /opt/jdk/lib/server/libjvm.so
13# Thread::call_run() in /opt/jdk/lib/server/libjvm.so
14# thread_native_entry(Thread*) in /opt/jdk/lib/server/libjvm.so
15# start_thread in /lib64/libpthread.so.0
16# clone in /lib64/libc.so.6
```
<!--Describe your changes.-->
If DEBUG build type, record size of each memory alloc and free, print no free size or no alloc size when query MemTracker is destructed, if necessary, record stack trace.
add global PointQueryExecutor memory tracker in ExecEnv, because memory may be shared between PointQueryExecutors of different pointer queries, but memory will not be shared between PointQueryExecutor and Fragment of the same pointer query.
If DEBUG build type, if query memory tracker not equal to 0 when query ends, BE will crash.
Like #15641, we should reduce the size of executables on macOS arm64. Otherwise, we can not run doris_be and doris_be_test with ASAN build type on macOS arm64 now.
Introduce libunwind get stack trace, cost is negligible and has line numbers.
use StackTraceCache, PHDRCache speed up, is customizable and has some optimizations.
Other stack trace tools remain: glog, boost, glibc, in case for need.
TODO:
currently support linux __x86_64__, __arm__, __powerpc__, not supported __FreeBSD__, APPLE
Note: __arm__, __powerpc__ not been verified
Support signal handle
libunwid support unw_backtrace for jemalloc
Use of undefined compile option USE_MUSL for later
1. Add hdfs file handle cache for hdfs file reader
Copied from Impala, `https://github.com/apache/impala/blob/master/be/src/util/lru-multi-cache.h`. (Thanks for the Impala team)
This is a lru cache that can store multi entries with same key.
The key is build with {file name + modification time}
The value is the hdfsFile pointer that point to a certain hdfs file.
This cache is to avoid reopen same hdfs file mutli time, which can save
query time.
Add a BE config `max_hdfs_file_handle_cache_num` to limit the max number
of file handle cache, default is 20000.
2. Add file meta cache
The file meta cache is a lru cache. the key is {file name + modification time},
the value is the parsed file meta info of the certain file, which can save
the time of re-parsing file meta everytime.
Currently, it is only used for caching parquet file footer.
The test show that is cache is hit, the `FileOpenTime` and `ParseFooterTime` is reduce to almost 0
in query profile, which can save time when there are lots of files to read.
daidai
Mryange
Jack
Mingyu Chen (Rayner)
linrrarity
Jover Zoengandzhaochangle
JimmyandSiyang Tang
nooneuseandyuanyuhao
Yixuan Wang
Gavin Chou
Luwei
Zhiguo WuandZhiguo Wu
Pxl
Qi Chen
zhiqiang
Xinyi Zou
TengJianPing
deardeng
AlexYue
Mryange
Adonis Ling
Xinyi Zou
zhiqiang
Qi Chen
plat1ko
shuke
chunping
zxealous
Tiewei Fang
Mingyu Chen
DeadlineFen
Pxl