### What problem does this PR solve?
Issue Number: N/A
Problem Summary: Add `h3_lat_lng_to_cell`, `h3_cell_to_lat_lng`, and
`h3_cell_to_boundary` using official H3 4.5.0. Coordinates use degrees
in latitude/longitude order. Invalid inputs return NULL; centers contain
two coordinates and boundaries preserve H3 vertex order without closing
the ring. Input NULL propagation uses the default execution framework.
FE has no separate Java H3 evaluator; constant folding through BE
remains supported.
Build H3 as a static C library from its 18 upstream core sources.
Generate its versioned API header in the build directory, export include
paths through the target, enable PIC, and inherit Doris sanitizer flags.
Only the H3 target relaxes third-party warning errors. The wrapper
avoids upstream standalone tools and global build options. Include the
upstream license attribution and NOTICE.
### Release note
Add three H3 SQL functions for mapping coordinates to cells and
retrieving cell centers and boundaries.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. Add three SQL functions.
- Does this need documentation?
- [ ] No.
- [x] Yes. SQL function documentation is needed; no documentation PR
yet.
Validation on Linux x86_64:
- ASAN BE build and clean FE build passed; H3 objects carry ASAN/UBSAN
flags.
- `run-be-ut.sh --run --filter=FunctionH3Test.* -j48`: all three tests
passed.
- `run-regression-test.sh --run -d
query_p0/sql_functions/spatial_functions -s test_h3`: generated results
with `-forceGenOut`, inspected them, and reran successfully. Includes
constant-folding modes, nullable/non-nullable table columns, all 16
resolutions, stored nested arrays, array expansion, filtering/grouping,
and 35,000 mixed rows.
- Compared 10,000 inputs per function against official C/Python H3
4.5.0: exact agreement. ClickHouse 26.3.9.8 (H3 3.7.1) matched cell IDs
and coordinates within tolerance (maximum coordinate error 8.29e-8
degrees).
- Clang-format 16 and targeted clang-tidy checks passed; no sanitizer
diagnostics during these tests. ARM/macOS builds were not run.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
### What problem does this PR solve?
Issue Number: None
Related PR: #67180, #67538, #67918
Problem Summary:
MATCH and SEARCH each had their own Boolean, phrase, term-expansion and
scoring executors, written separately for CLucene (V1/V2) and SNII
indexes. Every query optimization had to be implemented once per
executor, the formats drifted apart in behavior, and cross-field queries
could mix document IDs from different physical partitions.
This PR runs both on one engine:
- **One logical form.** MATCH (`query/logical`) and SEARCH
(`exprs/function/search`) lower their queries to the same logical nodes.
- **One query engine.** Boolean logic (AND/OR/NOT, minimum_should_match,
WAND), term expansion (prefix, wildcard, regexp), exact and sloppy
phrases and BM25 scoring are implemented once, independent of the index
format.
- **Formats behind an SPI.** CLucene and SNII only provide dictionaries,
postings, positions, norms, statistics and batched I/O through
`query/spi`. The old per-format executors are removed.
- **Layered source tree.** `be/src/storage/index/` is reorganized by
layer, and `build-support/check-header-deps.py` enforces the allowed
include directions.
Index formats and writers are unchanged, and existing indexes remain
readable. FE is unchanged.
#### Source layout
```text
be/src/storage/index/
├── index_*.h/.cpp shared index files: file reader/writer, compound reader, storage formats
├── ann/ bloom_filter/ zone_map/
└── inverted/
├── analysis/ analyzers, tokenizers, char and token filters, normalizers, gram schemes
├── common/ descriptors, caches, statistics, profiles, single flight
├── reader/ text index reader, index selection, collection statistics
├── query/ format-independent query engine
│ ├── logical/ MATCH lowering and logical nodes (gram/ builds gram queries)
│ ├── engine/ Query/Weight/Scorer: boolean, term, phrase, expansion, WAND, union, collect
│ ├── exec/ doc-id sets and sinks, term waves, chained conjunctions
│ ├── phrase/ exact and sloppy phrase matchers
│ ├── scoring/ BM25
│ └── spi/ the source interface each format implements
├── clucene/ CLucene (V1/V2) reader, searcher, writer, compaction
│ └── spi/ CLucene index source and postings cursor
└── snii/ SNII: format/, encoding/, io/, bkd/, reader/, writer/, compaction/, query/
be/src/exprs/
├── vsearch.h/.cpp VSearchExpr, the SEARCH expression node
└── function/search/
├── function_search.* SEARCH entry: Boolean assembly, DSL result cache, NULL handling
├── search_lowering.* SEARCH DSL to logical nodes; binds each field to an index
├── function_search_leaf.* compiles each leaf into a query on its bound index
├── search_predicate_collector.* collects scored leaves for BM25 statistics
└── variant_inverted_index_search.* SEARCH on VARIANT subcolumns and NESTED groups
```
#### Performance
The 18 production SQL cases (V2 and SNII: point, range, absent-term and
phrase queries) were compared with master on fresh dual-BE pairs: 8
pairs, 16 samples per case and binary, alternating order. Every case is
within the 5% limit for both `InvertedIndexFilterTime` and client wall
time; the largest ratios are 1.032 and 1.038. SNII phrase cases take
25-60% less FilterTime. TPC-H, TPC-DS and ClickBench are within noise.
<details><summary>Per-case median ratios (PR / master)</summary>
Run on 2026-10-14 against master 7ac0eb2fdb on a busy shared host
(median 90% CPU), which widens the intervals; all 4,608 result checks
pass.
| Case | FilterTime | Client wall |
| --- | ---: | ---: |
| `v2/long/early/full` | 1.030 | 1.024 |
| `v2/long/early/range` | 1.018 | 0.999 |
| `v2/long/absent/full` | 1.032 | 1.014 |
| `v2/long/absent/range` | 1.018 | 0.995 |
| `v2/long/mid/full` | 1.006 | 1.038 |
| `v2/long/mid/range` | 1.013 | 1.012 |
| `v2/short/tiny` | 0.965 | 1.018 |
| `v2/search/scored_phrase` | 1.010 | 0.990 |
| `v2/search/phrase_2` | 0.954 | 0.951 |
| `snii/long/early/full` | 0.950 | 0.994 |
| `snii/long/early/range` | 0.961 | 1.022 |
| `snii/long/absent/full` | 0.728 | 0.847 |
| `snii/long/absent/range` | 0.891 | 0.983 |
| `snii/long/mid/full` | 0.821 | 0.924 |
| `snii/long/mid/range` | 0.975 | 1.007 |
| `snii/short/tiny` | 1.020 | 0.985 |
| `snii/search/scored_phrase` | 0.404 | 0.971 |
| `snii/search/phrase_2` | 0.746 | 0.986 |
</details>
### What problem does this PR solve?
Issue Number: N/A
Related PR: #68369
Problem Summary:
Doris maintains separate handwritten SSE, NEON and scalar UTF-8
validators. Replace them with the simdutf dependency added in #68369,
keeping the existing boolean validation entry point and file-scan
validation setting. SIMD selection and short-input handling use
simdutf's runtime dispatch; no CPU-specific or length-specific dispatch
is added in Doris.
Remove the obsolete implementation and scalar-only API, update its
remaining test caller, and drop the old source/distribution license and
tooling exclusions. Bundle simdutf 9.2.0's complete BSD-3-Clause
ISA-detection notice in the binary distribution and reference it from
the license manifest. Fix the existing UTF-8 tests, whose
`sizeof(vector)` loops only exercised the first test case, and extend
coverage for Unicode boundaries, malformed and truncated sequences,
embedded NUL, empty/null buffers, SIMD boundaries, unaligned exact-sized
buffers, long inputs, independent row validation and the file-scan
bypass.
### Release note
None
### Check List (For Author)
- Test: Unit tests and static checks
- License follow-up: verified the bundled BSD notice byte-for-byte
against simdutf v9.2.0, checked both license-tool exclusion removals,
and executed the existing `build.sh` packaging function to confirm the
manifest and complete notice are included intact.
- License-Eye v0.8.0 passed with the repository configuration on the
four follow-up files and the newly unexcluded `utf8_check.cpp` (1 valid
source, 0 invalid, 4 ignored license/configuration files). The full
local worktree scan stopped on the existing `doris-skills` submodule
directory (`read doris-skills: is a directory`); the full repository
license check is left to CI.
- Passed all 9 UTF-8 utility tests in a standalone harness compiling the
actual modified implementation, test source and generated Thrift types.
Both the implementation and simdutf 9.2.0 were built with ASAN/UBSAN.
Passed with automatic dispatch and the supported icelake, haswell,
westmere and fallback implementations.
- Passed repository formatting and build-hygiene checks; clang-tidy
passed for the UTF-8 implementation and tests.
- Attempted the standard BE ASAN test runner with UTF-8, ColumnString,
SQL-function and Variant filters. The build stopped in unchanged
`common/cpp/aws_common.cpp` because the local AWS dependency lacks
`aws/core/auth/GeneralHTTPCredentialsProvider.h`; these integrated tests
did not run.
- clang-tidy for the one-line Variant test-helper update is blocked by
an existing unmatched `NOLINTEND` in unchanged
`be/src/core/types.h:576`.
- Behavior changed: No; validation semantics and the file-scan bypass
are preserved.
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: https://github.com/apache/doris/issues/64646
Related PR: None
Problem Summary:
Doris has no Japanese-aware tokenizer for the inverted index. Japanese
text has no spaces between words, so the existing parsers can't segment
it and `MATCH` / `MATCH_PHRASE` on Japanese columns end up with poor
recall and precision.
This PR adds a built-in `kuromoji` parser for Japanese, in the same
style as the existing Chinese IK analyzer. It's opt-in per column:
```sql
INDEX content_idx (`content`) USING INVERTED
PROPERTIES("parser" = "kuromoji", "parser_mode" = "search");
```
After indexing, MATCH, MATCH_PHRASE and TOKENIZE() run against the segmented Japanese terms.
How it works:
- Native C++ under `be/src/storage/index/inverted/analyzer/kuromoji/`, so there's no JVM on the indexing path. KuromojiAnalyzer / KuromojiTokenizer mirror the IK analyzer/tokenizer, with a Viterbi cost-model segmenter over the IPADIC connection-cost matrix.
- The dictionary is a process-wide singleton loaded once from ${inverted_index_dict_path}/kuromoji. An offline converter compiles raw IPADIC into a compact C++ runtime format (double-array trie + cost matrix + char/unknown tables) at build time, so no binary blob is committed.
- search (default), normal and extended modes are supported. No thrift/proto changes — parser and mode ride as strings in the index properties.
Dictionary source is mecab-ipadic-2.7.0-20070801 (NAIST-2003 license, the same lexicon Lucene kuromoji uses).
### Release note
Support Japanese text tokenization in the inverted index via a new kuromoji parser (`PROPERTIES("parser"="kuromoji")`), with `search/normal/extended` modes.
Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
```sql
CREATE TABLE test_jp (
id BIGINT,
content TEXT,
INDEX idx_content (content) USING INVERTED
PROPERTIES("parser" = "kuromoji", "parser_mode" = "search")
) ENGINE=OLAP
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES("replication_num" = "1");
INSERT INTO test_jp VALUES
(1, '東京都に住んでいます'),
(2, '日本語の形態素解析エンジン');
-- search-mode decompounding: 東京都 also matches 東京
SELECT id FROM test_jp WHERE content MATCH '東京'; -- expect: 1
SELECT id FROM test_jp WHERE content MATCH_PHRASE '形態素解析'; -- expect: 2
-- inspect segmentation directly
SELECT TOKENIZE('東京都に住んでいます',
'"parser"="kuromoji","parser_mode"="search"');
```
### What problem does this PR solve?
Issue Number: N/A (draft for early design review)
Related PR: N/A
Problem Summary:
The earlier revisions of this draft added the React application under
`ui-next/` next to the legacy AngularJS `ui/`, so the new UI was never
produced
by a normal build. Reviewers and users had to pass `CUSTOM_UI_DIST` by
hand,
and the packaged default was still the legacy application.
This revision finishes that transition instead of maintaining two
applications:
the React application takes over the `ui/` directory, the AngularJS
application is removed, and `./build.sh --fe` produces the new UI with
no flag
and no change to `build.sh`.
### What changed in the latest revision
- `ui/` now holds the React application; the AngularJS sources are
deleted and
the `ui-next/` directory name is gone. `build_ui()` already builds
`${DORIS_HOME}/ui` and packs `${DORIS_HOME}/ui/dist` into the FE
resources, so no build script change is required. `CUSTOM_UI_DIST` still
works for callers that package a prebuilt directory.
- `@testing-library/dom` is now an explicit dev dependency. React
Testing
Library only lists it as a peer dependency and re-exports `screen`,
`fireEvent`, `waitFor` and `within` from it, while `build_ui()` installs
with
`--legacy-peer-deps`, which does not install peer dependencies. Without
the
explicit entry the production build fails to type-check every test file.
- `enable_web_ui` now defaults to `true`. It gated the UI routes and the
SPA
forward while the new UI was optional; leaving it `false` would ship a
build
whose packaged UI answers 404 on every page.
- Also in this revision: the Playground toolbar and
object-tree refinements, and restoring the resolved identity in the
cookie
authentication path so `checkAdminAuth` no longer sees a null user. The
branch is rebased on master.
### Implemented scope
- English-only login and logout. A Doris user must have global `ADMIN`
privilege to use UI Next.
- Home version information and complete frontend/backend node status
tables.
- SQL Playground with formatting, a database/table/column object tree,
and
FE-managed JDBC sessions that preserve SQL session state across
statements.
Restored browser session IDs are verified with the owning FE before the
UI
reports the connection as ready.
- System proc navigation backed by the existing `/rest/v1/system` API.
- Active Sessions backed by the existing `/rest/v1/session` API.
- Log configuration, content, and verbose logger management backed by
the
existing `/rest/v1/log` API.
- Query Profile list and raw Text details backed by the existing
current-FE
`/rest/v1/query_profile*` APIs. Only `Task Type: QUERY` entries are
shown.
- Visual Profile for structurally compatible `MergedProfile` text.
Parsing is
performed in a bounded browser Worker, ELK produces the compound
Fragment
layout, and React Flow renders the read-only DAG. Missing
`MergedProfile`
data is reported as unavailable; incompatible data reports the parse
reason
instead of relying on a hard Doris-version gate.
- FE and BE Configuration tables backed by the existing manager
configuration
APIs. Mutable rows expose an ADMIN-only editor; immutable rows remain
read-only, including against direct mutation requests.
### FE backend scope
The FE addition remains limited to behavior that cannot live safely in
the
browser:
- Login and logout reuse `/rest/v1/login` and `/rest/v1/logout`. A thin
`/rest/v1/ui/me` endpoint bootstraps the authenticated ADMIN user and
CSRF
token.
- Home, System, Sessions, Log, Query Profiles, and Configuration reuse
existing
Doris HTTP controllers rather than adding parallel read controllers.
- Web SQL retains FE-local session ownership, one persistent JDBC
connection
per active session, execution/cancellation, bounded results, idle
cleanup,
and logout cleanup. It is exposed as the UI-independent
`/rest/v1/sql-sessions` API under `org.apache.doris.httpv2.websql`.
- Web SQL accepts HTTP Basic authentication or the existing Doris login
cookie;
both require global `ADMIN`. Cookie mutations additionally require CSRF.
- The existing Configuration manager endpoints now accept the
authenticated UI
cookie, preserve ADMIN authorization, use cluster-internal
authentication for
FE/BE fan-out, encode configuration values safely, and reject immutable
FE
changes server-side.
- `enable_web_ui=true` is an immutable FE-local master switch for UI
routes,
assets, login/bootstrap, and stateful SQL endpoints. Setting it to
`false`
returns 404 for those resources without stopping port 8030 or shared FE
HTTP
APIs.
- `enable_web_sql_session`, `web_sql_session_idle_timeout_seconds`,
`web_sql_max_sessions`, and `web_sql_max_result_bytes` are mutable.
Running
Web SQL operations read their current values without an FE restart.
Numeric
session limits reject non-positive values; the result-byte limit accepts
1 through 100 MiB.
- `StatementSubmitter` and the legacy Login controller remain unchanged
from
this PR's master baseline. The legacy Log mutation adds only a cookie
CSRF
check; its existing log behavior and response remain unchanged.
The Visual Profile parser and graph are adapted from
`apache/doris-website`
PR #4043 at commit `133f948c235995a917b2e1f6d4e9d764b6d62726` under the
Apache License 2.0. The website uploader, AI analysis, hCaptcha,
polling,
storage, recovery, and Docusaurus wrapper are not included.
### FE endpoints used by the UI
Enumerated from `ui/src/api/**`, split by whether the endpoint is added
here or
reused from existing Doris controllers.
Added by this PR:
| Endpoint | Method | Purpose |
| --- | --- | --- |
| `/rest/v1/ui/me` | GET | Bootstrap after login: current ADMIN user and
CSRF token (`UiSessionController`) |
| `/rest/v1/sql-sessions` | POST | Create a Web SQL session
(`WebSqlSessionController`) |
| `/rest/v1/sql-sessions/{id}` | GET, DELETE | Inspect and close a
session |
| `/rest/v1/sql-sessions/{id}/statements` | POST | Execute exactly one
statement |
| `/rest/v1/sql-sessions/{id}/cancel` | POST | Cancel the running
statement |
| `/rest/v1/sql-sessions/{id}/reset` | POST | Reset the connection state
|
| `/rest/v1/hardware_info/fe/version` | GET | Version-only route on the
existing `HardwareInfoController`, so Home does not trigger OSHI
hardware collection |
Reused, no new controller:
| Endpoint | Method | Page | Controller |
| --- | --- | --- | --- |
| `/rest/v1/login` | POST | Login | `WebConfigurer` |
| `/rest/v1/logout` | POST | Sign out | `WebConfigurer` |
| `/rest/v1/system?path=/frontends` | GET | Cluster Overview, Frontends
| `SystemController` |
| `/rest/v1/system?path=/backends` | GET | Cluster Overview, Backends |
`SystemController` |
| `/rest/v1/system?path=...` | GET | Proc System navigation |
`SystemController` |
| `/rest/v1/session` | GET | Sessions | `SessionController` |
| `/rest/v1/log` | GET | Log configuration and contents |
`LogController` |
| `/rest/v1/log` | POST | Add or delete a verbose logger, sent as the
legacy `add_verbose` / `del_verbose` form parameters | `LogController` |
| `/rest/v1/query_profile` | GET | Query Profile list |
`QueryProfileAction` |
| `/rest/v1/query_profile/text/{queryId}` | GET | Raw profile text,
shared by the Text and Visual tabs | `QueryProfileAction` |
| `/rest/v2/manager/node/configuration_info?type={fe\|be}` | POST |
Configuration read | `NodeAction` |
| `/rest/v2/manager/node/set_config/{fe\|be}` | POST | Configuration
update, ADMIN only | `NodeAction` |
`/rest/v1/ui/**` and `/rest/v1/sql-sessions**` go through `uiRequest`,
which
requires the login cookie and sends the CSRF header on mutations.
Everything
else goes through `legacyRequest` and retains each legacy controller's
existing
authorization behavior. The PR-added `/rest/v1/ui/**` and Web SQL
endpoints
require global `ADMIN`; reused legacy endpoints are not all globally
ADMIN-only. Web SQL additionally accepts HTTP Basic.
### Behavior and impact
- The FE web port now serves the React application. The AngularJS
application
is no longer part of the repository, so a legacy build is no longer a
rollback option; reverting this change is.
- Node.js 22.12 or newer and npm are required to build FE. The
repository's
x86, gcc10 and ARM compilation images are updated to provide Node.js
22.12.0. Neither Node.js nor npm is required at runtime.
- Web SQL sessions remain behind `enable_web_sql_session`.
### Validation
- `./build.sh --fe` with no flags builds the React application and packs
it:
the resulting `doris-fe.jar` `static/` payload matches `ui/dist` file
for
file.
- `npm run lint`, `npm run typecheck`, `npm run test` and `npm run
build` in
`ui/`.
- The branch is rebased on master and merges cleanly.
### Release note
The Doris FE Web UI is replaced with a new React application. Repository
compilation images now provide Node.js 22.12.0; the ARM image uses
AlmaLinux 8
because the official Node.js 22 ARM64 binary requires a newer glibc than
CentOS 7 provides.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (development and packaging steps are documented in
`ui/README.md`)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. The FE web port serves the new UI, and the AngularJS UI is
removed.
- Does this need documentation?
- [x] No. Development and packaging instructions are included in
`ui/README.md`.
- [ ] Yes.
### Follow-up
- Add CI coverage for the UI directory. No workflow currently runs its
lint,
typecheck, test or build, so frontend regressions cannot fail a build.
- Split the production bundle further; Vite currently reports large ELK
and
application chunks.
---------
Co-authored-by: Ragnaroos <Ragnaroos@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: morningman <yunyou@selectdb.com>
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #65586 (the branch-4.1 change this ports)
Problem Summary:
Hadoop's `FileSystem.CACHE` keys entries only by `(scheme, authority,
UGI)`. Different catalogs/TVFs accessing the same bucket/namenode with
different credentials could receive each other's cached `FileSystem`
instance (cross-credential contamination), so Doris forced
`fs.<schema>.impl.disable.cache=true` everywhere — which makes every
access (e.g. every JNI scanner split) create a brand-new `FileSystem`
instance (closing/releasing depends on GC).
This PR makes the cache credential-aware instead of disabling it. It is
the master counterpart of #65586; the 4.1 code no longer exists here
(the connector-plugin refactor moved storage properties into the
`fe-filesystem/*` SPI plugins and replaced the typed `StorageProperties`
hierarchy with `StorageAdapter`), so this is the same design re-layered
rather than a cherry-pick.
**BE/FE shared patch (hadoop-deps):**
- Ship a patched copy of `org.apache.hadoop.fs.FileSystem` (verbatim
from hadoop-common 3.4.2, hunks marked `DORIS-PATCH`): `Cache.Key`
carries one extra dimension read from the reserved conf property
`doris.fs.cache.key.<scheme>`, falling back to the scheme-less
`doris.fs.cache.key`. Absent/empty value = byte-for-byte vanilla
behavior.
- A second `DORIS-PATCH` in `loadFileSystems` binds the `ServiceLoader`
scan to `FileSystem.class.getClassLoader()` instead of the thread
context loader. `SERVICE_FILE_SYSTEMS` is a process-wide,
first-caller-wins registry; now that plugins share the kernel's
`FileSystem` (below), whichever plugin touched it first would otherwise
freeze the scheme registry to that plugin's classpath.
- maven-enforcer pins `hadoop.version=3.4.2`: upgrading hadoop fails the
build until the patched file is re-synced. The vendored file is recorded
in `LICENSE.txt` and excluded from checkstyle (scoped to
`org/apache/hadoop/fs/**` so `fe-connector-hms`'
`org/apache/hadoop/hive/**` stays checked).
**Why the key is per scheme.** One `Configuration` is routinely merged
from several storage definitions — a catalog can hold an object store
*and* HDFS — and every consumer merges them with `putAll`. Under a
single shared property name that merge is last-writer-wins: every scheme
ends up tagged with one arbitrary storage's fingerprint, which is both
unstable (map iteration order) and unsafe — two catalogs differing only
in their object-store credentials but sharing an HDFS definition would
collapse onto the same key. Distinct per-scheme names make the merge
lossless, so **no merge site has to know this mechanism exists** and no
consumer code changes: the fingerprint is set entirely inside the
storage layer, on the way out of the property getters.
**Load-order guarantees (four classpaths):**
- BE loads `be/lib/hadoop_hdfs/hadoop-deps.jar` before `lib/*.jar`
(existing).
- FE declares `hadoop-deps` as fe-core's first dependency (all
transitives excluded) and `start_fe.sh` prepends the jar explicitly —
`collectJars`/glob order does not deliver it
(`"hadoop-common-x.y.z.jar"` sorts *before* `"hadoop-deps.jar"`).
- Cloud meta-service: `build.sh` now also copies the module's own jar
(not just `copy-dependencies`' transitives) into
`cloud/output/lib/hadoop_hdfs/`, and `cloud/script/start.sh` prepends it
the same way. Without this the meta-service ran on the vanilla class and
silently ignored the key.
- **Plugins.** Filesystem plugins already resolve `org.apache.hadoop.*`
parent-first (`FileSystemPluginManager.FS_PARENT_FIRST_PREFIXES`), so
they see the FE copy. Connector plugins (iceberg/paimon/hudi/hive)
bundle their own `hadoop-common` and loaded it child-first, so they
would have run on a vanilla `FileSystem`. This PR adds
`org.apache.hadoop.` to
`ConnectorPluginManager.CONNECTOR_PARENT_FIRST_PREFIXES`, i.e. **no
connector pom or plugin zip changes**. Parent-first here is a delegation
*order*, not an exclusion: anything the kernel lacks still falls back to
the plugin's own jars, so `org.apache.hadoop.hbase.*` (hudi) and
`org.apache.hadoop.hive.*` still come from the plugin. Every
hadoop-namespace artifact the four plugins bundle resolves to the
identical GAV as the kernel's
(`hadoop-common/auth/annotations/hdfs-client/aws` 3.4.2,
`hadoop-shaded-guava` 1.2.0, `hadoop-shaded-protobuf_3_25` 1.5.0, and
paimon's `hadoop-huaweicloud` 3.1.1-hw-46, which fe-core also carries
via `${huaweiobs.version}`).
**FE fingerprint injection (storage layer only):**
- `FsCacheKeys` (fe-filesystem-api) computes a stable 32-hex SHA-256
fingerprint from the concrete class name (as salt) plus a sorted
identity map, exposed as `FileSystemProperties.fsCacheFingerprint()` and
published under `doris.fs.cache.key.<scheme>` for every scheme the
storage serves.
- **The identity is the effective credential set, not just the typed
one.** It is `matchedProperties()` (the user-supplied storage
definition, credentials included) plus every raw property that reaches
the effective `Configuration` without being a declared
`@ConnectorProperty` alias:
- `fs.` / `dfs.` / `hadoop.` / `juicefs.` — overlaid verbatim *after*
the storage's own map by
`IcebergCatalogFactory.buildHadoopConfiguration`,
`HudiScanPlanProvider.buildHadoopConf` and
`HdfsProperties.extractUserOverriddenHdfsConfig`;
- `paimon.s3.` / `paimon.s3a.` / `paimon.fs.s3.` / `paimon.fs.oss.` —
*re-keyed* onto `fs.s3a.` by `MetaStoreParseUtils.applyStorageConfig`
(every HMS-backed catalog) and `PaimonCatalogFactory`, so they never
appear verbatim yet decide the effective S3A credentials.
- Every key, value and the salt are **length-prefix framed** before
hashing (`len ':' value`, `null` → `-1:`), so the encoding is injective:
a value containing newlines or `=` cannot be smuggled to forge another
definition's fingerprint.
- The scheme set is `getSupportedSchemes()` ∪ `legacyCacheSchemes()`.
Neither alone is enough: the legacy set is wrong for the S3-compatible
dialects (COS declares `{cos, cosn}`, but Doris normalizes `cos://` to
`s3://` and the FileSystem is opened as `s3a`, so a fingerprint
published only under `cos` would never be read), while
`getSupportedSchemes()` is empty for providers with no scheme identity.
- `StorageAdapter` publishes it into `getBackendConfigProperties()` (now
a defensive copy) and into `hadoopStorageConfig`; the SPI providers
publish it into their own hadoop/backend maps in place of the removed
disable-cache keys. **Connector code is untouched.**
- REST vended credentials need no special handling either: a vended
overlay replaces the `...key.<scheme>` of the scheme it actually
re-credentials and leaves the other storages' entries intact.
- The blanket `fs.<schema>.impl.disable.cache=true` defaults are
removed; explicit user-provided values are still honored as an escape
hatch. The HDFS plugin's `HdfsConfigBuilder` keeps its per-instance
disable: it exists for manual lifecycle management
(`DFSFileSystem.close`), not credential isolation.
`OutFileClause` no longer mutates the returned backend map (it is a copy
now); it injects `fs.defaultFS` into the source properties and rebuilds
the descriptor.
**Notes:**
- Rolling upgrade: upgrade BE before FE (standard order). Old FE + new
BE stays safe because old FE still sends the disable flags; new FE + old
BE would re-expose the vanilla cache.
- `FileSystem.CACHE` has no eviction, so rotating vended credentials
accumulate entries over time — still strictly better than today's
new-instance-per-access behavior.
### Release note
Hadoop FileSystem cache is now enabled and keyed by a per-storage
credential fingerprint (`doris.fs.cache.key`) instead of being disabled
globally; explicit `fs.<schema>.impl.disable.cache` settings are still
honored.
### Check List (For Author)
- Test
- [x] Unit test:
- `DorisFileSystemCacheKeyTest` (hadoop-deps, 8) — patched class loaded,
vanilla default behavior, distinct keys isolate, equal keys share,
per-scheme key beats the generic one, generic key still honored without
a per-scheme entry, an unrelated scheme's key is ignored, service scan
ignores the thread context classloader
- `FsCacheKeysTest` (fe-filesystem-api, 7) — fingerprint stable and
order-independent, embedded separators cannot forge another property
set, key/value boundary unambiguous, `null` distinct from empty, salt
framed too, documented shape, per-scheme lower-cased property name
- `StorageAdapterFsCacheFingerprintTest` (fe-core, 13) — stability /
credential sensitivity, raw `fs.*` overlays changing the HDFS *and* S3
fingerprints, connector re-keyed `paimon.*` credentials changing the S3
fingerprint, smuggled separators, unrelated properties ignored, both
injection channels, per-scheme publication (incl. S3 dialects published
under `s3a`), a two-storage `putAll` merge keeping both fingerprints
independent, defensive copy, no blanket disable + explicit override
honored, patched `Cache.Key` on the FE classpath
- `ConnectorPluginHadoopPatchTest` (fe-core, 3) — a connector plugin
resolves the patched `FileSystem` from the kernel; without the
`org.apache.hadoop.` prefix it gets its own unpatched copy; parent-first
still falls back to the plugin when the kernel has nothing to offer
- `OutFileTest`, `HdfsPropertiesTest`, `AzureFileSystemPropertiesTest`
extended; updated HDFS / OSS-HDFS / JFS SPI parity goldens
- [ ] Regression test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test.
- Behavior changed:
- [ ] No.
- [x] Yes. Hadoop FileSystem caching is enabled again with
credential-aware keys; blanket cache disabling removed. Connector
plugins now resolve `org.apache.hadoop.*` from the FE kernel rather than
child-first.
- Does this need documentation?
- [ ] No.
- [x] Yes. `doris.fs.cache.key` is a reserved property; upgrade order
note (BE before FE).
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
### What problem does this PR solve?
When creating an AWS SDK V2 async client (e.g. S3AsyncClient), the SDK
requires a thread pool to manage asynchronous task scheduling, timeouts,
and retries (e.g. ScheduledExecutorService or async executor).
If no executor is explicitly provided, the AWS SDK creates its own
internal thread pool, which is expected to be shut down when
client.close() is invoked.
### Issue
In Doris, when using Paimon Catalog, some catalog implementations
provide an empty close() method. As a result, when a user executes DROP
CATALOG:
starting with Hadoop 3.4, the AWS SDK was upgraded to v2. Since #57226
upgraded HDFS to 3.4.2, the catalog runs into this issue.
Some Catalog instance is discarded,AWS SDK client.close() is never
called,The internally created thread pool cannot be shut down.
This leads to a thread leak
FYI https://github.com/aws/aws-sdk-java-v2/issues/3746
### Problem Analysis
The lifecycle of Catalog creation and destruction is complex and managed
internally by Paimon
Doris cannot reliably intervene in the call chain to enforce AWS SDK
client.close()
If each Catalog creates its own AWS SDK client with an internally
managed thread pool, threads will continue to leak as Catalogs are
repeatedly created and dropped
### Solution
This PR resolves the issue by introducing a shared executor strategy:
A Doris-managed shared thread pool is explicitly passed when creating
AWS SDK V2 clients
This prevents the AWS SDK from implicitly creating per-client internal
thread pools
The lifecycle of the executor is fully controlled by Doris and no longer
depends on Paimon Catalog’s close() implementation
With this approach, even if a Paimon Catalog’s close() method is a
no-op, the system will no longer leak threads.
None
### What problem does this PR solve?
remove memutil/gscoped_ptr/move/stl_util/charset/charmap/utf
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [x] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
1. add `flight_record_fe.sh` to listen the memory consume/gc events and
generate jfr file
2. add regression test action `flightRecord` to analyze the record
flight events for nereids (only support for jdk17+), if you need
flightRecord test, you should add `nonConcurrent` group for this test
background: some users want to find the bottleneck of the frontend, but they can not provide their query sql and schema because of privacy, so I embed async profiler to help they generate flame graph for frontend in their environment.
There has a script file `${FE_DEPLOY_PATH}/bin/profile_fe.sh` do the profile:
start profiler example
```shell
$ bin/profile_fe.sh
$ PROFILE_SECONDS=1 bin/profile_fe.sh
```
This is the real example:
```shell
$ PROFILE_SECONDS=1 bin/profile_fe.sh
DORIS_HOME: /Users/lanhuajian/github/doris/output/fe
JAVA: /Users/lanhuajian/Library/Java/JavaVirtualMachines/corretto-17.0.10/Contents/Home/bin/java
DorisFE pid: 29494
Begin profiling 1 seconds and generate flame graph to /Users/lanhuajian/github/doris/output/fe/log/profile_20240517154934.html...
Profiling for 1 seconds
Done
Generated flame graph to /Users/lanhuajian/github/doris/output/fe/log/profile_20240517154934.html
```
You can use web browser to open the generated html file to troubleshooting performance issues.
The flame graph html file would look like this:
<img width="1432" alt="image" src="https://github.com/apache/doris/assets/8806055/032a021e-6cbe-4e82-acf5-34a22dac3546">
**Note**: this profiler use the `async-profiler` tool, so it only support in Linux and MacOS
Add `JSON` datatype, following features are implemented by this PR:
1. `CREATE` tables with `JSON` type columns
2. `INSERT` values containing `JSON` type value stored in `String`, which is represented as binary format(AKA `JSONB`) at BE
3. `SELECT` JSON columns
Detail design refers [DSIP-016: Support JSON type](https://cwiki.apache.org/confluence/display/DORIS/DSIP-016%3A+Support+JSON+type)
* add JSONB data storage format type
* fix JsonLiteral resolve bug
* add DataTypeJson case in data_type_factory
* add JSON syntax check in FE
* add operators for jsonb_document, currently not support comparison between any JSON type value
* add ColumnJson and DataTypeJson
* add JsonField to store JsonValue
* add JsonValue to convert String JSON to BINARY JSON and JsonLiteral case for vliteral
* add push_json for MysqlResultWriter
* JSON column need no zone_map_index
* Revert "JSON column need no zone_map_index"
This reverts commit f71d1ce1ded9dbae44a5d58abcec338816b70d79.
* add JSON writer and reader, ignore zone-map for JSON column
* add json_to_string for DataTypeJson
* add olap_data_convertor for JSON type
* add some enum
* add OLAP_FIELD_TYPE_JSON type, FieldTypeTraits for it and corresponding cases or functions
* fix column_json offsets overflow bug, format code
* remove useless TODOs, add CmpType cases for JSON type
* add license header
* format license
* format be codes
* resolve rebase master conflicts
* fix bugs for CREATE and meta related code
* refactor JsonValue constructors, add fe JSON cases and fix some bugs, reformat codes
* modification be codes along code review advice
* fix rebase conflicts with master
* add unit test for json_value and column_json
* fix rebase error
* rename json to jsonb
* fix some data convert bugs, set Mysql type to JSON
This pr support
1. remove the generic type from operator, remove some NODE_TYPE from plan and expression
2. refactor Plan and NODE_TYPE generic type
3. support child class matching by TypePattern
4. analyze the code of operator and generate pattern makes it easy to create rules.
e.g.
```java
class LogicalJoin extends LogicalBinaryOperator;
class PhysicalFilter extends PhysicalUnaryOperator;
```
will generate the code
```java
interface GeneratedPatterns extends Patterns {
default PatternDescriptor<LogicalBinaryPlan<LogicalJoin, Plan, Plan>, Plan> logicalJoin() {
return new PatternDescriptor<LogicalBinaryPlan<LogicalJoin, Plan, Plan>, Plan>(
new TypePattern(LogicalJoin.class, Pattern.FIXED, Pattern.FIXED),
defaultPromise()
);
}
default <C1 extends Plan, C2 extends Plan>
PatternDescriptor<LogicalBinaryPlan<LogicalJoin, C1, C2>, Plan>
logicalJoin(PatternDescriptor<C1, Plan> child1, PatternDescriptor<C2, Plan> child2) {
return new PatternDescriptor<LogicalBinaryPlan<LogicalJoin, C1, C2>, Plan>(
new TypePattern(LogicalJoin.class, child1.pattern, child2.pattern),
defaultPromise()
);
}
default PatternDescriptor<PhysicalUnaryPlan<PhysicalFilter, Plan>, Plan> physicalFilter() {
return new PatternDescriptor<PhysicalUnaryPlan<PhysicalFilter, Plan>, Plan>(
new TypePattern(PhysicalFilter.class, Pattern.FIXED),
defaultPromise()
);
}
default <C1 extends Plan>
PatternDescriptor<PhysicalUnaryPlan<PhysicalFilter, C1>, Plan>
physicalFilter(PatternDescriptor<C1, Plan> child1) {
return new PatternDescriptor<PhysicalUnaryPlan<PhysicalFilter, C1>, Plan>(
new TypePattern(PhysicalFilter.class, child1.pattern),
defaultPromise()
);
}
}
```
and then we don't have to add pattern for new operators.
this function utilizing jsr269 to do something in compile time, and utilizing antlr4 to analyze the code of `Operator`, then we can generate corresponding pattern.
pattern generate steps:
1. maven-compiler-plugin in the pom.xml will compile fe-core three terms. first term will compile `PatternDescribable.java` and `PatternDescribableProcessor.java`
2. second compile term will compile `PatternDescribableProcessPoint.java`, and enable annotation process `PatternDescribableProcessor`, PatternDescribableProcessor will receive the event and know that `PatternDescribableProcessPoint` class contains the `PatternDescribable` annotation.
3. `PatternDescribableProcessor` will not process `PatternDescribableProcessPoint`, but find all java file exists in `operatorPath` that specify in pom.xml, and then parse to Java AST(abstract syntax tree).
5. PatternDescribableProcessor collect java AST and use `PatternGeneratorAnalyzer` to analyze AST, find the child class file for `PlanOperator` then generate `GeneratedPatterns.java` by the AST.
6. third compile term will compile `GeneratedPatterns.java` and other java file.
1. The clang format action will be triggered when a PR is submitted.
2. Skywalking eyes actions will be triggered when a PR is submitted and after merging to master branch.
#5146
Add histogram metrics into util/metrics.h. The data structure of histogram is implemented in util/histogram.h,
which could also be used in other situations that in need of histogram. Unit tests added as well.
1. Disable the MySQL client and LZO library by default when building the Doris.
MySQL client library is used for MySQL external table feature.
This feature will be replaced by the new ODBC external table soon.
LZO library is used to compress/decompress data of some old data format of Doris,
which is no longer used anymore.
2. Add missing license to some files.
3. For all non-Apache-License code, all are explained in NOTICE file and the corresponding license is declared.
4. Remove the js source code from webroot, it will be downloaded as thirdparty
1. upgrade gutil code from imapla to new verison， include `cpuinfo`, `spinlock` and `linux_syscall_support `
2. impliments arm version utf8 check code
3. remove incompatible code from stopwatch
After replacing Arena with MemPool, we can achieve one copy for string
value read from segment v2. We can exchange MemPool's chunk between
RowBlockV2 and RowBlock. This change only replace Arena, this work will
be done in other change list.
Mryange
Jack
HappenLee
Nishant Saharan
Calvin Kirs
Pxl
924060929
Mingyu Chen
924060929
Pxl
Shane
jiafeng.zhang
wangyongfeng
yiguoleiandZhengguo Yang
Mingyu Chen
Zhengguo Yang
Skysheepwang
Yingchun Lai
hffariel
lichaoyong
Seaven
xy720
Seaven
ZHAO Chun
lide
lide-reed
李超勇