32997 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Gabriel	035a843648	[feature](be) Add session controls for Parquet page indexes (#69971) ... ### What problem does this PR solve? Parquet output relies on implicit page-index and data-page-size defaults, so users cannot explicitly omit indexes or tune the target page size. Add two validated session variables shared by OUTFILE, EXPORT, Hive and Iceberg Parquet writers: - `enable_parquet_write_page_index`: write ColumnIndex/OffsetIndex metadata; defaults to `true`, preserving the current master behavior. - `parquet_write_data_page_size`: target uncompressed data-page size in bytes; defaults to `1048576`, accepts positive 32-bit integers, and applies with indexes enabled or disabled. Keep the existing 128 KiB target cap for Hadoop-framed LZ4. Other codecs use the requested target. Row-group size, dictionary-page size, and data-page/format versions are unchanged. Optional query fields preserve existing defaults when omitted by older FEs. ### Release note Add session controls for Parquet page-index output and target data-page size. ### Validation - 16 BE ASAN tests passed, including real file metadata/readback, index on/off, dictionary on/off, null/all-null columns, Parquet versions, Data Page V1/V2, page offset bounds, Hadoop-framed LZ4 limits, and Hive/Iceberg partition writers. - 9 FE tests passed for defaults, variable forwarding, Thrift serialization, valid bounds, and invalid input. - The new tests failed for the missing settings before implementation. - FE Checkstyle, clang-format 16, build hygiene, and clang-tidy passed. - No cluster SQL run or performance benchmark was performed locally; CI validation is pending. ### Check List (For Author) - Test: - [x] Unit Test - Behavior changed: - [x] Yes. Explicit controls are available; current defaults are preserved. - Does this need documentation? - [x] Yes. English and Chinese documentation: https://github.com/apache/doris-website/pull/4581 ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label	2026-11-04 13:06:31 +08:00
wudi	c0274bfb4a	[improvement](fe) Improve streaming job schema change error messages (#69725) ... ### What problem does this PR solve? When an unsupported upstream DDL pauses a streaming job, the existing `ErrorMsg` contains the unsupported-change reason, source table, and observed added/removed columns, but **does not contain the original upstream SQL or identify the mapped Doris target table**. For MySQL, the original DDL is available in the CDC logs only. Users therefore need to find that statement and resolve the target mapping before deciding how to adjust the Doris schema. This PR puts that diagnostic context directly in `ErrorMsg` and makes the recovery guidance consistent between the initial failure and a failed RESUME. | Information | Before | After | | --- | --- | --- | | Original MySQL DDL | Written to CDC logs; absent from `ErrorMsg` | Complete statement appended as `Source DDL: [...]`, including all clauses of a mixed DDL | | Table identification | Source table only | `source_table -> doris_db.actual_mapped_table` | | Same-name MODIFY/CHANGE | `(amount -> amount)` | `(amount)`; real renames retain both names | | PostgreSQL change details | Observed types/OIDs and column differences | Those details are retained, with the mapped target table added; no original SQL is available from the change event | | RESUME guidance | Confirmation explained on the initial failure, but omitted from the resumed failure hint | Both failure paths explain schema review and RESUME confirmation | | Target validation failure | Concrete error such as `missing columns [city]` | Retained alongside the diagnostic context and common recovery hint |	2026-11-04 11:08:18 +08:00
Gavin Chou	736649fcb4	[fix](regression) Isolate eliminate_distinct_constant database (#70105)	2026-11-04 10:44:51 +08:00
Mingyu Chen (Rayner)andGabriel	1adeb3a680	[fix](flight-sql) Return analyzed result schemas from Prepare and GetSchema (#69867) ... ### What problem does this PR solve? Issue Number: None Related PR: #68631, #68768 Problem Summary: **In short.** This ports branch-4.1 #68631 (also on branch-4.2) to master. A Flight SQL `CreatePreparedStatement` now returns the analyzed result schema of the statement, `GetSchema` is implemented for both statements and prepared statements, and a prepared execution is checked against the schema that Prepare advertised. Before this PR, Prepare always returned a fixed one-column placeholder, and `GetSchema` answered UNIMPLEMENTED. The analysis runs in the FE only and does not plan or execute the query. Original author: @Gabriel39 **Background.** Flight SQL clients learn the shape of a result before they fetch it. The Arrow Flight SQL JDBC driver prepares every statement and builds `ResultSetMetaData` from the dataset schema that `CreatePreparedStatement` returns. ADBC exposes the same schema as `execute_schema`, and `GetSchema` is the RPC that asks for it without running anything. **The problem, and what it cost.** On master: | | master today | with this PR | |---|---|---| | `CreatePreparedStatement` dataset schema | always `ResultMeta: Utf8`, whatever the statement | the statement's columns, names and Arrow types as BE / FE will send them; statements whose schema cannot be told before they run, and results with an untyped NULL column, keep the placeholder (see below) | | `GetSchema` for a statement or a prepared statement | UNIMPLEMENTED | the analyzed schema, or UNIMPLEMENTED for a statement that keeps the placeholder | | Invalid SQL | fails only at execution | fails at Prepare with INVALID_ARGUMENT | | A query whose privilege check fails | fails only at execution | still fails only at execution: Prepare keeps the placeholder, because only the check on the executed plan decides (see below), and GetSchema answers UNIMPLEMENTED without a schema | | Prepared handle used after the session's catalog / database changed, or after the result schema changed (e.g. another session's DDL) | runs the unqualified SQL in whatever namespace the session has now, with whatever columns come back | refused with NOT_FOUND "prepare again", and the result left by that execution is dropped | | Prepared handle presented by another session | resolved by the identity written in the handle | must start with the caller's own identity (from #68768) | | TIMEV2 / VARBINARY columns in GetTables and Prepare schemas | `Null`, while BE sends `float64` / `binary` | `Float64` / `Binary`, as BE sends them | A client that trusts the advertised schema saw one string column for every query, and could not ask for a schema at all without executing. **How this PR fixes it.** - `arrowflight/FlightSqlQuerySchema` analyzes the statement on a clone of the session variables: a USE / SWITCH prefix is resolved in a temporary namespace, the analysis `StatementContext` is closed on the same thread, and the connector statement scope it may have opened is closed explicitly. Each field gets the same `doris_type` markers and native Variant layout the execution side uses (`FlightSqlSchemaHelper.withDorisTypeMetadata` / `nativeVariantField`), and TIMESTAMPTZ carries the canonical session time zone that BE stamps on its batches. - Privileges are checked on the analyzed plan after the CTEs that execution inlines have been inlined (`PullUpCteAnchor`, `CTEInline`, as the rewriter does before its own check). A CTE used once is therefore checked only on the columns the query reads, and a CTE used more than once on every column it selects, which is also how execution checks them. The rewriter normalizes the plan further before it checks privileges. For example, it drops an ORDER BY under a derived table, so the analyzed plan can read a column that the executed plan does not. When the analyzed plan is denied, the statement keeps the placeholder and execution decides. No schema is disclosed in either case. - FE-side commands whose result header does not depend on running them now report that header without running: HELP, SHOW SNAPSHOT, SHOW CREATE TABLE (one header selector shared with `doRun`, including master's stream, plugin-view and system-table cases), SHOW PROC `/` and `/current_queries`, SHOW CONFIG, and EXPLAIN PLAN PROCESS (Rule / Before / After). - `ShowStoragePolicyCommand.getMetaData()` returned the four-column USING header for every form, while plain SHOW STORAGE POLICY answers the seven columns of `StoragePolicy.STORAGE_META_DATA`. It now returns the header of the form that runs. This also corrects the column definitions that MySQL `COM_STMT_PREPARE` reports for the statement. No other SHOW command declares a header that differs from the one it returns. - WARM UP SELECT keeps the placeholder. It answers OK instead of its statistics whenever planning proves its source empty: a predicate that folds to false, no partition left after pruning, or partitions without data. - A result with an untyped NULL column keeps the placeholder too, at any depth (`SELECT NULL`, `SELECT id, NULL FROM t`, `ARRAY(NULL)`). `DorisArrowTypeMapping` gives NULL_TYPE Arrow `Null`, but BE builds such a column as a `UInt8` flagged as a NULL literal and sends it as nullable `Bool`, so an advertised `Null` would expire the handle at every execution. A typed NULL (`CAST(NULL AS VARCHAR(20))`) is described as its type. - The prepared query kept in `FlightProtocolAdapter` records the SQL, the namespace and the advertised schema. A prepared execution re-analyzes the statement and compares the result with the advertised schema before it runs, and compares the schema it actually produced before it publishes a DoGet ticket. It is the session's next request whether or not these checks pass: before them, it drops what the previous request left, as a statement's execution does, while Prepare and GetSchema are metadata requests and leave it. On a mismatch, the handle is removed, the execution is cancelled, and the deferred coordinator, the channel result and the endpoints are dropped. A handle whose statement can no longer be described when it is re-analyzed, for example after a column grant was revoked, expires the same way, and the statement prepared again keeps the placeholder. A USE / SWITCH inside the statement rebinds only that handle. - `DorisArrowTypeMapping` maps TIMEV2 to `Float64` and VARBINARY to `Binary`, the types BE sends. TIMESTAMPTZ keeps the fixed `"UTC"` in GetTables, and AGG_STATE keeps `Null` there. - `FeServiceClient` rejects a remote Doris metadata timeout whose milliseconds do not fit in an int, instead of letting it wrap around. Where master differs from branch-4.x: - branch-4.x answers UNIMPLEMENTED at Prepare for statements whose schema it cannot tell before they run: DESC, SHOW DATA / PARTITIONS / QUERY STATS / PYTHON PACKAGES, other PROC nodes, COPY INTO, WARM UP CLUSTER, multi-statement requests that do not start with USE / SWITCH, `?` placeholders, and result types the FE cannot describe. Python ADBC falls back to executing without a prepared statement, but the Arrow Flight SQL JDBC driver prepares every statement and has no fallback, so on master these statements would fail over JDBC. `test_arrow_flight_session_lifecycle` runs DESC and `SET @multi = 2; SELECT ...` through JDBC. Here, these statements keep today's placeholder schema, and their handles are not checked at execution. - branch-4.x checks privileges on the analyzed plan as it is, so a CTE used once is checked on every column it selects, and any denial fails Prepare. Here, CTEs are inlined first, and a remaining denial keeps the placeholder. - branch-4.x advertises the statistics header for WARM UP SELECT and then rejects, as a schema change, an execution whose source turned out empty. Here, WARM UP SELECT keeps the placeholder. - branch-4.x advertises Arrow `Null` for an untyped NULL column, which BE sends as `Bool`, so every prepared execution of such a query is rejected as a schema change. Here, such a result keeps the placeholder. - The analysis and the checks run under the session's command lock (`FlightProtocolAdapter.runCommand` / `callCommand`) instead of `synchronized (connectContext)`, and the namespace is saved and restored with `getNameSpaceContext()` / `restoreNameSpaceContext()`. - EXPLAIN PLAN PROCESS has a result over Flight on master, so it gets a static header instead of UNIMPLEMENTED. PLAN REPLAYER and SHOW CREATE PROCEDURE do not exist on master and are not carried over. - SWITCH resolves the default database through the connector provider (`Env.defaultDatabaseOnUse`, now shared with `Env.changeCatalog`) instead of `EsExternalCatalog.DEFAULT_DB`. - TIMESTAMPTZ fields carry the canonical session time zone that BE stamps, where branch-4.x uses the raw session value. - Only the two handle checks from #68768 that do not depend on parameter binding are taken: the caller-prefix check and rebinding only the handle that ran the USE / SWITCH. Parameter binding itself is not part of this PR. - `GetTableTypes`, the other half of #68631, is implemented by #67899. Cost: every Prepare, every prepared execution and every `GetSchema` runs one more FE analysis of the statement. **Results.** On a local cluster built from this branch, every suite under `arrow_flight_sql_p0` passes: the new `test_schema_only` and `test_get_tables_schema`, plus the seven suites that run through Flight JDBC, `test_arrow_flight_session_lifecycle` among them. `datatype_p0/uuid/test_uuid_arrow_flight` passes too. `test_schema_only` checks the advertised schema of each kind of statement against what its execution actually returns. That covers queries, the side-effect-free frontend-side commands executed as prepared statements (SHOW STORAGE POLICY among them), a WARM UP SELECT on an empty table (executed in cloud mode, the only mode that warms up internal tables), and a `group_concat_state` AGG_STATE, which BE sends as Utf8. Co-authored-by: Gabriel <liwenqiang@selectdb.com>	2026-11-04 10:22:17 +08:00
Mingyu Chen (Rayner)	30ce25dc1f	[fix](be) Cancel Arrow Flight queries on the result BE when a result stream closes early (#69945) ... ### What problem does this PR solve? Issue Number: None Related PR: #68630, #67644 Problem Summary: #### In short This ports the BE half of branch-4.x's #68630 to master. When an Arrow Flight SQL client cancels or closes a result stream before its end, master keeps the query running: the BE thread serving the stream waits on the result buffer without ever checking whether the client is still there, and the unread result blocks stay in memory, holding back the query, until the query times out or the delayed buffer cleanup runs minutes later. With this PR the BE notices the cancelled stream within 20 ms, and when the stream is served by the BE that holds its result (the usual case), that BE cancels the query and frees all of the query's Flight result buffers there. The FE route that also reaches the query's other BEs (`cancelFlightQuery`, which changes `FrontendService.thrift`) comes in a follow-up PR. #### Background - A Flight SQL query leaves its result on the BEs. `GetFlightInfo` returns one endpoint per result buffer, and the client reads each endpoint with a `DoGet` stream. Since #67504 the FE closes the coordinator as soon as it has returned the `FlightInfo`, so it no longer cancels anything on behalf of an open stream. - `DoGet` is served by `ArrowFlightBatchLocalReader` when the result buffer is on the same BE, or by `ArrowFlightBatchRemoteReader`, which pulls the blocks from the result BE with the `fetch_arrow_data` BRPC. - The result buffer (`ArrowFlightResultBlockBuffer`) is keyed by the query id when parallel result sink is on (the default) and by the fragment instance id when it is off. When it is full, the result sink stops and the query waits for the reader. - `ResultBufferMgr` drops a buffer that nobody removed at `execution_timeout + 5 s`, or `result_buffer_cancelled_interval_time` (300 s by default) after the fragments writing it have finished, whichever comes first. Finishing with an error does not clear the unread blocks. `cancel_plan_fragment` is the BRPC the FE uses to cancel a query on a BE. #### The problem, and what it cost - **The local reader never looks at the client.** `get_arrow_batch` loops on 20 ms waits until a block arrives, the buffer is closed or it fails. It has no access to `ServerCallContext::is_cancelled()`, so a `DoGet` cancelled while the buffer is empty keeps a Flight server thread waiting until the query produces a block or ends. - **Closing a stream cancels nothing.** The readers do not override `RecordBatchReader::Close()`, and their destructor only logs. A client that stops reading (a `LIMIT` reached on the client side, a cancelled cursor, a dropped connection) leaves the query running and its unread blocks in the buffer. When the result does not fit in the buffer, the full buffer back-pressures the result sink and the fragments, with their memory, wait until the query times out; the unread blocks go only when the buffer is dropped. - **The remote reader cannot be interrupted.** It waits for `fetch_arrow_data` with `brpc::Join`, which returns only when the result BE answers or `arrow_flight_reader_brpc_controller_timeout_ms` (300 s by default) passes. - **An FE cancel leaves the results behind.** `cancel_plan_fragment` cancels the fragments but not the query's Flight buffers, so their unread blocks stay until the delayed cleanup 300 s later. - **A fetch for a removed buffer never completes.** When `fetch_arrow_data` cannot find the buffer, it logs and returns without running `done`, so the caller waits for its BRPC timeout instead of getting an error. #67644 fixes the same line. #### How this PR fixes it - **Bounded waits, cancellation checks in the reader.** `get_arrow_batch(result, eos)` waits at most 20 ms and then returns OK with no block and `eos = false`; `eos` now marks the real end. The local reader retries in a loop and checks `is_cancelled()` before each wait. `DoGetStatement` passes `context.is_cancelled()` to both readers. - **The reader closes itself.** `ArrowFlightBatchReaderBase` now implements `ReadNext` (which calls each reader's `ReadNextImpl`) and `Close()`. A stream counts as aborted when it is cancelled, closed before EOF, fails while reading, or is destroyed before EOF; a stream that reached EOF is never treated as aborted. - **Aborting a local stream cancels the query on this BE.** `ResultBufferMgr` keeps an index from query id to the query's Flight buffers, maintained under the buffer-map lock together with the map. `cancel_arrow_flight_buffers` removes every Flight buffer of the query from the map, then, outside the lock, fails each buffer (which wakes its waiting fetches and releases the back-pressure) and cancels the `QueryContext` it holds a weak reference to. Sibling buffers of the same query are cleared even after their `QueryContext` has expired. - **Remote waits can be interrupted.** The remote reader waits for its BRPC with a callback that checks `is_cancelled()` every 20 ms and calls `brpc::StartCancel` when the stream is gone. - **`cancel_plan_fragment` drops the Flight buffers of an aborted query.** It calls `cancel_arrow_flight_buffers` unless the reason is `LIMIT_REACH` or `FINISHED`, which can leave valid unread results that the client still has to read. - **`fetch_arrow_data` answers when the buffer is gone.** It completes the RPC with the lookup error. The line is identical to the one #67644 adds, so the two PRs merge in either order. Differences from branch-4.x: the two consistency checks between the index and the buffer map are `DORIS_CHECK` (an `if` and a `DCHECK` in 4.x), a null check on the reader's memory tracker that both constructors make unnecessary is not carried over, and the `ArrowFlightResultBlockBuffer` constructor moved to the .cpp so that `varrow_flight_result_writer.h`, which 13 TUs include, does not include `runtime/query_context.h`; only the .cpp needs its definition. The new lines also take a few clang-tidy fixes (the destructor calls `ArrowFlightBatchReaderBase::Close()` explicitly instead of a virtual call, `std::scoped_lock`), and the test starts its BRPC server inside `ScopedLeakCheckDisabler`, like the other BE tests that start one, because brpc registers process-lifetime built-in services on its first start. Not in this PR: the FE route. A remote reader's abort, and an abort on one result BE, still do not reach the query's other BEs; their fragments and buffers wait for the timeout as before. The follow-up adds `FrontendService.cancelFlightQuery` (same signature as branch-4.x), the BE broadcast to the FEs and the FE routing table, together with the FE unit test and the `test_flight_cancel_cleanup` regression test. | Situation | Before | After | |---|---|---| | Client cancels `DoGet` while the result buffer is empty (local reader) | Server thread waits until the query produces data or ends | Returns within 20 ms; the query is cancelled on this BE | | Client closes a local stream before EOF | Query keeps running, back-pressured until it times out if the result does not fit; unread blocks kept for minutes | Query cancelled on this BE; all of its Flight buffers there freed at once | | Client cancels while a remote fetch is pending | Waits for the result BE, up to 300 s | BRPC cancelled within 20 ms | | FE cancels the query (`cancel_plan_fragment`, abort reason) | Unread blocks kept until the delayed cleanup, 300 s later | Flight buffers freed at once; `LIMIT_REACH` / `FINISHED` still keep them | | `fetch_arrow_data` for a removed buffer | RPC never completes; caller waits for its timeout | Completes at once with an error | | Stream read to EOF, then closed | Nothing cancelled | Nothing cancelled | | New `DoGet` with the ticket of a stream aborted on its result BE | Reads the remaining blocks, or EOS | Fails: the buffer is gone | #### Results - `ArrowFlightCancelTest` (9 tests, new): early close and destruction free a back-pressured buffer and cancel the query; sibling buffers are freed after their `QueryContext` expired; EOF followed by close is not a cancellation; cancellation interrupts an empty-buffer wait; empty waits return for retry and do not end the stream; a conversion error cancels the query; a real Flight server sees a client-side `Cancel()` through `ServerCallContext` and cleans up. - `ArrowFlightRemoteCancelTest` (3 tests, new, against an in-process BRPC `PInternalService`): cancellation interrupts a pending `fetch_arrow_data`; an aborting `cancel_plan_fragment` frees the buffer and a following fetch completes with an error inside its 1 s timeout; `LIMIT_REACH` / `FINISHED` keep the unread results. - On a local single-BE cluster, branch-4.x's regression `test_flight_cancel_cleanup` (it aborts one endpoint after the first batch, by closing it and by an explicit cancel, and requires the query's pipelines to disappear within 20 s) fails on master with `Cancelled Flight query retained pipelines: [...: 1 pipeline fragment contexts in query ...]` and passes with this PR. The suite comes with the follow-up, because on a multi-BE cluster it also needs the FE route. The 20 `arrow_flight_sql_p0` suites and the 14 `external_table_p0/remote_doris` suites, which read results over Arrow Flight, pass before and after. - The 12 new tests plus `ResultBufferMgrTest` (6) and `ArrowResultBlockBufferTest` (4) pass 10 times in a row in one ASAN process. Ten single-point mutations of the change (dropping the buffer cleanup in `cancel_plan_fragment`, the `on_failure` call, the local cancel in `close()`, the cancellation check in the local read loop or in the BRPC wait, the sibling loop, the EOF flag, the `QueryContext` cancel, the cancel in the destructor, the `eos = false` on an empty wait) each turn their target tests red.	2026-11-04 10:15:45 +08:00
Gabriel	28cbcfc233	[fix](build) Enable Parquet modular encryption in Arrow (#70087) ... ### What problem does this PR solve? The third-party Arrow build leaves `PARQUET_REQUIRE_ENCRYPTION` at its default `OFF`. Parquet still installs encryption APIs and exports some encryption symbols, but its archive contains `encryption_internal_nossl.cc.o`. Reading a standard PME-encrypted Parquet file with the correct key then throws: ```text Calling encryption method in Arrow/Parquet built without OpenSSL ``` Linking `libssl.a` and `libcrypto.a` into the consumer does not replace those compiled stubs. This change supplies the build prerequisites for native encrypted Iceberg reads. The change: - Enables `PARQUET_REQUIRE_ENCRYPTION` and selects static OpenSSL. - Pins OpenSSL headers and both archives to the Doris third-party prefix, and fails before CMake if any required input is missing. - Installs `encryption_internal.h` and `internal_file_decryptor.h` from the same pinned Arrow source. Arrow normally excludes these internal headers, but module-level decryption is needed to retain the native Parquet scan path. Existing third-party prebuilt packages must be rebuilt to pick up these changes. SQL-level encrypted Iceberg read integration remains separate work. ### Release note None ### Check List (For Author) - Test: - [x] Manual test (details below) - Behavior changed: - [x] Yes. Third-party Parquet is built with PME enabled; missing owned OpenSSL inputs fail before configure. - Does this need documentation? - [x] No. This is the third-party build prerequisite, with no new SQL-facing configuration. #### Validation On Linux x86_64, extracted Arrow 24.0.0 from the archive pinned by `thirdparty/vars.sh`, applied the current Doris Arrow patches, and executed the changed `build_arrow()` function in an isolated prefix seeded with existing third-party dependencies. The complete Arrow build and install succeeded. CMake selected PME `ON` and OpenSSL 1.1.1s from that prefix. The rebuilt archive contains `encryption_internal.cc.o` / `openssl_internal.cc.o`, with no `nossl` object. A standalone C++ reader linked directly to the checked/rebuilt static archives, including the same prefix's OpenSSL archives, used the public Apache [Parquet test fixtures](https://github.com/apache/parquet-testing/tree/master/data): | Check | Before: checked PME-disabled package | After: rebuilt package | | --- | --- | --- | | `uniform_encryption.parquet.encrypted`, correct public test key | OpenSSL-disabled runtime error | All-column scan succeeds: 50 rows | | Same file, wrong key | OpenSSL-disabled runtime error | Authentication failure: `Failed decryption finalization` | | Same file, no key | Missing decryption-properties error | Missing decryption-properties error | | `alltypes_plain.parquet` | All-column scan succeeds: 8 rows | All-column scan succeeds: 8 rows | Additional checks: - Module-level AES-GCM round trip and rejection of a different data-page AAD pass. - Both installed internal headers match the source and compile/link with `InternalFileDecryptor` and the module decryptors. - Removing each required OpenSSL input rejects the build before CMake. The baseline function reaches CMake for the same incomplete inputs. - Captured Linux and Darwin invocations contain the PME option and explicit static OpenSSL paths; this is an argument check, not a macOS build. - `bash -n thirdparty/build-thirdparty.sh` and `git diff --check` pass. Validation limits: no full Doris BE build, Doris SQL test, AArch64 build or native macOS build was run. No runtime reader behavior is enabled by this PR alone. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label	2026-11-04 10:12:58 +08:00
Socrates	b5996fbf10	[chore](paimon) Upgrade paimon-rust and reserve ANN Thrift fields (#70075) ... ### What problem does this PR solve? Related PR: #69493 Problem Summary: Master still builds paimon-rust v0.4.0-rc1, while the Paimon vector-search work in #69493 needs v0.5.0-rc1 and additional wire fields. Upgrade the native dependency and reserve its Thrift layout on master so later changes cannot reuse the same identifiers for different meanings. The dependency change updates the source URL, archive name, source directory and MD5 checksum together. The schema change adds the following optional fields and their Paimon payload/options types: | Structure | Field | ID | | --- | --- | --- | | `TPaimonFileDesc` | `vector_payload` | 24 | | `TExternalSearchRequest` | `paimon_options` | 6 | | `TFileScanRangeParams` | `external_search_request` | 42 | `TPaimonVectorPayloadType` keeps `BUCKET_SPLIT_BYTES=0` and `MATERIALIZE_OPAQUE_BYTES=1`. Both payload members and all Paimon search-option members are optional. Master does not emit these fields yet; this PR reserves the protocol without enabling SQL/TVF ANN search or upgrading the Paimon Java SDK. Compatibility notes: - `TPaimonFileDesc.20` remains `original_file_path`; comments reserve branch-4.1's `blob_files=21`, `anchor_first_row_id=22` and `anchor_row_count=23`. The vector payload uses 24 rather than the conflicting 21 from the original #69493 diff. - Comments reserve the existing branch-4.1/4.2 shared-search fields `search_query=2`, `search_filter=3`, `vector_search_options=4` and `prefilter=5`. Their query types can be introduced independently on master. - `TFileScanRangeParams.41` remains `fluss_properties`. - The affected definitions were reviewed against current master and branches 3.1, 4.0, 4.1 and 4.2. Existing field IDs and enum values retain their meanings. ### Release note Upgrade the native Paimon Rust reader dependency to v0.5.0-rc1. ### Check List (For Author) - Test: No local build, Thrift generation or unit tests on master, as requested; CI validation pending. During the preceding branch-4.2 work, the archive checksum was verified and the v0.5.0-rc1 dependency was built with Rust 1.94.0. This is not a completed master BE build or BE unit-test run. - Behavior changed: Yes; the bundled native Paimon dependency is upgraded. The new Thrift fields are reserved and remain unset by master producers. - Does this need documentation: No	2026-11-04 10:12:43 +08:00
924060929	11bfb73889	[fix](metacache) Complete guarded external table lookup cleanup (#69806) ... ### What problem does this PR solve? Related PR: #69501, #69629, #69632, #69720 Problem Summary: Complete guarded cleanup of stale external table lookup state after confirmed remote absence, preserving refreshed or same-name recreated tables and unrelated numeric lookup routes. External metadata reads can discover a dropped table after resolving a cached handle while FE names, table objects and numeric ID routes remain stale. A refreshed names list can also omit a resident old object. In case-folding modes, targeted cleanup previously cleared unrelated numeric routes. Route confirmed provider absence through a typed SPI exception carrying the exact remote database/table identity. Capture the existing lookup observation before handle resolution and retain it through base/system schemas (including pinned system schemas), latest/pinned row counts, ordinary write-schema binding and row-level DML validation, partition-values and partition-name metadata (SHOW PARTITIONS and PARTITIONS), eager/deferred scan partition views, MVCC partition/freshness reads and deferred loads, and scan column/property/filter/estimation and synchronous/deferred split reads. System relations repair their shared source identity; query TVFs have no shared lookup identity. Missing files, transport/permission failures and unrelated resources do not authorize cleanup. Preserve live views, cache TTLs, HMS exception wrapping and comment display fallbacks. Runtime, names, object and metadata-refresh guards reject delayed cleanup after refresh or same-name recreation. Catalog-wide, metadata-only and explicit lookup refreshes use the catalog-owned generation and active count across their full interval, protecting replacement database/table objects admitted by normal eviction or published during hot/cold invalidation. Database refresh and truncate replay include their cold-cache branches; rename replay, overlapping refreshes and errors use the same owner guard. Existing per-table metadata generations remain part of the observation. Ordinary writes remain metadata-only and do not reload table names. Targeted wide invalidation retires only the target's ROOT-folded ID aliases while retaining unrelated numeric routes. Provider IO remains outside the catalog monitor. Pinned system schema reads use the existing schema-session lifecycle hooks. Their live statement owns the borrowed scope until all aliases and scans finish; the schema read does not prematurely close it. The guarded resolver preserves master's view classification before table-endpoint access, including connectors that return non-404 errors for views; live view deletion still permits guarded cleanup. Hive file-size estimates preserve confirmed table absence until guarded FE cleanup, then retain the unknown estimate fallback. Sampled ANALYZE file-size reads retain the same observation and their existing failure/session-closure behavior. Ordinary listing failures still degrade statistics to unknown. Native Hive table and table-scoped partition-name RPCs preserve exact-source absence under the existing HmsClientException boundary; both ordinary/fresh and cancellation-aware listings use the same RPC normalization. Individual partition absence, ordinary failures and successful empty lists keep their existing behavior. MTMV partition warmup retains its pre-handle MetadataRead through snapshot binding and partition enumeration. The combined interface retains current master's DEFAULT-column and partition-filter APIs. Native Iceberg write-shaping loads preserve exact-source table absence through the existing shared converter, while ordinary write exceptions, resource tracking and statement ownership keep their prior contracts. The shared absence exception bumps the connector plugin API to 29.0. No cache owner framework, persisted ID format, or journal format changes. ### Release note Repair stale external-table lookup state after confirmed table/view absence while preserving refreshed or recreated tables and unrelated numeric lookups.	2026-11-04 10:08:42 +08:00
hui lai	cbc30b9813	[fix](fe) Align Thrift wire IDs with branch-4.1 (#70057) ... ### What problem does this PR solve? Problem Summary: During a rolling upgrade from 4.1 to master, the same Thrift enum value or field ID can have different meanings on FE and BE: - `TSchemaTableType` diverges starting at value 62. For example, 4.1 sends `SCH_FILE_CACHE_INFO = 62`, which master interprets as `SCH_BLACKHOLE`. - `TRLTaskTxnCommitAttachment.firstErrorMsg` uses field 12 in 4.1 but field 13 in master. Master uses field 12 for the Kinesis progress struct, so mixed-version readers skip the error message because the wire types do not match. Align master's shared enum values with branch-4.1 (also matching branch-4.2), place master-only database properties and table stream values at 70–72, and preserve extensions/TSO at 73/74. Restore `firstErrorMsg` to field 12 and move the optional Kinesis progress field to 13. Compatibility review: checked both definitions on branch-3.1, branch-4.0, branch-4.1, branch-4.2 and master. Branch-4.0 already differs from 4.1 for `SCH_BLACKHOLE`, `SCH_COLUMN_DATA_SIZES`, `SCH_LOAD_JOBS` and `SCH_BE_COMPACTION_TASKS`; this master-only fix does not resolve that existing release-to-release mismatch. These corrections also change the affected wire IDs relative to unpatched master. ### Release note Fix schema table enum values and routine load commit attachment field IDs on master to match Doris 4.1 and 4.2 during rolling upgrades.	2026-11-04 10:06:53 +08:00
daidai	b65a564ea5	[fix](be) Route glibc allocation entry points through jemalloc (#69819) ... ### What problem does this PR solve? Problem Summary: On glibc < 2.25, dynamic TLS can be allocated by libc and freed by jemalloc, crashing during thread cleanup. Route all seven glibc allocation entry points through the existing jemalloc wrappers, preserve allocator API semantics, and add native interposition tests. ### Release note Fix a BE crash during dynamic TLS cleanup on older glibc systems and keep glibc allocation entry points paired with jemalloc. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason `./run-be-ut.sh --jemalloc-hook --run -j 2`: 12/12 passed on Linux x86_64/glibc 2.32. Full BE, older glibc, ARM64 and customer JNI E2E were not run. - Behavior changed: - [ ] No. - [x] Yes. Glibc allocation entry points now consistently use jemalloc, with allocator boundary semantics preserved. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label	2026-11-04 10:00:12 +08:00
Chenjunwei	b78c656c0c	[fix](iceberg) Commit every Iceberg file under the table's own location (#69392) ... ### What problem does this PR solve? Issue Number: None Related PR: #69082 Problem Summary: **In short.** DELETE, UPDATE and MERGE on an Iceberg table recorded the delete files they write (v2 position delete files and v3 deletion vector files) under the internal `s3://` address BE writes them at, instead of the table's own location (`oss://`, `s3a://`, `abfss://`, ...). Doris could not always read such a table back, and other engines and maintenance procedures resolved those files with the wrong file system. This PR commits delete files under the table's location. It also makes every Iceberg file Doris writes take both of its paths from one place in BE and pass one check in FE, so the next writer cannot repeat the mistake without failing. #### Background Every Iceberg sink FE sends carries two forms of the table's data location: - a normalized one BE can address the storage with (`output_path`): FE rewrites `oss://`, `cos://`, `cosn://`, `obs://`, `bos://`, `gs://`, `s3a://`, `s3n://` and Azure `abfs[s]://` / `wasb[s]://` locations (except OneLake) to `s3://`; - the raw one Iceberg metadata must record: `original_output_path` on `TIcebergTableSink`, `table_location` on `TIcebergDeleteSink`, both on `TIcebergMergeSink`. BE writes each file under the first form and reports it in `TIcebergCommitData.file_path`, which FE writes into the manifests as it is. So the reported path must be in the second form. #### The problem, and what it cost Reproduce on an Iceberg table located at `oss://bucket/...`: ```sql CREATE TABLE t (id INT, v STRING) PROPERTIES ('format-version' = '2'); INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c'); DELETE FROM t WHERE id = 1; SELECT file_path FROM t$delete_files; -- s3://bucket/.../data/delete_pos_xxx.parquet SELECT file_path FROM t$data_files; -- oss://bucket/.../data/... ``` - Whether Doris can still read the table depends on the catalog and storage binding. In a real Azure ADLS Hadoop catalog test with explicit SharedKey, Doris SELECT failed with `No storage properties found for schema: s3`, and Spark failed with `UnsupportedFileSystemException: No FileSystem for scheme "s3"`. - Other engines and maintenance operations such as `expire_snapshots` / `remove_orphan_files` may resolve the wrong file system from the recorded scheme, depending on the engine and FileIO. Doris's own `remove_orphan_files` refuses to run on such a table (`Cannot determine whether listed and reachable file locations are equivalent`). - Cause: data files were written under `write_path` and committed under `original_write_path` (`VIcebergPartitionWriter`), but `VIcebergDeleteSink` had no such split; it reported the `output_path`-based path it wrote. - Why it went unnoticed: the two writers mapped the paths in different ways, `VIcebergDeleteFileWriter` reported its write path by default, and FE accepted any path. CI did run on such a location: the Iceberg REST fixture's warehouse is `s3a://`, which FE normalizes to `s3://`. But the DML suites only checked that delete file paths contain `/data/delete_pos_`, and a stray `test_iceberg_update_delete_basic.out.org` in the repository records the wrong `s3://warehouse/wh/...` paths. #### How this PR fixes it 1. `[fix](be) Commit Iceberg delete files under the table's own location`, `[test](be) Verify Iceberg delete file commit paths` and `[test](be) Initialize the Arrow memory pool in the Iceberg position delete write test`: `VIcebergDeleteSink` writes under `output_path` and commits position delete files and deletion vectors under `table_location`. An FE unit test pins that both sinks carrying `table_location` name the same directory as `output_path`, and `DataSinks.thrift` documents the contract. The position delete write test sets up the Arrow memory pool the Parquet writer allocates from, so it also passes when run alone (CI review bot finding on this PR). 2. `[refactor](be) Take both paths of every Iceberg file from one output location`: - The new `IcebergOutputLocation` (header-only) holds one directory in both forms and gives the write path and the commit path of a file name, and of a partition subdirectory. - The data writer (`VIcebergTableWriter` / `VIcebergPartitionWriter`), the delete file writer and the Puffin path all take their paths from it. `VIcebergDeleteFileWriter` now reports the commit path itself instead of its write path. - A table or delete sink without both location fields is rejected in `open()` / `init_properties()` instead of committing a write path or a relative path. The FE of every branch (3.1, 4.0, 4.1, 4.2, master) sets both fields on every Iceberg sink. - Directories join file names with exactly one `/`. 3. `[improvement](fe) Reject Iceberg commit data reported outside the sink's location`: - `planWrite` registers on the transaction the raw location(s) each sink it builds carries, read back from the built sink. - `IcebergConnectorTransaction.addCommitData`, where every report from every writer enters FE, rejects a file that is not under a registered location, or has no path. FE then answers the report as not accepted, and BE deletes the files under the paths it wrote them at and cancels the query, so a wrong path fails the statement and leaves no orphan files. - Thrift comments state the contract, including that "the same directory" can differ in more than the scheme (Azure moves the account out of the authority). Comments only, no protocol change. 4. `[test](regression) Check that every Iceberg write records its files under the table location`: - The new `test_iceberg_write_file_locations` runs INSERT, INSERT OVERWRITE PARTITION, branch INSERT, DELETE, UPDATE, MERGE and `rewrite_data_files` on a v2 table, and INSERT, DELETE, UPDATE and MERGE on a v3 table. After each operation every file in `$all_files` must start with the table's LOCATION plus `/data/`. A new write operation belongs in this suite too. - `test_iceberg_update_delete_basic` checks the prefix of its delete files as well, and the stray `.out.org` is removed. | Scenario | Before | After | | --- | --- | --- | | DELETE / UPDATE / MERGE on a v2 table at `oss://`, `s3a://`, `abfss://`, ... | position delete files recorded as `s3://...` | recorded under the table's location | | The same on a v3 table | deletion vector (Puffin) files recorded as `s3://...` | recorded under the table's location | | A BE writer reports a file outside the location its sink carries | committed as reported | the statement fails; BE deletes the file | | A sink plan without `original_output_path` / `table_location` (no supported FE sends one) | data files committed as `/p=1/x.parquet`, delete files as `s3://...` | rejected before any file is written | | `write.data.path` ending with `/` | data files at `.../data//p=1/x.parquet` | `.../data/p=1/x.parquet`, as Iceberg's own location provider places them | Delete files that an older version already committed with `s3://` keep that address until they are rewritten or expired. Upgrade BE before FE, as usual. An upgraded FE rejects the `s3://` delete file paths that a BE without this fix reports, so until that BE is upgraded, DELETE / UPDATE / MERGE on a table whose location FE normalizes fails instead of committing the wrong path. #### Classes, and how they call each other - `IcebergWritePlanProvider.planWrite` (FE) builds the sink and registers the raw location(s) it carries with the transaction (`commitLocationsOf`). - `IcebergConnectorTransaction.addCommitData` (FE) accepts a reported file only under a registered location; `IcebergWriterHelper` then turns the fragments into `DataFile` / `DeleteFile`. - `IcebergOutputLocation` (BE, new) is one directory in its write form and its commit form. - `VIcebergTableWriter` builds the data location in `open()` and hands each `VIcebergPartitionWriter` its partition directory (`WriteInfo.location`). `VIcebergDeleteSink` writes position delete files through `VIcebergDeleteFileWriter` and deletion vectors into one Puffin file, both from its location. `VIcebergMergeSink` builds one of each. ``` FE planWrite ── sink: output_path + raw location ──────────────► BE │ addCommitLocation(raw location) IcebergOutputLocation(output_path, raw location) ▼ ├─ VIcebergTableWriter ─► VIcebergPartitionWriter data files IcebergConnectorTransaction ├─ VIcebergDeleteSink ──► VIcebergDeleteFileWriter position deletes ▲ addCommitData: file_path under a registered │ └─► Puffin writer deletion vectors │ location, else reject the report └─ VIcebergMergeSink ──► both of the above └──────────── TIcebergCommitData.file_path = commit path ◄───────┘ ``` Not changed: Thrift fields (comments only), the scan side, and how FE normalizes locations.	2026-11-04 09:59:11 +08:00
Chenyang Sun	6802f57005	[chore](build) Require new thrift/proto ids to land on master first (#70066) ... Require new thrift/proto ids to land on master first	2026-11-04 09:41:00 +08:00
yujun	0140035142	[fix](fe) Remove misleading missing-progress warnings for INSERT jobs (#70077) ... A successful `INSERT INTO dst SELECT ... FROM src` can repeatedly emit `ProgressManager` missing-meta warnings. Materialized view refreshes use the same INSERT execution path and can produce large volumes of these warnings. INSERT jobs have a load job ID but do not register scan progress. Reports may also arrive after a bulk load removes its progress entry. A missing entry therefore does not indicate an error. Remove the diagnostic entirely and continue updating only registered entries, consistent with `addTotalScanNums`. Backend statistics reporting and progress registration/cleanup remain unchanged.	2026-11-04 08:15:30 +08:00
Calvin Kirs	7866f5a36c	[fix](mtmv) Batch non-empty partition checks during MV compensation (#69966) ... ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Cloud MV partition compensation checks each valid mapped MV partition through a singleton non-empty request. Cached-empty or unknown partitions can therefore cause many serial MS version requests, repeated across related partition tables, even when the MV is not ultimately selected. Collect unique mapped valid MV partition IDs and call the existing bulk interface in batches of at most 5,000. Reuse the returned ID set across related tables, preserving partition removal, base-table compensation, cache handling and exception propagation. The 5,000-partition cap is a fixed code constant, not a runtime configuration. It bounds the three request ID lists to at most 165,000 bytes at the worst protobuf integer width, plus the remaining request fields. This is a request-size guard, not a claimed performance-optimal batch size. Tests verify that 1,020 candidates use one bulk call and 10,001 use three; these are mocked interface call counts, not a live Cloud benchmark.	2026-11-03 22:29:08 +08:00
Gavin Chouandgavinchou	6b3481f28b	[chore](fe) Upgrade BDB JE to 18.3.18 (#69970) ... ### What problem does this PR solve? Upgrade the FE BDB JE dependency to the latest Doris snapshot requested for all active target branches. ### What is changed and how does it work? - Update `je.version` in `fe/pom.xml` to `18.3.18-doris-SNAPSHOT`. ### Related PRs - Canonical Apache Doris master PR: https://github.com/apache/doris/pull/69970 ### Check List - [x] No local build or test was run, as requested for this dependency-only version update. Co-authored-by: gavinchou <gavinchou@apache.org>	2026-11-03 22:16:58 +08:00
Gabriel	856194ece0	[test](regression) Check ORC lazy reads using row results (#70059) ... ### What problem does this PR solve? Issue Number: N/A Related PR: N/A The ORC lazy materialization suite asserts exact counters from merged query profiles. Reader counters can be absent when an empty scanner supplies the profile merge template, making the suite fail independently of query correctness. Replace profile polling and metric assertions with full-row comparisons against an eager scan with ORC min/max filtering and lazy materialization disabled. Expected rows are filtered independently in Groovy. Exercise all four combinations of the two options with six predicates covering equality, partial and complete groups, empty results, tail reads, and ranges spanning groups. Disable result and condition caches during the checks and restore the modified session settings in `finally`. This suite now checks result equivalence rather than exact internal filtering counts. No production code is changed. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [x] Manual test (details below) - [ ] No need to test or manual test. Validation completed: - `git diff --check` passed. - Regression runner dry-run passed: `./run-regression-test.sh --run -d external_table_p0/hive -s test_orc_lazy_mat_profile -dryRun`. - An offline Groovy/H2 harness executed the suite against rows read from its existing ORC fixture: all 24 filtered queries passed. Fault injections detected missing rows, duplicate rows, incorrect payloads, ignored predicates, and an empty fixture. Session settings were restored after both successful execution and assertion failures; disabling Hive skipped all SQL. The dry-run and offline harness do not execute the Doris/Hive read path. Full Hive-backed integration validation has not been run locally. - Behavior changed: - [x] No production behavior change. Only regression assertions change as described above. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label	2026-11-03 18:37:08 +08:00
Mingyu Chen (Rayner)	1f71a66565	[fix](fe) Keep sibling capabilities on iceberg-on-HMS tables pinned at a snapshot (#69999) ... ### What problem does this PR solve? Issue Number: None Related PR: #69299, #69488 Problem Summary: #### In short An Iceberg table stored in HMS and read through an HMS catalog (`'type'='hms'`) loses every capability it inherits from the embedded Iceberg connector in any statement that pins the table at a snapshot. Since #69299 that is every statement that reads it: it gets no Top-N lazy materialization and no nested column pruning, and `DEFAULT(column)` in the conditions of `UPDATE` and `MERGE` silently matches nothing. This PR makes the snapshot overload of `HiveConnectorMetadata#getTableSchema` pass the inherited capabilities on, as the latest-schema overload already does. It is a master-only fix (branch-4.x has no connector SPI), left over from #69488 (review comment 4355508100). #### Background fe-core resolves a table-scoped capability (`PluginDrivenExternalTable.hasCapability`) from the catalog's connector or from the per-table set in the table's schema. It never asks the embedded connector that built the schema of an iceberg-on-HMS table. So `HiveConnectorMetadata` copies the capabilities that fe-core resolves per table (`SIBLING_INHERITABLE_CAPABILITIES`: auto and sampled analyze, Top-N lazy materialization, nested column pruning, field-id access paths, storage predicate pruning, nested column DDL and write-only defaults) from the owning connector into the per-table set of each schema it delegates. Iceberg declares seven of the eight; it does not declare sampled analyze. It did this only in `getTableSchema(session, handle)`. `getTableSchema(session, handle, snapshot)` returned the sibling's schema unchanged. fe-core uses that overload for explicit time travel. Since #69299 it also uses it for the latest-snapshot pin of every statement that reads an Iceberg table: the pin retains its schema (`retainSchema(true)`), `PluginDrivenMvccExternalTable` builds the schema at that pin, and `getSchemaCacheValue()`, where `hasCapability` reads the per-table set, returns it for the rest of the statement. #### The problem, and what it cost Reproduced on a local cluster with a standalone Hive 3.1.2 metastore. Two identical format-version 3 tables `(id INT, c INT, s STRUCT<a:INT, b:STRING>)` get two rows, then `ADD COLUMN d INT DEFAULT 42`, `RENAME COLUMN s.a TO a2` and a third row with `d = 7`. The same statements then run against one table through the Iceberg catalog and against the other through the HMS catalog. | Statement | Iceberg catalog | HMS catalog, master | HMS catalog, this PR | |---|---|---|---| | `EXPLAIN SELECT * FROM t ORDER BY id LIMIT 1`, also with `FOR VERSION AS OF` | `VMaterializeNode` | no lazy materialization | `VMaterializeNode` | | `EXPLAIN SELECT struct_element(s, 'a2') FROM t` | `all access paths: [s.a2]` | no pruning | `all access paths: [s.a2]` | | `UPDATE t SET c = 100 WHERE d = DEFAULT(d)` | updates rows 0 and 1 | updates nothing | updates rows 0 and 1 | | `MERGE INTO t USING (SELECT 0 AS id) s ON t.id = s.id AND t.d = DEFAULT(t.d) WHEN MATCHED THEN UPDATE SET c = 300` | updates row 0 | updates nothing | updates row 0 | | `DELETE FROM t WHERE d = DEFAULT(d)` | fails: `cannot be used in a DELETE` | deletes nothing | fails the same way | | `SELECT DEFAULT(d) FROM t` | fails: `must reference a column of the table that this statement writes` | returns NULL | fails the same way | The `UPDATE`, `MERGE` and `DELETE` rows fail silently: the statement succeeds and changes nothing. Reads of the renamed field return the right values in all three columns: on master the HMS catalog does not prune the struct at all, and with this PR it prunes it with field-id access paths, as the Iceberg catalog does. `INSERT ... VALUES (DEFAULT(d))` and `INSERT ... SELECT DEFAULT(d)` already wrote 42 through both catalogs. #### How this PR fixes it Both overloads now go through one private helper, `delegatedTableSchema`. It resolves the owning sibling once, asks it for the schema (at the pin, for the snapshot overload), and passes the result through the existing `reflectSiblingCapabilities`. The snapshot reaches the sibling unchanged. A hudi-on-HMS table is unaffected: the Hudi connector declares no capabilities, so its schema comes back as it was. A plain Hive table never reaches this branch. #### Results Through the HMS catalog, every statement in the table above now plans and answers as it does through the Iceberg catalog.	2026-11-03 18:27:49 +08:00
Mingyu Chen (Rayner)	e4c2dc261a	[fix](be) Reject Arrow timestamps outside the DATETIMEV2 and TIMESTAMPTZ ranges (#69697) ... ### What problem does this PR solve? Issue Number: None Related PR: #69581 #68516 Problem Summary: **In short.** An Arrow timestamp whose civil year is outside 0000..9999 is read into DATETIMEV2 or TIMESTAMPTZ as a different, valid-looking value: timestamp(us) 2005980768000000000, which is 65537-01-01, reads as a non-null 0001-01-01. This PR checks the year before the value is narrowed into Doris' 16-bit year field, as the ORC path already does, and fails the read otherwise. The review of #69581 found it on the Lance path; the code is shared by every Arrow reader. **Background.** `DataTypeDateTimeV2SerDe::read_column_from_arrow` and `DataTypeTimeStampTzSerDe::read_column_from_arrow` decode Arrow timestamps for ADBC, Arrow stream load, paimon-rust, remote Doris and, once #69581 lands, Lance, nested leaves included. Both convert epoch seconds with `DateV2Value::from_unixtime()`, which passes cctz's 64-bit year to `unchecked_set_time(uint16_t year, ...)` ("there's no overflow check since it's hot path"). #68516 added a check before this narrowing to the ORC path (`orc_timestamp_to_datetime()`). **The problem, and what it cost.** | Arrow timestamp(us) | DATETIMEV2 before | TIMESTAMPTZ before | |---|---|---| | 253402300800000000 (10000-01-01) | stored as year 10000, outside the type's range | error | | 2005980768000000000 (65537-01-01) | 0001-01-01 00:00:00 | 0001-01-01 00:00:00+00:00 | | a year before 0000 | wraps to 655xx or back into 0000..9999 | error, or a wrapped valid year | In the wrapped cases the read succeeds with a wrong value, and nothing in the result tells the user. **How this PR fixes it.** - DATETIMEV2: convert to civil time in the value's time zone (UTC for a timezone-naive Arrow timestamp, the session time zone for a zoned one) and reject a year outside 0000..9999 before packing, like `orc_timestamp_to_datetime()`. After packing, `is_valid_date()` is checked too: cctz counts year 0 as a leap year and Doris does not (`is_leap(0)` is false), so the civil date 0000-02-29 passes the year check, and before this PR it was stored as an invalid date. - TIMESTAMPTZ is stored in UTC, so `append_timestamptz_from_utc_epoch_micros` checks the instant against 0000-01-01 00:00:00 UTC .. 9999-12-31 23:59:59.999999 UTC before converting. Its runtime `is_valid_date()` check after packing stays, for the same leap day: timestamp(us) -62162121600000000 (0000-02-29 UTC) is inside the range but packs into an invalid value. The range message now says 0000-9999, the range `is_valid_date()` accepts. - Both fail the read with DataQualityError, as the other range errors of these readers do. NULL slots are still skipped before conversion. - Not changed: the Parquet INT64 / INT96 timestamp paths validate the UTC range before packing (`parquet_timestamp_micros()`, `parquet_int96_timestamp_micros()`), so they could not wrap. Arrow reads into the deprecated DATE / DATETIME (v1) types go through `VecDateTimeValue::from_unixtime()` with the same unchecked narrowing; they are left as they are. `orc_timestamp_to_datetime()` checks only the year, so an ORC timestamp on 0000-02-29 is still stored as an invalid date; it is not changed here. **Merge order: after #69581.** A NULL STRUCT, LIST or MAP row may hide any child value, an out-of-range timestamp included, because Arrow leaves the payload under a NULL parent unspecified. #69581 masks those child slots in the shared nested SerDes (`mask_arrow_children_hidden_by_null_parents`, df95b41a33) before any leaf SerDe reads them. Without it, the checks here fail a batch whose visible values are valid: a STRUCT with rows `[NULL, {ts: 0}]` whose NULL row carries 2005980768000000000 reads with DataQualityError on this PR alone, and reads both rows on a tree with both PRs. **Results.** | Arrow timestamp(us) | DATETIMEV2 after | TIMESTAMPTZ after | |---|---|---| | 253402300799999999 (9999-12-31 23:59:59.999999) | read | read | | -62167219200000000 (0000-01-01) | read | read | | -62162121600000000 (0000-02-29, not a Doris date) | DataQualityError | DataQualityError | | 10000-01-01, 65537-01-01, -62167219200000001 (year -1) | DataQualityError | DataQualityError | | zoned, at +08:00: 253402271999999999 / 253402272000000000 | 9999-12-31 23:59:59.999999 / DataQualityError | |	2026-11-03 18:21:39 +08:00
3 people	f109e130f7	[fix](arrow-flight) Support table type metadata and filters (#67899) ... ### What problem does this PR solve? Related to #67578 (partial A3). Flight SQL clients can obtain FlightInfo for GetTableTypes, but reading its ticket fails with UNIMPLEMENTED. GetTables also forwards only the first table_types entry to a service that treats only VIEW as a filter: [VIEW, BASE TABLE] loses base tables, while [BASE TABLE] or an unknown type can return excluded rows. Implement the type stream using the same Doris names as GetTables (BASE TABLE, SYSTEM VIEW and VIEW), sorted according to Flight SQL. Match the complete filter against the existing authorized listing before collecting optional schemas. Request only name/type metadata to avoid unused table statistics, and retain the session guard and Arrow buffer cleanup on failed sends. A type is the TABLE_TYPE that information_schema.tables reports for the object: table streams are VIEW, and external catalogs report their tables and views as BASE TABLE. The service lists every authorized object and the filter runs on that listing. Requested names match case-insensitively, and TABLE and SYSTEM TABLE, which Flight SQL calls commonly supported, select BASE TABLE and SYSTEM VIEW: the ADBC Flight SQL driver asks for `table` (Doris's own ADBC catalog lists a Doris source with `getObjects(..., {"table"})`), and JDBC tools ask for `TABLE`. Any other name matches only a type of that name, so UNKNOWN selects nothing. Because the filter now runs on the full listing, `FrontendServiceImpl.listTableStatus` refreshes a cloud follower's partition versions from the meta service only when UPDATE_TIME is requested, the one column that reads them. GetTables, which asks for names and types, never waits on those RPCs or is cut short when they fail, and information_schema.tables scans that do not read UPDATE_TIME skip the refresh too. Co-authored-by: morningman <yunyou@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com>	2026-11-03 18:21:21 +08:00
shuke	173a0e2064	[improvement](ci) Upgrade review runtime to Codex 0.166.0 (#69990) ... ### What problem does this PR solve? Related PR: https://github.com/shuke987/codex/pull/6 Upgrade the Code Review Runner and its runtime canary from Codex 0.162.1 to the validated [0.166.0 fork release](https://github.com/shuke987/codex/releases/tag/exec-goal-v0.166.0), retaining goal completion and same-session resume support. Codex 0.166.0 distinguishes `policy_capacity_unavailable` from ordinary overloads. Accept both canonical terminal capacity messages so the existing bounded recovery policy continues to apply. Preserve the original diagnostic and exact submission verification; unrelated failures remain non-retryable. Update the archive, version and CLI checksum pins together. The workflow changes are limited to seven package pin replacements. Existing model selection and delivery checks are preserved.	2026-11-03 17:45:20 +08:00
Gabriel	74b477d3c1	[fix](fe) Avoid database object loading for schemata scans (#70010) ... ### What problem does this PR solve? Issue Number: N/A Problem Summary: Queries such as: ```sql SELECT COUNT(*) FROM example_catalog.information_schema.schemata; ``` only need database and catalog names. However, `SchemaSchemataScanner` uses the default `getDbNames` RPC, whose FE handler calls `catalog.getAllDbs()` to also return database IDs. For external catalogs, this resolves every database object even though SCHEMATA never reads those IDs. A large namespace list can therefore churn a much smaller database-object cache and allocate substantial temporary metadata while waiting inside a synchronous BE-to-FE RPC. A warm database-name cache does not remove this work. This PR adds the optional `include_db_ids` request field, defaulting to `true`, and lets SCHEMATA set it to `false`. The FE then calls `catalog.getDbNames()` and omits the unused database IDs. Both paths share the existing privilege checks, pattern matching, and schema-name formatting. Callers that need IDs keep the object-enumeration path without additional intermediate lists. The new field uses ID 8; field 7 remains reserved. Definitions were checked on master and branch-3.1/4.0/4.1/4.2. Older FE nodes ignore the optional field and keep serving the existing response; older BE callers retain the default ID-inclusive behavior on a new FE. ### Validation - Added a 100,000-namespace test that rejects any database-object enumeration and verifies the result-array lengths and omitted IDs. - Covered privilege/pattern filtering, qualified schema names, empty/unavailable catalogs, existing ID-returning callers, and binary deserialization of requests without the new field. - Confirmed the new behavior tests fail on the original handler before implementing the change. All 7 targeted FE tests pass, including the existing `FrontendServiceImplTest#testGetDBNames`. ```bash bash run-fe-ut.sh --run 'org.apache.doris.service.FrontendServiceGetDbNamesTest,org.apache.doris.service.FrontendServiceImplTest#testGetDBNames' ``` - FE Checkstyle: passed with zero violations. - Changed BE scanner: clang-format 16 and C++ syntax-only compilation with regenerated Thrift headers passed. An in-process FE-handler benchmark uses the actual `ExternalCatalog` / `TestExternalCatalog` name and object-cache implementations, a synthetic metadata provider, and stubbed catalog routing and privilege checks. It compares the original handler source at master commit `bdb90856e6e468eb495e5bfd67010de90570bfbc` with the patched handler, using the same dependencies and fixture. Each trial starts with a warm name cache and cold object cache, whose capacity is 1,000. There are 100,000 synthetic namespaces plus the two system databases. Settings: JDK 17, 2 GiB heap, 4 active processors; discard one warm-up pair and report medians from three measured trials. | Metric | Original master | Patched SCHEMATA path | | --- | ---: | ---: | | FE handler elapsed time | 2,984.2 ms | 1,331.1 ms | | Cumulative current-thread allocation | 2,117.2 MiB | 726.2 MiB | | Database objects constructed | 100,002 | 0 | | Name-provider invocations | 1 | 1 | | Returned rows | 100,002 | 100,002 | In this fixture, handler time falls by 55.4% (2.24x faster) and cumulative allocation falls by 65.7%. Allocation is total bytes allocated during the call, not peak live heap. This benchmark excludes network transport, Thrift serialization, and SQL execution; it does not measure a deployed PostgreSQL/Iceberg environment. Full BE linking and a deployed end-to-end SQL test were not run locally. ### Release note Improve `information_schema.schemata` enumeration for large catalogs by avoiding unnecessary database-object loading. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [x] Manual test (FE-handler benchmark described above) - [ ] No need to test or manual test. - Behavior changed: - [x] No. SQL result semantics and existing ID-returning requests are preserved. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label	2026-11-03 17:39:12 +08:00
Calvin Kirs	1aa505cb0b	[fix](fe) Preserve non-UTF-8 prepared statement parameters (#69805) ... ### What problem does this PR solve? Related PR: #69314 JDBC `setBytes()` and Connector/ODBC binary bindings can send `MYSQL_TYPE_VAR_STRING` / `MYSQL_TYPE_STRING` rather than BLOB. FE decoded those bytes with Java's replacement UTF-8 decoder, silently changing arbitrary binary data. For example, `0080FF4100C328` became `00EFBFBDEFBFBD4100EFBFBD28` in `SELECT HEX(?)`. The BLOB handling added by #69314 did not cover these wire types. Decode string parameters strictly and retain non-UTF-8 payloads as binary literals. Bind those payloads through a BE-side cast to the original string category, preserving string function and comparison typing without replacing bytes. Valid UTF-8 retains the existing text literal types, and BLOB types retain their existing binary behavior. Use value-based binary literal child hashes so equivalent casts remain equal after rebinding and forwarding. ### Release note Preserve non-UTF-8 bytes in MySQL server-prepared string parameters, including JDBC `setBytes()` and Connector/ODBC binary bindings. ### Check List (For Author) - Test: - [x] Regression test: generated and replayed `prepared_stmt_p0/binary_parameters`; replayed `prepared_stmt_p0/send_long_data`. Covers all byte values, malformed UTF-8, parameter reuse, stored HEX/MD5, string expression predicates, NULL/empty values and ordinary Unicode text. - [x] Unit Test: 17 tests passed across `MysqlParameterLiteralTest` and `MysqlConnectProcessorLongDataTest`, including string wire aliases, cached types, uploads and forwarding. - [x] Manual test: Connector/J 8.2.0 server-prepared `setBytes`, `setBinaryStream` and `setBlob` preserve `0080FF4100C328`. - [x] Manual test: Linux Connector/ODBC 5.3.14 Unicode, 17 cases passed, including 1-MiB binary/Unicode inputs. An additional 17-case proxy-rewritten VARCHAR wire-type probe also passed. - [x] FE build and Checkstyle passed. - Behavior changed: - [x] Yes. Non-UTF-8 string-coded parameters preserve the input bytes instead of substituting U+FFFD. Valid UTF-8 text behavior is unchanged. - Does this need documentation? - [x] No. Runtime regression checks used the newly built master FE with an existing 4.1.4 BE and condition cache disabled. The predicate regression uses `CONCAT(value, '') = ?`; direct constant predicate pushdown on a current master BE was not validated. No BE, storage-format or protocol-field changes are included.	2026-11-03 17:04:13 +08:00
lihangyu	a35707da74	[fix](variant) Write Variant V2 sparse keys in byte order (#70001)	2026-11-03 16:37:26 +08:00
Gabriel	c7d1df40c0	[fix](external) Preserve source values and alias semantics in ES scans (#69672) ... ### What problem does this PR solve? Elasticsearch scans can return values that fail SQL predicates when indexed values differ from the values Doris reads, or when a scan switches all selected columns from docvalues to source. Parallel alias scans can also bypass alias filters/routing, and supported mapping shapes can fail during planning. For example, projecting a normalized keyword forces a source read for its siblings. A numeric alias then reads NULL although its docvalue is 5; a declared empty ARRAY remains nonnull although ES considers it missing; a date-only string reads local midnight instead of the indexed UTC midnight. Boolean empty strings and coerced integer strings must also remain readable: `""` means false and `"1.2e3"` means 1200. Keeping their indexed predicates while pruning SQL residuals returns incorrect rows. #### Execution flow 1. Select a populated mapping independently of alias response order. Distinguish typeless flags/root objects from legacy type wrappers, including a legacy type named `runtime`, and preserve array annotations. Accept empty/configuration-only alias backings during safety analysis while retaining standalone schema diagnostics. 2. Resolve the selected keyword query path and docvalue read path separately on every backing, including filter-only columns. Fetch index mapping defaults with the existing metadata request and honor per-field overrides for every alias backing; inherited `ignore_malformed` can remove indexed values while retaining source values. Ignore transformations on unused sibling paths and unrelated source filters. Missing/read-changing paths disable the affected docvalue entry. Root `enabled:false` and disabled source (`enabled:false` or `mode:disabled`) restrict every affected column. 3. Determine the final scan-wide read mode before building DSL. In source mode, STRING/ARRAY predicates and predicates on rounded numbers, aliases, scripted fields and default dates retain local residuals without narrowing the ES query. Eligible pure docvalue projections preserve their indexed-value behavior; scalar `_id` metadata remains pushable. 4. Keep searchability separate from value equivalence: supported doc-value-only fields remain queryable with `index=false`. Require existence for SQL inequalities. Normalize retained equality negation, double negation and Boolean negation inside the connector. Retained ranges keep the original ES query complement with an existence guard: an inverse existential range changes matching on arrays such as [1,2]. This preserves NULL rejection when optimizer NOT simplification is disabled. 5. Pass residual tracking and the existing early-LIMIT restrictions through the current scan contract. Native `esquery` remains remote-only: native-only expressions and independent AND conjuncts work; an indivisible Boolean expression requiring local native evaluation fails explicitly during planning. 6. Use both readers' shared parser for supported source representations. Boolean empty strings become false in scalar and ARRAY paths. Scalar numeric/Boolean/date singleton source arrays read their element; empty and null-only arrays read NULL, while STRING/JSONB and declared ARRAY containers remain intact. JSONB singleton, empty and null-element arrays retain their original JSON container. Declared ARRAYs preserve nullable members; numeric empty strings become SQL NULL in scalar and ARRAY paths, while Boolean empty strings remain false. Integer coercion follows the ES family: byte/short/integer strings use double rounding before truncation, while long strings retain exact decimal/exponent handling and 64-bit precision. Bounds are checked before narrowing. Declared integer ARRAY elements share the family-specific conversion. Numeric floating tokens are converted directly with checked bounds, preserving numeric INT64_MIN and the Java upper-endpoint conversion. Floating strings and narrow integer strings accept Java hexadecimal exponents, type suffixes and outer ASCII whitespace; FLOAT strings round directly to float rather than through double. Epoch-millis source dates accept zero, short/signed decimal strings and fractional numeric values, with checked bounds and second-precision floor semantics. Four-digit default-format years retain their calendar interpretation; pre-epoch formatted docvalues use the same floor. 7. For a source read involving disabled-source backings, pass an optional `read_full_source` flag through the existing ES property map. BE requests `_source:true`: disabled hits have missing values, while enabled alias backings retain their source. Ordinary source scans keep field projection and pure docvalue scans keep docvalue requests. 8. Scan aliases once by their original name so ES applies alias filters and search routing without duplicate shared-shard scans. Physical indices retain shard-parallel scans. #### Effects and tradeoffs This change preserves source values, supported decoding and SQL filtering across both readers and read modes. Source fallbacks can fetch more candidate rows because unsafe predicates remain local. Mixed aliases containing disabled-source backings can fetch additional source fields to avoid ES rejecting projected-source requests. Alias scans use one coordinating scroll to preserve alias semantics; physical-index parallelism remains. No throughput improvement is claimed. There are no new public SQL/catalog/session properties or SPI/Thrift/Protobuf definitions; the optional internal flag uses the existing flat property map. #### Known limitation Mixed-generation FE/BE execution during smooth upgrades is outside this change's scope. Source plans depending on the updated decoder require the updated BE for correct results; this PR does not add a backend-generation gate. ### Release note Fix Elasticsearch source/index value discrepancies, supported Boolean/floating/integer source decoding, default epoch-millis date decoding, disabled-source reads, legacy/configuration-only mapping handling, retained negation NULL semantics and alias filter/routing bypass. Preserve eligible doc-value-only queries and reject native-query combinations requiring local evaluation during planning. ### Check List (For Author) - Test: - [x] ES connector reactor package: 163 reported tests, 162 passed, one optional ES 9 compatibility class skipped; zero failures/errors. All nine reactor modules passed FE Checkstyle. - [x] Standard `build.sh --be` ASAN build and all 28 `EsScrollParserTest` unit tests passed. The combined parser/reader run reported 47 tests: 46 passed and one optional environment-dependent live smoke test skipped; the separate live regression run exercised both readers. Clang-format 16, scoped clang-tidy and build hygiene passed for all affected C++ files. The local BE ELF requires no GLIBC version above 2.10. - [x] Reproduced the reported alias/script, ARRAY, date-only, disabled-source, boolean, root-disabled, integer and retained-NOT failures on the previous head with live ES 8.11.3. FE/BE unit tests also demonstrated failures before the relevant fixes. Legacy ES6 type compatibility is covered by mapping-response unit tests. - [x] Seven live ES 8.11.3 suites passed 1,860 stored checks and 1,764 independent SQL comparisons, preserving 88 expected native planning errors. All 1,780 prior stored checks remain unchanged. - [x] `test_es_source_semantics`: 612 stored / 516 independent checks; `test_es_numeric_source_semantics`: 264 / 264; `test_es_mapping_boundaries`: 272 / 272; `test_es_source_fallback_types`: 304 / 304; `test_es_scalar_materialization`: 160 / 160; `test_es_source_value_boundaries`: 168 / 168. - [x] New source-boundary coverage reproduces all six review findings: JSONB singleton/empty/null-element containers, numeric-double INT64_MIN and upper endpoint, nullable declared ARRAYs, numeric empty strings, Java numeric grammar, direct FLOAT rounding, and inherited malformed/coercion settings with field overrides and mixed aliases. All eight reader/docvalue/parallel combinations passed. New BE and FE controls failed before their corresponding fixes. - [x] `test_es_epoch_millis` adds 80 stored checks and 80 independent comparisons across both readers, docvalue/parallel modes and UTC/+08:00. It covers normalized-sibling source fallback, zero/short/fractional epoch strings, numeric doubles, negative instants, NULL/empty/singleton arrays, local predicates and pure-docvalue controls. All outputs were generated by the repository runner and verified without generation; all 80 sections also match an independent date/predicate computation. New unit coverage checks legacy/V2 date types, declared arrays, calendar-year precedence and invalid numeric/civil-year bounds. - [x] New scalar-materialization coverage includes floating/Boolean/date singleton, empty and null-only arrays; narrow integer rounding boundaries and NaN, 64-bit precision controls, declared integer ARRAYs and retained range NOT on multi-value numeric docvalues. New coverage spans both readers, docvalue modes and parallel modes, source fallback plus pure-docvalue controls, disabled source indices/mixed aliases, populated disabled roots, non-UTC date-only values and retained NOT with optimizer simplification disabled. Session settings are restored. New outputs were generated by the repository runner and verified separately without generation; the final date control avoids dependence on the FE JVM timezone. - [x] Original FE package built with `build.sh --fe`; connector updates packaged through the documented module reactor recipe and deployed with the rebuilt BE for live validation. - Behavior changed: - [x] Yes: correctness fallbacks, supported mapping/decoding/query handling and native-only planning restrictions above. - Does this need documentation? - [x] No new public SQL syntax or configuration.	2026-11-03 16:25:39 +08:00
wudi	c8a75b273e	[fix](fe) Require ADMIN for remote CDC driver URLs (#69617) ... ### What problem does this PR solve? Issue Number: N/A Problem Summary: Users with permission to create CDC jobs can supply HTTP/HTTPS JDBC driver URLs that load code into the FE JVM. Require global ADMIN privilege when these URLs are supplied during CDC job creation, alteration, or `cdc_stream` construction, before source initialization or driver loading. Share the permission check across the FROM-TO and TVF paths. Preserve existing local driver permissions and S3 streaming behavior.	2026-11-03 16:10:48 +08:00
linrrarity	7acfd9bfc2	[improvement](be) Reduce transitive dependencies in shared headers (#69717) ... Related PR: https://github.com/apache/doris/issues/66901 ### What problem does this PR solve? Several shared BE headers transitively include protobuf, Thrift, thread-pool, and storage-reader implementations. Translation units that only need declarations must still parse these definitions, increasing compilation work and the rebuild scope when those headers change. Recent CI builds failed during CMake configuration because the reverse-reach counts for threadpool.h and rle_encoding.h exceeded the build-hygiene limits. This PR audits all 32 header-budget entries and reduces unnecessary dependencies without changing the budgets or exception lists. The changes: - Extract `ColumnIteratorOptions` into a dedicated header so `vexpr_context.h` does not require the full `column_reader.h` dependency chain. - Use forward declarations for `thread-pool`, `protobuf`, `cache`, and `WorkloadGroup `types where complete definitions are unnecessary. - Move `UniqueId` conversions, `protobuf` methods in numeric `SerDe`, and `selected construction/destruction` operations into `.cpp` files. - Extract `SyncSizeCallback` into a dedicated header to reduce RPC dependencies propagated through the runtime-filter interface. - Add direct includes at actual use sites and verify independent compilation. Object layouts, move semantics, explicit template instantiations, and per-value decoding implementations are preserved. Moving some definitions into .cpp files changes their visibility for inlining in ordinary separate compilation. Query runtime performance has not been measured in this PR. ### Dependency reductions The following counts come from Ninja dependencies after completed builds, covering 1,531 production translation units and 52 generated translation units: | Header | TUs Before | TUs Now | New Limit | |---|---:|---:|---:| | util/rle_encoding.h | 846 | 231 | 250 | | util/threadpool.h | 1,056 | 585 | 600 | | gen_cpp/types.pb.h | 1,862 | 1,225 | 1,250 | | gen_cpp/segment_v2.pb.h | 1,231 | 1,156 | 1,200 | | gen_cpp/internal_service.pb.h | 777 | 557 | 600 | | gen_cpp/FrontendService_types.h | 593 | 293 | 300 | | gen_cpp/cloud.pb.h | 325 | 30 | 50 | | gen_cpp/BackendService_types.h | 195 | 115 | 150 | | gen_cpp/data.pb.h | 228 | 90 | 100 | | storage/options.h | 502 | 293 | 350 | | io/fs/s3_file_system.h | 65 | 31 | 50 | | gen_cpp/Types_types.h | 2,078 | 2,073 | 2,100 | | gen_cpp/PaloInternalService_types.h | 1,093 | 1,093 | 1,100 | | util/brpc_closure.h | 62 | 35 | 50 | | gen_cpp/Status_types.h | 37 | 37 | 41 | | runtime/workload_group/workload_group.h | 43 | 38 | 47 | Across the 16 monitored heavy headers, measured header-to-TU dependency pairs decrease from 8,108 to 6,617, an 18.39% reduction. These removals affect 742 distinct translation units. The number of production compilation units remains unchanged. The transitive project-header closure of dependency.h decreases from 325 to 246 headers, a 24.31% reduction. This uses the hygiene script’s static include graph and is reported separately from the measured Ninja dependency counts. ### Query runtime performance ### TPC-H SF1000 | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 59.615 | -9.93% | | master 2026-10-13 | 60.776 | -11.65% | | master 2026-10-15 | 57.021 | -5.83% | | master 2026-10-20 | 55.877 | -3.90% | | master 2026-10-27 | 65.987 | -18.62% | | PR #69717 | 53.697 | — | ### SSB SF1000 | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 11.418 | +3.71% | | master 2026-10-13 | 11.749 | +0.79% | | master 2026-10-15 | 10.653 | +11.16% | | master 2026-10-20 | 10.773 | +9.92% | | master 2026-10-27 | 11.471 | +3.23% | | PR #69717 | 11.842 | — | ### ClickBench Configuration: cluster_type=single, ecs.c9i.4xlarge_16c32g, one 500 GB disk, full analyze, PR #69717 release package. | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 24.717 | -1.83% | | master 2026-10-13 | 26.406 | -8.11% | | master 2026-10-15 | 25.079 | -3.25% | | master 2026-10-20 | 24.629 | -1.48% | | master 2026-10-27 | 23.643 | +2.63% | | PR #69717 single rerun | 24.265 | — | ### TPC-DS SF1000 (1 TB) Comparisons use the PR result from 2026-10-29. The 2026-10-28 rerun is listed separately. | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 168.926 | -7.40% | | master 2026-10-13 | 170.676 | -8.35% | | master 2026-10-15 | 163.170 | -4.13% | | master 2026-10-20 | 163.926 | -4.58% | | master 2026-10-27 | 195.853 | -20.13% | | PR #69717 2026-10-29 main result | 156.423 | — | ### Release note None ### Check List (For Author) - Test: Three complete Release builds; standard ASAN BE/FE build and final incremental build; all BE UT sources compiled and linked; 29 selected unit tests passed with -j50 --coverage; 1,024-row service read, aggregate, shuffle join, sort and cast checks; header hygiene and changed-line format checks passed. Clang-tidy covered all 71 files and retained two existing warnings in relocated protobuf methods (function size and signedness). - Behavior changed: No - Does this need documentation: No	2026-11-03 15:08:06 +08:00
Jamie	cb0b56d408	[fix](cloud) Fix deadlock between tablet encryption check and meta lock writers (#69950) ... ### What problem does this PR solve? Issue Number: None Related PR: #64574 Problem Summary: `is_tablet_encrypted()`, which serves `/api/check_tablet_encryption`, holds the tablet `_meta_lock` read lock in `traverse_rowsets()` and takes it again through `tablet_schema()` in the visitor. Since #64574 `_meta_lock` is the writer-preferring `BthreadSharedMutex`, so a writer that arrives between the two reads deadlocks with the check: - The check holds the first read lock and waits for the second, which is blocked by the pending writer. - The writer waits for the first read lock to be released. The window is wide because each rowset's files are read from remote storage under the lock, and compaction and rowset sync take the write lock often. Once this happens the tablet stays stuck until the BE restarts: - The HTTP request never returns. - The compaction keeps renewing its lease but never finishes, so other BEs get `JOB_TABLET_BUSY` for this tablet. - Every later reader of this tablet's meta blocks behind the pending writer, including the periodic tablet sync. We hit this in a regression test: the compaction stopped right after its rowset sync, and the check timed out on all 10 retries over about 2h47m. This PR reads the schema once before the check and checks a snapshot of the rowsets (`get_snapshot_rowset()`) with no lock held, which also stops holding the meta lock across remote file reads. It adds a test-only sync point so a unit test can observe the lock at that moment. ### Release note Fix a deadlock between the tablet encryption check API and tablet meta writers such as compaction, which could block a tablet until the BE restarts. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> Unit test `CheckEncryptionActionTest.RowsetsAreCheckedWithoutHoldingMetaLock` checks that `_meta_lock` is free while a rowset is being checked. It fails on the code before this PR and passes with it. Manual test: a temporary unit test (not committed) started a writer from inside the rowset check, then waited for the check to return. On the code before this PR the check had not returned after 20s. With this PR it returned at once. - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214	2026-11-03 14:41:49 +08:00
daidai	61906ce9a1	[fix](be) Preserve signed decimal bounds in Iceberg ORC writes (#69888) ... ### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: ORC `Int128` statistics already contain two's-complement bits, but the Iceberg ORC writer negates negative values again before encoding them. A file containing `[-100, -1]` consequently reports bounds `[100, 1]`, so a negative predicate can prune a file that contains matching rows. ORC footer statistics also trim trailing fractional zeros. For example, `-1.00` is returned as `value=-1, scale=0`; a `DECIMAL(...,2)` manifest needs the unscaled value `-100`. Preserving the sign alone therefore leaves incorrect bounds. Restore each bound to the declared column scale with the existing ORC scaling utility, then serialize its original signed bits using minimal big-endian encoding. Preserve necessary sign prefixes such as `128 -> 00 80` and `-129 -> FF 7F`. The fix applies to newly collected bounds. Existing incorrect metrics require recomputation or data-file rewriting. ### Release note Correct decimal bounds emitted by Iceberg ORC writes to prevent incorrect predicate pruning for negative values and values with trailing fractional zeros. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Added BE tests cover 29 encoding cases, real ORC statistics at precisions 1/9/18/38, scale restoration through scale 38, nullable columns, and all-null columns. The external regression checks manifest pruning for negative and mixed decimal data across decimal32/64/128. Local validation: - An isolated codec harness built from the production function passed all 29 encoding cases. - Five real ORC files passed independent Iceberg 1.12.0 manifest write/read and predicate-pruning probes. - clang-format 16, build hygiene, changed-line clang-tidy, and Groovy 4.0.19 syntax checks passed. Static analysis used real ASAN UT compilation commands with PCH expanded as headers. - The full local BE-UT build was stopped during the monolithic dependency build before tests executed. - Regenerated `test_iceberg_write_stats2.out` with the standard regression runner after restoring decimal scales. Only `sql_4/5/7/9` changed, and the decimal bounds match both stored rows and the Parquet metrics. - Re-ran `test_iceberg_write_stats2` and `test_iceberg_orc_decimal_bounds` locally using this PR's CI FE/BE artifacts: 2 suites passed, 0 failed, 0 skipped. - Behavior changed: - [ ] No. - [x] Yes. Newly written ORC decimal bounds preserve the sign and the column scale. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label	2026-11-03 14:39:46 +08:00
wudi	deb079478d	[fix](fe) Report missing backend during stream load planning (#69989) ... ### What problem does this PR solve? Issue Number: None Problem Summary: A cloud Stream Load request can still arrive from a BE after scaling has removed that BE from FE metadata. Resolving the compute group then obtains a null backend and throws a NullPointerException, which reaches the caller as a generic INTERNAL_ERROR. Return a descriptive UserException containing the missing backend ID instead. The existing RPC exception handler converts it to ANALYSIS_ERROR. The request fails before plan generation, allowing the existing CDC task failure and automatic rescheduling flow to handle recovery. Add an RPC-level test for a missing backend with token authentication and a compute-group header. It checks the returned status and backend ID, absence of an execution plan, and ConnectContext cleanup.	2026-11-03 14:23:43 +08:00
lihangyu	a37fc36a0f	[opt](variant) Walk only non-null materialized slots per row in VariantAssembler (#69914)	2026-11-03 12:31:52 +08:00
minghong	156dbe71dc	[fix](MV) Accept constant query outputs when matching a materialized view (#69787) ... ### What problem does this PR solve? Issue Number: None Problem Summary: The materialized view rewrite keeps a query output which is a **constant** as it is, because the view does not have to output that column, but it only recognized a bare `Literal`. A constant alias such as `1 AS o_orderkey#97` (the shape constant propagation materializes above an aggregate) or a constant function such as `bin(1)` did not reach that path, so the rewrite looked for a counterpart in the view, did not find it, and rejected the candidate. The explain then reports `... not chose` and the query does not use the view even though the columns are pinned by predicates. Accept every constant expression there (`isConstant()` covers `Literal` and constant aliases/functions) so a constant query output needs no counterpart in the view. Aggregate functions over literals (`count(1)`) are not constant by definition and keep the previous behavior. ### Release note None ### Check List (For Author) - Test: - Unit Test: the materialized view unit tests (`MappingTest`, `MaterializedViewUtilsTest`, `MvExplorationSuiteTest`, `AsyncMaterializationContextTest`, `PartitionColumnTraceTest`) pass - Regression test: the whole `nereids_rules_p0/mv` directory. `Explain and check failed` does not appear: no materialized view rewrite case regressed. 10 suites of that directory fail in this local environment for unrelated reasons: 5 stream load failures (empty response body), 4 external table suites which cannot reach a Hive metastore, and 1 outfile suite without S3 credentials. - Behavior changed: No - Does this need documentation: No	2026-11-03 11:53:28 +08:00
Jerry Hu	4088359429	[fix](be) Reset quantile union state for empty window frames (#69964) ... ### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: A bounded ROWS window can become empty after a nonempty frame. The `quantile_union` reset previously restored only the first-input flag and retained the preceding `QuantileState`. Without a subsequent value to merge, an empty or NULL-only frame returned the preceding frame's quantiles. Reset both the first-input flag and the stored samples before each frame. The logical reset retains explicit sample-vector capacity for reuse, while `clear()` preserves its storage-release behavior. ### Release note Fix `quantile_union` over empty window frames: `quantile_percent` now returns NaN instead of values from the preceding frame. ### Check List (For Author) - Test: ASAN BE unit tests (47 passed); ASAN BE+FE build (including FE Checkstyle); three isolated SQL regression suites (1 suite passed each); clang-format, format check, build hygiene, changed-line clang-tidy (no warnings), and diff check. No performance benchmark was run. - Behavior changed: Yes (empty window frames no longer inherit the preceding quantile state). - Does this need documentation: No.	2026-11-03 11:46:56 +08:00
daidai	952ed6904f	[fix](be) Preserve Unicode across JNI string conversions (#69889) ... ### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: JNI `NewStringUTF` and `GetStringUTFChars` use modified UTF-8, while native scanner/writer parameters and paths use standard UTF-8. Passing a SQL predicate containing an emoji to `NewStringUTF` can corrupt or truncate it before the JDBC driver receives it. Returned maps also encode supplementary characters differently and cannot preserve a standard UTF-8 round trip. - Add validated standard UTF-8 ↔ UTF-16 conversion helpers and use them for both directions of parameter/statistics maps. Preserve supplementary characters and embedded NULs; reject malformed UTF-8 and unpaired UTF-16 surrogates with an input offset. - Migrate Paimon writer text, column/directory arrays, spill-path callbacks, and exception messages to these helpers. Reuse one array builder and preserve native spill-path matching and accounting. - Use standard UTF-8 for dynamic UDF cache signatures and plugin status JSON as well. - Use native temporary conversion buffers that do not require a Doris `ThreadContext`, and return native allocation failures as `Status`. Regressions exercise JNI-attached threads without a Doris context, including spill callbacks and accounting. - Fix the existing `String<Global>::new_string` template instantiation and keep the legacy modified UTF-8 APIs' encoding contract. ### Release note Fix Unicode corruption in JNI parameters, returned maps, and Paimon metadata and spill paths. Invalid text encodings now fail explicitly. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Validation: Focused ASAN_UT build and execution via `./run-be-ut.sh --run --filter='JniUtf8DeathTest.*:JniUtf8Test.*:JniPaimonUtf8DeathTest.*:JniPaimonUtf8Test.*:JniPaimonWriteBackendTest.*:PaimonWriteBackendFactoryTest.*:PaimonWriteBackendTest.*:PaimonTableSinkOperatorTest.*:PaimonTableWriterTest.*' -j 6`: **23 passed, 0 failed, 0 skipped**. This includes 13 real-JVM subprocess tests, 3 JNI allocation-failure tests, and 7 existing Paimon backend tests. The local test source list was temporarily narrowed; production libraries and ASAN configuration were retained, and the CMake selection was restored afterward. Full BE/regression suites, complete Paimon writes, and the customer-specific JDBC environment were not run. Style/static checks: clang-format 16, build hygiene, and clang-tidy checks passed. - Behavior changed: - [ ] No. - [x] Yes. JNI text boundaries preserve standard UTF-8; malformed input is rejected instead of being silently corrupted. Existing modified UTF-8 APIs retain their contract. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label	2026-11-03 11:27:09 +08:00
hui lai	31a036fbcd	[fix](load) Avoid warning spam when filtering non-local adaptive bucket tablets (#69979) ... ### What problem does this PR solve? During adaptive random bucket loading, `VNodeChannel::_set_adaptive_random_bucket_open_request()` normally skips selected tablets that do not belong to the current node channel. Logging a WARNING for every skipped tablet can flood both INFO and WARNING logs when many partitions and parallel writers are involved, increasing disk I/O and evicting useful diagnostic history. Change this expected filtering message to `VLOG_DEBUG`, which is disabled by default. Keep the existing diagnostic fields, tablet selection logic, and warnings for unknown partitions, unknown indexes, and invalid bucket sequences unchanged. ### Release note Avoid excessive BE INFO/WARNING logging when adaptive random bucket loading skips non-local tablets.	2026-11-03 10:48:18 +08:00
Gabriel	156bae0793	[fix](be) Read dictionary-encoded Parquet BOOLEAN columns (#69967) ... ### What problem does this PR solve? Issue Number: N/A Problem Summary: The native Parquet reader rejects every BOOLEAN dictionary page, although PLAIN dictionaries with RLE_DICTIONARY or legacy PLAIN_DICTIONARY data pages are valid. Reading a nullable BOOLEAN interoperability fixture fails with `Boolean type cannot have a dictionary page`. Create the existing fixed-length dictionary decoder for BOOLEAN and expand its LSB-first packed entries into tracked, contiguous one-byte values for SerDe. Validate the packed extent before allocation and preserve the common dictionary-index checks for reads, selections, and skips. The 257-byte interoperability fixture and synthetic tests cover byte boundaries, invalid extents, dictionary replacement, both dictionary encoding enums, V1/V2 pages, skipped/filtered values, and dictionary-to-PLAIN transitions. ### Release note Support reading dictionary-encoded BOOLEAN columns from Parquet files. ### Check List (For Author) - Test - [x] Regression test: S3 TVF reads compare against SQL literal rows, covering predicates, aggregates, NULL alignment, and lazy materialization on/off. Groovy compilation and a suite control-flow check using the real FE variable manager pass, including settings restoration after success or an injected failure; full SQL execution is pending CI. - [x] Unit Test: ASAN BE tests; all 199 tests in `ParquetV2NativeDecoderTest.*:NewParquetReaderTest.*` pass. The five new tests fail before the fix and pass after it. - [ ] Manual test - Behavior changed: - [x] Yes. Valid BOOLEAN dictionaries are readable instead of being rejected. - Does this need documentation? - [x] No. Compatibility fix for existing Parquet reads.	2026-11-03 10:36:46 +08:00
Gabriel	8ebf20dc3a	[feature](fe) Support MongoDB ObjectId and Trino JSON types (#69969) ... ### What problem does this PR solve? Issue Number: N/A MongoDB exposes ObjectId as a plugin-defined Trino type. Doris did not register these types in either plugin loader, rejected ObjectId during schema mapping, and discarded the connector's default hidden `_id` column. Register plugin-defined types in FE and BE, map ObjectId recursively to `VARBINARY(12)`, and expose the default MongoDB `_id` while retaining the hiding of other columns. Existing byte readers preserve all twelve bytes without UTF-8 conversion. Trino JSON columns also lacked a Doris type mapping. Map the built-in Trino JSON type recursively to native Doris JSON, and parse its UTF-8 JNI carrier into JSONB. Preserve SQL NULL independently of the JSON value `null`, including empty and repeated batches. Reuse the existing JSON parser and return malformed-input errors. Correct nested JSON text serialization so arrays, maps and structs contain JSON values with their original quoting; previously an object could become invalid text such as `["{"a":1}"]`. Coordinate typed ARRAY/MAP/STRUCT text parsing with that formatter, retaining JSON string quotes and using uppercase `NULL` for SQL-null JSON children versus lowercase JSON `null`. Both strict and non-strict casts retain the original JSON type and null semantics. Delegate complete child tokens to `from_string_as_nested`: scalar SerDes remove outer quotes, JSONB retains JSON quotes and accepts legacy SQL apostrophe wrappers, and Nullable handles SQL nulls and conversion errors using the child's paired null-token policy. Generic complex parsing no longer queries the concrete JSONB type. Require the fixture's final network-facing MongoDB process before marking it ready, and use bounded tmpfs mounts for both image-declared data directories to avoid orphaned anonymous volumes. Add an opt-in MongoDB 8.0.20 Docker fixture and five regression suites with **138 result checks and ten expected-error checks**. Coverage includes primitive and nested types, BSON documents and Extended JSON, all JSON value shapes, null and missing fields, numeric/date boundaries, Unicode and escapes, schema discovery/permissions, scalar and JSON predicates, joins, aggregates, subqueries, windows, views, internal-table joins, scalar JSON materialization, multi-page reads and repeated early termination. Golden files are generated and compared with the standard regression runner. Source-side MongoDB queries independently confirm critical counts and null semantics. MongoDB remains an operator-installed Trino 435 plugin; no MongoDB runtime dependency is added to Doris. Use the complete matching plugin ZIP on FE and BE. ObjectId retains existing VARBINARY operator/storage restrictions; use `from_binary(_id)` for hexadecimal filtering, joining, grouping or materialization. Nested JSON is readable through catalogs, while internal-table creation retains existing complex-type restrictions, including rejecting `ARRAY<JSON>`. BSON documents inferred automatically remain ROW/STRUCT; use an explicit `json` schema declaration to read a field as a whole JSON value. ### Validation - FE package and native ASAN BE built through `build.sh --fe` / `build.sh --be`. UI, Hive UDF, OBS/COS dependencies and unrelated Java plugins were excluded locally; the native build covers the initial connector and JSON changes. The final SerDe interface refactor was compiled and linked through the ASAN BE unit-test build; the standalone BE package and external integration suites were not rerun for that refactor. Final executable passed the GLIBC_2.17 compatibility gate. - `run-be-ut.sh --run`: **33 related BE unit tests passed**, including existing JSON serde, general complex-type parsing and formatting tests. Typed JSON text covers 66 combinations of container, value shape and strictness, plus 48 legacy apostrophe-wrapper combinations, deeper nesting and malformed children in both modes. Reverting JNI support reproduces three failures; the typed complex JSON test fails before the reader fix and passes after it. - `run-regression-test.sh --run -d external_table_p0/trino_connector/mongodb`: **five suites passed**, with **138 executed result checks and ten expected errors**, real MongoDB and the complete Trino 435 plugin ZIP. - Checkstyle/validate, clang-format 16 for all affected C++ files, header hygiene and clang-tidy passed. - **17 related regression suites passed** across the final comparison and an Elasticsearch rerun after adjusting only local fixture CPU/direct-memory limits. This includes the array-function documentation, JSON/array/aggregate coverage, all four Elasticsearch fixture versions, spill-enabled bucket-shuffle plans, HTTP TVF and FE/BE version metrics. - The generated-version test verifies refreshed numbers without recompiling a cached consumer. Warmed FE builds verify changed and restored release metadata after cache hits; bytecode inspection confirms numeric fields are read at runtime. - Gated MongoDB startup reproduces and fixes the initialization-server readiness race; repeated start/stop cycles leave no anonymous data volumes. - Independent MongoDB queries confirmed source-side null semantics and multi-page counts/sums. Refresh the affected array-function documentation and Elasticsearch golden results through the regression runner. Keep the bucket-shuffle regression focused on physical INTERSECT/EXCEPT nodes when spill is enabled by disabling their two spill rewrite rules. Pin the HTTP TVF INT96 fixture timezone to UTC to preserve its stored wall-clock values and existing golden results. Refresh the FE build-version class after Maven cache restoration so packaged FE/BE version metrics report the current revision. Keep numeric version fields runtime-initialized so cached consumers observe refreshed metadata. A standalone generator test covers replacing Version.class without recompiling its consumer and invokes the actual packaging function with a PATH-only JDK to verify both tool selection and deployed-jar metadata. The fixture README documents setup, JSON schema behavior, typed SQL text null spelling, coverage and current limitations. MongoDB integration suites are disabled by default and require the matching plugin on FE and BE. ### Release note Support MongoDB ObjectId columns, including the default `_id` and nested fields, as `VARBINARY(12)` through Trino connector catalogs. Support Trino JSON columns as native Doris JSON, including nested JSON fields, and preserve JSON values and SQL-null distinctions through typed complex-text round trips. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Manual test: real MongoDB fixture, complete connector ZIP, exact bytes and independent source-side JSON checks - Behavior changed: - [x] Yes. ObjectId and JSON fields become queryable with their native Doris types, the default hidden MongoDB `_id` is exposed, and nested JSON values retain their own quotes and null semantics. - Does this need documentation? - [x] Yes. Setup, JSON schema selection and limitations are documented in `docker/thirdparties/docker-compose/mongodb/README.md`. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label	2026-11-03 10:27:59 +08:00
daidai	3763f09897	[fix](fe) Skip table snapshot resolution for external views (#69836) ... ### What problem does this PR solve? External views currently resolve a table handle when loading the statement snapshot, before their SQL is expanded. Skip that table-only path for known views; their base tables retain normal snapshot pinning. Reject unsupported view scan parameters such as `@branch` and `@tag` instead of silently ignoring the requested selector. ### Release note Avoid table metadata requests for external views during snapshot binding, and reject unsupported view scan parameters. ### Check List (For Author) For this update: Checkstyle passed for both changed Java files. Selector-rejection unit tests were added but not run locally. - Test - [x] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [x] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label	2026-11-03 10:26:41 +08:00
zhaorongshengandzhaorongsheng	bdb90856e6	[improvement](hive) Push partition filters to HMS (#67725) ... ## What problem does this PR solve? Push eligible Hive partition equality/IN predicates into bounded HMS filtering without eagerly materializing every partition. Preserve conservative fallback for unsupported predicates, types, identifiers and unavailable views. This update addresses the current review findings: - Reuse filtered handles without losing stronger backend residual predicates. - Prepare complete partition views for MV creation and eligible refresh/rewrite consumers. - Distinguish unavailable coverage from native/all coverage during MV compensation. - Fence ROOT/ADMIN selective subsets against schema and connector changes. - Restrict filtered, unfiltered and empty refresh reads to their materialized pin. - Preserve connector-retained schema through deferred pin hydration. - Keep INSERT preloading outside the outer target lock; retry schema or target replacement before binding with refreshed table caches. - Avoid pre-binding constant folding of unbound CAST predicates. - Handle reserved Hive partition keys `in` and `between` through local fallback. - Display unknown full partition totals as `?` without enumerating solely for EXPLAIN. - Resolve the latest target integration conflict by preserving both partition-pruning and write-only-default capabilities and advancing the connector API to 28.0. ## Release note Improve selective Hive partition planning while preserving schema, snapshot and materialized-view consistency. Selective EXPLAIN output may show an unknown full partition total as `?`. ## Validation - Published head: `ae2f21cbb30469408970cfb81709d8bbe27a8998`. - Target snapshot integrated without rewriting published history: `0976b2cd5020607205969bf7986dcb551b6a7325`. - Before the final target integration, canonical `./run-fe-ut.sh --coverage --run ...` runs verify 39 selected classes: 945 tests, zero failures/errors, one existing disabled test. - After the final target-replacement correction, a fresh targeted canonical run passes 33 tests, zero failures/errors/skips, and 89 actual zero-violation Maven Checkstyle executions. - After the final target integration, fresh canonical reports verify 36 classes and 642 tests, zero failures/errors/skips. This includes SPI's own full suite, Hive/Iceberg, view/MVCC, planning and partition compensation, INSERT/defaults, MTMV task fixtures and SQL-cache version tests. - The first integration run passes 641 tests but exits 1 when the final mock-FE class hits an occupied ephemeral edit-log port before executing its test. The isolated single-method rerun passes the remaining test, exits 0 with `BUILD SUCCESS`, and performs 89 actual zero-violation Checkstyle executions. These results are aggregated from fresh reports, not a claim that the all-class reactor succeeded in one attempt. - Coverage includes connector/HMS/SPI, MVCC schema and partition views, sync/stream/batch handoffs, MV compensation, policy/replan boundaries, INSERT lock windows and real target replacement in both execution and planning-only modes. Previously failing CAST/shape classes are included. - Connector-import, metadata-funnel, build-hygiene and authored whitespace checks pass. - Local mock-FE port conflicts required targeted reruns. A temporary test-only non-ephemeral port selection was used and restored before committing. - Standard local `./build.sh --fe` packaging remains blocked before Maven by missing genuine native thirdparty prerequisites. No successful full packaging, live HMS/BE regression or complete refresh-scheduler run is claimed. - Extended review rounds were stopped at the author's request. All independently confirmed remaining findings are repaired and tested; no pipeline PASS receipt or new remote `/review` request is issued by this update. --------- Co-authored-by: zhaorongsheng <zhaorongsheng@corp.netease.com>	2026-11-03 10:03:20 +08:00
Jack	7cca01bc6c	[fix](be) Score MATCH on keyword inverted indexes like SEARCH (#69790) ... ### What problem does this PR solve? Issue Number: None Related PR: #69051 Problem Summary: score() of a MATCH on an untokenized (keyword) inverted index is always 0, while SEARCH on the same field scores BM25: INDEX idx_redirect (redirect) USING INVERTED -- one row ('BP') SELECT score() FROM t WHERE redirect MATCH 'BP'; -- 0 SELECT score() FROM t WHERE SEARCH('redirect:BP'); -- 0.2876821 Elasticsearch scores a match, term or match_phrase query on a keyword field with BM25 in query context: no norms and no frequencies, so freq = dl = avgdl = 1 and the score is the idf of the whole value (0.2876821 above, 0.6931471 for N=4, n=2). Only a bool filter scores 0. SEARCH already gives exactly these values. MATCH on keyword behaved like a filter, so `body MATCH 'apple' AND tag MATCH 'x'` ranked as if the tag clause were not there. Root cause: #69051 made the engine score keyword postings (unit frequency and norm) but kept MATCH out on purpose: can_score_match required an analyzed index, and MatchPredicateCollector returned early for indexes without an analyzer, so no statistics were collected for them. Fix: a keyword (non-analyzed) index is eligible for MATCH scoring regardless of support_phrase, and MatchPredicateCollector collects its whole value as one term (case and spaces kept), the same term the reader looks up and the same as SearchPredicateCollector collects. Analyzed indexes keep the existing support_phrase / lowercase-normalizer rule. Equality and IN stay pure filters. A keyword MATCH_PHRASE_PREFIX scores the constant 1 of a single-term prefix; MATCH_REGEXP is unchanged.	2026-11-03 09:05:21 +08:00
Mryange	4bc7f42682	[fix](be) Fix TIMESTAMPTZ schema change defaults (#69611) ... ### What problem does this PR solve? Issue Number: N/A Problem Summary: Adding a TIMESTAMPTZ column with DEFAULT CURRENT_TIMESTAMP could finish schema change but leave historical rows unreadable. Root cause: the backfill default was materialized without timezone information, and the batch ADD COLUMN and Protobuf paths did not consistently use the materialized value. Materialize TIMESTAMPTZ backfill defaults as UTC literals, use the schema-change conversion only for TIMESTAMPTZ/TIMESTAMP_NS in batch ADD COLUMN, preserving other types' existing conversion, and preserve the default expression in both Protobuf and Thrift. Partial updates evaluate the preserved expression for new rows instead of reusing the fixed backfill timestamp. ### Release note Fix unreadable historical TIMESTAMPTZ values after adding a column with DEFAULT CURRENT_TIMESTAMP. ### Check List (For Author) - Test: - [x] Regression test: test_timestamptz_default_schema_change passed with -forceGenOut and again normally (1 suite, 0 failures each), including TIMESTAMP_NS backfill and TIMESTAMP_NS/DATETIME partial-insert checks. - [ ] Unit Test: FE coverage added and compiled, not executed separately. - BE ASAN build passed for the BE changes. FE rebuilt successfully after the follow-up. `git diff --check` passed. C++ formatting checks passed. Clang-tidy reported existing diagnostics outside modified lines; the full scan was not clean. - Behavior changed: - [x] Yes. Historical rows receive readable TIMESTAMPTZ defaults; subsequent writes retain dynamic CURRENT_TIMESTAMP evaluation. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label	2026-11-02 20:01:02 +08:00
yujun	c9b73d2947	[fix](ivm) Plan EXPLAIN REFRESH COMPLETE with the read mode a refresh reads (#69851) ... `EXPLAIN REFRESH MATERIALIZED VIEW mv COMPLETE` for an IVM materialized view did not show the plan the refresh actually runs. The explain path installed `IvmRewriteContext.fullExplain`, which carried no stream read mode, so `hasFullRefreshStreamScans()` was false and `IvmFullRefreshMTMV` never rewrote the base-table scans. The statement therefore printed the normalized MV query over ordinary base-table scans, while a real COMPLETE refresh reads its base tables through IVM table streams in RESET mode (the first batch of a whole-MV refresh, and every table it does not partition by). That rewrite reshapes the plan, so the explained plan and the executed plan differed in exactly the part EXPLAIN is used to debug. Problem Summary: - Root cause: `IvmRewriteContext.fullExplain` passed an empty reset partition map and an empty non-PCT read mode, so the full-refresh stream rewrite rule was gated off during EXPLAIN. - Fix: the explain context now carries `StreamReadMode.RESET`, and the full-refresh rewrite reads every stream-scanned base table — including the tables the MV partitions by — in that mode under EXPLAIN, over the partitions the plan selects. - A base table without a usable IVM stream now fails the statement instead of being explained as a plain scan. EXPLAIN plans the same read the refresh runs, and the refresh fails the same way (a real COMPLETE refresh reconciles the stream before planning; EXPLAIN must not create one). Notes: - A complete refresh is not one plan: it may be split into batches, and the later batches read non-PCT tables at the stream offset (SNAPSHOT) and read tables the MV partitions by as ordinary scans. One EXPLAIN statement stands for the refresh as a whole, so it shows the read the refresh starts with — RESET for every stream read — and no reset partition scope is resolved for the explain plan. - No execution state is touched. EXPLAIN never builds an insert transaction, so no stream offset advances; the only added work at plan time is reading stream metadata.	2026-11-02 19:51:23 +08:00
Calvin Kirs	0976b2cd50	[opt](sql-cache) Preserve SQL cache across unrelated partition writes (#69883) ... ### What problem does this PR solve? Issue Number: None Problem Summary: SQL cache currently invalidates an OLAP query whenever the table visible version changes. For example, a cached query reading historical partition p1 is discarded after an insert into unrelated partition p2, even though its result is unchanged. Record the partition versions actually assigned to the scan, including selected empty partitions removed during planning. When the table version advances, reuse the cache only if those dependencies and the schema/partition topology still match. Use the same stable dependency identity for the BE cache key. Keep table-wide validation for metadata aggregates, variable-dependent keys and materialized-view rewrites whose physical scans can hide source dependencies. The change is FE-only and preserves the existing BE cache protocol. Regression cases use only 3–5 source rows and cover unrelated/relevant writes, empty partitions and branches, self joins, partition DDL, partial MV rewrites and eliminated empty MVs.	2026-11-02 18:08:28 +08:00
shuke	38a38dda22	[fix](ci) Use GPT-6 Sol for review capacity fallback (#69988) ... ### What problem does this PR solve? Related PR: #69953 When GPT-6.1 Sol encounters five capacity failures, review recovery currently switches to GPT-6 Astra, which has continued to encounter capacity failures in observed runs. Change the capacity fallback to `gpt-6-sol` and update the existing test fixtures and assertions to exercise that target. The production change is one workflow argument. The primary model remains `gpt-6.1-sol` with `xhigh`; the five-failure threshold, seven total capacity attempts, 300-second backoff, same-session recovery, shared budget, ledger and round preservation, head-change checks, and submission verification are unchanged.	2026-11-02 17:54:33 +08:00
Mingyu Chen (Rayner)	7e6f6cb3a3	[fix](fe) Resolve DEFAULT(column) on Iceberg tables from the write schema (#69488) ... ### What problem does this PR solve? Issue Number: None Related PR: #66538, #57531, #65851 Problem Summary: **In short.** On an Iceberg table, `DEFAULT(column)` anywhere except `UPDATE ... SET` and `MERGE ... VALUES` reads the default from the read schema, which carries no Iceberg write-default. This PR resolves it against the write schema that the statement pinned for its target, as branch-4.1 and branch-4.2 do. **Background.** An Iceberg `write-default` applies to rows written after it is set. So it lives in the writer schema that a write statement pins (`IcebergWriteSchemaContext`), not in the columns of the read schema. #65851 resolves `DEFAULT(column)` against that pinned schema in `UPDATE ... SET`, `MERGE ... UPDATE SET` and `MERGE ... INSERT VALUES` (`ConnectorWriteSchemaUtils`). Every other `DEFAULT(column)` either fails at binding or reaches `RewriteDefaultExpression` (#57531), which reads the default from the bound column. branch-4.1 #66538, also in branch-4.2, resolves these against the pinned write target too. It rejects `DEFAULT(column)` on an Iceberg column in a statement that does not write the table. **The problem, and what it cost.** Reproduced on a local cluster with a `file://` format-version 3 table: `CREATE TABLE t (id INT, c INT)`, `ALTER TABLE t ADD COLUMN d INT DEFAULT 42`, one row `(0, 0)`, so `d = 42`. | Statement | Master today | After this PR (same as branch-4.2) | |---|---|---| | `INSERT INTO t (id, c, d) VALUES (1, 1, DEFAULT(d))` | `Unknown column 'd' in 'table list'` | writes 42 | | `INSERT INTO t (id, c, d) SELECT 2, 2, DEFAULT(d) FROM t WHERE id = 0` | writes NULL | writes 42 | | `INSERT INTO t (id, c, d) SELECT 3, 3, DEFAULT(d)` | `Unknown column 'd'` | writes 42 | | `UPDATE t SET c = 100 WHERE d = DEFAULT(d)` | updates nothing: `d = NULL` makes the plan an empty set | updates every row, `d = 42` is pushed down to the scan | | `MERGE INTO t USING (SELECT 0 AS id) s ON t.id = s.id AND t.d = DEFAULT(t.d) WHEN MATCHED THEN UPDATE SET c = 300` | matches nothing | updates row 0 | | `SELECT DEFAULT(d) FROM t` | NULL | `DEFAULT(d) must reference a column of the table that this statement writes` | The `UPDATE` and `MERGE` cases fail silently: the statement succeeds and changes nothing. **How this PR fixes it.** - fe-core cannot tell from a column whether its connector keeps defaults in the read schema. Hive does: an HMS default shows up in both schemas. Iceberg does not. The new connector capability `SUPPORTS_WRITE_ONLY_DEFAULTS` says which kind a table is. The Iceberg connector declares it. The Hive connector passes it on to iceberg-on-HMS tables, like the other per-table capabilities, but only through the two-argument `getTableSchema`. Since #69299, a statement that pins the table at its latest snapshot reads the schema from the snapshot overload, which reflects none of them, so on an iceberg-on-HMS table such a statement keeps master's results (for example, `UPDATE t SET c = 100 WHERE d = DEFAULT(d)` still changes no row). Reflecting the sibling's capabilities there changes planning for all of them and is left to a follow-up PR. - For a target that declares it, `InsertUtils.normalizePlan` resolves `DEFAULT(column)` in `INSERT ... VALUES` and in the select lists of `INSERT ... SELECT`, including the replacements of `SELECT * REPLACE(...)`, against the pinned write schema. It does this by name, before the source relations are bound, so the name always refers to a target column, as in branch-4.2. In MySQL, `DEFAULT(column)` in a `SELECT` refers to a source column instead. A derived table in `FROM` is a query of its own and is left alone: `DEFAULT(column)` in it still names a column of its own `FROM` clause. - `RewriteDefaultExpression` takes the default for a `DEFAULT(column)` bound to such a table from the pinned write column with the same field id. That covers the `WHERE` of `UPDATE` and the `ON` of `MERGE`. A MERGE resolves it whatever its WHEN branches, also when they all delete, as branch-4.2 does. If the statement pinned no write schema for that table, it fails. A `DELETE` pins the writer schema too, for its delete sink, but it writes no row values, so `RowLevelDmlCommand` marks its target and `DEFAULT(column)` on it fails with `DEFAULT(d) cannot be used in a DELETE: the defaults of this table apply only to the rows that a statement writes`, as in branch-4.2, whose DELETE installs no write schema context. Master reads NULL there, so `DELETE FROM t WHERE d = DEFAULT(d)` deletes nothing; resolved to the write-default it would delete every row with `d = 42`. - A resolved default is cast to the type of its writer column before the expression around it is bound, as `RewriteDefaultExpression` already does for a bound `DEFAULT(column)`. The parsed literal `-1` is a TINYINT, so without the cast `bit_count(DEFAULT(c))` on an INT column would count 8 bits instead of 32. The external `UPDATE`, `MERGE` and changelog paths use the same helper. `RewriteDefaultExpression` rejects a bound `DEFAULT(column)` whose writer column has another type than the read column the statement was bound with, which happens only when the table changed outside Doris after Doris cached it, and asks to refresh the table: the expressions around it were bound against the read type. - Hive, Paimon and internal tables do not declare the capability, so they behave as before. - The new enum constant changes the connector plugin API surface, so `connector.plugin.api.version` goes from 26.0 to 27.0, as for the other recent changes to that surface (#69299, #69588, #69298, #69683 and #69835 took 22.0 to 26.0). **Where `DEFAULT(column)` resolves on an Iceberg target (direct Iceberg catalogs).** | Statement | Position of `DEFAULT(column)` | Result | branch-4.2 | |---|---|---|---| | `INSERT` / `INSERT OVERWRITE` | `VALUES` rows | the target column's write-default, typed as the column | same | | | the INSERT's own select lists (with or without `FROM`, set operation branches, `SELECT * REPLACE`) | the target column's write-default, by name, before binding | same, except `* REPLACE`, which branch-4.2 binds to the source | | | `GROUP BY` / `ROLLUP` outputs, a derived table in `FROM`, `WITH` definitions, subqueries, `WHERE` / `ON` | bound like any `DEFAULT(column)` (last row) | same, except derived tables, which branch-4.2 resolves against the target | | | a `VALUES` branch of a set operation | fails with an unknown column, as on master | same | | | a qualifier other than the target (`DEFAULT(src.c)`, `DEFAULT(ctl.db.t.c)`) in the positions above | fails: `column in assignment list is invalid` | same | | `UPDATE` | `SET` / `WHERE` | write-default | same | | `MERGE` | `ON` and `WHEN ... AND` conditions, whatever the WHEN branches, also when they all delete | write-default | same | | | `UPDATE SET` / `INSERT VALUES` | write-default | same | | `DELETE` (also `DELETE ... USING`) | anywhere | fails: `cannot be used in a DELETE` | fails: its DELETE has no write schema context | | any | a bound `DEFAULT(x.c)` | `x` is the written table: its write-default by field id; an OLAP or Hive table: that column's default; another Iceberg table: fails with `must reference a column of the table that this statement writes` | same | A bound `DEFAULT(column)` whose writer column has another type than the read column fails and asks to refresh the table. Where `DEFAULT(column)` is bound rather than resolved by name (the third row), it keeps its meaning from before this PR and names a column of its own `FROM` clause, as on master and branch-4.2: for example in `INSERT INTO t SELECT id, DEFAULT(d) FROM src GROUP BY id, d` with an OLAP or Hive `src`, it takes the default of `src.d`. **Results.** - Local cluster, with the table above: the table shows what each statement did before and after. `EXPLAIN UPDATE t SET c = 5 WHERE d = DEFAULT(d)` goes from an empty set to a scan with `d = 42` pushed down to Iceberg. - Unit tests: 6 new tests in `InsertUtilsTest`, 6 in the new `RewriteDefaultExpressionTest` and 1 in `RowLevelDmlCommandTest`. They cover VALUES, and INSERT ... SELECT both from a staging relation whose own column is also `name` and without FROM: each resolves to the pinned default, and an unknown column fails. `DEFAULT(column)` inside a derived table, qualified or not and also under a UNION branch, is left to binding. A `SELECT * REPLACE(DEFAULT(name) AS name)` replacement resolves like the rest of the select list and keeps the star's qualifier and `EXCEPT` list. The pinned write column wins over the read column. A statement that writes nothing fails. A table without the capability is unchanged. With master's `InsertUtils` and `RewriteDefaultExpression`, the 5 tests of the fix fail and the 2 tests of the unchanged path pass. The `SELECT * REPLACE` test fails with this PR's previous `InsertUtils`, where the replacement stays `DEFAULT(name)`. `IcebergConnectorTest`, `HiveConnectorMetadataSiblingDelegationTest` and `ConnectorPluginSurfaceTest` pass. So do the fe-core tests around the write schema: `ConnectorWriteSchemaUtilsTest`, `ExternalRowLevelUpdatePlanBuilderTest`, `ExternalRowLevelMergePlanBuilderTest`, `ChangelogRowLevelDmlTransformTest` and `PluginDrivenExternalTableTest`. Two of the tests pin the typing: `bit_count(DEFAULT(bits))` gets the default as an INT in VALUES and in SELECT, and fails without the cast; a BIGINT writer column over an INT read column is rejected, directly and under an IF bound to INT. `DEFAULT(column)` on the target of a DELETE fails even with a pinned write-default, and only a DELETE marks its target (an UPDATE does not); skipping the check, or marking UPDATE instead of DELETE, makes exactly these two tests fail. `PositionDeleteRowLevelDmlTransformTest`, `PhysicalPlanTranslatorIcebergRowLevelDmlTest`, `DeleteFromCommandTest` and `UpdateCommandTest` pass. - Regression: `test_iceberg_initial_defaults` now writes rows 20 to 22 with `DEFAULT(default_int)` in VALUES and in INSERT ... SELECT with and without FROM. It then updates them with `WHERE default_int = DEFAULT(default_int)` and `MERGE ... ON ... = DEFAULT(...)`, for Parquet and ORC, and checks the rows in the `.out`. A plain `SELECT DEFAULT(default_int)` must fail, and so must a `DELETE ... WHERE default_int = DEFAULT(default_int)` on rows 20 to 22, which hold the write-default. The suite passed locally before the DELETE block was added, and the whole suite, DELETE block included, has since passed in two External regression runs on CI. With master's FE it fails at the first of these statements: `Unknown column 'default_int' in 'table list'`.	2026-11-02 17:47:01 +08:00
Mingyu Chen (Rayner)	fe58e58386	[feature](cloud) Support declaring compute_group on routine load and async MV (#69749) ... ### What problem does this PR solve? Issue Number: None Related PR: #67010 Problem Summary: **In short.** This ports branch-4.1 #67010 (also on branch-4.2) to master: CREATE / ALTER ROUTINE LOAD and async materialized views accept a `compute_group` property in cloud mode, so a background job can be pinned to a compute group instead of inheriting one implicitly, and the declared group is re-checked before every task. **Background.** In cloud mode every query runs in a compute group. A routine load job snapshots the compute group of the session that created it, and an async MV has no compute group of its own: an automatic refresh runs wherever `admin` resolves to, and a manual `REFRESH` borrows the group of the session that triggered it. **The problem, and what it cost.** On master nothing lets a user choose where a background job runs: | | master today | with this PR | |---|---|---| | Routine load | runs in whatever group the creating session was on; ALTER cannot move it | `"compute_group" = "cg_etl"` on CREATE or ALTER pins it | | Async MV, automatic refresh | runs where `admin` resolves to | runs in the declared group | | Async MV, manual `REFRESH` | runs in the triggering session's group, so the same MV refreshes in different places depending on who triggered it | runs in the declared group; an MV that declares none keeps borrowing the session's group | | Async MV, `REFRESH ... WITH DRY RUN` and `EXPLAIN REFRESH` | run and plan where `admin` resolves to | run and plan in the declared group; an MV that declares none resolves as today | | A pinned group is dropped, or the owner's USAGE on it is revoked | (no pinning) | the next task fails with that reason instead of a later "no available BE found" | | `compute_group` in non-cloud mode, or `DEFAULT` | (no property) | rejected at CREATE / ALTER | Four smaller defects go with it on master: CREATE ROUTINE LOAD checks the `workload_group` in the session's compute group namespace although the job runs elsewhere; `RoutineLoadJob.plan()` builds the sink's plan context from the raw snapshot field rather than the effective group; and the statement re-parsed when a routine load job is loaded from metadata resolves its `workload_group` again. On image load that lookup runs before any workload group exists (`PersistMetaModules` reads `routineLoadJobs` before `workloadGroups`, and until then the Env holds the empty `WorkloadGroupMgr` its constructor created), and a failure there sets the job to the final `CANCELLED`, so a job created with `workload_group` is cancelled by the first restart that loads it from an image (shown by a unit test with an empty manager, not reproduced on a cluster). And when the partition refresh fails, which `RoutineLoadJob.update()` runs before it takes the job lock, it pauses a NEED_SCHEDULE job without that lock, so a STOP that completes in between is turned back into PAUSED on the master while replay keeps STOPPED (Kafka metadata failures and Kinesis shard listing alike). **How this PR fixes it.** Same design as #67010: - The declaration is stored in the existing property maps (`RoutineLoadJob.jobProperties`, `MTMV.mvProperties`) and written only when declared: no new persisted field, journal type or meta version, and an absent key keeps today's implicit resolution, which is also how a later binding model tells a pinned group from an inherited one. `DEFAULT` is rejected because that model reserves it for "follow the owner's default group"; non-cloud mode rejects the property, and `ConnectContext.setCloudCluster()` is a no-op outside cloud mode. - `ComputeGroupBindingUtil` holds the two checks: `validateDeclaredComputeGroup` at CREATE / ALTER (USAGE for the current user, then existence, the order of `USE @<group>`), and `checkComputeGroupBeforeTask` before every task, against the identity the task runs as. A missing group keeps `INTERNAL_ERR`, so a routine load job is paused retryably and resumes by itself when the group comes back (a group scaled to zero backends leaves the cluster map and returns); a revoked USAGE uses `CANNOT_RESUME_ERR`, so `ScheduleRule` does not keep resuming it. - Routine load resolves the effective group in `RoutineLoadJob.getCloudCluster()`, which backend selection, the task sent to BE, the workload group namespace, the plan context and the `ComputeGroup` column of SHOW ROUTINE LOAD all use, so every Kafka job follows it (Kinesis routine load is rejected in cloud mode, the only mode with compute groups). `RoutineLoadTaskScheduler` re-checks the declared group before backend allocation and before `beginTxn`. The partition refresh of a Kafka job runs the same check before its metadata request, which goes through the group's backends and, after RESUME, comes before any task: a dropped group pauses the job with `Compute group '<name>' not found.` instead of `no alive backends` and advice to check the Kafka properties. An auto resume attempt skips that check: it refreshes a job that is still PAUSED, which cannot be paused again with another reason, so the attempt goes on to NEED_SCHEDULE and the check before the first task records a revoked USAGE as `CANNOT_RESUME_ERR`. Every pause from the refresh (this check, a failed Kafka metadata request, a failed Kinesis shard listing) goes through `RoutineLoadJob.pauseIfNeedSchedule()`, which takes the job lock and checks NEED_SCHEDULE again, so it cannot undo a concurrent STOP. CREATE validates the workload group in the declared group's namespace. An ALTER that sets `compute_group` validates in that group the workload group the job has afterwards (the one the ALTER sets, or else the one it keeps), as the owner the tasks run as. An ALTER that sets `workload_group` alone is still not checked, as on master. The statement re-parsed at metadata load adopts both names without resolving either, because a failure there would cancel the job for good: the declared group can have been dropped (and the stored statement is the CREATE one, so a job moved to another group by ALTER still names its old group there), and on image load the workload groups are not loaded yet. - CREATE / ALTER ROUTINE LOAD start a suspended declared group the way `USE @<group>` does (`waitForAutoStart`), on the statement thread before any job lock, since they send their Kafka metadata requests (custom partitions, offset conversion) through that group's backends. An ALTER that sets `compute_group` and `kafka_offsets` together converts the offsets in the group it declares, because the job's current group may be gone. The scheduler keeps resolving backends by name without starting a suspended group, as master does for every cloud routine load job. - SHOW CREATE ROUTINE LOAD writes `compute_group` when the job declares one. It writes the declared value, which follows ALTER, and never the effective group, so a job recreated from the statement is pinned the same way, and the statement of a job that declares nothing does not pin the group it inherited. SHOW CREATE MATERIALIZED VIEW already renders every MV property. - An async MV applies the declared group, then the session's group, on the planning context of the task and on every refresh context, and re-checks the declared group before each refresh. Each refresh context reads the declaration once and binds, records and re-checks that value. `REFRESH ... INCREMENTAL WITH DRY RUN` runs outside the task, on the current FE, in a context of its own: it binds the declared group to that context and re-checks it the same way. `EXPLAIN REFRESH` binds it to its planning context without the re-check, as the task does for its own. Neither borrows the session's group for an MV that declares none, as on master. Where master differs from branch-4.x: - master's `MTMVTask` splits a refresh into the IVM incremental path and the partition / complete path, each with its own context. `applyComputeGroup()` is factored out of the existing `setupComputeGroup()`, and the re-check runs in both paths. - master's routine load also has a Kinesis source, but `KinesisRoutineLoadJob` rejects cloud mode, so a Kinesis job never has a compute group to declare. - `REFRESH ... WITH DRY RUN` (from master's IVM) and `EXPLAIN REFRESH` exist only on master. Both build their context in `RefreshMTMVCommand` instead of `MTMVTask`, so that command binds the declared group itself, and the dry run re-checks it. - SHOW CREATE ROUTINE LOAD writes the declared `compute_group`; branch-4.x does not. - An ALTER that sets `compute_group` checks the job's workload group in that group; branch-4.x does not, so there a job moved to a group without its workload group fails every task. - The Kafka partition refresh checks the declared group before its metadata request, except on an auto resume attempt; branch-4.x has no check there. - Each MV refresh context reads the declaration once to bind and re-check it; branch-4.x reads it twice. - The 4.x unit test for a task without a task context is not carried over: `MTMVJob` always creates a task with a context, and master's `setupComputeGroup()` already relies on it. - The three unit test classes are converted to JUnit 5, as master requires, and `MTMVComputeGroupTest` gains two cases for the master-only parts: `setupComputeGroup()` applies and records the declared group, and the MV re-check fails for a declared group that no longer exists while it ignores the session's group. `RefreshMTMVCommandTest` gains four for the dry run and EXPLAIN REFRESH contexts, and `RoutineLoadJobTest` one for SHOW CREATE ROUTINE LOAD. - `ConnectContextTest.testThreadInfoUsesSessionCloudCluster` (#68111) sets the session variable directly instead of calling `ConnectContext.setCloudCluster()`, which is now a no-op outside cloud mode; branch-4.x made the same change when it picked #68111 (#68948). **Results.** The unit tests below pass locally. The regression suites need a cloud cluster and were not run locally: `test_routine_load_compute_group` and `test_mtmv_compute_group` are p0 suites, so P0 runs their non-cloud rejection checks and cloud_p0 runs them in cloud mode (the routine load part also needs Kafka); `test_compute_group_binding_isolation` is a docker suite that starts its own two-group cloud cluster.	2026-11-02 17:46:35 +08:00
Mingyu Chen (Rayner)	2b8908b883	[feature](be) Count the changelog TTL cleanups in the BE metrics (#69868) ... ### What problem does this PR solve? Issue Number: #69907 Related PR: None Problem Summary: **In short.** The BEs clean up the changelog of a table by TTL, but `/metrics` cannot tell whether that cleanup runs, fails, or reclaims anything. This PR counts each cleanup on the BE as it finishes, in three new series: how many succeeded or failed, how many bytes of expired changelog they dropped, and when one last succeeded. **Background.** A table with `changelog.enable` keeps its changelog in a hidden changelog index, and `changelog.ttl_seconds` bounds how long the changelog is kept. The BEs remove what is older than that in a cumulative compaction of a tablet of that index (a changelog tablet): rowsets whose commit TSO is at or before the TTL cutoff are left out of the merge, so the output replaces them without their rows. Two submitters start these compactions, both as `CHANGELOG_COMPACTION`: the changelog TTL scanner, for tablets it finds holding expired rowsets, and the regular compaction producer; either compaction picks expired rowsets first. **The problem, and what it cost.** - No alert can watch the cleanup. The bvars `changelog_ttl_discovered`, `changelog_ttl_submitted` and `changelog_ttl_failed` are only on the brpc `/vars` page, and they count what the TTL scanner found and submitted, not what the compactions did: a cleanup that fails after it was submitted counts nowhere, and nothing counts the bytes removed. - The only changelog series in `/metrics`, `compaction_task_state_total{type="changelog"}`, is the number of changelog compactions running and pending. A cleanup that keeps failing lets the changelog grow without bound, and nothing on the BE says why. **How this PR fixes it.** - Three series, counted as a compaction that drops expired changelog ends, and registered at startup so they exist from 0: | Series | Type | Labels | Value | | --- | --- | --- | --- | | `doris_be_changelog_ttl_cleanup_total` | counter | `result`=success/failed | the compactions that drop expired changelog rowsets, counted as each ends | | `doris_be_changelog_ttl_reclaimed_bytes_total` | counter | | the total disk size of the expired rowsets the successful ones dropped | | `doris_be_changelog_ttl_last_success_timestamp_seconds` | gauge | | when one last succeeded, in seconds since the epoch | - A compaction is a cleanup when one of its inputs is expired changelog: at or below the visible version its TTL snapshot was taken at, with a commit TSO at or before the cutoff. That is exactly what `filter_changelog_ttl_rowsets` leaves out of the merge, and the two now ask one predicate, `Compaction::is_expired_changelog_rowset`, which `pick_expired_changelog_rowset` uses too. It is decided from the inputs rather than from what the merge filtered, so that a cleanup that fails before its merge, a cloud compaction that cannot refresh its tablet meta for example, is counted as failed. - Counted where the execution of the compaction ends: `CumulativeCompaction::execute_compact` in shared-nothing mode, `CloudCumulativeCompaction::record_compaction_success` and `record_compaction_failure` in cloud mode. A compaction that does not get to run has not started a cleanup and is not counted: one that finds the tablet under another cumulative compaction, whether in its preparation or as its execution takes the cumulative compaction lock again, and one whose job the meta service refuses. - In cloud mode the BE that runs the compaction counts it, so a query sums over the BEs; the shared storage is not measured. The dropped rowsets become stale rowsets, whose files the stale rowset sweep or the recycler deletes later; the bytes are counted when the rowsets leave the tablet. - The bvars stay as they are. **Results.** Before this PR, the `/metrics` of a BE has none of these series. After it, on a local shared-nothing cluster, a table with `changelog.ttl_seconds` = 1 and two rows inserted (regression `test_changelog_ttl_cleanup_metrics`): | | `cleanup_total{result="success"}` | `cleanup_total{result="failed"}` | `reclaimed_bytes_total` | `last_success_timestamp_seconds` | | --- | --- | --- | --- | --- | | BE started | 0 | 0 | 0 | 0 | | once the changelog rows have been cleaned up | 1 | 0 | 1112 | the time of the cleanup | The 1112 bytes are what the BE logs for that changelog compaction: `input_rowsets_total_size=1.09 KB`, `output_rowset_total_size=0`, `filtered_row_num=2`. **Classes, and how they call each other.** - `DorisMetrics`: the three series. - `Compaction`: `is_expired_changelog_rowset` (new), shared by `pick_expired_changelog_rowset` and `filter_changelog_ttl_rowsets`; `record_changelog_ttl_cleanup` (new) counts a cleanup from the inputs. - `CumulativeCompaction`, `CloudCumulativeCompaction`: call it as the execution ends. ``` CumulativeCompaction::execute_compact shared-nothing └─ CompactionMixin::execute_compact ─► merge: filter_changelog_ttl_rowsets ─┐ └─ record_changelog_ttl_cleanup(ok) │ is_expired_changelog_rowset CloudCumulativeCompaction::execute_compact cloud │ └─ CloudCompactionMixin::execute_compact ─► merge: same filter ─────────────┤ └─ record_compaction_success ─► record_changelog_ttl_cleanup(true) │ record_compaction_failure ─► record_changelog_ttl_cleanup(false) │ └─ DorisMetrics::changelog_ttl_* ◄─────────┘ (from the inputs) ``` Nothing else changes: no config, no thrift or proto change, and what the compactions pick and drop stays the same.	2026-11-02 17:46:09 +08:00
Mingyu Chen (Rayner)	9dc9dc7566	[feature](ivm) Report how a refresh task refreshed an IVM MV, and count the refresh tasks on the Master (#69856) ... ### What problem does this PR solve? Issue Number: #69907 Related PR: None Problem Summary: **In short.** A refresh of an IVM materialized view can keep the MV up incrementally or quietly recompute it, and nothing tells the two apart: `tasks("type"="mv")` does not say which kinds of work a task committed, and `/metrics` has no IVM series at all. This PR makes each refresh task of an IVM MV report how it refreshed the MV (`IvmRefreshMethod` in `tasks()`) and why it failed or fell back, keeps a refresh summary on the MV for the IVM status table that follows, and counts the tasks on the Master in five new counters. **Background.** An IVM MV is refreshed by a task that runs one or more attempts: the incremental attempt (the delta of its base tables' streams, after rebuilding the MV partitions an invalidation marked), a partition rebuild, and a whole-MV rebuild. A request that may fall back (AUTO, INCREMENTAL FALLBACK, PARTITIONS FALLBACK) moves on to the next attempt when one gives up, for example to COMPLETE after the delta meets a MIN/MAX boundary. `tasks()` already reports `RefreshMode` (which partitions the task covered), `IvmFallbackReason` (why the incremental attempt gave up, or why a strict request failed) and `IvmRebuiltPartitions`. **The problem, and what it cost.** - An operator cannot tell a refresh that kept the MV up incrementally from one that recomputed it. `RefreshMode` says `COMPLETE` for any whole-MV scope, and `IvmFallbackReason` is set even when the attempt that replaced the delta wrote nothing, or when a strict request failed. - No alert can watch IVM refreshes. The asynchronous MV counters (`async_materialized_view_task_success_num`, `_failed_num`, `_skip_num`) cover every MV and say nothing about the method, the fallbacks, or why tasks fail. An MV that falls back to COMPLETE on every run, the expensive failure mode of IVM, goes unnoticed until the load of the cluster shows it. - A task that the full MV task queue turns away never runs, and nothing counts it. - The task history keeps only `max_persistence_task_count` tasks, so how many times in a row an MV has failed, or when it last fell back, cannot be read back from it. **How this PR fixes it.** - `MTMVTask` notes each batch that commits data (the delta, a partition rebuild, a whole-MV rebuild) and decides `IvmRefreshMethod` once, as the task publishes its result: `NONE`, `INCREMENTAL`, `PARTITION_REBUILD`, `COMPLETE`, or `MIXED` when more than one kind committed, typically the rebuild of invalidated partitions before the delta. `tasks("type"="mv")` gets the `IvmRefreshMethod` column, NULL while a task runs and for a task of an MV that is not an IVM MV. - A task falls back when the incremental attempt gives up and an attempt replacing it commits data. Not fallbacks: building the first baseline of an MV no refresh has succeeded on, the rebuild of invalidated partitions before the delta, an explicit COMPLETE or PARTITIONS request, and a fallback that commits nothing. - A failed task reports why: the `IvmFailureReason` a request that may not fall back failed with, or else the stage it failed in, `QUERY_UNUSABLE`, `PARTITION_SYNC_FAILED`, `EXECUTION_FAILED` or `OTHER`, told apart by where the failure was thrown rather than by its message. - The method, the failure reason and whether the task fell back are journaled with the task, as new Gson fields that an older FE ignores. The MV keeps an `IvmRefreshSummary`, updated from each task result in `MTMV.addTaskResult`, which the Master and the replay of the same journal record run alike, from what the record carries: consecutive failures, the latest success and failure (with the reason and the error message, cut to 512 characters), the latest method and run time, and the fallback count with the latest reason. A canceled task leaves it as it is. Nothing reads it in this PR; `information_schema.ivm_mv_status` and the per-MV series of a follow-up PR do. - Five counters, counted on the Master as each task of an IVM MV ends. Every label combination is registered up front, so each of the 61 series exists from 0: | Series | Type | Labels | Value | | --- | --- | --- | --- | | `doris_fe_ivm_refresh_tasks_total` | counter | `result`=success/no_change/failed/canceled, `method`=none/incremental/partition_rebuild/complete/mixed | refresh tasks that ended; `no_change` is a SUCCESS that committed nothing | | `doris_fe_ivm_refresh_duration_seconds_total` | counter | `method` | the run time, StartTime to FinishTime, of the tasks that succeeded or failed; a canceled task has no finish time | | `doris_fe_ivm_fallbacks_total` | counter | `reason`=the `IvmFailureReason` values, `incomplete_refresh_snapshot` | tasks that fell back | | `doris_fe_ivm_refresh_failures_total` | counter | `reason`=the `IvmFailureReason` values, the four stages | tasks that failed | | `doris_fe_ivm_schedule_rejected_total` | counter | | tasks the full MV task queue turned away | Like the other FE event counters they live in the registry of every FE and only the Master counts in them, so a query sums them over the FEs; a restart or a new Master starts them from 0. - `AbstractTask.onRejected`, a new hook that `TaskProcessor.addTask` calls when the executor queue rejects a task. Both the instant dispatch (`JobScheduler`) and the timer dispatch (`DispatchTaskHandler`) add tasks through it, and `MTMVTask` counts the rejection for an IVM MV. A timer-dispatched task that is turned away is canceled as before, so it also counts as `canceled`. **Results.** Before this PR, `tasks()` has no method column and `/metrics` has none of these series. After it, on a local cluster, one IVM MV refreshed through each case (regression `test_ivm_refresh_method_metrics`): | Refresh | `tasks()`: Status, IvmRefreshMethod, IvmFallbackReason | Counters that moved, besides the run time | | --- | --- | --- | | first AUTO refresh of a new MV | SUCCESS, COMPLETE, NULL | `tasks{success,complete}` | | INCREMENTAL after an insert | SUCCESS, INCREMENTAL, NULL | `tasks{success,incremental}` | | INCREMENTAL with nothing new | SUCCESS, NONE, NULL | `tasks{no_change,none}` | | INCREMENTAL after deleting the row that holds a group's minimum | FAILED, NONE, MIN_MAX_BOUNDARY_HIT | `tasks{failed,none}`, `failures{min_max_boundary_hit}` | | INCREMENTAL FALLBACK in the same state | SUCCESS, COMPLETE, MIN_MAX_BOUNDARY_HIT | `tasks{success,complete}`, `fallbacks{min_max_boundary_hit}` | | COMPLETE | SUCCESS, COMPLETE, NULL | `tasks{success,complete}` | | INCREMENTAL after truncating a base partition and inserting into another | SUCCESS, MIXED, NULL | `tasks{success,mixed}` | The first refresh of a new MV finds every partition still to be filled and goes straight to COMPLETE; it counts no fallback. The MV holds what its base table does after each case. **Classes, and how they call each other.** - `IvmRefreshMethod`, `IvmRefreshFailureStage` (`org.apache.doris.mtmv.ivm`): the methods, and the stages a failure is reported by. - `MTMVTask`: notes committed batches (`recordCommitted`), decides whether giving up the incremental attempt is a fallback (`givingUpTheIncrementalAttemptIsAFallback`), moves the failure stage along, settles the method and the failure reason in `after()`, and counts the task (`recordIvmRefreshMetrics`) and a rejection (`onRejected`). - `IvmRefreshSummary` (`org.apache.doris.mtmv.ivm`): persisted on `MTMV` and updated by `update(MTMVTask)` from `MTMV.addTaskResult`; `MTMV.hasSucceededIvmRefresh` tells the first baseline apart. - `MetricRepo`: registers the five counters. `DoubleCounterMetric` (new) keeps a total in seconds without rounding each addition. - `AbstractTask.onRejected` (new, empty by default), called by `TaskProcessor.addTask`. ``` MTMVTask.run() worker thread ├─ executeIvmAttempt │ ├─ rebuild invalidated partitions ──► recordCommitted(PARTITION_REBUILD) │ └─ delta (IvmIncrRefreshManager) ───► recordCommitted(INCREMENTAL) │ gave up ─► fallingBack = givingUpTheIncrementalAttemptIsAFallback() ├─ PARTITIONS / COMPLETE attempts ───────► recordCommitted(...) ivmFallback once fallingBack └─ failureStage: QUERY_UNUSABLE ─► PARTITION_SYNC_FAILED ─► EXECUTION_FAILED, or an IvmFailureReason onSuccess / onFail / cancel ─► after() ├─ ivmRefreshMethod, ivmFailureReason settled ├─ recordIvmRefreshMetrics() ─► MetricRepo.COUNTER_IVM_* Master only └─ Env.addMTMVTaskResult ─► MTMV.addTaskResult ─► IvmRefreshSummary.update(task) (journaled as ADD_TASK; the replay calls addTaskResult the same way) TaskProcessor.addTask ── queue full ─► task.onRejected() ─► MTMVTask: COUNTER_IVM_SCHEDULE_REJECTED ``` Nothing else changes: no thrift or proto change, no new config, and the existing columns and series stay as they are.	2026-11-02 17:41:47 +08:00
Mryange	fcf143c1aa	[fix](function) Align timezone offset parsing across FE and BE (#68979) ... ### What problem does this PR solve? Issue Number: N/A Related PR: #68865 Problem Summary: Timezone offsets can be accepted or rejected differently by FE constant folding and BE runtime CAST. For example, a DATETIME suffix of `-14:00` is valid under the literal policy but is rejected by the BE session-zone resolver. Session offsets such as `+14:30` and `-12:30` can bypass validation through the preloaded BE cache, while FE CONVERT_TZ applies different minute restrictions to bare and UTC/GMT-prefixed offsets. Root cause: these entry points duplicate offset validation and mix session and literal policies. Introduce a shared FE parser using the existing timezone aliases and upstream canonicalization helper, retain separate session/DATETIME/TIMESTAMPTZ policies, validate BE cache endpoints, and construct parsed literal offsets through TimezoneUtils. Strict and non-strict BE CAST use the same checks. Numeric TIMESTAMPTZ offsets are limited to Java's `[-18:00, +18:00]`, retaining historical minute and second offsets within that range. Explicit named and compact suffix recognition is also corrected. This does not add a BE Java alias table. Session alias dispatch is already fixed by #68865. System timezone database differences, ignored external lookup return values and DST rounding order are outside this change. ### Release note Fix inconsistent timezone offset validation in CAST and CONVERT_TZ, reject out-of-range cached session offsets, and correctly recognize explicit timezone suffixes. Numeric TIMESTAMPTZ offsets beyond +/-18:00 are now rejected. ### Check List (For Author) - Test - [x] Regression test - `test_timezone_parser_contract`: generated output through the runner and replayed successfully (1 suite, 0 failures). Compared 55 inputs across DATETIME/TIMESTAMPTZ and five execution paths: 550 results without semantic differences, allowing UTC display-format differences. - Upstream `test_session_time_zone_short_ids`: passed after rebase, covering scans, point queries and stream load. - [x] Unit Test - Earlier focused FE tests: 76 tests, 0 failures/errors, 4 existing skips; BE TimezoneUtilsTest: 5/5 passed. - After rebase: TimeUtilsTest, CoordinatorContextTest and TimeZoneParserTest: 18 tests passed. - [x] Manual test - Rebuilt BE and FE after rebase to master `2d053c83252`; both services report the matching version and Alive=true. ACT session FROM_UNIXTIME on a stored column matches Australia/Darwin without a BE alias table. - Validation limit: the final FE cleanup removing redundant normalization/default arguments and reusing getCanonicalTimeZone has not been rebuilt. The latest regression ran on the preceding build; CI must validate the final source. - clang-format, git diff --check and BE build hygiene passed. The earlier clang-tidy run reported pre-existing modernization diagnostics; it was not a clean clang-tidy pass. - Behavior changed: - [ ] No. - [x] Yes. Offset boundaries and explicit suffix parsing are consistent across the scoped FE/BE paths; invalid endpoints are rejected. - Does this need documentation? - [x] No. This aligns existing offset policies and documents the TIMESTAMPTZ boundary change in the release note. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label	2026-11-02 17:40:38 +08:00
Mingyu Chen (Rayner)	599616cb0d	[fix](metrics) Escape label values in the Prometheus output of FE (#69837) ... ### What problem does this PR solve? Issue Number: #69907 Related PR: None Problem Summary: **In short:** the FE writes metric label values into its Prometheus output without escaping them. A table name may contain a backslash, a double quote or a line feed, and in cloud mode the Master FE labels four per-table series with the table name, so a single such table leaves the Master's whole `/metrics` response outside the text format, and Prometheus keeps no sample of a scrape it cannot parse. This PR escapes every label value that is not a literal through one function, as the format requires. Ordinary names are written exactly as before. #### Background In the Prometheus text format a label value is written between double quotes, and a backslash, a double quote or a line feed inside it must be written as `\\`, `\"` and `\n`; no other character is escaped. A scrape whose response does not parse fails as a whole. With `enable_unicode_name_support`, which is on by default, a table name may contain all three characters: `FeNameFormat.UNICODE_TABLE_NAME_REGEX` only requires the last character not to be white space. A local cluster with default settings accepts tables named ``` `a"b` ```, `` `a\b` `` and one with a line feed in the middle. The names of databases, users, workload groups, jobs and catalogs keep their restricted patterns. #### The problem, and what it cost - `PrometheusMetricVisitor` concatenates label values as they are, in every place that writes one: the labels of registered metrics (`visit`), both kinds of histograms, the cloud per-table series (`visitCloudTableStats`), the workload group series and the JVM GC series. - In cloud mode the Master FE exports `doris_fe_table_data_size`, `doris_fe_table_rowset_count`, `doris_fe_table_segment_count` and `doris_fe_table_row_count` for up to `prom_output_table_metrics_limit` (10000) tables, labeled with `db_name` and `table_name`. One table whose name holds one of the three characters makes the response invalid, and every metric of the Master FE is lost while that table is exported. - The coming monitoring of table streams and IVM materialized views labels series with the names of materialized views and streams, which would run into the same problem in both modes. #### How this PR fixes it - `escapeLabelValue` writes a backslash, a double quote and a line feed as `\\`, `\"` and `\n` and leaves every other character alone, which is the full set the format requires. - Every label value that is not a literal goes through it: the two histogram paths and `visit`, the database and table names of the per-table series (escaped once per table, not once per series), the workload group name and the GC name. The JVM GC and workload group names cannot hold these characters today; they go through the same function so that no label value in the class is written raw. - A name without these characters comes back as the same string, so the output of ordinary names does not change, byte for byte, and costs no extra allocation (`String.replace` returns the string itself when nothing matches). - A null label value is written as `null`, as the concatenation without escaping wrote it. Label values can be null: a table warm-up job has no source cluster, so the series of its start and finish times carry a null `src_cluster_name`. #### Results What the cloud per-table series is written as for each table name, and what `promtool check metrics` (Prometheus v3.2.0) says about it: | Table name | Before this PR | promtool | After this PR | promtool | | --- | --- | --- | --- | --- | | `t1` | `table_name="t1"` | OK | `table_name="t1"` | OK | | `a"b` | `table_name="a"b"` | `unexpected end of label value "a"` | `table_name="a\"b"` | OK | | `a\b` | `table_name="a\b"` | `invalid escape sequence '\b'` | `table_name="a\\b"` | OK | | `a\` | `table_name="a\"` | `label value "a\"} 100" contains unescaped new-line` | `table_name="a\\"` | OK | | `a`, line feed, `b` | the sample split over two lines | `label value "a" contains unescaped new-line` | `table_name="a\nb"` | OK | A parser that follows the format turns each escaped value back into the table name itself. #### Classes, and how they call each other - `PrometheusMetricVisitor`: the new private `escapeLabelValue`, called wherever a label value is written. - `MetricsTest`: `testPrometheusVisitorEscapesLabelValues` (a gauge with an escaped and a plain label, and both kinds of histograms), `testCloudTableStatsEscapeTableName` (the four per-table series, with a mocked `CloudTabletStatMgr`) and `testPrometheusVisitorWritesNullLabelValue` (a gauge and a labeled histogram with a null label value). ``` MetricsAction (/metrics) -> MetricRepo.getMetric(PrometheusMetricVisitor) |-- visitJvm GC name -> escapeLabelValue |-- DORIS_METRIC_REGISTER.accept -> visit labels -> escapeLabelValue |-- visitHistograms -> visitHistogram (both) labels -> escapeLabelValue |-- visitCloudTableStats db / table name -> escapeLabelValue `-- visitWorkloadGroup group name -> escapeLabelValue ``` Not changed: the JSON output (`type=json`), which builds its own strings and would need JSON escaping instead; the core output (`type=core`), which writes no labels; the metric names and the `# HELP` texts, which are literals.	2026-11-02 17:40:12 +08:00
morrySnow	0fe5cc1e31	[fix](changelog) Handle disabled TSO in the system table smoke test (#69824) ... The system-table smoke test queries every information_schema table. When experimental_enable_feature_changelog is false, querying tso_status intentionally reports "TSO feature is disabled", stopping the suite before the remaining metadata tables are checked. Assert that expected error for the disabled configuration and retain the regular query when enabled.	2026-11-02 17:21:22 +08:00