mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
f2ab237bf7
To fully understand the implementation of the PR, please refer to the following link（It is a Chinese document）: https://www.notion.so/V3-1-2c31293e1081807ca476dd5c87efb28e ### 1. PR Function Overview The core function of this PR is the implementation of **File Cache Admission Control**. * **Background**: Doris supports using File Cache to accelerate data access for remote storage (e.g., HDFS, S3). However, in certain scenarios (such as large-scale ETL jobs or heavy ad-hoc queries), reading massive amounts of cold data can evict existing hot data from the File Cache. This leads to "cache pollution" and a significant drop in cache hit rates. * **Goal**: Provide a mechanism to decide whether data scanned by a specific query is allowed to enter the File Cache, based on dimensions such as user identity and table information. If admission is denied, the query will read the data directly from remote storage without populating the cache. ### 2. Implementation Scheme Analysis The implementation consists of the following key components: #### 2.1. FE Side: Admission Decision The primary logic is located in the `createScanRangeLocations` method of `FileQueryScanNode.java`. * **New Configuration**: Introduced the `Config.enable_file_cache_admission_control` switch. * **New Manager**: Introduced `FileCacheAdmissionManager` (Singleton) to execute the specific admission judgment logic. * **Decision Flow**: 1. Before generating Scan Ranges, `FileQueryScanNode` retrieves the current User Identity (`userIdentity`), Catalog, Database, and Table information. 2. It calls `FileCacheAdmissionManager.getInstance().isAllowed(...)` to obtain a boolean result `fileCacheAdmission`. 3. It logs the decision result and the time cost in query plan. ``` | 0:VHIVE_SCAN_NODE(74) | | table: test_file_cache_features.tpch1_parquet.lineitem | | inputSplitNum=10, totalFileSize=205792918, scanRanges=10 | | partition=1/1 | | cardinality=1469949, numNodes=1 | | pushdown agg=NONE | | file cache request ADMITTED: user_identity:root@%, reason:user table-level whitelist rule, cost:37996 ns | | limit: 1 ``` #### 2.2. FE Side: Decision Propagation The decision result needs to be propagated from the `FileQueryScanNode` down to the underlying split assignment logic. * **SplitAssignment Modification**: * The constructor of the `SplitAssignment` class (located in `org.apache.doris.datasource`) is modified to accept a new `boolean fileCacheAdmission` parameter. * **SplitToScanRange Modification**: * The `splitToScanRange` method (or its corresponding Lambda expression) is updated to receive the `fileCacheAdmission` parameter. * This method is responsible for setting this value into the Thrift object. #### 2.3. Communication Protocol: Thrift Definition Update To pass the FE's decision to the BE, the Thrift definition (likely `TFileRangeDesc` or `TFileScanRangeParams` in `PlanNodes.thrift`) requires a new field. * **Inferred Change**: A new field `optional bool file_cache_admission`, is added to the `TFileRangeDesc` struct. #### 2.4. BE Side: Enforcement Although the analysis focuses on the FE, the complete loop requires enforcement on the BE side: * **FileReader**: The BE's `FileReader` (e.g., `HdfsFileReader` or `S3FileReader`) checks the `file_cache_admission` flag in the incoming `TFileRangeDesc` during initialization or reading. * **Cache Policy**: * If `file_cache_admission` is **true** (default): It uses the standard `FileCachePolicy`, where data not found in the cache is written to the Block File Cache after reading. * If `file_cache_admission` is **false**: It sets the `FileCachePolicy` to `NO_CACHE`, skips the cache writing step, reading directly from remote storage. This protects the existing cache from being polluted. ### 3. Summary This PR introduces an **Admission Control Manager** during the FE query planning phase and transparently passes this control signal through the **Split Assignment** and **Scan Range Generation** phases. This ultimately guides the BE side's file readers to **selectively** use the file cache. Co-authored-by: xuchenhao <419062425@qq.com> Co-authored-by: xuchenhao <48084123+xuchenhao@users.noreply.github.com> Co-authored-by: morningman <yunyou@selectdb.com>
84 lines
2.6 KiB
Bash
Executable File
84 lines
2.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
set -e
|
|
|
|
# Configuration
|
|
DB_HOST="localhost"
|
|
DB_USER="root"
|
|
DB_NAME="file_cache_admission_control"
|
|
DB_PASS=""
|
|
TABLE_NAME="admission_policy"
|
|
OUTPUT_FILE="rule_$(date +%Y%m%d_%H%M%S).json"
|
|
|
|
echo "=== Database Export Configuration ==="
|
|
echo "Database Host: $DB_HOST"
|
|
echo "Database User: $DB_USER"
|
|
echo "Database Name: $DB_NAME"
|
|
echo "Password: $(if [ -n "$DB_PASS" ]; then echo "Set"; else echo "Not set"; fi)"
|
|
echo "Table Name: $TABLE_NAME"
|
|
echo "Output File: $OUTPUT_FILE"
|
|
echo "====================================="
|
|
echo ""
|
|
|
|
# Query and convert to JSON (including long type timestamps)
|
|
QUERY=$(cat <<SQL
|
|
SELECT
|
|
JSON_ARRAYAGG(
|
|
JSON_OBJECT(
|
|
'id', id,
|
|
'user_identity', user_identity,
|
|
'catalog_name', IFNULL(catalog_name, ''),
|
|
'database_name', IFNULL(database_name, ''),
|
|
'table_name', IFNULL(table_name, ''),
|
|
'partition_pattern', IFNULL(partition_pattern, ''),
|
|
'rule_type', rule_type,
|
|
'enabled', CASE WHEN enabled = 1 THEN true ELSE false END,
|
|
'created_time', UNIX_TIMESTAMP(created_time),
|
|
'updated_time', UNIX_TIMESTAMP(updated_time)
|
|
)
|
|
) AS json_data
|
|
FROM ${TABLE_NAME}
|
|
SQL
|
|
)
|
|
|
|
# Execute query
|
|
if [ -n "$DB_PASS" ]; then
|
|
JSON_DATA=$(echo "$QUERY" | mysql -h $DB_HOST -u $DB_USER -p$DB_PASS $DB_NAME -N 2>/dev/null)
|
|
else
|
|
JSON_DATA=$(echo "$QUERY" | mysql -h $DB_HOST -u $DB_USER $DB_NAME -N)
|
|
fi
|
|
|
|
# Handle NULL
|
|
if [ "$JSON_DATA" = "NULL" ] || [ -z "$JSON_DATA" ]; then
|
|
JSON_DATA="[]"
|
|
fi
|
|
|
|
# Save to file
|
|
echo "$JSON_DATA" > "$OUTPUT_FILE"
|
|
|
|
# Format
|
|
if command -v jq &> /dev/null; then
|
|
jq '.' "$OUTPUT_FILE" | awk '
|
|
/^ {/ && NR > 3 {print ""}
|
|
{print}
|
|
' > "${OUTPUT_FILE}.tmp" && mv "${OUTPUT_FILE}.tmp" "$OUTPUT_FILE"
|
|
fi
|
|
|
|
echo "Export completed: $OUTPUT_FILE"
|