### What problem does this PR solve?
Related PR: #69312
Port #69312 from `branch-4.2` to current `master`.
- Remove Hive ACID read transactions, shared read locks, write-ID
snapshots, delta-file listing and the BE transactional reader. Reject
full ACID and insert-only reads in the Hive connector.
- Adapt the changes to master's connector modules and unified reader
initialization interfaces. Preserve ordinary Hive scanning and
Iceberg/Paimon ORC position-delete and deletion-vector handling.
- Reject standalone Avro and deprecated table markers before file access
or metadata-only counts. Keep LakeSoul unsupported and remove its
obsolete test service and examples.
- Keep the retired `skip_checking_acid_version_file` name as a
compatibility no-op for client session initialization, using the
existing empty-string fallback for variable reads.
- Preserve Iceberg/Hudi Avro metadata dependencies. The Thrift changes
add deprecation comments only; every definition and numeric ID is
unchanged.
### Release note
Remove support for Hive transactional reads, including insert-only
tables, and standalone Avro file reads. LakeSoul remains unsupported.
Iceberg and Hudi Avro metadata parsing remains supported. The retired
`skip_checking_acid_version_file` session variable remains accepted as a
compatibility no-op.
### Validation
- Hive connector: all 490 tests passed; HMS client: all 155 tests
passed.
- 20 targeted FE core tests passed: deprecated-read rejection,
partition-path parsing, generic file scan planning and Hive streaming
scans.
- Review follow-up: reproduced the legacy ACID session-variable failure
before the fix; all 8 focused compatibility, variable-export and
unsupported-read tests passed after the fix.
- FE Checkstyle passed with zero violations for every affected FE
module.
- All 12 affected C++ source/header files passed clang-format 16; build
hygiene and shell syntax checks passed.
- Syntax-only compilation passed for all 6 affected BE production
translation units and all 4 affected BE test translation units. Isolated
checks relaxed pre-existing unused-constant/shadow warnings. Full BE
linking and runtime tests remain for CI.
- Verified that Thrift changes are comments only and checked the
affected definitions against branch-3.1/4.0/4.1/4.2. `git diff --check`
passed.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test: formatting, syntax, build hygiene and Thrift
compatibility checks described above.
- Behavior changed:
- [x] Yes. Reject unsupported external reads explicitly.
- Does this need documentation?
- [x] No. This ports the existing removal to master.
## Proposed changes
The `apache_hdfs_broker` daemon under `fs_brokers/` has not been
modified for nearly two years, and users have migrated to the built-in
S3 / HDFS TVF, so the standalone broker component is no longer needed.
The in-tree SDKs under `sdk/` (`go-doris-sdk`, `java-doris-sdk`) are now
maintained out-of-tree in a separate `doris-sdk` repo and no longer need
to live here.
### Removed
- `fs_brokers/apache_hdfs_broker/` — broker daemon module
- `sdk/` — `go-doris-sdk` + `java-doris-sdk`
- `docker/runtime/broker/` — broker daemon docker image, and the
`hdfs-broker-doris` compose demo
### Build / CI / docker references cleaned up
`build.sh` (`--broker` / `BUILD_BROKER`), `build-for-release.sh`,
`.github/workflows/build-extension.yml` (`build-broker` job),
`.github/workflows/code-checks.yml`, `.gitignore`,
`docker/runtime/docker-build.sh`, `docker/runtime/all-in-one/Dockerfile`
+ `entry_point.sh`, `docker/runtime/README.md`, and `threat-model.md`.
### Kept (intentionally out of scope)
- `fs_brokers/cdc_client/` (not a broker; still built into BE)
- All FE/BE broker **functionality**: `BrokerMgr`, `BrokerDesc`, broker
load, `fe-filesystem-broker`, the BE filesystem broker module, and
`PaloBrokerService.thrift`
This PR only removes the standalone broker daemon component and the
in-tree SDK sources; the broker load/storage feature in FE/BE is
unchanged, so there is no metadata/edit-log compatibility impact.
Previously, we use RuntimeProfile class directly, and because there are multiple level in profile, so you can see there may be several RuntimeProfile instances be to maintain.
I created several new classes for profile:
class Profile:
The root profile of a execution task(query or load)
class SummaryProfile:
The profile that contains summary info of a execution task,
such as start time, end time, query id. etc.
class ExecutionProfile:
The profile for a single Coordinator. Each Coordinator will
have a ExecutionProfile.
The profile structure is as following:
Profile:
SummaryProfile:
ExecutionProfile 1:
Fragment 0:
Instance 0:
Instance 1:
...
Fragment 1:
...
ExecutionProfile 2:
...
You can see, each Profile has a SummaryProfile and one or more ExecutionProfile.
For most kinds of job, such as query/insert, there is only one ExecutionProfile. But for broker load job, will may be more than one ExecutionProfile, corresponding to each sub task of the load job.
How to use
For query/insert, etc:
Each StmtExcutor will have a Profile instance.
Each Coordinator will have a ExecutionProfile instance.
StmtExcutor is responsible for the SummaryProfile, it will update the SummaryProfile during the execution.
Coordinator is responsible for the ExecutionProfile, it will first add ExecutionProfile to the child of Profile, and update the ExecutionProfile periodically during the execution.
For Load/Export, etc:
Each job will hava a Profile instance.
For each Coordinator of this job, add its ExecutionProfile to the children of job's Profile.
Behavior Change
The columns of show load profile/show query profile and QueryProfile Web UI has changed to:
| Profile ID | Task Type | Start Time | End Time | Total | Task State | User | Default Db| Sql Statement | Is Cached | Total Instances Num | Instances Num Per BE | Parallel Fragment Exec Instance Num | Trace ID |
The Query Id and Job Id is removed and using Profile ID instead.
For load job, the profile id is job id, for query/insert, is query id.
Gabriel
Calvin Kirs
Gavin Chou
Yongqiang YANG
Gavin Chou
Mingyu Chen
wudongliang
shuke
Kang
Pxl