mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Calvin Kirs
4ea473c9ba
### Background Previously, BE depended on the HDFS official distribution through the thirdparty module. This approach directly compiled and packaged Hadoop’s full dependencies, including large and unnecessary modules such as YARN, MapReduce, etc. As a result: The build artifact contained many duplicate jars. The total size was unnecessarily large (hundreds of MB). BE only acted as an HDFS client, not a Hadoop runtime, so most of these dependencies were not needed. ### Changes in this PR Move all HDFS-related jars to FE for unified dependency management, since FE already uses HDFS extensively. Retain necessary native/C++ dependencies under thirdparty for BE. Do not build an uber-jar for HDFS (unlike other BE thirdparty libs). Exclude unused Hadoop modules (e.g., YARN, MR, tools). The resulting target structure now includes: ``` target/ ├── classes ├── lib/ │ └── (~57M total) ├── hadoop-deps.jar └── ... ``` Compared with the previous build, the total artifact size has been reduced by ~150 MB. ### Rationale Since BE only functions as a client to access HDFS, managing Java-side HDFS dependencies in FE is cleaner and more maintainable. If BE needs custom logic (e.g., optimized HDFS client), we can extend or override the relevant classes and control class loading order. Impact Build size reduced by ~150 MB Simpler dependency graph ### Notes Native/C++ HDFS dependencies remain managed under thirdparty. FE now provides necessary jars for BE at runtime. No need to modify existing C++ build logic.