mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Gabriel
7f00c96348
### What problem does this PR solve? `test_hive_compress_type_large_data` succeeds with the default file split size but fails after setting `file_split_size=8388608`: ``` [INTERNAL_ERROR] For now we do not support split compressed file ``` Hive's planner used the table-level TEXT/CSV splittability for every file. For example, a 20 MiB gzip file was assigned ranges starting at 0, 8 MiB and 16 MiB. Both native text/CSV readers reject compressed streams starting at a nonzero offset. The same partition can contain plain and compressed files, so a table-wide decision is insufficient. Determine splittability per file in the common range iterator used by synchronous, partition-batch and streaming scans. Recognized compressed TEXT/CSV files now produce one complete range starting at zero. Plain text still splits; ORC/Parquet keep their existing range planning. Hadoop codec splittability is not used because the native readers have different restart capabilities. Move the existing filename-to-codec inference into a shared, dependency-free foundation enum and delegate `Util.inferFileCompressTypeByPath` to it. This keeps planning and the codec sent to BE on the same suffix rules, without changing supported suffixes, Hive's LZ4 adjustment, LZO sidecar filtering, or the BE safety checks. ### Validation - The new mixed-file planner test fails on the original code (33 ranges instead of 13) and passes with the fix. It covers TEXT/CSV, ORC/Parquet, all recognized compression suffixes and uppercase gzip, empty files, plain-file offsets/lengths, and synchronous/batch/streaming enumeration. - `mvn -f fe/pom.xml -pl :fe-connector-hive -am test -Dmaven.build.cache.enabled=false`: 1,462 tests passed, zero failures/errors/skips; 510 tests in the Hive module. - Foundation tests cover unchanged codec recognition and unknown-suffix behavior; connector tests verify the shared enum names map to native Thrift codecs. - The existing large-data regression retains its cardinality/filter assertions, forces synchronous 8 MiB queries, and compares forced streaming queries with the default-size results. Groovy syntax compilation passed. - FE Checkstyle and `git diff --check` passed. A live Hive environment was not exercised locally. The external regression and full FE integration build remain for CI. ### Release note Fix Hive compressed-text queries failing when file split sizes are smaller than the compressed files. ### Check List (For Author) - [x] Unit Test - [x] Regression test extension - [x] Behavior changed: compressed native text files are read as whole streams - [x] No documentation needed