mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Gabriel
4bde8d0fb2
### What problem does this PR solve? Issue Number: close #69926 Problem Summary: Hive JSON queries that only need partition columns can return zero rows even when the selected files contain records. For example, `SELECT * ... WHERE dt = 'p1'` returns data while `SELECT COUNT(*) ... WHERE dt = 'p1'` returns zero. The V2 JSON reader derived its row count from the output block, which has no columns after physical field pruning. Writer-backed reader tests reproduce the same row-count loss for empty WAL projections. Track accepted JSON records independently of materialized columns, including batch progress, array expansion, error rollback, and ignore-malformed NULL rows. Preserve JSON-path record selection for empty projections, and let the existing TableReader materialize partition values using the returned row count. For WAL, save the deserialized source row count before materialization transfers column ownership, then pass it through the existing filter pipeline. This preserves the row count when no file columns are requested. The empty-projection reproduction is at the reader interface; production replay reachability of an empty projection remains unverified. Add ten JSON BE tests covering empty file schemas and projections, arrays, JSON roots/paths, malformed input, batch boundaries, EOF, and actual TableReader partition materialization. Add eight writer-backed WAL cases covering empty, single-column, and reordered full projections across multiple blocks, with and without schema discovery, plus repeated EOF reads. Add a WAL SQL regression that forces two original async-load transactions to abort, then verifies row counts, NULLs, aggregates, filters, and actual values after replay. Add a Hive SQL regression using the existing partitioned JSON fixture for COUNT, partition projections, grouping, and file-column filters with external COUNT pushdown enabled and disabled. ### Release note Fix incorrect zero counts and missing rows when querying only partition columns of Hive JSON tables. Preserve logical row counts for empty WAL reader projections. ### Check List (For Author) - Test - [x] Unit Test: 166 ASAN tests passed, including JSON reader, TableReader, WAL V2 reader, WAL reader/writer, and CSV/Text empty-projection controls. Before the JSON fix, nine of ten new tests failed; before the WAL fix, both empty-projection cases failed while six projection controls passed. - [x] WAL SQL regression: `test_group_commit_wal_row_count` passed twice on isolated Doris 4.1.5 with scanner V2 enabled. Generated the `.out` file with `run-regression-test.sh`, then reran for result comparison. Execution on master is pending CI. - [x] Hive SQL regression: added `test_hive_json_partition_only`; Groovy compilation passed. Full execution against the external Hive fixture is pending CI. - Behavior changed: - [x] Yes. Empty JSON and WAL file projections preserve logical row counts. Hive JSON partition-only queries produce the correct rows and aggregates. - Does this need documentation? - [x] No.