Gabriel
4bde8d0fb2
### What problem does this PR solve? Issue Number: close #69926 Problem Summary: Hive JSON queries that only need partition columns can return zero rows even when the selected files contain records. For example, `SELECT * ... WHERE dt = 'p1'` returns data while `SELECT COUNT(*) ... WHERE dt = 'p1'` returns zero. The V2 JSON reader derived its row count from the output block, which has no columns after physical field pruning. Writer-backed reader tests reproduce the same row-count loss for empty WAL projections. Track accepted JSON records independently of materialized columns, including batch progress, array expansion, error rollback, and ignore-malformed NULL rows. Preserve JSON-path record selection for empty projections, and let the existing TableReader materialize partition values using the returned row count. For WAL, save the deserialized source row count before materialization transfers column ownership, then pass it through the existing filter pipeline. This preserves the row count when no file columns are requested. The empty-projection reproduction is at the reader interface; production replay reachability of an empty projection remains unverified. Add ten JSON BE tests covering empty file schemas and projections, arrays, JSON roots/paths, malformed input, batch boundaries, EOF, and actual TableReader partition materialization. Add eight writer-backed WAL cases covering empty, single-column, and reordered full projections across multiple blocks, with and without schema discovery, plus repeated EOF reads. Add a WAL SQL regression that forces two original async-load transactions to abort, then verifies row counts, NULLs, aggregates, filters, and actual values after replay. Add a Hive SQL regression using the existing partitioned JSON fixture for COUNT, partition projections, grouping, and file-column filters with external COUNT pushdown enabled and disabled. ### Release note Fix incorrect zero counts and missing rows when querying only partition columns of Hive JSON tables. Preserve logical row counts for empty WAL reader projections. ### Check List (For Author) - Test - [x] Unit Test: 166 ASAN tests passed, including JSON reader, TableReader, WAL V2 reader, WAL reader/writer, and CSV/Text empty-projection controls. Before the JSON fix, nine of ten new tests failed; before the WAL fix, both empty-projection cases failed while six projection controls passed. - [x] WAL SQL regression: `test_group_commit_wal_row_count` passed twice on isolated Doris 4.1.5 with scanner V2 enabled. Generated the `.out` file with `run-regression-test.sh`, then reran for result comparison. Execution on master is pending CI. - [x] Hive SQL regression: added `test_hive_json_partition_only`; Groovy compilation passed. Full execution against the external Hive fixture is pending CI. - Behavior changed: - [x] Yes. Empty JSON and WAL file projections preserve logical row counts. Hive JSON partition-only queries produce the correct rows and aggregates. - Does this need documentation? - [x] No.
Guide for test cases
General Case
-
Write "def" before variable names; otherwise, they will be global variables and may be affected by other cases running in parallel.
Problematic code:
ret = ***Correct code:
def ret = *** -
Avoid setting global session variables or modifying cluster configurations in cases, as it may affect other cases.
Problematic code:
sql """set global enable_pipeline_x_engine=true;"""Correct code:
sql """set enable_pipeline_x_engine=true;""" -
If it is necessary to set global variables or modify cluster configurations, specify the case to run in a nonConcurrent manner.
-
For cases involving time-related operations, it is best to use fixed time values instead of dynamic values like the
now()function to prevent cases from failing after some time.Problematic code:
sql """select count(*) from table where created < now();"""Correct code:
sql """select count(*) from table where created < '2023-11-13';""" -
After streamloading in a case, add a sync to ensure stability when executing in a multi-FE environment.
Problematic code:
streamLoad { ... } sql """select count(*) from table """Correct code:
streamLoad { ... } sql """sync""" sql """select count(*) from table """ -
For UDF cases, make sure to copy the corresponding JAR file to all BE machines.
-
Do not create the same table in different cases under the same directory to avoid conflicts.
-
Cases injected should be marked as nonConcurrent and ensured injection to be removed after running the case.
-
Docker case run in a docker cluster. The docker cluster is new created and independent, not contains history data, not affect other cluster.
Docker case can add/drop/start/stop/restart fe and be, and specify fe and be num.
Example will see demo_p0/docker_action.groovy
Read the annotation carefully in the example file.
Also read the doris-compose readme.
-
Do not use raw
Thread.startorThread.startDaemonto run SQL in a case. Use the frameworkthreadorextraThreadhelper and callfuture.get()to wait for completion, so exceptions are reported to the case that created the thread.Problematic code:
def t = Thread.start { sql """insert into tbl values (1)""" } t.join()Correct code:
def future = thread("insert-thread") { sql """insert into tbl values (1)""" } future.get()
Compatibility case
Refers to the resources or rules created on the initial cluster during FE testing or upgrade testing, which can still be used normally after the cluster restart or upgrade, such as permissions, UDF, etc.
These cases need to be split into two files, load.groovy and xxxx.groovy, placed in a folder, and tagged with the restart_fe group label, example.