This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch branch-4.2 in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2a6f1521f03fdbe924667eedf4d66433c8568b41 Author: Gabriel <[email protected]> AuthorDate: Fri Oct 9 14:54:24 2026 +0800 [fix](regression) Bound large Parquet map output batches (#68808) ### What problem does this PR solve? Issue Number: N/A Related PR: N/A The large-string-map Parquet regression fixture contains two 1 GiB string keys in a column chunk larger than 2 GiB, with dictionary and PLAIN data pages. Materializing both rows into one output column requires a 4 GiB character buffer and can exceed the BE process memory limit during ASAN regression runs. Read this query with `batch_size=1` and disable aggregate pushdown using a statement-local hint. Both rows still go through full Map and string decoding, while each output batch contains only one large key. The query and its expected result remain unchanged. This retains the oversized Parquet column-chunk and mixed-encoding coverage; it no longer exercises accumulating both keys in one output ColumnString. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [x] Manual test - Compiled the modified suite with Groovy 4.0.19 successfully. - `git diff --check` passed. - Reviewed the batch-size validation and propagation to the native Parquet reader on branch-4.1. - The HDFS regression was not run locally: the original CI HDFS fixture environment is not configured. ASAN end-to-end validation remains for CI. - Behavior changed: - [x] No. Production code is unchanged; the existing test uses statement-local execution settings. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label --- .../suites/external_table_p0/tvf/test_hdfs_parquet_group0.groovy | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/regression-test/suites/external_table_p0/tvf/test_hdfs_parquet_group0.groovy b/regression-test/suites/external_table_p0/tvf/test_hdfs_parquet_group0.groovy index 7007ecefd3a..d964cb4a225 100644 --- a/regression-test/suites/external_table_p0/tvf/test_hdfs_parquet_group0.groovy +++ b/regression-test/suites/external_table_p0/tvf/test_hdfs_parquet_group0.groovy @@ -105,7 +105,10 @@ suite("test_hdfs_parquet_group0","external,hive,tvf,external_docker") { uri = "${defaultFS}" + "/user/doris/tvf_data/test_hdfs_parquet/group0/large_string_map.brotli.parquet" - order_qt_test_11 """ select count(arr) from HDFS( + // Read both 1 GiB keys one row per batch to avoid a 4 GiB output buffer allocation. + // Disable aggregate pushdown to retain full decoding of the >2 GiB column chunk. + order_qt_test_11 """ select /*+ SET_VAR(batch_size=1, enable_push_down_no_group_agg=false) */ + count(arr) from HDFS( "uri" = "${uri}", "hadoop.username" = "${hdfsUserName}", "format" = "parquet"); """ --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
