Gabriel39 opened a new pull request, #69615: URL: https://github.com/apache/doris/pull/69615
### What problem does this PR solve? Doris writes Parquet row groups for Iceberg data files but does not include their split offsets in the commit fragments or manifests. Readers therefore cannot use those known boundaries when planning splits. Collect each row group's first page offset from the closed writer's in-memory Footer, including dictionary pages. Carry the offsets in a new optional Thrift field and set them on Iceberg data files. Collect them even when column metrics are disabled; leave empty files and older BE fragments without offsets, and do not apply Parquet offsets to ORC or delete files. ### Release note Include Parquet row-group split offsets in Iceberg data-file metadata. ### Validation - Maven reactor build and all 31 `IcebergWriterHelperTest` tests passed, including optional-field decoding, Manifest round trip, per-file isolation and ORC exclusion. The initial offset conversion test failed before the fix. - 6 targeted BE tests passed under ASAN: dictionary/plain pages, multiple/single/empty row groups, metrics-disabled Parquet commits and unchanged ORC metrics behavior. The changed source/test objects and generated Thrift/unity objects were recompiled and linked with cached engine libraries. - FE Checkstyle, clang-format 16, BE build hygiene and connector import checks passed. - Checked the new optional field ID against master and release branches 3.1, 4.0, 4.1 and 4.2. - Extended the partitioned-write regression suite to assert nonempty split offsets, including with column metrics disabled. The external regression and Spark/Trino interoperability tests were not run locally. - A clean full BE rebuild remains for CI: extending the local run to unrelated existing tests hit the old cached RuntimeState object's commit-fragment layout after adding the Thrift field. ### Check List (For Author) - Test: Unit Test and Regression test - Behavior changed: Yes. Newly written Iceberg Parquet data files publish split offsets as optional metadata. - Does this need documentation? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
