venkata91 opened a new pull request, #18726: URL: https://github.com/apache/iceberg/pull/18726
## Problem - When one task reads several input partitions (for example, a storage-partitioned join on a subset of the partition keys), Spark 4.1 opens a reader per partition and hands the previous reader's metrics to the next one through `initMetricsValues`. - `BatchDataReader`, `RowDataReader` and `MergingSortedRowDataReader` inherit the no-op default, so `numSplits` and `numDeletes` only count the last reader in each task. Raised by @manuzhang in https://github.com/apache/iceberg/pull/17562#discussion_r4379002870. ## Fix - The three readers now override `initMetricsValues` and add the inherited values to their own. - Spark 4.2 isn't affected: it no longer groups input partitions inside `DataSourceRDD` and doesn't call `initMetricsValues`, so this change is 4.1 only. ## Testing - New `TestSparkReadMetrics` case runs a subset-key SPJ that puts 5 partitions into 2 tasks. Without the fix, `numSplits` is 2 instead of 5, on both the vectorized and row paths. - New `TestMergingSortedRowDataReader` case covers the merging reader, which fails the same way without the fix. - `TestSparkReaderDeletes`, `TestStoragePartitionedJoins`, `TestBaseReader` pass. `spotlessCheck` and `checkstyle` pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
