gerashegalov opened a new issue, #18708: URL: https://github.com/apache/iceberg/issues/18708
### Feature Request / Improvement Iceberg's Spark scan resolves the split target from the read option, session configuration, table property, or default, then may reduce it through adaptive sizing. The resulting target is passed to task-group planning, but [the adjustment is currently visible only in a debug log](https://github.com/apache/iceberg/blob/c71e7dfca17633ae2bce15c486278e8e07629cea/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java#L369-L385). Scan analysis and autotuning tools cannot reliably recover the target from Spark SQL metrics or event logs. This matters when diagnosing scan partition sizes; see [cudf-spark-tools#2206](https://github.com/NVIDIA/cudf-spark-tools/issues/2206). Please expose the **effective split target size in bytes** as a Spark scan driver metric, for example `effectiveSplitTargetSize`. Its value should be the target actually passed to task-group planning after configuration precedence and adaptive sizing are resolved, rather than an individual input partition's size. Useful checks would cover an adaptive target, a configured target with adaptation disabled, an explicit read option, and staged rewrite scans that use their resolved target directly. An initial contribution can target Spark 4.2, with older Spark integrations considered separately. ### Query engine Spark ### Willingness to contribute - [x] I can contribute this improvement/feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
