paulpaul1076 commented on issue #8729: URL: https://github.com/apache/iceberg/issues/8729#issuecomment-1750200744
@amogh-jahagirdar the value of spark.sql.adaptive.advisoryPartitionSizeInBytes is the default 64MB. 1) Do I understand it correctly that the size of the uncompressed data in my spark tasks is supposed to be around 64MB? 2) Also, do I understand it correctly that the we can't write one file from multiple tasks? If the answers to 1) and 2) is yes and yes, then the size of files (uncompressed) should be less than 64MB and if we compress it it's supposed to be even smaller, maybe around 5MB. But how does it end up being 100MB? Does iceberg do extra coalescing of tasks such that they grow larger than spark.sql.adaptive.advisoryPartitionSizeInBytes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
