kumarpritam863 commented on PR #11290: URL: https://github.com/apache/iceberg/pull/11290#issuecomment-2402743316
@bryanck Thanks for the quick response. we were able to optimise on the memory and performance also by adding some configurations like: "iceberg.tables.write-props.write.parquet.page-size-bytes" : "16384", "iceberg.tables.auto-create-props.write.parquet.page-size-bytes" : "16384", "iceberg.table.write-props.write.parquet.row-group-size-bytes" : "33554432", "iceberg.tables.auto-create-props.write.parquet.row-group-size-bytes" : "33554432" As the default values are set on the higher side. with these values we lowered the number of files, memory usage and boosted performance. Also one of the main reason for this design was that the current design has a lot of issues when it comes to ICR. Also if you can please just go though the doc once. Also we can connect and discuss on this whenever you are free. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
