yadavay-amzn commented on PR #16347: URL: https://github.com/apache/iceberg/pull/16347#issuecomment-4462004732
@nssalian Thanks for the heads up. I looked at #16327 -- the key difference is that your PR makes the fix opt-in via a new config flag (default `false`), while this PR fixes the behavior by default. My reasoning: the current behavior is a bug (row group size limit is silently ignored with compression). Users who set `write.parquet.row-group-size-bytes` expect it to be enforced regardless of codec. Making the fix opt-in means most users will never discover the config and continue hitting unbounded row groups. Happy to discuss the trade-off -- if maintainers prefer the opt-in approach for backward compatibility, I can close this. But I think the correct default is to enforce the configured limit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
