gtrettenero commented on code in PR #18724:
URL: https://github.com/apache/iceberg/pull/18724#discussion_r4431489675
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetIO.java:
##########
@@ -49,8 +57,81 @@
/** Methods in this class translate from the IO API to Parquet's IO API. */
class ParquetIO {
+ // number of bytes read from the end of a file to get the footer in a single
request; this matches
+ // the default footer read size used by the Arrow C++ Parquet reader
+ @VisibleForTesting static final int FOOTER_READ_SIZE = 64 * 1024;
Review Comment:
Open question for reviewers: 64 KiB matches the Arrow C++ default footer
read size and covers typical footers, but very wide schemas (roughly 1,000+
columns) have larger footers and need a second tail read. That case is no worse
than today (two requests either way), just no better. Would you prefer a larger
default (e.g. 256 KiB), a read property to tune it, or keeping it as is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
