kevinjqliu commented on PR #17284: URL: https://github.com/apache/iceberg/pull/17284#issuecomment-5442105130
nit: how about something like this in the pr description? I want to make sure others (and perhaps myself) reading this PR in the future can quickly understand the behavior change. ### Behavior change for `HadoopFileIO` This PR changes small Parquet reads through `HadoopFileIO` to use the length reported by `InputFile.getLength()`—which may have been initialized from Iceberg metadata—rather than retrieving the physical file size through `HadoopInputFile.getStat()`. Previously, Hadoop-backed reads could tolerate an incorrect metadata length because `getStat()` returned the physical file size. Other implementations, such as `S3InputFile`, already use the metadata-provided length when available. With eager fetching, files at or below the 1 MiB threshold use the reported length to size the read buffer, including files backed by `HadoopInputFile`. This makes Hadoop consistent with other `InputFile` implementations, but an incorrect metadata length that previously succeeded through `HadoopFileIO` may now cause the read to fail. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
