zeroshade commented on code in PR #2597:
URL: https://github.com/apache/iceberg-go/pull/2597#discussion_r4603766994
##########
table/internal/parquet_files.go:
##########
@@ -1927,17 +1927,35 @@ func resolveParquetStatsColumns(meta
*metadata.FileMetaData, statsCols map[int]S
return columns
}
+// ParquetColumnStatsPresence reads the optional fields from the footer because
+// Arrow's typed statistics can report an absent null count as zero or a
+// one-sided bound as a min/max pair.
+func ParquetColumnStatsPresence(meta *metadata.FileMetaData, rowGroup, column
int) (nullCount, bounds bool) {
+ chunk := meta.RowGroups[rowGroup].Columns[column].MetaData
Review Comment:
This reads the raw footer `ColumnChunk.MetaData`. For a column encrypted
with its own key, arrow-go's `ColumnChunkMetaData` reads stats from the
decrypted `ColumnMetaData`, but this field is nil (or has its stats stripped
under a plaintext footer). Such columns would report no null counts or bounds,
so they get no pruning, and imports partitioned on them would panic. That's
conservative, not wrong, and footer-key (uniform) encryption is fine. Could you
add a comment noting the limitation, or fall back to the typed stats when
`MetaData` is nil?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]