laskoviymishka commented on code in PR #2597:
URL: https://github.com/apache/iceberg-go/pull/2597#discussion_r4596798759


##########
table/internal/parquet_files.go:
##########
@@ -1927,17 +1927,35 @@ func resolveParquetStatsColumns(meta 
*metadata.FileMetaData, statsCols map[int]S
        return columns
 }
 
+// ParquetColumnStatsPresence reads the optional fields from the footer because
+// Arrow's typed statistics can report an absent null count as zero or a
+// one-sided bound as a min/max pair.
+func ParquetColumnStatsPresence(meta *metadata.FileMetaData, rowGroup, column 
int) (nullCount, bounds bool) {
+       chunk := meta.RowGroups[rowGroup].Columns[column].MetaData
+       if chunk == nil || chunk.Statistics == nil {
+               return false, false
+       }
+       stats := chunk.Statistics
+       if meta.Schema.Column(column).ColumnOrder() == 
parquet.ColumnOrders.TypeDefinedOrder {
+               return stats.IsSetNullCount(), stats.IsSetMinValue() && 
stats.IsSetMaxValue()
+       }
+
+       return stats.IsSetNullCount(), stats.IsSetMin() && stats.IsSetMax()

Review Comment:
   Could the import test also reach this deprecated `Min`/`Max` branch? 
`writePartialMetricsParquet` writes through pqarrow, which sets `ColumnOrders`, 
so only the `TypeDefinedOrder` branch runs 
([parquet_import_metrics_test.go#L72-L90](https://github.com/apache/iceberg-go/blob/4ab989f69e894c7fce6fa9acd8d9c96c3d3c9708/table/parquet_import_metrics_test.go#L72-L90)).
 A case with `ColumnOrders` cleared would cover it, and a string or binary 
source column would help too, since every case uses the single Int32 `amount`.



##########
table/internal/utils.go:
##########
@@ -321,6 +329,10 @@ func (d *DataFileStatistics) ToDataFile(opts DataFileOpts) 
iceberg.DataFile {
                                // prioritizing caller-supplied value.
                                fieldIDToPartitionData[field.FieldID] = 
partitionVal
                        case field.Transform.PreservesOrder() && 
sourceFieldFound:
+                               if _, incomplete := 
d.incompleteBounds[field.FirstSourceID()]; incomplete {

Review Comment:
   Should the `!set` and `stats == nil` branches also put the column into 
`incompleteBounds`, with a test next to 
`TestPartialParquetBoundsPartitionInference`? Those branches only write 
`invalidateCol` 
([parquet_files.go#L2003-L2019](https://github.com/apache/iceberg-go/blob/4ab989f69e894c7fce6fa9acd8d9c96c3d3c9708/table/internal/parquet_files.go#L2003-L2019)),
 so this guard never fires. With `Statistics = nil` on one row group or both, 
`FileToDataFile` returns `partition[1000]=<nil>` and no error. `main` does the 
same, but this PR is where partition inference starts requiring complete bounds.



##########
table/internal/utils.go:
##########
@@ -231,16 +231,24 @@ type DataFileStatistics struct {
        NullValueCounts  map[int]int64
        NanValueCounts   map[int]int64
        ColAggs          map[int]StatsAgg
+       incompleteBounds map[int]StatsAgg

Review Comment:
   Could we document the two meanings this map carries? `ToDataFile` checks 
only that the key is present, while `PartitionValue` reads the value, which is 
nil when no row group had bounds 
([parquet_files.go#L2079-L2083](https://github.com/apache/iceberg-go/blob/4ab989f69e894c7fce6fa9acd8d9c96c3d3c9708/table/internal/parquet_files.go#L2079-L2083)).
 Switching the guard to a `!= nil` check would quietly drop it for those 
columns.
   
   ```suggestion
        // incompleteBounds holds source columns whose bounds are missing in at 
least
        // one row group. Key presence is what blocks partition inference; the 
value
        // is the partial aggregate from the other row groups and may be nil.
        incompleteBounds map[int]StatsAgg
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to