slachiewicz commented on code in PR #2487:
URL: https://github.com/apache/iceberg-go/pull/2487#discussion_r4578561343


##########
table/arrow_scanner.go:
##########
@@ -3069,10 +3111,10 @@ func (as *arrowScan) recordsFromTask(ctx 
context.Context, task tblutils.Enumerat
                        return 
array.NewRecordBatch(invariants.outputArrowSchema, r.Columns(), r.NumRows()), 
nil
                }
 
-               return ToRequestedSchema(ctx, as.projectedSchema, readSchema, 
r, SchemaOptions{
+               return projectToRequestedSchema(ctx, as.projectedSchema, 
readSchema, r, SchemaOptions{
                        UseLargeTypes:   as.useLargeTypes,
                        TableProperties: invariants.tableProperties,
-               })
+               }, nil, partitionConstants)
        })
 
        err = as.processRecordsWithPlans(ctx, task, iceSchema, rowFilter, rdr, 
colIndices, pipeline, posSource, sink, filterPlans)

Review Comment:
   Fixed in 8a6e0d6: a predicate on a field the file leaves to its identity 
partition is folded to its result for that file's value, for the row filter, 
row-group pruning and `producePosDeletesFromTask`. Those files skip the plans 
cached by file schema. `TestDeleteOnIdentityPartitionColumnMissingFromDataFile` 
covers copy-on-write and merge-on-read; the merge-on-read case removed id 1 
from east before.



##########
table/arrow_scanner.go:
##########
@@ -3053,6 +3094,7 @@ func (as *arrowScan) recordsFromTask(ctx context.Context, 
task tblutils.Enumerat
                pipeline = append(pipeline, filterFunc)
        }
 
+       partitionConstants := identityPartitionConstants(as.metadata, 
task.Value.File)

Review Comment:
   Fixed in 8a6e0d6: that step now uses `projectToRequestedSchema` with the 
partition constants. 
`TestEqualityDeleteOnIdentityPartitionColumnMissingFromDataFile` covers it.



##########
table/arrow_utils.go:
##########
@@ -1276,7 +1280,12 @@ func (a *arrowProjectionVisitor) Struct(st 
iceberg.StructType, structArr arrow.A
                        dt := a.typeToArrowType(field.Type)
                        alloc := compute.GetAllocator(a.ctx)
 
+                       constant, hasConstant := a.partitionConstants[field.ID]
                        switch {
+                       case hasConstant && constant != nil:
+                               arr = defaultToArray(constant, field.Type, dt, 
structArr.Len(), alloc)

Review Comment:
   Fixed in 8a6e0d6: the value is converted to `field.Type` before it fills the 
column or is compared. 
`TestScanWidensPromotedIdentityPartitionColumnMissingFromDataFile` panicked 
before.



##########
table/arrow_scanner.go:
##########
@@ -2836,6 +2836,47 @@ func pruningFilterHasMissingInitialDefault(
        return false, nil
 }
 
+// identityPartitionConstants returns the data file's identity partition
+// values keyed by source field ID. A writer may leave an identity partition
+// column out of its data files, as Delta and Hive-style layouts do; the spec's
+// column projection rule 1 reads such a field from the partition value.
+func identityPartitionConstants(meta Metadata, file iceberg.DataFile) 
map[int]any {
+       if meta == nil || file == nil {
+               return nil
+       }
+       spec := meta.PartitionSpecByID(int(file.SpecID()))

Review Comment:
   Added in 8a6e0d6 as 
`TestIdentityPartitionValueUsesFileSpecAndWinsOverInitialDefault`: files 
written before the partition field was removed, a null partition value, and a 
later file that takes the default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to