hemanthboyina commented on code in PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#discussion_r3106981398
##########
core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java:
##########
@@ -559,6 +564,75 @@ private static List<PartitionStatistics>
sortStatsByPartition(
return entries;
}
+ /**
+ * Populates {@code total_record_count} for partitions where it can be
reliably computed from
+ * metadata without scanning data files.
+ *
+ * <p>Per the spec, {@code total_record_count} should stay optional (NULL
means unknown). It can
+ * be computed when a partition has no equality deletes and no V2 position
delete files — i.e.,
+ * deletes are tracked exclusively via deletion vectors (DVs) or the
partition has no deletes at
+ * all. In that case, the total is: {@code data_record_count -
position_delete_record_count}.
+ *
+ * <p>DV cardinalities are exact (each position maps to one deleted row), so
the subtraction
+ * yields a precise total without reading any data files.
+ *
+ * @param stats the computed partition statistics to enrich
+ */
+ @VisibleForTesting
+ static List<PartitionStatistics>
populateTotalRecordCount(Collection<PartitionStatistics> stats) {
+ List<PartitionStatistics> result =
Lists.newArrayListWithExpectedSize(stats.size());
+ for (PartitionStatistics partitionStats : stats) {
+ boolean hasEqualityDeletes =
+ partitionStats.equalityDeleteRecordCount() != null
+ && partitionStats.equalityDeleteRecordCount() > 0;
+ boolean hasV2PositionDeleteFiles =
+ partitionStats.positionDeleteFileCount() != null
+ && partitionStats.positionDeleteFileCount() > 0;
+
+ // Create a fresh BasePartitionStatistics to avoid projection issues on
read-back objects
Review Comment:
BasePartitionStatistics#internalSet has a null check
if (value == null)
{ return; }
this skips null values. I think this exists to prevent NPE from unboxing
required premitive fields. But it means we cannot reset total_record_count back
to null on objects that already have a non-null value from a previous stats
computation. Fresh BasePartitionStatistics objects start with totalRecordCount
= null by default, so we copy all fields and only set total_record_count when
it's computable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
