amogh-jahagirdar commented on code in PR #18925:
URL: https://github.com/apache/iceberg/pull/18925#discussion_r4565018381


##########
core/src/main/java/org/apache/iceberg/TrackedFileAdapters.java:
##########
@@ -583,7 +597,189 @@ private void wrapInternal(DataFile newFile) {
           newFile.content());
 
       this.file = newFile;
-      this.stats = hasContentStats(newFile) ? statsWrapper.wrap(newFile) : 
null;
+      this.stats = wrapStats(fileWithPartitionSourceStats());
+    }
+
+    private ContentStats wrapStats(DataFile dataFile) {
+      return hasContentStats(dataFile) ? statsWrapper.wrap(dataFile) : null;
+    }
+
+    private DataFile fileWithPartitionSourceStats() {
+      PartitionSpec spec = specsById.get(file.specId());
+      Preconditions.checkArgument(
+          spec != null, "Cannot find partition spec for spec ID: %s", 
file.specId());
+
+      Set<Integer> partitionSourceIds = Sets.newHashSet();
+      Set<Integer> floatingPointSourceIds = Sets.newHashSet();
+      for (PartitionField field : spec.fields()) {
+        if (field.transform().isVoid()) {
+          continue;
+        }
+
+        Preconditions.checkArgument(
+            field.transform().preservesOrder(),

Review Comment:
   Once we support stats-only fields, a `partition-value` field can store the 
bucket output directly. Its type is the partition field's result type, so we 
don't need the source type. The adapter would take the table's stats fields 
alongside the specs and fill the `partition-value` field from the partition 
tuple. `MapBackedContentStats` would also need to resolve the stats-only field 
types, since right now it only looks up types in the table schema.
   



##########
core/src/main/java/org/apache/iceberg/TrackedFileAdapters.java:
##########
@@ -583,7 +597,189 @@ private void wrapInternal(DataFile newFile) {
           newFile.content());
 
       this.file = newFile;
-      this.stats = hasContentStats(newFile) ? statsWrapper.wrap(newFile) : 
null;
+      this.stats = wrapStats(fileWithPartitionSourceStats());
+    }
+
+    private ContentStats wrapStats(DataFile dataFile) {
+      return hasContentStats(dataFile) ? statsWrapper.wrap(dataFile) : null;
+    }
+
+    private DataFile fileWithPartitionSourceStats() {
+      PartitionSpec spec = specsById.get(file.specId());
+      Preconditions.checkArgument(
+          spec != null, "Cannot find partition spec for spec ID: %s", 
file.specId());
+
+      Set<Integer> partitionSourceIds = Sets.newHashSet();
+      Set<Integer> floatingPointSourceIds = Sets.newHashSet();
+      for (PartitionField field : spec.fields()) {
+        if (field.transform().isVoid()) {
+          continue;
+        }
+
+        Preconditions.checkArgument(
+            field.transform().preservesOrder(),
+            "Cannot convert data file %s: partition field does not preserve 
order: %s",
+            file.location(),
+            field);
+        Type sourceType = tableSchema.findType(field.sourceId());
+        Preconditions.checkArgument(
+            sourceType != null,

Review Comment:
   Users can drop columns that are still referenced by historical partition 
specs. After the drop, those specs resolve the source column's type as 
`unknown`, so for files written under them we can't fill the source column 
bounds from the partition value. For identity and truncate the partition 
value's type is `unknown` too. For year, month, day and hour the partition type 
is fixed, but the source could be a date or any of the timestamp types, and 
their bounds are encoded in different units. Ideally we'd get the source 
column's type from an older schema, but expire snapshots can technically remove 
every schema that has it while files under that spec are still live. Arguably 
that's an issue in expire snapshots, because it keeps a spec while removing the 
schemas that spec relies on.
   
   To address this we could:
   - Fix expire snapshots to keep the schemas a live spec relies on, and pass 
the table's schemas by ID into `TrackedFileAdapters.forDataFile` along with the 
specs, so the adapter can take the source type from the latest schema that has 
the column.
   - Store the partition value directly as a stats-only `partition-value` 
field. This works for any transform, since its type is the partition field's 
result type.
   - Least preferable: require users to rewrite these specific files and their 
equality deletes, so that no live file is written under a spec with a dropped 
source column.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to