singhpk234 commented on code in PR #12270:
URL: https://github.com/apache/iceberg/pull/12270#discussion_r1972008956
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RemoveDanglingDeletesSparkAction.java:
##########
@@ -156,7 +162,12 @@ private List<DeleteFile> findDanglingDeletes() {
.or(
col("data_file.content")
.equalTo("2")
-
.and(col("sequence_number").$less$eq(col("min_data_sequence_number"))));
+
.and(col("sequence_number").$less$eq(col("min_data_sequence_number"))))
+ // dvs pointing to non-existing data files
+ .or(
+ col("data_file.file_format")
+ .equalTo(FileFormat.PUFFIN.name())
Review Comment:
I agree, and this is not a blocker, just brought up to see if we can handle
this. While this may not be an issue for java impl for iceberg DV but can this
happen or be potentially missed in other language implementation. If we put
this in a diff way like this, does iceberg consider a file with
's3://<tbl_location>/data-filea.parquet' and
's3a://<tbl_location>/data-filea.parquet' same, should we consider they are
handled same way through out ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
