singhpk234 commented on code in PR #10943:
URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720266361


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java:
##########
@@ -407,14 +422,35 @@ public Scan build() {
 
   private Scan buildBatchScan() {
     Schema expectedSchema = schemaWithMetadataColumns();
+    org.apache.iceberg.Scan scan =
+        buildIcebergBatchScan(false /* not include Column Stats */, 
expectedSchema);
+    if (pushedLimit > 0 && hasDeletes(scan)) {
+      LOG.info("Skipping limit pushdown: detected row level deletes");
+      pushedLimit = -1;
+    }

Review Comment:
   Was wondering if the filter is partition filter or no filter at all and we 
know the amount of files we are gonna scan and have enough confidence that we 
will get those number of records can we set this to false then, hinting spark 
that global limit is not required, though it will be a very niche use case and 
maybe not worth it 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org

Reply via email to