Re: [PR] Spark partial limit push down [iceberg]

github-actions[bot] commented on PR #10943: URL: https://github.com/apache/iceberg/pull/10943#issuecomment-2558674238 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Spark partial limit push down [iceberg]

github-actions[bot] closed pull request #10943: Spark partial limit push down URL: https://github.com/apache/iceberg/pull/10943 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Spark partial limit push down [iceberg]

github-actions[bot] commented on PR #10943: URL: https://github.com/apache/iceberg/pull/10943#issuecomment-2543386163 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1841589907 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/BaseRowReader.java: ## @@ -48,6 +50,17 @@ abstract class BaseRowReader extends BaseReader taskGro

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1841589128 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -405,14 +420,35 @@ public Scan build() { private Scan buildBatch

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1840791166 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -405,14 +420,35 @@ public Scan build() { private Scan build

Re: [PR] Spark partial limit push down [iceberg]

aokolnychyi commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1840750027 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -405,14 +420,35 @@ public Scan build() { private Scan buildBat

Re: [PR] Spark partial limit push down [iceberg]

aokolnychyi commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1840744978 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/BaseRowReader.java: ## @@ -48,6 +50,17 @@ abstract class BaseRowReader extends BaseReader taskG

Re: [PR] Spark partial limit push down [iceberg]

aokolnychyi commented on PR #10943: URL: https://github.com/apache/iceberg/pull/10943#issuecomment-2471409893 Ack, will try to get to this asap. I am partially off this week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Spark partial limit push down [iceberg]

aokolnychyi commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1838676561 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/BaseRowReader.java: ## @@ -48,6 +50,17 @@ abstract class BaseRowReader extends BaseReader taskG

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1838675045 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -405,14 +420,35 @@ public Scan build() { private Scan build

Re: [PR] Spark partial limit push down [iceberg]

aokolnychyi commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1838670199 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -405,14 +420,35 @@ public Scan build() { private Scan buildBat

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1837201737 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestLimitPushDown.java: ## @@ -0,0 +1,339 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1835113147 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestLimitPushDown.java: ## @@ -0,0 +1,339 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1835110026 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestLimitPushDown.java: ## @@ -0,0 +1,339 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1835109730 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestLimitPushDown.java: ## @@ -0,0 +1,339 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1835096536 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatch.java: ## @@ -114,16 +132,29 @@ private String[][] computePreferredLocations() {

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1779324357 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java: ## @@ -78,7 +78,19 @@ abstract class SparkPartitioningAwareScan e

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1779152923 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java: ## @@ -78,7 +78,19 @@ abstract class SparkPartitioningAwareS

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1778973376 ## parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java: ## @@ -1151,6 +1152,11 @@ public ReadBuilder withAADPrefix(ByteBuffer aadPrefix) { return t

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1778973584 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -407,14 +422,35 @@ public Scan build() { private Scan buildBatch

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1777470185 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -407,14 +422,35 @@ public Scan build() { private Scan build

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1777429830 ## parquet/src/main/java/org/apache/iceberg/parquet/VectorizedParquetReader.java: ## @@ -49,6 +49,7 @@ public class VectorizedParquetReader extends CloseableG

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1777471776 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -407,14 +422,35 @@ public Scan build() { private Scan build

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1777410265 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedColumnIterator.java: ## @@ -69,12 +69,20 @@ public boolean producesDictionaryEncod

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1777414232 ## parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java: ## @@ -1151,6 +1152,11 @@ public ReadBuilder withAADPrefix(ByteBuffer aadPrefix) { ret

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1721078438 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -407,14 +422,35 @@ public Scan build() { private Scan buildBatch

Re: [PR] Spark partial limit push down [iceberg]

singhpk234 commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720266361 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -407,14 +422,35 @@ public Scan build() { private Scan buildBatc

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720160710 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -407,14 +422,35 @@ public Scan build() { private Scan buildBatch

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720157880 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedColumnIterator.java: ## @@ -69,12 +69,20 @@ public boolean producesDictionaryEncodedVec

Re: [PR] Spark partial limit push down [iceberg]

singhpk234 commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720144903 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedColumnIterator.java: ## @@ -69,12 +69,20 @@ public boolean producesDictionaryEncodedVe

Re: [PR] Spark partial limit push down [iceberg]

singhpk234 commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720122775 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -407,14 +422,35 @@ public Scan build() { private Scan buildBatc

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720105551 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -407,14 +422,35 @@ public Scan build() { private Scan buildBatch

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720102633 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedColumnIterator.java: ## @@ -69,12 +69,20 @@ public boolean producesDictionaryEncodedVec

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720099765 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java: ## @@ -45,6 +45,10 @@ private SparkSQLProperties() {} "spark.sql.iceberg.

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720099440 ## parquet/src/main/java/org/apache/iceberg/parquet/VectorizedParquetReader.java: ## @@ -141,8 +148,15 @@ public T next() { advance(); } + lo

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720099159 ## parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java: ## @@ -1151,6 +1152,11 @@ public ReadBuilder withAADPrefix(ByteBuffer aadPrefix) { return t

Re: [PR] Spark partial limit push down [iceberg]

singhpk234 commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720037386 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java: ## @@ -45,6 +45,10 @@ private SparkSQLProperties() {} "spark.sql.iceberg

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720011663 ## parquet/src/main/java/org/apache/iceberg/parquet/VectorizedParquetReader.java: ## @@ -141,8 +148,15 @@ public T next() { advance(); } +

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720011247 ## parquet/src/main/java/org/apache/iceberg/parquet/VectorizedParquetReader.java: ## @@ -141,8 +148,15 @@ public T next() { advance(); } +

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r172122 ## parquet/src/main/java/org/apache/iceberg/parquet/VectorizedParquetReader.java: ## @@ -141,8 +148,15 @@ public T next() { advance(); } +

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r171179 ## parquet/src/main/java/org/apache/iceberg/parquet/VectorizedParquetReader.java: ## @@ -141,8 +148,15 @@ public T next() { advance(); } +

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1719987340 ## parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java: ## @@ -1151,6 +1152,11 @@ public ReadBuilder withAADPrefix(ByteBuffer aadPrefix) { ret

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1719977369 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java: ## @@ -117,7 +122,12 @@ private static class FileIterator implements CloseableIterator {

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1719976844 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedColumnIterator.java: ## @@ -69,12 +69,14 @@ public boolean producesDictionaryEncodedVec

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1719976513 ## parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java: ## @@ -1151,6 +1152,11 @@ public ReadBuilder withAADPrefix(ByteBuffer aadPrefix) { return t

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1719975918 ## .palantir/revapi.yml: ## @@ -1025,14 +1025,11 @@ acceptedBreaks: new: "class org.apache.iceberg.types.Types.NestedField" justification: "new Constr

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1719059739 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java: ## @@ -117,7 +122,12 @@ private static class FileIterator implements CloseableIterator

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1719058057 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedColumnIterator.java: ## @@ -69,12 +69,14 @@ public boolean producesDictionaryEncod

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r171905 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedColumnIterator.java: ## @@ -69,12 +69,14 @@ public boolean producesDictionaryEncod

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1719054286 ## parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java: ## @@ -1151,6 +1152,11 @@ public ReadBuilder withAADPrefix(ByteBuffer aadPrefix) { ret

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1719053493 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedColumnIterator.java: ## @@ -69,12 +69,14 @@ public boolean producesDictionaryEncod

Re: [PR] Spark partial limit push down [iceberg]

RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1719050871 ## .palantir/revapi.yml: ## @@ -1025,14 +1025,11 @@ acceptedBreaks: new: "class org.apache.iceberg.types.Types.NestedField" justification: "new C

Re: [PR] Spark partial limit push down [iceberg]

huaxingao commented on PR #10943: URL: https://github.com/apache/iceberg/pull/10943#issuecomment-2291604050 cc @aokolnychyi @RussellSpitzer @szehon-ho -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g