Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

github-actions[bot] closed pull request #11615: Spark : Derive Stats From Manifest on the Fly URL: https://github.com/apache/iceberg/pull/11615 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

github-actions[bot] commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2702375969 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

github-actions[bot] commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2686493099 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

huaxingao commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2615031769 Sorry, I've been really busy recently. I'll try to take a look soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

jeesou commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2614906052 HI @huaxingao , @RussellSpitzer, kindly have a look at the PR and suggest any update if there. -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

guykhazma commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2574163828 @huaxingao @RussellSpitzer friendly remainder, can you please review this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2532200924 Hi @RussellSpitzer ,@huaxingao can u please review the pr once, Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

guykhazma commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2511446282 @RussellSpitzer @huaxingao Just a friendly reminder to review the changes when you have a chance. Thanks! -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2503833787 Hi @RussellSpitzer @huaxingao , We’ve updated the implementation to use AggregateEvaluator from the aggregate pushdown code. In summary, we prepare a list of expressions for min, ma

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

guykhazma commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2502221330 @huaxingao yes, it is possible to reuse the logic from the aggregate pushdown by reusing the AggregateEvaluator instead of the current code to aggregate from the manifests. Something

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

huaxingao commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2499259216 > do you have any thoughts on this? I know you have dealt with similar issues before on the Aggregate pushdowns. @RussellSpitzer Sorry I just saw this. I disable aggregate p

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1855092790 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,10 +205,40 @@ protected Statistics estimateStatistics(Snapshot snaps

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1854320423 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,10 +205,40 @@ protected Statistics estimateStatistics(Snapshot snaps

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

guykhazma commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1854912088 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -248,6 +296,88 @@ protected Statistics estimateStatistics(Snapshot snapshot)

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

guykhazma commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1854912088 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -248,6 +296,88 @@ protected Statistics estimateStatistics(Snapshot snapshot)

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

RussellSpitzer commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1854422282 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,10 +205,40 @@ protected Statistics estimateStatistics(Snapshot sn

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1854296151 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,10 +205,40 @@ protected Statistics estimateStatistics(Snapshot snaps

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1854296151 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,10 +205,40 @@ protected Statistics estimateStatistics(Snapshot snaps

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1854296151 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,10 +205,40 @@ protected Statistics estimateStatistics(Snapshot snaps

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1854320423 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,10 +205,40 @@ protected Statistics estimateStatistics(Snapshot snaps

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1854296151 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,10 +205,40 @@ protected Statistics estimateStatistics(Snapshot snaps

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1854296151 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,10 +205,40 @@ protected Statistics estimateStatistics(Snapshot snaps

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1854296151 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,10 +205,40 @@ protected Statistics estimateStatistics(Snapshot snaps

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2492078503 @RussellSpitzer, thanks for the review comments,I will address them soon. As per @huaxingao implementation [here](https://github.com/apache/iceberg/blob/90be5d7360bc7ff274e7d00cb725

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

RussellSpitzer commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1852437532 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,10 +205,40 @@ protected Statistics estimateStatistics(Snapshot sn

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

RussellSpitzer commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1852423636 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -183,6 +193,7 @@ public Statistics estimateStatistics() { return e

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

RussellSpitzer commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1852425556 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java: ## @@ -355,4 +355,21 @@ public boolean reportColumnStats() { .defaultVa

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

RussellSpitzer commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1852421521 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,10 +205,40 @@ protected Statistics estimateStatistics(Snapshot sn

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

RussellSpitzer commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1852418237 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -248,6 +296,88 @@ protected Statistics estimateStatistics(Snapshot snap

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

RussellSpitzer commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1852416780 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -248,6 +296,88 @@ protected Statistics estimateStatistics(Snapshot snap

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

RussellSpitzer commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1852403873 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -248,6 +296,88 @@ protected Statistics estimateStatistics(Snapshot snap

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

RussellSpitzer commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1852396291 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -248,6 +296,88 @@ protected Statistics estimateStatistics(Snapshot snap

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

RussellSpitzer commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1852391179 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,10 +205,40 @@ protected Statistics estimateStatistics(Snapshot sn

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

RussellSpitzer commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1852388937 ## core/src/main/java/org/apache/iceberg/TableProperties.java: ## @@ -388,4 +388,8 @@ private TableProperties() {} public static final int ENCRYPTION_DEK_LE

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on code in PR #11615: URL: https://github.com/apache/iceberg/pull/11615#discussion_r1852219386 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java: ## @@ -355,4 +355,21 @@ public boolean reportColumnStats() { .defaultValue

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

saitharun15 commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2490258926 Hi, @huaxingao @karuppayya @aokolnychyi @RussellSpitzer Can you help review this PR -- This is an automated message from the Apache Git Service. To respond to the message, please