Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2025-02-20 Thread via GitHub
jeesou commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1964871553 ## api/src/main/java/org/apache/iceberg/Table.java: ## @@ -373,4 +374,14 @@ default Snapshot snapshot(String name) { return null; } + + /** + * Returns th

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2025-01-26 Thread via GitHub
jeesou commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1930017945 ## api/src/main/java/org/apache/iceberg/Table.java: ## @@ -373,4 +374,14 @@ default Snapshot snapshot(String name) { return null; } + + /** + * Returns th

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2025-01-06 Thread via GitHub
jeesou commented on PR #11040: URL: https://github.com/apache/iceberg/pull/11040#issuecomment-2574390763 Hi @karuppayya @amogh-jahagirdar friendly reminder, please check the comments once. -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-12-09 Thread via GitHub
jeesou commented on PR #11040: URL: https://github.com/apache/iceberg/pull/11040#issuecomment-2530451996 Hi @karuppayya @amogh-jahagirdar could you please have a look at the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-27 Thread via GitHub
jeesou commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1861546270 ## api/src/main/java/org/apache/iceberg/Table.java: ## @@ -373,4 +374,14 @@ default Snapshot snapshot(String name) { return null; } + + /** + * Returns th

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-27 Thread via GitHub
jeesou commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1861546270 ## api/src/main/java/org/apache/iceberg/Table.java: ## @@ -373,4 +374,14 @@ default Snapshot snapshot(String name) { return null; } + + /** + * Returns th

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1861360140 ## api/src/main/java/org/apache/iceberg/Table.java: ## @@ -373,4 +374,14 @@ default Snapshot snapshot(String name) { return null; } + + /** + *

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1861360140 ## api/src/main/java/org/apache/iceberg/Table.java: ## @@ -373,4 +374,14 @@ default Snapshot snapshot(String name) { return null; } + + /** + *

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1861360067 ## api/src/main/java/org/apache/iceberg/Table.java: ## @@ -373,4 +374,14 @@ default Snapshot snapshot(String name) { return null; } + + /** + *

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-27 Thread via GitHub
amogh-jahagirdar commented on PR #11040: URL: https://github.com/apache/iceberg/pull/11040#issuecomment-2503996649 Sorry for the delay @jeesou @karuppayya , this is on my list today for review -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-26 Thread via GitHub
jeesou commented on PR #11040: URL: https://github.com/apache/iceberg/pull/11040#issuecomment-2502877154 Hi @karuppayya , @amogh-jahagirdar kindly check the comment above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-14 Thread via GitHub
jeesou commented on PR #11040: URL: https://github.com/apache/iceberg/pull/11040#issuecomment-2477244116 Hi @karuppayya , @amogh-jahagirdar as per our discussion to introduce a config to let users decide if they are fine with best effort search, I was thinking of adding a kind of threshold

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-11 Thread via GitHub
github-actions[bot] commented on PR #11040: URL: https://github.com/apache/iceberg/pull/11040#issuecomment-2469334682 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-09-25 Thread via GitHub
karuppayya commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1775572057 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,9 +195,9 @@ protected Statistics estimateStatistics(Snapshot snapshot)

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-09-20 Thread via GitHub
amogh-jahagirdar commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1769162105 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,9 +195,9 @@ protected Statistics estimateStatistics(Snapshot sna

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-09-20 Thread via GitHub
amogh-jahagirdar commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1769162105 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java: ## @@ -194,9 +195,9 @@ protected Statistics estimateStatistics(Snapshot sna

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-09-12 Thread via GitHub
karuppayya commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1757325988 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java: ## @@ -293,6 +293,73 @@ public void testTableWithTwoColStats() throws NoSuch

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-09-11 Thread via GitHub
jeesou commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1756152681 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java: ## @@ -293,6 +293,73 @@ public void testTableWithTwoColStats() throws NoSuchTabl

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-09-11 Thread via GitHub
jeesou commented on PR #11040: URL: https://github.com/apache/iceberg/pull/11040#issuecomment-2345310369 LGTM I have picked up the PR changes and tested it out, Its working fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-08-30 Thread via GitHub
huaxingao commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1739562317 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java: ## @@ -318,46 +322,44 @@ public void testMultipleSnapshotsWithColStats() throw

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-08-30 Thread via GitHub
huaxingao commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1738093741 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java: ## @@ -293,6 +293,73 @@ public void testTableWithTwoColStats() throws NoSuchT

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-08-29 Thread via GitHub
amogh-jahagirdar commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1737878864 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java: ## @@ -293,6 +293,73 @@ public void testTableWithTwoColStats() throws

[PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-08-28 Thread via GitHub
karuppayya opened a new pull request, #11040: URL: https://github.com/apache/iceberg/pull/11040 Use the statistics of the snapshot being scanned, instead of the first statistics file. @huaxingao @RussellSpitzer @aokolnychyi Please help review -- This is an automated message from t