[GitHub] [iceberg] danielcweeks merged pull request #6674: Add support for special characters in snowflake identifiers for Snowflake Catalog

danielcweeks merged PR #6674: URL: https://github.com/apache/iceberg/pull/6674 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

[GitHub] [iceberg] Fokko merged pull request #6747: Build: Bump moto from 4.1.0 to 4.1.2 in /python

Fokko merged PR #6747: URL: https://github.com/apache/iceberg/pull/6747 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096628616 ## spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java: ## @@ -459,9 +492,157 @@ public void testAggregatePushDownForTimeTravel() {

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096627383 ## spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java: ## @@ -0,0 +1,467 @@ +/* + * Licensed to the Apache Software Foundation (AS

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096627360 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -158,6 +182,150 @@ public Filter[] pushedFilters() { return pushed

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096627307 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -158,6 +182,150 @@ public Filter[] pushedFilters() { return pushed

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096627292 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -158,6 +182,150 @@ public Filter[] pushedFilters() { return pushed

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096627279 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java: ## @@ -243,4 +243,15 @@ public boolean preserveDataGrouping() { .defaultValue(

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096627266 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java: ## @@ -243,4 +243,15 @@ public boolean preserveDataGrouping() { .defaultValue(

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096627249 ## api/src/main/java/org/apache/iceberg/expressions/AggregateEvaluator.java: ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096627231 ## api/src/main/java/org/apache/iceberg/expressions/AggregateEvaluator.java: ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096627160 ## api/src/main/java/org/apache/iceberg/expressions/AggregateEvaluator.java: ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096627129 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -158,6 +182,141 @@ public Filter[] pushedFilters() { return pushed

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096627078 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -158,6 +182,141 @@ public Filter[] pushedFilters() { return pushed

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096627039 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -158,6 +182,141 @@ public Filter[] pushedFilters() { return pushed

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096626750 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -158,6 +182,141 @@ public Filter[] pushedFilters() { return pushed

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096626715 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -158,6 +182,141 @@ public Filter[] pushedFilters() { return pushed

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096626648 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -158,6 +182,141 @@ public Filter[] pushedFilters() { return pushed

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096626615 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkLocalScan.java: ## @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096626563 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java: ## @@ -47,4 +47,8 @@ private SparkSQLProperties() {} public static final String

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096626529 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkReadOptions.java: ## @@ -90,4 +90,6 @@ private SparkReadOptions() {} public static final String VER

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096626413 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -158,6 +182,141 @@ public Filter[] pushedFilters() { return pushed

[GitHub] [iceberg] huaxingao commented on a diff in pull request #6622: push down min/max/count to iceberg

huaxingao commented on code in PR #6622: URL: https://github.com/apache/iceberg/pull/6622#discussion_r1096626306 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -158,6 +182,130 @@ public Filter[] pushedFilters() { return pushed

[GitHub] [iceberg] danielcweeks opened a new pull request, #6749: Prevent RESTCatalog AuthSession from expiring

danielcweeks opened a new pull request, #6749: URL: https://github.com/apache/iceberg/pull/6749 When using OAuth with RESTCatalog, the catalog's auth session is returned by `newSession` if credentials are not provided. This results in the main auth session being cached and eventually expir

[GitHub] [iceberg] Fokko merged pull request #6748: Build: Bump pre-commit from 3.0.1 to 3.0.4 in /python

Fokko merged PR #6748: URL: https://github.com/apache/iceberg/pull/6748 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] dependabot[bot] opened a new pull request, #6748: Build: Bump pre-commit from 3.0.1 to 3.0.4 in /python

dependabot[bot] opened a new pull request, #6748: URL: https://github.com/apache/iceberg/pull/6748 Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 3.0.1 to 3.0.4. Release notes Sourced from https://github.com/pre-commit/pre-commit/releases";>pre-commit's releases

[GitHub] [iceberg] dependabot[bot] opened a new pull request, #6747: Build: Bump moto from 4.1.0 to 4.1.2 in /python

dependabot[bot] opened a new pull request, #6747: URL: https://github.com/apache/iceberg/pull/6747 Bumps [moto](https://github.com/getmoto/moto) from 4.1.0 to 4.1.2. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog. 4.1.2

[GitHub] [iceberg] yabola commented on pull request #6742: support registerTable in GlueCatalog

yabola commented on PR #6742: URL: https://github.com/apache/iceberg/pull/6742#issuecomment-1416903235 @jackye1995 Thanks for pinging me. I agree with you : there is no requirement in a recovery use case, but this can be a requirement in automatic switch of metadata location. But I am not

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

jackye1995 commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1096608183 ## aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java: ## @@ -1314,55 +1294,27 @@ private void configureEndpoint(T builder, String en } } - @Vi

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

jackye1995 commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1096608183 ## aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java: ## @@ -1314,55 +1294,27 @@ private void configureEndpoint(T builder, String en } } - @Vi

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

jackye1995 commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1096608183 ## aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java: ## @@ -1314,55 +1294,27 @@ private void configureEndpoint(T builder, String en } } - @Vi

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

jackye1995 commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1096607921 ## aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java: ## @@ -366,36 +362,48 @@ public class AwsProperties implements Serializable { */ public static

[GitHub] [iceberg] jackye1995 commented on a diff in pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

jackye1995 commented on code in PR #6746: URL: https://github.com/apache/iceberg/pull/6746#discussion_r1096607340 ## aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java: ## @@ -366,36 +362,48 @@ public class AwsProperties implements Serializable { */ public static

[GitHub] [iceberg] github-actions[bot] commented on issue #5453: Issue after migrating to Spark 3.3.0 and Iceberg 14.0

github-actions[bot] commented on issue #5453: URL: https://github.com/apache/iceberg/issues/5453#issuecomment-1416882230 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[GitHub] [iceberg] JonasJ-ap opened a new pull request, #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

JonasJ-ap opened a new pull request, #6746: URL: https://github.com/apache/iceberg/pull/6746 ## Problem Addressed This PR fix the problem described in issue #6715 by using reflection to instantiate the httpclient configuration impl class to avoid runtime deps of both `url-connection-clie

[GitHub] [iceberg] srilman opened a new pull request, #6745: Python: Use Version Ranges for Various Dependencies

srilman opened a new pull request, #6745: URL: https://github.com/apache/iceberg/pull/6745 As discussed in #6620, this PR uses version ranges for some dependencies. This will unfix the versions for certain dependencies and allow users to use older versions when install PyIceberg to use as a

[GitHub] [iceberg] jackye1995 commented on pull request #6742: support registerTable in GlueCatalog

jackye1995 commented on PR #6742: URL: https://github.com/apache/iceberg/pull/6742#issuecomment-1416812241 Nice, I anticipated similar concerns as in that thread, that's why I'd like to just put it up and see how the community reacts to this. I think the conversation there was around

[GitHub] [iceberg] Fokko commented on issue #6475: Python: Improve PyArrow performance

Fokko commented on issue #6475: URL: https://github.com/apache/iceberg/issues/6475#issuecomment-1416747817 https://github.com/apache/arrow/pull/34015 Has just been merged. This will reduce the IO overhead by removing an unnecessary call. -- This is an automated message from the Apache Git

[GitHub] [iceberg] ajantha-bhat commented on pull request #6742: support registerTable in GlueCatalog

ajantha-bhat commented on PR #6742: URL: https://github.com/apache/iceberg/pull/6742#issuecomment-1416714745 - allowing GlueDataCatalog registerTable API for exiting Table There is also related on going work https://github.com/apache/iceberg/pull/5327 -- This is an automated mes

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6716: Spark 3.3: Implement Position Deletes Table

szehon-ho commented on code in PR #6716: URL: https://github.com/apache/iceberg/pull/6716#discussion_r1096506568 ## core/src/main/java/org/apache/iceberg/MetadataTable.java: ## @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contri

[GitHub] [iceberg] singhpk234 commented on pull request #6655: Spark: Handle ResolvingFileIO while determining LocalityPreference

singhpk234 commented on PR #6655: URL: https://github.com/apache/iceberg/pull/6655#issuecomment-1416694194 Thanks @jackye1995 @aokolnychyi @amogh-jahagirdar for the reviews ! > @singhpk234, would you be interested to cherry-pick this change to other query engine versions? sure

[GitHub] [iceberg] singhpk234 opened a new pull request, #6744: Spark: Backport handling ResolvingFileIO in determining locality - PR-6655

singhpk234 opened a new pull request, #6744: URL: https://github.com/apache/iceberg/pull/6744 Backports https://github.com/apache/iceberg/pull/6655 to Spark 2.4, 3.1, 3.2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [iceberg] jackye1995 commented on pull request #6742: support registerTable in GlueCatalog

jackye1995 commented on PR #6742: URL: https://github.com/apache/iceberg/pull/6742#issuecomment-1416693030 > https://github.com/apache/iceberg/pull/6591 already fixed this part for Glue too right? great, did not notice that one! In that case I think the only missing feature is

[GitHub] [iceberg] singhpk234 opened a new pull request, #6743: Flink: Backport handling ResolvingFileIO in determining locality - PR 6655

singhpk234 opened a new pull request, #6743: URL: https://github.com/apache/iceberg/pull/6743 Backports https://github.com/apache/iceberg/pull/6655 to Flink 1.14, 1.15 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th