Re: [I] Spark SQL Extensions: Rewrite manifests [iceberg]

2024-01-19 Thread via GitHub
zeddit commented on issue #1598: URL: https://github.com/apache/iceberg/issues/1598#issuecomment-1901892857 @rdblue @aokolnychyi would spark `rewrite_manifests()` respects the order of data files according to column statistics for a sorted iceberg table. I found the engine of spark always

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1460271555 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -503,6 +541,78 @@ public boolean namespaceExists(Namespace namespace) { return JdbcUtil.name

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-19 Thread via GitHub
pvary commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1460271751 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceContinuous.java: ## @@ -367,6 +382,8 @@ public void testSpecificSnapshotTimestamp() th

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1460271555 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -503,6 +541,78 @@ public boolean namespaceExists(Namespace namespace) { return JdbcUtil.name

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-19 Thread via GitHub
pvary commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1460271365 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceContinuous.java: ## @@ -58,9 +61,11 @@ public class TestIcebergSourceContinuous {

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-19 Thread via GitHub
pvary commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1460271093 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/MiniClusterResource.java: ## @@ -50,4 +51,18 @@ public static MiniClusterWithClientResource createWithClasslo

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-19 Thread via GitHub
pvary commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1460268581 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/assigner/SplitAssigner.java: ## @@ -115,4 +115,7 @@ default void onCompletedSplits(Collection complete

Re: [I] [doc] Word `Gague ` spelled incorrectly in https://iceberg.apache.org/docs/latest/flink-writes/#metrics [iceberg]

2024-01-19 Thread via GitHub
amogh-jahagirdar commented on issue #9527: URL: https://github.com/apache/iceberg/issues/9527#issuecomment-1901747331 Thanks @Aireed ! PRs are welcome, feel free and we can review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Core: Cleanup assertion messages in partition spec tests [iceberg]

2024-01-19 Thread via GitHub
amogh-jahagirdar commented on PR #9528: URL: https://github.com/apache/iceberg/pull/9528#issuecomment-1901739205 Unrelated flaky test failure which is being tracked by https://github.com/apache/iceberg/issues/9511 ``` TestSparkExecutorCache > testMergeOnReadMerge() > catalogName = test

Re: [I] Error while using bucket partitions [iceberg]

2024-01-19 Thread via GitHub
github-actions[bot] commented on issue #274: URL: https://github.com/apache/iceberg/issues/274#issuecomment-1901378482 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Support partitioned writes [iceberg-python]

2024-01-19 Thread via GitHub
jqin61 commented on issue #208: URL: https://github.com/apache/iceberg-python/issues/208#issuecomment-1901345425 Based on the existing discussion, there are 3 major possible directions for detecting partitions and writing each partition in a multi-threaded way to maximize I/O. It seems ther

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-19 Thread via GitHub
syun64 commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1901289940 Makes sense @Fokko . Thank you very much for taking the time to lay all these options where a user may have to handle compatible or incompatible schema updates. As you sug

Re: [PR] Build: Bump ray from 2.7.1 to 2.9.0 [iceberg-python]

2024-01-19 Thread via GitHub
dependabot[bot] closed pull request #235: Build: Bump ray from 2.7.1 to 2.9.0 URL: https://github.com/apache/iceberg-python/pull/235 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Build: Bump ray from 2.7.1 to 2.9.0 [iceberg-python]

2024-01-19 Thread via GitHub
dependabot[bot] commented on PR #235: URL: https://github.com/apache/iceberg-python/pull/235#issuecomment-1901271367 Superseded by #287. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[PR] Build: Bump ray from 2.7.1 to 2.9.1 [iceberg-python]

2024-01-19 Thread via GitHub
dependabot[bot] opened a new pull request, #287: URL: https://github.com/apache/iceberg-python/pull/287 Bumps [ray](https://github.com/ray-project/ray) from 2.7.1 to 2.9.1. Release notes Sourced from https://github.com/ray-project/ray/releases";>ray's releases. Ray-2.9.1 T

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-19 Thread via GitHub
syun64 commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1901252873 Makes sense @Fokko . Thank you very much for taking the time to lay all these options out for the cases where a user may have to handle schema updates. As you suggested, I

Re: [PR] Core: Use correct headers for Multi-Table commits [iceberg]

2024-01-19 Thread via GitHub
danielcweeks commented on code in PR #9523: URL: https://github.com/apache/iceberg/pull/9523#discussion_r1459891880 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -1022,11 +1022,17 @@ public void commitTransaction(SessionContext context, List commi

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-19 Thread via GitHub
Fokko commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1901165986 > Evolving schema would be nice, also would it be possible to evolve partitions? e.g. after a specific overwrite I want to have schema evolution and possibility to replace the tab

Re: [I] Sql Catalog: Implement `_commit_table` [iceberg-python]

2024-01-19 Thread via GitHub
Fokko commented on issue #286: URL: https://github.com/apache/iceberg-python/issues/286#issuecomment-1901129265 Haven't updated my main :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] Sql Catalog: Implement `_commit_table` [iceberg-python]

2024-01-19 Thread via GitHub
Fokko closed issue #286: Sql Catalog: Implement `_commit_table` URL: https://github.com/apache/iceberg-python/issues/286 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] Update site-ci with github action credentials [iceberg]

2024-01-19 Thread via GitHub
Fokko merged PR #9525: URL: https://github.com/apache/iceberg/pull/9525 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[I] Sql Catalog: Implement `_commit_table` [iceberg-python]

2024-01-19 Thread via GitHub
Fokko opened a new issue, #286: URL: https://github.com/apache/iceberg-python/issues/286 ### Feature Request / Improvement This would be nice :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Adding Snowflake's public documentation [iceberg-docs]

2024-01-19 Thread via GitHub
scottteal commented on PR #297: URL: https://github.com/apache/iceberg-docs/pull/297#issuecomment-1901025384 @Fokko do you know when this will be merged and deployed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Query optimization fails after upgrading to 1.4.0+ with nullif in predicate [iceberg]

2024-01-19 Thread via GitHub
jakelong95 commented on issue #9518: URL: https://github.com/apache/iceberg/issues/9518#issuecomment-1900989318 Apologies, I edited out a quick test I had done that I actually ran on a cluster that is on EMR 6.14 so it's expected that my test would succeed instead of hit the reported error

Re: [I] Query optimization fails after upgrading to 1.4.0+ with nullif in predicate [iceberg]

2024-01-19 Thread via GitHub
jakelong95 commented on issue #9518: URL: https://github.com/apache/iceberg/issues/9518#issuecomment-1900986922 Here's the obfuscated schema from `explain` - ``` column1,timestamp column2,timestamp column3,timestamp column4,string this_is_the_problem_column,string column

Re: [PR] API, Core, Spark: Change behavior of fastForward/replace to create the from branch if it does not exist [iceberg]

2024-01-19 Thread via GitHub
amogh-jahagirdar commented on code in PR #9196: URL: https://github.com/apache/iceberg/pull/9196#discussion_r1459608457 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestFastForwardBranchProcedure.java: ## @@ -247,4 +234,39 @@ public void testF

Re: [PR] Core: Fix setting updated parquet compression property [iceberg]

2024-01-19 Thread via GitHub
amogh-jahagirdar commented on code in PR #9503: URL: https://github.com/apache/iceberg/pull/9503#discussion_r1459520414 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -90,10 +90,19 @@ private static Map unreservedProperties(Map rawP private static Map per

Re: [PR] Core: Fix setting updated parquet compression property [iceberg]

2024-01-19 Thread via GitHub
amogh-jahagirdar commented on code in PR #9503: URL: https://github.com/apache/iceberg/pull/9503#discussion_r1459520414 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -90,10 +90,19 @@ private static Map unreservedProperties(Map rawP private static Map per

Re: [I] AWS: Support setting description for Glue table [iceberg]

2024-01-19 Thread via GitHub
singhpk234 commented on issue #9462: URL: https://github.com/apache/iceberg/issues/9462#issuecomment-1900823638 sounds good, please feel free to submit a change. happy to review it ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] Core: Mark NoSuchViewException as CleanableFailure [iceberg]

2024-01-19 Thread via GitHub
amogh-jahagirdar merged PR #9516: URL: https://github.com/apache/iceberg/pull/9516 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] Query optimization fails after upgrading to 1.4.0+ with nullif in predicate [iceberg]

2024-01-19 Thread via GitHub
singhpk234 commented on issue #9518: URL: https://github.com/apache/iceberg/issues/9518#issuecomment-1900807210 This is interesting can you please share the table schema as well ? looks like we are now hitting an Assetion / Precondition which essentially means we are pushing something down

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-19 Thread via GitHub
syun64 commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1900741649 Thank you for the great points @Fokko and @nicor88 . Just like @nicor88 mentioned, I think RTAS will be slightly different from overwrite in the sense that the schema, the

Re: [I] Flaky test: TestSparkExecutorCache > testMergeOnReadDelete() [iceberg]

2024-01-19 Thread via GitHub
manuzhang commented on issue #9511: URL: https://github.com/apache/iceberg/issues/9511#issuecomment-1900672359 The error says the equality delete file doesn't match the predicate, ie `streamCount(deleteFile) <= 1`. ```java int maxRequestCount = mode == COPY_ON_WRITE ? 3 : 1;

Re: [PR] Arrow, AWS, Core: Remove deprecated code for 1.5.0 release [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9505: URL: https://github.com/apache/iceberg/pull/9505#discussion_r1459231764 ## aws/src/test/java/org/apache/iceberg/aws/TestAwsProperties.java: ## @@ -1,50 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more co

Re: [I] Saving Polars Dataframe as Parquet to Iceberg [iceberg-python]

2024-01-19 Thread via GitHub
syun64 commented on issue #226: URL: https://github.com/apache/iceberg-python/issues/226#issuecomment-1900604096 Thank you for raising this @asheeshgarg . To me, the issue seems to be that some of the "large" types are not yet accounted for within PyIceberg type conversions. https:/

Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]

2024-01-19 Thread via GitHub
syun64 commented on issue #278: URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1900574539 That sounds good @Fokko I think having a _CreateMappingFromPyArrowSchma preorder visitor does a good job of separating out the two concerns above. > I think the ou

Re: [PR] Arrow, AWS, Core: Remove deprecated code for 1.5.0 release [iceberg]

2024-01-19 Thread via GitHub
ajantha-bhat commented on code in PR #9505: URL: https://github.com/apache/iceberg/pull/9505#discussion_r1459114356 ## aws/src/test/java/org/apache/iceberg/aws/TestAwsProperties.java: ## @@ -1,50 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or m

Re: [PR] Arrow, AWS, Core: Remove deprecated code for 1.5.0 release [iceberg]

2024-01-19 Thread via GitHub
ajantha-bhat commented on code in PR #9505: URL: https://github.com/apache/iceberg/pull/9505#discussion_r1459114356 ## aws/src/test/java/org/apache/iceberg/aws/TestAwsProperties.java: ## @@ -1,50 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or m

Re: [PR] API, Core, Spark: Change behavior of fastForward/replace to create the from branch if it does not exist [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9196: URL: https://github.com/apache/iceberg/pull/9196#discussion_r1459108230 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestFastForwardBranchProcedure.java: ## @@ -247,4 +234,39 @@ public void testFastForward

Re: [I] using pyiceberg with kerberized hive metastore [iceberg-python]

2024-01-19 Thread via GitHub
Ohadkaiser commented on issue #135: URL: https://github.com/apache/iceberg-python/issues/135#issuecomment-1900508161 Hi! Was there any changes to this issue in the last months? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] Arrow, AWS, Core: Remove deprecated code for 1.5.0 release [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9505: URL: https://github.com/apache/iceberg/pull/9505#discussion_r1459086268 ## aws/src/test/java/org/apache/iceberg/aws/TestAwsProperties.java: ## @@ -1,50 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more co

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-01-19 Thread via GitHub
pvary commented on PR #9464: URL: https://github.com/apache/iceberg/pull/9464#issuecomment-1900420653 Started the discussion on the Flink dev list: https://lists.apache.org/thread/ocm6cj0h8o3wbwo7fz2l1b4odss750rk In the letter I collected my findings - basically the 64k limit is defined

Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]

2024-01-19 Thread via GitHub
Fokko commented on issue #278: URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1900388501 What do you think of the following approach: > I think the problem here is that we don't have an API like in Spark where we can [nicely hide things](https://github.com/apac

Re: [PR] Arrow, AWS, Core: Remove deprecated code for 1.5.0 release [iceberg]

2024-01-19 Thread via GitHub
ajantha-bhat commented on PR #9505: URL: https://github.com/apache/iceberg/pull/9505#issuecomment-1900313696 I pushed the changed from AWS module as a separate commit in the same PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458887667 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcViewOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458878847 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcViewOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458875264 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -26,6 +26,88 @@ public class TestJdbcUtil { + @Test + public void testGetTableOrViewSql() {

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458873570 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -26,6 +26,88 @@ public class TestJdbcUtil { + @Test + public void testGetTableOrViewSql() {

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458869663 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -303,6 +287,325 @@ public static Properties filterAndRemovePrefix(Map properties, S return resul

Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]

2024-01-19 Thread via GitHub
syun64 commented on issue #278: URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1900264983 Thank you for adding the context @Fokko :) Just so that we make sure new readers aren’t confused, do you think it’s fair to say that we are talking about ‘assigning fresh I

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458868551 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -303,6 +287,325 @@ public static Properties filterAndRemovePrefix(Map properties, S return resul

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458866257 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -303,6 +287,325 @@ public static Properties filterAndRemovePrefix(Map properties, S return resul

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458864815 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -303,6 +287,325 @@ public static Properties filterAndRemovePrefix(Map properties, S return resul

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458863205 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -303,6 +287,325 @@ public static Properties filterAndRemovePrefix(Map properties, S return resul

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458860528 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -303,6 +287,325 @@ public static Properties filterAndRemovePrefix(Map properties, S return resul

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458859034 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -303,6 +287,325 @@ public static Properties filterAndRemovePrefix(Map properties, S return resul

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458852399 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -303,6 +287,325 @@ public static Properties filterAndRemovePrefix(Map properties, S return resul

Re: [PR] Arrow, Core: Remove deprecated code for 1.5.0 release [iceberg]

2024-01-19 Thread via GitHub
ajantha-bhat commented on PR #9505: URL: https://github.com/apache/iceberg/pull/9505#issuecomment-1900247829 Only thing left is deprecated code in `AwsProperties`. But looks like it will be a big diff. I will give it a try. -- This is an automated message from the Apache Git Service. To

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458848983 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -376,6 +680,15 @@ static boolean namespaceExists( return true; } +if (exists( Revie

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458847484 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -303,6 +287,325 @@ public static Properties filterAndRemovePrefix(Map properties, S return resul

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458846882 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -303,6 +287,325 @@ public static Properties filterAndRemovePrefix(Map properties, S return resul

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458839129 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -503,6 +541,78 @@ public boolean namespaceExists(Namespace namespace) { return JdbcUtil.namesp

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458835826 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -503,6 +541,78 @@ public boolean namespaceExists(Namespace namespace) { return JdbcUtil.namesp

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458831926 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -503,6 +541,78 @@ public boolean namespaceExists(Namespace namespace) { return JdbcUtil.namesp

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458820768 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcViewCatalog.java: ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458821775 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcViewCatalog.java: ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-19 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1458821089 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcViewCatalog.java: ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] Build: Bump actions/upload-artifact from 3 to 4 [iceberg]

2024-01-19 Thread via GitHub
nastra merged PR #9319: URL: https://github.com/apache/iceberg/pull/9319 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-19 Thread via GitHub
nicor88 commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1900185062 > The next step would be to automatically evolve the schema as well (with flags for compatible/incompatible changes). @Fokko evolving schema would be nice, also would it be po

Re: [I] Add format check for toml files. [iceberg-rust]

2024-01-19 Thread via GitHub
liurenjie1024 closed issue #161: Add format check for toml files. URL: https://github.com/apache/iceberg-rust/issues/161 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [I] Add format check for toml files. [iceberg-rust]

2024-01-19 Thread via GitHub
liurenjie1024 commented on issue #161: URL: https://github.com/apache/iceberg-rust/issues/161#issuecomment-1900119336 Closed by #167 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Update deploy script to use remote branch and add 1.4.3 updates [iceberg]

2024-01-19 Thread via GitHub
Fokko merged PR #9519: URL: https://github.com/apache/iceberg/pull/9519 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-19 Thread via GitHub
Fokko commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1900076148 @syun64 thanks for raising this. With the freshly merged write support, we can do: ```python cat = load_catalog('default') tbl = cat.load_table('default.some_table')

Re: [PR] Add formatting for toml files [iceberg-rust]

2024-01-19 Thread via GitHub
Fokko merged PR #167: URL: https://github.com/apache/iceberg-rust/pull/167 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spark 3.5: Support specifying filter in RewriteManifestsProcedure [iceberg]

2024-01-19 Thread via GitHub
bknbkn commented on code in PR #9447: URL: https://github.com/apache/iceberg/pull/9447#discussion_r1458505983 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteManifestsProcedure.java: ## @@ -118,4 +126,15 @@ private InternalRow[] toOutputRows(RewriteM

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-01-19 Thread via GitHub
pvary commented on PR #8907: URL: https://github.com/apache/iceberg/pull/8907#issuecomment-1900024854 > * abstracting behaviors common to table and view makes sense but not a blocker for this PR imho If the community decides against abstracting out the common part of the Metadata, th

[PR] Core: Use correct headers for Multi-Table commits [iceberg]

2024-01-19 Thread via GitHub
nastra opened a new pull request, #9523: URL: https://github.com/apache/iceberg/pull/9523 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [I] why Iceberg Spark add_files procedure does not support adding parquet files with parquet-mr 1.10 version? [iceberg]

2024-01-19 Thread via GitHub
hbpeng0115 closed issue #9418: why Iceberg Spark add_files procedure does not support adding parquet files with parquet-mr 1.10 version? URL: https://github.com/apache/iceberg/issues/9418 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] why Iceberg Spark add_files procedure does not support adding parquet files with parquet-mr 1.10 version? [iceberg]

2024-01-19 Thread via GitHub
hbpeng0115 commented on issue #9418: URL: https://github.com/apache/iceberg/issues/9418#issuecomment-195794 It's caused by the issue https://github.com/apache/iceberg/issues/9497, so it needs to be closed. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-01-19 Thread via GitHub
jbonofre commented on PR #8907: URL: https://github.com/apache/iceberg/pull/8907#issuecomment-183428 @pvary I agree. Maybe I wasn't clear, my point was: - abstracting behaviors common to table and view makes sense but not a blocker for this PR imho - having something consistent abo

Re: [PR] feat: add support for catalogs with glue implementation to start [iceberg-go]

2024-01-19 Thread via GitHub
wolfeidau commented on PR #51: URL: https://github.com/apache/iceberg-go/pull/51#issuecomment-1899957542 @Fokko hopefull that changes makes things a bit clearer, I have added more internal documentation as well on the interface to be more specific on behavior around these identifiers. 😅🤞

Re: [PR] feat: add support for catalogs with glue implementation to start [iceberg-go]

2024-01-19 Thread via GitHub
wolfeidau commented on code in PR #51: URL: https://github.com/apache/iceberg-go/pull/51#discussion_r1458556754 ## catalog/catalog.go: ## @@ -0,0 +1,55 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE f

Re: [PR] docs: Add release guide for iceberg-rust [iceberg-rust]

2024-01-19 Thread via GitHub
Xuanwo commented on PR #147: URL: https://github.com/apache/iceberg-rust/pull/147#issuecomment-1899954667 > cc @Xuanwo Any updates? Sorry for busy in other projects, I will revist this PR later today. -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] feat: add support for catalogs with glue implementation to start [iceberg-go]

2024-01-19 Thread via GitHub
wolfeidau commented on code in PR #51: URL: https://github.com/apache/iceberg-go/pull/51#discussion_r1458553326 ## catalog/glue.go: ## @@ -0,0 +1,168 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE fil

Re: [PR] Spark 3.5: Support specifying filter in RewriteManifestsProcedure [iceberg]

2024-01-19 Thread via GitHub
bknbkn commented on code in PR #9447: URL: https://github.com/apache/iceberg/pull/9447#discussion_r1458505983 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteManifestsProcedure.java: ## @@ -118,4 +126,15 @@ private InternalRow[] toOutputRows(RewriteM

Re: [I] GlueCatalog: Set Glue table input information based on Iceberg table metadata [iceberg-python]

2024-01-19 Thread via GitHub
mgmarino commented on issue #216: URL: https://github.com/apache/iceberg-python/issues/216#issuecomment-1899935248 It seems that this is also required to enable AWS Athena to query a table created with `pyiceberg` and not just useful for UI/CLI tools. ### Example Running a test

Re: [PR] Spark 3.5: Support specifying filter in RewriteManifestsProcedure [iceberg]

2024-01-19 Thread via GitHub
bknbkn commented on code in PR #9447: URL: https://github.com/apache/iceberg/pull/9447#discussion_r1458505983 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteManifestsProcedure.java: ## @@ -118,4 +126,15 @@ private InternalRow[] toOutputRows(RewriteM