Re: [PR] Set Glue Table Information when creating/updating tables [iceberg-python]

2024-01-22 Thread via GitHub
mgmarino commented on code in PR #288: URL: https://github.com/apache/iceberg-python/pull/288#discussion_r1462796650 ## pyiceberg/catalog/glue.py: ## @@ -84,17 +110,97 @@ def _construct_parameters( return new_parameters +GLUE_PRIMITIVE_TYPES = { +BooleanType: "boole

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-22 Thread via GitHub
jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1462810467 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -18,6 +18,22 @@ */ package org.apache.iceberg.jdbc; +import static org.apache.iceberg.jdbc.

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-22 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1462806870 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -18,6 +18,22 @@ */ package org.apache.iceberg.jdbc; +import static org.apache.iceberg.jdbc.Jd

Re: [PR] Set Glue Table Information when creating/updating tables [iceberg-python]

2024-01-22 Thread via GitHub
mgmarino commented on PR #288: URL: https://github.com/apache/iceberg-python/pull/288#issuecomment-1905409366 > @nicor88. I tried the same code you posted along with this PR, and it seemed to work on my side. Each time I ran the code, a new "Alice" and a new "Bob" were appended to the

Re: [PR] Build: Bump actions/cache from 3 to 4 [iceberg]

2024-01-22 Thread via GitHub
nastra merged PR #9532: URL: https://github.com/apache/iceberg/pull/9532 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Set Glue Table Information when creating/updating tables [iceberg-python]

2024-01-22 Thread via GitHub
mgmarino commented on code in PR #288: URL: https://github.com/apache/iceberg-python/pull/288#discussion_r1462802332 ## tests/catalog/integration_test_glue.py: ## @@ -279,6 +379,20 @@ def test_commit_table_update_schema( assert test_catalog._parse_metadata_version(table.met

Re: [PR] Build: Fix errorprone warning [iceberg]

2024-01-22 Thread via GitHub
nastra merged PR #9531: URL: https://github.com/apache/iceberg/pull/9531 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump spring-boot from 2.5.4 to 3.2.2 [iceberg]

2024-01-22 Thread via GitHub
dependabot[bot] commented on PR #9535: URL: https://github.com/apache/iceberg/pull/9535#issuecomment-1905396795 OK, I won't notify you again about this release, but will get in touch when a new version is available. You can also ignore all major, minor, or patch releases for a dependency by

Re: [PR] Build: Bump spring-boot from 2.5.4 to 3.2.2 [iceberg]

2024-01-22 Thread via GitHub
nastra commented on PR #9535: URL: https://github.com/apache/iceberg/pull/9535#issuecomment-1905396742 Closing as this requires a newer JDK -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Build: Bump spring-boot from 2.5.4 to 3.2.2 [iceberg]

2024-01-22 Thread via GitHub
nastra closed pull request #9535: Build: Bump spring-boot from 2.5.4 to 3.2.2 URL: https://github.com/apache/iceberg/pull/9535 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Build: Bump io.netty:netty-buffer from 4.1.68.Final to 4.1.106.Final [iceberg]

2024-01-22 Thread via GitHub
dependabot[bot] commented on PR #9536: URL: https://github.com/apache/iceberg/pull/9536#issuecomment-1905396470 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Build: Bump io.netty:netty-buffer from 4.1.68.Final to 4.1.106.Final [iceberg]

2024-01-22 Thread via GitHub
nastra closed pull request #9536: Build: Bump io.netty:netty-buffer from 4.1.68.Final to 4.1.106.Final URL: https://github.com/apache/iceberg/pull/9536 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Build: Bump com.diffplug.spotless:spotless-plugin-gradle from 6.13.0 to 6.24.0 [iceberg]

2024-01-22 Thread via GitHub
dependabot[bot] commented on PR #9538: URL: https://github.com/apache/iceberg/pull/9538#issuecomment-1905396313 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Build: Bump com.diffplug.spotless:spotless-plugin-gradle from 6.13.0 to 6.24.0 [iceberg]

2024-01-22 Thread via GitHub
nastra commented on PR #9538: URL: https://github.com/apache/iceberg/pull/9538#issuecomment-1905396282 closing as this requires a newer JDK -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Build: Bump com.diffplug.spotless:spotless-plugin-gradle from 6.13.0 to 6.24.0 [iceberg]

2024-01-22 Thread via GitHub
nastra closed pull request #9538: Build: Bump com.diffplug.spotless:spotless-plugin-gradle from 6.13.0 to 6.24.0 URL: https://github.com/apache/iceberg/pull/9538 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-22 Thread via GitHub
jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1462797359 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -303,7 +287,294 @@ public static Properties filterAndRemovePrefix(Map properties, S return res

Re: [PR] Set Glue Table Information when creating/updating tables [iceberg-python]

2024-01-22 Thread via GitHub
mgmarino commented on code in PR #288: URL: https://github.com/apache/iceberg-python/pull/288#discussion_r1462796650 ## pyiceberg/catalog/glue.py: ## @@ -84,17 +110,97 @@ def _construct_parameters( return new_parameters +GLUE_PRIMITIVE_TYPES = { +BooleanType: "boole

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-01-22 Thread via GitHub
nk1506 commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1462795981 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,302 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-01-22 Thread via GitHub
nk1506 commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1462794299 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,302 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-22 Thread via GitHub
jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1462794165 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcViewCatalog.java: ## @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-22 Thread via GitHub
jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1462793255 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -245,13 +270,17 @@ public List listTables(Namespace namespace) { row -> Jd

Re: [PR] Hive: Refactor hive-table commit operation to be used for other operations like view [iceberg]

2024-01-22 Thread via GitHub
nk1506 commented on PR #9461: URL: https://github.com/apache/iceberg/pull/9461#issuecomment-1905385023 @szehon-ho With issue https://github.com/apache/iceberg/issues/9514 we are discussing what would be the best approach to handle this? We have not concluded on anything yet which is why thi

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-01-22 Thread via GitHub
nk1506 commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1462790142 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -341,19 +359,20 @@ private void setHmsTableParameters( String hmsK

Re: [I] Flink: IcebergTableSink to write data into multiple iceberg tables [iceberg]

2024-01-22 Thread via GitHub
AraiYuno commented on issue #2208: URL: https://github.com/apache/iceberg/issues/2208#issuecomment-1905359791 > > Is there any update on this feature? > > I also need this feature in my own project, just when try to do sth like ETL from kafka to iceberg, I need to write data into diff

Re: [PR] Set Glue Table Information when creating/updating tables [iceberg-python]

2024-01-22 Thread via GitHub
HonahX commented on PR #288: URL: https://github.com/apache/iceberg-python/pull/288#issuecomment-1905350230 > @mgmarino @HonahX - I was testing this, and after the change I confirm that I can query the table in Athena (I'm still doing some deep dive on why the table is not droppable in athe

Re: [PR] Set Glue Table Information when creating/updating tables [iceberg-python]

2024-01-22 Thread via GitHub
HonahX commented on code in PR #288: URL: https://github.com/apache/iceberg-python/pull/288#discussion_r1462684106 ## tests/catalog/integration_test_glue.py: ## @@ -279,6 +379,20 @@ def test_commit_table_update_schema( assert test_catalog._parse_metadata_version(table.metad

Re: [PR] init writer framework [iceberg-rust]

2024-01-22 Thread via GitHub
ZENOTME commented on PR #135: URL: https://github.com/apache/iceberg-rust/pull/135#issuecomment-1905338963 > > This may make this interface easier to make sense. > > The real code could be like: > > ```rust > let writer = FileWriterHelper::new(builder_a) > .lay

Re: [PR] Core: rewrite should drop delete files by data sequence number partition wise [iceberg]

2024-01-22 Thread via GitHub
ajantha-bhat commented on code in PR #9454: URL: https://github.com/apache/iceberg/pull/9454#discussion_r1462755419 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -289,13 +321,38 @@ private void invalidateFilteredCache() { cleanUncommitted(Snapsh

Re: [PR] init writer framework [iceberg-rust]

2024-01-22 Thread via GitHub
Xuanwo commented on PR #135: URL: https://github.com/apache/iceberg-rust/pull/135#issuecomment-1905335724 > This may make this interface easier to make sense. The real code could be like: ```rust let writer = FileWriterHelper::new(builder_a) .layer(builder_b)

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-22 Thread via GitHub
ajantha-bhat commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1462750931 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -303,7 +287,294 @@ public static Properties filterAndRemovePrefix(Map properties, S return

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-22 Thread via GitHub
ajantha-bhat commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1462748153 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -245,13 +270,17 @@ public List listTables(Namespace namespace) { row ->

Re: [PR] init writer framework [iceberg-rust]

2024-01-22 Thread via GitHub
ZENOTME commented on PR #135: URL: https://github.com/apache/iceberg-rust/pull/135#issuecomment-1905325893 I found a new way to make this API can be used like the following: ``` let writer = FileWriterHelper::new(MockFileWriterBuilder) // build the file writer first

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-22 Thread via GitHub
pvary commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1462740690 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceContinuous.java: ## @@ -367,6 +382,8 @@ public void testSpecificSnapshotTimestamp() th

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-22 Thread via GitHub
pvary commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1462737223 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceContinuous.java: ## @@ -58,9 +61,11 @@ public class TestIcebergSourceContinuous {

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-22 Thread via GitHub
pvary commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1462735668 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/assigner/SplitAssigner.java: ## @@ -115,4 +115,7 @@ default void onCompletedSplits(Collection complete

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-01-22 Thread via GitHub
pvary commented on PR #9464: URL: https://github.com/apache/iceberg/pull/9464#issuecomment-1905302078 > Is the the conclusion to wait for them to provide that new utility or proceed and improve this PR according to our discussions? Let's give the community a few more days to decide. T

Re: [PR] chore(deps): Update env_logger requirement from 0.10.0 to 0.11.0 [iceberg-rust]

2024-01-22 Thread via GitHub
Xuanwo commented on PR #170: URL: https://github.com/apache/iceberg-rust/pull/170#issuecomment-1905300264 This PR should be safe to merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] Flink: IcebergTableSink to write data into multiple iceberg tables [iceberg]

2024-01-22 Thread via GitHub
MichaelHan0228 commented on issue #2208: URL: https://github.com/apache/iceberg/issues/2208#issuecomment-1905296090 > Is there any update on this feature? I also need this feature in my own project, just when try to do sth like ETL from kafka to iceberg, I need to write data into diff

Re: [PR] docs: Add release guide for iceberg-rust [iceberg-rust]

2024-01-22 Thread via GitHub
Xuanwo commented on code in PR #147: URL: https://github.com/apache/iceberg-rust/pull/147#discussion_r1462721064 ## website/src/release.md: ## @@ -0,0 +1,383 @@ + + +This document mainly introduces how the release manager releases a new version in accordance with the Apache req

Re: [PR] docs: Add release guide for iceberg-rust [iceberg-rust]

2024-01-22 Thread via GitHub
Xuanwo commented on code in PR #147: URL: https://github.com/apache/iceberg-rust/pull/147#discussion_r1462723178 ## website/src/release.md: ## @@ -0,0 +1,383 @@ + + +This document mainly introduces how the release manager releases a new version in accordance with the Apache req

Re: [PR] docs: Add release guide for iceberg-rust [iceberg-rust]

2024-01-22 Thread via GitHub
Xuanwo commented on code in PR #147: URL: https://github.com/apache/iceberg-rust/pull/147#discussion_r1462722701 ## website/src/download.md: ## Review Comment: > Then how about adding one section in `install.md` with title `Build from source`? I think they are just one way

Re: [PR] docs: Add release guide for iceberg-rust [iceberg-rust]

2024-01-22 Thread via GitHub
Xuanwo commented on code in PR #147: URL: https://github.com/apache/iceberg-rust/pull/147#discussion_r1462721064 ## website/src/release.md: ## @@ -0,0 +1,383 @@ + + +This document mainly introduces how the release manager releases a new version in accordance with the Apache req

Re: [PR] docs: Add release guide for iceberg-rust [iceberg-rust]

2024-01-22 Thread via GitHub
Xuanwo commented on code in PR #147: URL: https://github.com/apache/iceberg-rust/pull/147#discussion_r1462719672 ## website/src/release.md: ## @@ -0,0 +1,383 @@ + + +This document mainly introduces how the release manager releases a new version in accordance with the Apache req

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-22 Thread via GitHub
fqaiser94 commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1462661503 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriterFactory.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-01-22 Thread via GitHub
BsoBird closed pull request #9333: Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. URL: https://github.com/apache/iceberg/pull/9333 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-01-22 Thread via GitHub
BsoBird commented on PR #9546: URL: https://github.com/apache/iceberg/pull/9546#issuecomment-1905228503 @RussellSpitzer Hi. can you check this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-01-22 Thread via GitHub
BsoBird commented on PR #9546: URL: https://github.com/apache/iceberg/pull/9546#issuecomment-1905228051 New Commit by https://github.com/apache/iceberg/pull/9333 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-01-22 Thread via GitHub
BsoBird opened a new pull request, #9546: URL: https://github.com/apache/iceberg/pull/9546 Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Support force option on RegisterTable procedure [iceberg]

2024-01-22 Thread via GitHub
yabola commented on PR #5327: URL: https://github.com/apache/iceberg/pull/5327#issuecomment-1905187665 @abfisher0417 Thank you. If community agrees, I can complete this PR again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-22 Thread via GitHub
syun64 commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1905187291 Hi @Fokko - sounds like you beat me to it 😄 Please let me know if you need any additional heavy lifting on #284 . Happy to help as always. The reason I was curious if there

Re: [PR] Core: rewrite should drop delete files by data sequence number partition wise [iceberg]

2024-01-22 Thread via GitHub
zinking commented on code in PR #9454: URL: https://github.com/apache/iceberg/pull/9454#discussion_r1462652137 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -289,13 +321,38 @@ private void invalidateFilteredCache() { cleanUncommitted(SnapshotPro

Re: [PR] Core: rewrite should drop delete files by data sequence number partition wise [iceberg]

2024-01-22 Thread via GitHub
zinking commented on code in PR #9454: URL: https://github.com/apache/iceberg/pull/9454#discussion_r1462652137 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -289,13 +321,38 @@ private void invalidateFilteredCache() { cleanUncommitted(SnapshotPro

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1462645409 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -245,13 +270,17 @@ public List listTables(Namespace namespace) { row ->

Re: [I] `schema_id` not incremented during schema evolution [iceberg-python]

2024-01-22 Thread via GitHub
HonahX commented on issue #290: URL: https://github.com/apache/iceberg-python/issues/290#issuecomment-1905147806 > I noticed that Schema class __eq__ function does not check if the schema_ids are equal. I think this is the intended behavior. We consider two schemas equal if they shar

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-01-22 Thread via GitHub
szehon-ho commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1462602403 ## core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java: ## @@ -284,7 +284,7 @@ private Map tableOverrideProperties() { } } - protected static S

Re: [I] refactor: Remove `async_trait` in `Catalog` trait. [iceberg-rust]

2024-01-22 Thread via GitHub
liurenjie1024 commented on issue #139: URL: https://github.com/apache/iceberg-rust/issues/139#issuecomment-1905134581 > Could I work on this ? Please tell me what precisely is to be done here? The release of rust 1.75 enables a fancy feature so that we no longer need to use `async_tra

Re: [I] Speeding up rewrite_data_files encountered concurrent write issue. [iceberg]

2024-01-22 Thread via GitHub
manuzhang commented on issue #9521: URL: https://github.com/apache/iceberg/issues/9521#issuecomment-1905133774 You may try tuning the following configs to increase the chance of commit success for each group. ``` Property Default Description commit.retry.num-retries 4

Re: [I] refactor: Remove support of manifest list format as a list of file paths. [iceberg-rust]

2024-01-22 Thread via GitHub
liurenjie1024 commented on issue #158: URL: https://github.com/apache/iceberg-rust/issues/158#issuecomment-1905130859 > Could I work on this? Sure, thanks for contributing! @hiirrxnn -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] Make field access by name work for Avro schema and record apis in Iceberg generics module [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #571: URL: https://github.com/apache/iceberg/issues/571#issuecomment-1905060621 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Make field access by name work for Avro schema and record apis in Spark module [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #572: URL: https://github.com/apache/iceberg/issues/572#issuecomment-1905060649 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Support other data formats in Iceberg Pig reader [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #542: URL: https://github.com/apache/iceberg/issues/542#issuecomment-1905060581 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Make field access by name work for Avro schema and record apis in Avro generics module [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #570: URL: https://github.com/apache/iceberg/issues/570#issuecomment-1905060599 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Vectorize read of complex/nested data types [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #521: URL: https://github.com/apache/iceberg/issues/521#issuecomment-1905060540 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Update ORC version in Iceberg [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #515: URL: https://github.com/apache/iceberg/issues/515#issuecomment-1905060504 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Remove withPartitionPath from the public API [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #507: URL: https://github.com/apache/iceberg/issues/507#issuecomment-1905060488 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Add back ability to set custom name on transformed field [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #495: URL: https://github.com/apache/iceberg/issues/495#issuecomment-1905060457 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Should issue an error/warning message when no data file to delete [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #492: URL: https://github.com/apache/iceberg/issues/492#issuecomment-1905060431 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Respect commit.manifest.min-count-to-merge while appending manifests [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #490: URL: https://github.com/apache/iceberg/issues/490#issuecomment-1905060406 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] add optional stringType column after a new long type column ,when write data get an exception [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #464: URL: https://github.com/apache/iceberg/issues/464#issuecomment-1905060367 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] TimestampWriter isn't being used [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #475: URL: https://github.com/apache/iceberg/issues/475#issuecomment-1905060384 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Iceberg Pig reader should support all catalogs [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #541: URL: https://github.com/apache/iceberg/issues/541#issuecomment-1905060559 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Support External name mapping in Iceberg generic single message encoder [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #500: URL: https://github.com/apache/iceberg/issues/500#issuecomment-1905060474 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Write a dataFrame to a table while after add an 'optional longtype' column , then get some dirty data from the new column [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #463: URL: https://github.com/apache/iceberg/issues/463#issuecomment-1905060351 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] PARTITION_DATA_ID_START is hard-coded [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #449: URL: https://github.com/apache/iceberg/issues/449#issuecomment-1905060326 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Adding support for time-based partitioning on long column type [iceberg]

2024-01-22 Thread via GitHub
github-actions[bot] commented on issue #417: URL: https://github.com/apache/iceberg/issues/417#issuecomment-1905060298 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-22 Thread via GitHub
mas-chen commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1462568032 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceContinuous.java: ## @@ -58,9 +61,11 @@ public class TestIcebergSourceContinuous

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-22 Thread via GitHub
mas-chen commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1462568032 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceContinuous.java: ## @@ -58,9 +61,11 @@ public class TestIcebergSourceContinuous

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-22 Thread via GitHub
mas-chen commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1462571996 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/MiniClusterResource.java: ## @@ -50,4 +51,18 @@ public static MiniClusterWithClientResource createWithClas

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-22 Thread via GitHub
mas-chen commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1462568032 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceContinuous.java: ## @@ -58,9 +61,11 @@ public class TestIcebergSourceContinuous

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-22 Thread via GitHub
mas-chen commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1462566583 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceContinuous.java: ## @@ -367,6 +382,8 @@ public void testSpecificSnapshotTimestamp()

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-22 Thread via GitHub
mas-chen commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1462562799 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/assigner/SplitAssigner.java: ## @@ -115,4 +115,7 @@ default void onCompletedSplits(Collection compl

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-22 Thread via GitHub
mas-chen commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1462562799 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/assigner/SplitAssigner.java: ## @@ -115,4 +115,7 @@ default void onCompletedSplits(Collection compl

Re: [PR] feat: add support for catalogs with glue implementation to start [iceberg-go]

2024-01-22 Thread via GitHub
wolfeidau commented on code in PR #51: URL: https://github.com/apache/iceberg-go/pull/51#discussion_r1462459442 ## catalog/glue.go: ## @@ -0,0 +1,162 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE fil

Re: [I] fast_forward does not work for the first commit in Spark [iceberg]

2024-01-22 Thread via GitHub
amogh-jahagirdar closed issue #8849: fast_forward does not work for the first commit in Spark URL: https://github.com/apache/iceberg/issues/8849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[PR] Build: Bump pyarrow from 14.0.2 to 15.0.0 [iceberg-python]

2024-01-22 Thread via GitHub
dependabot[bot] opened a new pull request, #295: URL: https://github.com/apache/iceberg-python/pull/295 Bumps [pyarrow](https://github.com/apache/arrow) from 14.0.2 to 15.0.0. Commits https://github.com/apache/arrow/commit/a61f4af724cd06c3a9b4abd20491345997e532c0";>a61f4af MINO

Re: [PR] API, Core, Spark: Change behavior of fastForward/replace to create the from branch if it does not exist [iceberg]

2024-01-22 Thread via GitHub
amogh-jahagirdar merged PR #9196: URL: https://github.com/apache/iceberg/pull/9196 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] API, Core, Spark: Change behavior of fastForward/replace to create the from branch if it does not exist [iceberg]

2024-01-22 Thread via GitHub
amogh-jahagirdar commented on PR #9196: URL: https://github.com/apache/iceberg/pull/9196#issuecomment-1904974901 Thanks for reviewing @nastra @rdblue , merging -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] feat: add support for catalogs with glue implementation to start [iceberg-go]

2024-01-22 Thread via GitHub
wolfeidau commented on code in PR #51: URL: https://github.com/apache/iceberg-go/pull/51#discussion_r1462459442 ## catalog/glue.go: ## @@ -0,0 +1,162 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE fil

Re: [PR] feat: add support for catalogs with glue implementation to start [iceberg-go]

2024-01-22 Thread via GitHub
wolfeidau commented on code in PR #51: URL: https://github.com/apache/iceberg-go/pull/51#discussion_r1462427200 ## catalog/glue.go: ## @@ -0,0 +1,162 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE fil

Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]

2024-01-22 Thread via GitHub
anupam-saini commented on issue #278: URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1904757960 Hello, I would like to put up a PR as per the discussion above if no one has started working already. Please let me know if this is fine. Also, @syun64 and I work together

Re: [I] Failed to assign splits due to the serialized split size [iceberg]

2024-01-22 Thread via GitHub
javrasya commented on issue #9410: URL: https://github.com/apache/iceberg/issues/9410#issuecomment-1904744066 @pvary , thank you for that. You are right. It is all immutable so it makes sense that rewrite operation would create another snapshot and I should be using that not a prior one to

Re: [I] Caused by: java.net.SocketException: Connection reset [iceberg]

2024-01-22 Thread via GitHub
javrasya commented on issue #9444: URL: https://github.com/apache/iceberg/issues/9444#issuecomment-1904737183 Thank you for jumping in @amogh-jahagirdar . The way it is not literally unusable for us so I had to write my own S3FileIO together with all the nested classes so that eventually I

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-01-22 Thread via GitHub
javrasya commented on PR #9464: URL: https://github.com/apache/iceberg/pull/9464#issuecomment-1904732583 Is the the conclusion to wait for them to provide that new utility or proceed and improve this PR according to our discussions? -- This is an automated message from the Apache Git Serv

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-22 Thread via GitHub
bryanck commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1462366234 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/PartitionedAppendWriter.java: ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Flink: Added error handling and default logic for Flink version detection [iceberg]

2024-01-22 Thread via GitHub
stevenzwu commented on code in PR #9452: URL: https://github.com/apache/iceberg/pull/9452#discussion_r1462361510 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/util/FlinkPackage.java: ## @@ -18,16 +18,44 @@ */ package org.apache.iceberg.flink.util; +import java

Re: [I] [BUG] CLI fails with Glue catalog because of missing URI [iceberg-python]

2024-01-22 Thread via GitHub
stefnba closed issue #255: [BUG] CLI fails with Glue catalog because of missing URI URL: https://github.com/apache/iceberg-python/issues/255 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Flink: Added error handling and default logic for Flink version detection [iceberg]

2024-01-22 Thread via GitHub
gjacoby126 commented on code in PR #9452: URL: https://github.com/apache/iceberg/pull/9452#discussion_r1462334518 ## flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/util/FlinkPackage.java: ## @@ -18,16 +18,44 @@ */ package org.apache.iceberg.flink.util; +import jav

Re: [I] InMemory Catalog [iceberg-python]

2024-01-22 Thread via GitHub
kevinjqliu commented on issue #293: URL: https://github.com/apache/iceberg-python/issues/293#issuecomment-1904684492 Looks like there's also a `InMemoryCatalog` in the Java lib lgithub.com/apache/iceberg/blob/0f509d2d678db2d7322dafded58ec0ca6d7fb268/core/src/main/java/org/apache/iceberg/i

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-22 Thread via GitHub
jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1462324465 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -18,6 +18,22 @@ */ package org.apache.iceberg.jdbc; +import static org.apache.iceberg.jdbc.

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-22 Thread via GitHub
rdblue commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1462320114 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -18,6 +18,22 @@ */ package org.apache.iceberg.jdbc; +import static org.apache.iceberg.jdbc.Jd

  1   2   >