Re: [PR] API: Fix typo in RewriteManifestFiles java doc [iceberg]

2024-07-25 Thread via GitHub
Fokko merged PR #10778: URL: https://github.com/apache/iceberg/pull/10778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Deprecate rest.authorization-url in favor of oauth2-server-uri [iceberg-python]

2024-07-25 Thread via GitHub
Fokko commented on code in PR #962: URL: https://github.com/apache/iceberg-python/pull/962#discussion_r1690921688 ## pyiceberg/catalog/rest.py: ## @@ -118,7 +120,8 @@ class Endpoints: SIGV4 = "rest.sigv4-enabled" SIGV4_REGION = "rest.signing-region" SIGV4_SERVICE = "rest.sign

Re: [PR] Hive: close the fileIO client when closing the hive catalog [iceberg]

2024-07-25 Thread via GitHub
nastra commented on code in PR #10771: URL: https://github.com/apache/iceberg/pull/10771#discussion_r1690919530 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -512,7 +531,10 @@ private boolean isValidateNamespace(Namespace namespace) { public

Re: [PR] Deprecate rest.authorization-url in favor of oauth2-server-uri [iceberg-python]

2024-07-25 Thread via GitHub
Fokko commented on code in PR #962: URL: https://github.com/apache/iceberg-python/pull/962#discussion_r1690923706 ## pyiceberg/catalog/rest.py: ## @@ -290,7 +293,14 @@ def url(self, endpoint: str, prefixed: bool = True, **kwargs: Any) -> str: @property def auth_url(

Re: [I] Incorrect Avro schema generated for Tables with a `Transform::Day` partition causes manifest file parsing to fail [iceberg-rust]

2024-07-25 Thread via GitHub
sdd commented on issue #478: URL: https://github.com/apache/iceberg-rust/issues/478#issuecomment-2249618894 Thanks for looking at this so quickly @Fokko! I think we (and the apache_avro crate) should consider some more detailed error logging for manifest file parse failures as well. I

Re: [I] Incorrect Avro schema generated for Tables with a `Transform::Day` partition causes manifest file parsing to fail [iceberg-rust]

2024-07-25 Thread via GitHub
Xuanwo commented on issue #478: URL: https://github.com/apache/iceberg-rust/issues/478#issuecomment-2249634931 Agreed. Would you like to raise an issue for this? I do think we need some logging in iceberg crate. -- This is an automated message from the Apache Git Service. To respond to th

Re: [I] Add memory support for file_io [iceberg-rust]

2024-07-25 Thread via GitHub
sdd commented on issue #477: URL: https://github.com/apache/iceberg-rust/issues/477#issuecomment-2249649663 I have a local branch where I've modified FileIO to add an OpenDAL `Layer` that does in-memory caching of the underlying files using the `moka` crate. I can share the details if you

Re: [PR] Remove support for catalog_name in table identifier string [iceberg-python]

2024-07-25 Thread via GitHub
Fokko commented on PR #963: URL: https://github.com/apache/iceberg-python/pull/963#issuecomment-2249655208 Hey @sungwy thanks for raising this. Some historical context was added from the beginning to copy some of the concepts of how it works in Java/Spark, but to be frank, I don't think it

Re: [I] Formal verification discovers potential consistency issue [iceberg]

2024-07-25 Thread via GitHub
Vanlightly commented on issue #10720: URL: https://github.com/apache/iceberg/issues/10720#issuecomment-2249660794 I'm not sure how to test it directly with Spark, given the API does not allow me to set the validateFromSnapshot, so I created a test to demonstrate the case, using the lower-le

Re: [I] Incorrect Avro schema generated for Tables with a `Transform::Day` partition causes manifest file parsing to fail [iceberg-rust]

2024-07-25 Thread via GitHub
sdd commented on issue #478: URL: https://github.com/apache/iceberg-rust/issues/478#issuecomment-2249664181 > Agreed. Would you like to raise an issue for this? I do think we need some logging in iceberg crate. I've been thinking this for a long time too. I often sprinkle in `tracing

Re: [PR] DRAFT: Strawman proposal for PR merging [iceberg]

2024-07-25 Thread via GitHub
Fokko commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1690996188 ## site/docs/contribute.md: ## @@ -45,6 +45,17 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an issue, a

Re: [I] Add memory support for file_io [iceberg-rust]

2024-07-25 Thread via GitHub
Xuanwo commented on issue #477: URL: https://github.com/apache/iceberg-rust/issues/477#issuecomment-2249693360 That's nice! Please share it to me :heart: By the way, opendal has native support of moka and mini_moka, maybe also useful for caching. -- This is an automated message fr

Re: [PR] DRAFT: Strawman proposal for PR merging [iceberg]

2024-07-25 Thread via GitHub
emkornfield commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1691012917 ## site/docs/contribute.md: ## @@ -207,8 +218,8 @@ public class SnapshotManager implements ManageSnapshots { // existing code... @Override - public Manag

Re: [PR] DRAFT: Strawman proposal for PR merging [iceberg]

2024-07-25 Thread via GitHub
emkornfield commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1691014322 ## site/docs/contribute.md: ## @@ -45,6 +45,17 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an is

Re: [PR] DRAFT: Strawman proposal for PR merging [iceberg]

2024-07-25 Thread via GitHub
emkornfield commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1691018532 ## site/docs/contribute.md: ## @@ -207,8 +218,8 @@ public class SnapshotManager implements ManageSnapshots { // existing code... @Override - public Manag

[PR] Make compatibility example consistent. [iceberg]

2024-07-25 Thread via GitHub
emkornfield opened a new pull request, #10781: URL: https://github.com/apache/iceberg/pull/10781 * make implementation match definition in interface * clarify why adding a new interface method breaks compatibility. -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Minor: Make compatibility example consistent. [iceberg]

2024-07-25 Thread via GitHub
emkornfield commented on PR #10781: URL: https://github.com/apache/iceberg/pull/10781#issuecomment-2249719560 @Fokko would you mind taking a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] DRAFT: Strawman proposal for PR merging [iceberg]

2024-07-25 Thread via GitHub
Fokko commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1691042870 ## site/docs/contribute.md: ## @@ -45,6 +45,17 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an issue, a

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-07-25 Thread via GitHub
lirui-apache commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1691044481 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] DRAFT: Strawman proposal for PR merging [iceberg]

2024-07-25 Thread via GitHub
emkornfield commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1691057611 ## site/docs/contribute.md: ## @@ -45,6 +45,17 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an is

Re: [PR] Spark Action to Analyze table [iceberg]

2024-07-25 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1691054962 ## api/src/main/java/org/apache/iceberg/actions/ComputeTableStats.java: ## @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

[PR] [WIP] Fix issues related to having catalog_name in identifier [iceberg-python]

2024-07-25 Thread via GitHub
HonahX opened a new pull request, #964: URL: https://github.com/apache/iceberg-python/pull/964 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Remove support for catalog_name in table identifier string [iceberg-python]

2024-07-25 Thread via GitHub
HonahX commented on PR #963: URL: https://github.com/apache/iceberg-python/pull/963#issuecomment-2249883371 Thanks for raising the PR! I am also +1 on removing this. This seems unnecessary and will inevitably lead to the issue as @sungwy mentioned in the third point: > identifier_to_

Re: [PR] Infra, Docs: Publish Apache Iceberg 1.6.0 release [iceberg]

2024-07-25 Thread via GitHub
jbonofre commented on code in PR #10752: URL: https://github.com/apache/iceberg/pull/10752#discussion_r1691138323 ## site/docs/releases.md: ## @@ -71,6 +71,88 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ... ``` + +### 1.6.0 release +Ap

Re: [PR] Fix issues related to having catalog_name in identifier [iceberg-python]

2024-07-25 Thread via GitHub
HonahX commented on code in PR #964: URL: https://github.com/apache/iceberg-python/pull/964#discussion_r1691141095 ## tests/integration/test_writes/test_rest_behavior.py: ## @@ -0,0 +1,68 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor l

Re: [I] flink iceberg may occur duplication when succeed to write datafile and commit but checkpoint fail [iceberg]

2024-07-25 Thread via GitHub
maekchi commented on issue #10765: URL: https://github.com/apache/iceberg/issues/10765#issuecomment-2249927915 @pvary I also think this is different than #10526. I am only appending data. and perform with `exactly once` mode in Flink. Because there were other Flink apps, the

Re: [PR] UpdatePartitionSpec: Added ability to not set the new partition spec as default [iceberg]

2024-07-25 Thread via GitHub
shanielh commented on PR #10736: URL: https://github.com/apache/iceberg/pull/10736#issuecomment-2249962904 Hi, I've applied the changes you suggested within a new fixup commit, please review and let me know when it's good to squash and rebase. -- This is an automated message from the Apac

Re: [PR] Remove support for catalog_name in table identifier string [iceberg-python]

2024-07-25 Thread via GitHub
sungwy commented on PR #963: URL: https://github.com/apache/iceberg-python/pull/963#issuecomment-2249980644 Hi @HonahX and @Fokko thank you for the detailed reviews! Given those concerns, I think it would make sense to: 1. Introduce the fix @HonahX proposed to address #742 without introdu

Re: [I] Geospatial Support [iceberg]

2024-07-25 Thread via GitHub
desruisseaux commented on issue #10260: URL: https://github.com/apache/iceberg/issues/10260#issuecomment-2249997624 To summarize: Next to the _"Coordinate Reference System, ie mapping of how coordinates refer to precise locations on earth"_ sentences, it may be worth to said that OGC

Re: [PR] docs: installation of the new `iceberg_catalog_rest` added to the docs [iceberg-rust]

2024-07-25 Thread via GitHub
Xuanwo commented on PR #355: URL: https://github.com/apache/iceberg-rust/pull/355#issuecomment-2250010769 Hi, @liurenjie1024, this PR has been open for a while without any progress. I believe it's ready to be merged and we can continue to make improvements on it afterward. What do you think

Re: [PR] Concurrent table scans [iceberg-rust]

2024-07-25 Thread via GitHub
sdd commented on code in PR #373: URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1691246764 ## crates/iceberg/src/scan.rs: ## @@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> { field_ids.push(field_id); } -Ok(TableScan { +

Re: [PR] Fix issues related to having catalog_name in identifier [iceberg-python]

2024-07-25 Thread via GitHub
sungwy commented on code in PR #964: URL: https://github.com/apache/iceberg-python/pull/964#discussion_r1691348197 ## tests/catalog/test_rest.py: ## @@ -1226,3 +1226,25 @@ def test_catalog_from_parameters_empty_env(rest_mock: Mocker) -> None: catalog = cast(RestCatalog,

Re: [PR] Update .asf.yaml [iceberg]

2024-07-25 Thread via GitHub
findepi merged PR #10767: URL: https://github.com/apache/iceberg/pull/10767 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Fix issues related to having catalog_name in identifier [iceberg-python]

2024-07-25 Thread via GitHub
sungwy commented on PR #964: URL: https://github.com/apache/iceberg-python/pull/964#issuecomment-2250191429 +1 to this solution for 0.7.0 @HonahX . This is a creative way to mitigate the issue for the release in the short term. Thank you for taking your time to put together this solution so

Re: [PR] Update .asf.yaml [iceberg]

2024-07-25 Thread via GitHub
findepi commented on PR #10767: URL: https://github.com/apache/iceberg/pull/10767#issuecomment-2250192506 @ajantha-bhat thanks for your patience. Looks we're all set. And this is definitely the best first PR to merge :) -- This is an automated message from the Apache Git Service. To r

[PR] Add jbonofre as collaborator on the project [iceberg]

2024-07-25 Thread via GitHub
jbonofre opened a new pull request, #10782: URL: https://github.com/apache/iceberg/pull/10782 As discussed in another PR @findepi @Fokko @ajantha-bhat 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Support building with Java 21 [iceberg]

2024-07-25 Thread via GitHub
nastra commented on PR #10474: URL: https://github.com/apache/iceberg/pull/10474#issuecomment-2250236029 thanks for working on this @findepi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Support building with Java 21 [iceberg]

2024-07-25 Thread via GitHub
nastra merged PR #10474: URL: https://github.com/apache/iceberg/pull/10474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Infra, Docs: Publish Apache Iceberg 1.6.0 release [iceberg]

2024-07-25 Thread via GitHub
jbonofre commented on code in PR #10752: URL: https://github.com/apache/iceberg/pull/10752#discussion_r1691388502 ## site/docs/releases.md: ## @@ -71,6 +71,88 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ... ``` + +### 1.6.0 release +Ap

Re: [PR] Infra, Docs: Publish Apache Iceberg 1.6.0 release [iceberg]

2024-07-25 Thread via GitHub
jbonofre commented on PR #10752: URL: https://github.com/apache/iceberg/pull/10752#issuecomment-2250246912 @amogh-jahagirdar as requested by @nastra I added a `Dependencies` section -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

[PR] Fix parsing reference for nested fields [iceberg-python]

2024-07-25 Thread via GitHub
sungwy opened a new pull request, #965: URL: https://github.com/apache/iceberg-python/pull/965 Fixes: https://github.com/apache/iceberg-python/issues/953 This may be a backward incompatible change because it seemingly removes support for allowing the table name within the parser strin

Re: [I] Query on nested struct field with PyIceberg? [iceberg-python]

2024-07-25 Thread via GitHub
sungwy commented on issue #953: URL: https://github.com/apache/iceberg-python/issues/953#issuecomment-2250263129 @cfrancois7 thank you for raising this tricky issue, and @kevinjqliu for spending the time to step through the issue as well to do the root cause investigation! I've put u

Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]

2024-07-25 Thread via GitHub
bryanck commented on code in PR #10739: URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691439004 ## kafka-connect/kafka-connect-runtime/docker/docker-compose.yml: ## @@ -0,0 +1,107 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contri

Re: [I] Query on nested struct field with PyIceberg? [iceberg-python]

2024-07-25 Thread via GitHub
cfrancois7 commented on issue #953: URL: https://github.com/apache/iceberg-python/issues/953#issuecomment-2250346701 The first issue regarding the parsing is resolved by the PR. But the second issue related to the pyarrow command is still there: `ArrowInvalid: No match for FieldRef.Name(s

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691464398 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] Deprecate rest.authorization-url in favor of oauth2-server-uri [iceberg-python]

2024-07-25 Thread via GitHub
ndrluis commented on code in PR #962: URL: https://github.com/apache/iceberg-python/pull/962#discussion_r1691469582 ## pyiceberg/catalog/rest.py: ## @@ -290,7 +293,14 @@ def url(self, endpoint: str, prefixed: bool = True, **kwargs: Any) -> str: @property def auth_ur

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
Xuanwo commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691483403 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreem

Re: [I] Query on nested struct field with PyIceberg? [iceberg-python]

2024-07-25 Thread via GitHub
sungwy commented on issue #953: URL: https://github.com/apache/iceberg-python/issues/953#issuecomment-2250394041 > The first issue regarding the parsing is resolved by the PR. > > But the second issue related to the pyarrow command is still there: `ArrowInvalid: No match for FieldRef.

Re: [PR] feat(iceberg): Add memory file IO support [iceberg-rust]

2024-07-25 Thread via GitHub
fqaiser94 commented on code in PR #481: URL: https://github.com/apache/iceberg-rust/pull/481#discussion_r1691489187 ## crates/iceberg/src/io/storage_memory.rs: ## @@ -0,0 +1,43 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agre

Re: [PR] feat(iceberg): Add memory file IO support [iceberg-rust]

2024-07-25 Thread via GitHub
Xuanwo commented on code in PR #481: URL: https://github.com/apache/iceberg-rust/pull/481#discussion_r1691497746 ## crates/iceberg/src/io/storage.rs: ## @@ -79,6 +87,16 @@ impl Storage { ) -> crate::Result<(Operator, &'a str)> { let path = path.as_ref(); m

Re: [PR] Infra, Docs: Publish Apache Iceberg 1.6.0 release [iceberg]

2024-07-25 Thread via GitHub
amogh-jahagirdar commented on PR #10752: URL: https://github.com/apache/iceberg/pull/10752#issuecomment-2250397068 Thanks @jbonofre ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Infra, Docs: Publish Apache Iceberg 1.6.0 release [iceberg]

2024-07-25 Thread via GitHub
amogh-jahagirdar merged PR #10752: URL: https://github.com/apache/iceberg/pull/10752 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691464398 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] API: New API For sequential / streaming updates [iceberg]

2024-07-25 Thread via GitHub
jasonf20 commented on PR #9323: URL: https://github.com/apache/iceberg/pull/9323#issuecomment-2250437040 I rebased this PR again so it's mergable. Would appreciate reviving this PR. @rdblue Could you please have a look or let me know if there is someone else who could help? -- This is a

Re: [PR] Infra, Docs: Publish Apache Iceberg 1.6.0 release [iceberg]

2024-07-25 Thread via GitHub
jbonofre commented on PR #10752: URL: https://github.com/apache/iceberg/pull/10752#issuecomment-2250457287 @amogh-jahagirdar thanks ! I will do the announcement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691550143 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691550143 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691550143 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] feat(iceberg): Add memory file IO support [iceberg-rust]

2024-07-25 Thread via GitHub
fqaiser94 commented on code in PR #481: URL: https://github.com/apache/iceberg-rust/pull/481#discussion_r1691486557 ## crates/iceberg/src/io/storage.rs: ## @@ -79,6 +87,16 @@ impl Storage { ) -> crate::Result<(Operator, &'a str)> { let path = path.as_ref();

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691550143 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691550143 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-07-25 Thread via GitHub
amogh-jahagirdar commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1691584095 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3818,176 @@ components: type: integer description: "List of equality field IDs"

Re: [PR] Deprecate rest.authorization-url in favor of oauth2-server-uri [iceberg-python]

2024-07-25 Thread via GitHub
ndrluis commented on PR #962: URL: https://github.com/apache/iceberg-python/pull/962#issuecomment-2250532227 The deprecated decorator adds a message that does not make sense for the properties deprecation. I will add a new function and change where is a property deprecation and are using th

[PR] Update icebergVersion in the website/doc [iceberg]

2024-07-25 Thread via GitHub
jbonofre opened a new pull request, #10783: URL: https://github.com/apache/iceberg/pull/10783 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] Update icebergVersion in the website/doc [iceberg]

2024-07-25 Thread via GitHub
jbonofre commented on PR #10783: URL: https://github.com/apache/iceberg/pull/10783#issuecomment-2250548095 @amogh-jahagirdar can you please take a look ? Thanks ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-07-25 Thread via GitHub
amogh-jahagirdar commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1691584095 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3818,176 @@ components: type: integer description: "List of equality field IDs"

Re: [PR] Adding `add_files_overwrite` method [iceberg-python]

2024-07-25 Thread via GitHub
enkidulan commented on PR #810: URL: https://github.com/apache/iceberg-python/pull/810#issuecomment-2250554372 @sungwy #569 was merged last week, so I've updated this PR. Can you glance over it and say if adding `add_files_overwrite` makes sense to you? -- This is an automated message fro

Re: [PR] Update iceberg version on site to 1.6.0 [iceberg]

2024-07-25 Thread via GitHub
amogh-jahagirdar merged PR #10783: URL: https://github.com/apache/iceberg/pull/10783 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Deprecate rest.authorization-url in favor of oauth2-server-uri [iceberg-python]

2024-07-25 Thread via GitHub
ndrluis commented on PR #962: URL: https://github.com/apache/iceberg-python/pull/962#issuecomment-2250572751 @Fokko Done! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] AddPartitionSpec: A new way to set new partition specs [iceberg]

2024-07-25 Thread via GitHub
RussellSpitzer commented on code in PR #10737: URL: https://github.com/apache/iceberg/pull/10737#discussion_r1691611196 ## api/src/main/java/org/apache/iceberg/AddPartitionSpec.java: ## @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]

2024-07-25 Thread via GitHub
danielcweeks commented on code in PR #10739: URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691637442 ## kafka-connect/kafka-connect-runtime/docker/docker-compose.yml: ## @@ -0,0 +1,107 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more c

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
Xuanwo commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691637775 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreem

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
Xuanwo commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691637775 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreem

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
Xuanwo commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691637775 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreem

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-25 Thread via GitHub
nastra commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2250721645 @amogh-jahagirdar would you like to take another look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Hive: close the fileIO client when closing the hive catalog [iceberg]

2024-07-25 Thread via GitHub
nastra commented on code in PR #10771: URL: https://github.com/apache/iceberg/pull/10771#discussion_r1691695766 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -512,7 +531,10 @@ private boolean isValidateNamespace(Namespace namespace) { public

Re: [PR] Flink: support limit pushdown in FLIP-27 source [iceberg]

2024-07-25 Thread via GitHub
nastra commented on code in PR #10748: URL: https://github.com/apache/iceberg/pull/10748#discussion_r1691704734 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/source/reader/LimitableDataIterator.java: ## @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]

2024-07-25 Thread via GitHub
ajantha-bhat commented on code in PR #10739: URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691726716 ## kafka-connect/kafka-connect-runtime/docker/docker-compose.yml: ## @@ -0,0 +1,107 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more c

Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]

2024-07-25 Thread via GitHub
bryanck commented on code in PR #10739: URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691728921 ## kafka-connect/kafka-connect-runtime/docker/docker-compose.yml: ## @@ -0,0 +1,107 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contri

Re: [I] Failed to initialize S3FileIO when writing to minio using spark [iceberg]

2024-07-25 Thread via GitHub
nastra commented on issue #7396: URL: https://github.com/apache/iceberg/issues/7396#issuecomment-2250791863 @umathivagit you might want to look at the Spark [config](https://github.com/tabular-io/docker-spark-iceberg/blob/main/spark/spark-defaults.conf#L23-L29 ) that the [Spark quickstart

Re: [PR] DRAFT: Strawman proposal for PR merging [iceberg]

2024-07-25 Thread via GitHub
nastra commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1691733551 ## site/docs/contribute.md: ## @@ -45,6 +45,17 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an issue,

Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]

2024-07-25 Thread via GitHub
jbonofre commented on code in PR #10739: URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691734005 ## kafka-connect/kafka-connect-runtime/docker/docker-compose.yml: ## @@ -0,0 +1,107 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contr

Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]

2024-07-25 Thread via GitHub
ajantha-bhat commented on code in PR #10739: URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691739280 ## kafka-connect/kafka-connect-runtime/docker/docker-compose.yml: ## @@ -0,0 +1,107 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more c

Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]

2024-07-25 Thread via GitHub
danielcweeks commented on code in PR #10739: URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691741295 ## kafka-connect/kafka-connect-runtime/docker/docker-compose.yml: ## @@ -0,0 +1,107 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more c

Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]

2024-07-25 Thread via GitHub
jbonofre commented on code in PR #10739: URL: https://github.com/apache/iceberg/pull/10739#discussion_r1691755922 ## kafka-connect/kafka-connect-runtime/docker/docker-compose.yml: ## @@ -0,0 +1,107 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contr

Re: [PR] Support loading custom catalog impl [iceberg-python]

2024-07-25 Thread via GitHub
jackye1995 commented on PR #947: URL: https://github.com/apache/iceberg-python/pull/947#issuecomment-2250898833 @HonahX @kevinjqliu any further comments? Could this be merged? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Hive: close the fileIO client when closing the hive catalog [iceberg]

2024-07-25 Thread via GitHub
amogh-jahagirdar commented on code in PR #10771: URL: https://github.com/apache/iceberg/pull/10771#discussion_r1691797531 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -512,7 +531,10 @@ private boolean isValidateNamespace(Namespace namespace) {

Re: [PR] Hive: close the fileIO client when closing the hive catalog [iceberg]

2024-07-25 Thread via GitHub
amogh-jahagirdar commented on code in PR #10771: URL: https://github.com/apache/iceberg/pull/10771#discussion_r1691797531 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -512,7 +531,10 @@ private boolean isValidateNamespace(Namespace namespace) {

Re: [PR] Hive: close the fileIO client when closing the hive catalog [iceberg]

2024-07-25 Thread via GitHub
amogh-jahagirdar commented on code in PR #10771: URL: https://github.com/apache/iceberg/pull/10771#discussion_r1691803561 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -512,7 +531,10 @@ private boolean isValidateNamespace(Namespace namespace) {

Re: [I] Query on nested struct field with PyIceberg? [iceberg-python]

2024-07-25 Thread via GitHub
sungwy commented on issue #953: URL: https://github.com/apache/iceberg-python/issues/953#issuecomment-2250984811 Hi @cfrancois7 I investigated this a little more, and I think there could be a limitation on pyarrow in supporting predicate pushdown for nested fields. Here's a related open iss

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691830743 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691830743 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691830743 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
Xuanwo commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691852790 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreem

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
Xuanwo commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691852790 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreem

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-25 Thread via GitHub
Xuanwo commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1691852790 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreem

Re: [I] Query on nested struct field with PyIceberg? [iceberg-python]

2024-07-25 Thread via GitHub
kevinjqliu commented on issue #953: URL: https://github.com/apache/iceberg-python/issues/953#issuecomment-2251045032 Here's something interesting. I think the `pyarrow_filter` [here](https://github.com/apache/iceberg-python/blob/9b6503def4f07a70149da07378f4795c3a4b113b/pyiceberg/io/py

Re: [PR] Hive: close the fileIO client when closing the hive catalog [iceberg]

2024-07-25 Thread via GitHub
amogh-jahagirdar commented on PR #10771: URL: https://github.com/apache/iceberg/pull/10771#issuecomment-2251075082 Thanks @hussein-awala , and thank you @nastra for reviewing! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Hive: close the fileIO client when closing the hive catalog [iceberg]

2024-07-25 Thread via GitHub
amogh-jahagirdar merged PR #10771: URL: https://github.com/apache/iceberg/pull/10771 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

[PR] API: Define RepairManifests action interface [iceberg]

2024-07-25 Thread via GitHub
amogh-jahagirdar opened a new pull request, #10784: URL: https://github.com/apache/iceberg/pull/10784 Co-authored-by: Mathew Fournier <160646114+tabmatfourn...@users.noreply.github.com> Refer to #10445 for more details. This change defines a RepairManifest Action interface wher

  1   2   3   >