Re: [I] Implement remaining operations for Glue catalog [iceberg-go]

2024-12-28 Thread via GitHub
chil-pavn commented on issue #64: URL: https://github.com/apache/iceberg-go/issues/64#issuecomment-2564282392 Hey @zeroshade all of these looks like to be already implemented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] How to create namespace by iceberg-go? [iceberg-go]

2024-12-28 Thread via GitHub
chil-pavn commented on issue #231: URL: https://github.com/apache/iceberg-go/issues/231#issuecomment-2564280788 Hey @chengchengpei iceberg-go has implementations for Glue catalog. check the code [here] to create a namespace(https://github.com/apache/iceberg-go/blob/0921b84b53e3184a1867481bf

Re: [I] [Discussion] Add Avro library [iceberg-cpp]

2024-12-28 Thread via GitHub
zhjwpku commented on issue #17: URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2564282758 > I have created https://issues.apache.org/jira/browse/AVRO-4095 to work on removing the boost dependency from non-testing code path (at least binary-encoding). I'm inclined to use t

Re: [I] Discussion: make DataFile Serializable && Deserializable [iceberg-rust]

2024-12-28 Thread via GitHub
ZENOTME commented on issue #774: URL: https://github.com/apache/iceberg-rust/issues/774#issuecomment-2564287379 > Thanks @ZENOTME for raising this. In your question, the typical use case is compute engine, which needs to serialized `DataFile`. But in fact, what a compute engines required is

Re: [PR] Auth Manager API part 3: OAuth2 Manager [iceberg]

2024-12-28 Thread via GitHub
adutra commented on code in PR #11844: URL: https://github.com/apache/iceberg/pull/11844#discussion_r1898887643 ## core/src/main/java/org/apache/iceberg/rest/auth/AuthManagers.java: ## @@ -42,6 +57,9 @@ public static AuthManager loadAuthManager(String name, Map prope cas

Re: [I] feat: support metadata tables [iceberg-rust]

2024-12-28 Thread via GitHub
flaneur2020 commented on issue #823: URL: https://github.com/apache/iceberg-rust/issues/823#issuecomment-2564354143 i'd like to contribute to some of the implementation of these metadata tables, similar to the work done by @rshkv based on @xxchan's branch. I plan to start with the MA

Re: [I] [SPJ] Skweded partitions harm merge performances [iceberg]

2024-12-28 Thread via GitHub
szehon-ho commented on issue #11800: URL: https://github.com/apache/iceberg/issues/11800#issuecomment-2564342146 I think there are two 'not matched' here, 1 ) entries in A not matched in B 2) entries in B not matched in A. Case (1) is do-able. For case (2), it is harder bec

[I] Parameter type is not org.apache.avro.Schema for AvroSchemaUtil.toIceberg() [iceberg]

2024-12-28 Thread via GitHub
njalan opened a new issue, #11884: URL: https://github.com/apache/iceberg/issues/11884 ### Query engine Flink 1.18 Iceberg 1.6.1 ### Question In source code Parameter type is org.apache.avro.Schema for AvroSchemaUtil.toIceberg() But why in my coding it is org.apac

Re: [I] feat: support metadata tables [iceberg-rust]

2024-12-28 Thread via GitHub
rshkv commented on issue #823: URL: https://github.com/apache/iceberg-rust/issues/823#issuecomment-2564366022 @flaneur2020, apologies, I was actually just working on `manifests` and I should be able to put that up in a few days. If you want to go ahead with `entries` or `files` there'll be

Re: [PR] feat: support metadata table "snapshots" [iceberg-rust]

2024-12-28 Thread via GitHub
rshkv commented on code in PR #822: URL: https://github.com/apache/iceberg-rust/pull/822#discussion_r1898944524 ## crates/iceberg/src/metadata_scan.rs: ## @@ -0,0 +1,265 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] Update README.md to add link to iceberg-cpp [iceberg]

2024-12-28 Thread via GitHub
Fokko commented on PR #11882: URL: https://github.com/apache/iceberg/pull/11882#issuecomment-2564372425 Thanks @gabeiglio for adding this, and thanks @ebyhr for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Update README.md to add link to iceberg-cpp [iceberg]

2024-12-28 Thread via GitHub
Fokko merged PR #11882: URL: https://github.com/apache/iceberg/pull/11882 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Bump boto3 from 1.35.81 to 1.35.88 [iceberg-python]

2024-12-28 Thread via GitHub
Fokko merged PR #1476: URL: https://github.com/apache/iceberg-python/pull/1476 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

[PR] Build: Bump mkdocs-awesome-pages-plugin from 2.10.0 to 2.10.1 [iceberg]

2024-12-28 Thread via GitHub
dependabot[bot] opened a new pull request, #11885: URL: https://github.com/apache/iceberg/pull/11885 Bumps [mkdocs-awesome-pages-plugin](https://github.com/lukasgeiter/mkdocs-awesome-pages-plugin) from 2.10.0 to 2.10.1. Release notes Sourced from https://github.com/lukasgeiter/mkd

Re: [PR] Add iceberg_arrow library [iceberg-cpp]

2024-12-28 Thread via GitHub
kou commented on code in PR #6: URL: https://github.com/apache/iceberg-cpp/pull/6#discussion_r1899057489 ## cmake_modules/ThirdpartyToolchain.cmake: ## @@ -0,0 +1,142 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

[PR] Build: Bump software.amazon.awssdk:bom from 2.29.39 to 2.29.43 [iceberg]

2024-12-28 Thread via GitHub
dependabot[bot] opened a new pull request, #11886: URL: https://github.com/apache/iceberg/pull/11886 Bumps software.amazon.awssdk:bom from 2.29.39 to 2.29.43. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=soft

Re: [PR] feat: support metadata table "snapshots" [iceberg-rust]

2024-12-28 Thread via GitHub
xxchan commented on PR #822: URL: https://github.com/apache/iceberg-rust/pull/822#issuecomment-2564592025 @rshkv Thanks for the review and the follow-up work. I just want to wait for the maintainers to make sure the framework looks good before applying any suggestions :p BTW I thin

Re: [PR] Support Location Providers [iceberg-python]

2024-12-28 Thread via GitHub
Fokko commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1899070323 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] Implement column projection [iceberg-python]

2024-12-28 Thread via GitHub
Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1899070927 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1257,20 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_timestam

Re: [PR] Implement column projection [iceberg-python]

2024-12-28 Thread via GitHub
Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1899070896 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

2024-12-28 Thread via GitHub
Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1899070416 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

[PR] Introduce `MissingRequiredFilesToDeleteException` for Streaming Deletes [iceberg]

2024-12-28 Thread via GitHub
shanielh opened a new pull request, #11887: URL: https://github.com/apache/iceberg/pull/11887 Added a new exception, `MissingRequiredFilesToDeleteException`, to allow users to inspect missing file details during streaming deletes. - This exception extends `ValidationException` and

Re: [PR] feat: support metadata table "snapshots" [iceberg-rust]

2024-12-28 Thread via GitHub
rshkv commented on code in PR #822: URL: https://github.com/apache/iceberg-rust/pull/822#discussion_r1898945557 ## crates/iceberg/src/metadata_scan.rs: ## @@ -0,0 +1,267 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [I] [Discussion] Add Avro library [iceberg-cpp]

2024-12-28 Thread via GitHub
wgtmac commented on issue #17: URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2564339103 Yes, I also share the same feeling. Boost dependencies listed below are tightly used for gzip/zlib codec. I think it needs non-trivial work to replace them with the standard zlib libr

Re: [PR] Bump moto from 5.0.24 to 5.0.25 [iceberg-python]

2024-12-28 Thread via GitHub
Fokko merged PR #1475: URL: https://github.com/apache/iceberg-python/pull/1475 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] feat: search current working directory for config file [iceberg-python]

2024-12-28 Thread via GitHub
Fokko commented on code in PR #1464: URL: https://github.com/apache/iceberg-python/pull/1464#discussion_r1899066825 ## mkdocs/docs/cli.md: ## @@ -26,7 +26,7 @@ hide: Pyiceberg comes with a CLI that's available after installing the `pyiceberg` package. -You can pass the pat

Re: [PR] URL-encode partition field names in file locations [iceberg-python]

2024-12-28 Thread via GitHub
Fokko commented on code in PR #1457: URL: https://github.com/apache/iceberg-python/pull/1457#discussion_r1899068027 ## pyiceberg/partitioning.py: ## @@ -234,9 +234,11 @@ def partition_to_path(self, data: Record, schema: Schema) -> str: partition_field = self.fields

Re: [PR] URL-encode partition field names in file locations [iceberg-python]

2024-12-28 Thread via GitHub
Fokko commented on code in PR #1457: URL: https://github.com/apache/iceberg-python/pull/1457#discussion_r1899067852 ## pyiceberg/partitioning.py: ## @@ -234,9 +234,11 @@ def partition_to_path(self, data: Record, schema: Schema) -> str: partition_field = self.fields

Re: [PR] Fix read from multiple s3 regions [iceberg-python]

2024-12-28 Thread via GitHub
Fokko commented on code in PR #1453: URL: https://github.com/apache/iceberg-python/pull/1453#discussion_r1899068195 ## pyiceberg/io/pyarrow.py: ## @@ -362,6 +362,12 @@ def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) -> FileSyste "region": ge

Re: [PR] Fix read from multiple s3 regions [iceberg-python]

2024-12-28 Thread via GitHub
Fokko commented on code in PR #1453: URL: https://github.com/apache/iceberg-python/pull/1453#discussion_r1899068165 ## pyiceberg/io/pyarrow.py: ## @@ -362,6 +362,12 @@ def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) -> FileSyste "region": ge

Re: [PR] Fix read from multiple s3 regions [iceberg-python]

2024-12-28 Thread via GitHub
Fokko commented on code in PR #1453: URL: https://github.com/apache/iceberg-python/pull/1453#discussion_r1899068498 ## pyiceberg/io/pyarrow.py: ## @@ -1508,7 +1512,7 @@ def _record_batches_from_scan_tasks_and_deletes( if self._limit is not None and total_row_count >

Re: [PR] Fix read from multiple s3 regions [iceberg-python]

2024-12-28 Thread via GitHub
Fokko commented on code in PR #1453: URL: https://github.com/apache/iceberg-python/pull/1453#discussion_r1899068498 ## pyiceberg/io/pyarrow.py: ## @@ -1508,7 +1512,7 @@ def _record_batches_from_scan_tasks_and_deletes( if self._limit is not None and total_row_count >

Re: [PR] Support Location Providers [iceberg-python]

2024-12-28 Thread via GitHub
Fokko commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1899068942 ## pyiceberg/table/__init__.py: ## @@ -1627,6 +1632,67 @@ class AddFileTask: partition_field_value: Record +class LocationProvider(ABC): Review Comment:

Re: [I] Iceberg REST Spec Extensions for Access Decision Exchange [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] commented on issue #10395: URL: https://github.com/apache/iceberg/issues/10395#issuecomment-2564564980 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Support Securable Objects in Iceberg REST Catalog [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] commented on issue #10407: URL: https://github.com/apache/iceberg/issues/10407#issuecomment-2564564984 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Security improvements in the Iceberg REST specification [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] commented on issue #10537: URL: https://github.com/apache/iceberg/issues/10537#issuecomment-2564564993 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Old Avro file not found breaks Athena iceberg table [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] commented on issue #10560: URL: https://github.com/apache/iceberg/issues/10560#issuecomment-2564564998 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Added support for evolving the partition of the table [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] closed pull request #11275: Added support for evolving the partition of the table URL: https://github.com/apache/iceberg/pull/11275 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Added support for evolving the partition of the table [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] commented on PR #11275: URL: https://github.com/apache/iceberg/pull/11275#issuecomment-2564565026 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] New REST Catalog Spec/RFC [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] commented on issue #10075: URL: https://github.com/apache/iceberg/issues/10075#issuecomment-2564564946 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] [draft] HADOOP-18679. Add API for bulk/paged object deletion: Iceberg PoC [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] commented on PR #10233: URL: https://github.com/apache/iceberg/pull/10233#issuecomment-2564564963 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Support client-side purge in REST catalog [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] commented on issue #10089: URL: https://github.com/apache/iceberg/issues/10089#issuecomment-2564564954 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Add GitHub cpp-linter-action [iceberg-cpp]

2024-12-28 Thread via GitHub
wgtmac commented on PR #20: URL: https://github.com/apache/iceberg-cpp/pull/20#issuecomment-2564639921 > Hmm, in that screenshot, 'memory not found' seems like a spurious error? Will this be noisy? I have also noticed that. I need to learn more about its configuration and improve it

Re: [PR] Config for deciding whether to use Iceberg Time type [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] closed pull request #11174: Config for deciding whether to use Iceberg Time type URL: https://github.com/apache/iceberg/pull/11174 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Build: Bump calcite from 1.10.0 to 1.38.0 [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] closed pull request #11361: Build: Bump calcite from 1.10.0 to 1.38.0 URL: https://github.com/apache/iceberg/pull/11361 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Build: Bump calcite from 1.10.0 to 1.38.0 [iceberg]

2024-12-28 Thread via GitHub
dependabot[bot] commented on PR #11361: URL: https://github.com/apache/iceberg/pull/11361#issuecomment-2564565047 OK, I won't notify you again about this release, but will get in touch when a new version is available. You can also ignore all major, minor, or patch releases for a dependency

Re: [I] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] commented on issue #10391: URL: https://github.com/apache/iceberg/issues/10391#issuecomment-2564564973 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Build: Bump calcite from 1.10.0 to 1.38.0 [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] commented on PR #11361: URL: https://github.com/apache/iceberg/pull/11361#issuecomment-2564565039 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Config for deciding whether to use Iceberg Time type [iceberg]

2024-12-28 Thread via GitHub
github-actions[bot] commented on PR #11174: URL: https://github.com/apache/iceberg/pull/11174#issuecomment-2564565018 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Better error messages when creating a table with unsupported types [iceberg-python]

2024-12-28 Thread via GitHub
github-actions[bot] commented on issue #860: URL: https://github.com/apache/iceberg-python/issues/860#issuecomment-2564565597 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity oc

Re: [PR] Implement column projection [iceberg-python]

2024-12-28 Thread via GitHub
Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1899080083 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

[PR] Kafka-connect-runtime: remove code duplications in integration tests [iceberg]

2024-12-28 Thread via GitHub
wombatu-kun opened a new pull request, #11883: URL: https://github.com/apache/iceberg/pull/11883 I've found a lot of code duplicates in integration tests of module `iceberg-kafka-connect-runtime`. I propose to avoid such code duplication by making parent class `IntegrationTestBase` abstract