Re: [I] [HadoopCatalog]: [HadoopTableOperations]: Commit flow, renameToFinal does not actually check if lock acquired [iceberg]

2024-02-19 Thread via GitHub
BsoBird commented on issue #9485: URL: https://github.com/apache/iceberg/issues/9485#issuecomment-1953649333 @N-o-Z Hello, I am working on a fix related to HadoopTableOperations. I have a question for you. ``` if (!lockManager.acquire(dst.toString(), src.toString())) {

Re: [PR] Support merge manifests on writes [iceberg-python]

2024-02-19 Thread via GitHub
HonahX commented on code in PR #363: URL: https://github.com/apache/iceberg-python/pull/363#discussion_r1495367348 ## pyiceberg/table/__init__.py: ## @@ -944,7 +949,8 @@ def append(self, df: pa.Table) -> None: if len(self.spec().fields) > 0: raise ValueErro

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-19 Thread via GitHub
geruh commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495365248 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,97 @@ components: type: integer format: int64 +TypeValue: + oneOf: +-

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-02-19 Thread via GitHub
BsoBird commented on PR #9546: URL: https://github.com/apache/iceberg/pull/9546#issuecomment-1953639035 @RussellSpitzer Hello Sir, I have tried to limit the scope of the PR changes to what I consider to be the minimum. If there is still a need to further reduce the scope of the modificat

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-19 Thread via GitHub
geruh commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495360137 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: boo

Re: [I] SQL Catalog Implementation? [iceberg-rust]

2024-02-19 Thread via GitHub
JanKaul commented on issue #205: URL: https://github.com/apache/iceberg-rust/issues/205#issuecomment-1953580076 I could work on it. I already have an implementation in another repo. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

[I] PyArrowFileIO.parse_location(...) return error path for hdfs location. [iceberg-python]

2024-02-19 Thread via GitHub
luocan17 opened a new issue, #449: URL: https://github.com/apache/iceberg-python/issues/449 ### Apache Iceberg version None ### Please describe the bug 🐞 `parse_location` method in PyArrowFileIO return error `path` as follow. ![image](https://github.com/apache/ice

Re: [I] Add more scripts to make release process smoother. [iceberg-rust]

2024-02-19 Thread via GitHub
liurenjie1024 commented on issue #206: URL: https://github.com/apache/iceberg-rust/issues/206#issuecomment-1953565677 cc @Xuanwo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[I] Add more scripts to make release process smoother. [iceberg-rust]

2024-02-19 Thread via GitHub
liurenjie1024 opened a new issue, #206: URL: https://github.com/apache/iceberg-rust/issues/206 During [the discussion](https://lists.apache.org/thread/1rnk9c1ldjhd160bfxvdyk7mlkzrjj03) of the first release, we see some problems that we need to improve with our scripts: 1. Add doc fo

Re: [PR] Build: Bump mkdocs-material from 9.5.9 to 9.5.10 [iceberg-python]

2024-02-19 Thread via GitHub
HonahX merged PR #448: URL: https://github.com/apache/iceberg-python/pull/448 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Build: Bump moto from 5.0.1 to 5.0.2 [iceberg-python]

2024-02-19 Thread via GitHub
HonahX merged PR #447: URL: https://github.com/apache/iceberg-python/pull/447 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [I] Null values in metadata_log_entries after replacing Table [iceberg]

2024-02-19 Thread via GitHub
oneonestar commented on issue #9723: URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1953473457 Ok. https://github.com/apache/iceberg/issues/9758 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] iceberg materialized views [iceberg]

2024-02-19 Thread via GitHub
manuzhang commented on issue #8143: URL: https://github.com/apache/iceberg/issues/8143#issuecomment-1953451665 Proposal for materialized view is still under discussion https://github.com/apache/iceberg/issues/6420 -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-19 Thread via GitHub
stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1495209384 ## format/spec.md: ## @@ -1237,17 +1237,36 @@ Content file (data or delete) is serialized as a JSON object according to the fo | **`equality-ids`** |`JSON lis

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-02-19 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1495178420 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -149,26 +183,71 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-02-19 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1495168186 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -59,7 +61,7 @@ */ public class HadoopTableOperations implements TableOperations {

Re: [I] ValidationException: Missing required files to delete [iceberg]

2024-02-19 Thread via GitHub
fengguangyuan commented on issue #9741: URL: https://github.com/apache/iceberg/issues/9741#issuecomment-1953373456 > @fengguangyuan > > > but at some point a Compact task committed before the Overwrite task trying to call the internal method to commit metadata, obviously these overwr

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-19 Thread via GitHub
tomtongue commented on PR #9670: URL: https://github.com/apache/iceberg/pull/9670#issuecomment-1953348105 @nastra Reflected on your comments. If there's any part that I should change, please let me know. -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-19 Thread via GitHub
tomtongue commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494655518 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1052,9 +1039,10 @@ public synchronized void testDeleteWithSe

Re: [PR] refactor: rm async_trait and add trait_variant [iceberg-rust]

2024-02-19 Thread via GitHub
liurenjie1024 commented on PR #186: URL: https://github.com/apache/iceberg-rust/pull/186#issuecomment-1953347525 > @liurenjie1024 Yes, we can close this and reopen it after they release the rewrite feature. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] refactor: rm async_trait and add trait_variant [iceberg-rust]

2024-02-19 Thread via GitHub
liurenjie1024 closed pull request #186: refactor: rm async_trait and add trait_variant URL: https://github.com/apache/iceberg-rust/pull/186 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-19 Thread via GitHub
tomtongue commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494687079 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1052,9 +1039,10 @@ public synchronized void testDeleteWithSe

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495136815 ## open-api/rest-catalog-open-api.yaml: ## @@ -532,6 +532,100 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495136511 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495129410 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495132150 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495131124 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495129630 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495129505 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495129562 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495129505 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495129410 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [I] No retries on snapshot commit on eventual consistent file system [iceberg]

2024-02-19 Thread via GitHub
github-actions[bot] commented on issue #1398: URL: https://github.com/apache/iceberg/issues/1398#issuecomment-1953299793 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Add support for reading Avro and ORC data files in Iceberg Pig [iceberg]

2024-02-19 Thread via GitHub
github-actions[bot] closed issue #849: Add support for reading Avro and ORC data files in Iceberg Pig URL: https://github.com/apache/iceberg/issues/849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Pig should not rely on it own implementation of InputFormat [iceberg]

2024-02-19 Thread via GitHub
github-actions[bot] commented on issue #847: URL: https://github.com/apache/iceberg/issues/847#issuecomment-1953299679 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] Vectorized reads - enable vectorized reads when projecting identity partition columns [iceberg]

2024-02-19 Thread via GitHub
github-actions[bot] commented on issue #838: URL: https://github.com/apache/iceberg/issues/838#issuecomment-1953299661 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] Add support for reading Avro and ORC data files in Iceberg Pig [iceberg]

2024-02-19 Thread via GitHub
github-actions[bot] commented on issue #849: URL: https://github.com/apache/iceberg/issues/849#issuecomment-1953299690 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] Pig should not rely on it own implementation of InputFormat [iceberg]

2024-02-19 Thread via GitHub
github-actions[bot] closed issue #847: Pig should not rely on it own implementation of InputFormat URL: https://github.com/apache/iceberg/issues/847 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] Vectorized reads - enable vectorized reads when projecting identity partition columns [iceberg]

2024-02-19 Thread via GitHub
github-actions[bot] closed issue #838: Vectorized reads - enable vectorized reads when projecting identity partition columns URL: https://github.com/apache/iceberg/issues/838 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [I] AvroFileAppender doesn't report column-level stats [iceberg]

2024-02-19 Thread via GitHub
github-actions[bot] commented on issue #837: URL: https://github.com/apache/iceberg/issues/837#issuecomment-1953299643 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] AvroFileAppender doesn't report column-level stats [iceberg]

2024-02-19 Thread via GitHub
github-actions[bot] closed issue #837: AvroFileAppender doesn't report column-level stats URL: https://github.com/apache/iceberg/issues/837 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495097615 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Update NameMapping on update_schema() [iceberg-python]

2024-02-19 Thread via GitHub
syun64 commented on code in PR #441: URL: https://github.com/apache/iceberg-python/pull/441#discussion_r1495092898 ## pyiceberg/table/name_mapping.py: ## @@ -45,6 +45,18 @@ class MappedField(IcebergBaseModel): def convert_null_to_empty_List(cls, v: Any) -> Any: ret

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495092438 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495091041 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495091041 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495090469 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495087324 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495084521 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495083858 ## open-api/rest-catalog-open-api.yaml: ## @@ -2513,6 +2536,7 @@ components: - $ref: '#/components/schemas/RemovePropertiesUpdate' - $ref: '#/component

[I] Confusion about latest_schema_id in metadata_log_entries [iceberg]

2024-02-19 Thread via GitHub
oneonestar opened a new issue, #9758: URL: https://github.com/apache/iceberg/issues/9758 ### Apache Iceberg version main (development) ### Query engine None ### Please describe the bug 🐞 `latest_schema_id` in metadata_log_entries behaves strangely for ALTER

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1494943508 ## open-api/rest-catalog-open-api.yaml: ## @@ -532,6 +532,100 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/nam

Re: [PR] Core: Don't fail if catalog doesn't support views [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on PR #9754: URL: https://github.com/apache/iceberg/pull/9754#issuecomment-1953263483 Thanks, @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Core: Don't fail if catalog doesn't support views [iceberg]

2024-02-19 Thread via GitHub
rdblue merged PR #9754: URL: https://github.com/apache/iceberg/pull/9754 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[PR] Core: Fix EncryptingFileIO factory method. [iceberg]

2024-02-19 Thread via GitHub
rdblue opened a new pull request, #9757: URL: https://github.com/apache/iceberg/pull/9757 This implements @RussellSpitzer's suggestion to make `EncryptingFileIO.create` more defensive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495074341 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-19 Thread via GitHub
stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1495072935 ## format/spec.md: ## @@ -1237,17 +1237,36 @@ Content file (data or delete) is serialized as a JSON object according to the fo | **`equality-ids`** |`JSON lis

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495066733 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] API: Extend FileIO and add EncryptingFileIO. [iceberg]

2024-02-19 Thread via GitHub
rdblue merged PR #9592: URL: https://github.com/apache/iceberg/pull/9592 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] API: Extend FileIO and add EncryptingFileIO. [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on PR #9592: URL: https://github.com/apache/iceberg/pull/9592#issuecomment-1953240958 I'm going to merge this to unblock @ggershinsky and we can continue discussing the `create` method changes in a follow up. Thanks, @RussellSpitzer! -- This is an automated message from t

Re: [PR] API: Extend FileIO and add EncryptingFileIO. [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9592: URL: https://github.com/apache/iceberg/pull/9592#discussion_r1495065683 ## api/src/main/java/org/apache/iceberg/io/FileIO.java: ## @@ -42,6 +46,30 @@ default InputFile newInputFile(String path, long length) { return newInputFile(path);

Re: [PR] API: Extend FileIO and add EncryptingFileIO. [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9592: URL: https://github.com/apache/iceberg/pull/9592#discussion_r1495063672 ## api/src/main/java/org/apache/iceberg/encryption/EncryptingFileIO.java: ## @@ -0,0 +1,214 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

[PR] Build: Bump mkdocs-material from 9.5.9 to 9.5.10 [iceberg-python]

2024-02-19 Thread via GitHub
dependabot[bot] opened a new pull request, #448: URL: https://github.com/apache/iceberg-python/pull/448 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.9 to 9.5.10. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkd

[PR] Build: Bump moto from 5.0.1 to 5.0.2 [iceberg-python]

2024-02-19 Thread via GitHub
dependabot[bot] opened a new pull request, #447: URL: https://github.com/apache/iceberg-python/pull/447 Bumps [moto](https://github.com/getmoto/moto) from 5.0.1 to 5.0.2. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog. 5.0.

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1495060431 ## format/spec.md: ## @@ -1237,17 +1237,36 @@ Content file (data or delete) is serialized as a JSON object according to the fo | **`equality-ids`** |`JSON list o

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1495060431 ## format/spec.md: ## @@ -1237,17 +1237,36 @@ Content file (data or delete) is serialized as a JSON object according to the fo | **`equality-ids`** |`JSON list o

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-19 Thread via GitHub
jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1495060022 ## open-api/rest-catalog-open-api.yaml: ## @@ -1482,6 +1490,24 @@ components: explode: false example: "vended-credentials,remote-signing" +page-t

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-19 Thread via GitHub
jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1495059863 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: +

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-19 Thread via GitHub
jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1495059509 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: +

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495058736 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-19 Thread via GitHub
jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1495058516 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: +

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495056159 ## open-api/rest-catalog-open-api.yaml: ## @@ -2513,6 +2536,7 @@ components: - $ref: '#/components/schemas/RemovePropertiesUpdate' - $ref: '#/compo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495053841 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495048045 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495046660 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495047336 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495046660 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495046019 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495045393 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495041325 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495027556 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1495027555 ## open-api/rest-catalog-open-api.yaml: ## @@ -532,6 +532,100 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-19 Thread via GitHub
rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1494940519 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1616,12 @@ components: type: string example: [ "accounting", "tax" ] +NextPageToken: +

Re: [I] branch schema affected by main table schema [iceberg]

2024-02-19 Thread via GitHub
namrathamyske commented on issue #9737: URL: https://github.com/apache/iceberg/issues/9737#issuecomment-1953177938 When fetching data the schema which was associated with branch should be issued not table. But for operations like cherry pick from branch to main, it should resolve conflicts

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495022292 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495022292 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [I] refactor: Rename `ManifestListEntry`, `ManifestEntry` to make them more similar to java/python implementation. [iceberg-rust]

2024-02-19 Thread via GitHub
s-akhtar-baig commented on issue #145: URL: https://github.com/apache/iceberg-rust/issues/145#issuecomment-1953158388 Create changes in https://github.com/apache/iceberg-rust/pull/204. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Follow naming convention from Iceberg's Java and Python implementations [iceberg-rust]

2024-02-19 Thread via GitHub
s-akhtar-baig commented on PR #204: URL: https://github.com/apache/iceberg-rust/pull/204#issuecomment-1953157832 @liurenjie1024, I have refactored the code. Please take a look when you get a chance. Thanks! -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495010394 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495010775 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495010074 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495009376 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495009221 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495007496 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495007953 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,97 @@ components: type: integer format: int64 +TypeValue: + oneOf: +

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495007496 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495006843 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1495005890 ## open-api/rest-catalog-open-api.yaml: ## @@ -2513,6 +2536,7 @@ components: - $ref: '#/components/schemas/RemovePropertiesUpdate' - $ref: '#/component

Re: [PR] API: Extend FileIO and add EncryptingFileIO. [iceberg]

2024-02-19 Thread via GitHub
RussellSpitzer commented on code in PR #9592: URL: https://github.com/apache/iceberg/pull/9592#discussion_r1495000442 ## api/src/main/java/org/apache/iceberg/encryption/EncryptingFileIO.java: ## @@ -0,0 +1,214 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-19 Thread via GitHub
rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1494994843 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,97 @@ components: type: integer format: int64 +TypeValue: + oneOf: +

  1   2   3   >