Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1492051707 ## open-api/rest-catalog-open-api.yaml: ## @@ -1482,6 +1490,33 @@ components: explode: false example: "vended-credentials,remote-signing" +page-token

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1492051707 ## open-api/rest-catalog-open-api.yaml: ## @@ -1482,6 +1490,33 @@ components: explode: false example: "vended-credentials,remote-signing" +page-token

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1492044979 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1616,12 @@ components: type: string example: [ "accounting", "tax" ] +NextPageToken: +

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1492042191 ## open-api/rest-catalog-open-api.yaml: ## @@ -1482,6 +1490,33 @@ components: explode: false example: "vended-credentials,remote-signing" +page-token

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1492039126 ## open-api/rest-catalog-open-api.yaml: ## @@ -1482,6 +1490,33 @@ components: explode: false example: "vended-credentials,remote-signing" +page-token

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-15 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1491864036 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-15 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1491906201 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [I] GCS support? [iceberg-go]

2024-02-15 Thread via GitHub
thorfour commented on issue #60: URL: https://github.com/apache/iceberg-go/issues/60#issuecomment-1947678160 @zeroshade I took a stab at what it might look like in #66 let me know what you think. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] Make InMemoryFileIO map shared access across instances [iceberg]

2024-02-15 Thread via GitHub
amogh-jahagirdar commented on code in PR #9722: URL: https://github.com/apache/iceberg/pull/9722#discussion_r1491891341 ## core/src/test/java/org/apache/iceberg/inmemory/TestInMemoryFileIO.java: ## @@ -108,4 +111,18 @@ public void testOverwriteBeforeAndAfterClose() throws IOExc

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-15 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1491864036 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-15 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1491847698 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [I] Cannot read Parquet maps with NameMapping [iceberg]

2024-02-15 Thread via GitHub
github-actions[bot] commented on issue #1396: URL: https://github.com/apache/iceberg/issues/1396#issuecomment-1947539313 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Support writing to a branch [iceberg-python]

2024-02-15 Thread via GitHub
Gowthami03B commented on issue #306: URL: https://github.com/apache/iceberg-python/issues/306#issuecomment-1947458069 I had an offline chat with @kevinjqliu , I shall work on this to build off of the PR created by kevin. -- This is an automated message from the Apache Git Service. To resp

Re: [I] check if table exist [iceberg-python]

2024-02-15 Thread via GitHub
Gowthami03B commented on issue #406: URL: https://github.com/apache/iceberg-python/issues/406#issuecomment-1947454813 Can I take a stab at the `table_exists` method proposed here? @Fokko @djouallah @syun64 -- This is an automated message from the Apache Git Service. To respond to the mes

[PR] Build: Bump getdaft from 0.2.14 to 0.2.15 [iceberg-python]

2024-02-15 Thread via GitHub
dependabot[bot] opened a new pull request, #434: URL: https://github.com/apache/iceberg-python/pull/434 Bumps [getdaft](https://github.com/Eventual-Inc/Daft) from 0.2.14 to 0.2.15. Release notes Sourced from https://github.com/Eventual-Inc/Daft/releases";>getdaft's releases.

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

2024-02-15 Thread via GitHub
mrcnc commented on PR #8976: URL: https://github.com/apache/iceberg/pull/8976#issuecomment-1947419714 LGTM - Thanks for adding this! I look forward to using it 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Make InMemoryFileIO map shared access across instances [iceberg]

2024-02-15 Thread via GitHub
jackye1995 commented on code in PR #9722: URL: https://github.com/apache/iceberg/pull/9722#discussion_r1491698706 ## core/src/test/java/org/apache/iceberg/inmemory/TestInMemoryFileIO.java: ## @@ -108,4 +111,18 @@ public void testOverwriteBeforeAndAfterClose() throws IOException

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-15 Thread via GitHub
jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1491665454 ## open-api/rest-catalog-open-api.yaml: ## @@ -1482,6 +1490,33 @@ components: explode: false example: "vended-credentials,remote-signing" +page-t

Re: [PR] Support for REPLACE TABLE operation [iceberg-python]

2024-02-15 Thread via GitHub
syun64 commented on code in PR #433: URL: https://github.com/apache/iceberg-python/pull/433#discussion_r1491658399 ## pyiceberg/schema.py: ## @@ -1213,34 +1213,50 @@ def build_position_accessors(schema_or_type: Union[Schema, IcebergType]) -> Dict return visit(schema_or_typ

[PR] Support for REPLACE TABLE operation [iceberg-python]

2024-02-15 Thread via GitHub
anupam-saini opened a new pull request, #433: URL: https://github.com/apache/iceberg-python/pull/433 Closes #281 Proposed API: ``` with table.transaction() as transaction: transaction.replace_table_with(new_table) ``` TODO: - [ ] Update schema - [ ] Update sor

Re: [PR] Make InMemoryFileIO map shared access across instances [iceberg]

2024-02-15 Thread via GitHub
jackye1995 commented on code in PR #9722: URL: https://github.com/apache/iceberg/pull/9722#discussion_r1491519229 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryFileIO.java: ## @@ -28,22 +28,22 @@ public class InMemoryFileIO implements FileIO { - private final M

Re: [PR] Core: Add property to prevent loss of view representation when replacing a view [iceberg]

2024-02-15 Thread via GitHub
rdblue commented on PR #9620: URL: https://github.com/apache/iceberg/pull/9620#issuecomment-1947039792 Merged! Thanks for getting this done for the 1.5 release, @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Core: Add property to prevent loss of view representation when replacing a view [iceberg]

2024-02-15 Thread via GitHub
rdblue merged PR #9620: URL: https://github.com/apache/iceberg/pull/9620 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] AWS: Add url-connection-client to aws-bundle [iceberg]

2024-02-15 Thread via GitHub
94Sip commented on PR #9685: URL: https://github.com/apache/iceberg/pull/9685#issuecomment-1946981814 I have used v1.4.3 with Glue jobs and simply referenced the url connection client in the --extra-jars job parameter and had no problems, and I used the `http-client.type=urlconnection` Spar

Re: [PR] Make InMemoryFileIO map shared access across instances [iceberg]

2024-02-15 Thread via GitHub
jackye1995 commented on code in PR #9722: URL: https://github.com/apache/iceberg/pull/9722#discussion_r1491471174 ## core/src/test/java/org/apache/iceberg/inmemory/TestInMemoryFileIO.java: ## @@ -21,17 +21,18 @@ import java.io.IOException; import java.io.InputStream; import j

Re: [I] RewriteManifest with more options [iceberg]

2024-02-15 Thread via GitHub
zachdisc commented on issue #9615: URL: https://github.com/apache/iceberg/issues/9615#issuecomment-1946943705 Okay first cut at the simple version is in PR above -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-15 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1491465529 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -90,13 +93,45 @@ public class RewriteManifestsSparkAction

[PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-15 Thread via GitHub
zachdisc opened a new pull request, #9731: URL: https://github.com/apache/iceberg/pull/9731 ## What This adds a simple `sort` method to the `RewriteManifests` spark action which lets user specify the partition column order to consider when grouping manifests. Illustration:

Re: [I] Iceberg Glue - Timeouts (maybe others client side error cases) can result in missing metadata_location [iceberg]

2024-02-15 Thread via GitHub
kmurra commented on issue #9618: URL: https://github.com/apache/iceberg/issues/9618#issuecomment-1946863112 Adding a bit more results of investigation, server side timeouts are not retried. They are assigned a 4xx class error code by the SDK. However, a server side timeout may still resul

Re: [PR] Make InMemoryFileIO map shared access across instances [iceberg]

2024-02-15 Thread via GitHub
rahil-c commented on code in PR #9722: URL: https://github.com/apache/iceberg/pull/9722#discussion_r1491407208 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryFileIO.java: ## @@ -28,7 +28,7 @@ public class InMemoryFileIO implements FileIO { - private final Map in

Re: [PR] Make InMemoryFileIO map shared access across instances [iceberg]

2024-02-15 Thread via GitHub
rahil-c commented on code in PR #9722: URL: https://github.com/apache/iceberg/pull/9722#discussion_r1491407208 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryFileIO.java: ## @@ -28,7 +28,7 @@ public class InMemoryFileIO implements FileIO { - private final Map in

Re: [I] Null values in metadata_log_entries [iceberg]

2024-02-15 Thread via GitHub
nastra commented on issue #9723: URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1946724217 I'll take a closer look at this tomorrow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Core, Spark: Remove dangling deletes as part of RewriteDataFilesAction [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9724: URL: https://github.com/apache/iceberg/pull/9724#discussion_r1491387482 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java: ## @@ -103,17 +108,21 @@ import org.apache.iceberg.types.Conversions

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-15 Thread via GitHub
jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1491386167 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -81,6 +87,7 @@ public class JdbcCatalog extends BaseMetastoreCatalog private final Function, J

[PR] docstring: Fix missing commit [iceberg-python]

2024-02-15 Thread via GitHub
Fokko opened a new pull request, #432: URL: https://github.com/apache/iceberg-python/pull/432 While going through some code, I noticed that these are missing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Core: Only write view history when currentVersionId changes [iceberg]

2024-02-15 Thread via GitHub
amogh-jahagirdar commented on code in PR #9725: URL: https://github.com/apache/iceberg/pull/9725#discussion_r1491191209 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -504,6 +517,11 @@ static List updateHistory(List history, Set< } } +

Re: [PR] Core: Only write view history when currentVersionId changes [iceberg]

2024-02-15 Thread via GitHub
amogh-jahagirdar commented on code in PR #9725: URL: https://github.com/apache/iceberg/pull/9725#discussion_r1491185660 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -504,6 +517,11 @@ static List updateHistory(List history, Set< } } +

[PR] [DRAFT] Fix ASF links on homepage to comply with trademark [iceberg]

2024-02-15 Thread via GitHub
bitsondatadev opened a new pull request, #9729: URL: https://github.com/apache/iceberg/pull/9729 Working with @munabedan to get the css styleing right. Don't merge. Fixes: #9712 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] Fix the REST spec version [iceberg-rust]

2024-02-15 Thread via GitHub
Fokko opened a new pull request, #198: URL: https://github.com/apache/iceberg-rust/pull/198 This number indicates from which release the code was generated. For example, currently new endpoints are added to the spec, but they are not supported by iceberg-rust yet. -- This is an automated

[PR] Smooth out release steps [iceberg-rust]

2024-02-15 Thread via GitHub
Fokko opened a new pull request, #197: URL: https://github.com/apache/iceberg-rust/pull/197 Couple of small things: - The license check failed because the `dist/*` files were there - Add `dist/*` to gitignore since we don't want to push these files to the repo - Make `scripts/re

Re: [PR] [WIP] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-15 Thread via GitHub
tomtongue commented on PR #9670: URL: https://github.com/apache/iceberg/pull/9670#issuecomment-1946049882 Thanks for the review @nastra . No, there's one thing that just I update snapshotIsolation tests with `Awaitility`. -- This is an automated message from the Apache Git Service. To res

Re: [PR] [WIP] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-15 Thread via GitHub
tomtongue commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1490970717 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1166,7 +1151,7 @@ public synchronized void testDeleteWithSnap

Re: [PR] Build: Bump junit from 5.10.1 to 5.10.2 [iceberg]

2024-02-15 Thread via GitHub
nastra commented on PR #9699: URL: https://github.com/apache/iceberg/pull/9699#issuecomment-1945930246 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Spec: add task-type field to JSON serialization of file scan task. add JSON serialization for StaticDataTask. [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1490876417 ## format/spec.md: ## @@ -1237,17 +1237,37 @@ Content file (data or delete) is serialized as a JSON object according to the fo | **`equality-ids`** |`JSON list o

Re: [PR] Spec: add task-type field to JSON serialization of file scan task. add JSON serialization for StaticDataTask. [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1490873382 ## format/spec.md: ## @@ -1237,17 +1237,37 @@ Content file (data or delete) is serialized as a JSON object according to the fo | **`equality-ids`** |`JSON list o

Re: [PR] Spec: add task-type field to JSON serialization of file scan task. add JSON serialization for StaticDataTask. [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1490871701 ## format/spec.md: ## @@ -1237,17 +1237,37 @@ Content file (data or delete) is serialized as a JSON object according to the fo | **`equality-ids`** |`JSON list o

Re: [PR] Make InMemoryFileIO map shared access across instances [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9722: URL: https://github.com/apache/iceberg/pull/9722#discussion_r1490842855 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryFileIO.java: ## @@ -28,7 +28,7 @@ public class InMemoryFileIO implements FileIO { - private final Map inM

Re: [PR] Core: Add property to prevent loss of view representation when replacing a view [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9620: URL: https://github.com/apache/iceberg/pull/9620#discussion_r1490819640 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -510,5 +529,29 @@ static List updateHistory(List history, Set< private Stream changes(Class

Re: [PR] Core: Only write view history when currentVersionId changes [iceberg]

2024-02-15 Thread via GitHub
nastra merged PR #9725: URL: https://github.com/apache/iceberg/pull/9725 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] [AWS] S3AsyncFileIO Client integration [iceberg]

2024-02-15 Thread via GitHub
CsengerG commented on PR #7318: URL: https://github.com/apache/iceberg/pull/7318#issuecomment-1945734929 Hi all! Looks like this PR got stuck but I would like to continue this effort. I tried checking it out locally and rebasing but that seems to be a bit complex (main is 1100+ commits ah

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-15 Thread via GitHub
jbonofre commented on PR #9487: URL: https://github.com/apache/iceberg/pull/9487#issuecomment-1945703456 @danielcweeks Thanks ! Yeah, I assumed we won't have new schema update, but you are right, it's possible to have new updates in the future. So let me update the PR with schema versioning

Re: [PR] [WIP] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1490703703 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestUpdate.java: ## @@ -790,7 +768,7 @@ public synchronized void testUpdateWithSnapshotI

Re: [PR] [WIP] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1490702148 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java: ## @@ -1691,7 +1666,7 @@ public synchronized void testMergeWithSnapshotI

Re: [PR] [WIP] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1490699690 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1166,7 +1151,7 @@ public synchronized void testDeleteWithSnapsho

Re: [PR] Core: Only write view history when currentVersionId changes [iceberg]

2024-02-15 Thread via GitHub
nastra commented on PR #9725: URL: https://github.com/apache/iceberg/pull/9725#issuecomment-1945601409 @amogh-jahagirdar @rdblue I've rolled back all changes around version expiration and updating the history. I've improved testing coverage around these so that it's clearer that the size of

Re: [PR] Core: Only write view history when currentVersionId changes [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9725: URL: https://github.com/apache/iceberg/pull/9725#discussion_r1490603721 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -479,21 +488,21 @@ public ViewMetadata build() { metadataLocation); } -stati

Re: [PR] Core: Only write view history when currentVersionId changes [iceberg]

2024-02-15 Thread via GitHub
nastra commented on code in PR #9725: URL: https://github.com/apache/iceberg/pull/9725#discussion_r1490598286 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -504,6 +517,11 @@ static List updateHistory(List history, Set< } } + // ke