[I] Empty iceberg table created with PyIceberg in AWS Glue misses location and schema [iceberg-python]

2024-02-16 Thread via GitHub
zartstrom opened a new issue, #435: URL: https://github.com/apache/iceberg-python/issues/435 ### Apache Iceberg version None ### Please describe the bug 🐞 I want to create an empty [Iceberg](https://iceberg.apache.org/) table with [PyIceberg](https://py.iceberg.apache.or

Re: [I] JdbcCatalog createNamespace is not supported [iceberg]

2024-02-16 Thread via GitHub
jbonofre commented on issue #2825: URL: https://github.com/apache/iceberg/issues/2825#issuecomment-1947987393 @nastra I don't have permission to close this issue. Can you please close it ? Thanks ! -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [I] JdbcCatalog createNamespace is not supported [iceberg]

2024-02-16 Thread via GitHub
nastra closed issue #2825: JdbcCatalog createNamespace is not supported URL: https://github.com/apache/iceberg/issues/2825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-16 Thread via GitHub
jbonofre commented on PR #9487: URL: https://github.com/apache/iceberg/pull/9487#issuecomment-1948053941 @nastra @ajantha-bhat @danielcweeks @rdblue I updated the PR by introducing `SchemaVersion`, but I keep the same `jdbc.add-view-support` property for users. Some notes: 1. I didn't

[PR] Send X-Iceberg-Access-Delegation header to signal support for vended credentials/remote signing [iceberg-python]

2024-02-16 Thread via GitHub
nastra opened a new pull request, #436: URL: https://github.com/apache/iceberg-python/pull/436 Clients can optionally send this header to signal which delegated access pattern it can support. At this point the iceberg-python client can support `vended-credentials` and `remote-signing`, thus

Re: [I] JdbcCatalog createNamespace is not supported [iceberg]

2024-02-16 Thread via GitHub
ajantha-bhat commented on issue #2825: URL: https://github.com/apache/iceberg/issues/2825#issuecomment-1948177084 fixed via https://github.com/apache/iceberg/pull/3275 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Support for REPLACE TABLE operation [iceberg-python]

2024-02-16 Thread via GitHub
Fokko commented on PR #433: URL: https://github.com/apache/iceberg-python/pull/433#issuecomment-1948180604 @anupam-saini Thanks for working on this. I'm not sure if the following API is where people would expect it: ```python with table.transaction() as transaction: transact

Re: [I] Cannot load a binary column of many rows via the `to_arrow` method. [iceberg-python]

2024-02-16 Thread via GitHub
Fokko commented on issue #344: URL: https://github.com/apache/iceberg-python/issues/344#issuecomment-1948187897 Closing this one, since #409 has been merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] check if table exist [iceberg-python]

2024-02-16 Thread via GitHub
Fokko commented on issue #406: URL: https://github.com/apache/iceberg-python/issues/406#issuecomment-1948187320 @Gowthami03B That would be great! 👍 One note here: > important if we just want to check that a table exists in a namespace. I'd argue this is the same as calling lis

Re: [I] Cannot load a binary column of many rows via the `to_arrow` method. [iceberg-python]

2024-02-16 Thread via GitHub
Fokko closed issue #344: Cannot load a binary column of many rows via the `to_arrow` method. URL: https://github.com/apache/iceberg-python/issues/344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Make InMemoryFileIO map shared access across instances [iceberg]

2024-02-16 Thread via GitHub
nastra merged PR #9722: URL: https://github.com/apache/iceberg/pull/9722 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump getdaft from 0.2.14 to 0.2.15 [iceberg-python]

2024-02-16 Thread via GitHub
Fokko merged PR #434: URL: https://github.com/apache/iceberg-python/pull/434 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[PR] Reuse commit-uuid as the write-uuid [iceberg-python]

2024-02-16 Thread via GitHub
Fokko opened a new pull request, #437: URL: https://github.com/apache/iceberg-python/pull/437 This makes is easier to trace which data files are being written by which version of the table. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] Minimum Requirement of Data File Name in Apache Iceberg? [iceberg-python]

2024-02-16 Thread via GitHub
Fokko commented on issue #429: URL: https://github.com/apache/iceberg-python/issues/429#issuecomment-1948312519 Thanks @syun64 for raising this! > The partition 'path' undeniably has a very important significance in Iceberg. However, the data file paths/names are all stored in in the

Re: [I] Empty iceberg table created with PyIceberg in AWS Glue misses location and schema [iceberg-python]

2024-02-16 Thread via GitHub
Fokko commented on issue #435: URL: https://github.com/apache/iceberg-python/issues/435#issuecomment-1948319083 @zartstrom Thanks for raising this! 👍 I think this has been fixed in https://github.com/apache/iceberg-python/pull/288. Can you try this by installing [the latest RC `pip

Re: [PR] Feat: Add fail_if_exists param to create_table [iceberg-python]

2024-02-16 Thread via GitHub
Fokko commented on PR #415: URL: https://github.com/apache/iceberg-python/pull/415#issuecomment-1948321931 @hussein-awala Thanks! You're right, and I think `create_table_if_not_exists` would be a cleaner solution, see #406. -- This is an automated message from the Apache Git Service. To

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-16 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1492409536 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcTableOperations.java: ## @@ -182,18 +177,23 @@ private void createTable(String newMetadataLocation) throws SQLExceptio

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-16 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1492410874 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java: ## @@ -158,6 +161,92 @@ public void testInitialize() { jdbcCatalog.initialize("test_jdbc_catalog

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-16 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1492412706 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -25,31 +25,62 @@ import java.util.Map; import java.util.Properties; import java.util.Set; +import o

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-16 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1492427645 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -81,85 +114,242 @@ final class JdbcUtil { + TABLE_NAME + ")" + ")"; -

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-02-16 Thread via GitHub
ZENOTME commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1492434129 ## crates/iceberg/src/io.rs: ## @@ -240,9 +241,9 @@ impl InputFile { } /// Trait for writing file. -pub trait FileWrite: AsyncWrite {} +pub trait FileWrite: Asyn

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-16 Thread via GitHub
jbonofre commented on PR #9487: URL: https://github.com/apache/iceberg/pull/9487#issuecomment-1948364274 @nastra yes, actually, I plan to improve `V0` / `V1` and more support in a separate PR introducing `JdbcAdapter` allowing us to adapt to different backend (like I did here https://githu

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-02-16 Thread via GitHub
Xuanwo commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1492439815 ## crates/iceberg/src/io.rs: ## @@ -240,9 +241,9 @@ impl InputFile { } /// Trait for writing file. -pub trait FileWrite: AsyncWrite {} +pub trait FileWrite: Async

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-02-16 Thread via GitHub
ZENOTME commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1492445165 ## crates/iceberg/src/writer/file_writer/location_generator.rs: ## @@ -0,0 +1,99 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contri

Re: [PR] Website: Add release schedule on the releases page [iceberg]

2024-02-16 Thread via GitHub
jbonofre commented on PR #9666: URL: https://github.com/apache/iceberg/pull/9666#issuecomment-1948376773 @bitsondatadev @Fokko thoughts about this PR ? Thanks ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-02-16 Thread via GitHub
ZENOTME commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1492446872 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -0,0 +1,385 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribut

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-02-16 Thread via GitHub
ZENOTME commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1492447476 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -0,0 +1,385 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribut

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-02-16 Thread via GitHub
ZENOTME commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1492450927 ## crates/iceberg/src/writer/file_writer/track_writer.rs: ## @@ -0,0 +1,72 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-02-16 Thread via GitHub
ZENOTME commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1492447476 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -0,0 +1,385 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribut

Re: [PR] feat: complete the basic file scan [iceberg-rust]

2024-02-16 Thread via GitHub
ZENOTME commented on PR #199: URL: https://github.com/apache/iceberg-rust/pull/199#issuecomment-1948677398 For now, it just a draft for changed interface. I will complete them if this interface looks well. -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Core, Spark: Remove dangling deletes as part of RewriteDataFilesAction [iceberg]

2024-02-16 Thread via GitHub
manuzhang commented on code in PR #9724: URL: https://github.com/apache/iceberg/pull/9724#discussion_r1492660971 ## api/src/main/java/org/apache/iceberg/actions/RewriteDataFiles.java: ## @@ -99,6 +99,20 @@ public interface RewriteDataFiles boolean USE_STARTING_SEQUENCE_NUMB

[PR] feat: complete the basic file scan [iceberg-rust]

2024-02-16 Thread via GitHub
ZENOTME opened a new pull request, #199: URL: https://github.com/apache/iceberg-rust/pull/199 This PR aims to complete #125. After #66, we prepared the plan file(metadata) for the scan. This PR will uses these plan file to construct the stream for read record batch. This PR also cha

Re: [PR] Core, Spark: Remove dangling deletes as part of RewriteDataFilesAction [iceberg]

2024-02-16 Thread via GitHub
manuzhang commented on code in PR #9724: URL: https://github.com/apache/iceberg/pull/9724#discussion_r1492660971 ## api/src/main/java/org/apache/iceberg/actions/RewriteDataFiles.java: ## @@ -99,6 +99,20 @@ public interface RewriteDataFiles boolean USE_STARTING_SEQUENCE_NUMB

Re: [PR] Spark 3.5: Fix testDeleteFileThenMetadataDelete failure due to table not refreshed [iceberg]

2024-02-16 Thread via GitHub
manuzhang commented on code in PR #9551: URL: https://github.com/apache/iceberg/pull/9551#discussion_r1492670868 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java: ## @@ -166,6 +166,28 @@ public static Object[][

Re: [PR] Pass input file into iterators to get the file name [iceberg]

2024-02-16 Thread via GitHub
RussellSpitzer commented on code in PR #9691: URL: https://github.com/apache/iceberg/pull/9691#discussion_r1492700348 ## core/src/main/java/org/apache/iceberg/avro/AvroIterable.java: ## @@ -150,7 +153,7 @@ public D next(D reuse) { try { return reader.next(reuse);

Re: [PR] Pass input file into iterators to get the file name [iceberg]

2024-02-16 Thread via GitHub
RussellSpitzer commented on code in PR #9691: URL: https://github.com/apache/iceberg/pull/9691#discussion_r1492701634 ## core/src/main/java/org/apache/iceberg/avro/AvroIterable.java: ## @@ -150,7 +153,7 @@ public D next(D reuse) { try { return reader.next(reuse);

[PR] Allow creating metadata tables based on SerializableTable instances [iceberg]

2024-02-16 Thread via GitHub
pvary opened a new pull request, #9735: URL: https://github.com/apache/iceberg/pull/9735 Currently it is not possible to create metadata tables above serialized tables. This could be useful as a general feature, and specifically it is needed if we want to create Flink jobs to run table main

Re: [PR] Pass input file into iterators to get the file name [iceberg]

2024-02-16 Thread via GitHub
RussellSpitzer commented on code in PR #9691: URL: https://github.com/apache/iceberg/pull/9691#discussion_r1492708091 ## core/src/main/java/org/apache/iceberg/avro/AvroIterable.java: ## @@ -150,7 +153,7 @@ public D next(D reuse) { try { return reader.next(reuse);

Re: [PR] feat(catalog): Send X-Iceberg-Access-Delegation header to signal support for vended credentials [iceberg-go]

2024-02-16 Thread via GitHub
Fokko merged PR #67: URL: https://github.com/apache/iceberg-go/pull/67 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache

Re: [PR] Smooth out release steps [iceberg-rust]

2024-02-16 Thread via GitHub
nastra merged PR #197: URL: https://github.com/apache/iceberg-rust/pull/197 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-16 Thread via GitHub
stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1492723658 ## core/src/main/java/org/apache/iceberg/BaseFileScanTaskParser.java: ## @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [I] Null values in metadata_log_entries [iceberg]

2024-02-16 Thread via GitHub
amogh-jahagirdar commented on issue #9723: URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1948850563 I did a little investigation as well, I'm not sure the original context on why the snapshot log was being cleared. But here are my thoughts: It does indeed some like w

Re: [PR] WIP: don't reset snapshotLog when replacing table [iceberg]

2024-02-16 Thread via GitHub
amogh-jahagirdar commented on code in PR #9732: URL: https://github.com/apache/iceberg/pull/9732#discussion_r1492738591 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1246,7 +1246,6 @@ public Builder setRef(String name, SnapshotRef ref) { public Builder

Re: [PR] WIP: don't reset snapshotLog when replacing table [iceberg]

2024-02-16 Thread via GitHub
amogh-jahagirdar commented on code in PR #9732: URL: https://github.com/apache/iceberg/pull/9732#discussion_r1492738591 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1246,7 +1246,6 @@ public Builder setRef(String name, SnapshotRef ref) { public Builder

Re: [I] Updating a property map in a iceberg table [iceberg]

2024-02-16 Thread via GitHub
namrathamyske closed issue #9659: Updating a property map in a iceberg table URL: https://github.com/apache/iceberg/issues/9659 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[I] branch schema affected by main table schema [iceberg]

2024-02-16 Thread via GitHub
namrathamyske opened a new issue, #9737: URL: https://github.com/apache/iceberg/issues/9737 ### Apache Iceberg version main (development) ### Query engine None ### Please describe the bug 🐞 regarding this PR: https://github.com/apache/iceberg/pull/9131 - th

Re: [PR] Core: Only write view history when currentVersionId changes [iceberg]

2024-02-16 Thread via GitHub
rdblue commented on code in PR #9725: URL: https://github.com/apache/iceberg/pull/9725#discussion_r1492788744 ## core/src/test/java/org/apache/iceberg/view/TestViewMetadata.java: ## @@ -63,8 +62,11 @@ public void testExpiration() { ViewVersion v2 = newViewVersion(2, "select

Re: [PR] Core: Properly suppress historical snapshots when building TableMetadata with suppressHistoricalSnapshots() [iceberg]

2024-02-16 Thread via GitHub
rdblue commented on PR #9234: URL: https://github.com/apache/iceberg/pull/9234#issuecomment-1948966275 Thanks, @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-16 Thread via GitHub
rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1492792266 ## open-api/rest-catalog-open-api.yaml: ## @@ -1482,6 +1490,33 @@ components: explode: false example: "vended-credentials,remote-signing" +page-toke

Re: [PR] Core: Properly suppress historical snapshots when building TableMetadata with suppressHistoricalSnapshots() [iceberg]

2024-02-16 Thread via GitHub
rdblue merged PR #9234: URL: https://github.com/apache/iceberg/pull/9234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Allow creating metadata tables based on SerializableTable instances [iceberg]

2024-02-16 Thread via GitHub
szehon-ho commented on code in PR #9735: URL: https://github.com/apache/iceberg/pull/9735#discussion_r1492792745 ## core/src/main/java/org/apache/iceberg/SerializableTable.java: ## @@ -388,6 +388,12 @@ public Transaction newTransaction() { throw new UnsupportedOperationExce

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-16 Thread via GitHub
rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1492798196 ## open-api/rest-catalog-open-api.yaml: ## @@ -1482,6 +1490,33 @@ components: explode: false example: "vended-credentials,remote-signing" +page-toke

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-16 Thread via GitHub
rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1492800015 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1616,12 @@ components: type: string example: [ "accounting", "tax" ] +NextPageToken: +

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-16 Thread via GitHub
jackye1995 commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1492816128 ## core/src/main/java/org/apache/iceberg/FileScanTaskParser.java: ## @@ -21,116 +21,84 @@ import com.fasterxml.jackson.core.JsonGenerator; import com.fasterxml.jac

[I] com.esotericsoftware.kryo.KryoException: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.nio.ByteBuffer [iceberg]

2024-02-16 Thread via GitHub
adevore3 opened a new issue, #9738: URL: https://github.com/apache/iceberg/issues/9738 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 We were upgrading to spark 3.4.1 when we ran into this issue. Currently running on spa

Re: [PR] WIP: don't reset snapshotLog when replacing table [iceberg]

2024-02-16 Thread via GitHub
rdblue commented on code in PR #9732: URL: https://github.com/apache/iceberg/pull/9732#discussion_r1492829614 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1246,7 +1246,6 @@ public Builder setRef(String name, SnapshotRef ref) { public Builder removeRef

Re: [I] com.esotericsoftware.kryo.KryoException: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.nio.ByteBuffer [iceberg]

2024-02-16 Thread via GitHub
adevore3 commented on issue #9738: URL: https://github.com/apache/iceberg/issues/9738#issuecomment-1949035999 I forgot to mention I did try to register KryoSerializer folowing this SOF: https://stackoverflow.com/questions/52562671/registering-classes-with-kryo-via-sparksession-in-spark-2 do

Re: [PR] Allow creating metadata tables based on SerializableTable instances [iceberg]

2024-02-16 Thread via GitHub
pvary commented on code in PR #9735: URL: https://github.com/apache/iceberg/pull/9735#discussion_r1492831999 ## core/src/test/java/org/apache/iceberg/hadoop/TestTableSerialization.java: ## @@ -153,6 +163,22 @@ public void testSerializableMetadataTablesPlanning() throws IOExcept

Re: [PR] Allow creating metadata tables based on SerializableTable instances [iceberg]

2024-02-16 Thread via GitHub
pvary commented on code in PR #9735: URL: https://github.com/apache/iceberg/pull/9735#discussion_r1492832463 ## core/src/main/java/org/apache/iceberg/MetadataTableUtils.java: ## @@ -32,9 +32,16 @@ public static boolean hasMetadataTableName(TableIdentifier identifier) { publi

Re: [PR] Allow creating metadata tables based on SerializableTable instances [iceberg]

2024-02-16 Thread via GitHub
pvary commented on code in PR #9735: URL: https://github.com/apache/iceberg/pull/9735#discussion_r149288 ## core/src/main/java/org/apache/iceberg/MetadataTableUtils.java: ## @@ -32,9 +32,16 @@ public static boolean hasMetadataTableName(TableIdentifier identifier) { publi

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-16 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1492845781 ## open-api/rest-catalog-open-api.py: ## @@ -905,6 +973,28 @@ class CreateTableRequest(BaseModel): properties: Optional[Dict[str, str]] = None +class PlanConte

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-16 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1492846298 ## open-api/rest-catalog-open-api.py: ## @@ -905,6 +973,28 @@ class CreateTableRequest(BaseModel): properties: Optional[Dict[str, str]] = None +class PlanConte

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-16 Thread via GitHub
emkornfield commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1492846630 ## format/spec.md: ## @@ -1237,17 +1237,36 @@ Content file (data or delete) is serialized as a JSON object according to the fo | **`equality-ids`** |`JSON l

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-16 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1492847929 ## open-api/rest-catalog-open-api.py: ## @@ -905,6 +973,28 @@ class CreateTableRequest(BaseModel): properties: Optional[Dict[str, str]] = None +class PlanConte

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-16 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1492851000 ## open-api/rest-catalog-open-api.yaml: ## @@ -532,6 +532,100 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-16 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1492856785 ## open-api/rest-catalog-open-api.yaml: ## @@ -532,6 +532,100 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

[I] Spec inconsistency: partition_spec_id column in ManifestList vs. partition_specs in metadata.json [iceberg]

2024-02-16 Thread via GitHub
JFinis opened a new issue, #9739: URL: https://github.com/apache/iceberg/issues/9739 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 The spec is inconsistent with respect to the `partition_spec_id` column. Here the spec n

Re: [I] InMemoryCatalog's FiloIO in memory map isn't persistent in RestCatalog [iceberg]

2024-02-16 Thread via GitHub
jackye1995 closed issue #9604: InMemoryCatalog's FiloIO in memory map isn't persistent in RestCatalog URL: https://github.com/apache/iceberg/issues/9604 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] rewrite_data_files procedure fails with Premature end of Content-Length when using S3 client [iceberg]

2024-02-16 Thread via GitHub
paulpaul1076 commented on issue #9679: URL: https://github.com/apache/iceberg/issues/9679#issuecomment-1949240582 @nastra unfortunately this doesn't seem to be the only reason for the content-length exception. We now discovered that it still fails, even though I stopped using the direct str

[I] Spec is unclear w.r.t. optional fields in field_summary [iceberg]

2024-02-16 Thread via GitHub
JFinis opened a new issue, #9740: URL: https://github.com/apache/iceberg/issues/9740 ### Apache Iceberg version 1.4.3 (latest release) ### Query engine None ### Please describe the bug 🐞 I'm referring to the definition of `field_summary`, which is as follows

Re: [I] Minimum Requirement of Data File Name in Apache Iceberg? [iceberg-python]

2024-02-16 Thread via GitHub
syun64 commented on issue #429: URL: https://github.com/apache/iceberg-python/issues/429#issuecomment-1949301152 I like the idea! This is definitely helpful, and I think this opens up a lot more options when looking to use other engines in parallelizing writes. -- This is an automated mes

Re: [I] Minimum Requirement of Data File Name in Apache Iceberg? [iceberg-python]

2024-02-16 Thread via GitHub
syun64 closed issue #429: Minimum Requirement of Data File Name in Apache Iceberg? URL: https://github.com/apache/iceberg-python/issues/429 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-16 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1492973284 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,211 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-16 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1492976199 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,211 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-16 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1492979077 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,211 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-16 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1492979601 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,211 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-16 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1492980261 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,211 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-16 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1492981360 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,211 @@ components: type: integer format: int64 +BooleanTypeValue: + type

[I] "CREATE TABLE (REPLACE TABLE) ... AS SELECT" Support [iceberg-python]

2024-02-16 Thread via GitHub
syun64 opened a new issue, #438: URL: https://github.com/apache/iceberg-python/issues/438 ### Feature Request / Improvement In Spark SQL, we have the ability to combine the table overwrite using AS SELECT statement, with create_table or replace_table, as an atomic operation. Do

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-16 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1493011625 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,211 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-16 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1493016912 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,211 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-16 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1493017691 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,211 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-16 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1493017978 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,211 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-16 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1493018795 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,211 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

2024-02-16 Thread via GitHub
geruh commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1493023248 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,211 @@ components: type: integer format: int64 +BooleanTypeValue: + type: boo

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-16 Thread via GitHub
stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1493034865 ## core/src/main/java/org/apache/iceberg/FileScanTaskParser.java: ## @@ -21,116 +21,84 @@ import com.fasterxml.jackson.core.JsonGenerator; import com.fasterxml.jack

Re: [I] Getting the following error when using from spark thrift server [iceberg]

2024-02-16 Thread via GitHub
tkent commented on issue #3010: URL: https://github.com/apache/iceberg/issues/3010#issuecomment-1949418909 :wave: Recording one solution to this error. I ran into this using an Azure databricks cluster and the underlying error was what @RussellSpitzer noted. However, the runtime JAR for ice

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-16 Thread via GitHub
stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1493037575 ## format/spec.md: ## @@ -1237,17 +1237,36 @@ Content file (data or delete) is serialized as a JSON object according to the fo | **`equality-ids`** |`JSON lis

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-16 Thread via GitHub
stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1493037575 ## format/spec.md: ## @@ -1237,17 +1237,36 @@ Content file (data or delete) is serialized as a JSON object according to the fo | **`equality-ids`** |`JSON lis

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-16 Thread via GitHub
stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1493034865 ## core/src/main/java/org/apache/iceberg/FileScanTaskParser.java: ## @@ -21,116 +21,84 @@ import com.fasterxml.jackson.core.JsonGenerator; import com.fasterxml.jack

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-16 Thread via GitHub
stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1493034865 ## core/src/main/java/org/apache/iceberg/FileScanTaskParser.java: ## @@ -21,116 +21,84 @@ import com.fasterxml.jackson.core.JsonGenerator; import com.fasterxml.jack

Re: [PR] Send X-Iceberg-Access-Delegation header to signal support for vended credentials/remote signing [iceberg-python]

2024-02-16 Thread via GitHub
danielcweeks commented on code in PR #436: URL: https://github.com/apache/iceberg-python/pull/436#discussion_r1493048345 ## pyiceberg/catalog/rest.py: ## @@ -237,6 +237,7 @@ def _create_session(self) -> Session: session.headers["Content-type"] = "application/json"

[PR] Build: Bump pytest from 7.4.4 to 8.0.1 [iceberg-python]

2024-02-16 Thread via GitHub
dependabot[bot] opened a new pull request, #439: URL: https://github.com/apache/iceberg-python/pull/439 Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.4.4 to 8.0.1. Release notes Sourced from https://github.com/pytest-dev/pytest/releases";>pytest's releases. 8.0

Re: [PR] Build: Bump pytest from 7.4.4 to 8.0.0 [iceberg-python]

2024-02-16 Thread via GitHub
dependabot[bot] commented on PR #319: URL: https://github.com/apache/iceberg-python/pull/319#issuecomment-1949437791 Superseded by #439. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Build: Bump pytest from 7.4.4 to 8.0.0 [iceberg-python]

2024-02-16 Thread via GitHub
dependabot[bot] closed pull request #319: Build: Bump pytest from 7.4.4 to 8.0.0 URL: https://github.com/apache/iceberg-python/pull/319 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-16 Thread via GitHub
stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1493034865 ## core/src/main/java/org/apache/iceberg/FileScanTaskParser.java: ## @@ -21,116 +21,84 @@ import com.fasterxml.jackson.core.JsonGenerator; import com.fasterxml.jack

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

2024-02-16 Thread via GitHub
stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1493050495 ## core/src/main/java/org/apache/iceberg/FileScanTaskParser.java: ## @@ -21,116 +21,84 @@ import com.fasterxml.jackson.core.JsonGenerator; import com.fasterxml.jack

Re: [PR] Support for REPLACE TABLE operation [iceberg-python]

2024-02-16 Thread via GitHub
anupam-saini commented on code in PR #433: URL: https://github.com/apache/iceberg-python/pull/433#discussion_r1493051291 ## pyiceberg/schema.py: ## @@ -1213,34 +1213,50 @@ def build_position_accessors(schema_or_type: Union[Schema, IcebergType]) -> Dict return visit(schema_

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-02-16 Thread via GitHub
RussellSpitzer commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1493050575 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -53,13 +54,15 @@ import org.slf4j.LoggerFactory; /** - * TableOperations im

Re: [PR] Support for REPLACE TABLE operation [iceberg-python]

2024-02-16 Thread via GitHub
syun64 commented on code in PR #433: URL: https://github.com/apache/iceberg-python/pull/433#discussion_r1493052840 ## pyiceberg/schema.py: ## @@ -1213,34 +1213,50 @@ def build_position_accessors(schema_or_type: Union[Schema, IcebergType]) -> Dict return visit(schema_or_typ

  1   2   >